Pivotal Cloud Foundry

2016

Pivotal Cloud Foundry (PCF) offers a platform-as-a-service to enterprise companies who wish to speed up software innovation. Application that run on PCF can be monitored and troubleshooted with a component called Metrics. Since Metrics is part of PCF, it has a level of integration unavailable to competitors. With it, richer insights into app performance can be attained.

Challenge

As a research minded designer joining a new team, I wanted to learn the history of the product. I quickly concluded that there was little research being conducted and, when it was, it had problems. It treated internal staff as customers, the level of rigor applied was low and neither raw data nor findings were well documented (or at all sometimes). The team and the product could benefit if more stringent standards were applied and made visible. 

1. Plan & Recruit

After talking to key people on the team, I wrote a plan recommending both a usability study to set a baseline of performance for the current product and generative interviews to learn about users in greater detail. The PM agreed to the usability test and, with his help, wrote a screener based on the characteristics of our intended audience. Recruiting is often the lengthiest phase of research and I wanted to get started quickly.

2. Script & Pilot

With the input of our developers, I wrote a test script with scenarios and tasks that were realistic and believable. I concentrated on the elements and interactions which supported the main feature set and the areas which had the most assumption built into them. Once it was ready, I piloted the script with an internal staff member who fit our criteria. This gave me a chance to revise the flow, wording and timing.

3. Interview & Analyze

We tested four users remotely using a functioning prototype. I worked with my design pair to analyze and synthesize the findings which we presented to the full team. We presented a prioritized list of changes as well as a list of design explorations as next steps. The PM converted our recommendations into developer stories and the explorations became epics in the design tracker. All new and revised work would be tested in a future study.

Results

I felt successful with the project's outcome, but it was a short lived feeling. There were many long term questions to address: Could the process be sped up? How could recruiting scale to serve multiple studies rather than just one at a time? Would dedicated researchers be a wise investment in people? What resources and training, if any, should be offered? 

We continue to work through these questions as the level of rigor increases, additional research methods are put to use, and the organization begins to see the value in the work. One recent sign of success is an upcoming product manager division offsite meeting where research is a major area of discussion. I feel I've had a part in growing the influence of research and those conducting it at a company that historically has not placed much emphasis on it.