Voice of the Customer in our Product Development Process

(Another blog post - like “Product Management in 5 Questions” I wrote a bit ago in service of promoting the new SaaS-era Product practice at CrowdFlower — which eventually exited via acquisition by Appen. A bit basic perhaps — the cool thing is how much easier it is to schedule and administer this stuff with tools like LoopPanel, Monterey.ai and ChatGPT — but makes it even less valid to avoid it!)

January 27, 2016

Late last year, we hosted the Rich Data Summit. Fundamentally, it was an industry event where we brought in some of the smartest speakers in the data science landscape to share their views on the present and future of the field. After announcing the Beta release of our new AI offering, my attention turned to our customer panel on ‘The State of Data Science’, where in a moment of shameless self-interest, I used the fact I was tabbed as moderator to frame that session as an opportunity to understand the voice of the Data Scientist. It was important that every CrowdFlower employee in that room heard how they work, what interests, motivates (and de-motivates) them and their teams.

For product people, sessions where you learn about your core users are incredibly valuable. But most would agree it is disappointingly hard to get these kinds of sessions scheduled. We often have to spend many hours of logistics and protocol development (and give away a few Amazon gift cards of high denominations), all just to get a small handful of 15-to-60-minute interviews done.

But it’s completely worth it.

Here’s why and how those sessions are essential to the product development process.

1. You get a user story about an actual user

If you’re not familiar with the Agile concept of a User Story, Broadcom’s Rally Software can help. But the TL;DR is that in our process, Product people ask Engineers for code that fulfills a need for an actual (type of) person.

We might describe those people as “a new visitor who is frustrated with our security requirements” for example, but that persona doesn’t feel real until you actually talk to one such person. They make the User Story real.

Of course, it’s important you make sure to control for Cognitive Bias problems deriving from the small sample size and interview format, but once you have, you can do a better job of describing your user to other people on your team and discovering their real needs, even beyond the ones they can articulate.

2. You learn whether our goal is being met

Good product management starts with a thoughtful definition of the goal behind the time and money you’re proposing to spend. The reason why would merit a whole separate post, but suffice it to say, when you ask people “what is it you want to accomplish?” and “how would you go about doing that?”, it becomes very clear whether what you’re building can solve the problem you thought it would.

3. You learn whether your goal is the right goal

A happy outcome of any interview about whether a goal can be achieved is that you can also learn whether that goal actually matters to the people you think it should matter to.

In a user research session where you’ve crafted behavioral questions about what someone wants to do and tries to do, a good listener can also hear what the subject is pointedly not saying, or what they’re glossing over because they think something else is more important. When you notice that happening, you get a more valuable initiative gift-wrapped for you to work on instead.

4. You often see patterns very quickly

While it can be painful to get one interview scheduled, thankfully, it doesn’t take many of them to see the most valuable themes are for the people you’re trying to understand.

Specifically in usability testing (where you’re trying to figure out how easily people can complete a task you’ve laid out in your product), noted researcher Jakob Nielsen quantified why you often only need 5 such sessions. When you spend the time to schedule these interviews, it’s usually good to mix in both discovery-related, open-ended questions and task-oriented hypothesis tests on a related product or prototype. Hearing about both the problem as well as the symptom gives a better view as to which level’s common themes are most important.

5. You put yourself in a position to get lucky

I can count at least 4 times in my career where we’ve conducted behavioral interviews and uncovered something surprising that ended up being more valuable than the whole initial project. At OpenTable, for example, we realized we didn’t need to spend millions to acquire and load restaurant photos for all of our listings in search results — because when asked, people consistently said they only truly cared about seeing them for an individual restaurant that already met other criteria.

At CrowdFlower we saw that users didn’t want or need lengthy onboarding videos, product tours or tutorials for building a job that we had considered creating. They just wanted to get their hands dirty and needed us only to organize our job templates into a couple easy-to-understand use-case categories.

So, as the folks at Lean Startup would say, when you’ve got an important problem to solve for your customers, get out of the building. Nag, beg, demand, and work hard to get in front of just a couple real people. It may not go easily or even the way you thought it would, but it’s always worth the time.

And if you’re one of those customers at CrowdFlower, I’d love to hear from you — and our Product team just might happen to have a highly-denominated Amazon Gift Card with your name on it.