Product Design

Hypothesis-Driven Validation

How to reduce risk and build better products

Hilary Hayes

March 22, 2018

The journey from product concept to business launch is risky, messy, and often unsuccessful. No matter how you slice it, risk is an inherent part of product development. And the more leading edge the technology, the riskier it’s likely to be.

“But I have a really great idea!”

You may think you have a great idea, but the odds are against you:

72% of all new products end up failing. — Simon-Kucher & Partners, 2014
42% of failed startups were unsuccessful because their product or service didn’t address a real market need. — CB Insights, 2017
The Ill-Fated Microsoft Zune

Given these staggering statistics, anything you can do to reduce risk — and in particular, to ensure you’re addressing a known market demand — is extremely valuable.

Fortunately, there aretools and approaches you can use to reduce the likelihood that your product will fail. By introducing hypothesis-driven validation into your process, you can ride the uncertainty inherent in your work toward much more favourable odds.

Hypothesis-driven validation can be broken down into seven steps:

  1. Identify your assumptions
  2. Reframe assumptions as “hypotheses”
  3. Rank them in order of importance
  4. Design appropriate tests
  5. Conduct the tests
  6. Synthesize your learnings
  7. Act

Hypothesis-driven validation sounds commonsensical enough. Yet most businesses tend to avoid these activities, preferring instead to minimize risk through planning, analysis, financial modelling, and the like. These activities can provide all sorts of structure and confidence, yet ultimately they don’t compare to the nitty-gritty of testing assumptions, collecting evidence, and direct observation, which allow for emergent findings. When it comes to uncovering opportunities (not just mitigating risk in the narrow sense), testing beats planning every time.

Let’s walk through these one by one.

1. Identify Your Assumptions

Every product concept is built on a pile of assumptions. Before doing anything else, try to list as many assumptions about your product as you possibly can, breaking them down into the simplest statements possible.

Since many of your assumptions are interconnected and hard to eke out, it pays to consider your product in terms of feasibility, desirability, and viability. These lenses will help you think about all the things you may be taking for granted.

Feasibility: Can the product be built with the current technologies? Can all of its features be built? Does it sync up with existing platforms out in the market in the way you want it to? Have you chosen the right technologies?

Desirability: Is your product solving a customer need? How is your product helping the user? Will people even want it? Why will they want it?

Viability: Will producing your product be time-, cost-, and resource-effective? Does it fit into your company vision? How about your business model? Are there similar products in the market? What are people willing to pay for your product, and how does that match up to your profit goals?

2. Reframe Assumptions as Hypotheses

Once you have a long list of assumptions, reframe them as “We believe that…” statements, or clear hypotheses. This helps expose them as subjective opinions still in need of proof rather than objective facts. Say, for example, that you were developing an app that automatically sends users a pair of trousers every month:

Your Assumption: Customers will be happy with the trousers they are mailed through the app.

Reframed as a Hypothesis: We believe that customers will be happy with the trousers they are mailed through the app.

3. Rank Hypotheses in Order of Priority

To determine which hypotheses merit the most attention for testing, consider how significant it would be if that hypothesis were proven false. Would it be a tiny hiccup, or would it mean the product couldn’t be made at all? Could the issue be solved with a small pivot, or would it spell ruin for your potential business?

Get your team together and write out all your hypotheses out on Post-its. Throw them up on a wall and vote on what would be most threatening to the product’s success if the assumption turned out to be false.

4. Design the Tests

Paying greatest attention to the high-risk hypotheses, consider what the most applicable test methods might be for each one. Here are some sample test types:

Quantitative: Surveys (these can be as informal as a Google Form or Twitter poll), data analysis (of existing products, industries, etc.), A/B testing of prototyped experiences.

Qualitative: Wireframes, proofs of concept, and other prototypes for user feedback; in person or remote user/stakeholder interviews; ethnographic research; experience prototyping; roleplaying.

In a table, match each hypothesis (listed in order of importance) with its appropriate test method. You may use more than one method per hypothesis.

Now, build prototypes, write survey questions, prepare interview questions — do whatever you need to do to effectively test your hypotheses.

When it comes to uncovering opportunities, testing beats planning every time.

5. Test

Following the test plan you’ve outlined, commence testing. Depending on the chosen test methods, here’s how that might play out:

Surveys: Send your survey link to participants in your user group.

Interviews: Reserve a space, print your interview questions, arrange for a note taker.

Representative Designs: Build wireframes and/or prototypes, reserve a space to test, print test plans.

Start by testing the riskiest hypotheses and refrain from moving on to solutioning or making product modifications until you have gone through them all.

6. Synthesize your Learnings

Once you’ve conducted all of your tests, making sure to capture all your data along the way, debrief with your team to go through the data, synthesize it, and capture learnings.

Some of your hypotheses will have been straight up invalidated while others will be proven true. But more often than not, the reality is more murky: your hypothesis might have been true, but not for the reason you thought; or your hypothesis wasn’t true, but in showing it to be false exposed a deeper truth that hadn’t been articulated.

The “synthesis” step can be messy, but it pays to take time and pour over your results carefully.

7. Act!

With your learnings in hand, you can finally return to your product concept and see what refinements, revisions, iterations, and research might be necessary to reduce risk.

And now, as with many agile processes, refactor, rinse, and repeat to further refine your direction.

We’ve had a lot of success with Hypothesis-Driven Validation at Connected Lab, allowing small teams to accomplish a great deal on short timelines because of the rigour this process instills in identifying assumptions, prioritizing hypotheses, and validating/invalidating those hypothesis through testing. The process helps not only narrow down and refine a concept, but actually generate new insights about users more broadly. I hope you experience success with it too.

Thanks for reading! (I assume you did. Get it?)

This post was written in collaboration with Connected Lab Product Manager Nate Archer.

Related Posts

Product I Love: Google Photos

From the CN Tower to beautiful nature shots, read why Solutions Engineer Kevin Bralten's favourite product is Google Photos!

At Connected we are obsessed with great products. We often talk about the great products we worked on and the ones we wished we worked on. We sat down with our Solutions Engineer, Kevin Bralten to ask him about his favourite product. Funnily enough, once you finish reading this post, you’ll understand how we found it ironic that he was wearing his Google Cloud sweatshirt during our interview.

Connector Spotlight: Jacky Li

Senior Product Designer, Eco-Friendly Citizen, Goalie to our Heart. Meet the one and only Jacky Li!

He shoots, but he doesn’t score past our hockey-loving designer, Jacky Li!

Top 12 Collision 2019 Talks Every Practitioner Should Check Out

If you're a Designer, Engineer, Product Manager, or Business Strategist, this guide is for you.

With so many great talks happening at this 3-day conference, we decided to narrow down the list to highlight some to look out for based on your practice.