Designing Smarter Studies with ACH
“Data Science can help us understand the what; Research helps us understand the why.”
I’ve heard variations of this statement throughout my time as a Researcher. People often come to UXR aiming to understand the mechanisms driving user behavior or decision-making, a critical piece for product teams to understand before taking action on data. The prototypical example of this scenario is when a member of a product team comes to UXR and says “we’re seeing XYZ surprising trend - what could be going on here?”. One method I like to use to get from this question to a complete research & analysis plan is called the Analysis of Competing Hypotheses (ACH).
What’s “Analysis of Competing Hypotheses”?
ACH was developed in the 1970s by Richards (Dick) J. Heuer, Jr., a veteran of the Central Intelligence Agency, for use by the Agency. At the CIA, it’s used by analysts who are asked to make judgments in high-risk scenarios where the amount of information available is overwhelming. It also combats cognitive bias by providing structure for making decisions that encourage exhaustive thinking.
What does this have to do with UXR?
Although it might not sound like it at first, the ACH method is addressing a lot of the same problems we are when we plan to conduct research:
Our words matter Unlike a CIA analyst, we’re not usually deciding whether to infiltrate a crime ring, but the recommendations (or judgments) we deliver after conducting research carry real weight with our teams. The wrong conclusions and recommendations can potentially steer the team off course.
We have access to LOTS of information As hunters and gatherers of information, both CIA analysts and Researchers have the tools to collect all types of information. In UXR, the vastness of possibilities can sometimes make it easy to design large, wide-reaching studies that gather more information than we actually need or, on the flip side, make it challenging to figure out where to start to arrive at solid recommendations.
We’re human Cognitive bias is something that we’re all susceptible to. When asked to find the “why” with our research toolkit, it can be easy to focus on the most likely hypothesis and develop a research plan to evaluate its accuracy. When we do this, we sometimes fail to evaluate alternative hypotheses, which can weaken the impact of our final judgments, or we choose the wrong hypothesis to focus on entirely and end up with inconclusive results.
How do I actually use ACH?
I’ll walk through the way I’ve used elements of ACH for developing research and analysis plans. You’ll want to begin by opening up a spreadsheet. Here’s a template you can use!
Begin with the Question
To apply ACH to develop research plans, begin with the question you got from your product team: why might X be happening? Make sure you understand X intimately; this usually means looking at the data yourself but might also mean having a chat with your local DS team member. You can put this question at the top of your spreadsheet; it’s your anchor.
Brainstorm Hypotheses
I like to do this step in collaboration with other researchers or my product partners. What are our best guesses for why X is happening? It’s key at this stage to encourage devious thinking; no hypothesis is too far-fetched or too unlikely to write down. The more, the merrier! Taking this approach discourages us from focusing on the one "likely" hypothesis and only collecting evidence to prove its accuracy. You can list these hypotheses in the first column, one hypothesis per row.
Create the “Evidence”
This is where the magic happens. We’re going to create two columns to the right of our hypothesis column. The first column will be titled “Evidence to support” and the next column will be titled “Evidence against”.
Then we start filling it out and defining the potential evidence. Zeroing in on the first hypothesis, what evidence would we see if this hypothesis were true? I usually list the evidence as bullet points in a single cell at first. Once I’ve done the “support” column, I’ll move to the “against” column: what would we find if this hypothesis were false? It can be good to get a second set of eyes after you’ve completed your first pass; again, the goal is to be as exhaustive as we can.
The important thing to remember in this phase is that we’re not just listing definitive evidence; a hypothesis can rarely be completely disproved by any single data point (or even a collection of data points). Rather, list evidence that supports a specific conclusion; what evidence might suggest that this hypothesis is true or false?
Prioritize the Evidence
At this point, your document should look pretty full. You’ll have lots of hypotheses and lots of evidence that could support or refute each hypothesis. Now, we want to trim things down.
The creator of this technique calls this stage “Diagnostics”. The goal here is to consider each piece of evidence for each hypothesis and consider how important it is for supporting or refuting that given hypothesis. This is where we reign in our devious thinking from earlier. Not all pieces of evidence carry the same weight. I like to highlight or bold the pieces of evidence that seem especially important for that given hypothesis.
Turn Evidence into Research
Once you’ve done this for each hypothesis, you’ll start to notice some themes and patterns in the type of evidence you need. For each piece of hypothetical evidence, you can construct a way to gather that evidence. There are usually two ways:
Look at Existing Insights: We might already have this evidence from previous research studies or Data Science projects. It’s best to start here, to avoid going out and collecting information we already have.
Gather New Insights: If we don’t have existing data or research on a given hypothesis, this is where our research comes in. How would we gather this evidence if it’s out there? What method will work best?
Check Your Work
After you’ve taken your evidence and designed a study (or studies) to gather the necessary evidence if it’s out there, it can be helpful to revisit your hypothesis spreadsheet before finalizing your plan. For each piece of evidence, have you designed a way to gather it? If not, you can typically add an item to a survey question or a follow-up probe in your discussion guide to make sure you have the chance to gather it.
Analyze Your Findings
After you’ve completed your study, you can go back to your hypothesis spreadsheet to help structure your analysis. Go through each hypothesis, look for the evidence in your results and try to arrive at a conclusion for each hypothesis. What does your evidence tell you? Once you’ve done this for each hypothesis, you should be ready to construct your report and make recommendations, knowing you’ve explored and weighed many alternative hypotheses.
