A Lens for Prioritizing Research: Confidence
At each company I’ve worked out, I love that so many product teams are eager to use UXR while we’re developing a feature! I greatly appreciate that many Designers, Prototypers and Product Managers proactively reach out to discuss UXR opportunities for a new upcoming feature they’re working on. However, this often means we have more UXR requests than we can fulfill and need to prioritize vigorously. I’d like to share one lens (of many) I like to use when trying to figure out how to prioritize the work; we’ll call it the confidence lens.
The lens
I believe that companies hire smart people who will make pretty good decisions. Based on their wealth of experience and expertise, I think that in general, Designers build things that make sense, Product Managers prioritize features that are going to make a large impact and Prototypers implement reliable, usable interfaces.
For each given feature or project, we have some amount of confidence that can change over time. I’m using confidence in a pretty unscientific way; by confidence I mean, how likely do we think this feature is to successfully work the way we want it to upon launch? Usually, confidence at any given point in time is based on how much we know (or don’t know) about a given space. Think of it like this:
Also, each feature might have a threshold for how confident we need to be in order to “ship” the feature. We’re never 100% confident in a feature before we launch it and see how it performs, but there’s usually some level of confidence we want to have in a given feature before we ship it to our audience. This is more commonly referred to as risk but it can also be called a confidence threshold.
Some features feel high risk and therefore we’d need a high degree of confidence before we’d feel confident shipping it. This could be a feature that interrupts or greatly modifies the user’s experience or something that will impact all users. Other features might be deemed low risk. These are usually features that are opt-in, easy to ignore if you don’t like them or something that impacts only a small group of users or cases.
Applying the lens
Let’s run through some common scenarios and explain how we’d use this lens to prioritize UXR for them. These are just examples! There are plenty of edge cases and nuances not reflected here, but hopefully they’ll help you understand the lens a bit more.
Innovative & Unknown: A feature that’s brand new and innovative; something that’s not been done before at the company or comparative companies
Familiar but Complicated: A feature that most other experiences have, but might be complicated to execute technically
Small & Standard: A feature that is relatively small and is commonplace in social apps
You might imagine that our starting confidence point, before we’ve designed the feature, is different for these different features:
For this one, we have pretty low confidence to start because our knowledge is limited. Do people want this feature? How will it look and feel? Can we even build it?
For this feature, we might have relatively high confidence. We’ve seen it before in other apps, we know what it will look like but we’re still not at 100% confidence given the complicated tech implementation we foresee.
For this feature, we might have even higher confidence than the blue feature. This feature is well understood, small in size and implementation seems simple.
Imagining a world without UXR, we can anticipate what the confidence curves for these features could might look like:
As we work on this feature, we figure out more about how we’d make it real in the app, we discover and solve some tech challenges, and we dogfood it internally but still, we’re not very confident in this idea after we’ve built it.
We started with pretty high confidence but as we anticipated, we stumbled across a few technical obstacles that were hard to solve. For a while there, we weren’t sure we could make it work. A bunch of smart folks figured it out and our confidence rose again but we know more now, so the confidence is a bit lower than we started.
We felt like this feature was going to be simple and it was. Because the foundational components of this feature were easy to build quickly, we spend our time finding ways to make the experience even better than we’d imagined. We feel even more confident that this feature is going to succeed!
Okay, so back to where I started...
How can UXR use the confidence lens to prioritize work?
UXR is going to be the most impactful and make the biggest difference for the product when we do research that can meaningfully shift confidence curves upward and confidence thresholds are high.
There are two components to that idea:
“when”: UXR isn’t valuable at every stage of development. Sometimes, UXR is going to be most impactful when it’s done early - the questions we have fundamentally tell us if it’s even worth pursuing. Other times, like the hypothetical blue feature, UXR might be better to do later in development, once we’ve worked through some technical challenges and our questions center around how this functions at scale or how usable it is in low-performance contexts. If we time it wrong, we might be doing UXR long after the direction for the product is set. For example, UXR done only later on with the green feature might uncover we went in the wrong direction after the team has already invested a lot.
“shift upward”: For features that start with high confidence, like the pink feature, UXR might not have much room to lift confidence and we might be unable to prioritize it. We could run a usability study with users, but it’s not likely to catch anything the team hasn’t already thought of and confidence might not raise much. Instead, we might want to focus on features where we have low confidence over those with already high confidence; a relatively small UXR study could greatly improve the team’s confidence.
“thresholds are high”: We might prioritize UXR for something we have high confidence in because our confidence threshold is particularly high. Imagine the pink feature, the one we have high confidence in, was something with high adversarial risk (something that could be easily abused by trolls to make a negative experience for others), or it was something that completely changed the way all users complete key tasks. In these scenarios, we might choose to prioritize UXR for a feature that we have 85% confidence in because we actually need 95%.
