The Total Impact Matrix – Beyond Blind Bets

Impact is a central concept in product development, but it’s often unerdused or misused. In this article I want to suggest a thinking tool, The Total Impact Matrix, to help you use it better.

The impact of an idea is the sum of value it creates for our target audience — customers/users/partners — and for the business. Unlike outcomes that look at short term changes lower in the metrics tree, impact looks at the broader effect an idea may have.

Impact = Value to customers + Value to Business

• Value to customers/users/partners — how much does this idea address their needs (aka problems, opportunities, jobs/pains/gains) at a perceivably reasonable cost (here’s a deeper look at value-to-customer)
• Value to the business — how much does this idea grow revenue, profit, market share, or other business metrics. It’s possible to cheat here so we want to focus only on sustainable business growth. 

Some companies evaluate impact qualitatively — does the idea solve the user problem? Does it address the needs of the business? Others pick metrics and assess the change the idea will create. I favor the metrics approach because it reduces human biases, but impact assessments are not trivial, and focusing on a single metric can cause tunnel vision. We should think of Impact as multi-dimensional factor: there are the two types of value we mentioned, but there are also set of other important metrics we don’t want to degrade.

The Total Impact Matrix

Let’s assume we’re able to assess the value an idea delivers to the customers and to the business. The classic consultant move is to lay those on an X-Y matrix:

Now let’s fill out the space with colorful blocks and give them catchy titles:

We created an impact map of sorts with these main areas:

Do Nothing

This area is populated with ideas that create no added value for users or the business. We also have here ideas that create marginal improvements, but don’t justify the cost, complexity, or maintenance.

Incremental Improvements

These are ideas that add some value but don’t move the needle by much. Creating incremental improvements is not a bad thing, but over-investing in small, safe things can lead to stagnation and kill true innovation.

Business Improvements 

That’s an important category of ideas we build to help ourselves rather than the users (but without detracting user value). Examples include operational changes to reduce cost, payment flow improvements, promotions, and price adjustments. Some product experts will tell you always have to “solve user problems”, but I find this thinking overly narrow — your business has needs too, and it’s important that we address them. The problem starts if we only prioritize business improvement ideas over user-value ideas. 

Customer Satisfiers

These are ideas that address user/customer needs but may not drive short-term business impact. Still customer satisfiers make good business sense because they fuel loyalty, retention, and brand strength. For this reason, in good product companies most product ideas fall into this category.

Not all customer-satisfiers are good, though. Sometimes customers demand things that are not truly helpful and do not improve the product for the greater market. Following these requests may lead to product bloat and me-too products. 

Step-Function Growth

These are Ideas that clearly improve both customer experience and drive business growth. They are rare and you should feel a well-deserved sense of accomplishment for finding and launching them.

Big Wins

These are the breakthrough ideas that catapult the product forward in a very big way — usually major features or new products. These are very hard to find, but deliver very high impact when done right.

Um, Something’s Not Right

Product ideas face a sad, rarely discussed, reality: most don’t work. Perhaps that most clear indication of this phenomena comes from split A/B tests. Here are the success rates — % of ideas that created positive change — as measure by various companies and product units: 

Microsoft: 35% 

Slack monetizations: 30%

Netflix: 35%

VWO and Convert.com (experimentation platform) 14% of conversation rate improvements

Booking.com: 10%

Airbnb Search: 8%.

(sources: Trustworthy Online Controlled Experiments and other sources)

Experiment success rate is not one-to-one the same as idea success rate. Some ideas are good, but need iterations and improvements to create value. Some experiments fail due to bugs or bad design. Still, even with these factors the conclusion is that if you don’t test and refine your ideas, your chances of success are at best 50%, and often far lower. 

Another insight from split tests is that most ideas generate just a small positive effect, fewer create a medium effect, and far fewer deliver really big wins — an effect power law.

To reflect this new information in our matrix we need to change the X and Y scales. Here’s what a realistic impact matrix may look like:

(Note: To illustrate the point I stretched ‘zero’ to be an area on the graph. Conventionally this makes no sense, but in consultant-diagram-land we don’t fuss about such things) 

As the diagram shows, our chances of generating significant improvements (the green areas), are smaller than we think, while the possibility of landing in the no- or low-impact zones are quite significant. 

But It Gets Worse

Another lesson from A/B tests and real-world launches is that some ideas cause harm. Think of a new OS update that drains your phone’s battery, or redesign that annoys users. In rare cases—think Windows Vista or the Galaxy Note 7—they can trigger full-blown disasters.

So you know what happens next — we need to update the matrix again. 

Adding the negative options gives us three new impact areas, and they’re all bad:

• Unsustainable growth — This is the area of ideas that are simply not business-viable. For example launching a service where you lose money on each transaction. These ideas can fuel short-term growth (if you’re willing to burn through cash), but are hard to walk back later.
• Enschitification — Cory Doctorow coined the term Enschitification for what happens when product companies trade user value for business gains. Think Amazon crowding search results with ads, or Facebook filling feeds with paid posts. These ideas may boost revenue in the short run, but they erode trust and drive users away.
• Value Detractors — These ideas do nothing good; they’re pure net-zero. 

The Misleading Roulette Table

Imagine sitting at the roulette table in a casino. You place your chips on the table, thinking you understand the odds, but unbeknownst to you the roulette wheel is rigged against you and your chances of winning are far lower than you think. Such is the case in most prioritization exercises I see product companies do. What feel like solid bets often turn out to be do-nothing, minor improvements, or even value detractors. The more ambitious the idea, the less accurate the prediction. Still, consensus, rank, and a plethora of cognitive biases convince us that we can predict the future, and like a compulsive gambler, we keep betting hoping to cover our losses with a future big win.

The good news is that unlike the roulette, product development does not mandate blind bets. Small investments in research and product discovery — for example mapping out assumptions, interviewing users, running a usability test, launching an early adopter program — can greatly increase our odds to find the winners and drop the losers (here’s a full example of of product discovery in action). In other words you can cheat and “beat the house”. That’s what every successful company you know is doing. 

How to Use the Total Impact Matrix

I don’t consider the impact matrix a day-to-day working tool, but rather a conversation starter.

These are the key questions to consider:

• How intentional are we when choosing ideas? Do we have clear goals, or just go with what feels right?
• Which areas of the matrix are we mostly aiming for? Is it an healthy balance?
• How good are we at predicting winners?
• How much dev time is wasted on low- zero and negative-impact ideas?
• Are we measuring actual impact after launch?
• Should we invest more in research and product discovery?

Share this with the people who need to partake in the discussion, then drive the conversation.