Measuring Success: A Paradox and a Plan

You’ve taken your company in a new direction; the impact is a 10% growth in year-on-year revenue. Your new marketing program has increased leads by 5%. Your telecom costs have decreased by 5% because of your successful negotiations. Measuring results is an important part of our business lives. But measurements are meaningless without context, and adequate context can be hard to come by.

For example, look at that 10% growth in revenue. Should you have been able to grow more? Perhaps you’re at the front end of a humongous potential market. Who says 10% growth is a good amount, given the potential? You might compare your company to competitors, but who says they are any good? Or maybe you should have exploited your competitive advantages better. How much did you spend to achieve that 5% increase in leads? Is the market resistant, with leads difficult to generate? Are the leads qualified? Are you cannibalizing another part of your business?

You might be tempted to say, “We certainly did well, although we might have done even better.” I’m not sure this is right. If your company was set up well to enjoy 30% growth and the new CEO only achieved 10% growth, then that is arguably a failure, not “limited success.”

What’s missing in these measurements—and hard to ascertain—is a baseline, a sense of what the world would have looked like if you hadn’t acted, or a sense of what was possible versus what you achieved.

Context matters.

Why Measure Success in Digital Transformation?

I point this out because we’re tempted to look for the perfect, abstract, contextless measures of success for digital transformation, but those measures likely don’t exist. Digital transformation is broad and loosely defined; its success depends on what business-specific results a company achieves from it. The context for measuring success in digital transformation is what the company hoped to achieve, and the metrics measure how well those things were achieved.

Let me back up for a moment. First, we should ask why we want to measure success. One possible purpose is to assess the performance of those leading or implementing the transformation. That requires HR and organizational context, and I’ll leave it aside in this blog post. Another possible reason is to communicate results externally, say to shareholders or analysts. For that purpose, the company is trying to tell a story and will choose the metrics that seem to support that story best. I won’t cover that use of metrics either.

I’m not saying that you shouldn’t measure the initiative’s success, only that there will be different ways to look at it, and you will measure it based on what you wish to communicate about it. If you are doing sustainability disclosures, you will measure its impact on carbon emissions; if you are talking to the board about growing the business, then you’ll try to estimate its impact on revenues.

Another possibility is that you want to know whether your investments have been successful; that is, how much of a return you’ve realized. This goal is subject to the challenges I mentioned above. There is usually no reasonable baseline since you can’t know what the company would have looked like if you hadn’t invested in the transformation, and you don’t know what potential opportunities you missed.

If you are doing things right, the results of the transformation will be deeply embedded in results across the entire enterprise and therefore hard to untangle. After all, you are not making the investment just for the sake of the technology but because the technology is such an important part of your business, and you expect to use it to improve operating parts of your business. The transformation should impact your cost structure and revenues, risk posture, employees’ satisfaction, and ESG goals. Your return is all of those things.

Instead, I’ll focus on how to measure results on an ongoing basis, drive improvement and adjustment, and motivate employee behavior. Modern digital techniques are iterative and incremental; we no longer assume that we know the best route to accomplishing our desired outcomes upfront. Instead we start moving along a likely path and stop frequently to measure what we’ve achieved and make adjustments. So the critical measurements in digital transformation are those that show how well you are accomplishing your objectives. It’s less about measuring success than about measuring for success.

Measuring for Success

To measure our success against desired outcomes, we have to know our desired outcomes. And therein lies the challenge for many organizations. Digital transformation has become a catch-all for “becoming modern” or “catching up,” and many organizations are vague about what they hope to accomplish. In some cases, they don’t have a common understanding among their leaders. If so, they will have trouble measuring success—and perhaps trouble prioritizing, finding the urgency to move forward, and communicating to their employees what they should do. Everyone in the organization has their “day job”—their days are full already—so how can they devote time to change without a clear understanding of why that change is important? (Just telling them that change is important is not enough.)

When I talk about outcomes, I’m talking about business outcomes, not intermediate or instrumental outcomes. On the other hand, I’ve found that leaders sometimes take too limited a view of what a business outcome is, often associating it solely with immediate and predictable ROI. But other possible business outcomes include risk reduction, employee motivation, and nimbleness to address future needs. Something like “Migrate 80% of workloads to the cloud” is not a business outcome (although “Reduce the risk of security breaches by migrating 80% of workloads to the cloud” is a business outcome.) The trick is to keep asking why until you surface a true business impact.

Perhaps as a start, we can think of four categories of possible business outcomes:

1. Direct impact on revenue.
2. Direct impact on cost.
3. Reduction of risk.
4. Agility, which is essentially indirect influence on cost or revenue. (I define agility as the ability to address change [opportunities and challenges] quickly, inexpensively, creatively, and at low risk.)

Note that the third and fourth categories are difficult (impractical) to measure in terms of projected ROI (although they indirectly affect it).

Every company wants all four of these things. But based on where the company is, it will be focused on a subset of goals, and within these goals, focused on a subset of objectives that it cannot accomplish today, given its legacy ways of using technology. That is where we look for the desired outcomes of transformation.

Perhaps the company wants to grow revenues and has identified new geographies where it can sell its products but is held back because its infrastructure and software will not support globalization. Its objective for transformation might be to sell into those markets. (There’s a subtle difference between that and gaining the ability to sell into those markets, which is how we’re tempted to word it. But “gaining the ability” isn’t really a business outcome.) Or maybe the company has identified a material risk that its IT systems use such an old technology that there is only one living programmer who can make changes to it, and that programmer is doddering. Then its objective might be to reduce that risk by moving to a new technology platform.

From my last example, you can see that technology objectives can be business objectives. That’s OK—it’s likely, now that technology permeates business and drives many of its activities. The technology risk, in this case, is a business risk.

So success is measured in terms of accomplishing these primary goals, and measuring it continuously supports incremental adjustments to best accomplish those goals. If the primary goal is to increase revenues from outside the US, that’s the success measure. If the goal is to reduce risk from code written in mainframe APL, that’s the success measure. Why are you transforming?

It might sound like increasing revenues is the wrong success measure because it’s not completely under the control of IT. Actually, it’s the right measure precisely because it’s not completely under the control of IT. It is a shared goal between IT and the business unit involved, and as a result, it aligns them. This is why I was careful to distinguish between measures used for assessing employee performance and measures used for improving outcomes. We typically need to fight the impact of silos working with incompatible or contradictory incentives. Remember, we are using this measure to incrementally drive success, not to reward or punish people for their personal performance (in which case you would want to only measure things under their control).

By choosing business-level objectives and measuring against them, you ensure that the behaviors you drive get the desired results. If you measure only IT goals for IT, then you don’t know if IT’s success on those measures actually impacts company performance. On the other hand, if company performance is the target, then IT can freely change its activities to best support the company goal.

When it comes to the IT-only risk-reduction goal above, treating it as a business goal gives it the additional force and credibility to justify the fact that IT will use some of its capacity toward that goal. It’s not an IT goal but a business goal. In fact, the CFO and the board’s audit committee are invested in risk management as well as IT.

Some goals (e.g., IT nimbleness) have well-understood metrics that can be used—if they are a primary goal of the transformation (and thus also a business goal). There are the Accelerate metrics, which include (among others) release frequency and mean time to repair. Or if the stability of IT systems is an important business goal, then change failure rate and availability can be good success measures.

The important thing is to define the main purpose of the transformation (the answers to Why? And why is it urgent?) to focus the transformation effort, lend it urgency, and motivate employees. If you’ve done that, then the measures you use to gauge success—for inspecting and adapting to increase success—follow straightforwardly.