Measuring the Success of Your Transformation

How do you know whether you’re successful in your cloud journey and your digital transformation? Or, when you’re planning your digital initiative, what benefits should you expect and how will you see them reflected in the company’s performance? Given the uncertainty and change in today’s business environment, it’s sometimes hard to find a stable baseline to measure results against. On the other hand, the cloud and DevOps make it easy to continually learn and adjust—that is, to cope with all that uncertainty and change and still realize the benefits you intend. In this blog post I’ll propose a few frameworks that may help.

Measuring Against Primary Business Objectives

The cloud is not a strategy in itself; it’s a remarkably powerful tool for accomplishing business strategies. It’s important to connect the cloud initiative with the company’s highest strategic objectives; doing so will provide the business case for the cloud initiative, the urgency to drive execution, the list of priorities to address…and the criteria for measuring success. Many of the enterprises we engage with are looking to grow; some of them by entering new geographical markets, some by broadening their customer base, some by introducing new products, and some by deepening relationships with existing customers. The cloud can help with all of these things, but it’s hard to do everything at once (that is, it’s hard to make execution decisions about priorities). By choosing specific growth objectives you’ll be able to align your success measures with your business case and use them to guide your activities.

Or, if growth is not your main objective, perhaps you’re a manufacturing company that wants to use edge computing and machine learning for predictive maintenance, and you need to move to the cloud to do so. What success measures do you care about? Two things, probably—whether you’re successfully identifying problems with equipment and fixing them before the equipment fails, and whether you’re accomplishing the business objectives behind that predictive maintenance (saving maintenance costs, not losing revenue). You can define a measure for each.

To put it another way: What’s the business value of a hammer? It depends on what you’ll build with it! Its value follows from whether you use it build a new house to sell, or to fix a piece of furniture whose nails are coming out, or to keep it in the closet in case you need it (which, by the way, has value that is often not appreciated in the IT context). It’s the same with the cloud. Measure your results against what you really care about—fulfillment of your critical business objectives.

There’s a potential trap when thinking in terms of return on investment (ROI). What’s the ROI from a potential IT investment? Trick question—there’s no ROI, just a projected ROI. Measuring success against that projection is often more a test of how accurate the prediction was. What you really care about is the actual benefit versus the costs. And even measuring the actual ROI has pitfalls, because you often don’t have a good baseline to measure against since the environment is changing quickly around you. And finally, the value of many investments is in risk reduction or in creating options—both hard to measure in ROI terms. For these reasons, I’d suggest choosing operational measures that are more directly connected to your use of the technology.

Measuring Cost Reduction

There are three critical points to keep in mind with costs. The first is that when working in the cloud, costs are not simply a given—you have a vast number of levers available for actively managing costs. This is different from the company datacenter approach, where your hardware costs are fixed (actually, they’re step costs if you’re growing) and cash goes out of pocket up front. The second is that in the cloud you can and should expect to gain lots of transparency into your costs and unit economics. And third, your costs will vary based on how much you use the cloud, which in turn depends on the course your business takes. If you grow your business, your costs will of course rise; if you develop new products for your customers, then you’ll be paying the costs associated with those new products. You can’t just consider IT costs in isolation; they’re (hopefully) linked to new revenues or cost reductions in other parts of the business.

Your ability to manage costs continuously suggests that you’ll want to develop a baseline for your costs, then work to reduce them against that baseline. You’re probably already doing that with expenses like travel and telephone costs. It would be hard to predict exactly how much your phone usage will be next year, but you can take this year as a baseline and work to reduce those costs. It’s an empirical way to discover and control costs, rather than relying on very imprecise estimates.

You’ll have to make trade-offs on the amount of effort you want to put into actively reducing costs. You can, for example, have your software developers spend time optimizing their code to reduce its execution costs. There’s a discipline called FinOps that provides a structure for doing so. But, of course, there’s a trade-off with your developers’ time—you’ll pay an opportunity cost in the other valuable work they aren’t doing.

The second point makes cost measurement especially valuable. Using strategies in the cloud, such as resource tagging, you can slice and dice your costs in many ways: by business unit, product line, resource type, transaction type—whatever is useful. This gives you unprecedented insight into your cost drivers and unit economics. You might even learn things from this analysis that cause you to rethink business process or product mix.

The third point tells us that we have to move toward treating technology costs truly as variable costs—they should vary with revenues, their timing of your cash outflows should align with your revenues. Ideally—though this requires some sophistication—you can manage many of your technology costs as unit costs and work to increase your gross margins. Another useful way to look at costs is in terms of leanness: are you achieving outcomes with minimal waste? Total costs become less interesting because they’ll vary with the scale of your business, but unit cost measurements will show you how successful you are at streamlining your business and making it lean.

In many cases, you’ll be able to drive cost reductions in business operations that dwarf potential cost reductions in the IT budget. In the example above, predictive maintenance drives cost reduction in factory operations; digital interactions with customers might reduce your call center costs; robotic process automation can reduce the costs of your paper-handling processes. Just measuring IT costs misses those successes; it’s quite likely that an additional dollar of IT spend results in more than a dollar of cost reduction elsewhere in the business.

Measuring IT Delivery

Working in the cloud will improve IT’s ability to deliver for the rest of the business. While that sounds difficult to measure, the topic has been studied quite thoroughly and there are some good ideas on how to do it. In the book Accelerate, Dr. Nicole Forsgren and her co-authors show that a small set of IT metrics predicts business outcomes, and that these metrics are closely tied to certain IT practices. You’ll want to measure (1) release frequency—how often IT can deploy changes to IT systems, (2) change failure rate—how often those changes lead to problems, (3) cycle time for deploying changes—how long it takes to deliver IT capabilities once they’re ready to go, (4) mean time to repair problems, and (5) availability of systems. This might seem too small a set of measures to encompass everything that IT does, but Forsgren shows that they’re indicative of good IT practices such as automation, effective testing practices, and team accountability, which in turn yield business results.

Measuring Business Agility and Leanness

Let’s define agility as the ability to respond to changes in the business environment quickly, inexpensively, and at low risk. It’s a capability, a potential, to be called into use when needed (though that’s probably pretty often in our fluid business environment). Like risk (see below) it can be difficult to measure. Because it’s about potential (for capturing opportunities and avoiding losses), it often needs to be analyzed more like a financial option. And you have limited sample size to measure—how often do you bring a new product to market and how quickly have you done it in the past? Anecdotal evidence of agility might be the best you can do.

But this is less of a problem than it might seem. You can still guide the transformation journey to make sure you are gaining speed and agility—for example, by using the Accelerate metrics discussed above. You can also draw a value stream map of your product or feature delivery process to find measurable ways to reduce waste. Your typical time between sensing a change in the environment and responding to it is your lead time; the value stream includes things like governance processes, resource provisioning, team assembly, as well as technical delivery. The cloud is a powerful tool for removing waste in many parts of this process, but you’ll have to address governance and funding to make the most of it.

Measuring Risk Reduction

The cloud helps reduce many types of business risk. I’d focus on: (1) security risk, (2) resilience risk, and (3) compliance risk. The best approach to measurement here is to identify the sources of risk in each category and measurably reduce them.

Compliance risk can be managed through automated policy enforcement, continuous auditing, and transparent reporting; you can best measure it by counting the number of controls that are fully automated and the number that are continuously verified. To address resilience risk, you define potential disaster scenarios, simulate them if possible (or do a tabletop exercise), and verify that you can respond to each scenario successfully. You can also set up a maturity model for your nimbleness in responding to completely unexpected (untested) scenarios and measure your capabilities against it. Or, perhaps, you can itemize your main vulnerabilities—let’s say, legacy systems that weren’t created with resilient architectures or technologies—and measure your progress in modernizing them. For security, the approach is similar—itemize vulnerabilities and work to reduce them. You can also identify general solution areas and assess yourself against them—the quality of your identity validation, authentication and authorization, logging and log analysis, and so on.

Conclusion

Just moving to the cloud will have some benefits—better security and availability, for example. But that probably isn’t the reason you’re moving to the cloud. Whatever your reason is—gaining access to machine learning, better analytics, edge computing, or whatever—it’s probably a reason that requires you to actively build or modernize. And that means that you need to drive behavior as well as measuring and celebrating results. If you want to reduce costs, you have to work to reduce costs. If you want to add new services for customers, you’ve got to add the new services for customers. Getting the right data to guide these activities should be a key goal in measuring results.

So, for a CFO who wants to know how to measure the value they’re getting from the cloud, I say start by formulating business objectives—the why. Then work with the CIO to establish the plan for what you’ll have to do to get those benefits. Agree on how you’ll measure success. Then execute on that plan. (The plan does not have to be detailed or deterministic.) When I’m meeting with customers to talk about the value of the cloud and how it will be measured, it’s far better if I can meet with the CFO and CIO together. Finding ways to measure results is really just an aspect of gaining clarity on goals and formulating approaches to accomplish them.

Mark