AWS Cloud Enterprise Strategy Blog
Hitting the Target but Missing the Mark
When a measure becomes a target, it ceases to be a good measure.
— Goodhart’s law
As a developer fresh out of school, the first two service level agreements (SLAs) that I was accountable for were on-time delivery (OTD) and first time right (FTR). While the intention of the first time right measure was to reduce rework and prevent defects from flowing down the line, it also resulted in long planning cycles, reams of paper used for requirement gathering, and formal sign-offs from the “business” as if IT was an external vendor. Fortunately, I grew out of that measure, adopting a more incremental and experimental Agile approach. However, on-time delivery has been a harder one to shake off—I am still working on it!
Measurements drive behaviors, especially when they become (even implied) targets. If you dig deeper into organizational conflicts or failed initiatives, it is often the result of misplaced measurements and incentives tied to them. Why do customers often report being unhappy even when a supplier meets all the SLAs? Why do stakeholders often perceive a new system rollout a failure even if it always remained “green” on all the project management dashboards? I think the challenge is that organizations have good objectives, like creating new revenue streams, better customer experiences, and improving profitability, but it is incredibly hard to know if we meet those objectives when we break them down into multiple different initiatives with different indirect measures of success. Because of the fragmented ownership across organizational silos, every department or team contributing to these objectives will come up with their own measure for their piece of the value chain. This should work in theory, but often the whole turns out to be NOT greater than the sum of all parts.
Traditional Targets Promote Fixed Thinking
Let’s consider the two most common targets used in IT project delivery.
On Budget—Budgets are a static snapshot in time, often decided twelve to eighteen months in advance. Being hyper focused on staying on budget incentivizes spending all the budgeted funds by the end of the project. Savvy operators know that if you leave something on the table, you probably won’t get it in next year’s budget, which generally only goes down from there. This is why we see a rush to reallocate and use funds in the last quarter of the year. It is generally considered “bad planning” if you have to go back to your CFO and ask more money, so we often see sandbagging and significant contingencies built in project budgets.
One way I tried to address this was to often ask my leadership team, If you had X% more money now, what could you do? Could you go after an even bigger opportunity? Could we move faster? On the other hand, if you had X% taken away, what will you stop doing? While budget is a good measure, instead of making it a fixed annual target, it should be used to broaden the horizon of possibilities or to reign in the low impact work.
On-Time Delivery—Launch dates that are set months and years in advance for large initiatives tend to slip further the closer you get to them. So what if we went live with something on the date we said we would? Did it actually deliver the business outcome we were hoping to achieve? Could we have taken more time to drive adoption and proper change management? Could we have actually done this faster?
In his last shareholder letter, Jeff Bezos talked about the power of “wandering.”
“Sometimes (often actually) in business, you do know where you’re going, and when you do, you can be efficient. Put in place a plan and execute. In contrast, wandering in business is not efficient … but it’s also not random. It’s guided – by hunch, gut, intuition, curiosity, and powered by a deep conviction that the prize for customers is big enough that it’s worth being a little messy and tangential to find our way there. Wandering is an essential counter-balance to efficiency. You need to employ both. The outsized discoveries – the ‘non-linear’ ones – are highly likely to require wandering.”
Traditional Targets Are Prone to Gaming
The problem with volume-based goals is that they are prone to gaming. There is a fable about the nail factory. The management wanted the workers to produce more and decided to set the goal as number of nails produced. To meet this goal, employees produced thousands of really tiny nails that were hardly usable. Management then changed the goal to be the weight rather than the number. To no surprise, workers produced a gigantic nail that weighed a ton but was again entirely useless. As John Wooden said, “Never mistake activity for achievement.”
Paul Graham describes this challenge in his recent essay with respect to the education system, but his commentary applies to the world of business as well.
What Is the Answer, Then?
To start, we need to differentiate between measures and goals. At Amazon, we focus on input measures, things that you control and manage. Input measures should be actionable and should be used to help you make changes and decision as you go along.
Let us consider your enterprise’s cloud journey. Your outcome goals can be to increase revenue, launch new products, or improve your bottom line. You may have set some targets like migrate a number of applications within a certain time, shut down data centers, reduce time to market for new products, or bring in new customers to achieve your outcomes. But what are some good input measures to help you successfully navigate on your journey? I want to share a set of measures that we called the “Jedi Code” (we all loved Star Wars so our cloud initiative was branded as Cloud City!) in my previous role at A+E Networks:
- Deployment downtime—should be constantly trending down to zero
- Automated testing coverage for not only performance, load, and security but also functional—should be constantly trending up
- Percentage of immutable infrastructure—should be constantly trending up as a percentage of total footprint
- Alternating applications and components across regions monthly—this is to track the progress of moving away from disaster recovery to continuous availability
- Time spent patching—should be constantly trending down
- Number of AWS trained and certified team members
- Number of cross-team nominations for peer-nominated awards
Paying attention to input measures is like driving a car: you keep an eye on the speed, direction, fuel level, and tire pressure, and make changes as needed to reach your destination. However, if you set a goal of, say, driving at 65 mph, you can certainly hit that by always staying on the highway, but you will never reach your destination (and will run out of fuel because you cannot stop to refuel!).
Develop good measures to course correct and navigate rather than converting them into the goals to “hit.”
I would love to hear about some good measures that you are using in your transformation journey.
—Ishit