Announcing the New AWS Reimagine Report on AI

Two questions come up in almost every conversation we have with executives. The first is: What are other organizations doing to make AI work? The second: Has Amazon figured this out? Underlying both is the same question. What separates the organizations generating value from AI from the organizations that are not? I am an Executive in Residence at AWS. My colleagues and I are former CIOs, CTOs, CEOs, CFOs and other senior leaders from industry and public sector, and we spent the past year researching that question, nine months of it interviewing 154 leaders: 128 executives deploying AI across 27 countries and 23 industries, 23 Amazon leaders, and 3 researchers who study how organizations absorb shifts like this one. Today we are publishing a report on our findings: “Reimagine – Turning AI into Value” (aws.amazon.com/executive-insights/reimagine/).

One common challenge we found is illustrated by an experience we had here at AWS. A rebuild of an AWS service was originally scoped at thirty engineers for a year. Instead, with AI, five engineers did it in seventy days. But Rahul Pathak, who runs Data and AI Go-to-Market at AWS, explained why that alone wasn’t enough.

“Initially, we automated writing code, but our security review, our deployment process, the UI review was still the bottleneck. And so we didn’t get code out into production any quicker, even though we wrote it faster.”

The cost of doing the work fell to almost nothing. But the cost of deciding, funding, governing and approving it did not change. Amazon had faced a similar coordination problem much earlier in its history when it moved away from traditional development practices. Amazon solved it by organizing into two-pizza teams, teams no larger than two pizzas can feed, which streamlined communication and reduced the need for coordination. For twenty years that pattern was effective. In many situations it still is.

From where I sit, though, I can see that model strain. The cost of doing the work keeps falling, the cost of coordinating it does not, and the coordination grows as a share of the total time to deliver value. No one knows yet what will replace the two-pizza team. What I can tell you is that leaders here at Amazon are experimenting to find out. In other words, we found that Amazon is not exempt from anything in our report. Amazon’s leaders answered the same questions and face many of the same challenges as the other organizations we interviewed.

The organizations we interviewed are not all at the same stage of AI adoption. Some have already run into the wall Rahul describes, where the work got faster and everything around it did not. Others are redrawing how teams are shaped now that eight people can do what used to take a department. Others are still hunting for a first use case worth the effort. And some cannot tell whether what they deployed is working, because they didn’t agree in advance what success would look like. With almost every problem we ran into, we found organizations making progress. There was one exception, and I will come to it.

When building stops being the slow part

Many organizations told us they hit the same wall that Rahul had described. Once building gets fast, everything around it becomes the constraint. At Best Buy, the US electronics retailer, a feature had sat on the backlog for four years because building it was estimated at fourteen weeks. With AI it took a day. The company deliberately kept its AI teams outside its standard delivery and review process so they could move at the pace the technology now allowed. “We isolated a lot of our AI teams,” CIO Neal Sample told us. Instead, he told them, “we’re going to keep you free of the corporate overhead, of the red tape.” The team built the feature in isolation, then brought it back for product review, brand guidelines, and, in his words, all the other things you would normally do.

It is worth asking how many of your own AI wins arrived by going around a process rather than through it. At DraftKings, the approvals for adding capacity were built for hiring, which is slow. An agent can be spun up almost immediately, and it still waits on those same approvals. Brian Walker, their SVP of AI and Operations, watches that mismatch daily. Others are rebuilding their processes rather than working around them. Air New Zealand built an AI governance assistant so governance could keep pace. But we found that most organizations are still fixing AI rather than the process around it.

What happened to the savings

Speed was not the only gain organizations struggled to turn into business results. Most of the early use cases aimed AI at savings, on the assumption that the hours they saved would turn into growth. But nearly half of the interviews that describe efficiency gains contained no evidence of business impact tied to those gains. ManpowerGroup is one of the few that could put a number on it: AI cut recruiter screening time by 60 percent. Their Chief Growth and Innovation Officer Valerie Beaulieu told us that the saving has not yet translated into revenue. Take your own last efficiency win. Can you say where those saved hours went, and who committed to producing a business result from them? If nobody did, the capacity was likely absorbed rather than redirected. Some organizations closed the loop between saving and result. AT&T returns over five dollars for every dollar it spends on AI, and it got there by changing how it decides which AI work gets funded.

The part I got wrong

I have my own mistake to own. Most organizations run decisions through gates meant to reduce risk. I have spent my career arguing that those gates cost more than they protect. I retired portfolio planning at one company and stood down the change management review board at another, and I still think these uniform decision processes are wasteful. Running every decision through the same gate, at the same weight, regardless of what is at stake, is how organizations turn caution into paralysis and call it rigor. I would make those calls again.

What I got wrong is assuming the answer was removing the gate. The answer is making the degree of oversight proportional to the consequence of being wrong. Removing a gate and right-sizing it are different answers, and for two decades I treated them as the same one. Several leaders described a similar correction. The expertise that had earned them their position was calibrated to a world where execution was expensive, a world that taught all of us to pick a few bets carefully and plan them thoroughly before acting. Those instincts were right for a long time, and I still feel them. When building took months, careful planning was the cheapest insurance there was. Now that building takes days, the expensive part is deciding. The discipline is still worth having. It must just aim somewhere else now.

The problem nobody had an answer for

Which leaves the exception I mentioned earlier. AI takes over routine work, and routine work is how people build judgment. So how does anyone junior build judgment now? Andreas Plaul, Group CIO at Haufe, described a junior developer who can use AI all day and still not understand what the code does. Duncan MacDonald, CTO for Cloud Enablement at Standard Bank, said graduates now arrive with no context, and he is not sure any leader has worked out how to help them develop good judgment.

The same question applies outside engineering. How does a junior underwriter develop an instinct for risk when AI pre-scores every application? How does a junior analyst learn to spot a bad assumption when AI produced the analysis? Christina Mack, Chief Science Officer at IQVIA, has seen AI hand an executive and an intern identical output. The difference between the two individuals is what each one takes away from the output. Take away the repetition that built the skill, and the judgment that depended on it never forms. We raised this problem in interview after interview, and nobody had an answer.

What else we found

The report runs to ten chapters, covering so much more territory than is possible in a summary. Four of the threads:

• The more successful AI gets, the more the demand for it outruns the funding available, and funding processes are not built to reprioritize quickly.
• Every organization has access to the same AI models, so competitive advantage sits in their own data, and above all in what their people know that has never been written down anywhere a system could read it.
• AI costs rise and fall with how much people use it, so the spend is hard to forecast and often only becomes visible in a quarterly review.
• The skills that make an employee valuable are shifting, and people who spent careers building skills now find they are dated.

Each chapter of the report shares the challenges leaders described and the solutions they believe are working. We asked Amazon leaders the same questions we asked everyone else, including where AI has not worked for Amazon, and their answers appear in every chapter. Amazon has not told that story at this length before.

If you have hit any of these walls yourself, or broken through one, we would like to hear about it. We learned a great deal from the leaders who talked to us and from Amazon teams working the same problems, and the report raises at least as many questions as it answers. Many of those questions are being worked out right now inside these organizations, and inside Amazon, and we will keep testing what we found against what people tell us.

You can read the report at aws.amazon.com/executive-insights/reimagine.