Be the 5%: What we learned by shipping AI at scale

Only about 5% of AI initiatives successfully ship to production at scale. At Amazon, we’ve learned firsthand what separates that 5% from the rest. In this post, we share a tactical playbook for moving your AI pilot into production while keeping your customer and agent experience completely intact.

Many organizations excel at launching initial artificial intelligence (AI) prototypes. With modern tools, the distance from zero to an impressive internal demo has been compressed to effectively nothing. You write a few prompts, connect some basic documentation using retrieval-augmented generation (RAG), and suddenly you have a working conversational application.

However, scaling that pilot into a live environment without ruining customer trust is an entirely different challenge. According to data highlighted by MIT, only about 5% of AI initiatives successfully make it to production. The remaining 95% fail because moving from a prototype to a live customer-facing solution requires navigating hidden organizational hurdles without damaging your customer experience (CX). This figure may improve over time, but the takeaway remains true: most AI initiatives do not make it to production.

At a recent industry session, Hannah Bloking (Sr. Manager Solutions Architecture, Amazon Connect Customer) and Andrei Papancea (Sr. Manager, Software Development, Amazon Connect Customer) shared their firsthand experiences deploying large-scale AI solutions with Amazon Connect Customer, AWS’ agentic AI solution that helps companies deliver exceptional customer experiences at every touchpoint.

This post is for customer experience leaders, solutions architects, and AI practitioners who have a working prototype and need to take it live.

Navigating the transition trap 
The primary risk to your customer experience during an AI rollout is misconstruing prototype speed with production readiness. When you rush a conversational application into a production environment without proper preparation, you risk introducing friction for both your customers and your agents.

To protect your customer experience during this transition, you must recognize and actively bypass four common failure patterns:

1. Prioritizing the tool over the problem: Starting an initiative with the mandate to “use AI” rather than identifying a specific business problem leads to misalignment. Organizations often over-engineer obscure, highly complicated use cases, and ignore the issues causing the most customer pain.
2. Deploying unprepared knowledge bases: A standard RAG bot looking at generic wikis or standard operating procedures (SOPs) often fails at scale. The truly critical knowledge required to resolve complex customer issues isn’t documented; it lives entirely as human judgment inside the heads of your best agents. This needs to be mapped out and encoded as business processes.
3. Allowing governance to alienate users: Legal and security teams naturally focus on mitigating corporate risk. If they are not brought into the prototyping process early, the organization often over-indexes on rigid guardrails. The result is highly restrictive AI responses that actively frustrate users.
4. Building new operational silos: Introducing an AI tool with a standalone interface or isolated workflows creates an operational silo. This forces customers or agents to adapt to yet another disconnected tool, creating immediate adoption friction in the experience.

The playbook for a production-ready solution 
To transition your AI pilot into a production-ready system with your customer experience intact, focus on a structured, business-first framework.

1. Build an integrated foundation: An AI agent cannot successfully resolve customer needs if it is isolated from your core business logic. Imagine hiring a human front-line agent and putting them in front of a phone. If you refuse to give them access to your internal tools, the agent would be entirely unable to help the customer. Your AI requires the exact same operational access; verify your backend APIs and integrations are fully functional so the AI can execute actual business processes, such as modifying a reservation or updating an account.
2. Map content to impactful volume: Protect your user experience by focusing your deployment on high-volume, predictable drivers. Review your interaction data and target use cases that represent at least 5% of your total contact volume. Automating these core drivers cleanly provides immediate business value and creates a predictable baseline for your users.
3. Treat knowledge as a continuous product: Data curation is not a one-time project. To keep pace with a fast-moving enterprise, you must treat your internal knowledge as an evolving product with continuous ownership. Focus on capturing the unwritten reasoning and human judgment used by your tenured staff, structuring it so the AI can safely navigate ambiguous customer interactions.
4. Empower the people closest to the customer: Historically, software engineers have been the primary bottleneck for deploying workflow updates. By using tools that empower non-technical, business stakeholders to directly manage knowledge and refine conversational flows, you can iterate on the customer experience rapidly without waiting on lengthy code deployments.

Managing the human element: AI as a teammate 
Maintaining your customer experience requires looking beyond the technology to focus heavily on organizational change management.

Shift the perspective: Do not view AI as a replacement for humans. Instead, treat the AI agent as a new teammate designed to amplify your best human workers.

To ensure high adoption and a smooth transition, integrate these three deployment mechanisms into your launch plan:

1. Use focus groups: Do not rely strictly on metrics to judge the success of your pilot. AI project owners quickly become the experts in their own AI solutions and lose the perspective on how an everyday user interacts with the experience. Sit down and actively watch focus groups navigate your conversational application; seeing real human interactions firsthand will quickly expose gaps in your design.
2. Ruthlessly defend the scope: Avoid launching an unvetted AI solution directly to external customers. Instead, deploy the tool internally to your internal team first. This allows you to run controlled experiments, gather feedback, and refine conversational responses in a safe environment before expanding the solution externally.
3. Design for clarification: Customers rarely provide perfectly phrased requests when asking for assistance. A major reason AI responses fail in production is that the system tries to guess the user’s intent based on a poor utterance. Build robust conversational design into your workflows, ensuring the AI asks smart, clarifying questions to guide the user to the correct resolution.

Here is an example of Amazon Connect Customer’s visual no-code canvas. Learn more about how teams build these experiences without engineering support through a demo of the agentic CX designer (preview).

Conclusion: Move fast as your primary differentiator In a shifting technology landscape, customer expectations evolve rapidly. Model evolution and your technical architectures will inevitably change tomorrow. Therefore, your greatest organizational advantage is your ability to move and adapt quickly.

Do not stall your deployment by aiming for absolute perfection across every possible edge case. Instead, fall in love with the business problem, not the specific technical solution. Focus on shipping incremental, end-to-end functionality as fast as possible, utilizing existing organizational mechanisms to minimize disruption. Align top-down leadership goals with a rigorous focus on integrated business processes and human change management. This will allow you to securely guide your organization out of the pilot graveyard and straight into the successful 5%.

To hear more from Hannah and Andrei on the ultimate CX playbook, watch them discuss enterprise AI here. To learn more about deploying AI for your customer experience, visit Amazon Connect Customer. Ready to dive in? Earn a certificate for agentic CX designer (preview) skill validation.

About the Authors

Hannah Bloking is a Senior Manager of Solutions Architects leading the Applied AI Acceleration team for Amazon Connect Customer — AWS’s agentic customer experience solution. Her team scales Applied AI expertise across AWS’ global field through agentic solutions, field enablement programs, deep product expertise, and post-migration consulting — enabling AI and human teammates to work together to deliver superior customer experiences. Her specialties include scaling technical organizations with agentic AI, field enablement at scale, and turning complex technology into measurable customer outcomes.

Andrei Papancea is a technology leader at AWS, on the Amazon Connect Customer team, joining the company following its acquisition of NLX, where he served as CEO and Co-founder. He brings extensive experience in building, scaling, and delivering innovative enterprise products to the market. Alongside his industry leadership, Andrei is an Adjunct Professor at both Columbia University and New York University (NYU). His specialties include entrepreneurship, product strategy, and bridging the gap between academic theory and real-world technology applications.