How Nubank automated disaster recovery with the AWS DevOps Agent

Nubank serves more than 140 million customers across Brazil, Mexico, and Colombia and announced in 2026 its operation in the USA, over 60% of Brazil’s adult population. Its systems run thousands of microservices on Amazon Elastic Kubernetes Service (Amazon EKS) and process millions of transactions each day across accounts, cards, credit, investments, and insurance. Nubank tracks more than 300 business flows that it classifies as critical in its Business Impact Analysis (BIA). At this scale, every minute of downtime affects the financial lives of millions of people.

As an AWS DevOps Agent customer reference, Nubank shares how they replaced a slow, manual DR testing process with a platform it calls One Step Execution, and how the AWS DevOps Agent automates the evidence collection and reporting that used to take weeks.

The challenge: DR testing that didn’t scale

Nubank tests for two reasons. The first is regulatory: each country it operates in has its own regulators, and each defines the frequency, test type, and business flows to validate. The second reason matters more to Nubank’s engineers: the company tests to learn. Resilience testing gives teams visibility into vulnerabilities and confidence that recovery mechanisms work before a failure reaches customers.

The previous testing model made both goals hard to reach. A single test cycle pulled product, platform, and risk teams into the same room for more than four hours, and aligning the schedules of seven or more teams took weeks. The work itself was manual: an engineer injected a fault step by step, another collected evidence to confirm the failure, and others captured screenshots of the application to show whether the customer experience was affected. When it was, someone else gathered the technical evidence for the root cause. All of this was multiplied by the number of affected platforms and business flows. Post-test coordination and investigation ran another one to two weeks.

The manual approach also lowered the quality of the evidence. Low-fidelity results surfaced few vulnerabilities and produced little learning, so teams saw limited value in the exercise and it rarely fed the engineering roadmap.

The approach: Treat resilience as a product

Nubank started from the AWS Shared Responsibility Model. AWS is responsible for the resilience of the global infrastructure, Hardware, Regions, Availability Zones, and the managed compute, storage, database, and networking services. Nubank, as the customer, is responsible for the logical configuration, the data, and how its applications respond to failure. From that foundation, Nubank built One Step Execution, a platform that treats resilience as a product rather than a series of one-off exercises. The platform is built around a set of goals: reusable, canonical scenarios that work in any country and account; coverage of 160+ of the 300+ critical business flows identified in the BIA (2026); lower engineering effort and optimized cost per test; built-in traceability; and evidence that satisfies regulators.

How One Step Execution works

One Step Execution as a single run moves through the following steps:

1. Orchestrator: Argo Workflows acts as the orchestrator for the full test and its documetation.
2. Flow simulation: K6 as synthetic traffic runs in the environment to simulate customers exercising the critical business flows.
3. Fault injection: AWS Fault Injection Service (AWS FIS) injects the fault, marking the start of the experiment on the target.
4. Safe rollback: AWS FIS ends the fault based on failure conditions and time limits, so the experiment stays within safe bounds.
5. Recovery: The platform observes how the affected platforms and business flows recover.
6. Evidence collection: An internal service uses the Grafana plugin to capture screenshots of the relevant dashboard panels for each test.
7. Report: The AWS DevOps Agent generates the report from the test evidence. It uses Model Context Protocol (MCPs) connected to Nubank’s infrastructure to retrieve metrics, logs, and alerts from the test window.

The orchestrator runs the traffic simulation and the fault injection in parallel, and the rollback is automatic, triggered by both elapsed time and safety metrics. An end-to-end test that is safe and fully documented now takes about 40 minutes and costs a few dollars.

Where the AWS DevOps Agent fits

The AWS DevOps Agent is a managed service that acts as an always-available team member across software changes and environment operations on AWS, multi-cloud, and on-premises. In Nubank’s flow, the agent does the work that used to consume the most manual effort: it collects the evidence from each test window and generates the report automatically.

To make this possible, an internal Nubank service triggers the AWS DevOps Agent with a skill, a structured prompt that tells the agent exactly what to analyze for that DR scenario: which business flow was exercised, what fault was injected, the test window, and the specific recovery conditions to check (for example, whether traffic migrated to a healthy Availability Zone, whether replacement nodes came up within the expected time, or whether data was restored without loss). The same service connects the agent to Nubank’s internal MCP servers, giving it direct access to the metrics, logs, and alerts generated during the test window. With that context, the agent doesn’t just describe a dashboard screenshot, it correlates signals across systems to interpret whether the recovery behaved as expected and to flag what didn’t.

This is also where the agent’s value goes beyond its most common use case. Most teams use the AWS DevOps Agent for incident response, investigating a live problem to reduce mean time to resolution (MTTR) and for proactive incident prevention, analyzing historical patterns to recommend improvements before incidents occur. Nubank’s DR use case applies the same skill-and-MCP pattern to a different purpose: helping engineers produce and interpret results of a controlled experiment, rather than reacting to an uncontrolled one.

The same agent supports capabilities beyond DR testing, including autonomous incident response, reducing mean time to resolution (MTTR) by investigating operational problems across environments and proactive incident prevention, which analyzes historical patterns to recommend improvements before incidents occur.

The scenarios Nubank tests today

Nubank went from a single scenario in 2025 to three today, with a plan to reach six:

1. Availability Zone unavailable. Validates the high-availability strategy when an AZ goes down, confirming that critical components migrate to a healthy zone.
2. Cluster unavailable. Removes all nodes of a shard to evaluate how Karpenter provisions replacements and how the service recovers. This scenario replays a real incident from 2024.
3. Data destruction. With an internal service Nubank simulates data deleted or modified in  Amazon DynamoDB to measure downtime and confirm that restoration mitigates data loss.

Results

Nubank has measured the impact of One Step Execution against its previous manual model:

• Test cycle time dropped from about 4.3 hours to roughly 40 minutes, an 85% reduction at a cost of a few dollars per run.
• Engineering effort fell by 43% in 2026 across nine tests, with an estimated 50% reduction projected for 2027.
• Cost per cycle fell by 91% in 2026, with an estimated 98% reduction projected for 2027, as reusable business-flow simulations carry forward to later years.
• Business-flow coverage quadrupled in 2026, from ~40 to 160+ flows, and is projected to grow further in 2027

The platform has also demonstrated DR at real scale: a single execution terminated more than 900 Amazon EKS instances under synthetic traffic, producing formal evidence of recovery for the regulator.

“Disaster recovery testing used to mean coordinating 70+ engineers across 7+ teams, collecting evidence manually, and spending two weeks on post-test investigations. With the One Step Execution platform powered by AWS FIS and AWS DevOps Agent, we cut test cycle costs by 91% and engineering effort by 43%, while producing audit-ready documentation automatically. At our scale, every minute of unavailability isn’t just a red graph, it’s a direct, real impact on someone’s financial life.” – Vitor Puente, Engineering Director, Nubank

What’s next

Nubank is extending One Step Execution along three lines:

• Automatic scheduling. The risk team schedules an exercise and the execution runs in the background, with no live session required, using Amazon EventBridge Scheduler.
• On-call notification. The platform identifies who is on call for the affected services and notifies them before injecting a fault.
• Per-team reporting. Each team receives its own view of what broke in its domain, along with suggested improvements for the vulnerabilities found.

Together, these changes move disaster recovery from a slow, manual, multi-week effort to a one-step, minutes-long practice — one that scales resilience testing across countries while turning each test into a genuine source of learning and regulatory evidence. See full quote as customer reference in AWS DevOps Agent customers.