AWS DevOps & Developer Productivity Blog
Building a gated review pipeline for AI coding agents on AWS
AI coding agents have changed the economics of writing software. Tools like Kiro turn specifications into working code, tests, and documentation, and a single agent can open dozens of pull requests across your repositories in an afternoon. What they have not changed is the number of senior engineers available to review those pull requests. The AI Productivity Paradox research from Faros AI drew on telemetry from more than 10,000 developers. It found that teams with high AI adoption merged 98% more pull requests while review time rose 91%. The gains did not show up in company-level delivery metrics. The constraint moved downstream, from writing code, to reviewing it.
The intuitive fix is to add an AI reviewer to every pull request. The evidence says that alone is not enough. Human-Centric to Agentic Code Review, a study of 1.02 million pull requests across 207 open source projects, found that reviews involving AI agents reached decisions faster but did not improve quality. After AI reviewers took part, the strongest factor explaining review efficiency was not the tooling, but how humans and agents were wired together in the review flow.
This post presents a pattern for wiring together humans and agents as part of code review. It introduces a gated review pipeline with six controls that pairs deterministic checks with verified AI review, applies your policy through risk-tiered gates, and keeps humans in charge of every consequential decision. It describes the core components of each control, how to implement them with AWS services, and how to measure the effect on defect and rework rates.
Two kinds of controls, and one wiring pattern
The AWS Blog Balancing speed and safety: a control framework for AI coding agents establishes a control taxonomy for what agents produce, through author-time controls (steering documents, specifications, and agent hooks) and pipeline scanning. It also introduces an AI-assisted review control. In this post, we expand that control into an operational design pattern. The pattern includes a review panel that produces verified findings, a deterministic gate that applies policy to findings, risk tiers that set agent autonomy, and metrics that show whether defect and rework rates fell.
This pipeline adds two design rules to that foundation. The first is a deliberate pattern for human and agent collaboration, because the research cited earlier found that collaboration patterns shape review efficiency. The second is risk-tiered autonomy. A documentation fix and a payments change carry different risk, so they shouldn’t pass through the same gate.
Assign the initial tier deterministically from what the change touches, not from what the author declares. Certain critical paths and files should be assigned the highest risk tier regardless of the change, for example changes to build and pipeline definitions, the Control 4 policy bundle, steering files, and the panel reviewer prompts. Changes such as a new dependency or an AWS Identity and Access Management (IAM) policy change should also escalate the risk tier. Evaluate related changesets together so a high-risk change can’t be split into low-risk pieces.
| Tier | Change risk | Autonomy | Review depth | Evidence required |
| T1 | Low (docs, internal tooling) | Agent drafts, human approves | Light spot-check | Pull request + provenance tag (agent, model, requester) |
| T2 | Medium (standard features) | Agent drafts, engineer edits | Full review + tech-lead sign-off | Pull request + tests + approved spec or design notes |
| T3 | High (payments, auth, data, changes to pipeline configuration) | Agent assists, engineer implements | Two human reviewers, plus architecture review when the design changes | Full audit package |
The framework is methodology-agnostic. It composes naturally with structured approaches such as the AI-Driven Development Life Cycle (AI-DLC), but requires none of them. Where a team approves the specification before generation, that review settles whether this is the right change before any diff exists, leaving the pull request review to confirm it was built correctly.
Control 1: Standards consumable from one place
Code quality rules, architectural preferences, security standards, and compliance requirements each have an authoritative home where they’re owned, versioned, and published. Those homes can differ. A security team might keep a policy repository, compliance teams a wiki, and architects a decision log. All of these should feed a single consumption endpoint, so that every repository, pipeline, and agent pulls the same standards through the same mechanism instead of integrating with each source directly. Curating organizational knowledge into a form agents can consume is an easily underestimated investment. Funneling it through one consumption point means you make that investment once, not once for each repository and each source.
A single consumption point doesn’t require a single delivery mechanism. Retrieve situational standards on demand. Small, non-negotiable ones, such as production-safety guardrails, inclusive-language rules, and conventions for working with internal systems, belong in context on every interaction. Push those as baseline configuration, such as a global steering file or default agent configuration, so the agent doesn’t need to fetch them.
On AWS: Keep each standard in its authoritative home and put a single consumption layer in front of them. AgentCore Gateway, a capability of Amazon Bedrock AgentCore, can serve as that consumption point. It exposes each source as Model Context Protocol (MCP) tools from configuration alone, whether the source is a Git repository behind its provider’s MCP server, an HTTP endpoint, or Amazon Bedrock Knowledge Bases built over documents in Confluence or Amazon Simple Storage Service (Amazon S3). Every agent and tool then pulls the same versioned standards at runtime. The knowledge base is a retrieval layer for prose-style guidance that benefits from semantic search. It isn’t the system of record, and the owning team’s repository or document store remains authoritative. If you need evidence of which standard was in force for a given change, either sync a pinned copy into the repository as a pipeline step and record the version, or write the consumed standard version to an audit log at time of use. Deterministic consumers, such as linter configurations and the Control 4 policy bundle, should take the pinned path. A standards change then arrives as a version bump instead of failing every repository’s next build.
In the developer’s environment, the consumption format is whatever the agent reads, such as Kiro steering files or agent skills. Generate or sync those files from the gateway instead of writing them for each repository, so a local project becomes one more consumer of the same standards. For the always-on class of standards, generate the global steering files or default agent configuration from the gateway’s sources, and push them out through whatever provisions your developer environments. The gateway remains the on-demand path for situational standards.
Control 2: Deterministic gates first, with agent self-correction
The first gates to run are fast, deterministic, and hard-fail. These include formatting, lint, type checks, static application security testing (SAST), and secrets scanning. The remediation agent then self-corrects in a build, fail, fix loop until the stage passes or hits a maximum iteration cap, which hands the change to human review. Don’t use AI to review what a rule can catch. When the deterministic gates run first, the AI review stage sees only mechanically clean code and spends its reasoning on semantics instead of style. It also keeps most mechanical defects out of human review.
The self-correction loop is told to make the gates pass, and the shortest path to a passing gate is often a weaker gate. Treat suppression annotations, threshold changes, and removed or skipped tests in agent commits as change risk escalations.
The blog CI/CD Pipelines with agentic AI describes how Elastic does this. When a build fails, an agent steps in as a pipeline stage. It reads the failure logs, applies fixes from a deterministic tool allowlist, verifies the build, and commits, with auto-merge turned off whenever agent commits are present. In its first month, covering under half of Elastic’s dependencies, it fixed 24 broken pull requests and saved 20 days of development work.
On AWS: Run the same gates in two places. In the developer’s environment, Kiro agent hooks run them on file save and task completion. That catches defects before a pull request exists, where the fix loop is cheapest to run. The pipeline is the enforcement backstop. The same gates run in AWS CodeBuild on every pull request as a required status check, so a failing gate blocks the merge and nothing reaches review on the strength of a local run alone. Order them by cost and severity, as the predecessor post describes: secrets detection, SAST, software composition analysis (SCA), and then infrastructure as code (IaC) scanning.
When a pipeline gate fails, a remediation agent takes over. Because this agent commits to the repository, host it as a harness on AgentCore runtime, a capability of Amazon Bedrock AgentCore, outside the build, and call it from the failing CodeBuild action. The harness runs the agent loop in its own managed workspace. It treats the failure logs as untrusted data, applies fixes constrained by a tool allowlist, re-runs the gates, and commits to the source branch, reporting progress back to the pipeline. Authenticate that call with the pipeline’s IAM role so the automation runs on a workload identity you control, with no developer credential embedded in the build. Bound the loop with the harness’s execution limits. Cap fix attempts, set a wall-clock timeout, and give the agent a token budget. Count them per pull request, because each agent commit starts a new build. When any limit trips, the build stops and hands the failure to a human. Configure branch protection to dismiss approvals on new commits, so an agent fix always gets a fresh human review. For standing up the harness itself, see Hosting coding agents on Amazon Bedrock AgentCore.
Control 3: AI review as a verified panel
Run AI review as a panel of reviewers with distinct focus areas, for example, correctness, security, operational readiness, and, where the change was built from an approved specification, scope. Each seat on the panel is an agent that starts from its own clean context, which keeps it from inheriting assumptions about the code. Give each seat the diff as its focus, plus the full changed files and the dependencies they touch, so it reviews the change in context. An agent that sees only the changed lines can’t judge cross-file or architectural impact, and those are the defects most likely to survive the deterministic gates. From a security perspective, the code is untrusted input, so seats should be limited to sandbox environments with read-only tools.
Two sessions of the same model share the same blind spots. For each focus area on the panel, assign at least one agent to use a different model family than the author recorded in the provenance tag. For high-risk changes, you might run multiple agents with different models for the same focus area. Choose models by measurement. Evaluate candidates against past pull requests with known findings and prioritize models that score best for that focus area.
A two-stage verification step decides what reaches a human at the review gate. The first stage is deterministic. It confirms that the cited file and line exist and that the quoted code matches what’s there. A deterministic check can’t tell whether an anchored finding describes a real problem, which is a semantic judgment. The second stage can use a decision model, which returns one bounded answer (valid, invalid, or escalate to a human) with a confidence score. Because the answer is bounded, you can measure it. Evaluate the model against past findings with known outcomes, calibrate its thresholds, and route low-confidence cases to a human. Without this filter, volume works against you, as seen in one industrial deployment of a large language model (LLM)-based review tool. Average pull request closure time rose by more than 40%, even though developers acted on most of the automated comments.
On AWS: Run the panel seats as a single AWS CodeBuild batch build graph, one build per seat, each invoking an agentic coding harness in a fresh session and writing its findings to Amazon S3 under the commit’s key. Mount the checked-out repository so each seat has that context. The agents run inside CodeBuild because they only read code the build has already checked out, in an environment discarded after the run. The remediation agent in Control 2 writes to the repository, so it runs on AgentCore runtime under its own identity and limits.
A verification build that depends on the agents then reads their findings and runs the two-stage check. The deterministic pass checks each finding against the checked-out code and discards any finding whose file, line, or quoted code doesn’t match. A validity judgment then runs on what remains. The open source Strands Decider models fit that judgment, returning a bounded valid, invalid, or escalate classification with a confidence score. The verified, high-confidence set passes to the Control 4 gate, and low-confidence cases go to a human. For the security seat, you can use an open source harness such as the Visa Vulnerability Agentic Harness, which applies the same pattern to vulnerabilities. It uses deterministic stages for repeatable work and agentic stages for judgment, verifies each finding adversarially, and produces Static Analysis Results Interchange Format (SARIF) output that a gate can read. Run your chosen harness in a detection-only mode, so it emits findings without editing code, and feed its output to the same verification step and gate as the other agents.
Control 4: A deterministic policy gate on structured findings
The AI review panel in Control 3 never approves or blocks. It emits structured findings, each with a severity (important or minor), whether this change introduced it or it was pre-existing, and its evidence. Findings use a standard format such as SARIF, so deterministic and AI findings reach the gate in the same shape. A deterministic gate then applies your policy to that signal. It blocks on important findings, fast-paths clean changes, forces oversized changesets to split, and scales all of it by risk tier. Keeping policy in a deterministic component, apart from the model judgment it acts on, is the core of this pipeline. It makes the merge gate auditable, reproducible, and tunable for each repository. An empty finding set means the panel found nothing, not that nothing is there. Fast-path rules might shorten the queue for T1 changes but should never reduce human review depth at T2 or T3.
On AWS: Have the verification build write the verified findings to Amazon S3, keyed to the commit, as the record of that run. Then make the gate itself policy as code. A CodeBuild step or an AWS Lambda function evaluates the findings against rules written in Open Policy Agent or Cedar. The rules encode the preceding policy, scaled by risk tier. Oversized changes reach the gate less often when the specification breaks work into reviewable tasks. The gate reports to the pull request as a status check and posts a severity summary, for example “2 important, 4 minor,” before the verified findings. The rules ship in the pinned standards bundle from Control 1, so changing the merge bar means publishing a new policy version.
Control 5: Human judgment on a pre-triaged change
Human review remains a key pipeline component. What changes is what the human reads. Instead of a raw diff, the reviewer gets a change that the gates have made mechanically clean and the panel has triaged into ranked, evidence-backed findings. The reviewer no longer needs to catch what rules and agents already catch, and can focus on the judgments only a person can make. Is this change worth its risk? Does it fit the system’s design? Does it carry business context the agents never had? Are the important findings resolved, or only acknowledged?
Scale the depth to the tier: a T1 change needs one approver doing a light spot-check, a T2 change a full review with tech-lead sign-off, a T3 change two human reviewers plus architecture review when it changes a design boundary. The person who directed the agent is the change’s author and doesn’t count toward its reviewers. Review depth scales with risk, and the human decision is never delegated.
On AWS: Your existing pull request approval flow stays the baseline human gate. The panel’s verified findings land as review comments with a severity summary at the top, so the approver starts from a pre-triaged change. For T3 changes, require a second human reviewer before merge, and use your source provider’s approval rules to block approval by the requester. After merge, an AWS CodePipeline manual approval action adds a second recorded decision before production. It notifies approvers through an Amazon Simple Notification Service (Amazon SNS) topic routed to wherever they already work. Have the tooling that runs the agent, not the agent itself, write the provenance tag as commit trailers that record the agent, model, session, specification, standards version, and requester.
Figure 2: The pre-merge pipeline, where standards, deterministic gates, the verified review panel, and the policy gate run as pull request checks and branch protection enforces the gate and the tiered human review
Control 6: Post-merge verification and operations
The same logic extends after merge. Tests that ship with a change go through Controls 2 and 3 with the change. This control covers suite-improvement tests that an agent generates against existing code. Meta’s TestGen-LLM is an example. It admits a generated test only after the test clears filters showing that it measurably improves the existing suite. Admit these tests as you would untrusted contributions. They must build, pass reliably across repeated runs, and measurably improve the suite, through a coverage delta or a mutation score, before a human spends time on them. The filter removes tests that add nothing, but it can’t tell whether the asserted behavior is correct, because a test generated against existing code records what the code does today. That judgment belongs to a person, and it matters more here. A test that pins wrong behavior into the regression pack blocks the future change that fixes it. Tests that clear the bar join the regression pack that the deterministic gates run on every later change, so post-merge generation keeps strengthening the pre-merge gates. In operations, agents investigate on their own, reading telemetry, correlating signals, and proposing mitigation plans, while a person approves any remediation.
On AWS: Enforce the coverage or mutation threshold in a CodeBuild project that runs as an action in a dedicated stage of your CodePipeline. A generated test that doesn’t measurably improve the suite fails the build and never reaches the regression pack. This is the same mechanism as Control 2, pointed at a different signal. There, the build fails on lint, type, or SAST violations. Here, it fails on a coverage delta or mutation score that doesn’t clear the bar. For operations, AWS DevOps Agent fits this split. It’s read-only by default, investigating pipeline and operational failures and proposing mitigation plans for a person to approve.
Measuring whether it works
The research cited earlier found that AI review made decisions faster without making them better. Cycle time is the number most readily moved, and it can improve while quality degrades. Track review cycle time, because review is the constraint this approach exists to relieve, but only alongside the following quality measures. Avoid lines of code and pull request counts entirely, because they measure activity, not outcomes.
Measure instead:
- Defect escape rate and rework rate – Bugs found after merge, and changes that need follow-up fixes.
- Change failure rate and time to restore – The DevOps Research and Assessment (DORA) stability metrics, which show whether stability held as change volume grew.
- Finding acceptance rate per panel seat – A falling rate means the seat is getting noisier, or that reviewers have stopped engaging with its findings. Check the seat first.
- Panel recall – The share of escaped defects the panel would have flagged, found by replaying them through it. Acceptance shows what the panel gets right, and recall shows what it misses.
- Changeset size drift – Agents will absorb ever-larger diffs if you let them. Keep changes small even as more of them are automated.
- Cost per accepted finding – Model and compute spend divided by the findings that reviewers acted on, by seat and tier. It shows which seats earn their place and where tier policy should run fewer.
For the executive rollup, cost to serve software, defined as total delivery cost divided by units of value delivered, captures the whole system. Faster generation that creates more review burden shows up as a worse cost to serve. Pair cost to serve with tension metrics, such as security incidents and rollback rate, so the number can’t be improved by cutting controls. Developer experience work at Amazon reported a 15.9% year-over-year reduction on this measure in 2024.
On AWS: Measure from the signals the pipeline already emits instead of building a parallel system. Route CodePipeline and AWS CodeDeploy deployment events, plus your incident records, through Amazon EventBridge to AWS Lambda and into Amazon CloudWatch for the change failure rate and time-to-restore metrics. The DORA metrics sample is a useful reference for wiring this up. The review-quality metrics come from artifacts the pipeline already writes. Query the Control 4 findings store in Amazon S3 with Amazon Athena for acceptance rate per seat and changeset size drift. For that, the gate records each change’s tier and size in its report, and each finding’s outcome on the pull request (resolved or dismissed) is written back to the store. Defect escape and rework rates need one deliberate link. Tag post-merge defects and their follow-up fixes back to the commit that caused them, so the numbers attribute to a change.
Roll the executive view up in AWS Quick Sight, showing cost to serve software alongside the tension metrics that keep it from being gamed.
Figure 3: The post-merge pipeline re-runs the gates, admits generated tests, adds the T3 approval before production, and feeds delivery and review-quality metrics
An incremental adoption path
Roll the pipeline out in stages instead of all at once. Each stage produces the evidence you need to trust the next. A baseline tells you whether anything improved, airtight deterministic gates show what the panel adds beyond rules, and an advisory panel shows whether its findings deserve to block.
- Baseline first. Capture defect escape and rework for four weeks before you turn anything on, or you will have no evidence story. Expect escape rates to lag, because defects surface weeks after merge. Rework moves sooner.
- Deterministic gates next. Get Control 2 airtight with the agent self-correction loop. This step alone keeps most mechanical defects out of human review.
- Add the panel selectively. Pick starter repositories with high pull request volume and moderate (T2) risk, where you get signal fast without affecting a critical system. In this phase, the panel is advisory. It posts findings but doesn’t block anything, because the policy gate isn’t wired yet. Use this window to confirm its value. Watch the finding acceptance rate, and check whether the panel surfaces real issues your reviewers were missing before you let severity block anything.
- Wire the policy gate only after the panel has earned trust. When acceptance rates are stable and engineers act on the findings, let severity block merges.
- Extend to tests and operations last. The same evidence can widen agent autonomy over time, for example letting the remediation agent fix verified findings in categories where the panel has proven reliable. Teams with long-established trust in their pipeline sometimes go further and auto-merge low-risk changes, as the Kiro Crew team describes.
Figure 4: The staged rollout, from a measurement baseline through deterministic gates, a selective AI panel, the policy gate, and finally tests and operations, aligned to rising risk tiers
The same pipeline, two changes
The following table follows a T1 change and a T3 change through all six controls. The pipeline is the same for both. What changes is autonomy and review depth.
| Control | T1: fix a broken link in the README | T3: change refund-retry logic in the payments service |
| 1. Standards | Pulls the same standards, and the always-on security and compliance guardrails that are baseline config for every change | |
| 2. Deterministic gates | Formatting and link-check run, pass, nothing to self-correct | SAST, secrets, SCA, and IaC scans run. The agent self-corrects a flagged dependency, re-runs, and passes within the bounded loop |
| 3. AI review panel | Skipped by T1 policy, because the Control 2 link check already verified the fix | Full panel runs with repository context. The security and correctness seats flag an auth-scope concern on the retry path, which is verified and passed to the gate |
| 4. Policy gate | T1 policy: no important findings, fast-path to a single approver | T3 policy: an important finding blocks, oversized-diff check passes, routed for full review |
| 5. Human review | One approver, light spot-check, merge | Two human reviewers, and the auth-scope finding must be resolved, not only acknowledged. T3 adds the CodePipeline manual approval before production |
| 6. Post-merge | Nothing further | Agent-generated suite-improvement tests on the touched module must clear the coverage bar before a human looks. Operational signals are watched after deploy |
Conclusion
AI coding agents are accelerating code generation and making code review the constraint. Adding an AI reviewer doesn’t remove that constraint on its own. Outcomes depend on how humans and agents are wired together. In this post, we showed a pipeline that wires them together to speed up review while reducing risk. Deterministic rules catch the first issues. A verified panel reviews what remains and emits a structured signal, a deterministic gate applies your risk-tiered policy to that signal, and a human makes the final call on a pre-triaged change. The same pattern extends to generated tests and operations.
Before you implement this pattern, read the predecessor post, Balancing speed and safety: a control framework for AI coding agents, which covers the author-time controls this pipeline assumes. To implement Controls 1 and 2, explore Kiro steering files and agent hooks. For broader control guidance, see the AWS Well-Architected Agentic AI Lens. If you try this pattern, tell us what you learn in the comments.
If you’re attending AWS re:Invent 2026, join us in the workshop SEC416, where participants get hands-on experience building out this pattern.
