AI agent MLOps and development lifecycle on AWS

Getting an agent into production is a milestone worth celebrating. Keeping it there while it changes every week, without anyone quite noticing the day it got worse, is the engineering problem this module is about.

You'll likely discover the problem in a familiar order. The first deployment is exciting, the second routine, and somewhere around the fifth someone edits four words in a system prompt to fix a formatting complaint and ships it on a Friday. By Monday the agent has stopped reaching for one of its tools across a class of requests nobody tested. Nothing errored, latency was flat, dashboards stayed green, and the only signal was a slow rise in conversations that ended with somebody asking a colleague instead.

What you want is straightforward: ship changes to your agent with confidence, catch regressions before your users do, and roll back when something slips through. That is the shape of the discipline. An agent behaves as it does because of an interaction between a foundation model and a large amount of surrounding software, almost all of it yours. Any part can change on its own schedule, owned by a different person and reviewed by a different process, and a correct HTTP 200 tells you nothing about whether the answer was good.

What MLOps means 

Machine learning operations (MLOps) is the discipline of applying engineering practice to the machine learning lifecycle, so that models move from experiment into production, and stay reliable once they are there, through processes that are repeatable and automated rather than dependent on the person who happened to build them. It borrows heavily from DevOps and adds the concerns DevOps never had to address.

Four commitments distinguish it. Everything that shapes behavior is versioned, including data and configuration and not only code. Quality is established by measurement against held-out cases rather than asserted by whoever made the change. Promotion between environments is automated and gated on that measurement. And deployment starts a monitoring obligation, because a model accurate on the day it shipped will drift as the world moves on without it.

Agent MLOps keeps all four and inherits a fifth problem. In classical MLOps the versioned artifact is a trained model from a pipeline you control, and the discipline is largely about governing that pipeline. Nobody on your team trains a frontier model. What your team builds is everything wrapped around it, and that is where the engineering, the risk, and the release process live.

How the module is organized 

We start by naming the thing you release, since much of the confusion here traces back to an imprecise answer to that question. From there the module works outward: declaring the harness in code, capturing a release as a manifest, versioning each surface on AWS, structuring environments, building the pipeline, gating on evaluation, delivering progressively, coordinating across teams, and operating what you shipped.

 

Ask an engineering team what they deploy when they deploy an agent and the answers vary more than you might expect. Some say the model. Some say the application. Both make the release process harder to reason about than it needs to be, so it is worth being precise before we build anything.

What a harness is

The harness is the software system that surrounds a foundation model and turns it into an agent. It comprises the orchestration loop that decides when to call the model again and when to stop, the instructions that establish how the model should behave, the tool surface it acts through, the retrieval layer that supplies it with facts, the memory that carries context across turns and sessions, and the guardrails and policy that bound what it is permitted to do.

Adopting the term deliberately is worth the small effort of changing habits. Talking about the prompt, the tools, the knowledge base, and the memory as separate things invites teams to manage each through a separate process, which is the failure mode this module exists to prevent. Naming the whole gives you a single object to version, evaluate, promote, and roll back.

One word, two meanings

AWS now uses this word for a specific thing. The managed agent harness in Amazon Bedrock AgentCore reached general availability in June 2026, and AWS describes it in almost the terms above: if the model is the brain, the harness is the body, running the orchestration loop, executing tools, managing the context window, persisting state across turns, recovering from failures, and isolating each session.

Throughout this module, a lowercase harness means the concept: the software system around the model, and the thing you release. AgentCore harness, qualified, means the AWS resource. The concept is tool-agnostic and applies just as well to a harness you run on Amazon ECS. The resource is one way to build and version one, and Versioning Surfaces on AWS covers what it gives you.

Figure 1. The agentic harness. The model sits at the center and is the one component you procure rather than build. Everything inside the boundary is software your team writes, reviews, versions, and ships.

The harness carries most of the outcome

A persistent assumption in this space holds that agent quality is mostly a function of model choice, and that the way to improve an agent is to wait for a better model. The evidence points somewhere else entirely.

Across published agent benchmarks, and the evaluation write-ups accompanying most frontier model releases, the same pattern appears: an identical model scores dramatically differently depending on the scaffolding around it. Give a model a task inside a well-built harness, with a structured planning loop, carefully written tool descriptions, retrieval that surfaces the right context, room to check its own work, and sensible failure handling, and measured performance on the same benchmark can improve substantially. The gap between a well-harnessed agent and the raw model is frequently wider than the gap between two model generations.

The reason is clear once you look at what the harness does. It supplies context the model has no other way to obtain, converts a single forward pass into an iterative process where mistakes can be corrected, narrows an unbounded action space to a handful of well-described options, and handles the failures any probabilistic component will eventually produce. That last part is ordinary software engineering of the kind our industry has been doing for decades.

The implication for delivery is pointed. If most of the outcome comes from the harness, most of your release risk lives there too, and a process that carefully governs model selection while treating instruction edits as housekeeping has its attention in the wrong place. Everything that follows assumes the harness is the product, and deserves the full weight of release engineering.

The harness is the release unit

In practice this means a release is not "we deployed version 47 of the container," and it is not "we moved to the new model." A release is a complete, named, immutable description of every component in Figure 1, promoted as one object.

Framing it that way settles several recurring arguments. Does an instruction change need a version number? Yes, because instructions are part of the harness. Does a knowledge base refresh count as a release? Yes, even though no pipeline ran, which is a problem to solve rather than define away. Can you roll back a tool description independently of the model it was tuned against? Only by running a combination nobody has evaluated.

Four artifact classes, four cadences

The components of a harness do not change at the same rate, and they are rarely owned by the same people. Grouping them into four classes makes the delivery problem tractable.

Figure 2. The four artifact classes. Instructions and tools are the two your release pipeline fully governs. The model reaches production through that pipeline on timing the provider sets, and knowledge and memory reach it without a release pipeline run at all.

Two different pipelines are about to appear in the same paragraphs. The release pipeline is what this module builds: the automation that takes a commit, produces a harness version, gates it on evaluation, and promotes it. A data ingestion pipeline is what Module 9 builds: the automation that lands documents in the retrieval corpus. They are owned by different people, and confusing them makes the coordination problem later much harder to talk about.

With that distinction in place, the middle two rows of Figure 2 are the ones your release pipeline actually governs end to end. Instructions and tool changes originate in your repository, pass through your gates, and reach production when you decide they should. Everything in this module works cleanly for those two.

The other two rows are where the coordination problems live, and they are awkward in opposite directions. Knowledge and memory reach production with no pipeline run at all: an ingestion job completes, or the agent writes to memory during a session, and behavior changes immediately with no version bump and no gate. Coordinating with the data pipeline pulls that back under supervision using an event rather than a commit.

Foundation models are the inverse case. A model change does pass through your pipeline, since swapping a model identifier is a manifest change like any other, but you do not set the timing. Providers retire versions on their own schedule, so sooner or later one you depend on reaches end of life with a deadline attached and you run that release whether it suits you or not.

Quality is a distribution, not a boolean

Conventional software tests assert: given this input, the function returns exactly this value. Agent tests estimate. Given this input, the harness produces a response that is usually good, occasionally excellent, and sometimes poor, and running the same case twice can produce different results without anything having broken.

This shapes pipeline design more than most teams anticipate. Naive per-case pass/fail gating across a large suite will fail on healthy builds most of the time, engineers will learn within a fortnight that a red pipeline usually means nothing, and the gate you built to protect production becomes a slow formality everybody clicks past. Gating a non-deterministic system works the arithmetic through in full.

Reproducing a harness 

When an incident lands, the first question is always the same. What exactly was running? For a conventional service the answer is a container digest. For a harness you need the model identifier and its inference parameters, the exact instruction text, the tool schemas and descriptions as the model saw them, the state of the retrieval corpus at that moment, and whatever memory or session context shaped the turn.

Retrieval catches people out, because a knowledge base is a live, mutable store. Reproducing the agent as it existed last Tuesday means reproducing the corpus as it existed last Tuesday, and if ingestion overwrites in place then that state is gone. What goes in it therefore treats a corpus snapshot reference as a mandatory manifest field.

Even with all of that recorded, reproduction stays approximate. Providers patch models behind stable identifiers, sampling introduces variance, and a conversation that depended on timing may not replay identically. What you can realistically achieve is a configuration you can re-instantiate and a behavior distribution you can compare against, which in practice resolves the large majority of incidents. Chasing bit-exact reproduction costs far more than it returns.

Before we can talk sensibly about releasing a harness, we need to be able to create one on demand. Every component in Figure 1 has a corresponding set of AWS resources, and all of them belong in infrastructure as code, in the same repository as the instructions and the tool handlers, promoted through the same pipeline.

Declaring the harness in code is what makes two environments genuinely comparable, makes per-merge-request preview environments possible at all, and turns a rollback into an operation you can perform under pressure. It underpins the ephemeral environments, the deployment stages, and the reversal path described later, which is why the vocabulary is worth establishing early.

The principle

If it shapes agent behavior, it is declared somewhere you can review, version, and diff. The interesting question is not whether to declare it but which tier it belongs in, and The configuration boundary answers that. First, the tooling.

AWS CDK and AWS CloudFormation

AWS CloudFormation is the substrate. With AgentCore's general availability its resources gained CloudFormation support alongside Amazon VPC, AWS PrivateLink, and resource tagging, so an entire harness is expressible as a stack.

AWS CDK sits above it, and it is the default recommendation for teams whose estate is AWS-only, because you get compile-time checking of configuration values and constructs that encapsulate resource relationships which are verbose to express directly. A single runtime construct expands into the runtime resource, its versions and endpoints, and an artifact published to Amazon ECR or Amazon S3 depending on how you package.

Two practical notes before you build on this. The AgentCore CDK module is marked experimental, so pin your CDK version in the build and expect the occasional non-backwards-compatible change on upgrade. And make sure the deployment role carries iam:CreateServiceLinkedRole, which AgentCore requires and whose absence produces a confusing first-deployment error that has cost more than one team an afternoon.

Two AgentCore resource types carry most of the weight. AWS::BedrockAgentCore::Runtime declares a runtime you supply code for. AWS::BedrockAgentCore::Harness declares a managed agent loop where you specify the model, system prompt, and tools inline and AgentCore handles orchestration, tool execution, memory, and response generation. Both have first-class equivalents in every tool this section covers: CDK exposes aws_bedrockagentcore.CfnHarness and Terraform has aws_bedrockagentcore_harness, so the usual wait for non-CDK support does not apply here.

The AgentCore CLI is worth understanding even if you never adopt it as your deployment path, because agentcore deploy generates and applies CDK and CloudFormation underneath. It's a front end onto the same substrate rather than a parallel mechanism, so a team can start there and move to hand-written infrastructure later without re-platforming.

Declaring a managed harness

A managed harness inverts the usual relationship between code and configuration. Instead of packaging the orchestration loop into a container and declaring the container, you declare the model, the system prompt, the tool surface, the memory configuration, and the execution limits, and AgentCore runs the loop. Adding a tool or trying a different model becomes a configuration change, which is the whole argument for it.

The trade every managed abstraction asks you to make is unusually reversible here. Export harness to code generates the equivalent agent as editable Python using the Strands framework, mirroring the model, tools, skills, memory, limits, and filesystem mounts, and what comes out is an ordinary AgentCore Runtime agent you own. Start with configuration, move to code when configuration stops being enough, and keep the same deployment path throughout.

HashiCorp Terraform

HashiCorp Terraform, available in AWS Marketplace, earns its place when the harness is one workload inside an infrastructure practice that already runs on Terraform. That is common, and the argument is organizational rather than technical: a platform team with established module libraries, state conventions, policy sets, and a review workflow gets more from keeping agent infrastructure inside that practice than from the marginal ergonomics of a native tool.

Three capabilities matter here. Policy as code through Sentinel or Open Policy Agent encodes constraints that would otherwise stay tribal knowledge, such as a rule that harness execution roles never carry wildcard actions, evaluated on every plan before anything is applied. The plan output is itself a good review artifact for execution role and network changes, where security consequences are not obvious from the source. And workspaces backed by remote state give the same promotion story at the infrastructure layer that the manifest gives at the harness layer.

The threshold is straightforward. If agent infrastructure is greenfield and your team has no Terraform practice, CDK is the shorter path, and adopting a second toolchain for this alone adds surface without adding capability. Terraform earns its place when it is already the organization's infrastructure language, when policy enforcement across many teams is a requirement, or when the harness depends on resources your existing modules already manage.

Pulumi

Pulumi, available in AWS Marketplace, covers related ground from a different premise: infrastructure defined in general-purpose programming languages, with the expressive range that implies.

That expressiveness has an application which is easy to underrate. Much of the harness surface is not hand-written infrastructure but infrastructure derived from the harness definition. If twelve tools each need a handler function, an execution role scoped to that tool's permissions, a log group, and a Gateway target, the correct infrastructure is a pure function of the tool definitions. Expressing it as a loop over parsed definitions is testable with an ordinary unit test framework and stays correct as tools get added by people who never open the infrastructure directory.

One argument carries this pattern. Least-privilege IAM per tool is correct practice, and hardly anyone does it by hand, because writing twelve bespoke policies is tedious while copying the previous one takes seconds. Generating them from declared permissions makes the correct thing the easy thing, and Pulumi's CrossGuard then enforces guardrails across whatever the generator produced.

Pulumi earns its place when infrastructure is genuinely derived from application definitions, when your team's fluency sits in a programming language, or where testing infrastructure logic with a standard framework has real value. Where the harness is a fixed set of resources, CDK or Terraform gets you there with less machinery.

Choosing among them

Table 1. Any of these will build a correct harness. The choice comes down to what your organization already runs and how infrastructure changes get reviewed.

The configuration boundary

Not everything belongs in infrastructure code. Putting the system prompt in a CloudFormation template turns every wording change into a stack update, and stack updates are slow and produce diffs nobody wants to read. Five tiers, distinguished by how fast you need to change something and whether the change deserves a gate.

Table 2. A useful rule of thumb: if changing it should require an evaluation gate, it belongs in the manifest. If it is a lever you may need to pull during an incident, it belongs in Parameter Store or behind a flag.

The configuration bundle tier is the newest and changes this table most. A bundle lets you change what your agent says and which tools it reaches for — system prompts, model identifiers, tool descriptions, and arbitrary key-value pairs — without rebuilding or redeploying anything. Every update creates a new version, versions chain through parentVersionIds the way git commits chain through parents, updates carry a commit message, and branches organize lineage.

What that buys is a behavioral change with a gate in front of it and no deployment behind it. Instructions and tool descriptions have always sat awkwardly in the manifest tier, being the fastest-moving surface in the harness and the slowest to ship. A bundle version keeps the review and the gate and drops the rebuild and the stack update.

The threshold is whether your harness runs on AgentCore and reads its configuration at invocation. If it does, bundles are the shorter path for instructions and tool descriptions. If not, or if you need one artifact that also pins corpus state, Lambda versions, and guardrail versions together, the manifest described next is what you promote.

Secrets sit outside every one of them. Tool credentials, API keys, and database passwords belong in AWS Secrets Manager and are fetched at runtime by ARN, so only the ARN appears in an environment variable and the value never enters a template, a manifest, or a log line. For agents this matters more than usual, because a credential granting broad access is now reachable through natural language. Module 12 takes that up.

Infrastructure code creates the harness. The manifest describes which harness is running. You need both, and conflating them is one of the more common design mistakes in this area, because a stack template tells you what resources exist while telling you almost nothing about the instruction text or the corpus state that actually determined last Tuesday's answer.

Figure 3. Repository artifacts are normalized, hashed, and gathered into a manifest, which then drives change detection, deployment, and metric attribution. Reconciling the declared manifest against the deployed one is what surfaces harness-layer drift.

What goes in it

The manifest names every input to agent behavior by immutable reference rather than by pointer to something mutable. A path to an instruction file is a pointer, since the file at that path will differ next week. A content hash of that file is a reference. The whole value of the manifest rests on that distinction.

Several decisions in that document are load-bearing, and each one exists because its absence caused a specific class of problem somewhere.

Tool descriptions are hashed separately from tool schemas. The description is what the model reads when it decides whether to reach for a tool, so changing it changes behavior as surely as changing an instruction, without touching a line of executable code. Teams that hash a tool definition as one blob lose the ability to distinguish a behavioral change from an implementation change.

Retrieval is pinned by ingestion job and corpus snapshot rather than by knowledge base identifier alone. The identifier is stable forever, so it tells you nothing. The ingestion job plus a snapshot manifest in Amazon S3 tells you which documents were actually retrievable, and that's what makes a three-week-old incident diagnosable.

Orchestration parameters are in the manifest, not buried in code. Turn limits, retry policy, and escalation thresholds are harness behavior, they change independently of the container image, and treating them as configuration rather than as constants makes them tunable under a gate.

Finally, every release names the release it should be compared against. That sounds like bookkeeping until you try to write a relative evaluation gate, which needs an unambiguous answer to the question "better than what?" The answer isn't always the immediately preceding release, because after a rollback it is the last known-good one.

What the platform versions, and what the manifest adds

That manifest predates several AWS capabilities that now cover parts of it, and the honest position is that you should let the platform version what it can and spend your own effort on what it can't.

An AgentCore harness version is immutable and self-contained, capturing model, system prompt, tools, memory configuration, execution limits, and environment. A configuration bundle version captures prompts, model identifiers, and tool descriptions with a commit message and a parent chain. Between them, much of the document above is versioned by AWS, diffable through the API, and rollback-addressable without code you maintain.

Four things stay yours, and they're the four that decide whether an incident is tractable.

• The corpus snapshot and ingestion job. No AgentCore resource pins what was retrievable on a given date, and this is the field that most often separates a diagnosable incident from a frustrating one.

• The evaluation suite hash, evaluator set version, and baseline manifest. The platform runs evaluators. It doesn't record which version of your suite and which ruler produced the score a given release was gated on.

• The Git commit and tag. A harness version tells you what was running. It doesn't tell you which reviewed change produced it, which is the question an auditor is actually asking.

• Everything outside AgentCore. Lambda handler versions, guardrail versions, and the integration tier all version independently, and something has to name them together as one release.

So the manifest gets shorter and stays load-bearing. Where a harness version or a bundle version covers a component, the manifest records the version identifier rather than duplicating the content, and the hash boundary described next shrinks accordingly. Where nothing covers it, the manifest is the only record that exists.

What to hash, and how

Hashing is how the pipeline works out what changed and what work follows. Hash too coarsely and every commit triggers a full rebuild and evaluation run, so a typo fix costs an hour. Hash too finely and you spend more effort maintaining the hash graph than it saves.

The useful granularity is one hash per independently promotable thing: each instruction file, each tool schema, each tool description, each handler build context, and the evaluation suite. For a realistic harness that is fifteen to forty hashes, a manageable number to scan in a build log.

Normalize before hashing, or you'll chase false positives forever. Strip trailing whitespace, normalize line endings, and serialize JSON with sorted keys and fixed indentation. An instruction file that differs only because an editor added a final newline should never trigger a forty-minute evaluation run, and if it does twice, engineers will start looking for ways around the pipeline.

Keep environment-specific values outside the hash boundary. Account identifiers, ARNs that embed them, endpoint URLs, and log group names differ across environments by design, and inside the boundary they stop the staging and production manifests for identical code from ever matching, which breaks promotion logic that depends on this exact artifact having passed staging.

Declared and deployed

The manifest lives in two places doing two different jobs.

The declared manifest lives in Git. The build generates it from repository contents, commits it as a build artifact, and hands it to the deployment stage as input. It answers the question: what did we intend to deploy?

The deployed manifest lives in AWS Systems Manager Parameter Store, written by the deployment stage once every component has been successfully updated, under a path such as /harness/support/prod/current, with Parameter Store's own version history retaining what came before. It answers a different question: what is running right now?

Keeping them separate buys two capabilities. Comparing them detects harness-layer drift. And because the deployed manifest is readable at runtime, every trace, log line, evaluation record, and feedback event carries the manifest version that produced it, which is what lets you attribute a quality change to a specific release instead of a vague window of time.

Git as the source of truth

The manifest is only trustworthy if the repository it derives from is trustworthy, which makes the source control platform a genuine part of the delivery architecture rather than a place files happen to live. GitLab, available in AWS Marketplace, gives you several surfaces that matter specifically for harness delivery.

Approval rules scoped by path map onto the risk asymmetry in Figure 2. A change under agent/tools/*/handler/ is application code and needs an ordinary reviewer. A change to agent/prompts/system.md, or any description.md, is behavioral and should require approval from whoever owns agent behavior. Enforcing that as a rule rather than a convention is the difference between it happening and it happening most of the time.

Protected tags give the manifest a durable anchor. Every promoted manifest carries a gitTag, and if only the pipeline identity can create tags matching harness/v* then a manifest version cannot be fabricated after the fact, which is exactly the property an auditor is asking about when they want to know what was running in March.

Signed commits and protected branches close the loop on provenance. A harness whose instructions can be altered by anyone with repository write access has a security posture worth thinking about, particularly given that instructions are the primary defense against several of the attack classes Module 12 covers.

The package registry is a natural home for the built harness bundle, keeping the container image, the manifest, and the evaluation results for a given version together as one addressable release rather than scattered across three systems that have to be joined by convention.

The threshold for GitLab: if your team is small, everyone reviews everything, and CodePipeline with AWS CodeConnections covers you, GitLab is surface you don't need. It earns its place when several teams contribute to one harness with different approval requirements, when the audit trail has to satisfy somebody outside engineering, or when you already run it. 

The manifest is your abstraction. Underneath it sit AWS primitives with specific capabilities, and knowing precisely what each one does saves you from designing a deployment strategy that turns out not to be implementable.

AgentCore harness versions and endpoints

Start here if the harness is managed, because this is the cleanest versioning story on AWS and it asks the least of you.

Creating an AgentCore harness produces version 1 automatically. Each update creates a new immutable version holding a complete configuration: model, system prompt, tools, memory, limits, and environment. Endpoints reference versions, so you promote by retargeting an endpoint and roll back by pointing it at an earlier version, which takes effect immediately and rebuilds nothing.

That is the same shape as the runtime semantics below, because a harness is a managed abstraction over AgentCore Runtime driven through its own control plane. The difference is what a version contains. A runtime version wraps a container image you built, so prompts and tool definitions sit inside an artifact the platform can't see into. A harness version holds them as configuration the platform understands, and that visibility is what makes diffing, A/B testing, and trace-derived recommendations work on them directly.

AgentCore Runtime versions and endpoints

For harnesses deployed on Amazon Bedrock AgentCore Runtime, which is the path we recommend for new production agents, versioning is a first-class resource with clean semantics.

Creating an agent runtime produces version 1 automatically. Every subsequent update to the runtime configuration, whether that is a new container image, changed network settings, a modified protocol configuration, or updated environment variables, produces a new immutable version. These versions are complete and self-contained: they don't reference mutable state and they don't change after creation.

Endpoints are the routing layer, and each targets a specific runtime version. CreateAgentRuntimeEndpoint establishes one and UpdateAgentRuntimeEndpoint retargets it. The response carries liveVersion for what is currently serving and targetVersion for what it is moving to; during an update those differ while status passes through UPDATING on its way back to READY.

This gives you a clean blue/green model: publish a version, point a non-production endpoint at it, verify, then retarget production. Because versions are immutable and retained, rollback is one API call with a previous version number and takes effect immediately.

Amazon Bedrock agent versions and aliases

For harnesses built on the Amazon Bedrock agents primitive with action groups, the model is versions plus aliases. PrepareAgent snapshots the working draft, CreateAgentVersion produces an immutable numbered version capturing model, instructions, action groups, and knowledge base associations, and an alias is a stable name pointing at one version. Integration code references the alias, so promotion and rollback are alias updates.

An alias update is an atomic cutover: the alias points at one version, and updating it moves all traffic at once. That's clean and fast, and for many harnesses it's all you need, particularly combined with shadow evaluation, which does the risk reduction before any real traffic is involved.

A common assumption worth correcting

Teams frequently plan a canary strategy around splitting traffic across two agent versions at the alias, on the model of a Lambda weighted alias, and then discover during implementation that the API doesn't work that way.

The routingConfiguration on CreateAgentAlias and UpdateAgentAlias is an array with a maximum length of one item, and the item type carries agentVersion and provisionedThroughput and no weight field. There is nothing to weight against.

This is worth knowing early because it changes where you put the routing logic, not whether you can do progressive delivery. Weighted routing covers the mechanisms that do provide weighted routing on AWS.

Weighted routing

Four practical options, sitting at different layers with different properties.

Table 3. In practice the answer is often a combination: a weighted alias for the mechanical percentage ramp, session hashing to keep conversations coherent, and flags for capability-level control.

The AgentCore option deserves more than a table row, because it collapses three things this module otherwise asks you to build: the split, the stickiness, and the statistics.

Traffic is split by AgentCore Gateway, which assigns each request to a variant using the runtime session identifier in the X-Amzn-Amazon Bedrock-AgentCore-Runtime-Session-Id header, generating one if the caller supplies none. Assignment is sticky, so once a session lands on a variant every subsequent request in that session goes to the same place.

Two patterns, and the choice follows the change type. Configuration bundle variants run both arms on one runtime with different bundle versions, which suits a prompt edit, a model identifier change, or a tool description revision. Target-based routing registers separate endpoints or runtimes as distinct Gateway targets, which suits a code change or framework upgrade. Agents hosted outside AgentCore can take part too if you instrument them and register them as a Gateway target.

The scoring is what earns this a place in a release pipeline. An online evaluation configuration scores each session as it completes, and the service computes per-evaluator mean score, absolute and percent change, p-value, confidence interval, and a significance flag, with p below 0.05 marking significance. Results can be polled at any point without affecting validity. The rollback controller later asks you to choose sample sizes by judgment; this gives you a confidence interval instead.

Session stickiness deserves emphasis for everyone not on that path, because it's where naive percentage routing breaks agents. A stateless API routes each request independently and nothing notices. A multi-turn harness can't: if turn one goes to version 47 and turn two lands on 46, the second inherits conversation state produced by instructions it was never tuned against. The output is subtly incoherent rather than obviously broken, and close to impossible to reproduce, because it depended on a routing coin flip.

Off AgentCore the fix is a few lines of code: hash the session identifier into a bucket from 0 to 99 and compare it against the canary percentage. Ramping from 5 to 25 then only moves buckets 5 to 24, nobody is reassigned mid-conversation, and during an incident you can recompute which version served a session from the identifier alone. On AgentCore, Gateway already does this, and reimplementing it means maintaining a scheme the platform's own results won't agree with.

Instructions, tools, and retrieval

Instruction versioning

Three options, genuinely different rather than variations on a theme. Files in the repository let Git provide the versioning, so instructions diff well in review and pass through the same approval rules as everything else. Amazon Bedrock prompt management provides versioned, immutable prompt resources with variable substitution, and a console for people who should not be filing merge requests. AgentCore configuration bundles hold the prompt as immutable configuration read at invocation, with a commit message per version and branches for experimental lineage.

The choice follows who edits instructions and whether you need to vary them at runtime. Engineers working through merge requests: files in Git. A product or content team: prompt management, which gives them version history and you an ARN to pin. On AgentCore, wanting prompt changes to ship without a deployment and be testable against live traffic: bundles, also the only one of the three an A/B test can vary directly. Running more than one at once does not work, because two sources of instruction truth resolve at deploy time in an order no one wrote down.

Tool versioning

Publish a Lambda version for every tool handler and reference the qualified ARN rather than $LATEST. An unqualified reference means the manifest describes a moving target, and reproducibility disappears at the tool boundary. For tools exposed through AgentCore Gateway, version the target configuration alongside the handler.

Treat the tool description as a versioned behavioral artifact rather than documentation. It belongs next to the schema, goes through behavioral review, and is hashed independently. The most common silent harness regression is a description edited for clarity by somebody being helpful, which quietly changed which tool the model selects in a boundary case.

Retrieval versioning

Amazon Bedrock knowledge bases don't expose immutable content versions, because ingestion mutates the underlying index. What you get is an ingestion job identifier and whatever lineage your own pipeline records.

The practical pattern is an immutable corpus snapshot manifest written to Amazon S3 at the end of every ingestion job, listing document identifiers, source versions, and content hashes. It's a JSON file, so it costs almost nothing, and it is the only thing that will tell you what the agent could retrieve on a given date. Where retrieval correctness justifies the expense, some teams keep two knowledge bases with an indirection in the retrieval configuration, so a bad ingestion is reverted by pointing at the other. That doubles cost, making it a deliberate trade.

Choosing a versioning surface

Table 4. One row per harness component. If any row in your own system has no answer in the last column, that is the gap worth closing before the next production release.

The AgentCore delivery surface at a glance

You no longer need to build every piece of this from scratch. The table below maps what you can adopt from AgentCore against what it replaces in your pipeline, and how settled each capability is — because a release process should never depend on a capability whose shape is still moving unless your team has decided that trade deliberately.

Table 5. What AgentCore covers, and how settled each piece is. Treat preview capabilities as capable of changing shape, and keep the fallback described elsewhere in this module ready for anything you depend on.

Two reading notes. Anything marked preview is worth using and not yet worth making load-bearing, which means running it alongside a gate you control rather than in place of one. And coverage is not correctness: the platform will happily run an evaluator that measures the wrong thing, at a tolerance nobody calibrated, against a baseline that expired.

Three environments is the right default. The interesting engineering sits in what production-representative costs, and in the fourth environment most teams never build.

Development: optimize for iteration

Development exists so that an engineer can change an instruction and see the effect within seconds. Anything that adds latency to that loop is a direct tax on how many ideas get tried, and the number of ideas tried turns out to be one of the stronger predictors of eventual harness quality.

So keep the pipeline out of the inner loop entirely. Engineers run the harness locally or against a personal AgentCore Runtime endpoint, with a reduced corpus and ten to thirty scenarios that run in under a minute. AgentCore Runtime's direct code deployment path, which packages code and dependencies as a ZIP rather than building a container, removes the container build from the loop. Container builds are right for production and wrong when somebody is on their fourteenth instruction edit of the afternoon.

The AgentCore CLI covers most of this loop in one place: create scaffolds, dev runs locally, deploy pushes, and invoke, logs, traces, and evaluations cover the feedback half. Teams deep in their own tooling won't switch, and teams starting out get a working inner loop in an afternoon.

Two runtime behaviors will confuse anyone meeting them for the first time. Each session runs in its own microVM with isolated CPU, memory, and filesystem, terminated and sanitized when the session ends. Sessions run up to 8 hours with a default idle timeout of 15 minutes, after which compute stops and the next invocation provisions fresh compute under the same session. A reviewer who leaves a preview alone over lunch returns to what looks like a cold start, and that is correct behavior. Where the agent writes files it needs to keep, configure session storage.

Staging: the parity problem

Staging exists so the evaluation gate runs under conditions close enough to production that its result predicts production behavior. A great deal rests on how close is close enough, and that is where the cost lands.

Model parity is non-negotiable and free. Use the same model identifier, the same inference profile, and identical inference parameters. Temperature differences alone invalidate your results, and this is a surprisingly common misconfiguration, usually because somebody lowered temperature in staging to make tests more stable and thereby made them measure a different system.

Tool parity is non-negotiable and isn't free. The tools must carry the same schemas and descriptions, because those drive selection. Whether they reach the same backends is a judgment call: a tool that opens a pull request should hit a sandbox repository in staging, but the description the model reads must be byte-identical. The pattern that works is one handler with an environment-driven backend.

Retrieval parity is the expensive one, and it is where most staging environments quietly diverge. A full production corpus duplicated into staging doubles vector store and ingestion cost, and if the corpus contains customer data it drags staging into a compliance scope nobody budgeted for.

Table 6. Retrieval parity strategies. The stratified subset is the usual answer, but the sampling has to preserve topic distribution or your retrieval scores become quietly optimistic.

Whichever you choose, record it in the manifest. An evaluation score is only interpretable if you know which corpus produced it, and a team that has forgotten they are evaluating against twelve percent of the documents will systematically over-trust their retrieval numbers.

Account isolation

Use separate AWS accounts per environment under AWS Organizations. Beyond the usual blast-radius argument, two benefits are specific to harnesses.

Model invocation quotas are per-account. A staging evaluation run that fires several hundred invocations in a burst consumes real throughput, and in a shared account that burst throttles production. We have watched a routine evaluation run degrade production latency for eleven minutes because staging and production shared an account and nobody had modeled the concurrent token throughput.

Cost attribution also becomes trivial. Inference is the dominant line item in most harnesses, and knowing what fraction of it is evaluation spend rather than user-serving spend turns out to be essential for the conversations in Cost as a release signal. Cross-account allocation by tag is possible, but it is consistently more painful than account separation.

The pipeline assumes a deployment role in each target account, and that role is the only path in for harness changes. Engineers hold read access to production metrics, logs, and traces, and no write access to harness configuration. Drift, at two layers covers what to do when somebody gets write access anyway, because eventually somebody will.

Ephemeral environments are an infrastructure capability

This is the environment most teams skip, and adding it turns instruction engineering from a slow serialized activity into a parallel one.

Every merge request gets its own short-lived harness deployment at its own address, torn down when the request closes. A reviewer assessing an instruction change no longer reads a diff and imagines the effect. They open a link and talk to the agent. For instruction changes, where the diff is natural language, this is the difference between real review and a rubber stamp.

The capability depends entirely on the harness being declared in code. You can't spin up a complete environment per merge request unless something can create the runtime, its endpoint, the tool functions, their roles, and the observability wiring from a definition. Terraform and Pulumi both model this cleanly, through workspaces and stacks parameterized by the merge request identifier.

Pulumi expresses the same shape through stacks, with the merge request identifier as the stack name. Either way, share the staging retrieval corpus rather than ingesting a fresh one per merge request, because ingestion cost per preview would make the practice unaffordable. Previews exist to review instruction, orchestration, and tool changes, which are exactly the changes that do not need their own corpus.

Vercel, available in AWS Marketplace, is worth reaching for when what is reviewed includes a user-facing surface. Harness quality is partly a function of how responses render: streaming behavior, citation display, whether tool-call progress is visible, how a long answer degrades on a narrow screen. Its preview deployments give every merge request a front end wired to the corresponding harness preview endpoint, so a reviewer sees the change the way a user would. The threshold is whether your agent has a meaningful interface; consumed purely through an API, an endpoint URL and a scenario runner are shorter.

Promotion gates

Table 7. The manifest promoted to production is the identical artifact that passed staging. Nothing gets rebuilt at the production boundary.

Build once, promote the same bytes

If your production deployment rebuilds the container, re-resolves dependencies, or re-renders instructions, then the artifact you evaluated is not the artifact you shipped. Dependency resolution is not reproducible unless everything is pinned by hash, and "we evaluated a slightly different build" is a sentence that appears near the top of a lot of incident reports.

With the harness declared, the artifact defined, and the environments established, the pipeline is largely assembly. AWS CodePipeline provides orchestration, AWS CodeBuild runs the stages, and AWS CodeDeploy handles Lambda traffic shifting wherever the integration tier needs it. Two parts deserve real design attention: change-type detection and the evaluation gate.

Figure 4. The pipeline, with change-type routing. Comparing the new hash set against the deployed manifest tells you which stages need to run at all, which is what keeps the most frequent change type on the fastest path.

Repository layout

One structural decision matters more than the rest. The schema, the description, the permissions, and the handler sit together in a single directory per tool, so a merge request touching a tool touches one directory and a reviewer sees contract, model-facing text, permissions, and implementation together. Splitting descriptions into a separate documentation tree is how they drift out of sync, and how permissions get copied from whichever tool was written first.

If the harness is managed, or reads configuration bundles, agentcore.json sits alongside those directories and holds the harness definition and the bundle contents. Keep it in the repository rather than editing bundles in a console. A bundle version created by hand is a change with a commit message and no commit behind it, which is harness-layer drift wearing a convincing disguise.

For source, AWS CodeConnections links CodePipeline to GitLab, GitHub, or Bitbucket. AWS CodeCommit returned to general availability on 24 November 2025 and is reasonable where an AWS-native, IAM-integrated repository simplifies compliance. AWS has resumed active investment rather than holding it in maintenance, with Git Large File Storage and regional expansion on the roadmap, so the earlier advice to treat it as frozen no longer applies.

Build stage

The build produces the deployable artifact and the manifest describing it. Five things happen.

1. Normalize and hash every tracked artifact, producing the hash set that drives change detection.
2. Validate instructions. A lint pass enforces required sections, checks length against the model's context budget, verifies every declared variable is substituted, and scans for AWS Content Design System violations. Instructions are user-facing text in the sense that matters, since the model reads them and echoes their register back at your customers.
3. Validate tool schemas as JSON Schema, and cross-check them against handler signatures. A schema declaring a required parameter that the handler ignores produces a runtime failure that surfaces as an agent behaving inexplicably rather than as an error anyone can read.
4. Check instruction and tool consistency. This one is specific to harnesses and catches real bugs. Parse the tool set from the repository, parse tool references out of the instructions, and fail the build on either mismatch direction.
5. Build and publish. Container image to Amazon ECR referenced by digest, handler packages to Amazon S3 with published Lambda versions, and the manifest emitted as a pipeline artifact.

Both directions matter. Instructions referencing a removed tool produce hallucinated invocations, while a tool present but undescribed is usually a tool the model will never choose, which means a capability that shipped and does nothing.

Change-type detection

This is where a well-built harness pipeline separates itself from a slow one. Diffing the new hash set against the last deployed manifest tells you exactly which artifact classes changed and therefore which stages need to run.

Table 8. Change-type routing. Instruction-only changes skip the container build entirely, which is the difference between a four-minute path and a twenty-minute one for the change type you make most often.

Two rows repay a second look. A tool description change rebuilds nothing and still earns a full evaluation run weighted toward trajectory checks, because editing a description is a behavioral change to tool selection. And when only the evaluation suite changes, running it against the candidate tells you nothing, so re-baseline against the deployed production version first.

Both rows get cheaper again on a managed harness. A configuration bundle version is a control-plane write, so an instruction change produces no runtime version, no image, and no stack update, and the pipeline reduces to producing the bundle version, gating it, and pointing production at it. The evaluation run becomes the only slow part of the path, which is where the time should go.

Deploy and evaluate

The staging deployment reads the manifest, applies only the deltas change detection identified, publishes a new runtime version, and retargets the staging endpoint. It then polls until the endpoint reports READY with liveVersion matching the intended version. That poll isn't optional: evaluating against an endpoint mid-update attributes results to the wrong version, and the bug hunt is unpleasant precisely because everything looks correct.

The production deployment takes the manifest that passed staging, applies it to the production account through the assumed deployment role, and begins the progressive rollout described next. It rebuilds nothing.

Pipeline economics

Two budgets degrade silently, and both eventually push engineers into routing around the pipeline rather than through it.

The first is time. A ninety-minute pipeline will be bypassed, and the dominant cost is almost always evaluation: 200 scenarios at three repeats is 600 harness invocations, each involving several model calls and tool executions. Run scenarios concurrently, shard across parallel CodeBuild projects, and keep the full suite at the stage that gates production while the merge request stage runs a fast subset. Under 50 minutes from merge to first exposure is a reasonable target.

The second is money, and it surprises teams more often. Those 600 invocations cost real inference spend, and judge-model scoring adds more. For a team merging fifteen times a day, evaluation inference can plausibly exceed production inference. That isn't automatically wrong and may be an excellent trade, but it should be a decision rather than a discovery on an invoice. Teams running the full suite on every push routinely find evaluation consuming more than half their model spend; a twenty-scenario subset on push removes most of that.

GitLab as the pipeline surface

Git as the source of truth covered the repository side. On the pipeline side, GitLab adds two things worth having.

Protected environments with deployment approvals give a production gate enforced and auditable outside the pipeline definition, which matters when someone asks who approved a specific change and the answer has to outlive a pipeline log that rolls off after ninety days.

Merge request pipelines put evaluation results where the review is happening. Posting a score comparison table into the merge request, candidate against baseline per category with regressions highlighted, gets the quality signal in front of the reviewer as they decide. Results that live only in a pipeline console go unread.

Drift, at two layers

Infrastructure drift is the familiar case, and CloudFormation drift detection, Terraform plan, or Pulumi preview on a scheduled run will surface it.

Harness-layer drift is specific to this domain and none of those tools will catch it. Somebody edits an instruction in a console. Somebody attaches a different guardrail version while debugging and forgets to revert. Somebody triggers an ingestion job by hand. None of it is infrastructure, so none of it appears in a stack drift report, and all of it changes production behavior.

You detect it by reconciling declared against deployed. A scheduled Lambda function reads the deployed manifest from Parameter Store, queries the live configuration of every referenced resource, recomputes the hashes, and compares.

The right response to drift isn't to update the manifest so it matches reality, because that launders an unreviewed change into the record. Revert the resource to the declared state, then find out why the pipeline was bypassed. The answer is almost always that it was too slow for something urgent, which is fixable and a more useful finding than the drift itself.

Module 3 develops evaluation as a measurement discipline. Here we do something narrower and in some ways harder: wiring evaluators into a pipeline so they block bad releases without blocking good ones.

Response-only evaluation is half the picture

The instinct is to score the final response. Was it correct, was it helpful, was it well-formed. That is necessary, and it is roughly half of what you need.

Picture a harness diagnosing a failing deployment. Version A queries the deployment status, retrieves the runbook, reads the error logs, and answers correctly. Version B skips the runbook, reads the logs, guesses, and lands on the same answer. Response-only evaluation scores these identically, and they aren't identical: version B will be wrong the first time the runbook mattered, and you will have shipped that regression behind a green pipeline.

The inverse case is as instructive. Version C calls seven tools where three would do, including two expensive ones, and reaches the same correct answer. Response quality hasn't moved, latency is up forty percent, per-invocation cost has roughly tripled, and a response-only gate waves it through.

The path the harness took, its trajectory, is a first-class quality signal and frequently the leading indicator. Trajectory regressions tend to precede response regressions, because a model that has started selecting tools poorly keeps producing acceptable answers on easy cases for a while before the damage shows in output quality.

Amazon Bedrock AgentCore Evaluations

Since March 2026, you can run this through AgentCore Evaluations rather than building a custom scenario runner. It offers thirteen built-in evaluators covering response quality, safety, task completion, and tool usage, and runs in nine AWS Regions across North America, Asia Pacific, and Europe, which is a constraint to check before designing a pipeline around it.

Three modes, each mapping onto a different stage of this module. On-demand evaluation tests programmatically against spans you select, which is the pipeline gate. Online evaluation scores live production traces continuously, the monitoring mode described later. Batch evaluation runs asynchronously across a stored set of sessions, which turns shadow evaluation into something you configure rather than build. All three use the same evaluator identifiers, so what you gate on is what you monitor, and production numbers are directly comparable to pre-release ones.

Evaluations integrates with agent frameworks including Strands Agents and LangGraph through OpenTelemetry and OpenInference instrumentation, converting traces into a unified format for scoring, and results surface in Amazon CloudWatch alongside AgentCore Observability so quality and operational metrics share a dashboard.

Ground truth and trajectory evaluators

Ground truth pins evaluators to your expectations rather than to a model's general sense of what good looks like. Three forms, operating at different levels.

Table 9. Ground truth forms. The trajectory evaluators score programmatically, which has consequences for how you design gates.

The three trajectory evaluators differ in strictness, and choosing among them is a genuine design decision rather than a preference.

• TrajectoryExactOrderMatch requires the same tools in the same order with nothing extra. Reach for it where the sequence is the requirement, such as a workflow that must check authorization before acting. It is the right gate for safety-relevant orderings and the wrong gate for anything else, since it fails on harmless variation.

• TrajectoryInOrderMatch requires the expected tools in order while tolerating extras between them. This is the sensible default: the required steps happened in the required sequence, and the harness was free to do additional work along the way.

• TrajectoryAnyOrderMatch requires the expected tools to be present regardless of order, which suits cases where the steps are genuinely independent.

Because they score programmatically these evaluators are fast, effectively free, and deterministic. A trajectory evaluator either matches or it doesn't, which makes them the only evaluators suitable for hard per-case gates.

Look at the second assertion. Proposing a rollback but declining to execute one without confirmation is a safety property expressed in ordinary language, evaluated across a whole session, and close to impossible to state as a conventional test assertion. This is where agent evaluation earns its complexity.

Custom evaluators

Built-in evaluators cover general quality. Domain requirements need custom ones, and AgentCore supports two kinds with quite different economics.

Model-based custom evaluators take your prompt and your choice of model, which suits subjective domain judgments: whether generated infrastructure code follows your tagging standards, whether a response uses industry terminology correctly, whether an explanation is pitched at the right level.

Code-based custom evaluators run your logic in Python or JavaScript through Lambda-hosted functions, and suit anything checkable deterministically. Does the generated Terraform parse. Does the template validate. Is every cited document identifier real. Does the response contain anything matching a credential pattern.

A third option saves work for teams who already have an evaluation practice. AgentCore offers managed third-party evaluators from the DeepEval and AutoEval libraries. You select one by identifier and the service handles hosting, model selection, and library version, so a metric your team already trusts runs inside the same gate rather than a separate job with its own results format. You can also derive a custom evaluator from a built-in or third-party one to run its logic on your own model.

Push checks toward code-based evaluators aggressively. They are faster, cheaper, deterministic, and consume no model quota. A common and expensive mistake is using a judge model to check something a regular expression would settle, such as asking a language model whether a response contains valid JSON.

Suite design and maintenance

Four categories, in proportions that should shift as the harness matures.

Table 10. Suite composition. The regression share grows because every production incident should contribute a case, and a suite whose regression share is not growing is a suite that has stopped learning.

Adversarial scenarios deserve more repeats, typically five where others get three, with every run required to pass. The reasoning is asymmetry of consequence: a safety failure occurring one time in five is a safety failure, and averaging it away is the wrong statistical treatment.

Suite maintenance is easy to underbudget, and it decays in three distinct ways.

Evaluation set rot happens when scenarios written against a corpus that has since moved on start failing for reasons unrelated to the harness. A scenario expecting a citation of a superseded document is testing your document lifecycle rather than your agent. Audit the suite against the corpus quarterly.

Baseline drift happens when you always compare against the previous release, letting quality decline monotonically in increments that each pass the gate. A five percent tolerance applied twenty times is not a five percent decline. Anchor to a periodically refreshed absolute baseline alongside the immediate predecessor.

Overfitting happens because instruction engineering optimizes against whatever is measured. After months of tuning against the same 200 scenarios, scores improve more than real quality does. The mitigation is to keep adding scenarios sampled from production traffic, which online evaluation supplies for free, and to hold back a rotating validation subset that is never used for tuning.

Harvesting production sessions comes with an obligation. They contain whatever users typed, which for most harnesses means customer data, and the path-scoped approval rules give a broad group read access to the repository that would hold it. Redact or synthesize first: keep the shape of the input, the trajectory, and the expectation, and replace identifiers and free text carrying personal data. Module 12 takes up the wider question.

Gating a non-deterministic system

This section determines whether your pipeline gets trusted or quietly ignored, so it is worth reading carefully even if the rest felt familiar.

Figure 5. Two kinds of gate. Deterministic checks can block on a single failure; judge-scored metrics have to be aggregated, repeated, and compared against a named baseline.

The arithmetic in that figure catches out good teams. If each scenario passes ninety-nine percent of the time and you require all two hundred to pass, a healthy build clears the gate about thirteen percent of the time. At ninety-five percent it's effectively zero. Engineers learn that red means nothing, re-run until it goes green, and the gate becomes a slow random number generator. A gate people ignore is worse than no gate, because it consumes pipeline time and inference budget while providing assurance that isn't there.

Four techniques, applied together, fix this.

Repeat and aggregate

Run each scenario several times and score the aggregate. Three repeats is the practical minimum and five suits high-stakes categories. Gate on the mean rather than every individual run, because variance reduction is the point: with three repeats you can distinguish a real regression from sampling noise for most effect sizes that matter.

Separate hard gates from statistical gates

Deterministic checks get hard gates: trajectory evaluators, code-based evaluators, guardrail assertions, and schema validity. Judge-scored quality metrics get statistical gates on aggregates. Applying an all-must-pass rule to judge scores is the mistake described above, and the single most common way this goes wrong.

Gate relative to a baseline

Absolute thresholds are brittle. A rule that correctness must exceed 0.85 breaks the day you upgrade the judge model and every score shifts, and you lose a day establishing that the harness is fine and the ruler changed. Relative gates survive that: correctness must not fall more than 0.03 below the baseline manifest's score, under the same evaluator set version. This is why the manifest carries both baselineManifest and evaluatorSetVersion.

Weight by category

A regression in adversarial scenarios is not comparable to a regression in edge cases, so encode the difference.

The budgets block is worth adding early — it's the one most pipelines wish they'd started with. Latency, token consumption, and tool-call count are the three ways a harness regresses without a single quality metric moving, and Cost as a release signal explains why the token budget belongs in the gate rather than a monthly cost review.

Calibrate before you enforce

Run the suite five to ten times against a single unchanged harness version before setting any threshold. The spread you observe is your noise floor, and any tolerance tighter than it will fire on nothing, repeatedly. This takes an afternoon and saves months of pipeline distrust.

When the evaluator set changes

Every gate in this section is expressed relative to a baseline measured with a particular set of evaluators. That makes the evaluator set a dependency with a lifecycle of its own, and the one teams forget until it moves under them.

Three things change it. You add or reweight an evaluator. AWS improves a built-in evaluator, which the service does over time by design. Or the judge model behind a model-based evaluator is upgraded or deprecated, which is the forced migration described later applied to the instrument rather than the subject.

The consequence is worse than a model migration. A model change invalidates one harness's baseline. A ruler change invalidates every baseline you hold at once, including the ones you would use to work out whether the ruler change was a problem.

Treat it as its own release, with a procedure that mirrors the model migration:

1. Freeze promotion. Nothing ships against a baseline that is about to be re-measured.
2. Re-baseline the deployed production version under the new evaluator set, with enough repeats to establish a stable distribution per category. This is the same run as a re-baseline after an evaluation suite change, and the same row in the change-type table covers it.
3. Compare old ruler against new ruler on the same unchanged artifact. Score shifts here are properties of the evaluator rather than of the harness, and writing down the per-category delta is what prevents the next argument.
4. Recalibrate thresholds, absolute and relative. A tolerance of 0.03 means something different under a judge whose scores cluster differently.
5. Bump the evaluator set version and stamp it on the new baseline manifest. A gate comparing scores across evaluator set versions is comparing nothing at all.
6. Unfreeze, and keep the previous evaluator set available for one release, so a surprising result can be checked against the old instrument before anyone concludes the harness regressed.

The manifest already carries evaluatorSetVersion and baselineManifest for exactly this. Most teams populate both and never use them, then discover during a judge upgrade that those two fields were the plan the whole time.

The evaluation gate is a prediction. Production is the measurement. Progressive delivery is how you limit the cost of the prediction turning out to be wrong.

Why agent canaries run slower

Conventional canary analysis works because the failure signal arrives quickly and unambiguously. A bad deployment produces errors, elevated latency, or failed health checks within seconds, and a controller can act inside a minute.

Harness quality regressions produce none of those things. Every request returns successfully. Latency may be unchanged or even improve, because a version that stopped calling a tool is faster than one that calls it. The signal is that the answers got worse, and it arrives through a judge model minutes later, user feedback hours later, or a support ticket next week.

Three design consequences follow. Canary windows run in hours rather than minutes. The primary signal has to be a quality evaluator rather than an error rate. And because the window is long, you need a way to limit exposure that doesn't depend on catching the problem quickly.

Figure 6. A rollout sequence for a high-traffic harness, the signals the rollback controller watches, and the four categories of effect that survive a rollback regardless.

Shadow evaluation

Shadow evaluation is the strongest technique available, and the one teams reach for too late. Mirror a sample of production requests to the candidate version, discard its responses, and evaluate them offline against what the stable version returned.

The properties are excellent. There's no user exposure at all, since shadow responses are never served. You get the real traffic distribution, which a curated suite can't replicate, because production queries are messier, longer, and stranger than anything anyone writes by hand. And you get a paired comparison on identical inputs, a far stronger statistical position than comparing two populations that arrived at different times.

The constraint is side effects. A shadowed harness calling a tool that opens a pull request, sends a message, or writes to a ticket system will do all of it for real. Shadow execution therefore needs read-only tool variants or a side-effect-suppressing execution mode. The pattern that scales is a tool wrapper that reads a shadow flag from the invocation context and short-circuits mutating operations while returning realistic synthetic results, so the trajectory is preserved.

On AgentCore the scoring half is configuration rather than construction. Batch evaluation runs a stored set of sessions through your evaluators asynchronously, so the work reduces to mirroring traffic and suppressing side effects. The comparison uses the same evaluator identifiers as the gate and the production monitor, which matters more than the effort saved: a shadow result that can't be compared against both is a third opinion rather than a stronger one.

Running shadow at five to ten percent of traffic for a day before any real exposure is probably the highest-value addition most harness pipelines can make, because it catches the regressions curated suites systematically miss: failures that only appear on inputs nobody thought to write down.

Tuning the sequence to the change

Not every change deserves the full sequence in Figure 6, and applying it uniformly is how teams take two days to ship a typo fix. Tie the sequence to the change type from Table 8. A model change gets everything, including the longest shadow phase you can afford. An instruction change starts at canary. An infrastructure-only change goes straight to full after smoke tests. Hotfixes still run the full evaluation gate, with approval collapsed to one designated approver and a start at twenty-five percent.

Feature-flagged capability rollout

Version-level routing is coarse. It answers which build serves a request and nothing finer. What you often want is capability-level control: enable a new tool for internal users only, roll a new planning instruction out to one tenant, or disable an expensive research capability during a cost incident without rolling back a release that also contained three bug fixes.

LaunchDarkly, available in AWS Marketplace, provides that control plane. The harness evaluates flags at invocation time and assembles its own configuration accordingly.

Three capabilities version routing cannot provide. Targeting by attributes the deployment layer knows nothing about, such as tenant, plan tier, region, or an individual user. An instant kill switch for one capability, without reverting unrelated changes that shipped alongside it. And decoupling exposure from deployment, so a capability ships dark, gets verified against real data, and is turned on later by somebody who is not an engineer.

LaunchDarkly earns its place when exposure decisions are per-segment rather than per-percentage, when non-engineers make them, or when you need granularity finer than the deployment artifact. Where the requirement is simply that some percentage of traffic gets the new version, Lambda weighted aliases with CodeDeploy are the shorter path.

Two cautions. Flags are hidden configuration state, and a harness depending on twelve of them has a configuration space of several thousand combinations, almost none evaluated. Record the resolved flag state in every trace, as the code above does, so a session can be reproduced. And put an expiry date on every flag when you create it.

The rollback controller

Automated rollback needs its criteria specified before the deployment starts. Defining them during an incident reliably produces the familiar pattern where the threshold gets adjusted upward to avoid a rollback that has become inconvenient.

Table 11. Rollback signals, ordered by detection latency. The fast signals catch outages; the slow signals catch the regressions that actually characterize harness failures.

A cold cache makes a healthy candidate look expensive

Prompt caching keys on a stable prefix. An instruction change invalidates it, so for the opening stretch of a canary the candidate runs cold while the stable version runs warm.

Two rows in the table above will fire on that alone. Latency looks elevated and cost per session looks inflated, and neither has anything to do with quality. Teams who meet this for the first time during an incident usually roll back a perfectly good release and spend the next day looking for a regression that was never there.

Give the candidate a warm-up period before any latency or cost comparison starts, and compare only windows where both versions have had the same opportunity to cache. The quality signals need no such allowance, so they can start immediately.

Implement the controller as a Lambda function on a one-minute EventBridge Scheduler rule, or a Step Functions state machine when the rollout has phases with different criteria. Two implementation details save real pain later.

Require a minimum sample size before any comparison. At five percent exposure the first few minutes may contain a dozen sessions, and comparing quality scores across a dozen sessions produces confident nonsense. Fifty sessions is a reasonable floor for error rates, and several hundred for judge-scored quality.

Add hysteresis, and latch after firing. A metric oscillating around a threshold triggers repeatedly, so require the breach to persist across consecutive windows. Once a rollback has fired, never auto-advance that version again: a second attempt without a code change in between is the same failure repeated.

What rolling back does not undo

Reverting an endpoint restores the harness configuration. It doesn't undo what the harness did, and that gap is worth planning for, because it's easy to assume rollback is a complete remedy when it is half of one.

During the canary window the candidate called real tools. It opened pull requests, modified tickets, sent notifications, restarted services, and wrote records. Rolling back doesn't un-send a message or un-restart a service, so if the regression caused inappropriate actions you have a cleanup problem separate from the deployment problem, and usually larger.

Memory is the subtler case, and it produces one of the more confusing incidents in this space. A harness with long-term memory wrote to it during the canary. If the candidate stored malformed summaries or incorrect user facts, the rolled-back version reads that corrupted state and keeps behaving badly despite being the known-good build. The rollback completed, the version is correct, and the agent is still wrong, which sends people looking in the wrong place.

Corpus mutations persist if the harness, or an ingestion pipeline coupled to it, writes back to the retrieval store. Downstream consumer state isn't reverted at all: a workflow that branched, a record that was updated, or a human who decided something on the strength of an answer.

AgentCore Memory makes the second concrete rather than aspirational. Long-term memories are extracted by a strategy and stored under a namespace, a hierarchical path you define from variables including actorId, strategyId, and sessionId. Put a variant or manifest discriminator into that path and a canary's memory writes land somewhere addressable, so removing them becomes a scoped delete rather than wiping memory for every affected actor.

It pays to know which strategy is writing. The built-in strategies extract semantic facts, running summaries, user preferences, and episodic sequences, and you can supply your own extraction logic where a deterministic schema matters. A candidate that changed how the agent phrases conclusions corrupts a summary strategy long before a semantic one, and knowing which to purge is the difference between a targeted fix and a blunt one.

Design for irreversibility, not just for rollback

Stamp every side effect with the manifest version. Any tool invocation that mutates something writes the manifest version into the target system, whether that is a pull request label, a ticket field, or an audit record. When a bad version is identified, the affected actions become queryable instead of guessable.

Version memory writes. Tag memory records with the manifest version that produced them so a targeted purge is possible. Without this the only remediation for corrupted memory is a full wipe, which discards all the legitimate context alongside it.

Escalate approval requirements during canary. For the highest-consequence tools, require human confirmation while a candidate version is serving. It is slower, and considerably cheaper than the alternative.

Prefer shadow mode where mutating tools are involved. If a change touches tool selection around irreversible actions, extend the shadow phase rather than shortening it, since shadow is the only phase with no side effects at all.

Everything so far has assumed one team shipping one harness. Real systems have a data team on one cadence, a platform team on another, a model provider on a third, and increasingly several harnesses that call each other.

The dependency graph

Some changes must ship together, some in a particular order, and some are genuinely independent. Getting this wrong produces failures that look like quality problems and are actually sequencing problems, which makes them expensive to diagnose.

Table 12. Release coupling. The first three rows are enforceable automatically by the build-stage consistency check in Build stage, which is the cheapest safety net in this module.

Coordinating with the data pipeline

Corpus refreshes are the change class that reaches production without a release pipeline run and therefore without a gate. The delivery concern is making a refresh into a visible release event.

The mechanism is an event. On completion the ingestion pipeline emits to Amazon EventBridge with the ingestion job identifier, document counts by operation, affected data sources, and the corpus snapshot URI. A subscriber runs a retrieval-focused evaluation subset against production and publishes the result.

The design decision that matters is what the subscriber does when that evaluation fails. Blocking isn't usually available, since the content is already ingested and being served. What you can do is alert with specificity and hold the manifest update, so the deployed manifest still records the last validated corpus state until a human confirms. That preserves the invariant that the deployed manifest describes a validated configuration.

Where retrieval correctness justifies the expense, the parallel knowledge base pattern restores a real gate: ingest into the inactive base, evaluate against it, and switch the retrieval configuration only on success.

Coordinating with external tool surfaces

The four artifact classes near the start of this module split components by who controls their cadence. A fifth has grown quickly enough to deserve naming: tool surfaces you consume and do not own.

An AgentCore Gateway target fronting a third-party API, or a Model Context Protocol server operated by another team or vendor, sits inside your harness and changes on somebody else's schedule. Revise a tool description for clarity and the text your model reads changes. Add a parameter and the schema your handler validates against changes. No commit lands in your repository, no pipeline runs, and no gate stands in the way, which is precisely what makes corpus refreshes difficult.

The mitigations are ones already in this module, aimed at a new target.

• Pin what you can. Where a target exposes a version, record it in the manifest and treat an upgrade as a manifest change with a gate in front of it.

• Hash what you cannot pin. Fetch the tool schemas and descriptions as the model will see them at build time, hash them, and compare against the deployed manifest. This is the same reconciliation as harness-layer drift detection, and it belongs in the same scheduled function.

• Evaluate on change rather than on schedule. A detected change in an external tool surface should trigger the trajectory-weighted subset, on the same event-driven pattern the data pipeline uses.

• Fail visibly. A tool whose description drifted and whose selection rate then moved is a diagnosable incident. A tool whose description drifted silently is a quality mystery.

None of this gives you a veto over somebody else's release, which is why it's a coordination problem rather than a pipeline problem. What it gives you is a release event where there was none, the same thing the ingestion event gives you for the corpus.

Model deprecation and forced migration

This deserves explicit treatment, because it is the one release you do not schedule and carries more risk than most migrations you will run.

Foundation models have lifecycles. A version you depend on will eventually be deprecated with a migration deadline, and unlike every other change here you can't decline it. Your instructions were tuned against that model, your thresholds calibrated against its score distribution, and your trajectory expectations encode its tool-selection tendencies.

Treat it as a project rather than a release, and start when the notice arrives rather than when the deadline approaches.

1. Baseline. Run the full suite against the current model with enough repeats to establish a stable per-category distribution. This is the comparison you will spend the whole project defending against.
2. Measure the new model unchanged. Same instructions, same tools, new model identifier. Expect regressions, and read their pattern diagnostically: a broad decline suggests an instruction style mismatch, while a narrow decline in specific categories usually points at tool descriptions or output formatting.
3. Revise instructions against the new model. This is the actual work, and it is instruction engineering rather than configuration. Budget weeks rather than days for a mature harness.
4. Recalibrate thresholds. Score distributions shift between models, so absolute thresholds carried over will be either meaningless or unattainable.
5. Extend shadow evaluation. A model migration warrants the longest shadow phase you can afford, measured in days rather than hours, because the regression surface is broad and curated scenarios cover a narrow slice of it.
6. Migrate with human approval and the full rollout sequence. No compression.

Two mitigations reduce the pain of the next one. Reference models through inference profiles rather than raw identifiers, which gives you one place to change. And keep the model identifier and inference parameters in the manifest rather than in code, so a migration becomes a manifest change with a gate in front of it.

Multi-harness coordination

Where one harness calls another, the delivery problem compounds in ways single-harness guidance does not address.

Treat the interface between them as a contract, with the discipline you would apply to a service API. A supervising harness depends on a worker's output shape, latency envelope, and failure semantics, so version that contract and run consumer-driven contract tests in the worker's pipeline. A worker change that breaks a supervisor expectation should fail in the worker's pipeline rather than in production.

Two ordering rules avoid the common failure. Deploy workers before supervisors when the supervisor will consume a new capability, so it exists before anything depends on it. Deploy supervisors before workers when a capability is retired, so nothing depends on it by the time it disappears. Same discipline as database migration ordering, and it fails the same way when skipped.

Where the composition is a workflow rather than a call graph, AWS Step Functions can invoke a managed harness directly through an AgentCore InvokeHarness state. That puts ordering, retries, and error handling into a state machine you version and deploy with the rest of the infrastructure, rather than inside a supervisor's orchestration loop where they are harder to reason about during an incident.

For evaluation, a worker change should trigger the supervisor's suite as well as its own, because a worker whose trajectory changed may pass its own scenarios while breaking the supervisor's assumptions about timing or return shape.

Change freezes

Freeze policy is enforced as a Parameter Store flag read by the production deployment stage. When set, the pipeline runs source, build, staging deployment, and evaluation as normal, then holds at the production gate while releases queue in order.

Two refinements. Distinguish a freeze from an emergency freeze: the former holds planned releases while allowing approved hotfixes, the latter blocks everything. And when the freeze lifts, resist releasing the queue automatically. Five releases back to back with no observation window is five uncontrolled changes, and if quality drops you won't be able to attribute it.

Deployment starts the measurement period rather than ending it. Harness quality degrades without any deployment occurring at all, because the inputs change, the corpus ages, and the world moves on.

Continuous online evaluation

AgentCore Evaluations online mode samples live traces and scores them continuously using the same evaluators the pipeline gate used, publishing into CloudWatch alongside AgentCore Observability so quality sits next to latency and error rate on one dashboard.

That dashboard is fed by telemetry you do not have to build. AgentCore emits distributed traces, structured span-level logs, and metrics in OpenTelemetry format, routed to CloudWatch by default, with built-in metrics for session count, latency, duration, token usage, and error rate. The GenAI Observability console organizes it into Agents, Sessions, and Traces views. Agents running outside AgentCore Runtime get the same treatment by including the AWS Distro for OpenTelemetry, and the telemetry exports to Datadog, Grafana Cloud, or Elastic Observability without additional instrumentation. 

One setup step catches every team once. CloudWatch Transaction Search has to be enabled in the account before AgentCore spans and traces are visible at all. It's a one-time action, easy to miss because nothing errors, and the symptom is an observability stack that looks configured and shows no traces. Put it in the account bootstrap next to the service-linked role.

The sampling rate is a cost decision. One to five percent is typical, and it pays to weight it toward the sessions where quality matters most. Sessions that ended in escalation, sessions with unusually long trajectories, sessions from high-value tenants, and sessions served by a canary all deserve heavier sampling than routine traffic.

Alarm on the rolling mean rather than individual scores, over a window long enough to be stable. Seven days is a reasonable default for gradual drift, with a four-hour window during canary phases. That seven-day window catches the failure mode no per-release gate can see: a slow decline in which every release passed its gate and the cumulative effect is a materially worse agent.

Online evaluation has a second output that is easy to overlook. It produces a continuously growing corpus of real production sessions with quality scores attached, which is the raw material for suite maintenance. Low-scoring sessions, triaged and curated, become the regression scenarios that keep the suite representative. 

Feedback attribution

Explicit user feedback is the highest-quality signal available, and it's routinely wasted by not being joinable to a release.

Every feedback event needs the manifest version, the session identifier, and a trajectory reference. Without the manifest version you can see that satisfaction dropped last week but not which of four releases caused it; with it, you compute a feedback rate per version and compare directly.

Be careful with the statistics. Feedback skews heavily negative, volume is low relative to traffic, and the population choosing to give feedback on a canary is not a random sample of anything. Treat a feedback rate change as a trigger to investigate rather than an automated rollback signal, except at large sample sizes with a sustained effect. 

Cost as a release signal

Cost regression is the quality regression teams find last, usually on an invoice, and it's entirely preventable with a release gate.

A regression that increases tool calls per session from four to seven adds seventy-five percent more model invocations, and the token bill rises by considerably more. Every turn resends the accumulated transcript, so cumulative input tokens grow with the square of the turn count rather than linearly, making a four-to-seven shift closer to a threefold increase. Prompt caching softens the curve where the prefix is stable without flattening it. Latency rises too, and no quality metric need move, because the agent is still right by a more expensive route.

Track three metrics per manifest version, gate on them through the budgets block in Gating a non-deterministic system, and alarm on them during canary. 

• Mean tokens per session, split into input and output, since they are priced differently and a change that inflates context is a different problem from one that inflates generation.

• Mean tool calls per session, which is the leading indicator, since it moves before token count in most regressions and is cheap to compute from trajectory data you already have.

• Cost per successful session, which is the metric that actually matters. A version costing fifteen percent more per session while resolving thirty percent more of them without escalation is an improvement, and a raw cost metric would have flagged it as a regression.

That last point is why cost belongs beside quality rather than in a separate finance review. Cost is only interpretable relative to outcome, and the delivery pipeline is the one place both get measured against the same artifact. 

Two of the three are already emitted for harnesses on AgentCore. Token usage is a built-in metric rather than something you instrument, and tool-call counts come out of trajectory data in the traces, so the work is aggregation and attribution by manifest version.

Closing the loop: recommendations and validated change

Everything so far describes a pipeline that reacts to changes engineers decided to make. AgentCore optimization, in preview since April 2026, adds the step before that: deriving the change from what production already showed you.

It has three parts designed to cycle. Recommendations analyze real agent traces and evaluation output against a target evaluator you name, identify the failure patterns behind a score, and produce an optimized variant of the system prompt or tool descriptions. Configuration bundles version that variant. A/B testing validates it on live traffic. The result becomes the evidence for the next recommendation. 

Read against this module, that loop is the missing half of suite maintenance. Online evaluation already established that low-scoring production sessions are the raw material for regression scenarios. Recommendations use the same material to propose the fix, so one body of production evidence drives both what you test and what you change.

Two cautions, and the first matters more than the capability does.

• A recommendation is a hypothesis, not a validated change. It is generated against one target evaluator, so it optimizes for that evaluator and can regress others. It enters the pipeline at the same gate as a hand-written instruction edit, with the full suite, the repeats, and the baseline comparison. Nothing about its origin earns it a shortcut.

• Optimizing one evaluator at a time compounds into overfitting faster than manual tuning does, because it is so much cheaper to run. Run separate recommendation jobs against different evaluators and choose on the balance you actually want, and keep the held-back validation set genuinely held back. 

The prerequisite is instrumentation, the theme running through this section. Recommendations need traces, which need Observability enabled and Transaction Search turned on, and evaluation output, which needs online evaluation configured. Teams that built the observability foundation for their own reasons find this already available. Teams that skipped it find the loop has nothing to run on.

Incident response

Write the runbook before you need it. Under pressure the sequence that works is: identify the deployed manifest version from any trace, diff it against the previous manifest, choose between rollback, flag kill switch, and forward fix, then execute.

The flag kill switch is often the right first action and consistently under-used. If the regression traces to one capability, disabling that flag takes seconds and preserves the rest of the release, whereas a rollback reverts the entire manifest and takes any unrelated fixes with it. 

Once things are stable, the mandatory follow-up is a regression scenario. Every incident should produce at least one new case, carrying the input that triggered it and the expected correct behavior. A suite that does not grow after incidents will let the same defect through again, and building this habit pays off faster than almost anything else in the discipline.

Table 13. Failure modes worth recognizing early, ordered roughly by how often they show up.

MLOps applies engineering practice to the machine learning lifecycle so models reach production reliably and stay reliable. Agent MLOps keeps every one of those commitments and shifts the object they apply to, because the artifact you version is not a trained model. It is the harness.

That reframing does most of the work in this module. The harness is the orchestration, instructions, tools, retrieval, memory, and guardrails wrapped around a model you procure rather than build, and published benchmarks show this surrounding software carries far more of the outcome than model selection does. If most of the quality lives in the harness, most of the release risk does too.

Declare the harness in code, through AWS CDK, HashiCorp Terraform, or Pulumi, because infrastructure code is what makes environments comparable, preview environments possible, and rollback something you can perform. Capture each release as a content-addressed manifest naming every component by immutable reference, since without it you can't reproduce, attribute, or reverse anything with confidence.