AI agent exposure and integration on AWS

An agent that runs perfectly in isolation delivers nothing until something calls it. The interface layer decides whether that agent becomes a product or stays a demo, and it is where most of the operational work in a production agent deployment concentrates. Reasoning quality is what the team argues about in review. Exposure is what pages someone at three in the morning.

The reason is worth stating plainly, because it shapes everything that follows. Agent traffic breaks several assumptions that ordinary API infrastructure is built on. Requests are large, sometimes hundreds of kilobytes of accumulated context rather than a few hundred bytes of JSON. Responses take seconds instead of milliseconds, and they arrive incrementally. The same input can produce different output, which quietly invalidates every caching and testing habit you have. The most interesting part of the request payload is free text, which means schema validation tells you nothing about whether it is safe. And the cost of serving a request is not a rounding error, so throttling stops being a stability control and becomes a budget control.

This module works through the exposure surface in the order you will actually encounter it. It starts with the four patterns for reaching an agent, then goes deep on API design under agent-sized payloads, protocol selection beyond REST, the structure of the AWS Lambda tier that sits behind the gateway, gateway protection, logging, reliability across regions, invocation and streaming patterns, caching, progressive rollout and multi-tenant exposure. Partner tooling appears where it changes a decision rather than in a catalogue at the end, because the useful question is never whether a product exists; it is whether it removes work you would otherwise repeat.

Before designing an interface layer, identify the callers and what they need. A developer-facing agent answering IDE queries has different latency expectations, authentication mechanisms and payload characteristics than one processing overnight batch jobs or backing a customer-facing chat surface. Four patterns cover almost every case, and most production systems end up using two or three of them at once.

Direct API invocation

In the direct pattern, the caller invokes the agent through an AWS API, typically InvokeAgent or InvokeAgentWithResponseStream on Amazon Bedrock. The caller supplies the agent identifier, an alias that resolves to a specific version, a session identifier for conversational continuity and the input text. Amazon Bedrock returns the response either as a single payload or as a stream of events.

This works when the caller is an internal service inside your AWS account boundary, holds the right IAM permissions and can use an AWS SDK. AWS Lambda functions, Amazon ECS tasks and AWS Step Functions state machines are the usual direct callers. Skipping the gateway removes a network hop and a set of moving parts. What you give up is the indirection: the caller now knows your Amazon Bedrock resource identifiers, and access control is IAM only, with none of the richer options a gateway offers.

Direct invocation is the wrong choice for anything outside your account. It requires AWS credentials, couples the caller to your resource naming, and gives you no place to put per-caller quota, request shaping or content policy. There is also a subtle cost. When callers invoke Amazon Bedrock directly, every one of them ends up reimplementing retry, fallback and error mapping, and they will each get it slightly wrong in a different way.

Gateway-mediated invocation

Here callers reach the agent through Amazon API Gateway, which authenticates, throttles, validates, and forwards to an AWS Lambda function that invokes the agent. The gateway and function together add roughly 10 to 50 milliseconds, which is noise next to a multi-second model call. In exchange, you get usage plans and API keys, TLS termination, request validation, CORS for browser clients, access logging, and a clean separation between the contract you publish and the implementation behind it.

Amazon API Gateway offers two API types worth considering. REST APIs carry the full feature set: usage plans, API keys, JSON Schema request validation, mapping templates, Amazon Cognito user pool authorizers, AWS WAF association and per-method throttling. HTTP APIs cost less and add less latency, with a reduced feature set, which works when validation and quota live in the function instead. For agent endpoints, the deciding factor is usually usage plans. If you need per-caller quota enforced before the model is invoked, you need REST APIs.

The integration function is where customization belongs: enriching the request with caller context the client should not control, reshaping the response for different client types, emitting audit records, and mapping model errors to HTTP semantics that clients can act on. Section 4 covers how to structure that function once you have more than one kind of caller.

One boundary is worth drawing early. If the agent is an internal capability, Amazon API Gateway on its own is the whole story. If the agent is a product that external developers build against, the gateway covers enforcement but not the surrounding experience: self-service key provisioning, interactive documentation, per-consumer analytics and a place for a developer to see why their last request was rejected. Building those is a project, not a configuration change.

Tyk Technologies, available in AWS Marketplace, is the partner option for this case. It runs in front of or alongside Amazon API Gateway and adds a developer portal with self-service onboarding, an OpenAPI-driven documentation surface and usage analytics per consumer. For agent APIs specifically, two of its capabilities do real work. Its plugin pipeline attaches logic at several points in the request lifecycle, before and after authentication and around the upstream call, which is a clean place to translate a partner's payload shape into your canonical agent request rather than accumulating per-partner branches in your integration function. And its GraphQL support means a single control plane covers both the REST and GraphQL surfaces discussed in section 2.3 instead of splitting policy across two systems. The decision is about who consumes the agent and what they need to see, not about which gateway enforces better.

Event-driven invocation

In the event-driven pattern, invocation is triggered by a message on a queue or an event bus rather than by a synchronous call. Amazon SQS triggers an AWS Lambda function carrying the agent input and its context. Amazon EventBridge routes events from source systems to agent-invoking functions based on event patterns, which lets you add an agent to an existing event flow without touching the producers.

This fits anywhere the caller does not need an answer in the same breath. Document processing over a large corpus, overnight enrichment jobs, change-data-capture workflows that react to database updates. The pattern also gives you backpressure for free, because Amazon SQS bounds the invocation rate through the event source mapping concurrency setting, and it gives you retry and dead-letter handling without writing any of it.

There is a second benefit that matters more than it first appears. Decoupling the producer from the invocation means you can change the model, the prompt, the agent version, or even the entire agent framework without coordinating a release with every team that emits events. In a 14-module series about systems that change constantly, that property is worth a lot.

MCP server interfaces

Amazon Bedrock AgentCore can expose an agent as an MCP server: an endpoint that speaks the Model Context Protocol and can be discovered and called by any MCP-compatible client. To the caller, it looks like any other MCP tool, with a name, a description, and an input schema. Whether an Amazon Bedrock agent, an AWS Step Functions workflow or something else entirely sits behind, it is not the caller's concern.

This matters most in multi-agent architectures. An orchestrating agent can discover and call specialized agents through their MCP interfaces without compiling knowledge of how they are built. It is also the right answer for developer tooling. IDEs, CLI tools and internal developer portals increasingly speak MCP natively, and exposing an agent that way means you write no client integration at all.

The tradeoff is that MCP gives you a protocol, not a policy layer. Rate limiting, quota, tenant isolation and content policy still have to live somewhere. Amazon Bedrock AgentCore Gateway and Amazon Bedrock AgentCore Identity handle a good deal of this but treat an MCP endpoint the way you would treat any other public interface rather than assuming the protocol brought its own guardrails.

Table 1. The four exposure patterns. Most production systems run two or three of these at once, one per caller class.

Most of what you know about designing an HTTP API still applies. The parts that do not apply are the ones that catch teams out, so this section concentrates on them.

What makes an agent API different

Four properties separate an agent's endpoint from an ordinary microservice endpoint, and each one has a design consequence.

• Requests are large and grow over time. A conversation that starts at 2 KB is 40 KB by turn ten and 200 KB by turn fifty, because the context accumulates. Your p99 request size is not a stable number; it is a function of session age.

• Responses are slow and incremental. Time to first token matters more than total duration for anything a human is watching, which means the transport must support partial delivery, or the user experience is bad regardless of how fast the model is.

• The same request can produce different responses. Retries are not free, conditional requests are meaningless, and any cache you build must reason about semantic equivalence rather than byte equality.

• The interesting payload is free text. JSON Schema validation confirms the shape and tells you nothing about the content, which is where every serious attack against an agent lives.

The base design for a synchronous endpoint is unremarkable: a POST to a path that identifies the agent and optionally the session, a body carrying the message and any caller-supplied context, and a response carrying the answer, citations from retrieved chunks, token usage and the session identifier for the next turn. Amazon API Gateway applies a 29 second integration timeout by default on REST APIs, and HTTP APIs cap at 30 seconds. If your p99 exceeds that, you need the asynchronous pattern in section 6, not a quota increase request.

Two response fields deserve to be in the contract from day one, because retrofitting them later is expensive. Return the token usage for the turn so callers can reason about cost and about how close the session is to the context ceiling. And return an explicit termination reason, distinguishing a completed answer from one truncated by an output token limit, stopped by a guardrail or cut short by a tool failure. Clients that cannot tell these apart will treat a truncated answer as a complete one, and that is how agents give confidently wrong advice.

Carrying large context over HTTP

This is the design problem that surprises teams most often. A modern context window holds 200,000 tokens or more, which is roughly 800 KB of plain text. The transport you are putting it through was designed for form posts. Every hop on the path has its own ceiling, and the smallest one wins.

Figure 1. Payload ceilings on the request path and the three strategies for carrying agent-sized context through an HTTP gateway.

Amazon API Gateway accepts a 10 MB payload on both REST and HTTP APIs. AWS Lambda accepts 6 MB on a synchronous invocation and only 256 KB on an asynchronous one. Amazon SQS and Amazon EventBridge both stop at 256 KB per message. So the practical ceiling on a gateway-mediated synchronous path is 6 MB, and on an event-driven path it is 256 KB unless you do something about it. Neither number is close to what a full context window can hold, and the asynchronous number is smaller than a single verbose Terraform plan.

Three strategies cover the space, and mature systems use all three for different traffic.

Inline body

Serialize everything into the POST body. This is correct up to roughly 250 KB, which covers short-lived chat turns, single-document analysis and modest tool results. Past that it starts costing you in places that are not obvious: base64 encoding inflates binary attachments by a third, JSON parsing of a multi-megabyte body burns AWS Lambda duration you are paying for, and if access logging captures the body, you have just multiplied your Amazon CloudWatch bill by the size of your context.

The failure mode worth naming is the client that re-uploads the entire conversation on every turn. It works in testing, where conversations are three turns long, and it degrades continuously in production as sessions age. Instrument request size as a percentile distribution, not an average, and alarm on the p99 crossing half your ceiling.

Context by reference

The client uploads context to Amazon S3 using a presigned URL, then posts only the object key and a content hash. The gateway never sees the payload, so no ceiling on the HTTP path applies. This is the right answer for log bundles, infrastructure plans, build artifacts and multi-file diffs, which covers most of what an agent operating over existing systems actually receives.

Design the presigned URL narrowly. Scope it to a single key under a tenant-specific prefix, set a short expiry, and constrain the content length so a client cannot upload something unbounded. The code below issues a scoped upload URL and returns it alongside a reference token the caller passes back on the invocation request.

Listing 1. Issuing a scoped presigned upload and dereferencing it server side. The tenant prefix check is the control that stops one tenant from referencing another tenant's uploaded context.

Two details in that listing are load-bearing. The prefix check is what prevents a caller from passing someone else's reference, and it belongs on the server because the reference travels through the client. The hash check turns a silent substitution into a loud failure, which matters because the object is mutable between upload and invocation unless you make it otherwise.

The cost of this pattern is a second round trip before the agent starts working. For interactive traffic that is often unacceptable, which is why it pairs best with the asynchronous pattern or with clients that can upload ahead of time.

Server-side session

The context accumulates on the server and the caller posts only the new turn plus a session identifier. This is the pattern that scales best over long conversations, because request size stops being a function of session age. Amazon Bedrock AgentCore Memory handles this natively, and Amazon DynamoDB works fine when you want to own the representation.

What server-side sessions buy in payload size they charge back in state management. Session affinity now matters, because a request routed to a region that does not hold the session gets a cold conversation. Truncation becomes invisible: when the accumulated context exceeds the window, something has to be dropped or summarized, and if the caller cannot see that happening they will not understand why the agent forgot what it was told twenty turns ago. Surface a context utilization figure in the response and let clients decide when to start fresh.

Choosing between the three

Measure the p99 serialized context, not the mean. Under 250 KB, keep it inline and stop thinking about it. Between 250 KB and 6 MB, inline still works but log the body by reference only. Over 6 MB, or on any asynchronous path over 256 KB, you need the reference pattern. If context grows with every turn, you need server-side sessions regardless of the current size, because the trend is what will page you.

Protocols beyond REST: gRPC and GraphQL

REST over HTTP/1.1 is the default for good reasons: universal client support, human readable payloads, mature tooling and an OpenAPI contract anyone can consume. For agent traffic it also has two specific weaknesses. Streaming is an add-on rather than part of the protocol, and JSON is an expensive representation for the volume of text an agent moves. Two alternatives are worth understanding well enough to choose between.

Figure 2. How the three protocol families land on AWS, and the concerns none of them solve for you.

gRPC over HTTP/2

gRPC frames binary Protocol Buffers over multiplexed HTTP/2 streams. Bidirectional streaming is part of the protocol rather than something layered on top, which means a server can push tokens while the client pushes a cancellation or a mid-stream correction on the same connection. For agent-to-agent traffic inside a VPC, where the same context may traverse several hops, the combination of binary framing and header compression is a real reduction in bytes on the wire and in serialization CPU.

The constraint that shapes every gRPC decision on AWS is that Amazon API Gateway does not proxy gRPC. You terminate gRPC on an Application Load Balancer with a gRPC protocol target group, pointing at Amazon ECS or Amazon EKS. Amazon VPC Lattice is the other option, and it fits well for service-to-service calls that need per-service auth policies without a mesh. AWS Lambda is not a gRPC target, so choosing gRPC means choosing long-running compute for the integration tier.

Define the service so streaming is the default rather than an alternate method. A single server-streaming RPC handles both cases, because a non-streaming client simply reads until the stream ends.

Listing 2. A gRPC service definition for an agent. Modelling tool calls and citations as first-class stream events lets clients render intermediate progress instead of staring at a spinner.

Browsers cannot speak gRPC directly. If a browser client is in scope you need gRPC-Web and a translating proxy, at which point you are running the translation layer you were trying to avoid. The honest rule is that gRPC belongs behind your perimeter, and REST or GraphQL belongs in front of it.

Choosing gRPC also changes where traffic management lives, and this consequence is easy to underestimate. Amazon API Gateway is out of the picture, so throttling, retry policy, authentication, request transformation and observability all have to be expressed somewhere else. The default answer is to write them into each service, which works until you have four services and four slightly different retry implementations.

Kong, available in AWS Marketplace, is the partner option when the integration tier runs on Amazon EKS. Its ingress controller proxies gRPC natively over HTTP/2 and expresses rate limiting, authentication, request size limits and circuit breaking as Kubernetes resources, so agent endpoints and the conventional services beside them share one policy model and one set of dashboards. For an agent tier this matters more than for ordinary services, because the controls in sections 3.2 and 3.3 (per-tenant rate limiting, body size ceilings, structured 429 responses) are exactly the ones you lose when you leave Amazon API Gateway, and reimplementing them per service is how they drift apart. Amazon VPC Lattice covers similar ground without a cluster-scoped component, and is the better fit when services are spread across Amazon ECS, Amazon EKS and Amazon EC2 rather than concentrated in one cluster.

GraphQL through AWS AppSync

GraphQL exposes one endpoint and lets the caller specify exactly what it wants back. For agent integrations the compelling case is composition: a dashboard that needs the agent's answer, the user's recent deployments, the current incident list and the team's alert configuration can fetch all of it in one round trip, with the agent invocation running as one resolver among several. On a mobile client over a slow network, that difference is the whole user experience.

AWS AppSync is the managed implementation. Resolvers attach to AWS Lambda, HTTP endpoints, Amazon DynamoDB tables and other data sources. Authorization is configured per field, which is genuinely useful for agent APIs, because you can allow a broad audience to read cached agent summaries while restricting the field that triggers a new invocation to callers holding a specific Amazon Cognito group. Subscriptions push incremental results over WebSocket, which is how you stream tokens through GraphQL.

Listing 3. An AWS AppSync schema for streamed agent responses. The mutation that publishes deltas is restricted to IAM so only the resolver can call it, and the sequence number lets clients reassemble out-of-order delivery.

Before committing to GraphQL, check whether you actually need a query language. A large share of the teams who reach for it want one thing: fewer round trips. GraphQL delivers that and hands you a schema to govern, a depth limiter to tune and a caching story you have to build yourself.

KrakenD, available in AWS Marketplace, is the partner option when composition is the whole requirement. It assembles responses from several backends into one payload from declarative configuration, with no resolver code and no schema to version. For an agent integration the useful property is that it aggregates asynchronously and degrades per backend, so a client screen that combines the agent's answer with two conventional services still renders when one of the three is slow, and the agent call (the slowest and most expensive leg) does not block the rest. Its stateless design also means the aggregation layer scales independently of the agent tier. Prefer GraphQL when clients genuinely need different shapes from the same data. Prefer an aggregating gateway when every client wants the same shape and you are only trying to collapse round trips.

Two GraphQL-specific risks apply harder to agent APIs than to ordinary ones. Query cost is caller-controlled, so a deeply nested query that triggers several agent invocations is a denial-of-wallet vector rather than merely a slow request. Set depth and complexity limits, and count agent invocations explicitly against the caller's quota rather than counting HTTP requests. The second risk is that HTTP caching does not apply, since everything is a POST to one URL. Whatever caching you want has to be built at the resolver level, which section 8 covers.

Table 2. Protocol selection for agent endpoints. The integration timeout row is usually the one that decides it.

Request validation, idempotency, and cancellation

Amazon API Gateway request validation using JSON Schema models rejects malformed requests before they reach the function, which saves invocation cost and returns consistent errors. Use it, and understand its limit: it validates structure. A schema that requires a string field named message is satisfied by any string, including one engineered to redirect the agent. Structural validation is necessary and never sufficient.

What validation should enforce beyond shape is which fields the caller is allowed to set at all. Tenant identifier, system prompt overrides, model selection, tool allowlists and knowledge base filters must be asserted by your integration tier from the authenticated identity, never accepted from the body. The schema should reject requests containing those fields rather than silently ignoring them, because a rejected request tells you someone is probing and an ignored one does not.

Idempotency needs an explicit design because retries are not free and agent side effects are real. A client that retries a timed-out request against an agent with tool access can trigger the same deployment twice. Require an idempotency key on any request that can produce side effects, store it in Amazon DynamoDB with a conditional write, and return the original result on a repeat rather than re-running.

 

Listing 4. A conditional write claims the idempotency key. The in-progress branch matters: without it, two concurrent retries both miss the completed record and both invoke the agent.

Cancellation is the neglected half of this. When a user closes the browser tab, the model call is still running and still billing. Amazon API Gateway does not propagate client disconnects to an AWS Lambda integration, so on the synchronous path there is nothing to react to. On WebSocket and gRPC paths there is, and you should use it: check for a cancellation signal between streamed chunks and stop consuming the model stream when it fires. For long asynchronous jobs, expose an explicit cancel endpoint that writes a tombstone the worker checks between steps.

Authentication and authorization

Amazon API Gateway offers three mechanisms that suit agent endpoints, and the choice follows from who the caller is rather than from any property of the agent.

IAM authorization fits service-to-service calls inside your organization. The caller signs with AWS Signature Version 4 using a role granted execute-api:Invoke on a specific API, stage and resource. It adds no latency at the gateway, and every call lands in AWS CloudTrail without extra work. The limitation is that IAM tells you which role called, not which end user the call was made on behalf of, so a shared service role means you lose per-user attribution unless you carry it separately.

Amazon Cognito user pool authorizers fit user-facing integrations. The caller obtains a JWT after authenticating with a password, a social provider or SAML federation, and passes it in the Authorization header. The gateway validates it against the user pool without invoking a function, adding a few milliseconds. Claims from the token reach your function through the request context, which is where tenant identifier and group membership should come from.

AWS Lambda authorizers fit everything else: a non-Cognito identity provider, API key validation with business rules, or authorization that depends on request properties rather than identity alone. The authorizer returns an IAM policy document permitting or denying the call. Results are cached by a configurable key for a configurable duration, which is the single most important performance knob here. An uncached authorizer adds a full function invocation to every request.

One agent-specific caution about authorizer caching. If your policy depends on anything that changes faster than the cache TTL, such as a per-user token budget that resets or an entitlement that can be revoked mid-session, the cached policy will keep approving requests after the underlying decision has changed. Either keep the TTL short, or enforce the volatile part of the decision in the integration function where you can read current state.

Usage plans, rate limiting, and the cost dimension

Usage plans define sustained rate, burst capacity and quota per API key. For agent endpoints they do two jobs. They protect the model from traffic that would exceed your provisioned or on-demand throughput, and they stop one caller from consuming a share of capacity that belongs to everyone else.

Table 3. A usage plan hierarchy. The internal plan lives on a separate API rather than a separate key, so a misconfigured public key can never inherit it.

When a caller exceeds any limit, Amazon API Gateway returns 429 without invoking the function or the model, which is exactly where you want that decision made. Set the values against real throughput. Amazon Bedrock enforces both requests per minute and tokens per minute per model per region, and the token limit is usually the one you hit first. Provisioned Throughput reserves model units for your exclusive use and is the right answer for predictable production load. Cross-region inference profiles are the right answer for spiky load, because they spread a single request across the regions in a geography rather than requiring you to reserve for the peak.

Usage plans are enforcement, not communication. A caller who hits their quota gets a 429 and no way to see how close they were, how much of the window is left, or which of their integrations consumed it. Amazon CloudWatch answers those questions for you and not for them, and the gap is filled by support tickets.

Tyk Technologies closes that gap when the agent is consumed by people outside your team. Its developer portal gives each consumer a self-service view of their own quota consumption and key lifecycle, including rotation and revocation, without you building a billing-adjacent interface. Enforcement stays on Amazon API Gateway where it belongs, ahead of the model invocation. For an agent endpoint the portal has a second use worth noting: it is a reasonable place to publish the token accounting described below, so consumers can see that their cost is driven by context size rather than request count. That single piece of visibility prevents a recurring conversation in which a customer insists they are well inside their rate limit and cannot understand why their bill moved.

Requests per second is a poor proxy for load on an agent endpoint, and this is worth stating clearly. One caller sending 10 requests per second with 500 token prompts and another sending 2 requests per second with 100,000 token prompts look very different to the gateway and nearly identical to the model. Rate limits alone will let the second caller exhaust your token budget while comfortably inside quota. Track tokens per tenant in the integration tier, publish them as an Amazon CloudWatch metric dimensioned by tenant, and enforce a token budget alongside the request quota. The gateway limit is your first line, not your only one.

Protecting the gateway with AWS WAF

An agent endpoint attracts the same traffic every public API attracts, plus a category that is specific to it. Understanding which layer stops which class of problem prevents both false confidence and wasted effort.

Figure 3. What each layer of the path can actually stop. The important column is the second one in each box.

What AWS WAF is good at

Attach a web ACL to the Amazon API Gateway stage, or to the Amazon CloudFront distribution in front of it, and you get several controls that are genuinely valuable for agent traffic.

• Size constraints: Declare the maximum body you accept and reject anything larger before it reaches your function. For an agent this doubles as a cost control, because an oversized body is an oversized prompt.

• Rate-based rules: Aggregate on IP, on a header, on a query argument, on a JA4 TLS fingerprint or on a combination. The combination matters: rate limiting on IP alone is close to useless when your enterprise callers sit behind corporate NAT and your consumer callers sit behind carrier-grade NAT.

• AWS WAF Bot Control: Identifies known scrapers, headless browsers and automation frameworks. Agent endpoints are attractive scraping targets precisely because they synthesize information, so a bot that would be harmless against a static API is expensive against this one.

• Managed rule groups and geographic restriction: The baseline core rule set and known bad inputs group cost little and catch the background noise of the internet.

The body inspection limit, and why it matters here

This is the detail that changes how you think about AWS WAF for agents. AWS WAF does not inspect the entire request body. For Amazon API Gateway and Amazon CloudFront the default is the first 16 KB, raisable to 32, 48 or 64 KB through the web ACL association configuration at additional cost. For Application Load Balancer and AWS AppSync the limit is fixed at 8 KB and cannot be raised.

Put that next to a 200 KB agent request. Any content-based rule you write inspects somewhere between four and thirty percent of the payload. The remainder passes unexamined. An attacker who understands this simply pads the front of the body with benign text. Worse, the oversize handling setting on a body rule defaults to continuing evaluation with what was inspected, so the request is allowed rather than flagged.

There are three sane responses. Set the inspection limit to 64 KB on the agent endpoint and accept the cost, which is small relative to a model invocation. Set oversize handling to MATCH on a rule that counts rather than blocks, so you can see how much traffic exceeds inspection. And most importantly, stop expecting AWS WAF to be your content control. It is a volumetric and structural control that happens to also do some pattern matching.

Listing 5. A web ACL tuned for an agent endpoint. Note the raised inspection limit and the composite rate key. Rules that inspect content start in Count mode, because a false positive here blocks a legitimate user's entire conversation.

Where prompt injection is actually caught

It is tempting to write AWS WAF rules matching phrases like "ignore previous instructions". Do not build your defense on that. The space of effective injections is unbounded and expressible in any language, and a regex that catches last month's phrasing catches nothing this month while generating false positives against users discussing prompt injection legitimately. String matching against a semantic attack is a losing position.

Injection is caught in two places instead. The first is structural, in your integration function. Untrusted content must not be able to forge the structure of the prompt. Strip or escape role markers, conversation delimiters and anything resembling a tool result envelope from caller-supplied text, and wrap untrusted spans in explicit boundaries the model has been instructed to treat as data. This catches the mechanical attacks cheaply and deterministically.

The second is semantic, in Amazon Bedrock Guardrails. The prompt attack filter scores input tagged as untrusted, and the tagging is the part teams get wrong. If you send the whole prompt as one undifferentiated block, the filter has to evaluate your own system instructions as potential attacks, which produces false positives and pushes teams to lower the threshold until the filter does nothing. Tag the untrusted spans specifically.

Listing 6. Screening only untrusted spans with the ApplyGuardrail API. The exception handler is the important line: a guardrail that times out must not become an implicit allow.

Notice what neither layer covers. Injection that arrives through a retrieved knowledge base chunk, an MCP tool result or a prior turn already in the session never crosses the gateway at all. It enters below AWS WAF entirely. That is why the screening call above covers retrieved content and tool output, not just the user message, and it is why content ingested by the Module 9 pipeline needs its own controls rather than inheriting yours.

Logging requests and responses without creating two new problems

Access logging an ordinary API is uncontroversial. Access logging an agent is a decision with cost, privacy and compliance consequences, and the default configuration is wrong for almost every agent deployment.

Figure 4. What one agent turn actually contains, and the three sinks it should be split across.

The volume problem

Do the arithmetic before choosing a logging strategy. A moderately loaded agent turn carries a 2,000 token system prompt, 18,000 tokens of session history, 12,000 tokens of retrieved chunks, 6,000 tokens of tool output, a 300 token user message and a 1,500 token response. That is around 40,000 tokens, or roughly 160 KB of UTF-8. Log the request and response bodies in full and one million turns a month produces about 160 GB of log ingestion.

At Amazon CloudWatch Logs ingestion pricing that is a meaningful line item on its own, and it is not the whole cost. Storage accrues monthly. Subscription filters that fan logs out to analysis pipelines charge again. Amazon CloudWatch Logs Insights queries are priced on data scanned, so the log group you built for debugging becomes expensive to actually use. Teams commonly discover that observing the agent costs more than running it, and the discovery usually arrives with the monthly bill.

The comparison with an ordinary API is stark, and it is worth internalizing. A typical REST endpoint logs a body of a few hundred bytes. An agent logs a body three orders of magnitude larger. Any logging habit built on the first assumption fails on the second.

Table 4. Log sources on an agent path. The execution log row is where most surprise bills originate.

The sensitivity problem

Volume is the cheaper of the two problems. The prompt for a single turn is one of the most concentrated collections of sensitive data your system will ever assemble in one place, and it is assembled precisely so it can be sent somewhere and logged.

• Users paste things. Free-text fields collect credentials, API keys, connection strings, customer records and screenshots of dashboards. This is not an edge case, it is the median behavior of a user who wants help with a problem.

• Retrieved chunks belong to a tenant. The prompt contains knowledge base content the tenant is authorized to see. A shared debug log group hands that content to every engineer with read access, which is a cross-tenant disclosure even though no request crossed a tenant boundary.

• Tool results carry production data. An agent that queries a database or an incident system pulls real records into the prompt, and those records land in the log alongside everything else.

• Erasure obligations reach the log store. A deletion request under GDPR or similar regimes covers logs, not just the primary datastore. If your audit log is immutable by design and your privacy commitment says erasable, you have a conflict that has to be resolved in architecture rather than in policy language.

A tiered logging design

Split the problem into three sinks with three different retentions and three different access policies. Almost every operational question is answered by the first tier, which contains no content at all.

Tier one, metadata. Trace identifier, tenant, caller principal, agent alias, model identifier, latency broken down by phase, input and output token counts, guardrail verdict, cache outcome, finish reason, error class. Around 1 KB per turn. Goes to Amazon CloudWatch Logs as structured JSON, retained thirteen months, queried with Logs Insights. This tier answers latency questions, cost questions, error questions and capacity questions.

Tier two, hashed content. A SHA-256 over the normalized prompt and over the response, plus retrieved chunk identifiers, tool names and hashes of tool arguments. Enough to prove what ran and to detect that two turns were identical, not enough to read anything. Around 3 KB per turn, written to Amazon S3 partitioned by date and tenant, retained according to contract. This tier answers audit questions.

Tier three, sampled full fidelity. The complete request and response after redaction, kept for a small percentage of traffic plus every turn that errored or tripped a guardrail. Separate Amazon S3 bucket, AWS KMS encryption with a distinct key, a bucket policy that grants access to a named role rather than to the engineering organization, and a 30 day lifecycle expiry. This tier answers quality questions, which are the only questions that genuinely need content.

 

Listing 7. Tiered emission. Everything gets metadata and a hash; only a sampled slice and the failures get redacted content.

Two practical notes on that code. Amazon Comprehend PII detection is priced per unit of text and adds latency, so run it on the sampled path rather than on every turn, which is what the ordering above accomplishes. And Amazon Bedrock Guardrails sensitive information filters can mask on the way through the model, which handles the response side without a second service call. Use both: Guardrails on the live path, Amazon Comprehend on the archival path.

Finally, resist the reflex to fix all of this with a longer retention period and a bigger log group. The engineering decision that pays off is deciding, per field, whether you need the value, a hash of the value, or nothing at all. Most fields turn out to be nothing at all.

Once you have more than one kind of caller, a question arrives that has no obvious answer: does each consumer get its own AWS Lambda function, or does one function serve everything? Teams tend to answer it by accident, usually by copying the first function and editing it, and then live with the result for years.

Figure 5. Thin per-consumer adapters over one shared agent core.

One function or many

Consider what actually differs between a synchronous REST caller, an Amazon SQS consumer and an Amazon EventBridge target. The event shape differs. The response contract differs, because one returns a body and the others return nothing. The timeout differs by an order of magnitude. The memory profile differs, because streaming and batch processing have different working sets. The concurrency behavior differs, because one is user-facing and needs headroom while the other is throughput-oriented and should be capped. The IAM permissions differ, because the queue consumer needs sqs:DeleteMessage and the REST handler does not.

What does not differ is everything that matters about the agent: how you resolve a caller to a tenant, how you assemble context, which model you call, how you retry, how you apply guardrails, how you shape citations, how you account for tokens.

That split is the design. Per-consumer adapters that are thin enough to read in one screen, over a shared core that owns everything about the agent. The test for whether you have drawn the line correctly is simple: if changing something about the agent's behavior forces you to edit more than one function, the line is in the wrong place.

The single-function approach fails specifically because it collapses the operational envelope. One function means one timeout, so the interactive path inherits the batch path's 900 seconds and a hung request occupies a concurrency slot for fifteen minutes. One function means one reserved concurrency pool, so a batch backlog starves interactive traffic. One function means one execution role, so the REST handler carries queue delete permissions it will never use. And one function means one deployment, so a change for the batch path can break the interactive path on the same release.

The adapter and core split in practice

An adapter does three things and then gets out of the way: parse its event shape into a canonical request, call the core, and encode the result into its response contract. Errors are translated at that boundary too, because an HTTP caller needs a status code and a queue consumer needs a decision about whether to retry.

Listing 8. Two adapters, one core. The synchronous adapter maps exceptions to status codes; the queue adapter maps them to retry decisions, and reports partial batch failures so one poisoned message does not replay a whole batch.

The queue adapter's handling of ContentBlocked deserves attention. A content policy block is deterministic, so retrying produces the same block three more times and then a dead-letter entry that looks like an infrastructure failure. Terminal failures must be recorded as terminal at the adapter boundary. Only transient conditions belong in the batch item failure list.

Packaging the shared core

Two mechanisms exist for sharing code across functions, and they behave differently enough that the choice matters.

AWS Lambda layers attach a versioned artifact to a function without including it in the deployment package. Updating the layer and repointing functions at the new version is a configuration change rather than a code deployment, which is fast. The catch is that layers are versioned by an ARN with an integer suffix, so promoting a core change means updating every function's configuration, and it is easy to end up with three functions on three layer versions without noticing.

An internal package built into each deployment artifact through your build pipeline makes the dependency explicit and visible in a lockfile. Deployment packages are larger, and updating the core means rebuilding and redeploying every function that uses it. In exchange you get a version pin you can read, review and roll back like any other dependency.

For most teams the internal package is the better default, because it makes core version drift a build-time fact rather than a runtime surprise. Use layers for genuinely static dependencies, such as a large SDK or a set of certificates, where the update cadence is measured in months.

There is a third option that only becomes attractive at a certain scale. When several agents across several teams each need the same routing, fallback and token accounting behavior, a shared library still lets every team upgrade on their own schedule, and the copies drift anyway. The version pin that made the library honest is also what lets three teams run three different retry policies against the same model quota.

Portkey AI, available in AWS Marketplace, is the partner option that turns that shared logic into a network hop with its own configuration instead of a dependency each team pins. Routing rules, provider fallback chains, load balancing across model endpoints, retry policy and per-request token accounting move into the gateway, so a change to fallback behavior takes effect for every agent without a coordinated release. It also gives you one place to see cost and latency broken down by agent, model and consumer, which is otherwise assembled by hand from several Amazon CloudWatch metric namespaces. The threshold is worth stating plainly: this trade pays off when the number of consuming teams exceeds the number of people who understand the library, and it is a poor trade before then, because you have added a hop, a failure mode and an operational surface to eliminate duplication that does not yet exist.

Whichever you pick, version the core independently and treat a core release like a library release. Semantic versioning, a changelog, and a canary function that exercises the new version against a fixed set of prompts before the interactive adapters take it. The core is where a subtle change in prompt assembly silently degrades answer quality across every consumer at once.

Configuration that must differ per function

The reason to have separate functions is precisely so these settings can differ. If they are identical across your adapters, you have paid the cost of separation without collecting the benefit.

Table 5. The operational envelope is what separation actually buys.

When one function is the right answer

None of this argues for splitting by default. A single function is correct when there is one consumer, when consumers share a latency profile and a permission set, or when the whole system is small enough that the coordination cost of multiple deployables outweighs the isolation benefit. Splitting a function that serves 50 requests a day across two callers is over-engineering.

The signals that you have outgrown one function are concrete. You find yourself branching on event shape at the top of the handler. The timeout is set for the slowest consumer and everyone else inherits it. The execution role has accumulated permissions that only one code path uses. A deployment for one consumer requires regression testing for another. When two or more of those are true, split.

Split by consumer characteristics rather than by feature. One function per API endpoint is a common mistake that produces a dozen near-identical functions differing only in which field they read, all with the same timeout and the same role, with the shared logic copy-pasted rather than extracted. That is the worst of both designs.

An agent endpoint is only as available as the least available thing behind it, and there are more things behind it than behind an ordinary API. Model capacity, a knowledge base, a vector store, several tool endpoints and a session store all have to work for a turn to complete.

Model failover strategies

When a model invocation fails, whether from throttling, a transient service error or a content filter intervention, the integration tier has to decide what happens next. Three strategies compose rather than compete.

Retry with jittered exponential backoff handles transient errors and brief throttling. Use full jitter rather than fixed backoff, because synchronized retries from a fleet of functions reproduce the exact traffic spike that caused the throttling. The constraint is your remaining time budget: on a synchronous path with a 29 second ceiling and a model that typically takes eight seconds, you have room for one retry, not three. Compute the budget from the deadline rather than hardcoding an attempt count.

Fallback to an alternative model trades response quality for availability. Falling back from a frontier model to a faster, smaller one when throughput is exhausted keeps the endpoint answering. Two things make this safe. Mark the response so the caller knows a fallback was used, because a silently degraded answer presented as a normal one is worse than an error. And validate that your prompts actually work on the fallback model, since a prompt tuned for one model's instruction following can behave quite differently on another.

Portkey AI expresses this routing as configuration rather than code. You declare an ordered list of model targets with conditions, and the gateway handles the failover, the retry budget and the load balancing across them, including across providers and across regions. For a team running several agents this removes a class of bug that is genuinely hard to test: the fallback path that was written once, never exercised, and turns out to be broken on the day capacity runs out. It also emits per-attempt telemetry, so you can see how often the primary model is actually failing rather than inferring it from a latency graph.

What a model gateway does not do is make the fallback safe. Prompt compatibility with the secondary model, the response marking that tells callers a fallback was used, and the decision about which degradations are acceptable are all still yours. A gateway that silently swaps models on your behalf produces a quality regression with excellent uptime numbers, which is a harder problem to notice than an outage.

Graceful degradation is what happens when everything fails. Return a structured error that says what happened, whether retrying will help, and roughly when. A caller that receives {"error": "capacity_exhausted", "retryable": true, "retryAfterSeconds": 30} can back off intelligently. A caller that receives a generic 500 will retry immediately and make things worse.

There is an agent-specific option worth adding. When the model is unavailable but the knowledge base is not, you can answer with retrieval alone: return the top matching chunks with their sources and a clear statement that these are search results rather than a synthesized answer. For a documentation or runbook agent this is often genuinely useful, and it is far better than an error page.

Running agents across regions

Multi-region for an agent is not multi-region for a web application, and the difference trips up teams who have done the latter successfully. Amazon Route 53 and a second stack get you a healthy health check and an agent that answers badly.

Figure 6. What replicates across a region boundary and what does not.

What does not cross the boundary

• Model availability. Not every model, and not every version of every model, is available in every region. A failover region that lacks your primary model forces either a model substitution, with the quality change that implies, or an outage. Verify availability per region as part of your deployment pipeline rather than discovering it during a failover.

• Agent and alias identifiers. Amazon Bedrock agents are regional resources with regional identifiers. The same agent definition deployed to two regions produces two different agent IDs. Any caller or configuration that hardcodes an agent ID is a failover bug waiting to happen. Resolve identifiers from configuration keyed by region at startup.

• Knowledge base content. There is no cross-region replication for an Amazon Bedrock knowledge base. Each region needs its own ingestion run against its own vector store, which means two freshness timelines. If the secondary syncs on a slower cadence, a failover moves users to an agent working from older documents, and nothing in the response will indicate that.

• Guardrail configuration. Guardrails are regional and versioned per region. Drift between the two is silent and behavioral: the same input is blocked in one region and allowed in the other. Deploy guardrails from the same source of truth and assert version equality in a health check.

• Provisioned throughput. Model units are purchased per region. Failing over into a region where you hold no reservation puts your entire production load onto on-demand capacity, which throttles under exactly the conditions where you needed the failover.

Session and context across regions

This is the part that determines whether failover is graceful or jarring. Conversational state lives somewhere, and where it lives decides what a failed-over user experiences.

If sessions live in Amazon Bedrock AgentCore Memory, they are regional. A user who fails over starts a new conversation, with no memory of the last twenty turns. For a single-shot query agent that is fine. For a long-running assistant it is a visible regression, and users will report it as the agent losing its mind rather than as a regional failover.

If you own the session store, Amazon DynamoDB global tables replicate it with typically sub-second lag. Sub-second is not zero, and the failure mode is specific: a user who fails over mid-conversation may land on a replica missing their last turn, so the agent responds to the turn before. Write the turn sequence number into the session and have the client send the sequence it believes is current. When they disagree, tell the user the conversation was interrupted rather than silently answering the wrong question.

The data residency question sits on top of all of this and is genuinely constraining. If a tenant's contract says their data stays in the EU, a failover to a US region is a contract breach even though it improved availability. Model residency by tenant, not by application, and be prepared for the answer that some tenants get a single-region service with lower availability because that is what they asked for. Module 12 covers the governance side; here the point is that the routing layer needs to know about it, which usually means residency-aware routing rather than pure latency-based routing.

Two escape hatches before you build a second stack

Cross-region inference profiles let a single request be served from whichever region in a geography has capacity, without you deploying anything additional. This solves throughput and transient model availability, which is what most teams actually need when they start talking about multi-region. It does not solve an outage of your own gateway and function tier, and the request data does leave the source region for the duration of the call, so it interacts with residency commitments.

Active-passive with a colder secondary keeps the second region's gateway and functions deployed and warm while refreshing its knowledge base on a slower cadence. It costs far less than symmetric deployment. It is only honest if you tell the caller during failover that answers may be based on older sources, which is a one-line response field and a banner in the client.

Table 6. Component-by-component replication plan. The idempotency row is the one most often missed, and it is the one with real-world side effects.

Circuit breakers

A circuit breaker stops a failing dependency from consuming your capacity while it fails. The integration tier tracks the failure rate of each dependency over a rolling window; past a threshold the circuit opens and subsequent calls fail fast instead of waiting for a timeout. After a cool-down the circuit half-opens and allows a single trial call, closing on success.

Because AWS Lambda instances share no memory, the state lives outside: Amazon DynamoDB when a small amount of write latency is acceptable, Amazon ElastiCache for Redis when it is not. Read the state before making the dependency call and update it after.

Break the circuit per dependency and per tenant, not globally. One tenant's misbehaving MCP tool server should not open a circuit that stops every other tenant from using their own working tools. And distinguish failure classes carefully: a throttling response means capacity is constrained and backing off helps, while a validation error means your request is wrong and backing off changes nothing. Only the first should count toward opening a circuit.

The most consequential choice in an agent endpoint's design is whether the caller waits. It follows from the expected response time, the caller's tolerance, and whether the caller can poll or receive a callback.

Synchronous invocation

The caller sends a request, waits, and receives the answer in the response body. Simple to implement, simple to reason about, and correct when the response time sits reliably under the gateway timeout and a human is waiting.

“Reliably” is doing real work in that sentence. Agent response time is not normally distributed. A query that triggers three tool calls and a knowledge base retrieval takes several times as long as one answered from the model alone, and the distribution has a long tail. Size against p99, not p50, and instrument the phases separately: time to first token, time in tool calls, time in retrieval, total. When latency degrades you need to know which phase moved.

Even well under the timeout, a synchronous request can feel bad. Eight seconds of spinner is a poor experience regardless of whether the request succeeded. Streaming, covered in section 7, is the answer, and for user-facing agents it is close to mandatory.

Asynchronous invocation

When response time may exceed the gateway ceiling, or the caller is a batch system submitting many requests and collecting results later, the caller submits a job and receives an identifier immediately. It then polls a status endpoint or receives a callback.

The straightforward implementation puts Amazon SQS between submission and processing. The submission function writes the job and returns the identifier. A processing function consumes the queue, invokes the agent and writes the result to Amazon DynamoDB keyed by job identifier. The status endpoint reads that table and returns either a pending state or the completed response.

Three details separate a working implementation from a durable one. Give the result table a TTL so completed jobs expire rather than accumulating. Return a terminal state for failures rather than leaving the job pending forever, because a client polling a job that will never complete is a support ticket. And put a cap on polling: return a suggested poll interval in the submission response and honor it with 429 if the client ignores it, or a few impatient clients will generate more load than the agent work itself.

AWS Step Functions is the better fit once the workflow has more than one step. A state machine can submit, wait using the task token integration, catch and retry per error type, run a compensating action when a tool call fails halfway, and notify on completion through Amazon SNS or a callback function. The execution history also gives you a readable audit trail of what the workflow did, which is worth a lot when explaining an agent's behavior after the fact.

Choosing between them

 

Table 7. Synchronous against asynchronous. Cancellation and payload ceiling are the two rows teams discover late.

For anything a person watches, streaming is the single change with the biggest payoff. Instead of waiting for the complete answer, the client renders tokens as they are generated. Total duration is unchanged; perceived latency drops from seconds to a few hundred milliseconds, and the user gets continuous evidence that something is happening.

Server-sent events

SSE pushes a sequence of events to the client over one long-lived HTTP response. The server sets Content-Type: text/event-stream and writes events as they become available. Browsers support it natively through EventSource, and a fetch-based reader handles the POST case that EventSource does not.

There is an architectural constraint here that catches people out. Amazon API Gateway buffers the integration response, so an AWS Lambda function behind it cannot stream. AWS Lambda response streaming works through a function URL or the InvokeWithResponseStream API, not through Amazon API Gateway. In practice you either put an AWS Lambda function URL behind Amazon CloudFront for the streaming endpoint while keeping the gateway for everything else, or you use a WebSocket API, or you terminate on a container behind an Application Load Balancer where streaming is native.

Stream more than text. Emit events for tool invocations so the client can show what the agent is doing, for citations as they are resolved, and a terminal event carrying the finish reason and token usage. A user watching "searching the runbook index" appear understands a four second pause; a user watching an idle cursor does not.

 

Listing 9. Streaming through an AWS Lambda function URL. The error branch matters: once bytes have been written, the HTTP status is fixed at 200, so failures have to be expressed in the event stream itself.

That last point is the streaming failure mode worth burning into memory. A response that fails halfway has already sent a 200. Any client that treats the status code as the outcome will treat a truncated, wrong answer as a successful one. Every streaming contract needs a terminal event, and every client needs to require it before considering the response complete.

AWS AppSync is the managed alternative for GraphQL clients, using the subscription pattern from Listing 3. The resolver publishes deltas through an IAM-restricted mutation and subscribed clients receive them over WebSocket. Delivery is not strictly ordered, which is why the sequence number is in the schema.

WebSocket APIs for bidirectional traffic

When the user may interrupt, correct mid-response or send a follow-up while the agent is still generating, Amazon API Gateway WebSocket APIs give you a persistent bidirectional connection. Three routes: $connect for authentication and connection registration, $disconnect for cleanup, and a message route that invokes the agent and pushes chunks back through the Amazon API Gateway Management API using the connection identifier.

Authenticate on $connect and store the resolved identity against the connection identifier in Amazon DynamoDB. Re-deriving identity on every message is wasteful, and trusting a client-supplied identity on the message route is a straightforward impersonation bug. Store the tenant, the principal and the connection's authorized scopes at connect time and look them up per message.

Two operational notes. Amazon API Gateway WebSocket connections have a maximum duration of two hours and an idle timeout of ten minutes, so long conversations need reconnection handling that restores session state rather than starting fresh. And a disconnect during generation leaves a model call running with nowhere to send output. Check connection liveness between chunks and abandon the generation when the client is gone, or you will pay for tokens nobody receives.

Caching is hard in the ordinary case. Caching a probabilistic response expressed in natural language is a different problem wearing the same name and treating it as the familiar problem is how teams ship caches that serve confidently wrong answers.

Figure 7. Three cache tiers with different risk profiles, and the similarity threshold that governs the riskiest one.

Why the usual approach does not transfer

Ordinary HTTP caching rests on assumptions that agents violate one by one.

• Determinism. A cache assumes the same request produces the same response, so a stored copy is as good as a fresh call. With sampling enabled, an agent produces different text each time. Even at temperature zero, a change in retrieved chunks or tool output changes the answer.

• A complete key. For a static resource, the URL is the key. For an agent, the response depends on the question, the tenant, the caller's permissions, the knowledge base version, live tool state, the model, the sampling parameters, the system prompt version and the guardrail configuration. Every one of those omitted from the key is a class of wrong answer.

• Cheap misses. A cache miss on a web page costs a database query. A cache miss on an agent costs a model invocation that may be several cents and several seconds, which makes the incentive to cache aggressively much stronger and therefore much more dangerous.

• Equality. "How do I roll back a deployment?" and "what's the rollback procedure for a deploy?" are the same question with no bytes in common. Exact matching misses almost every real repeat, which is what pushes teams toward semantic matching and its associated risks.

Composing the cache key

Before considering similarity, get exact caching right, because the key composition problem is identical for both and much easier to see here.

Listing 10. Cache key composition. The is_cacheable check is the more important half: a large share of agent traffic should never be cached at all.

The kb_version field deserves emphasis because it connects directly to Module 9. Every knowledge base ingestion run invalidates every grounded answer produced from the previous version. Rather than tracking which cached entries touched which documents, include the knowledge base version in the key and let a new ingestion produce a clean namespace. The old entries expire on their TTL. This costs a cache-cold period after each sync, which is the correct price for never serving an answer grounded in a document that has since been corrected.

Semantic caching and the threshold problem

Semantic caching embeds the incoming question and retrieves the nearest cached question by vector similarity. Above a threshold, serve the stored answer. This catches the paraphrases that exact matching misses, and on a support or documentation agent the hit rate improvement is large.

It also introduces a failure mode with no equivalent in ordinary caching. A near miss does not produce a cache miss. It produces a confident, fluent, completely wrong answer to a question nobody asked, delivered with all the authority of a real one.

The trap is that embedding similarity captures topic, not intent. Consider these pairs, all of which score above 0.95 in common embedding spaces:

• "Can I deploy to production?" and "Can I not deploy to production?"

• "How do I enable MFA?" and "How do I disable MFA?"

• "What is the retention policy for staging?" and "What is the retention policy for production?"

• "Roll back the payments service" and "Roll back the payouts service"

Negation, antonyms and single-token entity substitutions barely move a cosine score, and each one flips the correct answer completely. Where the agent drives an action rather than answering a question, the last pair is not a quality problem, it is an incident.

Portkey AI, available in AWS Marketplace, offers semantic caching as configuration rather than code, alongside the routing capabilities covered in section 5.1. Enabling it removes the engineering work of embedding each question, storing vectors, running the nearest-neighbor lookup and expiring entries, which is several weeks of work to build well and an ongoing operational surface once built. Since the same gateway already sees every request and response, the cache also gets accurate hit rate and cost-avoided reporting without you instrumenting it.

What it does not remove is the part that determines whether the cache helps or harms. The similarity threshold, the namespace boundaries that keep tenants apart, and the false hit rate are still your decisions and your measurements, and the failure mode described above (a fluent, confident answer to a question nobody asked) arrives exactly the same way through a managed cache as through one you wrote. Treat a managed implementation as a faster route to the same decisions, not as a way to skip them.

Three things make semantic caching safe enough to run. First, set the threshold from your own labeled data rather than inheriting a vendor default. Sample production question pairs, label whether the same answer is correct for both, and choose the point where false hits fall below your tolerance. For low-stakes content 0.93 may be fine; for anything operational, expect to land above 0.97 and to accept a much lower hit rate.

Second, treat the band below your threshold as a prompt cache opportunity rather than a response cache miss. A near match means you have a relevant previous exchange, which is useful context to include in a fresh invocation even though it is not a substitute for one.

Third, verify near hits when the stakes justify it. A cheap fast model asked whether two questions have the same answer costs a fraction of the full invocation and catches exactly the negation and entity-substitution cases that embeddings miss.

Listing 11. A three-band lookup. The middle band is where most of the value sits: a near match improves the fresh invocation instead of replacing it.

Prompt caching: the tier with no correctness risk

Amazon Bedrock prompt caching is a different mechanism and deserves to be evaluated first, because it is the only tier that saves money without any possibility of serving a wrong answer. It caches the model's internal representation of a stable prompt prefix, so a long system prompt, a set of tool schemas or a large document that repeats across requests is not reprocessed each time. The model still runs and still produces a fresh response. What you save is input token cost and time to first token.

Getting value from it is an exercise in prompt ordering. Everything stable goes at the front, in a fixed order, with cache checkpoints after the stable segments. Everything variable, meaning the user's message and anything derived from it, goes at the end. A single variable token early in the prompt, such as a timestamp or a request identifier in the system preamble, invalidates the entire prefix and reduces your hit rate to zero.

For an agent with a 2,000 token system prompt, 3,000 tokens of tool schemas and 12,000 tokens of stable policy documentation, prompt caching removes 17,000 input tokens of processing per turn while changing nothing about the answer. That is usually a larger and safer win than a response cache, and it should be exhausted before the response cache is considered.

Invalidation and poisoning

Two invalidation triggers are easy to miss. Permission changes are the first: when a user's access is revoked, cached answers generated under their previous permissions remain servable if the key does not include the permission set. Including the sorted role list, as Listing 10 does, handles this by producing a different key rather than requiring an eviction sweep. Guardrail and policy changes are the second: a policy updated to block a topic must not be served around by entries cached before the change, which is why guardrail version belongs in the key.

Cache poisoning is the security consideration specific to semantic caching. If an attacker can get a response cached and then get their question matched by other users' questions, they have planted an answer. The mechanics are less exotic than they sound: ask a question phrased to sit near a common query, get a response influenced by injected content, and wait.

The defenses are structural. Never cache a response that a guardrail flagged, even at a low severity. Never share a cache namespace across tenants, which the tenant field in the key enforces. Require a minimum number of independent occurrences before an entry becomes eligible for semantic matching, so a single crafted question cannot become an attractor. And keep TTLs short enough that a poisoned entry has a bounded lifetime, measured in hours rather than days.

Measuring whether the cache is helping

Hit rate on its own is a misleading metric here, because it is trivially maximized by lowering the threshold, and lowering the threshold is exactly what makes the cache dangerous. Track the following together and treat any hit rate improvement accompanied by a false hit rate increase as a regression.

Table 8. Cache health. The false hit rate is the only one that can make the cache a net negative, and it is the only one that requires deliberate measurement.

The honest default

Start with prompt caching, which has no correctness risk and usually delivers the largest share of the savings. Add exact caching for the narrow set of stateless, non-personalized queries that genuinely repeat. Add semantic caching only after you have measured that paraphrase traffic is significant, and only with a threshold derived from your own labeled pairs. Many production agents should never run a semantic response cache at all, and that is a legitimate outcome rather than a missed optimization.

Agent endpoints have to change without breaking the callers already using them. New model versions, revised system prompts, re-ingested knowledge bases and altered tool configurations all change behavior, and unlike a schema change none of them will show up in a contract test. A rollout strategy is how you find out before your users do.

Amazon Bedrock agent versions and aliases

Amazon Bedrock agents version natively. Preparing an agent creates an immutable snapshot of its configuration: foundation model, instructions, action groups and associated knowledge bases. An alias maps a stable name to a version, and your integration tier references the alias rather than the version number. Promoting a new version is an alias update, so callers see no change to the identifier they hold. Keep separate aliases for production, staging and development.

One correction worth making explicitly, because it appears in a lot of otherwise-good material. An Amazon Bedrock agent alias does not split traffic across versions. The routing configuration accepts at most one version entry, so an alias points at exactly one version at a time. There is no percentage canary at the alias level.

Progressive rollout therefore has to happen a layer up, and there are three workable places to put it.

• AWS Lambda weighted aliases. An AWS Lambda alias can point at two published versions with a traffic weight between them. Publish an integration function version pinned to the new agent alias, point the AWS Lambda alias at both, and start at a small percentage. AWS CodeDeploy automates the shift with automatic rollback on an Amazon CloudWatch alarm, which is the closest thing to a turnkey canary in this stack.

• Routing in the integration tier. Select the agent alias per request from a deterministic hash of the tenant or session identifier. This gives sticky assignment, which matters more for agents than for stateless services: a user whose turns alternate between two agent versions mid-conversation gets an incoherent experience.

• Amazon Bedrock AgentCore A/B testing. For agents on AgentCore Runtime, target-based routing splits traffic across named endpoints through AgentCore Gateway, with assignment sticky by session identifier and online evaluation scoring each session. When your agent already runs on AgentCore, this is the least code.

Whichever mechanism you use, stickiness by session is the requirement that distinguishes agent canaries from ordinary ones. Hash the session identifier, not the request.

Amazon API Gateway stages and rollback

Stages give you a second versioning layer at the API level. A stage captures the resource definitions, integration configuration, usage plan associations and stage variables at a point in time, and different stages can carry different throttling limits and logging configuration. Stage variables parameterize the integration target, so the same API definition can point at a different AWS Lambda alias per stage.

Deployment history makes rollback a single API call, with no code change or infrastructure update. Pair it with Amazon CloudWatch alarms on 4xx and 5xx rates so the rollback can be automatic. For agents, add two alarms that a conventional API would not need: a guardrail intervention rate alarm, because a prompt change that starts tripping content policy shows up there long before it shows up in error rates, and a p99 latency alarm, because a new model version can be materially slower while remaining entirely healthy.

Behavioral regressions are the reason agent rollouts need more than error rate monitoring. A new version that answers every request with a 200 and a fluent but unhelpful response is invisible to conventional alarms. Run the evaluation suite from Module 5 against the canary in production, comparing its outputs to the stable version on the same inputs, and gate promotion on the result rather than on the absence of errors.

Feature flags for targeted rollout

Percentage rollout answers how many callers get the new version. It does not answer which ones, and for agents that distinction matters, because the callers most likely to surface a regression are rarely a random sample. The tenant with the largest knowledge base, the one whose questions are longest, and the one who opted into early access are each worth more as a canary than a random one percent of traffic.

LaunchDarkly, available in AWS Marketplace, evaluates flag rules against caller attributes and returns the value that decides which agent configuration a request gets. Instead of a percentage you target: a new model version enabled first for the customers who opted into early access, then for a service tier, then for everyone, with assignment sticky per tenant so nobody's conversation alternates between configurations. Because the flag is evaluated at request time rather than baked into a deployment, a quality problem is reverted in seconds without publishing an agent version, updating an alias or shifting an AWS Lambda weight. That difference in rollback latency is the argument for it: the alias and weighted-alias mechanisms above are minutes, and a flag is one toggle.

For agents this is most valuable at the prompt and tool-configuration level rather than at the model level. Prompt changes are frequent, low-ceremony and disproportionately likely to cause subtle regressions, and a flag gives you a rollback path that does not require publishing a new agent version. Evaluate flags once at the start of the request and carry the result through the turn, so a flag flipped mid-generation cannot produce a response assembled from two configurations.

Most production agent deployments serve multiple tenants from shared infrastructure. Isolation has to hold for context, for throughput and for the retrieved content that grounds every answer.

Tenant isolation in the integration tier

The integration tier is the enforcement point. After authenticating, resolve the tenant from the token claims or the API key's usage plan association, and carry it into every downstream operation: session attributes on the invocation, knowledge base retrieval filters, tool invocation scopes and audit records.

The rule that prevents most of the failures in this area is short. The tenant identifier is derived from the authenticated principal and never read from the request body. A request that contains a tenant field should be rejected rather than have the field ignored, because a rejection tells you someone is testing your boundary and silence does not.

Amazon Cognito user pools support this through custom attributes and group memberships. An AWS Lambda authorizer reads group membership from the token, resolves it to a permission set, and returns both a policy and a context map that the integration function reads. Putting the resolved tenant in the authorizer context rather than re-deriving it downstream means there is exactly one place where that decision is made.

Per-tenant rate limiting and token budgets

Usage plans attach to API keys, and API keys are issued per tenant, so Amazon API Gateway enforces each tenant's throttle and quota independently. A tenant generating a burst is throttled without affecting anyone else.

Request-count limits are only half the control, for the reason covered in section 3.2. Track tokens per tenant in the integration tier and enforce a budget alongside the request quota. Amazon ElastiCache for Redis is well suited: an atomic increment against a per-tenant counter with a rolling window costs under a millisecond and gives you a sliding-window limiter that request-count quotas cannot express.

Decide deliberately what happens when a tenant exhausts their token budget mid-conversation. Hard-stopping produces an error in the middle of a working session, which users experience as a break rather than as a limit. Degrading to a smaller model, or reducing the retrieval depth, keeps the session alive at lower cost and lower quality. Either is defensible; the failure is not choosing, and discovering the behavior during an escalation.

Data isolation for tenant knowledge bases

When tenants share knowledge base infrastructure, each tenant's documents must be retrievable only by that tenant. Amazon Bedrock Knowledge Bases supports metadata filtering at retrieval time: documents are tagged with their tenant during ingestion, and the retrieval request carries a filter restricting results to matching documents. The filter is assembled in the integration tier from the authenticated tenant, so there is no request a caller can construct that retrieves another tenant's content.

Assemble the filter server side and never merge it with a caller-supplied filter. If callers can pass their own filter expressions, the combination logic becomes a security control, and combination logic is exactly where these bugs live. Accept caller filters as a separate field and apply them as an additional constraint on an already-scoped result set, never as part of the same expression.

For tenants with strict isolation requirements, a shared knowledge base with metadata filtering may not satisfy their auditors regardless of how correct it is, because the control is logical rather than physical. A separate knowledge base and vector index per tenant costs more and scales worse, and it is sometimes the only answer that passes review. Price that tier accordingly rather than absorbing it.

An agent endpoint needs the observability any production API needs, shaped by variable response times, incremental delivery, multi-step reasoning and tool calls that each contribute independently to latency.

Metrics that matter

 

Table 9. Metrics for an agent endpoint. Every row is something a conventional API dashboard would not show you.

Dimension these by tenant and by agent alias. A p99 that looks acceptable in aggregate regularly hides one tenant having a bad time, and an alias dimension is what lets you compare a canary to stable on the same graph.

Distributed tracing

AWS X-Ray traces the full path through Amazon API Gateway, AWS Lambda and Amazon Bedrock. Enable it on the stage and the functions to get a service map with latency attribution per component. Add annotations for agent alias, tenant, session and the tools invoked, since annotations are indexed and can be filtered on, while metadata cannot.

Trace context has to survive the asynchronous hops or you lose the transaction. Carry the trace identifier through the Amazon SQS message attributes, the Amazon EventBridge detail and the job record, and restore it in the processing function. Without that, a submission and its completion are two unrelated traces and you cannot answer how long the whole thing took.

One caution specific to this domain. Amazon Bedrock agent traces expose the model's intermediate reasoning, which is valuable for debugging and inappropriate to surface to callers or to log wholesale. Summarize traces into step descriptions for the client, as Listing 9 does, and route the raw trace to the tier-three sink under the same access controls as any other full-fidelity content.

Dashboards and alarms

Put the gateway, function, model and business metrics on one dashboard. A latency spike is far more interpretable when you can see simultaneously that it coincides with a retrieval slowdown for one tenant who just re-ingested a large corpus.

Use Amazon CloudWatch anomaly detection rather than fixed thresholds for latency and error rate. Agent response time varies with query complexity, and complexity varies with time of day and day of week, so a fixed threshold either alarms constantly or catches nothing. Anomaly detection learns the baseline including its periodicity.

Keep fixed thresholds for the things that genuinely have hard limits: token budget consumption against quota, queue age against your processing SLA, and dead-letter queue depth, which should alarm on any non-zero value.

Agent exposure is the layer that turns a capable agent into something people can actually use, and it is where the assumptions built into ordinary API infrastructure quietly stop holding. The four patterns, direct API, gateway-mediated, event-driven and MCP server, map onto different caller types, and most real systems run several at once.

Payload size is the first constraint that bites. Agent context outgrows the transport long before it outgrows the model, so pick deliberately between inline bodies, Amazon S3 references and server-side sessions, and measure the p99 rather than the mean. Protocol choice follows: REST across organizational boundaries, gRPC inside the perimeter where streaming and byte efficiency compound, GraphQL through AWS AppSync where one client screen composes several sources.

Behind the gateway, split the AWS Lambda tier into thin per-consumer adapters over a shared agent core. The separation exists so timeouts, memory, concurrency, roles and alarms can differ per consumer; if they are identical you have paid for the split without collecting on it. AWS WAF handles volume, rate and structure, and its body inspection limit means it never sees most of an agent request. Prompt injection is caught structurally in your own code and semantically in Amazon Bedrock Guardrails, with untrusted spans tagged explicitly so the filter has something to work with.

Logging agent traffic at the default settings creates a cost problem and a privacy problem at the same time. Split it into metadata, hashed content and a sampled full-fidelity tier, each with its own retention and access policy. Across regions, accept that model availability, agent identifiers, knowledge base content, guardrail versions and Provisioned Throughput do not replicate, and check whether a cross-region inference profile solves your actual problem before building a second stack. 

Caching deserves the most care. A probabilistic natural-language response breaks the assumptions caching rests on, and semantic matching turns a cache miss into a confident wrong answer. Exhaust prompt caching first, which has no correctness risk, then exact caching with a complete key, and treat semantic caching as an optimization you have to earn with measurement.

Partner tooling shows up in this stack at four distinct points, all of it procurable in AWS Marketplace, and each one has a threshold below which the AWS primitive is the shorter path. A portal layer such as Tyk Technologies earns its place when external developers consume the agent as a product. Kong earns its place when the integration tier runs on Amazon EKS and gRPC has taken Amazon API Gateway out of the picture. An aggregating gateway such as KrakenD earns its place when every client wants the same composed shape and GraphQL would be overkill. A model gateway such as Portkey AI earns its place when routing and fallback logic would otherwise be duplicated across more teams than can keep a shared library in step. None of them removes a decision described in this module; they remove the work of implementing one you have already made.

Module 11 takes this into the full lifecycle: how an agent moves through development, staging and production, and how version promotions are coordinated with the data pipeline changes and infrastructure updates they depend on.