AWS Big Data Blog

AI-powered cost optimization agent for Amazon Kinesis Data Streams

Customers running multiple Amazon Kinesis Data Streams often struggle to estimate the cost impact of switching between capacity modes. As accounts grow to tens or hundreds of streams, manually reviewing Amazon CloudWatch metrics for each stream and comparing pricing across Provisioned, On-demand Standard, and On-demand Advantage becomes impractical. Teams often stay on their current mode, unsure whether switching would save money or cost more, leaving potential savings unquantified. On-demand Advantage is an account-level setting that unlocks additional capabilities and a different pricing structure for on-demand streams in an AWS Region. However, without a clear, data-driven comparison, the decision to enable it remains difficult to justify.

In this post, we show you how to deploy an AI-powered agent built on Amazon Bedrock. The agent automatically analyzes every Kinesis Data Stream in your account and compares costs across all three capacity modes. It tells you exactly which streams to move to On-demand and whether your account qualifies for On-demand Advantage pricing, all on a daily or weekly schedule with zero manual intervention. As we showed in Kinesis On-demand Advantage saves 60%+ on streaming costs, choosing the right mode can save over 60 percent on streaming costs. This agent automates that analysis for you.

What is the Kinesis Mode Optimizer Agent?

The Kinesis Mode Optimizer Agent is an open source, serverless solution that uses Amazon Bedrock AgentCore, a platform to build, connect, and optimize agents at scale, with any framework or model. The agent autonomously analyzes your Kinesis Data Streams usage. It collects 7 days of Amazon CloudWatch metrics for every stream in the Region and discovers Enhanced Fan-Out (EFO) consumers. It then computes a three-way cost comparison (On-demand Standard, On-demand Advantage, Provisioned) and generates per-stream recommendations along with an account-level On-demand Advantage assessment.

The agent strongly prefers on-demand modes for their operational simplicity (automatic scaling, no capacity planning, and no throttling risk).

Results are stored as both a visual HTML report and machine-readable JSON in Amazon Simple Storage Service (Amazon S3).

Architecture

The solution uses the following architecture:

Architecture diagram of the Kinesis Mode Optimizer Agent showing Amazon EventBridge, a Scheduler Lambda, Amazon Bedrock AgentCore, a Tool Lambda, and downstream Kinesis Data Streams, CloudWatch, and Amazon S3


Figure 1: Architecture of the Kinesis Mode Optimizer Agent

The architecture flow includes the following steps:

  1. Amazon EventBridge Schedule triggers the Scheduler Lambda, an AWS Lambda function, on your configured cadence (daily, weekly, or custom cron).
  2. Scheduler Lambda invokes the Amazon Bedrock AgentCore harness with the instruction to analyze streams and generate a report.
  3. Amazon Bedrock AgentCore Gateway, a capability of Amazon Bedrock AgentCore powered by Claude Sonnet, interprets the request and routes it to the appropriate Model Context Protocol (MCP) tools exposed by the Tool Lambda.
  4. Tool Lambda performs the heavy lifting, fanning out to three downstream services:
    1. Kinesis Data Streams – Lists streams in the Region, describes each stream (shard count, mode, retention), and discovers Enhanced Fan-Out consumers per stream.
    2. CloudWatch – Pulls 7 days of metrics per stream (IncomingBytes, OutgoingBytes, throttle events) and computes three-way cost comparison.
    3. Amazon S3 – Generates per-stream recommendations and an account-level Advantage assessment, then stores the final HTML and JSON reports.
  5. Amazon Bedrock AgentCore harness summarizes the findings and returns them to the caller.

The entire stack is deployed using AWS Cloud Development Kit (AWS CDK) with a single cdk deploy command.

Prerequisites

Before you begin, verify that you have the following:

  • AWS CDK – npm install -g aws-cdk.
  • Python 3.12+.
  • aws-cdk-lib >= 2.251.0 – for AgentCore L2 constructs.
  • AWS Command Line Interface (AWS CLI) configured with credentials that have permissions to deploy the required resources.
  • Amazon Bedrock model access – verify you have access to Claude Sonnet 4.5 (or your chosen model) in the target Region. Check in the Amazon Bedrock console under Model access.

Walkthrough

In the following sections, you deploy the Kinesis Data Streams Mode Optimizer Agent and test it against your Kinesis streams.

Step 1: Clone the repository

git clone https://github.com/aws-samples/sample-kinesis-optimizer-agent.git
cd sample-kinesis-optimizer-agent

Step 2: Install CDK dependencies

cd infra
pip install -r requirements.txt

Step 3: Set your target Region

The stack deploys to whatever Region is set in AWS_DEFAULT_REGION. Set it before running any CDK commands:

# Linux/macOS
export AWS_DEFAULT_REGION=us-east-1

# Windows PowerShell
$env:AWS_DEFAULT_REGION="us-east-1"

Step 4: Bootstrap CDK (first time per account/Region)

cdk bootstrap aws://<ACCOUNT_ID>/<REGION>

Step 5: Deploy

cdk deploy

Optionally customize the schedule and bucket name:

# Weekly instead of daily
cdk deploy --parameters ReportSchedule="rate(7 days)"

# Custom bucket name
cdk deploy --parameters ReportBucketName=amzn-s3-demo-bucket

Step 6: Test the agent

You can test the agent with the AWS CLI:

aws bedrock-agentcore invoke-harness \
    --harness-arn <HARNESS_ARN> \
    --runtime-session-id $(uuidgen) \
    --messages '[{"role":"user","content":[{"text":"Generate an optimization report"}]}]' \
    --region us-east-1

Step 7: View reports

Reports are stored in Amazon S3 at:

s3:// amzn-s3-demo-bucket/kinesis-optimization-reports/YYYY/MM/DD/HHMMSS-<report-id>.html
s3:// amzn-s3-demo-bucket/kinesis-optimization-reports/YYYY/MM/DD/HHMMSS-<report-id>.json

The HTML report includes a per-stream action table with priority indicators, detailed cost breakdowns, and the account-level Advantage recommendation.

Sample HTML report showing the per-stream action table with priority indicators and cost breakdowns


Figure 2: Sample HTML report with the per-stream action table

Sample report showing the account-level On-demand Advantage recommendation


Figure 3: Account-level On-demand Advantage recommendation in the sample report output

Multi-Region deployment

The agent is Region-specific. When deployed to a Region, it analyzes only the streams in that Region. To cover multiple Regions, change the environment variable and repeat the deployment:

export AWS_DEFAULT_REGION=eu-west-1
cdk bootstrap aws://<ACCOUNT_ID>/eu-west-1
cdk deploy

Each deployment is independent, with its own agent, Amazon S3 bucket, and schedule.

Clean up

To remove the stack from a Region:

cd infra
cdk destroy

Note: The Amazon S3 bucket has a RemovalPolicy.RETAIN setting and isn’t deleted with the stack. Delete it manually if you no longer need the reports.

Conclusion

In this post, you deployed an AI-powered agent that autonomously analyzes your Amazon Kinesis Data Streams and recommends the optimal capacity mode for each stream. The agent alleviates the manual effort of reviewing CloudWatch metrics across dozens or hundreds of streams and produces actionable, cost-aware recommendations on a recurring schedule.

By shifting from manual capacity reviews to autonomous, scheduled optimization, you gain three key benefits. First, you can reduce streaming costs by identifying streams that should switch modes. Second, you alleviate throttling risk by catching under-provisioned streams before they impact performance. Third, you free your team from repetitive operational work. All of this is achievable with a single cdk deploy.

To get started, clone the sample-kinesis-optimizer-agent repository and deploy it to your account today.

About the authors

Masudur Rahaman Sayem

Masudur Rahaman Sayem

Masudur is a Streaming Data Architect at AWS with over 25 years of experience in the IT industry. He collaborates with AWS customers worldwide to architect and implement sophisticated data streaming solutions that address complex business challenges. As an expert in distributed computing, Sayem specializes in designing large-scale distributed systems architecture for maximum performance and scalability. He has a keen interest and passion for distributed architecture, which he applies to designing enterprise solutions at internet scale.

Roy (KDS) Wang

Roy (KDS) Wang

Roy is a Senior Product Manager with Amazon Kinesis Data Streams. He is passionate about learning from and collaborating with customers to help organizations run faster and smarter. Outside of work, Roy strives to be a good dad to his new son and builds plastic model kits.