AWS Big Data Blog

Category: Technical How-to

Multiply pre-recorded streams with Amazon Managed Service for Apache Flink

Multiply pre-recorded streams with Amazon Managed Service for Apache Flink

Validating a streaming pipeline beyond current peak takes production-scale test data. This post shows how the stream multiplier pattern on Amazon Managed Service for Apache Flink turns a small pre-recorded sample into sustained, production-scale load while preserving the statistical properties of real traffic.

Automate ETL pipeline creation with Amazon Bedrock AgentCore and AWS Glue

Automate ETL pipeline creation with Amazon Bedrock AgentCore and AWS Glue

Agentic Data Ingestion is an open-source solution that uses AI agents on Amazon Bedrock to automate data pipeline creation, from file upload to queryable tables. Point it at a file or database table, and foundation models propose the field mapping, generate and test the AWS Glue transform, and run the job behind human approval gates.

Define metrics once and keep them correct with the dbt Semantic Layer on Amazon Redshift

Define metrics once and keep them correct with the dbt Semantic Layer on Amazon Redshift

Amazon Redshift has no native semantic layer, yet teams still need consistent, governed metric definitions across BI tools. This post shows how to define a metric once with the open-source dbt Semantic Layer (MetricFlow) on Amazon Redshift so it stays correct at any grain, and proves a non-additive weekly average comes out right ($280, not $550).

Deliver multi-tenant AI analytics at scale with AgentCore and Lake Formation

Deliver multi-tenant AI analytics at scale with AgentCore and Lake Formation

Governed, multi-tenant data access is a key requirement for SaaS and B2C products that embed AI-powered analytics. This post shows how to combine Amazon Cognito and AWS Lake Formation on Amazon Bedrock AgentCore to enforce row- and column-level data access for external users, with a minimal-privilege Lambda acting as a credential broker.

Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg

Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg

Learn how to migrate Google BigQuery workloads to an open lakehouse on AWS built on Apache Iceberg and Amazon S3. This post explains why an open lakehouse future-proofs your data, presents the target architecture on AWS, and walks through a step-by-step migration: assess, design, convert, migrate, cut over, and optimize.

From Problem to Fix in Minutes: Automating Search Relevance Tuning with the Amazon OpenSearch Service Relevance Agent

The Amazon OpenSearch Service Search Relevance Agent turns a low NDCG score into a better search query in minutes. Describe the problem through the Ask AI chat, and the agent analyzes your User Behavior Insights data, generates an improved query, and validates it with an offline Search Relevance Workbench experiment, all without leaving the UI.

Trace security threats in multi-agent pipelines with Amazon OpenSearch Service

Trace security threats in multi-agent pipelines with Amazon OpenSearch Service

Learn how to trace two invisible threats in a multi-agent AI pipeline, prompt injection propagation and cross-tenant data exfiltration, using Amazon OpenSearch Service and Amazon Bedrock. Instrument every agent hop with embeddings, provenance, and PII detection, reconstruct the attack path, and add guardrails that stop these attacks before they reach production.

Build cross-account streaming pipelines with the Amazon Kinesis Producer Library and Kinesis Client Library

Build cross-account streaming pipelines with the Amazon Kinesis Producer Library and Kinesis Client Library

Learn how to build an enterprise-scale cross-account streaming pipeline with Amazon Kinesis Data Streams using the Kinesis Producer Library (KPL) and Kinesis Client Library (KCL) across three separate AWS accounts, with resource-based policies, cross-account IAM, and a shared customer managed AWS KMS key for maximum security isolation.

Monitoring MWAA-orchestrated ETL pipelines with Amazon OpenSearch Service

Monitoring MWAA-orchestrated ETL pipelines with Amazon OpenSearch Service

Troubleshooting a multi-service, MWAA-orchestrated ETL pipeline often means hunting across scattered Amazon CloudWatch log groups. This post shows how to centralize your ETL logs in Amazon OpenSearch Service and query them in plain language through an MCP server on Amazon Bedrock AgentCore, so you find root causes faster.