AWS Big Data Blog

Category: Analytics

Multiply pre-recorded streams with Amazon Managed Service for Apache Flink

Multiply pre-recorded streams with Amazon Managed Service for Apache Flink

Validating a streaming pipeline beyond current peak takes production-scale test data. This post shows how the stream multiplier pattern on Amazon Managed Service for Apache Flink turns a small pre-recorded sample into sustained, production-scale load while preserving the statistical properties of real traffic.

Automate ETL pipeline creation with Amazon Bedrock AgentCore and AWS Glue

Automate ETL pipeline creation with Amazon Bedrock AgentCore and AWS Glue

Agentic Data Ingestion is an open-source solution that uses AI agents on Amazon Bedrock to automate data pipeline creation, from file upload to queryable tables. Point it at a file or database table, and foundation models propose the field mapping, generate and test the AWS Glue transform, and run the job behind human approval gates.

Amazon Athena introduces system tables for query execution observability

Amazon Athena introduces system tables for query execution observability

Amazon Athena now offers system tables: SQL-queryable, per-query execution metadata stored as Apache Iceberg in your own account. This post walks through what system tables are, how to turn them on, and how to use them for usage analysis, cost attribution, performance trends, audit, and capacity optimization without custom pipelines.

Store and process large strings in Amazon Redshift SUPER columns

Storing large semi-structured data such as JSON and arrays in a data warehouse often means splitting or truncating your data. With PartiQL and the SUPER data type, Amazon Redshift is now equipped to handle these issues. In this post, we walk you through three key SUPER data type enhancements that simplify semi-structured data workflows and reduce common migration challenges.

Deliver multi-tenant AI analytics at scale with AgentCore and Lake Formation

Deliver multi-tenant AI analytics at scale with AgentCore and Lake Formation

Governed, multi-tenant data access is a key requirement for SaaS and B2C products that embed AI-powered analytics. This post shows how to combine Amazon Cognito and AWS Lake Formation on Amazon Bedrock AgentCore to enforce row- and column-level data access for external users, with a minimal-privilege Lambda acting as a credential broker.

Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg

Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg

Learn how to migrate Google BigQuery workloads to an open lakehouse on AWS built on Apache Iceberg and Amazon S3. This post explains why an open lakehouse future-proofs your data, presents the target architecture on AWS, and walks through a step-by-step migration: assess, design, convert, migrate, cut over, and optimize.

Implement least-privilege access with designations in Amazon SageMaker Unified Studio

Implement least-privilege access with designations in Amazon SageMaker Unified Studio

Amazon SageMaker Unified Studio now offers designations: named permission sets that give administrators granular, least-privilege control over what each project member can do. Learn how designations work and how to create and assign your first one from the console and the AWS CLI.

Optimizing Apache HBase availability on Amazon EMR, Part 1: Coordinator startup, RegionServer recovery, and regions in transition

Optimizing Apache HBase availability on Amazon EMR, Part 1: Coordinator startup, RegionServer recovery, and regions in transition

A repeatable process for keeping Apache HBase on Amazon EMR highly available. Part 1 covers triaging an HBase coordinator that is slow to start, a RegionServer that leaves the cluster, and regions that stay in transition, with the web UI pages, commands, and log signatures that point to each cause.

Introducing time series collections for the next generation of Amazon OpenSearch Serverless

Introducing time series collections for the next generation of Amazon OpenSearch Serverless

The next generation of Amazon OpenSearch Service serverless now supports time series collections. Define the boundaries for cost, performance, security, access, and data retention using simple controls for data tiering, time field selection, and data retention, then turn raw events into insights with the next-generation OpenSearch UI.