AWS Big Data Blog
Category: Analytics
Build a self-healing ETL pipeline with Amazon Bedrock AgentCore
Build a self-healing ETL pipeline on AWS Glue and Amazon Bedrock AgentCore that automatically corrects bad records using domain knowledge, writes the fixes back to the data lake, and can open pull requests to prevent the same errors from recurring.
Choose the right streaming ingestion path for Apache Iceberg tables on AWS
Choosing the right streaming ingestion path for Apache Iceberg tables shapes your data lake’s latency, cost, and capabilities. This post compares five AWS paths against eight technical criteria and closes with a decision framework you can apply to your own workload.
Managed cross-Region disaster recovery for Amazon OpenSearch Service
Amazon OpenSearch Service now offers managed cross-Region disaster recovery for domains on OpenSearch 3.7 or later. It replicates data from a primary domain to a secondary domain in another Region with API-driven switchover and failover, providing an RPO under 1 minute and an RTO under 20 minutes.
Optimizing Apache HBase availability on Amazon EMR, Part 2: Diagnose and tune compaction
Compaction keeps HBase read performance healthy, but when it runs during peak hours it drives latency spikes, write stalls, and RegionTooBusyException errors. In Part 2 of this series, you learn how to confirm compaction is the cause on Amazon EMR, tune the right parameters in the least-invasive order, and know when to stop and contact AWS Support.
Build a multi-team AI and analytics environment on Amazon SageMaker Unified Studio with Terraform
Learn how to use the Amazon SageMaker Unified Studio Terraform module to provision a multi-team AI and analytics platform (domains, blueprints, project profiles, and projects) for data analytics, machine learning, and generative AI teams in a single terraform apply, with shared governance and least-privilege security defined entirely as code.
How Samsung Ads federated Snowflake Iceberg tables using Amazon Cognito
Learn how Samsung Ads federates Snowflake-managed Apache Iceberg tables into AWS Lake Formation and queries them from Amazon Athena using Amazon Cognito for OAuth2 machine-to-machine authentication, with no second copy of the data to maintain.
Allocate shared Amazon MSK costs to Kafka topics for cost visibility
Amazon MSK bills at the cluster level, with no per-topic breakdown. This post shows how to allocate shared MSK cluster costs to individual Kafka topics by combining AWS Cost and Usage Report billing with per-topic CloudWatch or Prometheus metrics, giving each team a usage-based view of its spend.
Governance compliance beyond the catalog in Amazon SageMaker Unified Studio
Amazon SageMaker provides rich governance signals such as metadata forms, data quality scores, glossary coverage, and subscription data, but they live in separate APIs. This post shows how to join them into a single enriched Iceberg table and build a compliance scorecard you can query, score, and adapt to your governance policy.
How Intuit migrated from Apache Hive Metastore to AWS Glue Data Catalog
How Intuit migrated its data catalog from Apache Hive Metastore to AWS Glue Data Catalog on its journey to an open, secure lakehouse. This post covers the challenges, the phased bidirectional-sync solution, and the benefits: reduced operational overhead, improved data discovery, and tighter integration with AWS analytics services.
46% fewer nodes, 2× faster indexing: Confluent on Amazon OpenSearch Service
How Confluent modernized its Amazon OpenSearch Service observability hot tier, ingesting hundreds of terabytes of logs per day, by migrating from r6gd to the purpose-built oi2 instance family. A measure-first approach cut the fleet 46%, roughly halved indexing latency, and cleared write backpressure, lowering cost even at a higher per-node price.









