AWS Big Data Blog

Today we are announcing that Amazon has signed a definitive agreement to acquire DuckLabs, the Amsterdam-based company behind the open-source analytical database DuckDB. We expect the transaction to close shortly, subject to customary closing conditions. Hannes Mühleisen and Mark Raasveldt, who created DuckDB and co-founded DuckLabs, will continue leading the team and the open-source project’s technical direction as part of AWS. The DuckDB open-source project will also continue to be driven by the DuckLabs team, remain open source under the independent Foundation (the non-profit that oversees DuckDB), and available under the MIT license as it does today.

Compaction keeps HBase read performance healthy, but when it runs during peak hours it drives latency spikes, write stalls, and RegionTooBusyException errors. In Part 2 of this series, you learn how to confirm compaction is the cause on Amazon EMR, tune the right parameters in the least-invasive order, and know when to stop and contact AWS Support.

Building a data foundation ready for agentic AI relies on a unified lakehouse and interoperability protocols like Model Context Protocol (MCP). This post walks through four pillars for building an agentic AI data foundation on AWS: a unified lakehouse, context engineering, multi-agent interoperability, and data governance.

Learn how to use the Amazon SageMaker Unified Studio Terraform module to provision a multi-team AI and analytics platform (domains, blueprints, project profiles, and projects) for data analytics, machine learning, and generative AI teams in a single terraform apply, with shared governance and least-privilege security defined entirely as code.

Learn how Samsung Ads federates Snowflake-managed Apache Iceberg tables into AWS Lake Formation and queries them from Amazon Athena using Amazon Cognito for OAuth2 machine-to-machine authentication, with no second copy of the data to maintain.

Amazon MSK bills at the cluster level, with no per-topic breakdown. This post shows how to allocate shared MSK cluster costs to individual Kafka topics by combining AWS Cost and Usage Report billing with per-topic CloudWatch or Prometheus metrics, giving each team a usage-based view of its spend.

Amazon SageMaker provides rich governance signals such as metadata forms, data quality scores, glossary coverage, and subscription data, but they live in separate APIs. This post shows how to join them into a single enriched Iceberg table and build a compliance scorecard you can query, score, and adapt to your governance policy.

How Intuit migrated its data catalog from Apache Hive Metastore to AWS Glue Data Catalog on its journey to an open, secure lakehouse. This post covers the challenges, the phased bidirectional-sync solution, and the benefits: reduced operational overhead, improved data discovery, and tighter integration with AWS analytics services.

How Confluent modernized its Amazon OpenSearch Service observability hot tier, ingesting hundreds of terabytes of logs per day, by migrating from r6gd to the purpose-built oi2 instance family. A measure-first approach cut the fleet 46%, roughly halved indexing latency, and cleared write backpressure, lowering cost even at a higher per-node price.

Validating a streaming pipeline beyond current peak takes production-scale test data. This post shows how the stream multiplier pattern on Amazon Managed Service for Apache Flink turns a small pre-recorded sample into sustained, production-scale load while preserving the statistical properties of real traffic.

Agentic Data Ingestion is an open-source solution that uses AI agents on Amazon Bedrock to automate data pipeline creation, from file upload to queryable tables. Point it at a file or database table, and foundation models propose the field mapping, generate and test the AWS Glue transform, and run the job behind human approval gates.