AWS Big Data Blog
AWS and DuckLabs: Building the future of analytics together
Today we are announcing that Amazon has signed a definitive agreement to acquire DuckLabs, the Amsterdam-based company behind the open-source analytical database DuckDB. We expect the transaction to close shortly, subject to customary closing conditions. Hannes Mühleisen and Mark Raasveldt, who created DuckDB and co-founded DuckLabs, will continue leading the team and the open-source project’s technical direction as part of AWS. The DuckDB open-source project will also continue to be driven by the DuckLabs team, remain open source under the independent Foundation (the non-profit that oversees DuckDB), and available under the MIT license as it does today.
Deliver petabytes of Apache Kafka data to Amazon S3 at up to 60% lower cost with Amazon MSK Express brokers
Amazon MSK Express brokers now deliver Apache Kafka topic data to Amazon S3 at up to 60% lower cost, with no Kafka Connect to run. You create a channel on a topic and pay only for the data delivered. In this post, we cover the two challenges of delivering Kafka data to Amazon S3 today and how Express brokers and data delivery to Amazon S3 address each one.
Migrate Apache Solr to Amazon OpenSearch Service with the AI-driven Migration Assistant
Migrate a 600,000-document Apache Solr catalog to Amazon OpenSearch Serverless end to end using the AI-driven agent mode in the Migration Assistant for Amazon OpenSearch Service, from assessment through validated backfill.
Analyze Anthropic usage, spend, and access with AWS Glue, Amazon S3 Tables, and AWS Quick
The new AWS Glue connector for Anthropic reads Claude usage, cost, and API key data into AWS without custom code. This post shows how to land it in Amazon S3 Tables as Apache Iceberg tables, join it with your business data, query it with Amazon Athena, and visualize AI spend and governance in AWS Quick.
Introducing enhanced snapshots for Amazon Managed Service for Apache Flink
Enhanced snapshots for Amazon Managed Service for Apache Flink are now generally available. An enhanced snapshot is an AWS resource with its own ARN, decoupled from the application that created it, so you can start another application from it, export it to Amazon S3, and import savepoints for blue/green deployments, disaster recovery, and migrations.
Upgrading and troubleshooting Amazon MWAA with AWS agent skills
AWS publishes MWAA agent skills in the open source AWS agent toolkit: procedures your AI coding assistant can run to upgrade Amazon MWAA from Airflow 2.x to 3.x and troubleshoot failures, with a safety confirmation on every environment change.
Understanding span queries in Amazon OpenSearch Service
Among OpenSearch query types, span queries give you precise control over where terms appear in a document and how they relate to one another. Learn how span queries work, when to use them instead of phrase queries, and how to compose them to enforce exact term order, position, and containment that phrase queries cannot express.
Optimizing PySpark UDFs on AWS Glue 6.0 with native Arrow UDFs
Python UDFs in PySpark can be slow because data is serialized between the JVM and the Python runtime. We benchmark five approaches on AWS Glue 6.0 (standard Python, Arrow-optimized Python, vectorized pandas, native Arrow UDFs, and Spark SQL) across 10M to 100M rows, and separate transport cost from compute cost to explain the ranking.
Multi-statement transactions for Apache Iceberg tables in Amazon Redshift
Amazon Redshift now supports multi-statement transactions for Apache Iceberg tables, so you can run several DML statements against one Iceberg table inside a single BEGIN…COMMIT and publish them as one atomic Iceberg snapshot. This post explains why one commit per statement makes multi-step pipelines fragile, what you can run in a transaction, and how it performs.
Build a self-healing ETL pipeline with Amazon Bedrock AgentCore
Build a self-healing ETL pipeline on AWS Glue and Amazon Bedrock AgentCore that automatically corrects bad records using domain knowledge, writes the fixes back to the data lake, and can open pull requests to prevent the same errors from recurring.
Choose the right streaming ingestion path for Apache Iceberg tables on AWS
Choosing the right streaming ingestion path for Apache Iceberg tables shapes your data lake’s latency, cost, and capabilities. This post compares five AWS paths against eight technical criteria and closes with a decision framework you can apply to your own workload.










