AWS Big Data Blog

AWS and DuckLabs: Building the future of analytics together

Today we are announcing that Amazon has signed a definitive agreement to acquire DuckLabs, the Amsterdam-based company behind the open-source analytical database DuckDB. We expect the transaction to close shortly, subject to customary closing conditions. Hannes Mühleisen and Mark Raasveldt, who created DuckDB and co-founded DuckLabs, will continue leading the team and the open-source project’s technical direction as part of AWS. The DuckDB open-source project will also continue to be driven by the DuckLabs team, remain open source under the independent Foundation (the non-profit that oversees DuckDB), and available under the MIT license as it does today.

Deliver petabytes of Apache Kafka data to Amazon S3 at up to 60% lower cost with Amazon MSK Express brokers

Deliver petabytes of Apache Kafka data to Amazon S3 at up to 60% lower cost with Amazon MSK Express brokers

Amazon MSK Express brokers now deliver Apache Kafka topic data to Amazon S3 at up to 60% lower cost, with no Kafka Connect to run. You create a channel on a topic and pay only for the data delivered. In this post, we cover the two challenges of delivering Kafka data to Amazon S3 today and how Express brokers and data delivery to Amazon S3 address each one.

Analyze Anthropic usage, spend, and access with AWS Glue, Amazon S3 Tables, and AWS Quick

Analyze Anthropic usage, spend, and access with AWS Glue, Amazon S3 Tables, and AWS Quick

The new AWS Glue connector for Anthropic reads Claude usage, cost, and API key data into AWS without custom code. This post shows how to land it in Amazon S3 Tables as Apache Iceberg tables, join it with your business data, query it with Amazon Athena, and visualize AI spend and governance in AWS Quick.

Introducing enhanced snapshots for Amazon Managed Service for Apache Flink

Introducing enhanced snapshots for Amazon Managed Service for Apache Flink

Enhanced snapshots for Amazon Managed Service for Apache Flink are now generally available. An enhanced snapshot is an AWS resource with its own ARN, decoupled from the application that created it, so you can start another application from it, export it to Amazon S3, and import savepoints for blue/green deployments, disaster recovery, and migrations.

Upgrading and troubleshooting Amazon MWAA with AWS agent skills

Upgrading and troubleshooting Amazon MWAA with AWS agent skills

AWS publishes MWAA agent skills in the open source AWS agent toolkit: procedures your AI coding assistant can run to upgrade Amazon MWAA from Airflow 2.x to 3.x and troubleshoot failures, with a safety confirmation on every environment change.

Optimizing PySpark UDFs on AWS Glue 6.0 with native Arrow UDFs

Optimizing PySpark UDFs on AWS Glue 6.0 with native Arrow UDFs

Python UDFs in PySpark can be slow because data is serialized between the JVM and the Python runtime. We benchmark five approaches on AWS Glue 6.0 (standard Python, Arrow-optimized Python, vectorized pandas, native Arrow UDFs, and Spark SQL) across 10M to 100M rows, and separate transport cost from compute cost to explain the ranking.

Multi-statement transactions for Apache Iceberg tables in Amazon Redshift

Multi-statement transactions for Apache Iceberg tables in Amazon Redshift

Amazon Redshift now supports multi-statement transactions for Apache Iceberg tables, so you can run several DML statements against one Iceberg table inside a single BEGIN…COMMIT and publish them as one atomic Iceberg snapshot. This post explains why one commit per statement makes multi-step pipelines fragile, what you can run in a transaction, and how it performs.