AWS Big Data Blog

AWS and DuckLabs: Building the future of analytics together

Today we are announcing that Amazon has signed a definitive agreement to acquire DuckLabs, the Amsterdam-based company behind the open-source analytical database DuckDB. We expect the transaction to close shortly, subject to customary closing conditions. Hannes Mühleisen and Mark Raasveldt, who created DuckDB and co-founded DuckLabs, will continue leading the team and the open-source project’s technical direction as part of AWS. The DuckDB open-source project will also continue to be driven by the DuckLabs team, remain open source under the independent Foundation (the non-profit that oversees DuckDB), and available under the MIT license as it does today.

How Klaviyo processes 3 billion daily events at scale with Amazon MSK Express

How Klaviyo processes 3 billion daily events at scale with Amazon MSK Express

Learn how Klaviyo migrated its event pipeline from RabbitMQ to Amazon MSK Express brokers, ingesting roughly 3 billion events per day. A small team scaled the broker fleet multi-fold for Black Friday/Cyber Monday 2025, delivered 22.7 billion messages (up 25% year over year), then scaled back down.

Announcing in-place upgrades from Standard brokers to Express brokers for Amazon MSK

Announcing in-place upgrades from Standard brokers to Express brokers for Amazon MSK

Amazon MSK now supports in-place upgrades from Standard brokers to Express brokers. You can modernize an existing MSK Provisioned cluster through a fully managed workflow, with no new cluster, no application changes, and no downtime. This post shows how to assess readiness, run the rolling upgrade, validate, and roll back if needed.

Understanding and optimizing materialized views in Amazon Redshift

Understanding and optimizing materialized views in Amazon Redshift

Materialized views in Amazon Redshift speed up expensive queries, but every refresh costs compute whether or not a query reads the result. This post shares a diagnostic view, v_mv_health, and the sp_populate_mv_definitions procedure that surface each MV’s refresh eligibility, cost, staleness, dependencies, and usage in one result set.

Introducing Amazon EMR Serverless Flex: Run non-urgent Spark jobs at a discount

Introducing Amazon EMR Serverless Flex: Run non-urgent Spark jobs at a discount

Amazon EMR Serverless Flex is a new execution option that runs non-urgent Spark batch jobs on spare compute capacity at a fixed 60% discount compared to standard pricing. Flex handles Availability Zone selection, instance diversification, driver protection, and automatic retries, so jobs complete even when spare capacity is reclaimed.

How Moonfrog scales analytics and lowers query cost with Amazon Redshift RG instances

How Moonfrog scales analytics and lowers query cost with Amazon Redshift RG instances

Moonfrog Labs migrated a 148 TB Amazon Redshift warehouse from RA3 to Graviton-based RG instances with a single elastic resize. See how the team sized capacity for growing analytics demand, verified warehouse and data lake access, and measured faster loads and lower query cost.

Guardrails and cost control for your agentic workloads with Amazon Redshift

Guardrails and cost control for your agentic workloads with Amazon Redshift

AI agents are becoming major consumers of warehouse data, and two risks decide whether they reach production: excessive agency and unbounded consumption. Learn how to run agentic analytics on Amazon Redshift safely and without runaway costs, using one model: six guardrails, three enforcement layers, and one cost envelope.

Amazon S3 Tables with dbt: Iceberg-based data transformations using AWS Glue

Amazon S3 Tables with dbt: Iceberg-based data transformations using AWS Glue

Amazon S3 Tables, a capability of Amazon S3, bring built-in Apache Iceberg support to object storage. This post shows how to configure and run dbt transformations on S3 Tables with the AWS Glue dbt-glue adapter, using incremental merge and append strategies across a staging and mart architecture.

Introducing automatic version upgrades for AWS Glue jobs

Introducing automatic version upgrades for AWS Glue jobs

by Keerthi Chadalavada, Chris Kha, Pramod Chunduri, Liyuan Lin, Malinda Malwala, Mohit Saxena, Mukul Prasad, Ria Katoch, Rishabh Nair, Shrey Malpani, Shruti Gupta, Wei Tang, and XiaoRun Yu on in Advanced (300), Announcements, AWS Glue Permalink Share

Today we’re announcing automatic version upgrades for AWS Glue jobs. Set a job’s version to auto, and AWS Glue checks for breaking changes, asks for your approval, then moves the job to the latest stable version and validates the upgraded runs. Learn how auto upgrade works and how to use it across a fleet of jobs with an AI agent.

Long-term Amazon Redshift observability with Datadog and Amazon S3 Tables

Long-term Amazon Redshift observability with Datadog and Amazon S3 Tables

Amazon Redshift keeps only 7 days of system table history. This post pairs Datadog Data Observability with the new Amazon Redshift system table integration with Amazon S3 Tables, so you can retain query history for the long term and analyze lineage, query performance, and data quality without adding load to your warehouses.