Skip to main content

AWS for Data

A technical leader’s guide to open data architectures with Apache Iceberg

Missing alt text value

Introduction

The data landscape has undergone a profound transformation over the past decade. Data volumes are growing faster than ever, accelerated by AI workloads that both consume and generate massive amounts of data. And with the recent rise of AI agents, the number of tools and engines that need access to data has multiplied, even while most teams are being asked to deliver more with less. What began with traditional data warehouses optimized for structured analytics has evolved into a complex ecosystem of technologies, and sharing data reliably across them remains one of the hardest problems to solve.

Apache Iceberg addresses this directly. It is an open table format that brings the reliability of a data warehouse to the flexibility of a data lake, while ensuring that any compatible engine, tool, or agent can safely work with the same data without reformatting or duplication. Iceberg addresses many of the challenges that data engineers have long faced, including schema evolution without downtime, ACID transactions on data lakes, time travel capabilities, and efficient metadata management at petabyte scale.

This guide serves as your introduction to Apache Iceberg. Whether you're looking to migrate from legacy systems, optimize existing deployments, or design a new data architecture for agentic AI, you'll learn what Iceberg is, why it matters now more than ever, and how to get started with Iceberg on AWS.

The journey to open data architectures

From data warehouses to data lakes

For decades, enterprise analytics were built on data warehouses: structured, governed, and performant, but limited in flexibility. By the early 2000s, data lakes built on cloud object storage like Amazon Simple Storage Service (Amazon S3) emerged as a complement, unlocking the ability to store any data in raw form and serve multiple analytics engines from a single repository. Today, more than a million data lakes run on Amazon S3, and they remain a core part of how organizations manage data at scale.

As data lake use cases expanded into mission-critical analytics, organizations needed capabilities that traditional data lakes lacked or required expensive dedicated resources to manually implement, including ACID transactions, safe concurrent writes, and schema evolution. By the late 2010s, many organizations were running both a data warehouse and a data lake, often moving data between them.

The emergence of open table formats

Open table formats resolve this tension by bringing database-like capabilities to data lakes. Critically, open table formats enable interoperability, meaning any compatible tool, engine, or agent can safely work with the same data without reformatting or duplication.

Open table formats underpin lakehouse architectures, which combine data lakes with a centralized governance layer, allowing them to act like data warehouses. Lakehouse architectures are scalable, flexible, and provide a shared system for storing and processing data. They make it easier to consolidate datasets into high-quality data products, with broad analytics tool compatibility and centralized access controls.

Apache Iceberg is the open table format that made lakehouse architectures possible at scale. Created at Netflix in 2017 to solve the challenges of operating petabyte-scale data lakes, Iceberg was contributed to the Apache Software Foundation in 2018 and is now supported by every major cloud provider and data platform vendor.

Iceberg: Built for the cloud, born on S3

In 2017, Netflix was running one of the largest data platforms in the world. Their systems landed petabytes of data into Amazon S3 each day, organized across hundreds of thousands of tables with millions of partitions. Their data was managed using Apache Hive tables, but their massive scale queries were slow, writes could corrupt tables, and concurrent engines had no way to coordinate safely. Netflix engineers decided to create a new table format purpose-built for cloud object storage. That project became Apache Iceberg, and within a few years, it went from an internal experiment to one of the most widely adopted projects in big data.

Learn more: Netflix’s journey to an Apache Iceberg-only data lake

How Apache Iceberg works

Apache Iceberg brings the reliability, performance, and governance capabilities of a data warehouse to data lakes. It does this through a metadata-first architecture that changes how analytics engines interact with data.

Metadata as the core innovation

Open table formats improve legacy approaches by adding a metadata layer on top of raw files. What distinguishes Iceberg is how it structures that metadata and what it enables.

Apache Iceberg organizes every table into three layers: a catalog that serves as the entry point for all table operations, a metadata layer that tracks every file and every change, and the data files themselves. The catalog points to the current metadata, ensuring that any compatible engine can locate and safely operate on the same table without coordination outside the format.

Each Iceberg table maintains three layers of metadata:

  • Metadata files capture the table's complete state at a specific moment, including the schema, partition layout, sort order, and full history of snapshots.
  • Manifest lists reference groups of manifest files and record the changes introduced by each write operation.
  • Manifest files contain detailed information about individual data files, including file locations, row counts, partition values, and column-level statistics like minimum and maximum values.

When a user submits a query, Iceberg evaluates the manifest metadata first. It reads the manifest-level partition summaries and per file column-level statistics within each manifest to determine which data files contain relevant records and only opens those files. This means the engine skips irrelevant files entirely, scanning only the data that matters. In cloud environments where cost is tied to the amount of data scanned, this translates directly into cost savings and faster response times.

Learn how Amazon Ads achieved a 22% reduction in processing time, 20% decrease in compute costs, and 32% savings in storage costs by leveraging Iceberg’s file layout optimizations on Amazon S3.

ACID transactions

Atomicity, consistency, isolation, and durability (ACID) transactions are what make data lakes suitable for workloads that previously required a data warehouse. Every write operation in Iceberg produces a new immutable snapshot, which represents the state of a table at a given moment. Readers always see a consistent state of the table and never encounter partial writes, even when multiple processes are writing concurrently. When conflicts occur, Iceberg detects them and fails cleanly rather than corrupting data.

Time travel

Because every write produces an immutable snapshot, Iceberg naturally preserves a complete history of table states. This means you can query the exact state of a table as it existed at any point, enabling audit trails, historical analysis, and reproducible analytics. When an erroneous write corrupts a table, you can roll back to a previous state in seconds rather than rebuilding from scratch.

Tables accumulate snapshots over time, and while this history enables powerful capabilities, it requires active management like expiring old snapshots and removing orphan files to maintain table health and control storage costs.

Schema and partition evolution

Data structures rarely remain static. Iceberg was designed to allow schemas to evolve in a safe, consistent, and forward-compatible way by managing schema and partition changes through metadata rather than data rewrites. Columns can be added, removed, renamed, or retyped without reprocessing existing files. Partition strategies can shift while maintaining full compatibility with historical data, allowing your data architecture to evolve alongside your business.

Hidden partitioning

Traditional data lakes require users to know the partition scheme to write efficient queries, meaning you must explicitly filter on those derived columns. Iceberg removes this requirement by using partition transforms to automatically derive partition values from source columns and prunes data based on the values in the query itself. Users write natural filters on original columns, and Iceberg handles the rest.

Row-level update and delete operations

Iceberg supports MERGE INTO, UPDATE, and DELETE operations directly on data lake tables without full table rewrites. This enables streaming pipelines, regulatory deletion requests, and corrections at scale.

Open at every layer

Apache Iceberg is an open source project guided by community governance, not controlled by any single vendor. Its openness runs through every layer of the architecture: data is stored in open file formats (Parquet, ORC, Avro), the Iceberg REST Catalog provides a standardized API for table discovery, and any compatible engine can work with the same tables without reformatting or duplication. Organizations are free to build their own data architectures without fear of vendor lock-in or committing to any single proprietary format.

Why Apache Iceberg matters now more than ever

As organizations prepare their data foundations for agentic AI, building on open standards has never been more important. AI agents need to discover, access, and reason over data autonomously, and they can't do that efficiently when every data source requires a custom integration.

Open table formats like Apache Iceberg give agents a consistent, well-structured interface to access and work with large-scale data. Features like ACID transactions, schema evolution, snapshot isolation, and built-in time travel ensure that every query, whether initiated by a human analyst or an autonomous agent, sees a complete and consistent view of the data. When combined with open protocols like the Model Context Protocol (MCP) for agent-to-tool communication and open source frameworks like Apache Kafka and Apache Spark for data movement and processing, the result is a data architecture where agents can operate more efficiently across systems with a much lower risk of being throttled by data silos.

Iceberg also provides the flexibility organizations need to stay future-proof. Schemas can evolve and partitioning strategies can change as requirements shift, without breaking downstream consumers. And because Iceberg is a 100% open-source, community-driven table format, it isn't tied to any single vendor's roadmap. You retain the freedom to swap tools and frameworks as workloads scale and the AI landscape continues to evolve, while agents are much more likely to get reliable access to high-quality data.

Beyond flexibility, Iceberg can deliver meaningful cost-efficiency. Consolidating workloads on a single copy of governed data eliminates redundant ETL pipelines and reduces storage costs from duplicate datasets spread across siloed systems. With native support across popular processing frameworks like Apache Spark, Apache Flink, and Presto, Iceberg lets organizations choose the most performant engine for each task, accelerating both analytics and AI initiative without adding cost.

Getting started with Apache Iceberg on AWS

AWS offers the broadest native Apache Iceberg support of any major cloud provider, with Iceberg-compatible services at every layer of the data stack, so that you can build the open data architecture that best serves your business. From streaming, ingestion, and purpose-built storage to data processing, SQL analytics, AI/ML, and governance, you get an open, performant, and trusted foundation for open Iceberg architectures.

The first step is often to identify a use case. For many organizations, that can look like one of the following:

Next step

For step-by-step guidance on building your Iceberg architecture, including migration patterns, storage optimization, catalog configuration, and governance setup, see the AWS Prescriptive Guidance: Using Apache Iceberg on AWS.

Note

Portions of this guide were informed by and adapted from Unleash the Power of Apache Iceberg on AWS, written by Imtiaz Sayed, Stefano Sandona, Angel Conde Manjon, Leonardo Gomez, Noritaka Sekiyama, Rui Qing Zhang, and Sandeep Adwankar.

Did you find what you were looking for today?

Let us know so we can improve the quality of the content on our pages