How Zeta Global delivers near real-time, AI-powered marketing insights with Amazon S3 Tables

Zeta’s AI marketing cloud serves thousands of clients simultaneously, delivering personalized campaigns across email, social, connected TV, display, and mobile. For these campaigns to work, the underlying data must be fresh within seconds, not hours. “In many ways data is our product, so having it secure, reliable, and fresh is critical to what we do,” said Sekhar Sahu, Principal Engineer and Data Platform Lead at Zeta Global.

Zeta Global’s data landscape expanded rapidly due to customer growth and a series of acquisitions. These acquisitions brought tremendous strategic value to the organization, but also introduced a broad set of data sources, schemas, infrastructure, and operational models. The result was data scattered across disparate systems and silos that became increasingly difficult to manage. During peak loads, lag times stretched from minutes to hours, forcing customers to make campaign decisions on stale data and risk wasted ad spend. At the same time, dedicated infrastructure resources for each major client drove up compute costs, threatening the company's ability to maintain competitive pricing.

Zeta’s data platform team recognized that the path forward needed more than incremental fixes. “Our vision was to unify everything into one common layer where all our data lands. Every team can access the governed data and use the right compute engine for the right use case," said Sahu. The ability to leverage multiple compute engines at the right price point rather than being tied into a single vendor’s tooling was important to the team, as was the ability to cut redundant ETL pipelines.

The team had three goals for the new architecture. First, the data lakehouse should be the source of truth for all data, raw or aggregated. Second, all processing, joins, and transformations must run in the lakehouse, not in copies. Finally, the lakehouse should target roughly 15 minutes of end-to-end freshness. Anything requiring sub-minute latency would run on a streaming path fed from the same events, with the lakehouse still serving as the system of record. Where a downstream application cannot tolerate that read latency, aggregated tables are replicated to a low-latency store rather than recomputed elsewhere.

Zeta built its modern lakehouse architecture on Amazon S3 because of its durability, scalability, and native support for open table formats (OTFs). The team evaluated several OTFs against criteria that included an open catalog interface, hidden partitioning, snapshot isolation for concurrent writers, and a credible path for semi-structured data and cost-effective updates. Apache Iceberg met all these needs, and its REST catalog endpoint and support for the Variant data type, deletion vectors, and row lineage in Version 3 confirmed the decision.

“We saw that the industry is converging towards Iceberg,” added Soumil Shah, Lead Software Engineer at Zeta Global. “Many of the tools we already used were building native support for Apache Iceberg, which gave us the confidence that it was the right choice.”

To manage Iceberg tables at scale, Zeta chose Amazon S3 Tables, purpose-built storage for Apache Iceberg in Amazon S3. The alternative — self-managing compaction and maintenance across thousands of tables and more than 150 AWS accounts — would require a dedicated engineering team. “We looked at the engineering time and effort needed to scale compaction and maintenance across all of our AWS accounts and regions,” said Shah. “That is a whole project in itself, which is why it makes sense to go to a managed service.” 

In the new architecture, data flows into the lakehouse through Amazon Managed Streaming for Apache Kafka (Amazon MSK), where Kafka consumers write events to Amazon S3. Apache Spark jobs on Amazon EMR transform raw data and load it into Iceberg tables, with AWS Step Functions orchestrating the pipeline. Spark on EMR, Amazon Athena, Snowflake, and Trino all read the same tables through the Iceberg REST endpoint, with no copies and no engine-specific formats. AWS Glue Data Catalog serves as the central metadata store, and AWS Lake Formation enforces governance and access controls. The platform manages 3–5 petabytes of data across roughly 5,000 Iceberg tables, and expects to reach tens of thousands as workload migrations continue.

With over 150 AWS accounts, Zeta adopted a multi-account federated catalog architecture. Each account maintains its own Amazon S3 Tables, and consumers access shared datasets through a governed central account without replicating data. “We don’t have to copy data across these different accounts. There is only a single copy of the data, and consumers read from that single copy, which is important,” said Sahu.

To handle the semi-structured nature of marketing data, where every client has a unique schema that changes frequently, Zeta adopted the Iceberg V3 Variant data type. Variant stores semi-structured data natively, reducing storage for that data by 30 percent compared to storing the data as string JSON. “Having all customers as top-level columns or a fixed schema is impossible for us, and that's where Variant comes into the picture, for semi-structured data or frequently changing schema,” said Shah.

The lakehouse architecture reduces compute costs by 50 percent and cut processing times from roughly 30 minutes to 1–2 minutes — a 95 percent improvement that moved data freshness from hours to minutes. The platform now runs 18,000 Spark jobs daily with a 98 percent success rate. "Our new architecture is much faster, much better, much cheaper, and much more reliable," said Shah. “AWS really helped us get to where we are today.”

With a data foundation centralized in an open lakehouse, the company can now support near real-time segmentation use cases that were previously impossible when processing took hours. Zeta has also built AI models and agents directly on top of the lakehouse. "Everything — all AI — is going to be powered from the lakehouse," said Shah. For a company whose mission is to help brands find, grow, and keep customers, delivering insights and activating them in real time strengthens its ability to deliver on that promise.