AWS Storage Blog
How Tubular Labs reclaimed 50% of engineering capacity by rebuilding their 70TB pipeline on Apache Iceberg and Amazon S3 Tables
Customer Story | Amazon S3 Tables – Learn how Tubular Labs (part of Chartbeat Inc.) reclaimed 50% of engineering capacity by replacing fragile, file-based data pipelines with a Common Pipeline Runtime (CPR) built on Apache Iceberg and Amazon S3 Tables.
By Tubular Labs Engineering (part of Chartbeat Inc.), in collaboration with the AWS Solution Architecture team
Tubular Labs (part of Chartbeat Inc.) tracks video engagement across YouTube, Facebook, Instagram, Twitch, and TikTok, providing social video intelligence that helps customers anticipate content trends and act on shifting audience behavior. Their analytical dataset has grown to 70TB containing 25 billion rows, with approximately 2 billion row updates occurring daily. At that scale, their file-based pipeline architecture was accumulating operational complexity: fragile retries, inconsistent data across destinations, duplicated infrastructure logic. As a result, engineers were spending more time on maintenance than building new features.
In this post, we share how Tubular Labs evolved this complex system to a more sustainable architecture by building a Common Pipeline Runtime (CPR) on Apache Spark, Apache Iceberg, and Amazon S3 Tables.
Operational complexity at billion-row scale
As Tubular Labs’ video analytics platform grew from millions to tens of billions of tracked videos, their pipeline infrastructure accumulated operational debt. Managing this scale while maintaining query performance, operational reliability, and cost efficiency required rethinking their pipeline architecture.
Legacy architecture limitations
The legacy pipelines ran large Spark jobs on Amazon EC2 Spot Instances and relied on disk-based checkpointing for recovery. Individual jobs frequently read from two sources and merges were handled by a custom implementation tightly coupled to the business logic. These design choices presented several operational challenges:
Non-idempotent pipelines. Pipelines lacked built-in idempotency, making retries require careful consideration. When upstream sources changed between runs, rerunning an interrupted pipeline could produce different results or create duplicate rows. On-call engineers needed deep service-specific knowledge to safely recover from failures.
Atomic writes required custom implementation. Writing Parquet files directly to Amazon S3 buckets without transactional guarantees meant partial writes during interruptions. Each service implemented atomicity differently, creating maintenance overhead and inconsistency.
Multi-phase writes increased failure surface area. Pipelines write data to multiple destination systems like analytics, search, and streaming. Without atomic multi-phase commit protocols, failures during a phase left data inconsistent across systems. This became particularly problematic when running on Amazon EC2 Spot Instances, where instance preemption could interrupt writes mid-operation.
Code duplication across services. Each pipeline independently implemented Change Data Capture (CDC), offset management, retry logic, and write coordination. This required infrastructure improvements to change dozens of services, creating version skew and duplicated effort.
Debugging difficulties. Without standardized logging, metrics, or failure patterns, diagnosing production issues required service-specific expertise. Mean time to repair (MTTR) varied significantly based on which engineer was on-call and their familiarity with each pipeline, and was often measured in hours.
Implicit assumptions created fragility. Pipelines made assumptions that seemed reasonable initially, such as fixed file counts, local disk checkpointing, and manual offset management, but required reconsideration at scale, particularly on ephemeral infrastructure like Spot Instances.
Business requirements driving architecture change
The goal was to sustain the growing dataset while improving reliability, reducing time-to-recovery, and accelerating development velocity. The new architecture needed to:
- Enable safe and automated retries through idempotent operations
- Provide atomic writes across multiple destinations
- Centralize infrastructure complexity in a common runtime
- Standardize observability across all pipelines
- Support both full and incremental processing modes
Solution overview
To meet these business requirements while addressing the limitations of the legacy architecture, Tubular Labs chose Amazon S3 Tables with Apache Iceberg as the managed data infrastructure and built a Common Pipeline Runtime (CPR) to centralize pipeline logic across dozens of services. S3 Tables handle Iceberg compaction, snapshot maintenance, and transactional semantics automatically, which freed up engineering time. CPR establishes clear separation between business logic (transformations and validations) and infrastructure logic (CDC, idempotency, atomic writes, retries). The runtime owns infrastructure concerns, exposing a minimal interface that services implement to provide their specific transformations.
Architecture
The solution runs on Amazon EKS with Apache Spark for distributed processing. The DataReader and Publisher depend on Apache Iceberg for transactional semantics, while S3 Tables provide the managed Iceberg layer handling automatic compaction, maintenance, and storage optimization, eliminating the need to operate these manually.
The following figure shows the AWS infrastructure supporting the solution:
Figure 1: Amazon EKS with Apache Spark processing data from multiple sources into Amazon S3 Tables
CPR consists of three primary components:
- DataReader (runtime-provided): Reads source tables in either FULL (complete table) or NET_CHANGES (CDC since last successful run) mode. Abstracts offset management, incremental read logic, and upstream change detection.
- DataProcessor (service-provided): Applies business transformations and validations. Services implement a small interface defining their transformation logic, validation rules, and identifier columns for merge operations.
- Publisher (runtime-provided): Writes to target destinations (Iceberg tables on S3 Tables) using Write-Audit-Publish (WAP) patterns, Iceberg branches/tags, and atomic commit semantics.
The following figure shows how data flows through CPR’s three-stage processing pipeline:
Figure 2: Common Pipeline Runtime: Three-stage data flow powered by Apache Iceberg and Amazon S3 Tables
The Publisher uses Iceberg’s Write-Audit-Publish (WAP) pattern. New data is written to an isolated branch, validated against the complete dataset, and only promoted to main via atomic fast-forward if validation passes. The runtime also standardizes idempotent retries and observability through Amazon CloudWatch, so engineers can safely rerun interrupted pipelines without duplicating data.
Why Amazon S3 Tables and Apache Iceberg
Tubular Labs chose Apache Iceberg for its ACID transactions with snapshot isolation, schema evolution without data rewrites, and built-in time travel for debugging data quality issues. For details on Iceberg’s full capabilities, see the Apache Iceberg documentation.
S3 Tables extends the benefits of Iceberg with fully managed operations. With 2 billion daily updates generating thousands of small files, manual compaction would require significant operational overhead. Instead, S3 Tables maintain optimal file sizes for query performance automatically.
Results
The managed Iceberg operations of S3 Tables, combined with CPR’s centralized runtime logic, delivered improvements across operational efficiency and scale.
Faster and safer recovery from failures. Mean time to repair (MTTR) for infrastructure-related failures dropped from 12–16 hours to approximately 40 minutes. Because CPR’s idempotent, atomic writes guarantee that a retry cannot corrupt or duplicate data, on-call engineers no longer need to perform complex, deep diagnosis before triggering a retry, nor do they need to conduct data cleanup after a failure. Recovery steps were simplified to just rerunning the pipeline.
Reduced pipeline runtime. Before pipeline modernization, the write and merge job alone took 6–8 hours to complete, now end-to-end pipeline execution, including multi-billion-row merges, validation, and atomic publish, finishes in under two hours.
Reclaimed engineering capacity. Before this work, the business faced frequent and prolonged delays in data becoming available for queries. Roughly 50% of the team’s engineering capacity was consumed by triaging incidents or fixing data corruption and performing data recovery caused by bugs and race conditions in the legacy pipelines. Committed feature delivery was routinely delayed by one to two weeks because of interruptions from on-call activities. By eliminating entire categories of consistency bugs and making retries safe, CPR frees this capacity for feature development.
Simplified pipeline code. Pipelines that previously contained 500+ lines of infrastructure code now implement only 50–100 lines of business logic. One pipeline migration removed 700 lines of custom CDC, retry, and write coordination code. Centralizing CDC also eliminated the need for Kafka bridges in many pipelines.
Conclusion
Built on Amazon S3 Tables, Apache Iceberg, and Apache Spark, the Common Pipeline Runtime developed by Tubular Labs provides a sustainable architecture for managing 70TB analytical datasets with 2 billion daily updates. The impact is most visible in operational resilience and reclaimed engineering time. If you’re managing large-scale data pipelines with heavy update workloads, the principles that made this work are broadly applicable:
- Separate concerns rigorously: Business logic and infrastructure should be cleanly separated with explicit interfaces
- Make idempotency foundational: Safe, atomic retries eliminate the need for deep diagnosis and post-failure data cleanup, collapsing recovery time and eliminating entire categories of operational complexity
- Centralize CDC: Don’t duplicate change detection logic across services
- Leverage modern table formats: Apache Iceberg provides capabilities impossible with traditional file-based approaches
- Use managed services: S3 Tables eliminate operational overhead for compaction, maintenance, and optimization
To learn more about Amazon S3 Tables, visit https://aws.amazon.com/s3/features/tables.