AWS Big Data Blog
How Twilio modernized billing at 70% lower cost with Amazon Redshift
This post is co-written with Sean McKibben from Twilio.
This post describes how Twilio’s Commerce Platform team modernized their billing pipeline by migrating from a legacy MySQL system to a modern data architecture on AWS. The new billing platform is built on Amazon Redshift Serverless, with streaming ingestion using Amazon Managed Streaming for Apache Kafka (Amazon MSK) and Zero-ETL integration with Amazon Aurora MySQL.
“Our billing transformation with Amazon Redshift empowered the business to deliver new ways of selling which were impossible to consider before. The fact that it did this while simultaneously reducing costs by 70% and reducing operational overhead told us how powerful the approach was.”
— Sean McKibben, Principal Software Engineer, Twilio
Twilio is a cloud communications platform with APIs for messaging, voice, email, and video. It serves over 400,000 active customer accounts and more than 12 million developers worldwide to build unique, personalized experiences for their customers. Twilio’s Commerce Platform business unit owns shopping, metering, and rating domains, processing tens of billions of usage events per month across 200 million price points. The platform supports Twilio’s Opportunity-to-Revenue (OTR) initiative, a company-wide program to modernize the full billing lifecycle from customer intake through invoicing.
Challenges with the legacy billing system
Twilio built its billing system on Amazon Aurora MySQL, which reliably powered 70 partitioned database instances and converted billions of events into invoices across messaging, voice, email, and video. As Twilio’s business grew and data volumes expanded significantly, the team recognized an opportunity to evolve the architecture to meet new demands.
Following are the primary challenges with the legacy platform:
- Processing slowdowns, with backlogs sometimes taking hours to clear.
- Operational fragility because of frequent ETL failures requiring engineers to spend more time troubleshooting pipelines than delivering new features.
- Higher infrastructure costs that were disproportionate to the business value delivered. It also lacked the flexibility to smoothly deliver features like backdated changes to pricing models or complex organizational lifecycles without manual intervention.
The billing system needs to generate invoice line items with financial-grade consistency. There can be no duplicate charges or missing events. The existing system could not support new pricing models without re-architecting, and the finance team could not run analytical queries without impacting production.
Twilio needed a distributed data warehouse architecture that could handle both transactional billing integrity and analytical flexibility. The architecture had to process billing calculations in minutes rather than hours and let finance and product teams query data without impacting production.
Overview of the solution
Twilio’s Commerce Platform team evaluated several approaches, including traditional data warehouses, data lakes with query engines, and split architectures with separate operational and analytical systems. Traditional data warehouses are optimized for analytics at the expense of real-time data consistency while operational databases handle transactional workloads but struggle with complex analytical queries at scale.
With Amazon Redshift, Twilio bridged that gap, helping queries return consistent results even for concurrent write workloads, while still maintaining high analytical performance. The key capabilities that drove Twilio to adopt Amazon Redshift Serverless are as follows:
- Serverless operations — Amazon Redshift Serverless eliminates provisioning, capacity planning, and failover management. Compute scales automatically in response to workload demand, optimizing cost without manual intervention.
- Real-time streaming ingestion — Billing events stream continuously through Amazon MSK. Amazon Redshift streaming ingestion delivers near real-time event data directly into the data warehouse with no requirement of intermediate landing zones.
- Transactional integrity — Using serializable isolation with Amazon Redshift helped deliver consistent query results across concurrent workloads, thereby preventing data anomalies that would require custom reconciliation logic.
- Materialized views AUTO refresh — Materialized views (MV) with AUTO refresh incrementally aggregate billing events by customer, product, and day, delivering near real-time rated data to downstream workgroups without batch processing.
- Zero-ETL integration — Amazon Aurora zero-ETL integration with Amazon Redshift with history mode enabled automatically replicates full change history from Amazon Aurora MySQL databases into Amazon Redshift with no custom change data capture (CDC) pipelines required.
- Distributed data warehouse — With Amazon Redshift data sharing, multiple workgroups securely share data across workloads. This provides workload isolation by running analytical queries on an independent consumer workgroup without slowing down invoice processing, along with cost visibility per workload.
- Integration with existing data lake — Amazon Redshift unload to Amazon Simple Storage Service (Amazon S3) and direct writes to Amazon S3 Iceberg tables indexed with AWS Glue Data Catalog eliminated Spark ETL jobs for data lake delivery, allowing broader organizational access to the data.
- Transformation orchestration with dbt — dbt (data build tool) is used for orchestrating Amazon Redshift MV refreshes and stored procedure execution, managing the dependency graph across more than 200 million price points with auto-generated documentation and full calculation lineage. Amazon Redshift full SQL compatibility with dbt Core allows them to define metering, rating, and bookkeeping logic as reproducible SQL directed acyclic graphs (DAGs).
The following diagram illustrates the architecture of the solution implemented by Twilio:
The key implementation steps for the architecture are as follows:
1. Ingestion layer
Twilio’s billing events originate from product teams across messaging, voice, email, and video. The data from these source systems are ingested into the billing system using the following ingestion pipelines:
Streaming ingestion
Each product emits usage events to Amazon MSK topics within their Amazon MSK Serverless clusters. Billing events from these topics flow into Amazon Redshift using streaming ingestion. AUTO refresh option for Amazon Redshift MVs provides real-time aggregation as events land, incrementally aggregated by customer, product, and date, thereby processing billions of events daily.
Serializable transaction isolation, combined with deduplication logic in the materialized views, allows each event to be counted exactly once.
Zero-ETL integration
Twilio’s product catalog and pricing configuration data live across multiple Amazon Aurora MySQL clusters. This legacy system contains customer account data, pricing configurations, and historical billing records that must be accessible to the new engine for reconciliation. The legacy system priced everything with whatever configuration was current at the time of processing, which worked well for forward-looking billing. However, the team needed the flexibility to reprice historical usage for corrections, promotional adjustments, and contract changes.
Using Zero-ETL with history mode for integrating Amazon Aurora databases with Amazon Redshift, every price update and configuration change is automatically captured with timestamps. As a result, the team can reconstruct any bill from any point in time after history mode was enabled. They can also reprice historical usage with the exact configuration that should have applied, or model what invoices would look like under different scenarios.
Additionally, Zero-ETL replicates data from the legacy MySQL billing shards into Amazon Redshift Serverless with CDC, eliminating custom extraction pipelines. This allowed:
- Usage and pricing comparison between the new engine and legacy system, required for Sarbanes-Oxley (SOX) compliance.
- Billing accuracy validation during the parallel-run period.
- A unified query layer across all shards without managing shard-aware queries.
COPY command to load from Amazon S3
Finalized invoice data is exported from the invoicing platform, Zuora, in Apache Parquet format to Amazon S3, providing completed invoice line items, customer charges, and billing summaries that need to be available for customer self-service and console experiences. This invoice data in Amazon S3 is loaded into Amazon Redshift tables using COPY from Parquet on Amazon S3.
2. Transformation layer
The billing transformation logic runs on open source dbt Core with SQL DAGs defining the following transformation pipelines:
Metering
Metering deduplicates raw billing events using transaction identifiers and then aggregates events by customer, product, and time period. It also handles edge cases like late-arriving events and event corrections. dbt Core orchestrates the ingestion while auto-maintained incrementally refreshed Materialized Views aggregate billing events.
Rating
Rating applies pricing rules to metered aggregates. It manages over 200 million price points across multi-tiered volume discounts, promotional discounts that stack in specific orders, multi-product bundles, and bifurcated taxes. Each pricing rule is a discrete SQL model, and dbt orchestrates the dependency graph between them.
Bookkeeping
dbt Core orchestrates SQL transformation DAGs that generate priced aggregate artifacts formatted as invoice line items for the invoicing platform. The Bookkeeper system also creates read-only Iceberg tables from rating system data shares for data lake delivery. dbt cleans and transforms invoice data returned from the invoicing platform through Amazon S3 in Parquet format before it is written to Amazon DynamoDB for customer console experiences.
3. Storage layer
The billing engine uses multiple Amazon Redshift Serverless workgroups with AI-driven scaling, primarily 128 Redshift Processing Units (RPUs) base, up to 1,024 for heavy workloads. Amazon Redshift data sharing distributes rated data across workgroups without ETL or data copies, across all consumers in real time.
As part of the billing pipeline, three Amazon Redshift Serverless workgroups serve distinct workloads:
Metering workgroup
The Amazon Redshift Serverless workgroup hosts the auto-refreshing MVs that deduplicate and aggregate billing events and stores over half a trillion events available for immediate querying. It writes curated financial datasets directly to Amazon S3 as Iceberg tables, indexed with AWS Glue Data Catalog.
Rating workgroup
This Amazon Redshift Serverless workgroup processes pricing configuration information, from aggregated Zero-ETL sources. It then applies pricing to usage aggregates to create the central system of record for rated usage. These core datasets are then shared with Bookkeeper for use-case-specific delivery.
Bookkeeper workgroup (invoice generation)
The Amazon Redshift Serverless workgroup for invoice generation receives rated data through data shares from the rating workgroup. Data shares from this workgroup are also used to feed the Customer Usage API workgroup. This also stores priced aggregate artifacts formatted as invoice line items for the invoicing platform. It processes the invoicing platform’s return data to Amazon S3 and writes to Amazon DynamoDB for customer console experiences.
In addition, the following Amazon Redshift Serverless workgroup serves Twilio’s end customers through their public APIs:
Customer Usage API workgroup
This dedicated Amazon Redshift Serverless workgroup receives rated usage data and product catalog configurations through stored procedures from the source systems and metered aggregates through data shares from the bookkeeper workgroup. It serves near real-time billing queries for the customer-facing usage API without impacting the billing pipeline’s compute resources.
4. Consumption layer
Processed billing data serves five downstream consumers:
- Invoicing platform — The Bookkeeper workgroup delivers priced aggregates as invoice line items for customer billing.
- Twilio Public APIs — The Customer Usage API workgroup powers near real-time billing queries for customers, backed by live data shares from the Metering workgroup.
- One Admin — Internal billing operations dashboards served directly from Amazon Redshift with sub-minute response times.
- One Console — Customer self-service experiences, for example balance inquiries, CSV invoice downloads, and so on, are served from Amazon DynamoDB. The Bookkeeper workgroup loads finalized invoice data returned from the invoicing platform to DynamoDB for low-latency API serving end-users.
- Product Teams — Amazon Athena queries curated financial datasets on Iceberg tables through the AWS Glue Data Catalog, providing broader organizational access without direct Amazon Redshift access.
Results
Twilio’s modernized billing engine cut infrastructure costs by 70% while accelerating billing recalculations by 12x, all while processing billions of events a month. Twilio launched the new billing system in production in mid-2026, providing rated usage for real customer invoices through Zuora. This new engine modernized on Amazon Redshift Serverless delivered measurable improvements across every dimension, which include:
- 70% lower infrastructure cost — Ongoing costs for the billing system dropped by 70%, while the system scaled to store almost two trillion events that the legacy architecture could not accommodate.
- 12x faster billing recalculation — Full month recalculations lowered from over 6 hours down to 30 minutes, along with retroactive pricing changes and scenario modeling that were previously impractical.
- Over 80% reduction in processing time — 80-85% operational gain across the pipeline, with the team meeting business service level agreements (SLAs).
- Sub-minute financial reporting — Finance dashboards moved from daily batch refreshes to sub-minute response times, providing near real-time visibility into billing data.
- Complete pricing flexibility — The rating engine now manages Twilio’s complete catalog of 200 million price points, up from a constrained subset, with no duplicate charges or missed events.
- Leaner operation — A small team manages what previously required a much larger operations team.
Most significantly, engineers shifted from firefighting pipeline failures to building new product capabilities, a cultural transformation driven by reliability and simplicity of the serverless architecture.
Conclusion
In this post, we showed how Twilio modernized their legacy billing system by adopting Amazon Redshift Serverless as the foundation for a financial-grade, real-time billing system at scale.
The modernized architecture is driven by streaming ingestion using Amazon MSK along with Amazon Redshift Zero-ETL from Amazon Aurora MySQL. With direct writes to Amazon S3 Iceberg tables cataloged with AWS Glue Data Catalog and dbt-driven transformations, Twilio built a billing engine that scales with their business. The new solution helped engineering teams to focus on billing product innovation rather than infrastructure operations. Learn more about how Amazon Redshift helped Twilio redesign their billing engine from their talk at AWS re:Invent 2025, where they co-presented with the Amazon Redshift and Amazon Athena service teams.
To learn more about the AWS services used to build modern data solutions on AWS, refer to the AWS public documentation and stay up to date through the AWS Big Data Blog.
