AWS for Industries
How Zeta created a cloud native marketing platform on AWS
Zeta Global is a leading cloud-based marketing technology company that empowers enterprises to acquire, grow, and retain customers. The Zeta Marketing Platform (ZMP) enriches your data with 225+ million consumer profiles. It offers proprietary and transactional signals, providing individualized experiences across the customer journey.
Zeta needed to accelerate the pace of innovation to keep up with changing customer needs in the advertising space. AWS provides services that remove the heavy lifting of infrastructure management typical with data centers, and has less turnaround time to provision resources. In addition, advertising workloads are extremely time sensitive. Having resources available as needs change for the customer is of prime importance.
In this blog, we show how various technical and functional needs were addressed by ZMP. We also review design features that enabled Zeta to meet business requirements.
The ZMP platform evolution
Zeta’s platform requires nearly continuous, reliable performance and delivery for its customers. Zeta chose Amazon Web Services (AWS) to achieve and maintain this performance level by taking advantage of the scalability and resources offered by AWS. Making operations more efficient, lowering costs, and delivering stability with scalability has enabled Zeta to set customer expectations and increase satisfaction.
As the ZMP grew and added features with new capabilities, it evolved to become a true omnichannel platform. It serves customers’ marketing needs across email, display, video, website, social media, and much more. The ZMP processes 1 trillion content consumption signals. They surface 2,500 demographic and behavioral attributes per individual with 225+ million US individuals. For real-time marketing, Zeta wanted to design a platform that scaled with demand, provided timely responses, and securely handled customer data.
For Zeta’s platform, using a traditional approach to application design with infrastructure procured in data centers would be challenging. Automating on-demand compute scaling during peak campaigns wasn’t a solution, because data centers need to acquire resources from vendors and on-demand capacity is limited. In addition, traditional virtual machines are inflexible for variable memory allocation and storage needs. Managed services on AWS enabled Zeta to keep the pace of expansion of their platform, while off-loading the operations tasks to AWS.
Figure 1. High-level cross-Region Zeta Marketing Platform architecture
Secure customer data handling
ZMP processes about 2 TB of data coming from customers everyday. Ingesting terabytes of data from multiple sources securely while effectively managing the variance in transfer volumes can impact performance. As illustrated in Figure 1, Zeta uses AWS Transfer Family for SFTP to securely transfer customer data into Amazon S3. This enables Zeta to provision FTP servers and scale servers on demand to meet transfer needs.
Zeta uses Amazon API Gateway to manage its APIs and provide secure customer access to the platform AWS CloudFormation. User authentication occurs at the ZMP application, which is passed from FTP to the ZMP app via API Gateway. AWS Lambda is used to transfer files from SFTP to S3. Amazon CloudWatch is used to maintain the transfer log from SFTP to S3.
Automating infrastructure provisioning and cost optimization
Automated launching of resources using CloudFormation templates helped Zeta reduce operational costs and monitor services, resources, and usage.
Infrastructure provisioning involves several steps; documentation is one of the most important ones. During provisioning in AWS, it’s important to tag a resource or a service. Tags such as environment, service role, team ownership, and more, provide instant inventory and tracking. Tagging enables Zeta to make sound budgeting and cost allocation decisions.
AWS resources and service provisioning are automated using Salt, Terraform, and CloudFormation. Mandatory tags are created to understand dependencies, usage, cost allocation, and budgeting.
Zeta keeps the costs predictable by implementing scaling limitations in their AWS Auto Scaling groups configurations. AWS Cost Explorer is used to analyze infrastructure spending by service and resource. Wherever predictable compute capacity can be provisioned Reserved Instances are used, which helped save up to 30% in computation costs in the last year.
Processing terabytes of data every day
The data ZMP receives on a daily basis must be ingested, cleansed, transformed for data analytics, and run through machine learning models.
Processing that amount of incremental data requires peak performance. This is critical when new campaigns are processed with a data-processing expectation that it complete within the hour, regardless of size. The platform analyzes billions of structured and unstructured data points to predict consumer intent, by using sophisticated artificial intelligence to personalize experiences at scale.
The resources and performance required by this type of processing is managed by monitoring the performance and scaling resources using AWS Auto Scaling groups. When the performance threshold for CPU utilization and available memory is reached, an Auto Scaling group is triggered to allocate more resources. The automation of infrastructure scaling using performance thresholds permits Zeta to meet their service level agreements (SLAs) for data processing performance levels.
By using Apache Kafka lags and thresholds such as queue length, Auto Scaling groups can spin up new resources for data processing. This helps handle a variety of requests during peak times, and reduces the waste of resources during off-peak hours.
People Service is a module of the ZMP that is used for creating and maintaining identity records, including attributes. It is responsible for creating and maintaining index sets between records. The network team monitors network volume spike with a threshold of 500k packets per second to scale servers using CloudWatch metrics. CloudWatch metrics can also be used to provision other resources, limit sudden traffic spikes, or send notifications to the operations team.
Conclusion
In this blog post, we showed how Zeta has built a strong foundation for its marketing platform on AWS. A key objective in Zeta’s 2025 growth plan is to significantly increase the number of clients on their cloud platform. This growth will be possible based on scalability of services, resources, and the rapid provisioning and implementation available through AWS. Using the elasticity of the cloud and services provided by AWS, you can create a completely cloud native marketing platform on AWS. With artificial intelligence (AI) innovations and new services, there will be even more opportunities in the future.
For further reading:
- Top re:Invent 2021 announcements for the advertising and marketing technology industry
- Running Ad Tech Workloads on AWS with Aerospike at Petabyte Scale