This Guidance helps ad-tech companies capture Open Real-Time Bidding (OpenRTB) events and establish a foundation for near real-time and batch analytics. During a programmatic advertising transaction, a demand-side platform (DSP) service generates a series of events. Capturing these events helps ad-tech companies keep their budgets updated and understand signals for optimizing the bid response. By tracking events, such as a successful win bid, advertisers can better measure the effectiveness of their campaigns. They can also analyze these events to make informed decisions for future bids.
For an overview of RTB and a description of the events that advertisers generate to win a bid request, visit Guidance for Building a Real Time Bidder for Advertising on AWS.
Please note: [Disclaimer]
Architecture Diagram
Step 1
The supply-side platform (SSP) receives an ad request from a publisher and launches an auction.
Step 2
An OpenRTB bid request is sent to a DSP public endpoint that is configured on an Elastic Load Balancer.
Step 3
The bidder application service receives the bid request. This service runs on Amazon Elastic Kubernetes Service (Amazon EKS) within an Amazon Virtual Private Cloud (VPC).
Step 4
The bid request-response event capture service is hosted on a different container pod in the same cluster. To reduce latency of the bid response, combine the publishing of the bid request and response event as a single call per bid asynchronously to the bid request-response event capture service endpoint.
Step 5
Post bid events capture service is hosted on a separate container pod that exposes the service to SSPs. This service is used to receive post bid events.
Step 6
Implement the event capture service in Java to take advantage of Amazon Kinesis Producer Library (KPL). KPL simplifies implementation of an asynchronous producer application and reduces costs for sending data to the Amazon Kinesis Data Streams API.
Step 7
The event messages are routed to Kinesis Data Streams through a dedicated VPC endpoint.
Step 8
Amazon Kinesis Data Firehose consumes these aggregated records and deaggregates and sends individual events to Amazon Simple Storage Service (Amazon S3) for long-term storage and analytics.
Step 9
Amazon CloudWatch captures application logs for traceability.
Step 10
AWS Key Management Service (AWS KMS) stores and manages encryption keys used for securing persisted data in Kinesis Data Streams and Amazon S3.
Step 1
The supply-side platform (SSP) receives an ad request from a publisher and launches an auction.
Step 2
An OpenRTB bid request is sent to a DSP public endpoint that is configured on an Elastic Load Balancer.
Step 3
The bidder application service receives the bid request. This service runs on Amazon Elastic Kubernetes Service (Amazon EKS) within an Amazon Virtual Private Cloud (VPC).
Step 4
The bid request-response event capture service is hosted on a different container pod in the same cluster. To reduce latency of the bid response, combine the publishing of the bid request and response event as a single call per bid asynchronously to the bid request-response event capture service endpoint.
Step 5
Post bid events capture service is hosted on a separate container pod that exposes the service to SSPs. This service is used to receive post bid events.
Step 6
Implement the event capture service in Java to take advantage of Amazon Kinesis Producer Library (KPL). KPL simplifies implementation of an asynchronous producer application and reduces costs for sending data to the Amazon Kinesis Data Streams API.
Step 7
The event messages are routed to Kinesis Data Streams through a dedicated VPC endpoint.
Step 8
Amazon Kinesis Data Firehose consumes these aggregated records and deaggregates and sends individual events to Amazon Simple Storage Service (Amazon S3) for long-term storage and analytics.
Step 9
Amazon CloudWatch captures application logs for traceability.
Step 10
AWS Key Management Service (AWS KMS) stores and manages encryption keys used for securing persisted data in Kinesis Data Streams and Amazon S3.
Well-Architected Pillars
The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.
The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.
-
Operational ExcellenceRead the Operational Excellence whitepaper
The bidder application uses a microservices approach, meaning that the bid request-response event capture and post bid events capture services are decoupled from the bidder service. This enables independent release cycles and scaling mechanisms based on the transactions-per-second (TPS) requirements. For example, the post bid event capture TPS is multiple orders of magnitude less than other components and can be configured independently.
-
SecurityRead the Security whitepaperData at rest in the Amazon S3 bucket is encrypted with AWS KMS keys. Data in transit is encrypted and transferred over HTTPS. The Amazon EKS clusters run in a VPC, and the connectivity between Amazon EKS and Kinesis is routed through a secure VPC endpoint.
-
ReliabilityRead the Reliability whitepaper
The bid request-response and post bid event capture services are decoupled from the code bidder application. The bidder application needs to implement throttling, retry, and backpressure accordingly. Additionally, the architecture promotes a stateless implementation using containers. The KPL implements throttling, retry, and asynchronous publish of events.
-
Performance EfficiencyRead the Performance Efficiency whitepaper
All components of this Guidance are co-located in a single Region and Availability Zone, and you can use automated deployments to deploy the architecture into any Region that meets your data residency and latency requirements.
-
Cost OptimizationRead the Cost Optimization whitepaper
With serverless services, you pay only for the resources you consume. The Amazon EKS cluster is the only component in this architecture that does not use serverless services. To meet the high TPS workload, we recommend a hybrid consumption model for the Amazon EKS cluster. Amazon Elastic Compute Cloud (Amazon EC2) Spot instances provide up to a 90% discount compared to on-demand instances and can support additional bursts in workloads. Additionally, the services in the bidder application run within an Amazon VPC and are deployed in a single Availability Zone to minimize data transfer costs.
-
SustainabilityRead the Sustainability whitepaper
With Amazon EKS, you can implement scalability and elasticity. By decoupling components through microservices, you can allocate the right instance type and the right number of instances to dynamic workloads, helping you to not overprovision resources.
Implementation Resources
A detailed guide is provided to experiment and use within your AWS account. Each stage of building the Guidance, including deployment, usage, and cleanup, is examined to prepare it for deployment.
The sample code is a starting point. It is industry validated, prescriptive but not definitive, and a peek under the hood to help you begin.
Related Content
Guidance for Building a Real Time Bidder for Advertising on AWS
Disclaimer
The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.
References to third-party services or organizations in this Guidance do not imply an endorsement, sponsorship, or affiliation between Amazon or AWS and the third party. Guidance from AWS is a technical starting point, and you can customize your integration with third-party services when you deploy the architecture.