Allows demand and sell-side platforms to query real-time data to inform bidding or auction logic through the Chrome Protected Audience API (PAAPI)
This Guidance demonstrates how to deploy the Google Chrome Privacy Sandbox Key/Value service within a trusted execution environment (TEE) on AWS. The Key/Value service allows implementers to fetch real-time signals to inform remarketing to custom audiences through the Protected Audience API (PAAPI). This real-time data helps ad buyers determine how to bid and helps sellers pick winning bids in a privacy-enhanced way. This Guidance intends to simplify the implementation of the Key/Value service while optimizing for cost and latency.
Please note: [Disclaimer]
Architecture Diagram
-
Overview
-
Data Loading
-
Overview
-
This architecture diagram shows an overview of how to deploy Privacy Sandbox’s Protected Audience API Key/Value service. For the data loading component, open the other tab.
Click to enlarge
Key/Value Service Deployment
Step 1
Privacy Sandbox’s Protected Audience API Key/Value service is deployed using a Google-provided terraform in your Amazon Virtual Private Cloud (Amazon VPC) environment.This Key/Value service runs within Amazon Elastic Compute Cloud (Amazon EC2), and an Amazon EC2 Auto Scaling group scales the service horizontally to meet the workload demand. The data is secured in an AWS Nitro Enclaves trusted runtime environment. The communication between the parent Auto Scaling group and the enclave is done using a secure local channel.
Step 2
Deploy additional AWS resources using a Google-provided terraform that provisions services and resources to load Key/Value data (in the form of delta or snapshot files) into Privacy Sandbox’s Key/Value service application.
Files are generated using the Privacy Sandbox data command line interface (CLI) application, which is hosted on Amazon Elastic Container Service (Amazon ECS). The Key/Value service subscribes to data updates using Amazon Simple Notification Service (Amazon SNS) and Amazon Simple Storage Service (Amazon S3) event notifications.
For low-latency updates, the Key/Value service listens to an Amazon SNS topic by long polling a subscribed Amazon Simple Queue Service (Amazon SQS) queue. VPC endpoints powered by AWS PrivateLink provide private connectivity between the Key/Value service VPC and other AWS resources.
Functional Flow
Step 3
During the Privacy Sandbox auction, an end user’s Chrome or Android browser (enabled with Privacy Sandbox features) sends a lookup request to the Key/Value service application’s endpoint URL. AWS WAF protects the application from common web exploits, and Elastic Load Balancing distributes the traffic to healthy application endpoints.Step 4
The Key/Value service application has secure outbound internet access through a NAT gateway to Google’s key management system, operated by a coordinator. The Key/Value service obtains a private key from Google’s key management system to decrypt the requests from the client device. Note that this is not AWS Key Management Service (AWS KMS).Step 5
The Key/Value service application looks up the matching data for the keys and returns it in encrypted form to the user’s Chrome or Android browser.Key/Value Service Deployment
Step 1
Privacy Sandbox’s Protected Audience API Key/Value service is deployed using a Google-provided terraform in your Amazon Virtual Private Cloud (Amazon VPC) environment.This Key/Value service runs within Amazon Elastic Compute Cloud (Amazon EC2), and an Amazon EC2 Auto Scaling group scales the service horizontally to meet the workload demand. The data is secured in an AWS Nitro Enclaves trusted runtime environment. The communication between the parent Auto Scaling group and the enclave is done using a secure local channel.
Step 2
Deploy additional AWS resources using a Google-provided terraform that provisions services and resources to load Key/Value data (in the form of delta or snapshot files) into Privacy Sandbox’s Key/Value service application.
Files are generated using the Privacy Sandbox data command line interface (CLI) application, which is hosted on Amazon Elastic Container Service (Amazon ECS). The Key/Value service subscribes to data updates using Amazon Simple Notification Service (Amazon SNS) and Amazon Simple Storage Service (Amazon S3) event notifications.
For low-latency updates, the Key/Value service listens to an Amazon SNS topic by long polling a subscribed Amazon Simple Queue Service (Amazon SQS) queue. VPC endpoints powered by AWS PrivateLink provide private connectivity between the Key/Value service VPC and other AWS resources.
Functional Flow
Step 3
During the Privacy Sandbox auction, an end user’s Chrome or Android browser (enabled with Privacy Sandbox features) sends a lookup request to the Key/Value service application’s endpoint URL. AWS WAF protects the application from common web exploits, and Elastic Load Balancing distributes the traffic to healthy application endpoints.Step 4
The Key/Value service application has secure outbound internet access through a NAT gateway to Google’s key management system, operated by a coordinator. The Key/Value service obtains a private key from Google’s key management system to decrypt the requests from the client device. Note that this is not AWS Key Management Service (AWS KMS).Step 5
The Key/Value service application looks up the matching data for the keys and returns it in encrypted form to the user’s Chrome or Android browser. -
Data Loading
-
This architecture diagram shows the data loading component and demonstrates patterns for ingesting the first-party data needed for real-time auction and bidding. For an overview, open the other tab.
Click to enlarge
Step 1
Your first-party data likely exists in multiple formats, and your system of records is part of your existing real-time bidding application. To support ad targeting through Privacy Sandbox APIs, your first-party data needs to be generated or copied to Privacy Sandbox’s Protected Audience API Key/Value service.
Step 2
Data to be stored in the Key/Value service is first sent to one of the three data collection services. Amazon SQS supports an API-to-API-based integration pattern. Amazon S3 supports a file-based integration pattern. Amazon Kinesis Data Streams supports an enhanced fan-out pattern for loading data to the Key/Value service.
Step 3
Real-time data flowing in through Amazon SQS and Kinesis Data Streams is processed by a data formatter application running in a Docker container and is hosted in Amazon ECS.Step 4
Batch data files uploaded to Amazon S3 generate an event invocation and run an Amazon EventBridge rule, which in turn implements an Amazon ECS task.Step 5
The data formatter application sends data to an S3 bucket and Amazon SNS for consumption by the Key/Value service.
Step 6
The Key/Value service is deployed separately, as explained in the overview architecture diagrams, using an infrastructure-as-code terraform stack. This stack provisions AWS resources that listen to incoming data rows by means of Amazon S3 or Amazon SNS, and it loads them into the Key/Value service running on Nitro Enclaves.Step 1
Your first-party data likely exists in multiple formats, and your system of records is part of your existing real-time bidding application. To support ad targeting through Privacy Sandbox APIs, your first-party data needs to be generated or copied to Privacy Sandbox’s Protected Audience API Key/Value service.
Step 2
Data to be stored in the Key/Value service is first sent to one of the three data collection services. Amazon SQS supports an API-to-API-based integration pattern. Amazon S3 supports a file-based integration pattern. Amazon Kinesis Data Streams supports an enhanced fan-out pattern for loading data to the Key/Value service.
Step 3
Real-time data flowing in through Amazon SQS and Kinesis Data Streams is processed by a data formatter application running in a Docker container and is hosted in Amazon ECS.Alternatively, you can also use an AWS Lambda function to process the data formatter application. The application uses AWS SDK and Google’s data CLI tool to receive and convert the incoming format to delta format, which the Key/Value service can read.
Step 4
Batch data files uploaded to Amazon S3 generate an event invocation and run an Amazon EventBridge rule, which in turn implements an Amazon ECS task.The Amazon ECS task runs the data formatter application, which uses AWS SDK, AWS Command Line Interface (AWS CLI), and Google’s data CLI tool to receive and convert the incoming format to delta format.
Step 5
The data formatter application sends data to an S3 bucket and Amazon SNS for consumption by the Key/Value service.
Step 6
The Key/Value service is deployed separately, as explained in the overview architecture diagrams, using an infrastructure-as-code terraform stack. This stack provisions AWS resources that listen to incoming data rows by means of Amazon S3 or Amazon SNS, and it loads them into the Key/Value service running on Nitro Enclaves.
Well-Architected Pillars
The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.
The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.
-
Operational ExcellenceRead the Operational Excellence whitepaper
Amazon CloudWatch, which provides logging and tracing (through AWS X-Ray) for the Key/Value service running in Nitro Enclaves and Amazon ECS, makes it easy for you to understand the performance characteristics. Additionally, Amazon Elastic Container Registry (Amazon ECR) lets you store and version containers for deployment to Amazon ECS, and Amazon EC2 Auto Scaling makes it easy to manage the scaling and compute of the Key/Value service running in Nitro Enclaves.
-
SecurityRead the Security whitepaper
AWS Identity and Access Management (IAM) policies enable you to scope services and resources to the minimum privilege required to operate. Amazon S3 encrypts data at rest using server-side encryption with Amazon S3 managed keys. AWS WAF provides distributed denial of service protection for the public endpoint to prevent malicious attacks, and you can easily create security rules to control bot traffic. Additionally, this Guidance uses Amazon VPC endpoints to enable the services running in Nitro Enclaves and Amazon ECS to privately connect to supported AWS services.
-
ReliabilityRead the Reliability whitepaper
This Guidance stores data in Amazon S3 before uploading it into the Key/Value service, so you can use S3 Versioning to preserve, replicate, retrieve, and restore every version of an object. This enables you to recover from unintended user actions and application failures. Additionally, this Guidance uses Auto scaling groups to spread compute across Availability Zones, and Elastic Load Balancing distributes traffic to healthy Amazon EC2 instances. Finally, Amazon SNS and Amazon SQS, which serve as an endpoint for low-latency data updates, provide a push-and-pull-based system to deliver data and facilitate data persistence before it is processed downstream.
-
Performance EfficiencyRead the Performance Efficiency whitepaper
Amazon ECS reduces the complexity of scaling containerized workloads on AWS. By using Amazon ECS containers that run on AWS Graviton processors, you can achieve higher throughputs and lower latencies for requests. Additionally, Amazon SNS can publish messages within milliseconds, so upstream applications can send time-critical messages to the Key/Value service through a push mechanism.
-
Cost OptimizationRead the Cost Optimization whitepaper
As a serverless service, Amazon S3 automatically scales your data storage, enabling you to optimize your costs based on the storage you actually use, and you can avoid the costs of provisioning and managing physical infrastructure. Additionally, Amazon S3 provides different storage classes that help you further optimize storage and reduce costs, and you can set up lifecycle rules to delete data after a period of time. Amazon ECS also helps optimize costs by dynamically scaling compute resources up and down based on demand, so you only pay for the resources used. Finally, by decreasing the number of empty receives to an empty queue, Amazon SQS long polling enhances the efficiency of processing inventory updates, further optimizing costs.
-
SustainabilityRead the Sustainability whitepaper
Instances powered by AWS Graviton processors enable you to serve more requests at lower latency and with up to 60 percent less energy than comparable Amazon EC2 instances, helping you lower the carbon footprint of your workloads. Additionally, Amazon SQS long polling improves resource efficiency by reducing API requests, minimizing network traffic, and optimizing resource utilization.
Implementation Resources
The sample code is a starting point. It is industry validated, prescriptive but not definitive, and a peek under the hood to help you begin.
Related Content
[Title]
Disclaimer
The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.
References to third-party services or organizations in this Guidance do not imply an endorsement, sponsorship, or affiliation between Amazon or AWS and the third party. Guidance from AWS is a technical starting point, and you can customize your integration with third-party services when you deploy the architecture.