[SEO Subhead]
This Guidance details the best practices for integrating the Amazon Ads and Amazon Selling Partner APIs. From creating a reporting workflow with AWS Step Functions, to storing and processing data using Amazon Simple Storage Service (Amazon S3) with AWS Glue, this Guidance covers the best approach to designing an Amazon Ads and Amazon Retail data pipeline on AWS. The data accumulated from this pipeline can be combined with data retrieved from Amazon Marketing Cloud (AMC) through Amazon Marketing Cloud Insights on AWS to formulate an Amazon Ads and Amazon Retail data lake on AWS. This data lake can then be visualized with a business intelligence tool, like Amazon QuickSight.
Please note: [Disclaimer]
Architecture Diagram
[Architecture diagram description]
Step 1
Amazon EventBridge is used to schedule a job which starts an AWS Step Functions state machine. The state machine processes a series of AWS Lambda functions to facilitate the creation of reports.
Step 2
The state machine invokes a Lambda function (Create Request Initiation) to create a report request from the Amazon Ads API or Amazon Selling Partner API. API credentials should be stored within AWS Secrets Manager and used when making calls to the APIs.
The state machine then moves into a series of polling steps, invoking a Lambda function (Check Report Status) to check the report request status, before finally downloading the report. The metadata for each report downloaded is stored in Amazon DynamoDB.
Step 3
The Lambda function (Download Report) writes the report into a raw Amazon Simple Storage Service (Amazon S3) bucket with a prefix which contains the specific report type and report date. Lambda uses AWS Key Management Service (AWS KMS), managed by AWS, to encrypt the reports as they’re written to the Amazon S3 bucket.
Step 4
A Step Functions state machine is invoked by notifications from Amazon S3 objects as they’re inserted into the bucket. When no more objects are received after a given time, the data transformation Step Functions state machine starts and invokes a Lambda function.
Step 5
A Lambda function (Update Metadata) stores the task token for Step Functions initiation ID in the DynamoDB table. Data is processed utilizing an AWS Glue job to read the data from the raw Amazon S3 bucket, and transformed to a usable format.
Step 6
The AWS Glue job writes the transformed data to the standard Amazon S3 bucket along with AWS Glue Data Catalog metadata. The AWS KMS Customer Managed Key (CMK) is used to encrypt the bucket contents and the AWS Glue metadata.
Step 7
Amazon Simple Notification Service (Amazon SNS) is used to send notifications as new data is transformed into the standard Amazon S3 bucket.
Step 1
Amazon EventBridge is used to schedule a job which starts an AWS Step Functions state machine. The state machine processes a series of AWS Lambda functions to facilitate the creation of reports.
Step 2
The state machine invokes a Lambda function (Create Request Initiation) to create a report request from the Amazon Ads API or Amazon Selling Partner API. API credentials should be stored within AWS Secrets Manager and used when making calls to the APIs.
The state machine then moves into a series of polling steps, invoking a Lambda function (Check Report Status) to check the report request status, before finally downloading the report. The metadata for each report downloaded is stored in Amazon DynamoDB.
Step 3
The Lambda function (Download Report) writes the report into a raw Amazon Simple Storage Service (Amazon S3) bucket with a prefix which contains the specific report type and report date. Lambda uses AWS Key Management Service (AWS KMS), managed by AWS, to encrypt the reports as they’re written to the Amazon S3 bucket.
Step 4
A Step Functions state machine is invoked by notifications from Amazon S3 objects as they’re inserted into the bucket. When no more objects are received after a given time, the data transformation Step Functions state machine starts and invokes a Lambda function.
Step 5
A Lambda function (Update Metadata) stores the task token for Step Functions initiation ID in the DynamoDB table. Data is processed utilizing an AWS Glue job to read the data from the raw Amazon S3 bucket, and transformed to a usable format.
Step 6
The AWS Glue job writes the transformed data to the standard Amazon S3 bucket along with AWS Glue Data Catalog metadata. The AWS KMS Customer Managed Key (CMK) is used to encrypt the bucket contents and the AWS Glue metadata.
Step 7
Amazon Simple Notification Service (Amazon SNS) is used to send notifications as new data is transformed into the standard Amazon S3 bucket.
Well-Architected Pillars
The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.
The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.
-
Operational ExcellenceRead the Operational Excellence whitepaper
Operational excellence is the focus on running and monitoring systems effectively, and we accomplish that here with the help of Step Functions, which is used to visualize the overall workflow when the API report is created and downloaded. Additionally, Amazon CloudWatch can be used in this Guidance to help you visualize metrics and logs across all services used. Lastly, Amazon SNS lets you know when reports have succeeded or failed.
-
SecurityRead the Security whitepaper
AWS Identity and Access Management (IAM), CloudWatch, AWS KMS, and Secrets Manager collectively protect data, systems, and assets in ways that improves your security posture. First, IAM policies are scoped down to provide least privilege access required for the services to function properly. By scoping IAM policies to the minimum permission required, you limit unauthorized access to resources. Second, AWS KMS provides encryption for both data at-rest and data in-transit using Pretty Good Privacy (PGP) encryption of data files. Third, Secrets Manager provides secure storage for API keys and hashes. AWS KMS and Secrets Manager are used to ensure that data and sensitive information is secured properly using encryption and hashing. And fourth, CloudWatch monitors logs and metrics across all services used to detect unexpected and unwanted behaviour.
-
ReliabilityRead the Reliability whitepaper
Lambda, Amazon S3, and DynamoDB help ensure your workloads perform their intended functions correctly and consistently, as well as help your workloads recover quickly from failure. With Lambda, you can initiate code without managing infrastructure. It also runs your function in multiple Availability Zones to ensure that it is available to process events in case of a service interruption in a single zone. Amazon S3 stores multiple copies of data across Availability Zones and provides 99.999999999% (11 9's) of durability of the data stored within the buckets. Furthermore, DynamoDB provides a serverless data store and stores multiple copies of data across Availability Zones while maintaining consistent and fast performance.
-
Performance EfficiencyRead the Performance Efficiency whitepaper
Lambda and AWS Glue help you structure and streamline your allocation of IT and computing resources. Lambda supports parallelization and concurrency with automatic scaling and can handle many concurrent processes to support the ingestion of multiple data pipeline workflows. AWS Glue jobs can scale both horizontally and vertically by letting you set the number of workers and the worker type to support small and large extract, transform, and load (ETL) processes.
-
Cost OptimizationRead the Cost Optimization whitepaper
AWS Glue is a serverless ETL service and only incurs cost when an AWS Glue job is occurring. When no AWS Glue job occurs, AWS Glue does not incur cost. And, AWS Glue scales both vertically and horizontally, allowing you to tune your ETL processes to be the most cost efficient. AWS Glue is cost efficient, resilient, highly performant, and scales to support any dataset transformation size. In addition, Amazon S3 stores reporting data and allows you to choose how data is stored into different tiers. You can configure Amazon S3 Lifecycle Policies to ensure older data is stored on storage tiers with less cost.
-
SustainabilityRead the Sustainability whitepaper
Lambda offers autoscaling, as well as the ability to scale from zero to peak demands to ensure only the required capacity is utilized. Also, DynamoDB removes unused resources from the database based on time to live (TTL), set by you, so only necessary storage is utilized. DynamoDB supports both provisioned and on-demand capacity to ensure only the necessary capacity is used to support the workload. Data stored in DynamoDB is compressed so minimal storage space is utilized. These services help ensure the lowest footprint of compute and data storage while lowering overall carbon generation, as services are only utilized as-needed with the most performant resources available.
Implementation Resources
A detailed guide is provided to experiment and use within your AWS account. Each stage of building the Guidance, including deployment, usage, and cleanup, is examined to prepare it for deployment.
The sample code is a starting point. It is industry validated, prescriptive but not definitive, and a peek under the hood to help you begin.
Related Content
[Title]
Disclaimer
The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.
References to third-party services or organizations in this Guidance do not imply an endorsement, sponsorship, or affiliation between Amazon or AWS and the third party. Guidance from AWS is a technical starting point, and you can customize your integration with third-party services when you deploy the architecture.