Guidance for Connecting Data Sources for Advertising and Marketing Analytical Workloads on AWS

Step 1
Amazon EventBridge schedules a job that starts an AWS Step Functions state machine. The state machine processes a series of AWS Lambda functions to facilitate report creation.

Step 2
The state machine invokes a Lambda function (Create Request Execution) to create a report request from the Amazon Ads API or Amazon Selling Partner API.

Step 3
Store API credentials within AWS Secrets Manager, and use them when making calls to the APIs. The state machine then moves into a series of polling steps, invoking a Lambda function (Check Report Status) to check the report request status before downloading the report.

Step 4
Amazon DynamoDB stores the metadata for each downloaded report.

Step 5
The Lambda function (Download Report) writes the report into a raw Amazon Simple Storage Service (Amazon S3) bucket with a prefix that contains the specific report type and report date. Lambda uses the Amazon-managed AWS Key Management Service (AWS KMS) key to encrypt the reports as they’re written to the S3 bucket.

Step 6
A Step Functions state machine is invoked by notifications from S3 objects as they’re inserted into the bucket. When no more objects are received after a given time, the data transformation Step Functions state machine starts and invokes a Lambda function.

Step 7
A Lambda function (Update Metadata) stores the task token for the Step Functions execution ID in the DynamoDB table. An AWS Glue job processes the data and reads the data from the raw S3 bucket and transforms it to a usable format.

Step 1
EventBridge schedules a job that starts the Step Functions state machine, which includes the launch of Amazon Appflow.

Step 2
The Amazon AppFlow flow starts by opening the connection to the external data provider and requesting data. The external provider responds with data to Amazon AppFlow.

Step 3
Amazon AppFlow puts the data into the raw S3 bucket and specified prefix. Amazon AppFlow uses an AWS KMS key to encrypt objects written to the raw S3 bucket.

Step 4
A Step Functions state machine is initiated by notifications from Amazon S3 as the Guidance stores objects in the bucket. When no more objects are received after a given time, the data transformation Step Functions starts and continues with the common flow.

Step 5
A Lambda function stores the task token for the Step Functions execution ID in the DynamoDB table and invokes the AWS Glue job. The AWS Glue job runs, reads data from the raw bucket, and transforms it. AWS Glue writes the transformed data to the clean S3 bucket. AWS Glue Data Catalog metadata is also written out. The AWS KMS customer managed key (CMK) created by this stack encrypts the bucket contents and the AWS Glue metadata.

Step 6
By storing the Step Functions workflow metadata and execution information, DynamoDB builds unified observability. Amazon Simple Notification Service (Amazon SNS) then generates observability notifications.

Step 7
Use the following AWS services for security and access: AWS Identity and Access Management (IAM) enables least privilege access to specific resources and operations. AWS KMS provides encryption for data at rest and data in transit. Secrets Manager provides hashing keys for PII data. Amazon CloudWatch monitors logs and metrics across all services used in this Guidance.

Step 1
External data sources push raw data files (such as CSV) into a daily partitioned Landing Zone S3 bucket. Refer to external documentation to set up push job inputs like S3 bucket location, access key, and schedule frequency.

Step 2
Create a rule in EventBridge to schedule a Step Functions standard workflow for data processing at required frequency.

Step 3
In the workflow, use a Lambda function to do file-level processing, such as Pretty Good Privacy (PGP) decryption. Place the decrypted file in a different S3 bucket prefix.

Step 4
Use AWS Glue jobs to process the decrypted data files in the Landing Zone S3 bucket, and write data in a separate Processed Zone S3 bucket. Write the object in read optimized Apache Parquet format. Apply attribute level transformation like SHA256 hashing to secure sensitive data. Apply partitioning scheme as needed to optimize reads.

Step 5
The AWS Glue crawler executes from the workflow to catalog the read optimized data in the Data Catalog.

Step 6
Use a Lambda function to do post processing activities, such as moving the source data files to an "archive" prefix location as part of clean-up.

Step 7
Use Amazon SNS to publish a workflow complete event and notify operators and users using email. Use HTTP or Topic options to integrate with other observability tools.

Step 8
Use the following AWS services for security and access: IAM enables least privilege access to specific resources and operations. AWS KMS provides encryption for data at rest and data in transit. Secrets Manager provides hashing keys for PII data. CloudWatch monitors logs and metrics across all services used in this Guidance.

Step 1a
Use AWS Glue and pre-built or marketplace connectors to extract data needed for advertising and marketing analytical use case from the relational database management system (RDBMS) in batch mode. AWS Glue will retrieve the data from data stores and load it to an S3 bucket. Amazon S3 is configured as a target for storing remote database files in parquet format.

Step 1b
Use AWS Database Migration Service (AWS DMS) to replicate data stored in compatible relational databases (on-premises or on a cloud) to AWS.

Step 2
A rule in EventBridge schedules a Step Functions standard workflow for post upload processing at required frequency.

Step 3
AWS Glue jobs and workflows do the row-level processing of the decrypted data files and write data in a separate S3 bucket. Write the object in read optimized Apache Parquet format. Apply attribute level transformation like SHA256 hashing to secure sensitive data. Apply custom partitioning scheme as needed to optimize reads.

Step 4
The AWS Glue crawler executes from the workflow to catalog the read optimized data in the Data Catalog.

Step 5
Publish a notification to Amazon SNS and notify operators of success or failure of the workflow. Use the HTTP or Topic option to integrate with other observability tools.

Step 6
Use the following AWS services for security and access: IAM enables least privilege access to specific resources and operations. AWS KMS provides encryption for data at rest and data in transit. Secrets Manager provides hashing keys for PII data. CloudWatch monitors logs and metrics across all services used in this Guidance.

Step 1
AWS Transfer Family securely migrates data stored in file systems (on-premises or on a cloud) that is needed for advertising and marketing analytical use cases.

Step 2
Raw files from remote servers are uploaded as-is into a raw S3 bucket.

Step 3
Transfer Family managed workflows complete post-upload processing of files, including decrypting, error checking, and formatting changes.

Step 4
Lambda completes custom post processing of data before sending it to storage.

Step 5
Processed data that is ready for analysis by applications and other data consumers is stored in a standard S3 bucket.

Step 6
Transfer Family managed workflows will invoke a Lambda function when post upload processing steps fail for a file.

Step 7
Amazon SNS publishes an exception event to notify users through email or other observability tools.

Step 8
Use the following AWS services for security and access: IAM enables least privilege access to specific resources and operations. AWS KMS provides encryption for data at rest and data in transit. Secrets Manager provides hashing keys for PII data. CloudWatch monitors logs and metrics across all services used in this Guidance.

Step 1
Install and configure AWS DataSync Agent on a virtual machine in the public cloud where the source Object Storage is hosted.

Step 2
The DataSync Agent and DataSync allow discovery and scheduling of data transfer for both the initial sync and continuous, ongoing sync.

Step 3
Configure DataSync to store the replicated data in a Landing Zone S3 bucket.

Step 4
Create a rule in EventBridge to schedule a Step Functions standard workflow for data processing at required frequency.

Step 5
In the workflow, use a Lambda function to do any necessary file- or object-level decryption, and invoke an AWS Glue task to normalize the data.

Step 6
Use AWS Glue jobs and workflows to do data processing of the decrypted data files, and write data in a separate S3 bucket.

Step 7
Write the object in read optimized Apache Parquet format. Apply data. Apply custom partitioning scheme as needed to attribute level transformation like SHA256 hashing to secure sensitive optimize reads.

Step 8
Create an AWS Glue crawler, and add it to the workflow to catalog the read optimized data in Data Catalog.

Step 9
Use another Lambda function to do post-processing activities, such as moving the source data files to an "archive" prefix location as part of clean-up and to save on storage costs.

Step 10
Use Amazon SNS to publish a workflow complete event and notify operators and users using email. Use HTTP or Topic option to integrate with other observability tools.

Step 11
Use the following AWS services for security and access: IAM enables least privilege access to specific resources and operations. AWS KMS provides encryption for data at rest and data in transit. Secrets Manager provides hashing keys for PII data. CloudWatch monitors logs and metrics across all services used in this Guidance.