What does this AWS Solutions Implementation do?

This solution automatically receives new AWS Data Exchange dataset revisions and initiates downstream processing when new data becomes available from a data provider.

This solution only receives dataset revisions from one data provider at a time. If you want to receive data from multiple providers, deploy the solution multiple times. Data providers can use the AWS Data Exchange Publisher Coordinator solution to automate publishing of their AWS Data Exchange dataset revisions.

AWS Solutions Implementation overview

The diagram below presents the architecture you can automatically deploy using the solution's implementation guide and accompanying AWS CloudFormation template.

AWS Data Exchange Subscriber Coordinator | Architecture Diagram
 Click to enlarge

AWS Data Exchange Subscriber Coordinator Solutions Implementation architecture

The AWS CloudFormation template deploys all of the serverless infrastructure necessary to coordinate the download steps required for AWS Data Exchange. This includes the following:

  • Three AWS Lambda functions to start an AWS Data Exchange job, periodically check its progress, and get the job for the downstream hook function.
  • An AWS Step Functions to orchestrate the AWS functions that download new revisions to the AWS Data Exchange dataset. 
  • An Amazon CloudWatch Events rule to identify the publishing of any new revisions by publishers.
  • AWS Identity and Access Management (IAM) roles that provide an AWS Step Functions and three associated Lambda functions with permissions to initiate their API calls.

AWS Data Exchange Subscriber Coordinator

Version 1.0.0
Released: 12/2020
Author: AWS

Estimated deployment time: 5 min

Source Code  CloudFormation template 
Use the button below to subscribe to updates for this Solutions Implementation.
Note: To subscribe to RSS updates, you must have an RSS plug-in enabled for the browser you are using.
Did this Solutions Implementation help you?
Provide feedback 

Features

End-to-end automation

After the assets have been delivered to your S3 bucket; this solution initiates a Lambda hook function. This Lambda function can be configured to notify you of the new data, start an AWS Glue extract, transform, and load (ETL) job, or begin another processing workflow depending on your business needs.

Visualization

AWS Step Functions provides observability over each orchestration step, every run.

Reliability

Through seamless integration with multiple Amazon services, this solution provides consistent product management for your data product subscriptions.
Build icon
Deploy a Solution yourself

Browse our library of AWS Solutions Implementations to get answers to common architectural problems.

Learn more 
Find an APN partner
Find an APN Partner

Find AWS certified consulting and technology partners to help you get started.

Learn more 
Explore icon
Explore Solutions Consulting Offers

Browse our portfolio of Consulting Offers to get AWS-vetted help with solution deployment.

Learn more