What does this AWS Solutions Implementation do?

This solution automates the steps to create a new revision for an existing AWS Data Exchange dataset in the AWS Marketplace. Instead of manually uploading and downloading dataset revisions, simply store the data you want to publish in an Amazon Simple Storage Service (Amazon S3) asset bucket and create a manifest file that references each of the files you plan to publish. When you upload the manifest file to the Amazon S3 manifest bucket, this solution initiates the process to publish a new revision. It invokes the creation of a new job in AWS Data Exchange, adds each of the files referenced in the manifest file to a new revision, then finalizes the revision once all of the assets have been added.

To retrieve the new data to their Amazon S3 bucket location as soon as the data publication completes, subscribers can use the AWS Data Exchange Subscriber Coordinator solution.

AWS Solutions Implementation overview

The diagram below presents the architecture you can automatically deploy using the solution's implementation guide and accompanying AWS CloudFormation template.

AWS Data Exchange Publisher Coordinator | Architecture Diagram
 Click to enlarge

AWS Data Exchange Publisher Coordinator Solutions Implementation architecture

The AWS CloudFormation template creates an Amazon S3 bucket for submitting manifests when a dataset is ready for publication. It also deploys all of the serverless infrastructure necessary to coordinate the publication steps required for AWS Data Exchange. This includes the following:

  • An Amazon S3 bucket where manifest files can be uploaded to.
  • Four AWS Lambda functions: including one function that initiates the AWS Step Functions; and three functions to create a new revision of an AWS Data Exchange product, monitor job run, and finalize and publish the revision.
  • An AWS Step Functions to orchestrate the management of the publishing process. 
  • AWS Identity and Access Management (IAM) roles that provide an AWS Step Functions and associated AWS Lambda functions with permissions to run their API calls.

 

AWS Data Exchange Publisher Coordinator

Version 1.0.0
Released: 12/2020
Author: AWS

Estimated deployment time: 5 min

Source Code  CloudFormation template 
Use the button below to subscribe to updates for this Solutions Implementation.
Note: To subscribe to RSS updates, you must have an RSS plug-in enabled for the browser you are using.
Did this Solutions Implementation help you?
Provide feedback 

Features

Automation

This solution automatically adds all the files to a new dataset revision and finalizes your revision in the product catalog so that all of your subscribers will be notified of the new dataset revision.

Visualization

AWS Step Functions provides observability over each orchestration step, on every run.

Reliability

Through seamless integration with multiple Amazon services, this solution provides consistent product management for your data product publications.
Build icon
Deploy a Solution yourself

Browse our library of AWS Solutions Implementations to get answers to common architectural problems.

Learn more 
Find an APN partner
Find an APN Partner

Find AWS certified consulting and technology partners to help you get started.

Learn more 
Explore icon
Explore Solutions Consulting Offers

Browse our portfolio of Consulting Offers to get AWS-vetted help with solution deployment.

Learn more