AWS Data Exchange Publisher Coordinator automates the steps to create a new revision for an existing AWS Data Exchange dataset in the AWS Marketplace. Instead of manually uploading and downloading dataset revisions, simply store the data you want to publish in an Amazon Simple Storage Service (Amazon S3) asset bucket and create a manifest file that references each of the files you plan to publish. When you upload the manifest file to the Amazon S3 manifest bucket, this solution initiates the process to publish a new revision. It invokes the creation of a new job in AWS Data Exchange, adds each of the files referenced in the manifest file to a new revision, then finalizes the revision once all of the assets have been added.
To retrieve the new data to their Amazon S3 bucket location as soon as the data publication completes, subscribers can use the automatic export revisions feature for AWS Data Exchange. For more information, refer to the AWS Data Exchange User Guide.
Overview
The diagram below presents the architecture you can build using the example code on GitHub.
AWS Data Exchange Publisher Coordinator architecture
AWS Data Exchange Publisher Coordinator creates an Amazon S3 bucket for submitting manifests when a dataset is ready for publication. It also deploys all of the serverless infrastructure necessary to coordinate the publication steps required for AWS Data Exchange. This includes the following:
- An Amazon S3 bucket where manifest files can be uploaded to.
- Four AWS Lambda functions: including one function that initiates the AWS Step Functions; and three functions to create a new revision of an AWS Data Exchange product, monitor job run, and finalize and publish the revision.
- An AWS Step Functions to orchestrate the management of the publishing process.
- AWS Identity and Access Management (IAM) roles that provide an AWS Step Functions and associated AWS Lambda functions with permissions to run their API calls.
AWS Data Exchange Publisher Coordinator
Version 1.0.0
Released: 12/2020
Author: AWS
Additional resources
Features
Automation
Reliability
Visualization
Browse our library of AWS Solutions to get answers to common architectural problems.
Find AWS Partners to help you get started.
Find prescriptive architectural diagrams, sample code, and technical content for common use cases.