Skip to main content

Guidance for Incremental Data Exports from Amazon DynamoDB to Amazon S3

I'm ready to deploy

Overview

This Guidance shows how the Amazon DynamoDB continuous incremental exports feature can help capture and transfer ongoing data changes between DynamoDB tables. By starting with a full export to set up a new table and then applying incremental updates, users can keep their tables synchronized across different AWS accounts and AWS Regions. This approach offers an alternative to traditional disaster recovery (DR) solutions, especially for situations where data recovery needs to happen within 30 minutes or longer. Unlike global tables, which require constant updates to a secondary table, the export method allows for creating and loading the secondary table only when needed for recovery. This flexibility makes it easier to manage data and maintain multiple copies effectively.

How it works

This architecture diagram demonstrates a serverless workflow to achieve continuous data exports from Amazon DynamoDB to Amazon Simple Storage Service (Amazon S3) using the DynamoDB incremental exports feature.

Download the architecture diagram

Deploy this Guidance

Ready to deploy? Review the sample code on GitHub for detailed deployment instructions to deploy as-is or customize to fit your needs.

Go to sample code

Well-Architected Pillars

The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.

The services used in this Guidance provide capabilities for infrastructure as code, workflow orchestration, serverless computing, and observability. Specifically, the AWS Cloud Development Kit (AWS CDK) allows the Guidance to be deployed using infrastructure as code, making it easy to manage and update. Step Functions provides a visual workflow with capabilities for orchestration, debugging, retries, and monitoring. Lambda enables complex functionality, such as time manipulation. All logs are centralized in CloudWatch, and X-Ray provides end-to-end traceability. Together, these services make this Guidance easy to deploy, manage, and troubleshoot.

Read the Operational Excellence whitepaper

This Guidance includes services with capabilities for access control, encryption, and configuration management. For example, AWS Identity and Access Management (IAM) allows granular control over AWS services to help ensure least privilege access, and the IAM roles used by Step Functions are scoped to only the necessary resources and services. The Amazon S3 bucket and Amazon SNS topic enforce SSL connectivity. In addition, encryption at rest is enabled using AWS KMS, while TLS helps ensure encrypted communication between services. This Guidance also uses the Systems Manager Parameter Store to store state information, further reducing the need for human interaction.

Read the Security whitepaper

Step Functions incorporates built-in mechanisms to handle exceptions and automatically retry when necessary while Systems Manager manages the required state information. These services allow the Guidance to automatically recover from known failure modes through the retry capabilities inherent to Step Functions, as well as the state information stored in the Systems Manager Parameter Store. Furthermore, as the workflow is entirely serverless, there is no need to manually manage capacity.

Read the Reliability whitepaper

This Guidance uses advanced serverless technologies so that the most suitable service is used for each specific task. For example, it uses the Systems Manager Parameter Store for managing state information and Lambda for handling complex time-related operations.

Read the Performance Efficiency whitepaper

The various services used within this Guidance, including Step Functions, can be fine-tuned through the adjustment of multiple parameters. This flexibility enables users to control the frequency at which the incremental exports are executed. For instance, implementing longer export windows would result in fewer runs, albeit potentially larger export files. Additionally, users can use the lifecycle management features of Amazon S3 to transition older data objects to more cost-effective storage classes, further optimizing the overall expenditure.

Read the Cost Optimization whitepaper

In a traditional, server-based infrastructure, organizations often need to provision computing resources to handle peak workloads, leading to significant over-provisioning and underutilization of hardware during off-peak periods. This inefficient use of resources can have a detrimental impact on the environment. By adopting a serverless approach, this Guidance helps ensure that computing resources are only used when actively required, automatically scaling the underlying infrastructure based on real-time demands. This dynamic scaling optimizes resource utilization and reduces the overall carbon footprint and energy consumption associated with the infrastructure.

Read the Sustainability whitepaper

Disclaimer

The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.