[SEO Subhead]
This Guidance shows how the Amazon DynamoDB continuous incremental exports feature can help capture and transfer ongoing data changes between DynamoDB tables. By starting with a full export to set up a new table and then applying incremental updates, users can keep their tables synchronized across different AWS accounts and AWS Regions. This approach offers an alternative to traditional disaster recovery (DR) solutions, especially for situations where data recovery needs to happen within 30 minutes or longer. Unlike global tables, which require constant updates to a secondary table, the export method allows for creating and loading the secondary table only when needed for recovery. This flexibility makes it easier to manage data and maintain multiple copies effectively.
Please note: [Disclaimer]
Architecture Diagram
[Architecture diagram description]
Step 1
The Amazon EventBridge Scheduler is configured to execute at a frequency exceeding the required export window with the capability to catch up on any missed exports if necessary.
Step 2
The AWS Step Functions workflow is initialized by retrieving the current state information from the AWS Systems Manager Parameter Store. This allows for the execution of state-specific actions, such as handling a paused workflow scenario.
Step 3
Perform any necessary sanity checks, such as verifying the existence of the table and confirming that point-in-time recovery (PITR) is enabled.
Step 4
The state information is evaluated to determine if a full export is required for the capture of the complete dataset from the table as the initial step.
Step 5
To initiate the export of the table, the workflow invokes the Amazon DynamoDB API.
Step 6
The export operation starts writing the data, along with the associated manifest and summary, to the specified Amazon Simple Storage Service (Amazon S3) bucket and prefix.
Step 7
The workflow pauses execution and waits for the full export operation to complete before proceeding to the next step. Additional validation checks can be performed at this stage to confirm whether the full export was successfully executed.
Step 8
State information is written to the Parameter Store to record the outcome.
Step 9
A notification is sent to the end user through an Amazon Simple Notification Service (Amazon SNS) topic.
Step 10
The workflow execution concludes, terminating in either a successful or failed state.
Step 11
The incremental export operation first verifies that PITR has remained continuously enabled since the completion of the full export. If PITR is disabled at any point, the PITR history is wiped, resulting in a gap in the data.
Step 12
The incremental export operation requires two time values. The start time value is passed as input to an AWS Lambda function, along with the desired export window. The Lambda function then returns the end time value, as well as any other necessary information.
Step 13
The workflow invokes the DynamoDB API to export the table with extra parameters as required by the incremental export API. This is then followed by a similar flow as before: the exported data is written to the Amazon S3 bucket.
Step 14
The workflow waits for the incremental export to finish before moving to the next step. More checks can be executed here to check whether the incremental export completed successfully or not.
Step 15
State information is written to the Parameter Store to record the outcome of the operation.
Step 16
A notification is sent to the end user through an Amazon SNS topic.
Step 17
The workflow execution concludes, terminating in either a successful or failed state
Step 18
AWS Key Management Service (AWS KMS), AWS X-Ray, and Amazon CloudWatch Logs are used to provide encryption keys, traceability, and logging capabilities, respectively.
Step 1
The Amazon EventBridge Scheduler is configured to execute at a frequency exceeding the required export window with the capability to catch up on any missed exports if necessary.
Step 2
The AWS Step Functions workflow is initialized by retrieving the current state information from the AWS Systems Manager Parameter Store. This allows for the execution of state-specific actions, such as handling a paused workflow scenario.
Step 3
Perform any necessary sanity checks, such as verifying the existence of the table and confirming that point-in-time recovery (PITR) is enabled.
Step 4
The state information is evaluated to determine if a full export is required for the capture of the complete dataset from the table as the initial step.
Step 5
To initiate the export of the table, the workflow invokes the Amazon DynamoDB API.
Step 6
The export operation starts writing the data, along with the associated manifest and summary, to the specified Amazon Simple Storage Service (Amazon S3) bucket and prefix.
Step 7
The workflow pauses execution and waits for the full export operation to complete before proceeding to the next step. Additional validation checks can be performed at this stage to confirm whether the full export was successfully executed.
Step 8
State information is written to the Parameter Store to record the outcome.
Step 9
A notification is sent to the end user through an Amazon Simple Notification Service (Amazon SNS) topic.
Step 10
The workflow execution concludes, terminating in either a successful or failed state.
Step 11
The incremental export operation first verifies that PITR has remained continuously enabled since the completion of the full export. If PITR is disabled at any point, the PITR history is wiped, resulting in a gap in the data.
Step 12
The incremental export operation requires two time values. The start time value is passed as input to an AWS Lambda function, along with the desired export window. The Lambda function then returns the end time value, as well as any other necessary information.
Step 13
The workflow invokes the DynamoDB API to export the table with extra parameters as required by the incremental export API. This is then followed by a similar flow as before: the exported data is written to the Amazon S3 bucket.
Step 14
The workflow waits for the incremental export to finish before moving to the next step. More checks can be executed here to check whether the incremental export completed successfully or not.
Step 15
State information is written to the Parameter Store to record the outcome of the operation.
Step 16
A notification is sent to the end user through an Amazon SNS topic.
Step 17
The workflow execution concludes, terminating in either a successful or failed state
Step 18
AWS Key Management Service (AWS KMS), AWS X-Ray, and Amazon CloudWatch Logs are used to provide encryption keys, traceability, and logging capabilities, respectively.
Get Started
Deploy this Guidance
Well-Architected Pillars
The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.
The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.
-
Operational ExcellenceRead the Operational Excellence whitepaper
The services used in this Guidance provide capabilities for infrastructure as code, workflow orchestration, serverless computing, and observability. Specifically, the AWS Cloud Development Kit (AWS CDK) allows the Guidance to be deployed using infrastructure as code, making it easy to manage and update. Step Functions provides a visual workflow with capabilities for orchestration, debugging, retries, and monitoring. Lambda enables complex functionality, such as time manipulation. All logs are centralized in CloudWatch, and X-Ray provides end-to-end traceability. Together, these services make this Guidance easy to deploy, manage, and troubleshoot.
-
SecurityRead the Security whitepaper
This Guidance includes services with capabilities for access control, encryption, and configuration management. For example, AWS Identity and Access Management (IAM) allows granular control over AWS services to help ensure least privilege access, and the IAM roles used by Step Functions are scoped to only the necessary resources and services. The Amazon S3 bucket and Amazon SNS topic enforce SSL connectivity. In addition, encryption at rest is enabled using AWS KMS, while TLS helps ensure encrypted communication between services. This Guidance also uses the Systems Manager Parameter Store to store state information, further reducing the need for human interaction.
-
ReliabilityRead the Reliability whitepaper
Step Functions incorporates built-in mechanisms to handle exceptions and automatically retry when necessary while Systems Manager manages the required state information. These services allow the Guidance to automatically recover from known failure modes through the retry capabilities inherent to Step Functions, as well as the state information stored in the Systems Manager Parameter Store. Furthermore, as the workflow is entirely serverless, there is no need to manually manage capacity.
-
Performance EfficiencyRead the Performance Efficiency whitepaper
This Guidance uses advanced serverless technologies so that the most suitable service is used for each specific task. For example, it uses the Systems Manager Parameter Store for managing state information and Lambda for handling complex time-related operations.
-
Cost OptimizationRead the Cost Optimization whitepaper
The various services used within this Guidance, including Step Functions, can be fine-tuned through the adjustment of multiple parameters. This flexibility enables users to control the frequency at which the incremental exports are executed. For instance, implementing longer export windows would result in fewer runs, albeit potentially larger export files. Additionally, users can use the lifecycle management features of Amazon S3 to transition older data objects to more cost-effective storage classes, further optimizing the overall expenditure.
-
SustainabilityRead the Sustainability whitepaper
In a traditional, server-based infrastructure, organizations often need to provision computing resources to handle peak workloads, leading to significant over-provisioning and underutilization of hardware during off-peak periods. This inefficient use of resources can have a detrimental impact on the environment. By adopting a serverless approach, this Guidance helps ensure that computing resources are only used when actively required, automatically scaling the underlying infrastructure based on real-time demands. This dynamic scaling optimizes resource utilization and reduces the overall carbon footprint and energy consumption associated with the infrastructure.
Related Content
[Title]
Disclaimer
The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.
References to third-party services or organizations in this Guidance do not imply an endorsement, sponsorship, or affiliation between Amazon or AWS and the third party. Guidance from AWS is a technical starting point, and you can customize your integration with third-party services when you deploy the architecture.