New – Convert Your Single-Region Amazon DynamoDB Tables to Global Tables

[This post has been updated on March 17th 2020. You can now update update existing global tables to the new replication model]

Hundreds of thousands of AWS customers are using Amazon DynamoDB. In 2017, we launched DynamoDB global tables, a fully managed solution to deploy multi-region, multi-master DynamoDB tables without having to build and maintain your own replication solution. When you create a global table, you specify the AWS Regions where you want the table to be available. DynamoDB performs all of the necessary tasks to create identical tables in these regions and propagate ongoing data changes to all of them.

AWS customers are using DynamoDB global tables for two main reasons: to provide a low-latency experience to their clients and to facilitate their backup or disaster recovery process. Latency is the time it takes for a piece of information to travel through the network and back. Lower latency apps have higher customer engagement and generate more revenue. Deploying your backend to multiple regions close to your customers allows you to reduce the latency in your app. Having a full copy of your data in another region makes it easy to switch traffic to that other region in case you would break your regional setup or in case of an exceedingly rare regional failure. As our CTO, Dr. Werner Vogels wrote: “failures are a given, and everything will eventually fail over time.”

Starting today, you can convert your existing DynamoDB tables to global tables with a few clicks in the AWS Management Console, or using the AWS Command Line Interface (AWS CLI), or the Amazon DynamoDB API. Previously, only empty tables could be converted to global tables. You had to guess your regional usage of a table at the time you created it. Now you can go global, or you can extend existing global tables to additional regions at any time.

Your applications can continue to use the table while we set up the replication. When you add a region to your table, DynamoDB begins populating the new replica using a snapshot of your existing table. Your applications can continue writing to your existing region while DynamoDB builds the new replica, and all in-flight updates will be eventually replicated to your new replica.

To create a DynamoDB global table using the AWS Command Line Interface (AWS CLI), I first create a local table in the US West (Oregon) Region (us-west-2):

aws dynamodb create-table --region us-west-2 \
                          --table-name demo-global-table \
                          --key-schema AttributeName=id,KeyType=HASH \
                          --attribute-definitions AttributeName=id,AttributeType=S \
                          --billing-mode PAY_PER_REQUEST

The command returns:

{
    "TableDescription": {
        "AttributeDefinitions": [
            {
                "AttributeName": "id",
                "AttributeType": "S"
            }
        ],
        "TableName": "demo-global-table",
        "KeySchema": [
            {
                "AttributeName": "id",
                "KeyType": "HASH"
            }
        ],
        "TableStatus": "CREATING",
        "CreationDateTime": 1570278914.419,
        "ProvisionedThroughput": {
            "NumberOfDecreasesToday": 0,
            "ReadCapacityUnits": 0,
            "WriteCapacityUnits": 0
        },
        "TableSizeBytes": 0,
        "ItemCount": 0,
        "TableArn": "arn:aws:dynamodb:us-west-2:400000000003:table/demo-global-table",
        "TableId": "0a04bd34-bbff-42dd-ae18-78d05ce641fd",
        "BillingModeSummary": {
            "BillingMode": "PAY_PER_REQUEST"
        }
    }
}

Once the table is created, I insert some items:

aws dynamodb batch-write-item --region us-west-2 --request-items file://./batch-write-items.json

(The json file is available as a gist)

Then, I update the table to add an additional region, the US East (N. Virginia) Region (us-east-1):

aws dynamodb update-table --region us-west-2 --table-name --replica-updates
  '
    [
      {
        "Create": {
          "RegionName":  "us-east-1"
        }
      }
    ]
  '

The command returns a long JSON, the attributes you need to pay attention to are:

{
...
        "TableStatus": "UPDATING",
        "TableSizeBytes": 124,
        "ItemCount": 3,
        "StreamSpecification": {
            "StreamEnabled": true,
            "StreamViewType": "NEW_AND_OLD_IMAGES"
        },
        "LatestStreamLabel": "2019-10-22T19:33:37.819",
        "LatestStreamArn": "arn:aws:dynamodb:us-west-2:400000000003:table/demo-global-table/stream/2019-10-22T19:33:37.819"
    }
...
}

I can make the same update in the AWS Management Console I select the table to update and click Global Tables.

Enabling streaming is a requirement for global tables. I first click on Enable stream, then Add region:

I choose the region I want to replicate to, for this example EU West (Ireland) and click Create replica.

DynamoDB asynchronously replicates the table to the new region. I monitor the progress of the replication in the AWS Management Console. Table’s status will eventually change from Creating to Active. I also can check the status by calling the DescribeTable API and verify TableStatus = Active.

After a while, I can query the table in the new region:

aws dynamodb get-item --region eu-east-1 --table-name demo-global-table --key '{"id" : {"S" : "0123456789"}}'

{
    "Item": {
        "firstname": {
            "S": "Jeff"
        },
        "id": {
            "S": "0123456789"
        },
        "lastname": {
            "S": "Barr"
        }
    }
}

Starting today, you can update existing local tables to global tables. You can also update your existing global tables to take advantage of this new capability with a few click in the console. The update itself will take a few minutes at most. Your table will be available for your applications during the update process.

Other Improvements
We are also simplifying the internal mechanism used for data synchronization. Previously, DynamoDB global tables leveraged DynamoDB Streams and added three attributes (aws:rep:*) in your schema to keep your data in sync. DynamoDB now manages replication natively. It does not expose synchronization attributes to your data and it does not consume additional write capacity:

• Only one write operation occurs in each region of your global table, which reduces the consumed replicated write capacity that is required on your table.
• Because of that, a second DynamoDB Streams record is no longer published.
• The three aws:rep:* attributes that were previously populated are no longer inserted in the item record.

These changes have two consequences for your apps. First, they reduce your DynamoDB costs when using global tables, because no extra write capacity is required to managed the synchronization. Secondly, when your application is relying on the three technical items (aws:rep:*), your application requires a slight code change. In particular, DynamoDB Mapper must not require the aws:rep:* attributes to exist in the item record.

With this change, we are also updating the UpdateTable API. Any operation that modifies global secondary indexes (GSIs), billing mode, server-side encryption, or write capacity units on a global table are applied to all other replicas asynchronously.

Availability
The improved Amazon DynamoDB global tables is available today in the 13 regions where Amazon DynamoDB global tables is available, and more regions are planned in the future. As of today, the list of AWS Regions is us-east-1 (Northern Virginia), us-west-2 (Oregon), us-east-2 (Ohio), us-west-1 (Northern California), ap-northeast-2 (Seoul), ap-southeast-1 (Singapore), ap-southeast-2 (Sydney), ap-northeast-1 (Tokyo), eu-central-1 (Frankfurt), eu-west-1 (Ireland), eu-west-2 (London), GovCloud (US-East), and GovCloud (US-West).

There is no change in pricing. You pay only for the resources you use in the additional regions and the data transfer between regions.

This update addresses the most common feedback that we have heard from you and will serve as the platform on which we will build additional features in the future. Continue to tell us how you are using global tables and what is important for your apps.