Auto Scale DynamoDB With Dynamic DynamoDB

Update (2017): DynamoDB now supports auto scaling out of the box! Check out my new blog post or read the new documentation.

Amazon DynamoDB is a fully-managed NoSQL database. When you create a DynamoDB table, you provision the desired amount of request capacity, taking in to account the amount of read and write traffic and the average size of each item. You can change this capacity (up or down) as you gain a better understanding of your application’s requirements.

Many independent developers have built and published tools to facilitate the use of DynamoDB in a wide variety of environments (see my DynamoDB Libraries, Mappers, and Mock Implementations post for a fairly complete list).

Today I would like to tell you about Dynamic DynamoDB, an open source tool built by independent developer Sebastian Dahlgren. This flexible and highly configurable tool manages the process of scaling the provisioned throughput for your DynamoDB tables.

New CloudFormation Template
Sebastian has created a CloudFormation template that you can run to start using Dynamic DynamoDB with just a couple of clicks. You can configure Dynamic DynamoDB to scale your tables up and down automatically, and you can restrict scaling activities to certain time slots. You can scale read and write throughput capacity independently using upper and lower thresholds, and you can set minimums and maximums for each value. Finally, Dynamic DynamoDB supports a circuit-breaker. Before it performs any scaling activities, it can verify that your application is up and running. This final check will avoid spurious scale-down activities if your application is experiencing other problems.

This template lives at https://raw.github.com/sebdah/dynamic-dynamodb/master/cloudformation-templates/dynamic-dynamodb.json .

The template launches a t1.micro EC2 instance with the Dynamic DynamoDB package pre-installed. The instance is managed by an Auto Scaling group, and will be replaced if it fails. Because Dynamic DynamoDB makes calls to the DynamoDB API on your behalf, the template prompts you for your AWS credentials:

As always, use of an IAM user is advised. You’ll need full access to DynamoDB APIs and resources, and read-only access to CloudWatch APIs and resources.

The template also requests an EC2 key pair for SSH access and the name of an S3 bucket for storage of the Dynamic DynamoDB configuration.

Configuring Dynamic DynamoDB
After the template has done its thing, SSH to the newly created EC2 instance, log in as ec2-user, and edit the configuration file in /etc/dynamic-dynamodb/dynamic-dynamodb.conf (you have lots of configuration options):

Dynamic DynamoDB runs as a service (dynamic-dynamodb) and can be stopped, started, and restarted if necessary. The configuration file is automatically backed up to S3 on each start or restart. The service will start automatically if it finds a valid configuration file in S3. You will have to start it yourself (sudo service dynamic-dynamodb start) after you first edit the configuration file.

Dynamic DynamoDB can be started in “dry run” mode. In this mode it will check the tables and make scaling decisions based on the configuration information, but it will not actually make any changes to the provisioned throughput.

Dynamic DynamoDB in Action
AWS customer tadaa (case study) uses DynamoDB to power their iPhone photo app. Here’s what they had to say about it:

when we moved from our self-administered database to Amazon DynamoDB, we eliminated the burdens of scaling in favor of predictable response times and infinite table sizes. That was a very easy business decision to make.

They use Dynamic DynamoDB to automatically adjust their DynamoDB capacity. Earlier this week they tweeted the following picture to show it in action:

The blue line represents the average amount of write capacity consumed. The red line represents the amount of write capacity that is provisioned. As you can see, Dynamic DynamoDB is able to alter the amount of provisioned capacity in response to changing conditions. After the developers at tadaa deployed an updated version of their code, the amount of write capacity consumed by their application declined precipitously.  Dynamic DynamoDB detected this change, and reduced the amount of provisioned write capacity accordingly.

Sebastian does not charge for Dynamic DynamoDB but you will pay the usual charges for the t1.micro instance. You can also install Dynamic DynamoDB on an existing instance if you’d like.

Read Sebastian’s CloudFormation Template documentation to get started with Dynamic DynamoDB.

— Jeff;