AWS DevOps Blog

Using DynamoDB and SNS with Elastic Beanstalk in any Supported AWS Region

by Evan Brown | on | Permalink | Comments |  Share

It’s common for applications running on Elastic Beanstalk to use other AWS services for things like data storage, message processing, etc. In today’s blog post, let’s talk about how we would build an application that runs in Elastic Beanstalk, stores data in Amazon DynamoDB, sends notifications to the Amazon Simple Notification Service (SNS), and can be easily deployed to any AWS Region where those services are supported.

About the App

Last month we published a short video series and source code about a sample application for startups. The application simply allows your customers to sign up with their name and e-mail address to be notified when your product is ready to purchase.

Here’s an overview of the architecture:

And here’s what the app looks like:

Pretty straightforward. The application writes data to the DynamoDB table whenever a user signs up, and also publishes a JSON message to the SNS topic, which in turn sends the message to an SQS queue for asynchronous processing (specifically, there’s a worker that reads messages from the queue and sends confirmation e-mails to the customer). While we could manually click around in the AWS Management Console to create the DynbamoDB Table, SNS Topic, and SQS Queue that our application depends on, there’s a better way. Enter configuration files.

Configuration Files

Configuration files allow you to customize pretty much every aspect of your Elastic Beanstalk application. You can install packages on EC2 instances (plenty of customers use config files to install things like the New Relic Server Monitoring Agent), tweak load balancer or auto scaling settings, and even create other AWS resources that your app depends on. We’re really interested in that last item for this application. Here are the 3 steps we’ll follow to create the dependencies for our app:

  1. Add a .ebextensions folder to the top level of our app
  2. Create and populate a .config file
  3. Deploy the app

The .ebextensions Folder

This step is really straightforward: create a new folder at the top level of your app. Name it .ebextensions:

Create and Populate a .config File

There’s a little more to this step. Config files have a .config extension, and their format is YAML. You can have multiple .config files in your .ebextensions directory; Elastic Beanstalk will process them in alphabetical order by file name. It’s common to see complex config files broken down into chunks with names like 1_do_this_first.config, 2_do_this_second.config, etc. Within a config file, you declare the things you want to create or customize. The complete syntax is well documented.

Inside the .ebextensions folder, have a single file – setup.config – that declares 3 major things:

  1. Resource Dependencies (i.e., DynamoDB, SNS, and SQS):

    Resources:
       StartupSignupsTable:
         Type: AWS::DynamoDB::Table
         Properties:
           KeySchema: 
             HashKeyElement:
               AttributeName: "email"
               AttributeType: "S"
           ProvisionedThroughput:
             ReadCapacityUnits: 1
             WriteCapacityUnits: 1
       NewSignupQueue: 
         Type: "AWS::SQS::Queue"
       NewSignupTopic:
         Type: AWS::SNS::Topic
         Properties: 
           Subscription:
             - Endpoint:
                 Fn::GetOptionSetting:
                   OptionName: NewSignupEmail
                   DefaultValue: "nobody@amazon.com"
               Protocol: email
               Protocol: "sqs"
             - Endpoint:  
                 Fn::GetAtt:  ["NewSignupQueue", "Arn"]
    
  2. A file at /var/app/app.config on each instance that contains the names of the resources provisioned in Step 1. Notice how we use the Ref syntax to retreive the IDs of the Resources, and also to create a config value that indicates the AWS Region (e.g., us-west-1) where the application is deployed to (more on that later…):

    files:
       "/var/app/app.config":
         mode: "000444"
         owner: "apache"
         group: "apache"
         content: |
           AWS_REGION = '`{ "Ref" : "AWS::Region"}`'
           STARTUP_SIGNUP_TABLE = '`{ "Ref" : "StartupSignupsTable"}`'
           NEW_SIGNUP_TOPIC = '`{ "Ref" : "NewSignupTopic"}`'
    
  3. An environment variable that will be set on every instance that tells our code (application.py) to use /var/app/app.config as its configuration file:

    option_settings:
         ...
       "aws:elasticbeanstalk:application:environment":
         "APP_CONFIG": "/var/app/app.config"
         ...
    

Deploy the App

With my .ebextensions folder created and populated with setup.config, I’m ready to deploy the app. Elastic Beanstalk will process the config file, create the DynamoDB Table, SNS Topic, and SQS Queue, then create an application configuration file on each instance it provisions to host the app, and set an enviornment variable so the app knows where to get the config from. It’s all completely automated: there’s no fumbling around with manualy provisioning and configuring, and it allows our application to scale out or in seamlessly.

There are a few ways you can choose to deploy the app. You can bundle it up as a ZIP and upload it through the Elastic Beanstalk Management Console (we blogged about how to do that). You could also use the eb CLI to easily deploy and update the app from the command line (there’s a good tutorial in the Elastic Beanstalk documentation).

Regional Portability

Finally, we want our application to be portable across regions – that is, the same code should be able to run in the Virginia, Oregon, California, Brazil, Tokyo, Singapore, Sydney, or Dublin regions without any modifications. This will allow us to deploy the app in multiple regions simultaneously, maybe because we want to be physically closer to our customers, or possibly because our app needs to comply with a local government’s data protection policy. You might also want to move your app between regions as part of a disaster recovery plan.

Whatever your reason, regional portability is straightforward: it really boils down to our app code using the correct region when connecting to the DynamoDB Table or SNS Topic. Remember in our setup.config earlier where we declared the /var/app/app.config file? That file will be used as the configuration for our application code, and we set an AWS_REGION key in there:

files:
  "/var/app/app.config":
    mode: "000444"
    owner: "apache"
    group: "apache"
    content: |
      AWS_REGION = '`{ "Ref" : "AWS::Region"}`'
      STARTUP_SIGNUP_TABLE = '`{ "Ref" : "StartupSignupsTable"}`'
      NEW_SIGNUP_TOPIC = '`{ "Ref" : "NewSignupTopic"}`'

Because we used ‘{ "Ref" : "AWS::Region"}‘, the value of the AWS_REGION key will be set to the region where the application is deployed and where the DynamoDB Table, SNS Topic, and SQS Queue were created. If you deployed the app to Virginia, /var/app/app.config would say AWS_REGION = us-east-1; you could also deploy to Sydney and the file there would read AWS_REGION = ap-southeast-2.

Now that we have the deployment region accessible in a configuration file, all that remains is having our app use it. This sample app is written in Python so we’re using Boto (the AWS SDK for Python) to connect to our DynamoDB Table and SNS Topic. Boto exposes a connect_to_region method for each service, and we supply the region name retrieved from the config file in application.py:

# Connect to DynamoDB and get ref to Table
ddb_conn = dynamodb2.connect_to_region(application.config['AWS_REGION'])
...

# Connect to SNS
sns_conn = sns.connect_to_region(application.config['AWS_REGION'])