How to Build Dynamic Dashboards Using Lambda and DynamoDB Streams: Part 1

Imagine driving your car without the benefit of a moving speedometer, or cooking a casserole with an oven thermometer that’s always stuck on one temperature, or setting your alarm clock only to find the numbers never change. You expect real-time feedback from things you use every day, why should the dashboards on our applications be any different?

Today, using some new tools from Amazon Web Services, I’ll show you how to make boring, static dashboards reflect changes in real time.

DynamoDB Streams and Lambda

You’ve probably heard of Amazon DynamoDB, a fast NoSQL database service. Coming soon, you’ll be able to take advantage of a new feature called DynamoDB Streams. DynamoDB Streams provide a time-ordered sequence of item level changes in any DynamoDB table.

AWS Lambda is a compute service that runs your code in response to events and automatically manages compute resources for you. The beauty of Lambda is that you, as a developer, don’t have to spin up any compute instances to handle this code — Lambda runs automatically at scale, handling the compute capacity for you so that thousands of functions can run in parallel, and performance remains consistently high regardless of the frequency of events.

Because DynamoDB Streams and Lambda are currently only in preview mode, part one of this post will give you an overview of the app in action and highlight just a few simple code snippets along the way. When DynamoDB Streams and Lambda are publicly available, I’ll release part two of this post, in which I’ll fill in the details of this application and walk you through how to build it for yourself.

In the meantime, I highly suggest you get on the list for access to both Streams and Lambda. If you happen to have Streams and Lambda access early, leave a comment below if you want help implementing this yourself before part two comes out.

• For DynamoDB streams, you can request access at
  https://aws.amazon.com/dynamodb
• For Lambda, you can request access at
  http://aws.amazon.com/lambda

The Dashboard

For this demonstration, I’ve built a simple application based on a fictitious scenario where people can text votes for their favorite color to a phone number provided by our friends at Twilio. An Amazon EC2 sever running Node.js receives these responses and places them in a DynamoDB table. When Lambda notices a change in the table, it will update JSON values in a data file that our dashboard polls to dynamically update a graph.

Although you’re simply voting for a color and updating a basic pie chart, you can easily take the techniques discussed here and build out your own custom dashboards suited to your specific needs.

To see a finished version in action, visit http://voteapp.s3-website-us-east-1.amazonaws.com/. You can watch the dashboard respond to your vote by sending “RED”, “BLUE”, or “GREEN” to 1–512–337–1954.

Twilio

I started by creating a free account at Twilio, and chose a number from a pool they provided. You can do the same if you want — and in fact, if you build this out when part two is released, you’ll need to have your account set up.

Our EC2 Server for Twilio Responses

I needed an EC2 server to parse SMS messages from the users. Twilio provides the conduit from which SMS messages will flow from the users to the server.

Before launching the instance, I created a role for it. A role allows the instance to access AWS resources — like DynamoDB — without having to explicitly pass in access keys or do authentication from the codebase. You’ll find roles to be handy; they’ll keep you from having to store authentication details in code, which in turn will keep your applications more secure.

The role for this demonstration uses a policy that gives the instance “put_item” access to the specific DynamoDB table.

Once I created the role, I launched an instance with that role applied. I used a t2.micro — which keeps me in the AWS free tier. But in a production environment, or in an environment where you anticipate a lot of traffic, you might want to use something more robust, and even follow AWS best practices by placing your application behind a load balancer to distribute your traffic to multiple EC2 instances.

As you’ll see in a moment, the dashboard itself is hosted on Amazon Simple Storage Service (Amazon S3), which ensures high resiliency for handling large amounts of web traffic.

I configured Twilio to send traffic to the server’s public IP address. I installed Node.js and Express on the server to handle the callbacks. Using the AWS SDK for JavaScript in Node.js, I was able to write the appropriate data to the DynamoDB table, with just a few lines of code.

In this case, the app saves the phone number of the voter, the timestamp of the entry, and the message — which is a vote for RED, GREEN, or BLUE.

Then, still using Twilio, the app sends a confirmation message back to the user, thanking him or her for voting.

When the app adds a new vote to the table, Lambda notices the change, and using Node.js inside a Lambda event, tallies up the votes for RED, GREEN, and BLUE, independently. Upon final tally, it writes the results to a storage bucket on S3.

Since Lambda runs independently of any servers you launch, you don’t have to worry about provisioning extra horsepower to make this happen. Your Lambda function runs as Node.js code and can take full advantage of built-in libraries like the AWS SDK.

Lambda doesn’t care how the data gets into DynamoDB, only that the table has been updated. For example, I could allow votes from additional sources, like a web form, an email campaign, or a mobile app. I eliminate the risks and headaches of writing redundant code to tally votes across these various platforms and input sources by using Lambda to monitor the DynamoDB table. You can see a snippet of the Node.js code that tallies the vote and writes to S3 below:

The Dynamic Dashboard

Amazon S3 provides an interesting feature that lets you serve static websites from an S3 bucket. Like Lambda, S3 provides additional functionality without your needing to spin up a web server or provision any new services.

S3 will give you a DNS endpoint — in this case, voteapp.s3-website-us-east-1.amazonaws.com, but you could also configure a custom domain name using Amazons Route53 that points to your bucket.

By using S3, the app can serve huge amounts of traffic with extremely low latency without load balancing or worrying about overloading an EC2 server.

Although the site is technically static, it can still contain server-side scripts. By using some JavaScript and the Chart.js charting library, I am able to build a dynamic dashboard.

The app polls the data file that Lambda wrote for us, looking for updates, and then display those updates to the user in the form of an updated chart.

Coming Soon: Part Two

When DynamoDB Streams and Lambda are publicly available, I’ll post part two of this series, in which I’ll dive into the code and show you how to build out the entire stack for yourself. I’ll use Node.js, but if you’re a Python fan, you’ll also be happy to know you can use Python to accomplish the same thing.

I hope you enjoyed this look at the power of Lambda and DynamoDB Streams. Please leave a comment below if you need help getting access to these exciting new services.