AWS Case Study: Trendsmap.com

Stateless Systems developed Trendsmap.com entirely upon Amazon Web Service (AWS), with Amazon Elastic Compute Cloud (Amazon EC2) as its primary service. Amazon EC2 functions as Trendsmap.com’s processing and Web serving platforms. The Stateless Systems developers built the core of the Website’s foundation in Ruby and additional components with MySQL, Memcached, Nginx, and Sphinx. To handle the large amount of data processed by Trendsmap.com, the developers created a custom parallel data processing pipeline to control workloads among its Amazon EC2 instances. The application incorporates four database servers and more than 20 Amazon EC2 instances in total for processing.

Trendsmap.com relies on several different features within Amazon EC2, including Amazon Elastic Block Store (Amazon EBS). The Stateless Systems developers use this flexible block level storage to house the Website’s growing dataset, which increases in size by many gigabytes every day. When Stateless Systems needs access to short term processing of large volumes of data for Trendsmap.com, it calls up Amazon EC2 Spot Instances. Amazon EC2 Spot Instances allow Stateless Systems to save money by bidding on unused instances to temporarily augment the Website’s computing resources.

Stateless Systems also utilizes Amazon EC2 Elastic IP addresses. Lead developer John Barratt explains the convenience of this Amazon EC2 feature: “The Elastic IPs have allowed us to transfer public services from one host to another very easily and without any downtime. And, they also allow for no fuss movement of servers that have been whitelisted by external services such as Twitter, without having to get them to update their configuration.”

In addition, Trendsmap.com uses the Elastic Load Balancing feature to distribute traffic between its multiple Amazon EC2 instances. This feature is particularly popular with the Stateless Systems developers because they recently credited Elastic Load Balancing with helping them to avert a potential service outage involving Trendsmap.com.

When asked why Stateless Systems decided to build Trendsmap.com on AWS, Barratt answers, “The flexibility of AWS, in particular with respect to the flexibility offered from different Amazon EC2 instances, and the ease with which we could deploy new ones, or change existing instances. Because we deal with large volumes of data, the flexibility that Amazon EBS volumes provide has been particularly useful and is one of the single biggest factors that steered us toward AWS.”

In 2009, Trendsmap.com began operations by storing and processing approximately 4 million Tweets per day. Today, the Website stores and processes over 40 million Tweets per day. Stateless Systems credits the scalability of AWS with helping it to achieve Trendsmap.com’s impressive expansion. Barratt says, “AWS has given us the ability to easily scale out our compute and storage requirements as both the functionality and the data volume has grown over the last two years.”

To learn more, visit http://trendsmap.com/.

Added May 18, 2011