Trendsmap.com reveals Twitter trends in real-time. The website allows visitors to view the most often used words in current Twitter posts from a global perspective or drilled-down to a local level. Trendsmap.com is a product of the web workshop Stateless Systems, which creates innovative web-based projects that provide a wide variety of content and web services, from coupon-code sharing to a content management system. Stateless Systems is headquartered in Melbourne, Australia and employs 12 software developers and web designers.
When Stateless Systems decided to build Trendsmap.com, the team needed the flexibility to spin up multiple instances quickly and the ease of use that would enable quick deployment. "We deal with large volumes of data," says Stateless Systems lead developer John Barratt. "We need the flexibility that you can only find in the cloud."
Stateless Systems developed Trendsmap.com entirely upon Amazon Web Service (AWS), with Amazon Elastic Compute Cloud (Amazon EC2) as its primary service. Amazon EC2 functions as Trendsmap.com’s processing and web serving platforms. The Stateless Systems developers built the core of the website’s foundation in Ruby and additional components with MySQL, Memcached, Nginx, and Sphinx.
To handle the large amount of data processed by Trendsmap.com, the developers created a custom parallel data processing pipeline to control workloads among its Amazon EC2 instances. The application incorporates four database servers and more than 20 Amazon EC2 instances in total for processing.
Trendsmap.com relies on several different features within Amazon EC2, including Amazon Elastic Block Store (Amazon EBS). The Stateless Systems developers use this flexible block level storage to house the site’s growing dataset, which increases in size by many gigabytes every day. When Stateless Systems needs access to short-term processing of large volumes of data for Trendsmap.com, it calls up Amazon EC2 Spot Instances. Amazon EC2 Spot Instances allow Stateless Systems to save money by bidding on unused instances to temporarily augment the site’s computing resources.
Stateless Systems also uses Amazon EC2 Elastic IP addresses. John Barratt explains the convenience of this Amazon EC2 feature: “The Elastic IPs have allowed us to transfer public services from one host to another very easily and without any downtime. And, they also allow for no-fuss movement of servers that have been whitelisted by external services such as Twitter, without having to get them to update their configuration.”
In addition, Trendsmap.com uses the Elastic Load Balancing feature to distribute traffic between its multiple Amazon EC2 instances. Stateless Systems developers recently credited Elastic Load Balancing with helping them to avert a potential service outage involving Trendsmap.com.
The benefits of Amazon EC2 and Amazon EBS convinced Stateless Systems to use AWS. “Between the flexibility of AWS, and the ease with which we could deploy new ones, or change existing instances was what sold us on AWS," Barratt says. "Because we deal with large volumes of data, the flexibility that Amazon EBS volumes provide has been particularly useful and is one of the single biggest factors that steered us toward AWS.”
In 2009, Trendsmap.com began operations by storing and processing approximately 4 million tweets per day. Today, the site stores and processes more than 40 million tweets per day. Stateless Systems credits the scalability of AWS with helping it to achieve Trendsmap.com’s impressive expansion. Barratt says, “AWS has given us the ability to easily scale out our compute and storage requirements as both the functionality and the data volume has grown over the last two years.”
To learn more about how AWS can help you process and store big data, visit our Big Data page: http://aws.amazon.com/big-data/.