AWS Case Study: TweetDeck

With TweetDeck, users can keep an eye on many searches simultaneously, organize groups of people they follow into columns, filter updates out by keywords, schedule their updates to send at a later time to target different time zones, and much more. TweetDeck products are available over the desktop (Windows, Linux, Mac), the Web (with our HTML5 Chrome client), iPhone, iPad and Android platforms.

TweetDeck is a leading Twitter client outside of twitter.com itself, with over 10% of Twitter's active users, who are believed to be some of the most influential and important users within the Twitter user base.

I have managed infrastructure and data services at TweetDeck for the last two years and have architected and deployed TweetDeck's Amazon Web Services (AWS) presence.

How have you incorporated Amazon Web Services as part of your architecture? What services are you using and how?

• TweetDeck has a backend REST API and website deployed over Amazon Elastic Compute Cloud (Amazon EC2), which uses Amazon Elastic Load Balancing (Amazon ELB) and Autoscale to manage each cluster's size and distributed requests.
• We deploy a mix of high-memory, small, and micro instances – each tuned for hosting large in-memory databases, generic application serving, or I/O bound queue-based jobs.
• Amazon SimpleDB is used as a storage layer for user accounts and certain synced data.
• We use an Amazon Relational Database Service (Amazon RDS) instance in conjunction with our website.
• We use Amazon Elastic Block Store (Amazon EBS) for backup and high availability of our databases, with frequent cron-ed snapshots to ensure data integrity.
• Amazon Simple Storage Service (Amazon S3) is used for storage of our tracked analytics data, which makes it easy to perform large-scale analysis via Amazon Elastic MapReduce.

What programming languages and/or tools did you use to build this solution? Did you use any AWS SDKs?

• Our infrastructure and REST API are implemented in Python, primarily using the Twisted event-based networking framework (http://twistedmatrix.com ).
• Our website is built using Django (http://djangoproject.com ).
• We host a large in-memory key-value database cluster using Redis (http://redis.io ).
• Our distributed task-queue is built with Celery (http://celeryproject.org ) and uses RabbitMQ (http://rabbitmq.com ).
• We used Memcached for caching website and certain API content (http://memcached.org ).
• Our primary http server endpoint for the Website and tracking systems are built using Nginx (http://nginx.net).

Why did you decide to use AWS?
We are a small company and couldn't spare the time to configure physical machines directly. Using a mature and full-featured solution like AWS allows us to deploy complex production services, and scale upwards or downwards with ease.

Can you share some metrics on your usage of AWS today?
We have about 50 Amazon EC2 instances running, and our API receives about 700 requests per second.

Have you learned any valuable lessons during this development process that you'd like to pass on to other developers?
Constantly evaluate your metrics and how your costs break down and correspond to your metrics. Usage profiles for your services change, so certain costs that may have remained quite low with one traffic profile could become significant as certain aspects of your system grow relative to others.

Do you have any future plans to incorporate other AWS solutions?
We're taking a look at Amazon Route53 to manage our Domain Name System (DNS).

Is there anything else you would like to add?
Everyone must agree that being able to launch a 100 GB Redis cluster in a few minutes is pretty cool.

To learn more, visit http://www.tweetdeck.com/ .

Added July 18, 2011