AWS Case Study: Live Talkback

I'm the CTO and co-founder of Live Talkback, responsible for building the product and designing the architecture to support broadcast-scale audiences.

Recently, we created the audience buzzer for Britain's Got Talent (http://talent.itv.com/2011/buzzoff). The buzzer allows the audience to "buzz off" acts they don't like, and instantly see how many other people are buzzing. We’ve already had millions of buzzes from people wanting to buzz off acts ranging from a man with buckets on his head to a child comic, via singers, dancers and magicians.

We believe that the future of TV is social and interactive. Harnessing the cloud to power these services is pushing the boundaries of what is possible, and ultimately, making TV better.

How have you incorporated Amazon Web Services as part of your architecture? What services are you using and how?
Almost the entire Live Talkback/Tellybug infrastructure is hosted on Amazon Web Services (AWS). The stack is a fairly typical LAMP architecture, using the Django Web framework, Memcached, MySQL and Ubuntu, with HAProxy providing the load balancing. We use Cassandra for handling the high peak write loads during broadcasts..

We use the following solutions from AWS:

• Amazon Relational Database Service (Amazon RDS) for our MySQL servers
• Amazon Elastic Compute Cloud (Amazon EC2) instances for all the other machines
• Amazon CloudFront for CDN distribution
• Amazon CloudWatch metrics (along with our own monitoring solution)

What programming languages and/or tools did you use to build this solution?
Most of the stack is written in Python. Flash is used for the buzzer widget that is embedded into Facebook and the ITV.com Website. We use various bits of open-source code to talk to AWS, including S3FS, the Amazon EC2 command line tools, and the Python boto library.

Why did you decide to use AWS?
Hosting in the cloud was essential for the success of our business. It has enabled us to focus on building the product rather than administering servers.

Social TV interactions have huge peak-to-trough demand swings. This means that the load on the buzzer servers goes up by about 20x between the hour before the show and the hour of the show, and we see buzz rates quadrupling in just 5 minutes after the show starts.

Handling such demand without the scalability and elasticity that AWS provides would be totally uneconomic. The infrastructure is all scaled out and back as needed to cope with the load. At our peak, we designed to cope with 50,000 requests per second hitting the proxies, which requires over 150 servers running to handle the load.

AWS has also made it possible for us to test the infrastructure with simulations of hundreds of thousands of simultaneous users, by using the cloud to create lots of test machines directing traffic at the servers. Without testing at full load, it's impossible to be ready when the show goes live.

How has AWS helped your business?
AWS has given Live Talkback the flexibility and scaling capability needed to deliver social TV interaction for our partners. With AWS, Live Talkback has been able to build a website that can handle 130 billion requests a month—all on a credit card budget.

Can you provide any quantifiable metrics related to your use of AWS?

• Over 1.5 million buzzes from the audience
• Infrastructure tested to 30,000 requests/s, using ~150 Amazon EC2 nodes

Have you learned any valuable lessons during this development process that you’d like to pass on to other developers?
The main lessons learned were:

Scaling needs testing. Until you've tested under real load levels, you cannot assume things will scale up. We found several places where bits of the system became bottlenecks as we scaled up.

Use automated provisioning tools like Chef, Puppet, or CloudFormation (we use Chef). Being able to consistently configure your servers in the cloud is essential, especially when load increases and you suddenly need lots of them.

With the flexibility and cost structure enabled by AWS, there are suddenly lots of opportunities to do new things. We're doing things with broadcast scale audiences that would have been impossible a few years ago.

Do you have any future plans to incorporate other AWS solutions?

Yes, we're looking at Amazon Domain Name Service (Amazon DNS) and Amazon Elastic MapReduce.

To learn more, visit http://www.livetalkback.com/ .

Added May 31, 2011