Category: Success Stories

Over the summer months, we’d like to share a few stories from startups around the world: what are they working on and how they are using the cloud to get things done. Today, we’re profiling Sportaneous from New York City!

The Story
I first learned about Sportaneous after reading about NYC Big Apps, an application contest launched by the city of New York, BigApps is a competition that is organized by Mayor Bloomberg. Its goal is to reward apps that improve NYC by using public data sets released by the local government.

Sportaneous jumped at the opportunity to enter the contest because their applications already offered users a database of public sports facilities to choose from, many of which are obtained from Park & Recreation data. Sportaneous makes it easy for busy people to play sports or engage in group fitness activities. Through the Sportaneous mobile app and website, a person can quickly view all sports games and fitness activities that have been proposed in her surrounding neighborhoods. The user can choose to join whichever game best fits her schedule, location, and skill level. Alternatively, a Sportaneous user may spontaneously propose a game herself (for example, a beginner soccer game in Central Park three hours from now), which allows all Sportaneous users in the Manhattan area to join the game until the maximum number of players has been reached.

From the over 50 applications that entered the NYC BigApps competition, Sportaneous won two of the main awards: the Popular Vote Grand Prize, based on over 9,500 people voting for their favorite app in the contest and the Second Overall Grand Prize voted on by a panel of very distinguished judges, including Jack Dorsey (Co-founder, Twitter), Naveen Selvadurai (Co-founder, Foursquare) and prominent tech investors in NYC.

Here is a video of Sportaneous in action:

From the CEO
I spoke to Omar Haroun, CEO and Co-Founder at Sportaneous about how they got started and ended up using AWS. He shared a bit about their humble beginnings and how their growth plans continued to include AWS:

We initially bootstrapped the service using a single EC2 instance.  We used an off-the-shelf AMI with a backing EBS volume, so we could fine-tune the machine’s configuration as we started higher traffic numbers.  We wanted a low cost, reliable hosting option which we knew had the ability to scale gracefully (and very quickly) when needed.  EC2 allowed us to get up and running in a matter of hours, without forcing any design compromises which we’d later regret.

As traffic has grown and we’ve begun preparing for a public launch, we’re planning to move our MySQL databases to RDS and to take advantage of some additional ELB features (including SSL termination). RDS was also a no-brainer.  We realize that any data-loss event would be devastating to our momentum, but we don’t have the resources for a full time DBA (or even a database expert).  RDS and its cross-AZ replication takes a huge amount of pressure off of our shoulders.

Behind the Scenes
I asked Omar to tell me a bit about the technology behind Sportaneous. Here’s what he told me:

Our web app is written in Scala, using the very awesome Lift Framework. Our iPhone App is written in Objective-C.  Both web app and iPhone app are thin clients on top of a backend implemented in Java, using the Hibernate persistence framework. Our EC2 boxes (which serve both our web app and our backend) run Jetty behind nginx.

He wrapped up on a very positive note:

Using EC2 with off the shelf AMIs, we went from zero to scalable, performant web app in under two hours.

The AWS Startup Challenge
We’re getting ready to launch this year’s edition of our own annual contest, the AWS Startup Challenge. You can sign up to get notified when we launch it, or you can follow @AWSStartups on Twitter.

— Jeff;

Felix Geisendrfer of Debuggable wrote in to tell me about Transloadit. He told me that they now support real-time video encoding using a number of Amazon EC2 instances.

Whereas prior transcoding solutions waited for the entire video to be uploaded, Transloadit runs the encoding process directly on the incoming data stream, using the popular node.js library to capture each chunk of the file as it is uploaded, piping it directly into the also-popular ffmpeg encoder. Because transcoding is almost always faster than uploading, the video is ready to go shortly after the final block has been uploaded. As Felix noted in his email to me, “Since Ec2 can encode videos much faster than most people can upload them, that essentially cuts the encoding time to 0.”

Felix told me that they implemented Transloading using a number of AWS services including EC2, Elastic Load Balancing, Elastic Block Storage, and Amazon S3. They have found that the c1.medium instance type delivers the best price/performance for their application, and are very happy that their Elastic Load Balancer can deliver data to the instances with minimal delay. They are able to deploy data directly to a customer’s S3 bucket, and are looking in to Multipart Upload and larger objects.

Transloadit also offers image resizing, encoding, thumbnailing, and storage (to S3) services (which they call robots).

Pricing for Transloadit is based on the amount of data transferred and usage is billed monthly. You can even sign up for a free trial and receive $5 of credits toward your usage of Transloadit.

— Jeff;

Many fields in industry and academia are experiencing an exponential growth in data production and throughput, from social graph analysis to video transcoding to high energy physics. Constraints are everywhere when working with very large data sets, and provisioning sufficient storage and compute capacity for these fields is challenging.

This is particularly true for biological sciences after the recent quantum leap in DNA sequencing technology. These advances represented a step change for the field of genomics, which had to learn quickly about how housing and processing terabytes of data through complex, often experimental workflows.

Processing data of this scale for a single user is challenging, but moving to the cloud meant Michigan State University were able to provide real world training to whole groups of new scientists using Amazon’s EC2 and S3 services.

Titus Brown writes about his experiences of running a next-generation sequencing workshop using Amazon’s Web Services in a pair of blog posts:

“Students can choose whatever machine specs they need in order to do their analysis. More memory? Easy. Faster CPU needed? No problem.

All of the data analysis takes place off-site. As long as we can provide the data sets somewhere else (I’ve been using S3, of course) the students don’t need to transfer multi-gigabyte files around.

The students can go home, rent EC2 machines, and do their own analyses — without their labs buying any required infrastructure.”

After the two week event:

“I have little doubt that this course would have been nearly impossible (and either completely ineffective or much more expensive) without it.

In the end, we spent more on beer than on computational power. That says something important to me.”

A great example of using EC2 for ad-hoc, scientific computation and reaping the rewards of a cloud infrastructure for low cost, reproducibility and scale.

~ Matt

We developed our first proof of concept using EC2 and S3 back in 2006.
From the financial point of view, AWS made prototyping in early states and real world scenarios really affordable. From the technical point of view, AWS took care of the “hardware” part for us, thus allowing the developers to focus on development. We saved a lot of time, and efforts, and the “time to market” of our solution was considerably shorter.
From the experience of working for and communicating with established TV broadcasting companies, we knew that upscaling is and always has been a main issue. At the same time, we think that downscaling is also to be considered due to financial constraints. This is why we have developed a video load balancing solution which is able to decide how many streaming instances are needed. This also allows our customers to benefit from the AWS “pay as you go” payment model.

Can you give us some more technical details?
Using high-CPU EC2 instances, the encoding cloud receives jobs from SQS, the Amazon Simple Queue Service; it then transcodes them into various formats and stores the results on S3. The videos are delivered by our playout cloud using Wowza Flash Streaming technology, which is also powered by Amazon EC2. Asset management is done using our IVMS (Incredible Video Management System) written in Django, persisting its data on a MySQL on EBS (where regular snapshots on S3 save us backup trouble). To prevent spikes in server load and reduce latency, we deploy SvM-Video-Workflow static files and rendered JSON on S3 and use the content delivery service Cloudfront where possible.

What significant benefits have you experienced?
In comparison to the main market solutions for film distribution on the web, we were able to save about 80% of the usual running costs. Using AWS has enabled us to develop our projects in small core teams and still deliver on time, with the advantage of saving a great deal of headaches for the admins.

I remember you told me about the day when you were featured on SPIEGEL online. How was it?
On May 8, 2009, just thirty minutes before launch, dctp.tv got  featured on SPIEGEL online. Our phone rang, ordering us to change the entire imprint. With shaky hands, we found ourselves deploying a new version in a matter of minutes and praying the cloudfront edge-servers updated on time. And yet, watching the server stats handling 50 hits per second and realizing that the load balancing module and the streaming cloud were working as they should was so enjoyable it made it all worthwhile.
All in all we had been working on our load balancing algorithms for many months without the opportunity of real world testing. The launch turned out to be our breakthrough, as everything worked as expected. And it has kept on doing just this ever since.
(comment from Simone: next time you need to load test your app, you should try Soasta.com)

Nikolai, do you have any suggestion for other AWS users?
Our experience is that the AWS team genuinely listens to the users’ needs and requirements.
A lot of the AWS features that we use today came to tackle problems that had been reported by users when we first started. So good work, and keep in touch

I believe our readers will find this story inspiring. Thanks Nikolai.

– Simone (@simon)

Everyday, we hear new stories about a cool new startup and its success story.

Today, It was WalkScore.com. The website offers some great information about which neighborhood/city is more walkable than the rest (San francisco was #1 and Seattle was #6). Walk Score calculates the walkability of an address by locating nearby stores, restaurants, schools, parks, etc. I think it is a great site for those who are consider moving to a new neighborhood or those who simply like the car-free lifestyle, especially because the gas prices are setting new records everyday.

This Seattle-based innovative startup hit almost all the major newspapers, blogs and websites last week from SF Chronicle to Washington Post, Los Angeles Times Blog to ABC News, From USA Today to MSNBC.

In an email, Matt Lerner from WalkScore says:

We would never have weathered being the #1 story on Yahoo! yesterday if it weren’t for Amazon!  THANK YOU!

We’re a hybrid-philanthropy business which means we prioritize social good over profit–and therefore we’re on a pretty tight infrastructure budget   What’s so great about Amazon cloud computing is that it was very cheap from an infrastructure and dev standpoint for us to scale up quickly.  In a nutshell we have only one physical web server and didn’t want to deal with the expense of a hardware upgrade so:

• We set up 4 EC2 instances to serve the walkability heat map tiles you see overlaid on top of the Google maps. Here is Seattle for example.
• We moved all of our images, CSS, and JS files to Amazon S3 which took a big load off of our one web server.
• We were able to accommodate a spike of about 80K unique visitors during a three hour period thanks to Amazon

One great article which I would like to highlight is How We Built a Web Hosting Infrastructure on EC2. Its a nice read if you are trying to host your website on Amazon EC2.

— Jinesh

From time to time, potential users of AWS ask me about the best way to set up a highly scalable architecture using Amazon EC2, S3, SimpleDB, and SQS. I’d like to challenge readers of this blog to document their AWS-powered architectures in a blog post, preferably with a diagram, and to leave comments with a link back to their posts. I’ll collect them all up in a future post.

Here are a few that I have already:

Doug Kaye described the architecture behind GigaVox Audio Lite in his post, Amazon for Infrastructure-on-Demand. Doug used EC2, S3, and SQS to build the highly scalable podcast processing system behind The Conversations Network.

Doug’s implementation regulates the number of EC2 instances in use by tracking the amount of time it takes to process each work item in the queues which drive the Transcoding and Assembly processes.

Don MacAskill described SmugMug’s master controller (SkyNet) in SkyNet Lives! (aka EC2 @ SmugMug). Don’s post doesn’t include a block diagram, but it does include a cool usage graph (included at right).

Don’s master controller watches 30 to 50 factors in order to make high quality scaling decisions. It was called RubberBand until it became sentient and attempted to take over the world launch several hundred Extra-Large EC2 instances simultaneously. It was then renamed SkyNet.

The architecture behind online social media dating site Zoosk is described in Elastic Computing with Amazon Web Service, written by Zoosk CEO Shayan Zadeh.

Per the blog post, they use SQS to maintain a queue of uploaded photos. The photos are processed on EC2 and then uploaded to S3. The graph in the blog post indicates that they are adding approximately 4 TB of new photos every month.

The AWS Developer Connection has some worthwhile how-to articles as well. In Monster Muck Mashup – Mass Video Conversion Using AWS, Mitch Garnaat shows how to use SQS, EC2, and S3 to do video conversion in a scalable way.

The article Auto-scaling Amazon EC2 with Amazon SQS also has a whole lot of really good information.

Once again, I invite you to write an architecture post of your own and to leave a link to it in the comments. I would also like to see posts which make reference to load management tools such as Scalr, RightScale, and Elastra.

Updates (before I write the next big post):

• David Kavanagh wrote in to tell me that there’s a good picture in his recent article, Automated Server Pool Management in Java.
• Scaling on EC2 tells the story of WebMynd, which looks like a pretty cool Firefox extension.
• Mike Brittain has a lot of great details in his blog post about How We Built a Web Hosting Infrastructure on EC2.

— Jeff;

We’ve been busy cooking up some new customer success stories to add to our collection. Here’s what’s new:

Hasbro, the mega producer of games and toys, recently came to Digitaria to help them produce an online marketing campaign around the first-ever Monopoly Here and Now: World Edition. The campaign web site allowed game-lovers worldwide to vote for their city to be included in the game’s new edition.

As the story says, “Looking for an economical solution, Digitaria used Amazon Web Services and open source software to produce the site’s back-end infrastructure from a single Amazon EC2 image. With this, they created and launched a new database, application server, caching server, and load balancer instances all within a matter of minutes.”

Read the success story…

The Talk Market is a new shopping channel lead by  seasoned television veterans Matt Singer and Amanda Eilian, whom together have sold over 500,000 products on QVC, the TV shopping channel. Additionally, they have produced television and music productions with celebrities such as George Clooney, James Earl Jones, Paul McCartney, Billy Joel, and Ray Charles.

Read the success story…

Familybuilder.com provides genealogy and family-oriented applications within online social networks. The company’s flagship product, Family Tree, is a leading social genealogy application on Facebook, Bebo, and MySpace.

Read the success story…

— Jeff;