AWS Startups Blog
Heyzap: Getting as Close to Elastic as Possible
Guest post by Samantha Weber & Chris Zutler, Heyzap
What is Heyzap?
Heyzap builds tools for mobile developers to help them visualize, manage, and optimize the monetization of their apps all in one convenient dashboard. We’ve built a simple SDK for Android and iOS, and provide advertising, publishing, mediation, and cross-promotion services to mobile developers worldwide.
“Wanting to find a long term solution, we began researching other options and decided to try AWS for its flexibility and low cost.”
Our Journey
In our early days, Heyzap was powered by dedicated servers from a major provider. Within a few days of leasing servers, we were up and running. Unfortunately, we quickly began to experience several issues, including repetitive hard drive failures that required redirecting engineering resources away from our core projects. Soon the servers couldn’t handle the traffic load, and we were facing serious outages on our website because adding new servers took too long with our existing service provider. Wanting to find a long term solution, we began researching other options and decided to try AWS for its flexibility and low cost.
Working with AWS
Heyzap is powered by a suite of AWS tools and services. Some of our favorites include Amazon Elastic Compute Cloud (EC2), Amazon Elastic Block Store (EBS), and Amazon Redshift. Scaling flexibility and hourly billing has allowed us to turn up our servers, at any size, in minutes instead of days. This is important because it’s not uncommon for us to see double or even quadruple our base volume during peak hours. Plus we’re often running multiple background jobs that require serious CPU power.
Scaling
Customization and scalability are critical to our mobile ad network since we handle more 600 k requests per minute. As we’ve grown, AWS has continued to meet our needs and added products and features at just the right time to help us scale up and out.
To begin with, AWS helped us resolve bottlenecks that we experienced with MongoDB. Like most databases, MongoDB works best when an active dataset fits in RAM. When we started to reach the max (70 GB) on the largest RAM instance available on EC2, we knew we would need to start sharding our dataset. Before we started this project, Amazon announced a new instance class with 244GB of RAM which allowed us to simply scale up and conserve engineering resources.
Another huge upgrade in EC2 has been Amazon Elastic Block Storage (EBS) I/O performance. Originally all EBS volumes were backed up with magnetic storage. While these volumes were flexible and easy to backup, they didn’t work well for high-I/O applications. To increase performance, we started using software RAID across several EBS volumes. This worked well for two to five volumes; however, we soon hit a performance ceiling. AWS came to the rescue with Provisioned IOPS, which performed at four times the rate of our fastest RAID volumes. Since then, Amazon has added lower cost GP SSD volumes with similar performance characteristics.
EC2 Spot Instances
Heyzap uses custom Autoscaling and Spot-Instance bidding scripts to make sure we always have enough resources online and are not overspending during off-peak hours.
With Spot Instances, Heyzap is able to bid on excess capacity at significantly lower prices than normal AWS rates. The only downside has been that Amazon has the right to terminate them on short notice, which potentially can result in excess load on a cluster.
To mitigate this risk, we look at spot price history to see if some instance types have fewer “price spikes” that might cause us to be outbid. If we need a server with a lot of CPU power, but care less about RAM or attached storage, there are usually several instances types that will meet our needs. By choosing from a larger pool of instances, we are able to avoid having too many servers shut down at once.
Our current script logic compares the most recent and historic prices of all the “good-enough” instances and bids on whichever looks the most stable at a price that’s likely to last at least several hours. We also make sure to vary the instance types enough so that losing all of a given type won’t cause overload.
If none of the instances that meet our requirements are price stable or have low enough cost, we fall back to demand servers. By adding a pool of reserved servers, we maintain a healthy, highly available cluster of instances that are at far below AWS list prices.
Amazon Redshift
Every month Heyzap holds a companywide hack day. Roughly two years ago, several of our engineers collaborated and built a unified analytics database to help us make faster and more informed business decisions. This hack day project looked great and had obvious value, but the database backend couldn’t keep up after a few days. That’s when we started using Amazon Redshift, which worked perfectly for this type of workload. Amazon Redshift allows our project to thrive internally, and it continues to be our “secret weapon” for investigating problems and making the best decisions for Heyzap.
The Future
We see great things in the future with AWS. Two newer Amazon offerings that we’re looking to take advantage of are EC2 Container Service (ECS) and AWS Lambda. ECS appears to be a great way to manage and scale our increasingly service-centric architecture, while Lambda could replace a whole host of servers used for background jobs. Both are prime candidates for future hack day projects.
Heyzap wants to make mobile advertising transparent and simple for developers to integrate into their apps. AWS has helped us scale so millions of people can discover the apps they love. We’re excited about the future of Heyzap Mediation, and hope you’ll help us spread the word.