Sauce Labs – OnDemand Testing Service on EC2

Late last month I spent some time on the phone with John Dunham and Steve Hazel of Sauce Labs to learn more about their Sauce OnDemand testing service. The product is built around the popular Selenium testing tool and can actually make use of existing Selenium scripts for functionality and performance testing. John and Steve said that I can think of it as “robotically controlled browsers as a service.”

We talked about the fact that the cloud is the natural place for testing resources, since the amount of usage within a particular organization is subject to extreme fluctuations. John told me that the dedicated test resources in a typical organization are idle 99% of the time. This inefficient use of capital is of great concern to CIOs and CTOs, since the return on idle resources is zero.

By allocating EC2 instances on demand, their customers can now test in hours instead of days. Steve said that many customers see a 10- to 20-fold improvement (reduction) in test times because they can simply throw additional resources at the problem. He told me that this works really well until it becomes an “accidental load test” of the system under test! They perform a continuous screen capture from each test machine and store all of the video in Amazon S3. To date they have run over 3 million tests and have retained all of the videos. This feature is called Sauce TV.

Architecturally, the system uses a single Extra Large (m1.xlarge) EC2 instance to coordinate an elastic pool of High CPU Medium (c1.medium) test servers. They’ve built 5 or 6 different Amazon Machine Images (AMIs) with different combinations of operating systems and browsers (IE6, IE7, Firefox, and so forth — complete list is here). They keep a few instances idle so that they can start testing as soon as the need arises. They make use of historic usage data to guide their pre-scaling process, further adjusted to reflect actual demand. They are able to run high-scale tests in short order without charging extra.

Scale-wise, the largest test has consumed about 250 instances and the overall pool has grown as large as 400 instances.

You can sign up for a free 30-day trial of Sauce OnDemand. After that, you can get 1,000 minutes of testing per month for $49 (additional minutes cost $0.05 each). There are also some enterprise plans.

Sauce Labs has implemented a number of improvements to Selenium including SSL and Unicode support, cross-browser file uploads, and an OS-level popup eliminator. They plan to make these enhancements available to the Selenium project.

John told me that The Motley Fool is really sold on the idea of cloud-based testing with Sauce OnDemand. Here’s a presentation from Dave Haeffner, Quality Assurance Manager at The Motley Fool:

–Jeff;

Modified 2/11/2021 – In an effort to ensure a great experience, expired links in this post have been updated or removed from the original post.