AWS News Blog
Virtual Stress-free Testing in the Cloud
|
I am giving a presentation at upcoming FutureTest conference on “Testing in the Cloud”. Using Amazon EC2 for Software Testing is one of “Lowest-hanging fruit” use cases. In this blog post, I will review some of the highlights of my presentation – some of the cool things the cloud brings to the world of application testing.
Instant Virtual Test Lab
Our customers love the fact that they can spawn up Test boxes when they need them, within minutes and terminate them when they are done with testing. The time it takes to procure new hardware, configuration, upfront investment, under-utilization of testing resources were major obstacles to good quality application testing and hence was often either ignored or was incomplete. The fact that they can now get 10, 15 or even 1000s of server instances encourages better and more complete testing and thereby creating more confidence. Instant test labs for a 3-month Test phase when they spawn few dozens instances to do their traditional usability and functional tests. Outsourcing/QA companies love it because they can now charge their customers for the infrastructure they consumed that month (usage-based costing). The day does not seem far when Cloud becomes the ideal testing infrastructure of the future.
Testing as a Service
Companies are like SOASTA, LoadStorm, Browsermob are building their businesses to provide Testing-as-a-Service. SOASTA spawned 650 EC2 Servers to simulate load from two different availability zones to stress test a music-sharing website QTRAX. After the 3-month iterative process of test-analyze-fix-test cycle, QTRAX can now serve 10M hits/hour and handle 500K concurrent users. This was a significant improvement from their first test which failed at 500 concurrent users. This “Let’s run it again” philosophy encourages iterative process and helps you focus on your fixes while abstracting out all the complexity of generating load and analyzing results (because they provide you with real-time graphs on a dashboard. Likewise, LoadStorm users can select a target server, build out test plan and run a load test right from their website. Browsermob, built by the core-developer of Selenium project, has created a client parallelization testing tool that spawned 2000 real browsers on 334 High-CPU XL instances for a media company, 1-cast, to test complex streaming and AJAX features of the website. These vendors offer Testing as a Service and reduce the complexity of traditional enterprise tools (like LoadRunner) and have pay-as-you-go models (charging either by bandwidth consumed or testing hours usage) thereby reducing huge upfront licensing cost. Not to mention saving time in learning and configuring these systems and maintaining and operating physical test labs. This “Push it to the Cloud” philosophy helps us to focus on building cool features in our applications and outsource QA while not losing control (unlike traditional outsourcing/offshoring models)
Throw-away boxes
It was funny when I was talking to some developers about testing. One of the developer jumped and said “If you mess up the configuration, simply dump the instance and start a new one. Time is precious, dude!” I knew developers from server-less start-up companies in our ecosystem, who start their dev boxes every morning and run it for 12 hours (the average developer uptime) per day and shut down every night before they go home. But I never thought that one can actually use Amazon EC2 to create thousands of Test environments in the Cloud – all fresh and new – and dump them, if they mess up and/or recreate it in the next test/sprint cycle. When you are testing your mobile application on different device platforms or testing your database-agnostic and app server-agnostic middleware app on different deployment configurations, the Cloud becomes an ideal platform to create-dump-recreate environments as you need.
Virtualization (AMIs) for Repros
Creating Test Environments with one-click has lots of advantages. The on-going battle between testers, developers and operators of I cannot reproduce the bug could be resolved by few clicks and recreating the environments and sharing Amazon Machine Images. It is a one click BundleInstance request (for Windows) using our Management Console and 3 command line calls to bundle/upload/register an AMI (Linux). Of course, its not advisable to create AMIs for every bug (thats insane) but at least severe production bugs could be reproduced very quickly.
Automation through Web Services
One of the best technology benefits of Cloud Computing is Automation via Web Service Interfaces. It is now easy to setup a system that builds your code at 2AM, runs it on 2 Large Instances for 2 Hours, run all your regression and unit tests to make sure that all the code is compiling fine. (Total Cost ~$1/day). One can also setup to replicate their environment and test the application by running clients from different Geographical Regions (US/EU) and Availability Zones.
Mechanical Turk for Innovative Testing
Using the on-demand workforce to help you in testing your app:
- Workers create actual test scenarios (Selenium)
- Workers enable Usability testing
- Worker help analyze results from Cross-browser testing (present a screenshot and ask a turker to compare the pages)
- Worker analyzes your test results/log files
- Worker tests for broken links on your website
- Workers participate in surveys that rate look-and-feel, navigation, search features of your website
You will find variety of customer stories and actual HITs on Mturk.com. Last year’s Start-Up Challenge Contest nominee UserTesting.com uses Amazon Mechanical Turk to create real videos to capture user behavior.
Parallelization (Client and Server)
Browsermob can start up 1000s of real Firefox client browsers (no http simulated traffic) and test your applications the way it should be tested. All of this happens in parallel. Imagine if you could do this on server-side as well. In other words, run all your tests (Unit Tests, Integration Tests, Regression Tests, browsers etc) in parallel. Cost of 1 server that runs for 1000 hours is same as 1000 servers for 1 hour. But in latter case, Quality will be verified in minutes instead of hours and days. Cloud Computing inspires us to think parallel.
To reduce the uncertainty in today’s Era of Tera (we never know when your app will be successful), we have to perform all the stress and load tests necessary prior to launch so that we can set the right expectations (SLAs). Customers all around the globe will access your web application. If the website is slow and does not respond, it amounts to bad customer service.
Take advantage of all the technologies that make the cloud (Virtualization for reuse/repros, Web services for automation, crowdsourcing for creating test scenarios) to stress/load/performance test your web applications thereby improving quality and speed (response time) of your application while keeping your costs down.
James Whittaker wrote series of posts on “Future of testing” which I highly recommend everybody to read. He predicts that the next generation of Testing is TestSourcing (after Crowdsourcing, Outsourcing and Insourcing) in which the focus will be on Tests (and not Testers, Testing and Tools). Embellishing on it, I think:
Testing in the Cloud – Use the power of the people (on-demand workforce to create standardized re-usable test scenarios) of your apps and the power of cloud computing tools like Amazon EC2 (on-demand hosted Virtualization) and Amazon S3 (for storing your logs/regression results).
How do you see Testing in the Cloud evolve?
If you would like to learn more about these thoughts, Catch me at FutureTest Conference in New York!
— Jinesh