Scientific Computing with EC2 Spot Instances

Do you use EC2 Spot Instances in your application? Do you understand how they work and how they can save you a lot of money? If you answered no to any of these questions, then you are behind the times and you need to catch up.

I’m dead-serious.

The scientific community was quick to recognize that their compute-intensive, batch workloads (often known as HPC or Big Data) were a perfect fit for EC2 Spot Instances. These AWS customers have seen cost savings of 50% to 66% when compared to running the same job using On-Demand instances. They are able to make the best possible use of their research funds. Moreover, they can set the Spot price to reflect the priority of the work, bidding higher in order to increase their access to cycles.

Our friends over at Cycle Computing have used Spot Instances to create a 30,000 core cluster that spans 3 AWS Regions. They were able to run this cluster for nine hours at a cost of $1279 per hour (a 57% savings vs. On-Demand). The molecular modeling job running on the cluster consumed 10.9 compute years and had access to over 30 TB of RAM.

Harvards Laboratory for Personalized Medicine (LPM) uses Amazon EC2 Spot Instances to run genetic testing models and simulations, and stretch their grant money even further. One day of engineering allowed them to save roughly 50% on their instance costs moving forward.

Based on the number of success stories that we have seen in the scientific arena, we have created a brand new (and very comprehensive) resource page dedicated for scientific researchers using Spot Instances. We’ve collected a number of scientific success stories, videos, and other resources. Our new Scientific Computing Using Spot Instances page has all sorts of goodies for you.

Among many new and unique things you will find:

• A case study from Harvard Medical School. They run patient (virtual avatar) simulations on EC2. After one day’s worth of engineering effort, they now run their simulations on Spot Instances and have realized a cost savings of over 50%. Some of the work described in this case study is detailed in a new paper, Biomedical Cloud Computing With Amazon Web Services.
• A video tutorial that will show you how to use the newest version of MIT’s StarCluster to launch an entire cluster of Spot Instances in minutes. This video was produced by our friends at BioTeam.
• A video tutorial that will show you how to launch your Elastic MapReduce jobs flows on Spot Instances.
• Detailed technical and business information about the use of Spot Instances for scientific applications including a guide to getting started and information on migrating your applications.
• Common architectures (MapReduce, Grid, Queue, and Checkpoint) and best practices.
• Additional case studies from DNAnexus, Numerate, University of Melbourne/University of Barcelona, BioTeam, CycleComputing, and EagleGenomics.
• A list of great Solution Providers who can help you get started if you need a little extra assistance migrating to Spot Instances.
• Documentation and tutorials.
• Links to a number of research papers on the use of Spot Instances.
• Other resources like our Public Data Sets on AWS and AWS Academic programs.

Spot Instances work great for scientific Research, but there are a huge number of other customers out there that also love Spot. As an example Spot works really well for loads of other use cases like analytics, big data, financial modeling, geospatial analysis, image and media encoding, testing, and web crawling. Check out this brand new video for more information on common use cases and example customers who leverage them.

Again, if you don’t grasp the value of Spot Instances, you are behind the times. Check out our new page and bring yourself up to date today.

If you have a scientific computing success story of your own (with or without Spot) or have feedback on how to make Spot even better, we’d love to hear more about it. Please feel free to post a comment to the blog or to email it to us at spot-instance-feedback@amazon.com.

Finally, if you are excited about Spot and want to join our team, please contact Kelly OMara at komara@amazon.com to learn more about the team and our open positions.

— Jeff;

Modified 2/11/2021 – In an effort to ensure a great experience, expired links in this post have been updated or removed from the original post.