Get Started with the Project

6 Steps  |  60 Minutes

Linux_HPC_Cluster_aws-project
architectural-diagram
aws-arch_hpc-cluster

Amazon Web Services (AWS) makes it easy for you to access on-demand and scalable compute capacity to power your High Performance Computing (HPC) workloads. Examples of HPC workloads customers have run on AWS include CFD simulations, computational chemistry workloads, genomics analysis, weather forecasting, and more. In this project, you will deploy an HPC cluster on AWS using CfnCluster, an open-source solution that enables you to rapidly provision and manage HPC clusters in the cloud. These cloud-based clusters deliver many similar characteristics of traditional on-premises clusters while leveraging uniquely cloud-based capabilities. For example, CfnClusters are fully elastic, meaning that compute nodes are only launched when there is work to be done. 

What you'll accomplish:

Install CfnCluster, an open-source tool that enables you to create and configure an elastic HPC cluster on Amazon EC2. The CfnCluster source code is published on GitHub and documentation is available at Read the Docs.

Configure and launch an elastic HPC cluster on Amazon EC2 using CfnCluster. You can customize your first HPC cluster by selecting the deployment region, VPC, instance type, volume type, minimum and maximum number of compute nodes, and preferred batch scheduler.

Submit and run a parallel MPI job. A simple parallel “hello world” code is compiled and submitted to the scheduler to launch the job, enable the compute nodes, and execute the MPI job.

Take an EBS snapshot of the shared disk for reusability with Amazon EBS. A snapshot of the shared disk is created, the cluster is shut down, and a new cluster is launched with the “hello world” software pre-installed and ready to run.

What you'll need before starting:

An AWS Account: You will need an AWS account to set up your HPC environment. Sign up for AWS.

Skill level: Previous experience with HPC scheduling software is recommended to successfully complete this project.

AWS Experience: Familiarity with Amazon EC2, S3, and VPC is recommended to successfully complete this project.

Cost Estimate:

The total cost to complete this Linux HPC project is estimated to be less than $1. This cost assumes that you accept the recommended configurations and that you terminate all resources within 24 hours of completing the project. 

Your use case may require different configurations that will impact your bill. For example. the CfnCluster config file calls out instance type, maximum queue size, and EBS volume sizes. When CfnCluster is launched with customized configuration settings, the costs will vary depending on set up and usage. Typically, you would launch a cluster only for the time period you need to use it, to minimize the cost of running large workloads.

To see a breakdown of the services used and their associated costs, see Services Used and Costs.

Get Started with the Project

This paper describes a range of HPC applications that are running today on AWS and provides best practices.

Learn more about HPC capabilities and find resources to help get you started

Need more resources to get started with AWS? Visit the Getting Started Resource Center to learn more.