StarCluster

Customer Apps>StarCluster
StarCluster is a utility for creating and managing general purpose computing clusters hosted on Amazon's Elastic Compute Cloud (EC2). StarCluster minimizes the administrative overhead associated with obtaining, configuring, and managing a traditional computing cluster used in research labs or for general distributed computing applications. StarCluster utilizes Amazon's EC2 web service to create and destroy clusters of Linux virtual machines on demand.

Details

Company: MIT
Inquiry e-mail address: star@mit.edu
Amazon Web Services Used: Amazon S3
Solution URL: http://web.mit.edu/stardev/cluster/
Audience: Developers
Pricing: Free of charge
How does this application use Amazon Web Services?: All that's needed to get started with your own personal computing cluster on EC2 is an Amazon AWS account and StarCluster.
Created On: September 11, 2009 2:17 PM GMT
Last Updated: July 13, 2010 4:32 PM GMT

About

StarCluster is a utility for creating and managing general purpose computing clusters hosted on Amazon's Elastic Compute Cloud (EC2). StarCluster minimizes the administrative overhead associated with obtaining, configuring, and managing a traditional computing cluster used in research labs or for general distributed computing applications. StarCluster utilizes Amazon's EC2 web service to create and destroy clusters of Linux virtual machines on demand.

To get started, the user creates a simple configuration file with their AWS account details and a few cluster preferences (e.g. number of machines, machine type, ssh keypairs, etc). After creating the configuration file and running StarCluster's "start" command, a cluster of Linux machines configured with the Sun Grid Engine queuing system, an NFS-shared /home directory, and OpenMPI with password-less ssh is created and ready to go out-of-the-box. Running StarCluster's "stop" command will shutdown the cluster and stop paying for service. This allows the user to only pay for what they use.

Dependencies

StarCluster has a minimal set of dependencies listed below:
  • Registered and fully configured Amazon EC2 account.
  • Python2.4+
  • Paramiko (python module for ssh, developed against v 1.7.6)
  • Boto (python module for aws, developed against v 1.9d)

Software Included in the StarCluster EC2 AMI

StarCluster comes with a publically available AMI on EC2 that includes an extremely minimal software stack for distributed/parallel computing. Currently, the AMI is based on Ubuntu 9.04 and comes in both i386 and x86_64 flavors. The AMIs include the following software:
  • OpenMPI - Library used for writing/running parallel applications
  • Sun Grid Engine - Queuing system for scheduling jobs on the cluster and handling load balancing.
  • NFS - Network File System for sharing folders across the cluster.
  • SciPy - Scientific algorithms library for Python (compiled against ATLAS for 8-cpu instance types)
  • NumPy - Fast array and numerical library for Python (compiled against ATLAS for 8-cpu instance types)
  • IPython - An advanced interactive shell for Python.
  • and more ...
AMI ids: ami-d1c42db8 (i386), ami-a19e71c8 (x86_64)
©2014, Amazon Web Services, Inc. or its affiliates. All rights reserved.