AWS Case Study: Globus Online

Steve Tuecke, Globus Online Project Lead and Deputy Director of the Computation Institute at the University of Chicago and Argonne National Laboratory, explains the team’s decision to use Amazon Web Services (AWS): “When we started this effort in 2009, best practices for how to build reliable, scalable Software-as-a-Service [SaaS] applications on AWS fit well with what we wanted to accomplish.” According to Tuecke, the following factors made AWS a better solution than other hosting providers:

• Flexibility. Tuecke says, “It is easy to temporarily stand up several development and test instances of the site for various purposes, such as integration testing, load testing, quality assurance, etc.”
• Reliability. Multiple regions and availability zones give the team the ability to easily support a more reliable service.
• Cost. Tuecke notes, “The flexibility of the AWS model of paying only for what we use suits us well.”
• Global Footprint. The global footprint of AWS will make it easy for the service to move beyond the U.S. to Europe and Asia-Pacific – although there are already many Globus Online users as far afield as Australia and New Zealand.
• Leadership. Tuecke says, “It was clear that AWS was and is the industry leader for cloud hosting.”

The Globus Online system currently comprises several related but distinct components, which use Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Load Balancing, and Amazon Simple Storage Service (Amazon S3):

1. Platform services (User, profile, and group management, and the web site). A collection of instances running across several availability zones in the U.S. East region, including web server, load balancer, database, and backup.
2. File transfer service. A collection of instances running in the U.S. East region, comprising transaction database, transfer agents, history database, transfer REST API server, command line interface (CLI) server, and backup.
3. Operations. The team runs two Nagios servers on Amazon EC2 instances: one in the U.S. East to monitor all of the service’s instances, and the other in the U.S. West region to monitor the health of the primary Nagios server. In addition, they use Chef for provisioning all of the servers.
4. Development and Test. Tuecke says, “All of our development and testing is done on AWS, using replicas and subsets of the above services.”

The vast majority of Globus Online is programmed in Python, running on Ubuntu Linux servers, with Cassandra and Postgres databases.

The team is currently using about 10 instances of varying sizes across multiple availability zones to host their production service. Tuecke notes, “We have tens of additional instances running at any given time for development and test environments.”

Since its launch in November 2010, Globus Online has moved over 1.5 petabytes of data and acquired over 2,600 registered users, numbers that are steadily growing. According to Tuecke, cost savings, man hours saved, and reduced time-to-market are all “sizable.”

As the team introduces new services, they plan to evaluate and consider adopting additional AWS products, including Amazon CloudFront, Amazon Relational Database Service (Amazon RDS), Amazon Simple Email Service (Amazon SES), Amazon Route 53, and reserved instances of Amazon EC2.

The team has been pleased with AWS’s ability to provide a good mix of low-level control and flexibility. Tuecke says, “We have studied and tried other cloud providers, and their models have typically been too rigid to fit our needs. AWS has allowed us to tailor our approach to our particular needs, and to our team’s skills.”

To learn more, visit http://www.globusonline.org/ .

Added November 10, 2011