Category: Amazon EC2


Parallel Computing With MATLAB On Amazon EC2

by Jeff Barr | on | in Amazon EC2 |

Update (2017) –  This blog post is here for historical purposes only.   The white paper has been superseded by the Cloud Center web application from MathWorks.“

Jeff;


 

Ec2Mathworks released a whitepaper on how to run MATLAB parallel computing products – Parallel Computing Toolbox and MATLAB Distributed Computing Server on Amazon EC2. This step by step guide walks you through the steps of installation, configuration and setting up clustered environments using these licensed products from MathWorks on Amazon EC2. It shows how you can create an AMI with MATLAB products bundled in and run them in the cloud.

Whitepaper is available free on Mathworks website:

MATLAB users will learn about the key aspects of using the EC2 service from their desktop MATLAB session and using Parallel Computing Toolbox to send parallel MATLAB computations to the EC2 service.

System administrators will learn the key technical details required for setting up MATLAB Distributed Computing Server on the EC2 service, including licensing and network setup. They will also learn how to configure their users desktops to enable the use of the EC2 service for MATLAB computations.

This is the ‘Era of Tera’. We have terabytes of data to compute. Data is never going to be less. Parallel Computing models are the answer to the future. Cloud computing is making it easier for the masses.

I am very excited because this is going to open up powerful MATLAB tools to any developer for not only research but also production applications. Students might be able to do their lab exercises without a lab and impress their professors by turning in the assignments before time. Professors will be able to teach courses using MATLAB by “turning on” a switch that creates their “Instant Labs” for the duration of the course without even contacting the College IT department for resources. Enteprises might be able to crunch the complex BI data over the weekend for a monday morning meeting.

Also, Mathworks Consulting and Support is available for those who are starting up with a MATLAB project on Amazon EC2.

Do you have a cool MATLAB usecase?

— Jinesh

Update: On licensing, MathWorks is aware of the requirement and would be willing to discuss this further with people that are interested in using MDCS on EC2. Please send us an email at evangelists at amazon d0t com with your usecase.

SOASTA – Load Testing From the Cloud

by Jeff Barr | on | in Amazon EC2 |

I met Tom Lounibos, CEO of SOASTA, at the Palo Alto stop of the AWS Start-Up Tour. Tom gave the audience a good introduction to their CloudTest product, an on demand load testing solution which resides on and runs from Amazon EC2.

Soasta_record Tom wrote to me last week to tell me that they are now able to simulate over 500,000 users hitting a single web application. Testing at this level gives system architects the power to verify the scalability of sites, servers, applications, and networks in advance of a genuine surge in traffic.

Here are a few of their most recent success stories:

  • Hallmark tested their e-card sites in preparation for the holiday season, and are ramping up testing to over 200,000 simultaneous users using CloudTest.
  • Marvel Entertainment is doing extensive cloud testing in order to get ready for the release of the sequel to IronMan.
  • A division of Procter & Gamble is using cloud testing to get ready for new releases of their web site.

Based on this video, it looks like it is very easy to create a test, run it, and to process and analyze the results.

Soasta_clip_editor The first step is to record a new test consisting of one or more user scenarios.  Next, the raw test is edited to generalize it and to specify test data, parameters, and variable substitutions. A drag and drop test creation tool is used to create real-world test scenarios on a multi-track timeline, The system under test can be monitored in various ways while the test is run. Once completed, the test results can be viewed and analyzed.

Pricing for CloudTest starts at $1000 per hour.

— Jeff;

Big Day for Amazon EC2: Production, SLA, Windows, and 4 New Capabilities

by Jeff Barr | on | in Amazon EC2 |

My colleagues and I have spent the week building up anticipation for this post on Twitter. After you read this post I am sure that you will agree that the wait was worthwhile.

The hallways at Amazon have been buzzing with excitement of late. After working for years to build and to run our line of highly scalable infrastructure web services we are happy to see that developers large and small are putting them to good use.

Here’s what’s happening today:

  • Amazon EC2 is now in full production. The beta label is gone.
  • There’s now an SLA (Service Level Agreement) for EC2.
  • Microsoft Windows is now available in beta form on EC2.
  • Microsoft SQL Server is now available in beta form on EC2.
  • We plan to release an interactive AWS management console.
  • We plan to release new load balancing, automatic scaling, and cloud monitoring services.

Let’s take a look at each of these items in turn.

Production – After a two year beta period, Amazon EC2 is now ready for production. During the beta we heard and responded to an incredible amount of customer feedback, adding support for powerful features such as Availability Zones, Elastic Block Storage, Elastic IP Addresses, multiple instance types, support for the OpenSolaris and Windows operating systems, and (as of today) a Service Level Agreement. Regular EC2 accounts are allowed to run up to 20 simultaneous instances. Requests for hundreds and even thousands of additional instances are granted all the time and can be made here.

SLA – The new EC2 Service Level Agreement works at the Region level. Each EC2 Region (there’s only one right now but there will be more in the future) is divided into a number of Availability Zones. The SLA specifies that each Region will be available at least 99.95% of the time. Per the SLA, a Region is unavailable if more than one of its Availability Zones does not have external connectivity.

Ec2_windows_yeah Windows Support – Beta level support for Microsoft Windows is now available on EC2, in the form of 32 and 64 bit AMIs, with pricing starting at $0.125 per hour. Microsoft SQL Server is also available in 64 bit form. All of the powerful EC2 features listed above can be used with the new Windows instances and we’ll be adding support for DevPay in the near future.

Once launched, the Windows instances can be accessed using the Windows Remote Desktop or the rdesktop client. I’ve spent some time using Windows on EC2 and it works really well. I used the EC2 command line tools to launch a 32 bit instance, opened up an additional port in the security group, and then logged in to it using Remote Desktop.

 

We’ll be running Windows on EC2 at next week’s PDC in Los Angeles, so be sure to stop by and to say hello if you are there. Rumor has it that we’ll be giving out a really cool badge to the people who stop by our booth.

RightScale founder Thorsten von Eiken has written up a helpful post which outlines the differences between Windows and Linux with respect to launching, accessing, bundling, and using the Elastic Block Store. He also describes current and planned support for Windows in their products.

Elasticfox_gsg We’ve updated ElasticFox with a number of new features, including direct access to EBS and Elastic IP addresses from the main tab, one-click AMI bundling on Windows, better key and security group management, and the ability to directly launch Remote Desktop sessions. There’s also a brand-new (and very helpful) ElasticFox Getting Started Guide.

We are looking forward to seeing how our customers will put Windows to work. We expect to see ASP.Net sites, media transcoding, HPC (High Performance Computing), and more. I’ve talked to a number of developers who will deploy hybrid web sites using a mix of Linux and Windows servers. This really underscores the open and flexible nature of EC2.

We are also planning to offer some new capabilities in 2009 to make managing cloud-based applications even easier. As usual, we’ll start with a private beta and you can express your interest here.

Management Console – The management console will simplify the process of configuring and operating your applications in the AWS cloud. You’ll be able to get a global picture of your cloud computing environment using a point-and-click web interface.

Load Balancing – The load balancing service will allow you to balance incoming requests and traffic across multiple EC2 instances.

Automatic Scaling – The auto-scaling service will allow you to grow and shrink your usage of EC2 capacity on demand based on application requirements.

Cloud Monitoring – The cloud monitoring service will provide real time, multi-dimensional monitoring of host resources across any number of EC2 instances, with the ability to aggregate operational metrics across instances, Availability Zones, and time slots.

Amazon CTO Werner Vogels has done a very nice job of explaining why services of this type are needed to build highly reliable and highly scalable applications. His blog is a must read for those interested in cloud computing. Werner has spent so much time talking about AWS of late that I’ve asked him to be an honorary member of my team of AWS evangelists!

I think it is important to note that load balancing, automatic scaling, and cloud monitoring will each be true web services, with complete APIs for provisioning, control, and status checking. We’ll be working with a number of management tool vendors and developers to make sure that their products will support these new services on a timely basis.

So, there you go. What do you think?

— Jeff;

Amazon EBS (Elastic Block Store) – Bring Us Your Data

by Jeff Barr | on | in Amazon EC2 |

A few months ago I talked about our plans to offer a persistent storage feature for Amazon EC2. At that time I indicated that the service was in a limited alpha release with a small number of customers. Since then the alpha testers have been putting the service to good use and have provided us with a lot of very helpful feedback.

As of today, the Amazon Elastic Block Store (EBS) is now open and available to all EC2 users.

EBS gives you persistent, high-performance, high-availability block-level storage which you can attach to a running instance of EC2. You can format it and mount it as a file system, or you can access the raw storage directly. You can, of course, host a database on an EBS volume. In fact, Eric Hammond has already written an article, Running MySQL on Amazon EC2 with Elastic Block Store.

EBS volumes can range in size from 1 GB to 1 TB. You can mount many of them on the same instance, and even stripe (aka RAID 0) your data across them to increase performance.

The volumes can be attached to any single instance within a single EC2 availability zone. They are also automatically replicated within the zone.

During the beta you can create up to 20 EBS volumes consuming a maximum of 20 TB of space. You can make a request for additional volumes here.

You can snapshot a volume to Amazon S3 with ease, and then, if needed, create new volumes (of the same or different sizes) using the snapshot as a base. Of course, if you create a new volume with a size that doesn’t match the size of the volume where you took snapshot, you will have to resize the new file system. When you create a new volume based on an S3 snapshot, the data is loaded lazily; there’s no need to wait for the snapshot to load.

EBS usage is charged based on storage and on I/O requests. Storage costs $0.10 per GB per month and I/O requests cost $0.10 per million. Snapshot storage is charged at Amazon S3 rates. The AWS Simple Monthly Calculator has been updated to reflect the new features so that you can estimate your costs with ease:

Aws_calc_ebs_3

All of the EBS functionality can be accessed through the EC2 APIs, through the EC2 Command Line tools, through ElasticFox, and via a number of third-party tools and libraries.

The popular ElasticFox extension for Firefox has been updated with full support for EBS. You can see all of your volumes and your snapshots on a new tab:

Elasticfox_ebs

You can create volumes and attach them to running instances using simple dialog boxes:

Elasticfox_create_vol

Elasticfox_attach_vol

You can create a snapshot with a single click:

Elasticfox_snapshot

And then create a new volume from the snapshot just as easily:

Elasticfox_create_from_snapshot

Third party tool and library support is already starting to appear. In fact, I’ve created a separate post, Amazon EBS – Tool and Library Support, which I will be updating a couple of times in the next day or so as announcements are made.

Also, Amazon CTO Werner Vogels has written a really good post which includes some great insights into the architectural and philosophical considerations behind our line of storage services.

And there you go!

— Jeff;

Amazon EBS – Tool and Library Support

by Jeff Barr | on | in Amazon EC2 |

This is a companion post to my earlier post — Amazon EBS (Elastic Block Store) – Bring Us Your Data. In the other post you can read about the features of EBS. This post goes into more detail on the tool and library support that has been built by our community of third-party developers.

Here are some tools:

And some libraries (some of the third parties will finalize their support in a day or two):

— Jeff;

PS –  I’ll be updating this post a couple of times in the wake of the EBS launch so come back again soon.

Jollat – Cross-Platform AWS Manager Client

by Jeff Barr | on | in Amazon EC2 |

JollatAndras wrote to tell me about Jollat, a new graphical cross-platform (Windows, Mac, and Linux) management client for Amazon EC2 and S3. Available for free download (with a purchase option), the client includes a number of interesting features.

On the S3 side, Jollat handles bucket creation in both the US and EU zones, upload and download of multiple files, log file configuration and management, and an access control list (ACL) editor.

On the EC2 side, Jollat’s image manager makes it easy to find and launch any AMI (Amazon Machine Image). Once launched, instances can be accessed using an embedded SSH client. The tool also manages availability zones, IP addresses, and key pairs.

You can see Jollat in action by watching the video.

— Jeff;

JBoss Releases on Amazon EC2

by Jeff Barr | on | in Amazon EC2, Conference |

By now many of you are aware that Red Hat Enterprise Linux is fully supported by Red Hat on Amazon EC2. You can read more about the offering at http://www.redhat.com/solutions/cloud/. Jeff Barr blogged about this in November, 2007 (aws.typepad.com/aws/2007/11/red-hat-enterpr.html).

Im posting this from Boston, where I am attending the Red Hat Global Summit — more specifically helping with a hands-on lab that teaches developers and IT staff how to deploy Red Hat Enterprise Linux (RHEL) on Amazon EC2. (It’s really easy.) Its been fun to meet enterprise developers from all over the world, and surprising to find out that no matter what country the developer is in awareness about Cloud Computing is high.

JBoss
Perhaps you already saw the posts in other blogs Red Hat announced that their JBoss Enterprise Application Platform is available in beta form as a service within the Amazon Elastic Compute Cloud (Amazon EC2).

Traditionally we think of Java application servers as building blocks that live in a hallowed enterprise data center; however with this announcement yet another one of those essential technologies is running fully supported by the vendor in the Cloud. In mission-critical applications support is essential–and for Red Hat products that means 24×7 operational support plus developer support. See www.redhat.com/support/policy/sla/production for a menu of offerings to choose from.

This is all quite amazing. Just over two years ago Amazon Simple Storage Service launched, followed in August of 2006 by Amazon Elastic Compute Cloud. In the short span of time since 2006 weve seen Cloud Computing grow from an idea to of course we use it for many organizations. With the advent of powerhouse enterprise infrastructure and applications, it seems inevitable that line-of-business applications in the cloud will become commonplace.

Getting started is easy, with just three steps:

  1. Sign up for Amazon EC2
  2. Purchase a subscription to Red Hat Enterprise Linux (RHEL) on Amazon EC2 or purchase a subscription to JBoss on Amazon EC2
  3. Deploy your applications on the newly-minted application server; then optionally make a custom AMI from this image and save it as your own private version in Amazon S3.

You can learn more at aws.amazon.com/partners/redhat.

Mike

More EC2 Power

by Jeff Barr | on | in Amazon EC2 |

Ec2_high_cpu Amazon EC2 users now have access to a pair of new “High-CPU” instance types. The new instance types have proportionally more CPU power than memory, and are suitable for CPU-intensive applications. Here’s what’s now available:

The High-CPU Medium Instance is billed at $0.20 (20 cents) per hour. It features 1.7 GB of memory, 5 EC2 Compute Units (2 virtual cores with 2.5 EC2 Compute Units Each), and 350 GB of instance storage, all on a 32-bit platform.

The High-CPU Extra Large Instance is billed at $0.80 (80 cents) per hour. It features 7 GB of memory, 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each), and 1,690 GB of instance storage, all on a 64-bit platform.

The AWS Simple Monthly Calculator now supports these new instance types.

We’ve been working with a number of tool vendors to line up early support for this important new feature. I plan to update the blog post several times in the coming days as this support becomes available.

— Jeff;

Cloud Studio

by Jeff Barr | on | in Amazon EC2 |

Cloudtools Alexsey and Tatyana from Cloud Services dropped me an email to tell me about the beta release of their new Cloud Studio product.

Cloud Studio is a Java application for the management of Amazon EC2 instances. It features a multi-pane interface with a list of available AMIs, a list of running instances, and access to keypairs, security groups, and  IP addresses. Menu options are provided for image registration and deletion, keypair manipulation, security group editing, and IP address assignment.

The application can be run standalone or it can be run from within Eclipse.

You can see a Flash demo on the home page, or you can simply download it.

— Jeff;

Redundant Disk Storage Across Multiple EC2

by Jeff Barr | on | in Amazon EC2 |

M_david_preparing_for_ec2_persisten XML Hacker M. David Peterson has put together a really interesting article.

As part of his work at 3rd and Urban, he has implemented redundant, fault-tolerant, read-write disk storage on Amazon EC2 using a number of open source tools and applications including LVM, DRBD, NFS, Heartbeat, and VTUN.

Mark notes that “the primary focus of this paper is to present both a detailed overview as well as a working code base that will enable you to begin designing, building, testing, and deploying your EC2-based applications using a generalized persistent storage foundation, doing so today in both lieu of and in preparation for release of Amazon Web Services offering in this same space.”

The article provides complete implementation details and links to source code for the scripts that Mark developed.

You can read the article, and you can also follow progress via the discussion group.

— Jeff;