Category: Amazon EC2*


Manage Amazon EC2 With New Web-Based AWS Management Console

Today were announcing the availability of the Web-based AWS Management Console, which in this first release provides management of your Amazon EC2 environment via a point-and-click interface. A number of management tools already exist: for example a popular Firefox extension known as Elasticfox; however as you read more of this post I believe youll agree that the new console is compelling–especially when its time to log in as a new AWS developer.

For starters, its easier than ever to gain access to your Amazon EC2 environment. The console provides access via your Amazon username and password. No more certificates or public/secret keys to manage! If youre like me, I never seem to have my own computer at hand when I need to check the status of the Amazon EC2 farm, or for that matter when I need to launch a new instance. Its a lot easier to log in with a username and password than to use those same credentials to retrieve my keys, configure Firefox (if its even on the borrowed computer) and then log in.

Then theres the new point-and-click AJAX user interface for managing Amazon EC2 resources. No more page refreshes every time something updates; and a timer refreshes management console components, such as the status of running instances, every few seconds.

The AWS community creates an amazing selection of innovative Amazon Machine Images, or AMIs. In fact, the count is now a staggering 1200 AMIs and growing! Thats quite a menu to choose fromespecially if you are a first-time user. The new Launch Instance Wizard walks you through starting your first instance; offering a short list of Linux and Windows server choices. Choose one of these AMIs, and then the wizard even suggests which ports to open in the firewall. Its smart enough to suggest that you open SSH (port 22) for Linux images, and RDP (port 3389) for Windows instances. The wizard even suggests settings that restrict Amazon EC2 access to your computer only.

And as I hinted in the opening paragraph, this is just the first in a set of Console interfaces that will provide a UI layer on top of AWS infrastructure services. Well be adding additional Amazon Web Services in the future.

The console feature list is extensive, and provides intuitive management of all these things:

  • AMI Management: browse and search AMIs, launch instances from AMIs, deregister and register AMIs
  • Instance Management: launch, reboot, terminate, get console output, RDP/SSH help, etc.
  • Security Group Management: create and delete security groups, add and remove permissions, configure firewall settings, open and close ports
  • Elastic IP Management: create and release IP Addresses, associate IPs to instances
  • Elastic Block Store: create, delete, attach and detach volumes. Take snapshots and manage snapshots.
  • Key Pair management: create and delete public/private key pairs.

If youd like to take a six minute tour of the console, I created a video (Flash format).

Finally, do you have a feature suggestion, or some other type of feedback? Feedback links are at the bottom of each page, and we welcome your input.

Mike

JumpBox – Ready To Use Applications For EC2

Jumpbox Config Page for TwikiI spoke with the good folks at JumpBox earlier this week. They told me that they are now supporting Amazon EC2 with a lineup of 12 public AMIs (Amazon Machine Images) containing pre-built and pre-configured open source applications. You can launch blogging tools, CRM tools, development tools, and lots more.

You can follow the directions in the tutorial to get started. I was able to start up a Twiki site in less than 5 minutes. Each JumpBox includes a configuration page which is accessible via HTTPS on port 3000. Using the page I set up my computer name, entered my email address for event notifications, set my time zone, entered my password, and agreed to the license agreement. After a 10 second wait for configuration, my Twiki was up and running!

26 additional packages are available to JumpBox subscribers. It is important to note that these are all single-instance applications that aregreat for workgroups and web sites with modest amounts of traffic. They are perfect for trying out new applications and for getting off the ground in a big hurry.

All in all, this is pretty powerful stuff. If you are putting a web startup together you can have your blog, bug tracker, project manager, wiki, and content management system up and running in the first hour of business.

— Jeff;

Architecting for the Cloud

Steve from MindTouch emailed me a while back about a really interesting write-up on how they moved their Wiki farm to Amazon EC2.Steve said that in the spirit of helping others do the same, we did a complete write up about it.

Mindtouch_architecture The article includes an architecture diagram, but more importantly it also drills into implementation details–complete with configuration settings that they used for HAProxy, Apache (with multi-tenant Deki), Memcache, and Lucene. MindTouch also implemented auto scaling, which is covered briefly.

If youre thinking about architecting an application for Amazon EC2, theres nothing like seeing someone else’s implementation, which you can read about here.

Oh, and one of my favorite features of MindTouchs wiki software is the Save to PDF feature. Made it easy to print out the paper.

Mike

Amazon EC2 Crosses the Atlantic

You can now launch Amazon EC2 instances in Europe!

We’ve created a new region for Europe, separate and distinct from the existing region in the United States.For fault tolerance, data separation, and stability, each EC2 region is an entity unto itself; issues within one region won’t affect the other one. This means that Amazon Machine Images (AMIs), security groups, and SSH keypairs must be created anew in each region. We’re working on tools to make it easy to move this information between regions. Also, as we learn more about how customers use multiple regions, we will add APIs to make it even easier for them to do so. There’s a new Feature Guide to Amazon EC2 Regions with a lot of helpful information, including some Frequently Asked Questions.

With the exception of support for Microsoft Windows and for Amazon DevPay (both of which will be ready before too long), every feature of EC2 is available in the new region, including Elastic Block Storage and Elastic IP Addresses.

The command line tools have been updated. The new ec2-describe-regions command lists all of the available regions, along with the endpoint URL needed to access them. A number of commands now accept the –region option. For example, this option allows ec2-run-instances to be used to launch instances in any available region.

The new region is now open for use by all interested developers. Pricing for EC2 instances, EBS storage, and I/O requests are slightly higher than in the US, reflecting differences in our operating costs in the regions.

Of course, we think that this new region will enable developers to do an even better job of serving their customers, giving them the ability to locate processing power in closer proximity to their user base while also helping them to meet EU data privacy requirements. In conjunction with the European version of S3, developers residing in or targeting markets in Europe now have powerful, local processing resources at hand.

Now that the dust has settled, I’ve got a few more related links:

— Jeff;

Oracle and AWS Webinar

Not very long ago Amazon Web Services and Oracle announced that many Oracle products are now licensed to run in the Amazon Cloud. If this interests you (and I know from emails that many of you are interested), then keep reading.

On Thursday December 11,2008 at 10:00am PST (1:00pm EST) I’m going to co-host a webinar, along with Bill Hodak from Oracle. Our agenda is simple: introduce Amazon Web Services to Oracle’s community, explain the Oracle offering to AWS customers, and mostly allow you to ask questions in real time. I expect that these three segments will last about 20 minutes each, with the most important ssegment being your questions.

Space at this event is limited, so reserve your spot today. See you there!

— Mike

Parallel Computing With MATLAB On Amazon EC2

Update (2017) –  This blog post is here for historical purposes only.   The white paper has been superseded by the Cloud Center web application from MathWorks.“

Jeff;


 

Ec2Mathworks released a whitepaper on how to run MATLAB parallel computing products – Parallel Computing Toolbox and MATLAB Distributed Computing Server on Amazon EC2. This step by step guide walks you through the steps of installation, configuration and setting up clustered environments using these licensed products from MathWorks on Amazon EC2. It shows how you can create an AMI with MATLAB products bundled in and run them in the cloud.

Whitepaper is available free on Mathworks website:

MATLAB users will learn about the key aspects of using the EC2 service from their desktop MATLAB session and using Parallel Computing Toolbox to send parallel MATLAB computations to the EC2 service.

System administrators will learn the key technical details required for setting up MATLAB Distributed Computing Server on the EC2 service, including licensing and network setup. They will also learn how to configure their users desktops to enable the use of the EC2 service for MATLAB computations.

This is the ‘Era of Tera’. We have terabytes of data to compute. Data is never going to be less. Parallel Computing models are the answer to the future. Cloud computing is making it easier for the masses.

I am very excited because this is going to open up powerful MATLAB tools to any developer for not only research but also production applications. Students might be able to do their lab exercises without a lab and impress their professors by turning in the assignments before time. Professors will be able to teach courses using MATLAB by “turning on” a switch that creates their “Instant Labs” for the duration of the course without even contacting the College IT department for resources. Enteprises might be able to crunch the complex BI data over the weekend for a monday morning meeting.

Also, Mathworks Consulting and Support is available for those who are starting up with a MATLAB project on Amazon EC2.

Do you have a cool MATLAB usecase?

— Jinesh

Update: On licensing, MathWorks is aware of the requirement and would be willing to discuss this further with people that are interested in using MDCS on EC2. Please send us an email at evangelists at amazon d0t com with your usecase.

SOASTA – Load Testing From the Cloud

I met Tom Lounibos, CEO of SOASTA, at the Palo Alto stop of the AWS Start-Up Tour. Tom gave the audience a good introduction to their CloudTest product, an on demand load testing solution which resides on and runs from Amazon EC2.

Soasta_record Tom wrote to me last week to tell me that they are now able to simulate over 500,000 users hitting a single web application. Testing at this level gives system architects the power to verify the scalability of sites, servers, applications, and networks in advance of a genuine surge in traffic.

Here are a few of their most recent success stories:

  • Hallmark tested their e-card sites in preparation for the holiday season, and are ramping up testing to over 200,000 simultaneous users using CloudTest.
  • Marvel Entertainment is doing extensive cloud testing in order to get ready for the release of the sequel to IronMan.
  • A division of Procter & Gamble is using cloud testing to get ready for new releases of their web site.

Based on this video, it looks like it is very easy to create a test, run it, and to process and analyze the results.

Soasta_clip_editor The first step is to record a new test consisting of one or more user scenarios.  Next, the raw test is edited to generalize it and to specify test data, parameters, and variable substitutions. A drag and drop test creation tool is used to create real-world test scenarios on a multi-track timeline, The system under test can be monitored in various ways while the test is run. Once completed, the test results can be viewed and analyzed.

Pricing for CloudTest starts at $1000 per hour.

— Jeff;

Big Day for Amazon EC2: Production, SLA, Windows, and 4 New Capabilities

My colleagues and I have spent the week building up anticipation for this post on Twitter. After you read this post I am sure that you will agree that the wait was worthwhile.

The hallways at Amazon have been buzzing with excitement of late. After working for years to build and to run our line of highly scalable infrastructure web services we are happy to see that developers large and small are putting them to good use.

Here’s what’s happening today:

  • Amazon EC2 is now in full production. The beta label is gone.
  • There’s now an SLA (Service Level Agreement) for EC2.
  • Microsoft Windows is now available in beta form on EC2.
  • Microsoft SQL Server is now available in beta form on EC2.
  • We plan to release an interactive AWS management console.
  • We plan to release new load balancing, automatic scaling, and cloud monitoring services.

Let’s take a look at each of these items in turn.

Production – After a two year beta period, Amazon EC2 is now ready for production. During the beta we heard and responded to an incredible amount of customer feedback, adding support for powerful features such as Availability Zones, Elastic Block Storage, Elastic IP Addresses, multiple instance types, support for the OpenSolaris and Windows operating systems, and (as of today) a Service Level Agreement. Regular EC2 accounts are allowed to run up to 20 simultaneous instances. Requests for hundreds and even thousands of additional instances are granted all the time and can be made here.

SLA – The new EC2 Service Level Agreement works at the Region level. Each EC2 Region (there’s only one right now but there will be more in the future) is divided into a number of Availability Zones. The SLA specifies that each Region will be available at least 99.95% of the time. Per the SLA, a Region is unavailable if more than one of its Availability Zones does not have external connectivity.

Ec2_windows_yeah Windows Support – Beta level support for Microsoft Windows is now available on EC2, in the form of 32 and 64 bit AMIs, with pricing starting at $0.125 per hour. Microsoft SQL Server is also available in 64 bit form. All of the powerful EC2 features listed above can be used with the new Windows instances and we’ll be adding support for DevPay in the near future.

Once launched, the Windows instances can be accessed using the Windows Remote Desktop or the rdesktop client. I’ve spent some time using Windows on EC2 and it works really well. I used the EC2 command line tools to launch a 32 bit instance, opened up an additional port in the security group, and then logged in to it using Remote Desktop.

 

We’ll be running Windows on EC2 at next week’s PDC in Los Angeles, so be sure to stop by and to say hello if you are there. Rumor has it that we’ll be giving out a really cool badge to the people who stop by our booth.

RightScale founder Thorsten von Eiken has written up a helpful post which outlines the differences between Windows and Linux with respect to launching, accessing, bundling, and using the Elastic Block Store. He also describes current and planned support for Windows in their products.

Elasticfox_gsg We’ve updated ElasticFox with a number of new features, including direct access to EBS and Elastic IP addresses from the main tab, one-click AMI bundling on Windows, better key and security group management, and the ability to directly launch Remote Desktop sessions. There’s also a brand-new (and very helpful) ElasticFox Getting Started Guide.

We are looking forward to seeing how our customers will put Windows to work. We expect to see ASP.Net sites, media transcoding, HPC (High Performance Computing), and more. I’ve talked to a number of developers who will deploy hybrid web sites using a mix of Linux and Windows servers. This really underscores the open and flexible nature of EC2.

We are also planning to offer some new capabilities in 2009 to make managing cloud-based applications even easier. As usual, we’ll start with a private beta and you can express your interest here.

Management Console – The management console will simplify the process of configuring and operating your applications in the AWS cloud. You’ll be able to get a global picture of your cloud computing environment using a point-and-click web interface.

Load Balancing – The load balancing service will allow you to balance incoming requests and traffic across multiple EC2 instances.

Automatic Scaling – The auto-scaling service will allow you to grow and shrink your usage of EC2 capacity on demand based on application requirements.

Cloud Monitoring – The cloud monitoring service will provide real time, multi-dimensional monitoring of host resources across any number of EC2 instances, with the ability to aggregate operational metrics across instances, Availability Zones, and time slots.

Amazon CTO Werner Vogels has done a very nice job of explaining why services of this type are needed to build highly reliable and highly scalable applications. His blog is a must read for those interested in cloud computing. Werner has spent so much time talking about AWS of late that I’ve asked him to be an honorary member of my team of AWS evangelists!

I think it is important to note that load balancing, automatic scaling, and cloud monitoring will each be true web services, with complete APIs for provisioning, control, and status checking. We’ll be working with a number of management tool vendors and developers to make sure that their products will support these new services on a timely basis.

So, there you go. What do you think?

— Jeff;

Amazon EBS (Elastic Block Store) – Bring Us Your Data

A few months ago I talked about our plans to offer a persistent storage feature for Amazon EC2. At that time I indicated that the service was in a limited alpha release with a small number of customers. Since then the alpha testers have been putting the service to good use and have provided us with a lot of very helpful feedback.

As of today, the Amazon Elastic Block Store (EBS) is now open and available to all EC2 users.

EBS gives you persistent, high-performance, high-availability block-level storage which you can attach to a running instance of EC2. You can format it and mount it as a file system, or you can access the raw storage directly. You can, of course, host a database on an EBS volume. In fact, Eric Hammond has already written an article, Running MySQL on Amazon EC2 with Elastic Block Store.

EBS volumes can range in size from 1 GB to 1 TB. You can mount many of them on the same instance, and even stripe (aka RAID 0) your data across them to increase performance.

The volumes can be attached to any single instance within a single EC2 availability zone. They are also automatically replicated within the zone.

During the beta you can create up to 20 EBS volumes consuming a maximum of 20 TB of space. You can make a request for additional volumes here.

You can snapshot a volume to Amazon S3 with ease, and then, if needed, create new volumes (of the same or different sizes) using the snapshot as a base. Of course, if you create a new volume with a size that doesn’t match the size of the volume where you took snapshot, you will have to resize the new file system. When you create a new volume based on an S3 snapshot, the data is loaded lazily; there’s no need to wait for the snapshot to load.

EBS usage is charged based on storage and on I/O requests. Storage costs $0.10 per GB per month and I/O requests cost $0.10 per million. Snapshot storage is charged at Amazon S3 rates. The AWS Simple Monthly Calculator has been updated to reflect the new features so that you can estimate your costs with ease:

Aws_calc_ebs_3

All of the EBS functionality can be accessed through the EC2 APIs, through the EC2 Command Line tools, through ElasticFox, and via a number of third-party tools and libraries.

The popular ElasticFox extension for Firefox has been updated with full support for EBS. You can see all of your volumes and your snapshots on a new tab:

Elasticfox_ebs

You can create volumes and attach them to running instances using simple dialog boxes:

Elasticfox_create_vol

Elasticfox_attach_vol

You can create a snapshot with a single click:

Elasticfox_snapshot

And then create a new volume from the snapshot just as easily:

Elasticfox_create_from_snapshot

Third party tool and library support is already starting to appear. In fact, I’ve created a separate post, Amazon EBS – Tool and Library Support, which I will be updating a couple of times in the next day or so as announcements are made.

Also, Amazon CTO Werner Vogels has written a really good post which includes some great insights into the architectural and philosophical considerations behind our line of storage services.

And there you go!

— Jeff;

Amazon EBS – Tool and Library Support

This is a companion post to my earlier post — Amazon EBS (Elastic Block Store) – Bring Us Your Data. In the other post you can read about the features of EBS. This post goes into more detail on the tool and library support that has been built by our community of third-party developers.

Here are some tools:

And some libraries (some of the third parties will finalize their support in a day or two):

— Jeff;

PS –  I’ll be updating this post a couple of times in the wake of the EBS launch so come back again soon.