Category: Amazon EC2*


New Amazon EC2 Feature: Resource Tagging

It is really easy to start up that first Amazon EC2 instance, and then another, and another as you find more and more ways to put it to use. It is really easy to create some EBS volumes, attach them to your instances, and to store lots and lots of data on them. The same goes for other EC2 resources such as security groups, and EBS snapshots.

As your usage starts to grow from one instance and one application to many instances spanning multiple applications, it can be difficult to track which instances are assigned to which application, which EBS volumes store what data, and so forth.

We’ve just released a very powerful tagging feature to allow you to tag your EC2 resources (and also certain shared resources) using up to ten key-value pairs per resource. Each tag consists of a key (up to 128 characters) and a value (up to 256 characters). The tags are stored in the AWS cloud as part of your AWS account, and are private to the account.

You can tag the following types of resources: EC2 instances, Amazon Machine Images (AMIs), EBS volumes, EBS snapshots, and Amazon VPC resources such as VPCs, subnets, connections, and gateways. You can tag existing resources and you can tag new resources right after you create them.

You can manipulate your tags using three new API calls:

CreateTags allows you to tag one or more EC2 resources with one or more tags.

DescribeTags gives you the tags associated with one or more resources. The returned tags can be filtered by resource identifier, resource type, key, or value. You can, for example, retrieve all of the tags for a given resource, or you can retrieve all of the resources (regardless of type) with a given tag.

DeleteTags allows you to delete a set of tags from a set of resources.

The existing EC2 “Describe” APIs (DescribeInstances, DescribeVolumes, and so forth) return the tags associated with each of the resources in the response.

You can also manipulate them using the EC2 command-line tools (ec2-create-tags, ec2-describe-tags, and ec2-delete-tags) and from the AWS Management Console.

You can create and delete tags using the AWS Management Console. You can view the tags associated with any resource, you can use a tag as a column in any resource list, and you can filter any view by tag.

Here’s a tour of the new console features. I’ve tagged EC2 instances, but you can also tag many other types of resources.

You can tag new instances as part of the Request Instances Wizard:

You can also tag existing instances (I’m sure that you’ve been a part of at least one “Project Phoenix” in your career):

You can select an instance to see its tags:

The tag names can be used as columns, allowing you to hide and show them, and to see the values:

In the following illustration I have created tags named Use Case and Project on some EC2 instances:

As you can see, a column that represents a tag also includes a filter control:

Clicking the control gives you the ability to filter rows (instances in this case) by the value of the tag:

You can also use the filtering menu to find the items that don’t have a particular tag, or that have a tag with an empty value. You can use this to locate resources that are allocated but not assigned to a particular role, product, or user, for example.

Here’s a list of instances that I have tagged with a Use Case of Production:

Read more about the tagging in the newest version of the EC2 User Guide.

Next feature: Idempotent Instance Creation.

— Jeff;

New Amazon EC2 Feature: Idempotent Instance Creation

The Amazon EC2 API includes functions which create resources such as instances, disk volumes and snapshots, IP addresses, and key pairs.

Some of these functions create the resources in a synchronous fashion and you can determine the success or failure of the request by examining the value returned by the call.

Other functions work in an asynchronous fashion. Making the call initiates an action that may take a fairly long time (seconds or minutes) to complete.  When the call returns you cannot know if the request has succeeded or not. Timeouts and connection errors can further muddy the water; you don’t want to unnecessarily retry a request if there’s an associated cost for the resource. You don’t want to create two EC2 instances when you only needed one.

To provide you with better control in this situation, we’ve just a released a somewhat esoteric (yet very useful) feature called idempotent instance creation.

Performing an idempotent operation more than once yields the same result as applying it just once. Washing your dog is idempotent (you always end up with a clean dog); feeding your dog is not (your dog will get fat).

The EC2 RunInstances function now supports idempotency. If you are launching EC2 instances as part of a higher level process, this feature should help you to build management and control applications that are more robust.

To call RunInstances in an idempotent fashion, you need to create a client token. A client token is a case-sensitive string of up to 64 ASCII characters. You should use a unique client token for each new instance.

Once you have a properly formed client token, you simply pass it along as an additional parameter to RunInstances. The function will ignore the second and subsequent requests that have the same token. You must use the same set of parameters each time you call the function. If you don’t you will get an IdempotentParameterMismatch error.

Read more about idempotency in the newest version of the EC2 Developer Guide.

Next feature: Filtering.

— Jeff;

New Amazon EC2 Feature: Bring Your Own Keypair

You can now import your own RSA keypair (or the public half, to be precise) for use with your Amazon EC2 instances.

Why would you want to do this? Here are a couple of reasons:

  1. Trust – By importing your own keypair you can ensure that you have complete control over your keys.
  2. Security -You can be confident that your private key has never been transmitted over the wire.
  3. Management of Multiple Regions – You can use the same public key across multiple AWS Regions.

You can upload RSA keys (which can be 1024, 2048, or 4096 bits long) in a variety of formats including OpenSSH public key format, Base64 encoded DER format, or the SSH public key file format specified in RFC 4716. The ssh-keygen tool (part of the standard OpenSSH installation) is a handy way to create keys.

Read more about the import key feature in the newest version of the EC2 User Guide.

Update:Paul Maunder documented the process of uploading the same keypair to multiple EC2 regions. Thanks, Paul!

— Jeff;

AWS For High Performance Cloud Computing – NASA, MATLAB

It is great to see our customers putting EC2’s new Cluster Compute instance type to use in High Performance Computing (HPC) scenarios. Here are two example applications:

MathWorks / MATLAB

The MATLAB team at MathWorks tested performance scaling of the backslash (“\”) matrix division operator to solve for x in the equation A*x = b. In their testing, matrix A occupies far more memory (290 GB) than is available in a single high-end desktop machinetypically a quad core processor with 4-8 GB of RAM, supplying approximately 20 Gigaflops.

Therefore, they spread the calculation across machines. In order to solve linear systems of equations they need to be able to access all of the elements of the array even when the array is spread across multiple machines. This problem requires significant amounts of network communication, memory access, and CPU power. They scaled up to a cluster in EC2, giving them the ability to work with larger arrays and to perform calculations at up to 1.3 Teraflops, a 60X improvement. They were able to do this without making any changes to the application code.

Here’s a graph showing the near-linear scalability of an EC2 cluster across a range of matrix sizes with corresponding increases in cluster size for MATLAB’s parallel backslash operator:

Each Cluster Compute instance runs 8 workers (one per processor core on 8 cores per instance). Each doubling of the worker count corresponds to a doubling of the number of Cluster Computer instances used (scaling from 1 up to 32 instances). They saw near-linear overall throughput (measured in Gigaflops on the y axis) while increasing the matrix size (the x axis) as they successively doubled the number of instances.

NASA JPL

A team at NASA’s Jet Propulsion Laboratory developed the ATHLETE robot. Each year they put the robot through autonomous field tests as part of the D-RATS (Desert Research and Training Studies) along with autonomous robots from other NASA centers. The operators rely on high-resolution satellite imagery for situational awareness while driving the robots. JPL engineers recently developed and deployed an application designed to streamline the processing of large (giga-pixel) images by leveraging the massively parallel nature of the workflow. The application is built on Polyphony, a versatile and modular workflow framework based on Amazon SQS and Eclipse Equinox. In the past, JPL has used Polyphony to validate the utility of cloud computing for processing hundreds of thousands of small images in an EC2-based compute environment. JPLers have now adopted the cluster compute environments for processing of very large monolithic images. Recently, JPLers processed a 3.2 giga-pixel image of the field site (provided courtesy of USGS) in less than two hours on a cluster of 30 Cluster Compute Instances. This demonstrates a significant improvement (an order of magnitude) over previous implementations, on non-HPC environments.

We’re happy to see MathWorks and JPL deploying Cluster Compute Instances with great results. It’s also exciting to see other customers scaling up to 128-node (1024 core) clusters with full bisection bandwidth. I’ll be writing up more of these stories in the near future, so stay tuned. If you have a story of your own, drop me an email or leave a comment.

— Jeff;

Introducing The Amazon Linux AMI

Yes, you read that right. We now have a Linux AMI tuned for AWS!

Many of our customers have asked us for a simple starting point for launching their Linux applications inside of Amazon EC2 that is easy to use, regularly maintained, and optimized for the Amazon EC2 environment. Starting today, customers can use Amazon Linux AMI to meet these needs.  This just adds to the great selection of AMI options in Amazon EC2 that range from free to paid, giving you access to the operating systems and environments you need.

Available in 32 and 64 bit form in all of the AWS Regions, Amazon Linux starts out as lean and mean as possible; no unnecessary applications or services are running. You can add more packages as needed, and you can do so very quickly and easily from a package repository that we host in Amazon S3.

The AWS command-line tools and libraries are pre-installed and ready to use. We’ve also integrated Ubuntu’s CloudInit to simplify the process of customizing each instance after it boots up. You can use CloudInit to set a default locale, set the hostname, generate and set up SSH private keys, and to set up mount points. You can also run custom commands and scripts on initial startup or on each reboot, as desired.

The Amazon Linux AMI can be booted from the AWS Management Console‘s Request Instances page at the usual charge for Linux/UNIX instances. This is a supported AMI; customers who use AWS Premium Support will be able to ask for help with installation and usage problems. Of course, everyone can use the forums to ask for help or to report bugs.

We will provide patches and security updates as needed. We also update the Amazon Linux AMI on a regular basis, and we’ll create a new set of AMIs each time we do so.

If you’ve used other Linux AMIs in the past, this one should hold few surprises. Nevertheless, here are a few things to keep in mind:

  1. Log in as ec2-user rather than as root.
  2. For S3-backed instances the first ephemeral volume is mounted at /media/ephemeral0.
  3. Complete release notes are available in file /etc/image-release-notes.
  4. The system is running kernel 2.6.34.

Read the Amazon Linux AMI User Guide [PDF] to learn more.

Update 1:

You can find the current AMI ID’s for each region on the Amazon Linux page. Once you have the ID you can search for it using the Community AMIs tab in the AWS Management Console‘s EC2 Request Instances Wizard:

You can do the same search using the Images tab in Elastic Fox:

Update 2:

The source code is available for reference purposes. Open up the user guide and search for the section labeled Accessing Source Packages for Reference for full information on how to download it to your instance.

— Jeff;

New Amazon EC2 Micro Instances – New, Low Cost Option for Low Throughput Applications

I cant tell you how many of you have told me youd like to run smaller applications at lower cost on EC2. These applications are typically low traffic/low throughputweb applications, web site hosting, various types of periodic cron jobs and the like.

Im happy to say we have now built an instance type exactly for these purposes, called Micro instances, starting at $0.02 (two cents) per hour for Linux/Unix and $0.03 (three cents) per hour for Windows.

Micro Instances (t1.micro) provide a small amount of consistent CPU resources and allow you to burst CPU capacity when additional cycles are available. They are available now in all Regions. You can buy Reserved Micro Instances and you can acquire Micro Instances on the Spot Market. Interestingly enough, they are available in both 32 and 64 bit flavors, both with 613 MB of RAM. The Micro Instances have no local, ephemeral storage, so you’ll need to Boot from EBS.

CloudWatch can be used to watch the level of CPU utilization to understand when the available CPU bursting has been used within a given time period. If your instance’s CPU utilization is approaching 100% then you may want to scale (using Auto Scaling) to additional Micro instances or to a larger instance type. In fact, at this low a price you could run CloudWatch configured for Auto Scaling with two Micro instances behind an Elastic Load Balancer for just under the price of one CloudWatch-monitored Standard Small instance.

While designed to host web applications and web sites that don’t receive all that much traffic (generally tens of requests per minute, depending on how much CPU time is needed to process the request), I’m pretty sure that you’ll be able to put this new instance type to use in some interesting ways. Here are some of my thoughts:

  1. DNS servers, load balancers, proxies, and similar services that handle a relatively low volume of requests.
  2. Lightweight cron-driven tasks such as monitoring, health checks, or data updates.
  3. Hands-on training and other classroom use.

Feel free to post your ideas (and your other thoughts) in the comments.

Update: The AWS Simple Monthly Calculator now includes the Micro instances. The calculation at right illustrates the costs for a three year Reserved Instance running Linux/Unix full time.

 

— Jeff;

AWS Management Console Support for the Amazon Virtual Private Cloud

The AWS Management Console now supports the Amazon Virtual Private Cloud (VPC). You can now create and manage a VPC and all of the associated resources including subnets, DHCP Options Sets, Customer Gateways, VPN Gateways and the all-important VPN Connection from the comfort of your browser.

Put it all together and you can create a secure, seamless bridge between your existing IT infrastructure and the AWS cloud in a matter of minutes. You’ll need to get some important network addressing information from your network administrator beforehand, and you’ll will need their help to install a configuration file for your customer gateway.

Here are some key VPC terms that you should know before you should read the rest of this post (these were lifted from the Amazon VPC Getting Started Guide):

VPC – An Amazon VPC is an isolated portion of the AWS cloud populated by infrastructure, platform, and application services that share common security and interconnection. You define a VPC’s address space, security policies, and network connectivity.

Subnet – A segment of a VPC’s IP address range that Amazon EC2 instances can be attached to.

VPN Connection – A connection between your VPC and data center, home network, or co-location facility. A VPN connection has two endpoints: a Customer Gateway and a VPN Gateway.

Customer Gateway – Your side of a VPN connection that maintains connectivity.

VPN Gateway – The Amazon side of a VPN connection that maintains connectivity.

Let’s take a tour through the new VPC support in the console. As usual, it starts out with a new tab in the console’s menu bar:

The first step is to create a VPC by specifying its IP address range using CIDR notation. I’ll create a “/16” to allow up to 65536 instances (the actual number will be slightly less because VPC reserves a few IP addresses in each subnet) in my VPC:

The next step is to create one or more subnets within the IP address range of the VPC. I’ll create a pair, each one covering half of the overall IP address range of my VPC:


The console shows all of the subnets and the number of available IP addresses in each one:

You can choose to create a DHCP Option Set for additional control of domain names, IP addresses, NTP servers, and NetBIOS options. In many cases the default option set will suffice.

And the next step is to create a Customer Gateway to represent the VPN device on the existing network (be sure to use the BGP ASN and IP Address of your own network):

We’re almost there! The next step is to create a VPN Gateway (to represent the VPN device on the AWS cloud) and to attach it to the VPC:


The VPC Console Dashboard displays the status of the key elements of the VPC:

With both ends of the connection ready, the next step is to make the connection between your existing network and the AWS cloud:

This step (as well as some of the others) can take a minute or two to complete.

Now it is time to download the configuration information for the customer gateway.


The configuration information is provided as a text file suitable for use with the specified type of customer gateway:

Once the configuration information has been loaded into the customer gateway, the VPN tunnel can be established and it will be possible to make connections from within the existing network to the newly launched EC2 instances.

I think that you’ll agree that this new feature really simplifies the process of setting up a VPC, making it accessible to just about any AWS user. What do you think?

–Jeff;

Amazon EC2 Price Reduction

We’re always looking for ways to make AWS an even better value for our customers. If you’ve been reading this blog for an extended period of time you know that we reduce prices on our services from time to time.

Effective September 1, 2010, we’ve reduced the On-Demand and Reserved Instance prices on the m2.2xlarge (High-Memory Double Extra Large) and the m2.4xlarge (High-Memory Quadruple Extra Large) by up to 19%.  If you have existing Reserved Instances your hourly usage rate will automatically be lowered to the new usage rate and your estimated bill will reflect these changes later this month.  As an example, the hourly cost for an m2.4xlarge instance running Linux/Unix in the us-east Region from $2.40 to $2.00. This price reduction means you can now run database, memcached, and other memory-intensive workloads at substantial savings. Here’s the full EC2 price list.

As a reminder, there are many different ways to optimize your costs. When compared to On-Demand instances, Reserved Instances enable you to reduce your overall instance costs by up to 56%.  You pay a low, one-time fee to reserve an instance for a one or three year period. You can then run that instance whenever you want, at a greatly reduced hourly rate.

For background processing and other jobs where you have flexibility in when they run, you can also use Spot Instances by placing a bid for unused capacity. You job will run as long as your bid is higher than the current spot price.

— Jeff;

 

 

Happy 4th Birthday Amazon EC2

I almost missed a really important anniversary! Yesterday marked Amazon EC2‘s fourth birthday. Here are some of the ways that EC2 has grown and changed in the last four years:

Category 2006 2010
Regions One Four
Availability Zones One Ten Availability Zones
Instance Types One Nine
Pricing Models One Three
Storage Ephemeral Storage Ephemeral Storage
Elastic Block Store
Operating Systems Linux Linux, Windows, OpenSolaris
Management Tools Command-Line Tools Command-Line Tools
AWS Management Console
Third-Party Tools
Ancillary Services Elastic Load Balancing, Auto Scaling, CloudWatch
High Performance Computing Elastic Map Reduce, Cluster Compute Instances

We’ve done quite a bit, but we’re not resting, not for a minute. We have a lot of open positions on the AWS team, including a really interesting developer position within the EC2 team. This developer will focus on EC2’s dynamic market pricing features. In addition to experience with Ruby, Perl, Java, C, or C++, candidates should have some experience building large-scale distributed systems and an interest in operational scheduling, optimization, and constraint satisfaction. You can read more here and you can send your resume directly to amazon-ec2-spot-jobs@amazon.com.

While I am on the subject of anniversaries, eight years ago this month I abandoned my full-time consulting practice to take a development position with the Amazon Associates Team, with the agreement that I could spend some of my time helping out with the effort to create and market the E-Commerce Service (which has since become the Product Advertising API). A few months in, I was asked if I would mind speaking at a conference. I guess I did ok, because they asked me to do another one, and before too long they invited me to apply for the position of Web Services Evangelist. I took on that title in the spring of 2003 and have been spreading the word about our web service efforts ever since. All things considered, this is a really awesome place to work. Day after day, week after week, things get more and more exciting around here. The pace is quick and I do my best to keep up. We do our best to understand and to meet the needs of our customers with regard to features, reliability, scale, business models, and price. I get to work with and to learn from a huge number of world-class intellects. If this sounds like the kind of place for you, check out our list of open jobs and apply today!

— Jeff;

Use Your Own Kernel with Amazon EC2

You can now use the Linux kernel of your choice when you boot up an Amazon EC2 instance. 

We have created a set of AKIs (Amazon Kernel Images) which contain the PV-Grub loader. This loader simply chain-boots the kernel provided in the associated AMI (Amazon Machine Image). Net-net, your instance ends up running the kernel in the AMI instead of the kernel specified in the boot process.

You need to install an “EC2 compatible” kernel and create an initrd (initial RAM disk) as part of your AMI. You also need to create a menu (/boot/grub/menu.lst) for the Grub boot loader. Once you’ve done this you can create the AMI and then launch instances by using one of the PV-Grub “kernels” as described above. You may find this document to be helpful if you want to learn more about the Linux boot process.

To be compatible with EC2, a Linux kernel must support Xen’s pv_ops (paravirtual ops) infrastructure with XSAVE disabled or the Xen 3.0.2 interface. The following kernels have been tested and/or have vendor support:

  • Fedora 8-12 Xen kernels
  • SLES/openSUSE 10x, 11.0, and 11.1 Xen kernels
  • SLES/openSUSE 11.x EC2 Variant
  • Ubuntu EC2 Variant
  • RHEL 5.x
  • CentOS 5.x

Other kernels may not start reliably within EC2. We’re working with the providers of popular AMIs to make sure that they will start to use PV-Grub in the near future.

You can read more about this in our “Enabling User Provided Kernels in Amazon EC2” document.

— Jeff;

PS – You could (if you are sufficiently adept) use this facility to launch an operating system that we don’t support directly (e.g. FreeBSD). If you manage to do this, please feel free to let me know.