Category: Amazon EC2*


AWS Management Console – Now with Amazon CloudWatch Support

The AWS Management Console now has complete support for Amazon CloudWatch. You can enable CloudWatch for any or all of your EC2 instances using the console and data will be available in a moment or two. You can select one or more running EC2 instances to see the CloudWatch data in graphical form. You can observe CPU utilization, disk reads, disk writes, and network traffic (both in and out). If you select more than one EC2 instance, the console will automatically display aggregated values.You can also get a larger and more detailed view of the data.

Here are some pictures of the console in action:


Among other uses, you can use the new CloudWatch support to monitor and tune your Auto Scaling rules.

The new release of the AWS Management Console also centralizes a number of actions on EC2 instances in a new Instance Actions menu:

It is flexible, colorful, and informative and you can start to use it now!

— Jeff;

Introducing Amazon Virtual Private Cloud (VPC)

Amazon Virtual Private Cloud (Amazon VPC) lets you create your own logically isolated set of Amazon EC2 instances and connect it to your existing network using an IPsec VPN connection. This new offering lets you take advantage of the low cost and flexibility of AWS while leveraging the investment you have already made in your IT infrastructure.

This cool new service is now in a limited beta and you can apply for admission here.

Heres all you need to do to get started:

  1. Create a VPC. You define your VPCs private IP address space, which can range from a /28 (16 IPs) up to a /18 (16,384 IPs). You can use any IPv4 address range, including Private Address Spaces identified in RFC 1918 and any other routable IP address block.
  2. Partition your VPCs IP address space into one or more subnets. Multiple subnets in a VPC are arranged in a star topology and enable you to create logically isolated collections of instances. You can create up to 20 Subnets per VPC (you can request more using this form). You can also use this form to request a VPC larger than a /18 or additional EC2 instances for use within your VPC.
  3. Create a customer gateway to represent the device (typically a router or a software VPN appliance) anchoring the VPN connection from your network.
  4. Create a VPN gateway to represent the AWS end of the VPN connection.
  5. Attach the VPN gateway to your VPC.
  6. Create a VPN connection between the VPN gateway and the customer gateway.
  7. Launch EC2 instances within your VPC using an enhanced form of the Amazon EC2 RunInstances API call or the ec2-run-instances command to specify the VPC and the desired subnet.

Once you have done this, all Internet-bound traffic generated by your Amazon EC2 instances within your VPC routes across the VPN connection, where it wends its way through your outbound firewall and any other network security devices under your control before exiting from your network.

IP addresses are specified using CIDR notation, where the value after the slash represents the number of bits in the routing prefix for the address. Youre currently limited to one VPC per AWS account, however, if you have a use case requiring more, let us know and well see what we can do.

Because the VPC subnets are used to isolate logically distinct functionality, weve chosen not to immediately support Amazon EC2 security groups. You can launch your own AMIs and most public AMIs, including Microsoft Windows AMIs. You cant launch Amazon DevPay AMIs just yet, though.

The Amazon EC2 instances are on your network. They can access or be accessed by other systems on the network as if they were local. As far as you are concerned, the EC2 instances are additional local network resources — there is no NAT translation. EC2 instances within a VPC do not currently have Internet-facing IP addresses.

Requirements to interoperate with our VPN implementation include:

  • Ability to establish IKE Security Association using Pre-Shared Keys (RFC 2409).
  • Ability to establish IPSec Security Associations in Tunnel mode (RFC 4301).
  • Ability to utilize the AES 128-bit encryption function (RFC 3602).
  • Ability to utilize the SHA-1 hashing function (RFC 2404).
  • Ability to utilize Diffie-Hellman Perfect Forward Secrecy in Group 2 mode (RFC 2409).
  • Ability to establish Border Gateway Protocol (BGP) peerings (RFC 4271).
  • Ability to utilize IPSec Dead Peer Detection (RFC 3706).

Optional capabilities that we recommend include:

  • Ability to adjust the Maximum Segment Size of TCP packets entering the VPN tunnel (RFC 4459).
  • Ability to reset the Dont Fragment flag on packets (RFC 791).
  • Ability to fragment IP packets prior to encryption (RFC 4459).

Weve confirmed that a variety of Cisco and Juniper hardware/software VPN configurations are compatible; devices meeting our requirements as outlined in the box at right should be compatible too. We also plan to support Software VPNs in the near future. If you want us to consider explicitly validating a device not on this list, please add your request to the Customer Gateway support thread located here.

Amazon VPC functionality is accessible via the EC2 API and command-line tools. The ec2-create-vpc command creates a VPC and the ec2-describe-vpcs command lists your collection of VPCs. There are commands to create subnets, customer gateways, VPN gateways, and VPN connections. Once all of the requisite objects have been created, the ec2-attach-vpn-gateway connects your VPC to your network and allows traffic to flow. While most organizations will likely leave the VPN connection (and VPC) up and running indefinitely, you can drop the connection, terminate the instances, and even delete the VPC if you would like.

You only pay for what you use. Pricing is on a pay-as-you-go basis. VPCs, subnets, customer gateways, and VPN gateways are free to create and to use. You simply pay an hourly charge for each VPN connection you create, and for the data transferred through those VPN connections. EC2 instances within your VPC are priced at the normal On-Demand rate. Well honor the hourly rate for any Reserved Instances that you have but during the beta we cannot guarantee that Reserved Instances will always be available for deployment within your VPC.

Imagine the many ways that you can now combine your existing on-premise static resources with dynamic resources from the Amazon VPC. You can expand your corporate network on a permanent or temporary basis. You can get resources for short-term experiments and then leave the instances running if the experiment succeeds. You can establish instances for use as part of a DR (Disaster Recovery) effort. You can even test new applications, systems, and middleware components without disturbing your existing versions.

As is the case with many of our betas, this one is launching in a single Availability Zone in the US-East region. You can use Amazon CloudWatch to monitor your instances, but you cant use Elastic IP addresses, Auto Scaling or Elastic Load Balancing just yet.

Recall that all traffic from your instances routes through the VPN connection. For now, this includes traffic to other Amazon Web Services such as EC2 instances outside of your Amazon VPC, Amazon S3, Amazon SQS, and Amazon SimpleDB. You can create Elastic Block Store (EBS) volumes and attach them to your instances. EBS volumes created within your cloud can be moved to standard EC2 instances and vice-versa.

I do want to mention a few of the things on our road map as well. First, we’re planning to let you directly reach the Internet from your VPC. In early discussions with potential users, we learned that most of them wanted to completely isolate their EC2 instances, routing all of the traffic back to their data center, so we gave this feature the highest priority. Later on, we’ll let you decide if and how you want to expose your VPC to the Internet. Second, we’re planning to let you specify the IP address of individual Amazon EC2 instances within a subnet. During this beta, Amazon EC2 instances are automatically assigned a random IP from the subnet’s designated IP address range. Third, we’re evaluating ways to allow you to filter traffic per subnet, kind of like how you might implement router ACLs. We’re already working on these items and on other additions to the core functionality we’re releasing today. If you have opinions on these items, or anything else you’d like to see in the service, e-mail us or post to the forum. This service is for you; we really need your feedback!

We think you can put Amazon VPC to immediate use and cant wait to hear about new and imaginative use cases for it. Please feel free to leave a comment on this blog or to send us some email.

— Jeff;

Lower Pricing for Amazon EC2 Reserved Instances

Our customers are putting the Amazon EC2 Reserved Instances to use in many different ways. Here are some of the usage patterns that they’ve told us about:

  • Steady State Usage – These customers have applications which require a fixed number of servers to be available at all times. Reserved Instances are advantageous for customers who are currently using their own hardware or who are using On-Demand instances full-time.
  • Low to Medium Annual Utilization – These customers have applications which run less than 100% of the time. The breakeven point can be calculated based on anticipated instance usage at the effective hourly rate. Reserved Instances offer a cost savings over On-Demand instances even at relatively low utilization rates.
  • Variable Usage – These customers have applications with unpredictable or fluctuating usage patterns. They can use a combination of Reserved and On-Demand instances to minimize their net costs. This is especially valuable when EC2 instances are frequently launched and then terminated—we minimize costs by always charging the lowest applicable price for each instance.
  • Standby Capacity – These customers use Reserved Instances as a reliable source of standby capacity with availability at a moment’s notice. The Reserved Instances are an integral part of their disaster recovery plan.

Given the many ways that our customers have already put them to use, I am happy to tell you that we’ve lowered the prices for newly purchased Amazon EC2 Reserved Instances! On a three year term, you can now get an m1.small instance for an effectively hourly rate of just $0.043 per hour (4.3 cents). The new pricing is now in effect.

Here are the new US prices (the instance prices for the EU have also been reduced):

Three Year Term
Instance Type Instance Price Hourly Charge Effective Hourly Rate*
m1.xlarge $2800.00 $0.24 $0.347
m1.large $1400.00 $0.12 $0.174
m1.small $350.00 $0.03 $0.043
c1.xlarge $2800.00 $0.24 $0.347
c1.medium $700.00 $0.06 $0.087
One Year Term
Instance Type Instance Price Hourly Charge Effective Hourly Rate*
m1.xlarge $1820.00 $0.24 $0.448
m1.large $910.00 $0.12 $0.224
m1.small $227.50 $0.03 $0.056
c1.xlarge $1820.00 $0.24 $0.448
c1.medium $455.00 $0.06 $0.112

You can purchase Reserved Instances from the AWS Management Console:

Or through ElasticFox:

— Jeff;

* – The Effective Hourly Rate is computed based on full-time (24×7) usage.

Cirrhus 9 Qualified Machine Image Webinar – August 12th

Mike from Cirrhus9 wrote to let me know that they’ll be conducting a webinar on August 12th to discuss their new Qualified Machine Image (QMI) for the Life Science and Pharmaceutical industries. The QMIs are Amazon EC2 AMIs with complete installation and operation documentation.

Currently in beta testing, the QMI is designed to help organizations meet the FDA’s 21 CFR Part 11 requirements for validation. 

The webinar is free but registration is a must.

— Jeff;

What Should Adam Do?

Its always interesting to see what people use Amazon Web Services for. This blog post is on one hand a look at an interesting example; however it is also a chance to participate in a social experiment. Adam Ginsburg from Sydney Australia set up a form to let you vote on what he should do next.

There a number of other things going on here. First of all, Adam made a YouTube video that shows how easy it is to set up Lotus Forms Turbo on Amazon EC2. That makes sense, because Adam works for IBM, and I work for AWS. So of course we were on a call about IBM software that runs as Amazon Machine Images (AMIs). We are both really excited about what on-demand pricing for IBMs server lineup offers in terms of new opportunities for both System Integrators and enterprises that want to innovate.

The conversation eventually got around to Adams experiment. He was able to set up a temporary server for just this event, and in a day or two the server can be torn down with no residual financial effects. Thats one of the beauties of AWS.

Adam posted the experiment on Twitter, and the interesting thing is that as of this moment “do nothing” is winning in the votes, and Adam swears that he is not cooking the results. (Perhaps I just did, though). You have one day to vote dont waste your chance to demonstrate the power of AWS…ummm…voting.

Update: Adam finished the experiment and took down the temporary server. You can view the results at www.youtube.com/watch?v=t2TJ8e2l4PI

Mike;

Elastic Load Balancing, Auto Scaling, and CloudWatch Resources

Here are some good resources for current and potential users of our Elastic Load Balancing, Auto Scaling, and Amazon CloudWatch features:

Version 1.8a of the popular Boto library for AWS now supports all three of the new features. Written in Python, Boto provides access to Amazon EC2, Amazon S3, Amazon SQS, Amazon Mechanical Turk, Amazon SimpleDB, and Amazon CloudFront. The Elastician Blog has some more info.

 

The Elastician Blog also has a good article with a complete example of how to use CloudWatch from Boto. After creating the connection object, one call initiates the monitoring operation and two other calls provide access to the collected statistics.

 

The Paglo monitoring system can now make use of the statistics collected by CloudWatch. You will need to install the open source Paglo Crawler on your EC2 instances. More info on Paglo can be found here.

 

The IT Architects at The Server Labs have put together some great blog posts. The first one, Setting up a load-balanced Oracle Weblogic cluster in Amazon EC2, contains all of the information needed to set up a two node cluster. The second one, Full Weblogic Load-Balancing in EC2 with Amazon ELB, shows how to use the Elastic Load Balancer to front a pair of Apache servers which, in turn, direct traffic to a three node Weblogic cluster to increase scalability and availability.

 

Speaking of availability and durability, you should definitely check out the DZone reference card on the topic. The card provides a detailed yet concise introduction to the two topics in just 6 pages. Topics covered include horizontal scalability, vertical scalability, high availability, measurement, analysis, load balancing, application caching, web caching, clustering, redundancy, fault detection, and fault tolerance.

 

Author and blogger Ramesh Rajamani wrote a detailed paper on the topic of Dynamically Scaling Web Applications in Amazon EC2. Although the paper predates the release of the Elastic Load Balancer and Auto Scaling, the approach to scaling is still valid. Ramesh shows how to use Nginx and Nagios to build a scalable cluster.

 

The Serk Tools Blog has a post on Amazon Elastic Load Balancer Setup. The post includes an architectural review of the Elastic Load Balancer service, detailed directions to create an Elastic Load Balancer instance, information about how to set up a CNAME record in your DNS server, and directions on how to set up health checks.

 

Arfon Smith wrote a blog post detailing his experience moving the Galaxy Zoo from HAProxy to Elastic Load Balancing. He notes that it took him just 15 minutes to make the switch and that he’s now saving $150 per month.

 

Update: After I wrote this post, two more good resources were brought to my attention!

Shlomo Swidler of of MyDrifts.com wrote to tell me about his post. He covers the two-level elasticity of Elastic Load Balancing and describes some testing strategies. The first level of elasticity is provided by DNS when it maps the CNAME of an Elastic Load Balancer instance to the actual endpoint of the instance. Shlomo correctly points out that this allows inbound network traffic to scale. The second level is provided by the Elastic Load Balancer itself as it distributes traffic across multiple EC2 instances. The latter sections of the post provide a testing strategy for a system powered by one or more Elastic Load Balancer instances.

 

The Typica AWS library for Java has included CloudWatch support for a few months. You can read this post to learn more about enabling and fetching CloudWatch metrics through Typica.

 

I hope you find these resources to be helpful!

— Jeff;

Amazon Elastic MapReduce Now Available in Europe

Earlier this year I wrote about Amazon Elastic MapReduce and the ways in which it can be used to process large data sets on a cluster of processors. Since the announcement, our customers have wholeheartedly embraced the service and have been doing some very impressive work with it (more on this in a moment).

Today I am pleased to announce Amazon Elastic MapReduce job flows can now be run in our European region. You can launch jobs in Europe by simply choosing the new region from the menu. The jobs will run on EC2 instances in Europe and usage will be billed at those rates.

Because the input and output locations for Elastic MapReduce jobs are specified in terms of URLs to S3 buckets, you can process data from US-hosted buckets in Europe, storing the results in Europe or in the US. Since this is an internet data transfer, the usual EC2 and S3 bandwidth charges will apply.

Our customers are doing some interesting things with Elastic MapReduce.

At the recent Hadoop Summit, online shopping site ExtraBux described their multi-stage processing pipeline. The pipeline is fed with data supplied by their merchant partners. This data is preprocessed on some EC2 instances and then stored on a collection of Elastic Block Store volumes. The first MapReduce step processes this data into a common format and stores it in HDFS form for further processing. Additional processing steps transform the data and product images into final form for presentation to online shoppers. You can learn more about this work in Jinesh Varia’s Hadoop Summit Presentation.

Online dating site eHarmony is also making good use of Elastic MapReduce, processing tens of gigabytes of data representing hundreds of millions of users, each with several hundred attributes to be matched. According to an article on SearchCloudComputing.com, they are doing this work for $1,200 per month, a considerable savings from the $5,000 per month that they estimated it would cost them to do it internally.

We’ve added some articles to our Resource Center to help you to use Elastic MapReduce in your own applications. Here’s what we have so far:

You should also check out AWS Evangelist Jinesh Varia in this video from the Hadoop Summit:

— Jeff;

PS – If you have a lot of data that you would like to process on Elastic MapReduce, don’t forget to check out the new AWS Import/Export service. You can send your physical media to us and we’ll take care of loading it into Amazon S3 for you.

Scaling to the Stars

Recently I blogged about The Server Labs, a consultancy that specializes in high-performance computing including on Amazon Web Services.

Heres another story that I found fascinating: nominally it is about how The Server Labs uses Amazon Web Services as a scale-out solution that also implements Oracle databases; however its really about space exploration (or should I say nebula computing). It began with an email asking whether there would be a problem running up to 1,000 Amazon EC2 High-CPU Extra-Large instances.

The Server Labs is a software development/consulting group based in Spain and the UK that works closely with the European Space Agency, and they needed to prove the scalability of an application that they helped build for ESA’s Gaia project. In addition to the instances, they also requested 2 large and 3 X-Large instances to host Oracle databases that coordinate the work being performed by the high-CPU instances.

Gaias goal is to make the largest, most precise three-dimensional map of our Galaxy by surveying an unprecedented number of stars – more than one billion. This, by the way, is less than 1% of all stars! The plan is to launch a mission in 2011, collect data until 2017; and then publish a completed catalog no later than 2019.

I had the opportunity to see a PowerPoint deck created and presented by The Server Labs founder, Paul Parsons, and their software architect, Alfonso Olias, who is currently assigned to this project.

The deck explained that the expected number of samples in Gaia is 1 billion stars x 80 observations x 10 readouts, which is approximately equal to 1 x 1012 samplesor as much as 42 GB per day transferred back to Earth. Theres a slide in the deck that says Put another way, if it took 1 millisecond to process one image, the processing time for just one pass through the data on a single processor) would take 30 years.

As the spacecraft travels, it will continuously scan the sky in 0.7 degree arcs, sending the data back to Earth. Some involved algorithms will come into play in order to process the data; and the result is a fairly complex computing architecture that is linked to an Oracle database. Scheduling the cluster of computational servers is not quite so complicated, and is based on a scheduler that is focused on keeping each machine as busy as possible.

However the amount of data to process is not steadyit will increase over time. Which means that infrastructure needs will also vary over time. And of course idle computing capacity is deadly to a budget.

The opportunity to solve large computational problems usually turns to grid computing. No difference this time either except that as mentioned above, the required size of the grid is not constant. Because Amazon Web Services is on-demand, its possible to apply just enough computational resources to the problem at any given time.

In their test, The Server Labs set up an Oracle database using an AWS Large Instance running a pre-defined public AMI. Then they mounted 5 EBS volumes of 100 GB each, and mounted them to the instance.

Then they created Amazon Machine Images (AMIs) to run the actual analysis software. These images were based on large instances and included Java, Tomcat, the AGIS software and an rc.local script to self-configure an instance when its launched.

The requirements break down as follows:

To process 5 years of data for 2 million stars, they will need to run 24 iterations of 100 minutes each, which works out to 40 hours running a grid of 20 Amazon EC2 instances. A secondary update has to be run once and requires 30 minutes per run, or 5 hours running a grid of 20 EC2 instances.

For the full 1 billion star project numbers extrapolate out more or less as follows: They calculated that they will analyze 100 million primary stars, plus 6 years of data, which will require a total of 16,200 hours of a 20-node EC2 cluster. Thats an estimated total computing cost of 344,000 Euros. By comparison, an in-house solution would cost roughly 720,000 EUR (at todays prices) which doesnt include electricity or storage or sys-admin costs. (Storage alone would be an additional 100,000 EUR.)

Its really exciting to see the Cloud used in this manner; especially when you realize that an entire set of problem solutions that were beyond economic possibility before the Cloud became a reality.

Mike

Webinar: How to Create Secure Test and Dev Environments on the Cloud

Amazon Web Services, CohesiveFT, and RightScale will participate in a webinar titled “How to Create Secure Test and Dev Environments on the Cloud.”

Along with Michael Crandell and Edward Goldberg of RightScale, Simone Brunozzi of Amazon Web Services and Patrick Kerpan of CohesiveFT will show you how you can save time and money by running your entire testing application testing infrastructure in the cloud. They will discuss creation of an agile approach to rapid prototyping, creation of a test and development environment which replicates the final deployment environment, and will also show how to build a secure VPN environment.

The webinar is free but registration is required.

— Jeff;

Setting up a Load-Balanced Oracle Weblogic Cluster in Amazon EC2

Update (January 29, 2016) – This blog post is six years old and many of the original links are now out of date. Take a look at the newer (albeit not by much) Oracle AMIs page for some alternatives.

— Jeff;


Oracle recently released several middleware Amazon Machine Images (AMIs) to the community. I want to point out a detailed blog entry by Paul Parsons from The Server Labs that describes how to run a Weblogic Server on Amazon EC2, incorporating the load balancing feature inside the Monitoring, Auto Scaling and Elastic Load Balancing Service for Amazon EC2.

The following paragraphs are straight out of Pauls full blog post, which of course you should read in its entirety if this topic is relevant to your interests.

Oracle recently made available a set of AMI images suitable for use with the Amazon EC2 cloud computing platform. I found the two images (32-bit and 64-bit) that contain Weblogic (along with Oracle Enterprise Linux 5 and JRockit) the most interesting of the lot. This article will explain how to set up a basic two-node Weblogic cluster using the 32-bit Weblogic image provided by Oracle with an Amazon Elastic Load Balancer (ELB). In future articles, I will demonstrate how to set up a more complicated cluster with Apache Web Server instances balancing the load between many weblogic cluster members.

You can set up a Weblogic cluster in EC2 in very little time which makes it great for testing complicated Weblogic setups without having to mess around on your local machine or trying to scrape together the necessary hardware. This type of configuration would also be suitable for deploying a production application, though youd have to check the licensing implications with Oracle if you wanted to do this.

Note that this article assumes a basic level of familiarity with using Amazon web services.

Mike