Category: Compute*


New Asia Pacific (Sydney) Region in Australia – EC2, DynamoDB, S3, and Much More

It is time to expand the AWS footprint once again, with a new Region in Sydney, Australia. AWS customers in Australia can now enjoy fast, low-latency access to the suite of AWS infrastructure services.

New Region
The new Sydney Region supports the following AWS services:

We also have an edge location for Route 53 and CloudFront in Sydney.

This is our ninth Region; see the AWS Global Infrastructure Map for more information. You can see the full list in the Region menu of the AWS Management Console:

Customers
Over 10,000 organizations in Australia and New Zealand are already making use of AWS. Here’s a very small sample:

  • The Commonwealth Bank of Australia runs customer-facing web applications on AWS as part of a cloud strategy that has been underway for the past five years. The seamless scaling enabled by AWS has allowed their IT department to focus on innovation.
  • Brandscreen, a fast-growing Australian start-up,has developed a real-time advertising trading platform for the media industry.They use Elastic MapReduce to process vast amounts of data to test out machine learning algorithms. They store well over 1 PB of data in Amazon S3 and add another 10 TB every day.
  • MYOB uses AWS to host the MYOB Atlas, a simple website builder that enables businesses to be online within 15 minutes.  They currently have more than 40,000 small and medium-sized businesses using Atlas on the AWS cloud.
  • Halfbrick Studios hosts the highly acclaimed Fruit Ninja game on AWS. They use DynamoDB and multiple Availability Zones to host tens of millions of regular players.

AWS Partner Network
A number members of the AWS Partner Network have been preparing for the launch of the new Region. Here’s a sampling (send me email with launch day updates):

  • Canonical is working to bring the official Ubuntu AMIs to our new Region. The latest supported images for Ubuntu Server 10.04 LTS, 11.10, 12.04 LTS and 12.10 have been migrated over. Daily images have been turned on for the new region. The Amazon Quickstart list is also populated with the proper image ID’s.
  • The enStratus cloud management platform is available in the new Region.
  • RightScale‘s cloud management platform (see my interview with RightScale CEO Michael Crandell to learn more) is available in the new Region.
  • Acquia provides hosted Drupal (again, see my interview with Acquia’s Tom Erickson to learn more) to over 2,400 customers. They are working to ensure that their service will be available to customers in the new Region.
  • ESRI is the leading provider of Geographic Information Systems, with over one million users in more than 350,000 organizations. They are making their ArcGIS platform available in the new Region.
  • The CloudBerry Labs S3 Explorer supports the new region.
  • MetaCDN provides global cloud-based content delivery, video encoding and streaming services. They are working to ensure that their video encoding, persistent storage and delivery services will be available to customers in the new Region.
  • Bulletproof launched Bulletproof Managed AWS in October, removing a significant barrier to AWS entry for enterprise and government customers that require enterprise management. Bulletproof’s Managed AWS includes 24/7 proactive incident response, AWS Enterprise Support and 24/7 application and database management.

We already have a vibrant partner ecosystem in the region. Local Systems Integrators include ASG, Bulletproof Networks, Fronde, Industrie IT, The Frame Group, Melbourne IT, SMS IT and Sourced Group.

On the Ground
In order to serve enterprises, government agencies, academic institutions, small-to-mid size companies, startups, and developers, we now have offices in Sydney, Melbourne, and Perth. We will be adding a local technical support operation in 2013 as part of our global network of support centers, all accessible through AWS Support.

Listen to Andy
AWS Senior Vice President Andy Jassy will be speaking at our Customer Appreciation Day (November 13, 2012). You can register for and attend the live event if you are in Sydney, or you can watch the live stream from anywhere in the world.

— Jeff;

The Amazon EC2 Spotathon

Stephen Elliott Talks About EC2 Spot Instances

If you have used EC2 Spot Instances in a unique way, you need to enter the Amazon EC2 Spotathon. This is an open-ended coding challenge designed to allow you to show off what you have done (and are doing) with Spot Instances.

We’re looking for entrants from a wide variety of industries including (but most definitely not limited to) life sciences, financial services, media, consumer apps, business analytics, gaming, and web analytics.

We want to hear about what you application does, how it incorporates Spot Instances, how much money you have saved, the performance benefits you’ve measured, and the computational scale that you’ve been able to achieve.

The Grand Prize is $2,500 in AWS Credit (which you can use to run even more Spot Instances) and the Runner-Up Prize is another $1,000 in AWS Credit.

I’m really looking forward to seeing what you come up with. The submission deadline is November 16, 2012 so enter today!

— Jeff;

Raising the Roof – More Provisioned IOPS for EBS

When we launched Provisioned IOPS for EBS earlier this year (blog post, EBS home page) we promised even more IOPS (Input/Output Operations Per second) over time.

Time moves fast on the AWS team, and I’m happy to announce that you can now provision up to 2,000 IOPS per EBS volume (double the previous limit of 1000). This is an order of magnitude more IOPS than you can expect from a high-end 15,000 RPM disk drive. If you’ve been RAIDing multiple volumes together for even higher performance, you may be able to achieve the desired level of performance with far fewer volumes.

To recap, Provisioned IOPS volumes allows you to set the level of throughput that you need.  EBS will consistently deliver the desired performance over the lifetime of the volume.  For even higher performance, you can stripe multiple Provisioned IOPS volumes together, giving you the ability to deliver thousands of IOPS per logical volume to your EC2-powered application. These volumes deliver consistent performance and are well-suited to database storage, transaction processing, and other heavy random I/O loads.  For maximum performance and to fully utilize the IOPS provisioned on an EBS volume, you can request the launch of EBS Optimized EC2 instances. An EBS-Optimized instance is provisioned with dedicated throughput to EBS. The m1.large, m1.xlarge, and m2.4xlarge instance types are currently available as EBS-Optimized instances. m1.large instances can transfer data to and from EBS at a rate of 500 Mbit/second; m1.xlarge and m2.4xlarge instances can transfer data at a rate of 1000 Mbit/second. This is additional throughput, and doesn’t affect other general purpose network throughput already available on the instance.

Our friends over at SAP have been using EBS Provisioned IOPS to host SAP HANA (a realtime data platform) on AWS. I was recently chatting with some of our colleagues there and they told me that they have used Provisioned IOPS to achieve over 20,000 IOPS in some of their test scenarios. They also enjoy the predictable performance that comes with the use of Provisioned IOPS.

The team at Parse moved their MongoDB database over to Provisioned IOPS EBS volumes last month. As you can see from their blog post, end-to-end latency was halved and they no longer see periodic latency spikes due to MongoDB write locks. Kevin Lacker, CTO of Parse, told us:

Amazon’s Provisioned IOPS helped us improve our MongoDB query speed by over 60% and drastically reduced latency spikes to our clusters.  We are thrilled to welcome the new 2000 P-IOPS volumes.

Weve heard from 10gen, the developer of MongoDB, about their usage of EBS Provisioned IOPS. 10gen has many customers running MongoDB workloads of up to 20,000 IOPS on a single instance.  For these workloads, 10gen has told us that the high performance and consistency of EBS Provisioned IOPS are a great fit.

Update: Charity Majors of Parse wrote a really nice post, MongoDB with Amazons EBS and Provisioned IOPS. She discusses their move from RAID 10 to a new striped configuration using Provisioned IOPS volumes, which simplified their architecture while also improving performance. The change reduced database latency and greatly improved their cold-start time.

If you are still learning about Provisioned IOPS, be sure to watch this quick (and yet highly informative) video (but replace Arun’s “1000” with “2000”):

We also have a recorded webinar that will help you to learn more about running high performance databases on EC2 using EBS:

Finally, be sure to check out our EBS Provisioned IOPS Benchmarking Guide to learn more about the best tools and techniques for measuring and evaluating the performance of your EBS Provisioned IOPS volumes.

— Jeff;

AWS Elastic Beanstalk – Ruby Support and VPC Integration

AWS Elastic Beanstalk now supports Ruby applications and Amazon Virtual Private Cloud (VPC). Let’s take a look at each of these important and powerful new features.

Ruby Support
Elastic Beanstalk continues to expand the list of supported platforms (Java, PHP, Python, and .NET) and now adds Ruby applications. If youre a Ruby developer, you can now focus on your application and let Elastic Beanstalk handle the details of provisioning, deployment, monitoring, and scaling.

Elastic Beanstalk runs Ruby applications on the Passenger application server. This means you can develop and test locally and seamlessly deploy to Elastic Beanstalk without changes to your application code. The Elastic Beanstalk Developer Guide is a good place to learn more about the Ruby support. It also includes a walkthrough for Rails and Sinatra applications.

If you havent used Elastic Beanstalk before, heres a quick tour of its application management capabilities:

Choose your runtime and deploy using Git and the eb command line or the AWS Management Console:

Customize your environment using YAML configuration files and a Gemfile.

Use snapshot logs to debug your application.

Let Elastic Beanstalk automatically monitor the health of your application and auto-scale it.

Seamlessly integrate your Elastic Beanstalk application with Amazon RDS.

VPC Integration
You can now run Elastic Beanstalk applications within a Virtual Private Cloud (VPC).  You can now define and provision a private, virtual network in the cloud and connect it to your corporate network using a VPN connection. This allows you to run many new types of applications on Elastic Beanstalk. For example, you can run your intranet applications such as a trouble ticketing application or a reporting site on Elastic Beanstalk. Here’s the network topology:

Once you have your VPC set up, you simply provide the VPC ID and the subnet IDs to Elastic Beanstalk and your application gets automatically provisioned inside the VPC.

To learn more about how to launch Elastic Beanstalk environments inside a VPC, visit the Elastic Beanstalk Developer Guide. We also have an example CloudFormation template that creates a VPC that you can use to launch Elastic Beanstalk environments.

— Jeff;

PS – We are hiring software development engineers, product managers, and BI engineers. If you are passionate about building the best developer experience, get in touch with us at aws-elasticbeanstalk-jobs@amazon.com.

New EC2 Second Generation Standard Instances and Price Reductions

We launched Amazon EC2 with a single instance type (the venerable m1.small) in 2006. Over the years we have added many new instance types in order to allow our customers to run a very wide variety of applications and workloads.

The Second Generation Standard Instances
Today we are continuing that practice, with the addition of a second generation to the Standard family of instances. These instances have the same CPU to memory ratio as the existing Standard instances. With up to 50% higher absolute CPU performance, these instances are optimized for applications such as media encoding, batch processing, caching, and web serving.

There are two second generation Standard instance types, both of which are 64-bit platforms:

  • The Extra Large Instance (m3.xlarge) has 15 GB of memory and 13 ECU (EC2 Compute Units) spread across 4 virtual cores, with moderate I/O performance.
  • The Double Extra Large Instance (m3.2xlarge) has 30 GB of memory and 26 ECU spread across 8 virtual cores, with high I/O performance.

The instances are now available in the US East (Northern Virginia) region; we plan to support them in the other regions in early 2013.

On Demand pricing in the region for an instance running Linux starts at $0.58 (Extra Large) and $1.16 (Double Extra Large). Reserved Instances are available, and the instances can also be found on the EC2 Spot Market.

Price Reductions
As part of this launch, we are reducing prices for the first generation Standard (m1) instances running Linux in the US East (Northern Virginia) and US West (Oregon) regions by over 18% as follows:

Instance Type New On Demand Price Old On Demand Price
Small $0.065/hour $0.08/hour
Medium $0.13/hour $0.16/hour
Large $0.26/hour $0.32/hour
Extra Large $0.52/hour $0.64/hour

There are no changes to the Reserved Instance or Windows pricing.

Meet the Family
With the launch of the m3 Standard instances, you can now choose from seventeen instance types across seven families. Let’s recap just so that you are aware of all of your options (details here):

  • The first (m1) and second (m3) generation Standard (1.7 GB to 30 GB of memory) instances are well suited to most applications. The m3 instances are for applications that can benefit from higher CPU performance than offered by the m1 instances.
  • The Micro instance (613 MB of memory) is great for lower throughput applications and web sites.
  • The High Memory instances (17.1 to 68.4 GB of memory) are designed for memory-bound applications, including databases and memory caches.
  • The High-CPU instances (1.7 to 7 GB of memory) are designed for scaled-out compute-intensive applications, with a higher ratio of CPU relative to memory.
  • The Cluster Compute instances (23 to 60.5 GB of memory) are designed for compute-intensive applications that require high-performance networking.
  • The Cluster GPU instances (22 GB of memory) are designed for compute and network-intensive workloads that can also make use of a GPGPU (general purpose graphics processing unit) for highly parallelized processing.
  • The High I/O instance (60.5 GB of memory) provides very high, low latency, random I/O instance performance.

With this wide variety of instance types at your fingertips, you might want to think about benchmarking each component of your application on every applicable instance type in order to find the one that gives you the best performance and the best value.

— Jeff;

The AWS Report – Matt Lull of Citrix

In this episode of The AWS Report, I spoke with Matt Lull, Managing Director, Global Strategic Alliances, for Citrix to learn more about their cloud strategy. We talked about their line of virtualization products including Xen, XenServer, CloudBridge, and the Citrix NetScaler.

After that we talked about the concept of desktop virtualization, and Matt told me “Work isn’t a place you go anymore, it is a thing that you do.” From there we wrapped up with a discussion about AWS re:Invent.

— Jeff;

Launch EC2 Micro Instances in a Virtual Private Cloud

Judging from the number of requests that I have had for this particular combination of EC2 features, I’m expecting this to be a very popular post.

You can now launch EC2 micro (t1.micro) instances within a Virtual Private Cloud (VPC). The AWS Free Usage Tier now extends to t1.micro instances running inside of a VPC.

The micro instances provide a small amount of consistent CPU power, along with the ability to increase it in short burst when additional cycles are available. They are a good match for lower throughput applications and web sites that require additional compute cycles from time to time.

With this release, you now have everything that you need to create and experiment with your very own Virtual Private Cloud at no cost. This is pretty cool and I’m sure you’ll make good use of it.

— Jeff;

 

SAP HANA One – Now Available for Production Use on AWS

Earlier this year I briefly mentioned SAP HANA and the fact that it was available for developer use on AWS.

Today, SAP announced HANA One, a deployment option for HANA that is certified for production use on AWS available now in the AWS Marketplace. You can run this powerful, in-memory database on EC2 for just $0.99 per hour.

Because you can now launch HANA in the cloud, you don’t need to spend time negotiating an enterprise agreement, and you don’t have to buy a big server. If you are running your startup from a cafe or commanding your enterprise from a glass tower, you get the same deal. No long-term commitment and easy access to HANA, on an hourly, pay-as-you-go basis, charged through your AWS account.

What’s HANA?
SAP HANA is an in-memory data platform well suited for performing real-time analytics, and developing and deploying real-time applications.

I spent some time watching the videos on the Experience HANA site as I was getting ready to write this post. SAP founder Hasso Plattner described the process that led to the creation of HANA, starting with a decision to build a new enterprise database in December of 2006. He explained that he wanted to capitalize on two industry trends — the availability of multi-core CPUs and the growth in the amount of RAM per system. Along with this, he wanted to exploit parallelism within the confines of a single application. Here’s what they came up with:

Putting it all together, SAP HANA runs entirely in memory, eschewing spinning disk entirely except for backup. Traditional disk-based data management solutions are optimized for transactional or analytic processing, but not both. Transactional processing is oriented around and optimized for row-base operations: inserts, updates, and deletes. In contrast, analytic processing is tuned for complex queries, often involving subsets of the columns in a particular table (hence the rise of column-oriented databases). All of this specialization and optimization is needed due to the fact that accessing data stored on a disk is 10,000 to 1,000,000 times slower than accessing data stored in memory. In addition to this bottleneck, disk-based systems are unable to take full advantage of multi-core CPUs.

At the base, SAP HANA is a complete, ACID-compliant relational database with support for most of SQL-92. At the top,  you’ll find an analytical interface using Multi-Dimensional Expressions (MDX) and support for SAP BusinessObjects. Between the two is a parallel data flow computing engine designed to scale across cores. HANA also includes a Business Function Library, a Predictive Analysis Library, and the “L” imperative language.

So, what is HANA good for? Great question! Here are some applications:

Real-time analytics such as data warehousing, predictive analysis on Big Data, and operational (sales, finance, or shipping) reporting.

Real-time applications such as core process (e.g. ERP) acceleration, planning and optimization, and sense and response (smart meters, point of sale, and the like).

As an example of what can be done, SAP Expense Insight uses HANA and it is also available in the AWS Marketplace. It offers budget visibility to department managers in real-time, across any time horizon.

The folks at Taulia are building a dynamic discounting platform around HANA One. They’re already using AWS to streamline their deployment and operations; HANA One will allow them to make their platform even more responsive.

This is an enterprise-class product (but one that’s accessible to everyone) and I’ve barely scratched the surface. You can read this white paper to learn more (you may have to give the downloaded file a “.pdf” extension in order to open it).

Deploy HANA Now
As I mentioned earlier, SAP has certified HANA for production use on AWS. You can launch it today and you can get started now.

You don’t have to spend a lot of money. You don’t need to buy and install high-end hardware in you data center and you don’t need to license HANA. Instead, you can launch HANA from the AWS Marketplace and pay for the hardware and the software on an hourly, pay-as-you-go basis.

You’ll pay $0.99 per hour to run HANA One on AWS, plus another $2.50 per hour for an EC2 Cluster Compute Eight Extra Large instance with 60.5 GB of RAM and dual Intel Xeon E5 processors, bringing the total software and hardware cost to just $3.49 per hour, plus standard AWS fees for EBS and data transfer.

To get started, visit the SAP HANA page in the AWS Marketplace.

— Jeff;

Amazon EC2 Spot Instance Bid Status

We want to make EC2 Spot Instances even easier to use. One way we are doing this is by making the bidding and processing more open and more transparent.

You probably know that you can use Spot Instances to bid for unused capacity, allowing you to obtain compute capacity at a price that is based on supply and demand.

When you submit a bid for Spot capacity, your request includes a number of parameters and constraints. The constraints provide EC2 with the information that it needs to satisfy your bid (and the other bids that it is competing with) as quickly as possible. EC2 stores and then repeatedly evaluates the constraints until it is able to satisfy your bid. The following constraints (some mandatory and some optional) affect the evaluation process:

  • Max Price – The maximum bid price you are willing to pay per instance hour.
  • Instance Type – The desired EC2 instance type.
  • Persistent – Whether your request is one-time or persistent.
  • Request Validity Period – The length of time that your request will remain valid.
  • Launch Group – A label that groups a set of requests together so that they are started or terminated as a group.
  • Availability Zone Group – A label that groups a set of requests together so that the instances they start will launch in the same Availability Zone.
  • Availability Zone – An Availability Zone target for the request.

Spot Life Cycle
Each bid has a life cycle with multiple states. Transitions between the states occur when constraints are fulfilled. Here’s the big picture:

We want to give you additional information so that you can do an even better job of making Spot Bids and managing the running instances. You might find yourself wondering:

  • Why hasn’t my Spot Bid been fulfilled yet?
  • Can I change something in my Spot Bid to get it fulfilled faster?
  • Why did my Spot Instance launch fail?
  • Is my Spot Instance about to be interrupted?
  • Why was my Spot Instance terminated?

Spot Instance Bid Status
In order to give you additional insight in to the evaluation process, we are making the Spot Bid Instance Status visible through the AWS Management Console and the EC2 APIs. The existing DescribeSpotInstanceRequests function will now return two additional pieces of information – bidStatusCode and bidStatusMessage.This infomation is updated every time the Spot Bid’s provisioning status changes or is re-evaluated (typically a few seconds, but sometimes up to 3 minutes).

  • bidStatusCode is designed to be both machine-readable and human-readable.
  • bidStatusMessage is human-readable. Each bidStatusCode has an associated message:

You can find the complete set of codes and messages in the Spot Instance documentation. Here are some of the more interesting codes:

  • pending-evaluation – Your Spot request has been submitted for review and is pending evaluation.
  • fulfilled – Your Spot request is fulfilled and the requested instances are running.
  • marked-for-termination – Your Spot Instance is marked for termination because the request price is lower than the fulfillment price for the given instance type in the specified Availability Zone.

You can click on the Bid Status message in the AWS Management Console to see a more verbose message in the tooltip:

What is $100 Worth of Spot Good For?
If you are wondering about the value of Spot Instances, the new post, Data Mining the Web: $100 Worth of Priceless, should be helpful. The developers at Lucky Oyster used the Common Crawl public data set, EC2 Spot Instances, and a few hundred lines of Ruby to data mine 3.4 billion Web pages and extract close to a Terabyte of structured data. All in 14 hours for about $100.

Learn About Spot
I recently interviewed Stephen Elliott, Senior Product Manager on the EC2 team, to learn more about the Spot Instances concept. Here’s our video:

Stephen and his team are interested in your feedback on this and other Spot Instance features. You can email them at spot-instance-feedback@amazon.com .

If you are new to Spot Instances, get started now by signing up for EC2 and watching our HOWTO video. To learn even more, visit our EC2 Spot Instance Curriculum page.

— Jeff;