Category: Compute*


AWS Elastic Beanstalk – Environment Resource Support + Updated PHP Runtime

AWS Elastic Beanstalk can now provision and configure AWS resources to power your application. In conjunction with a new version of our PHP runtime, you get more control and more flexibility with less code.

In October, we announced the Elastic Beanstalk configuration files and talked about how they can help you configure EC2 instances without creating and maintaining custom AMIs. We have extended these configuration files to allow you to provision and configure resources such as SQS queues and DynamoDB tables for your Elastic Beanstalk application. The configuration file format is YAML-based, and you can find the details in the AWS Elastic Beanstalk Developer Guide.

We are also announcing an updated PHP runtime that supports configuration files as well as the seamless integration with Amazon RDS and Amazon VPC – see my previous blog post for details. The PHP runtime supports 5.3 and 5.4. You can also install dependencies using Composer. Visit the Elastic Beanstalk Developer Guide to learn more about the updated PHP runtime and for walkthroughs of how to deploy CakePHP and Symfony2.

I’d lke to show you how simple and powerful it is to use Elastic Beanstalk to create and configure the new environment resources. Ill use the updated PHP runtime and show you how to configure DynamoDB backed session management. You can find the full example in the AWS Elastic Beanstalk Developer Guide.

Here’s what you need to do:

  • Launch an Elastic Beanstalk PHP environment (see the AWS Elastic Beanstalk Developer Guide).
  • Create an IAM role with enough permission to read and write to the DynamoDB table (lets call it SessionRole).
  • Add an .ebextensions directory to your application.
  • In that directory, create a configuration file called dynamodb.config with the following content:
    Resources:
      SessionTable:
        Type: AWS::DynamoDB::Table
        Properties:
          KeySchema:
            HashKeyElement:
              AttributeName: Index
              AttributeType: S
          ProvisionedThroughput:
            ReadCapacityUnits: 1
            WriteCapacityUnits: 1
  • Add the role to the Elastic Beanstalk environment (this and all of the subsequent snippets also belong in the dynamodb.config file):
    AWSEBAutoScalingLaunchConfiguration:
        Properties:
          IamInstanceProfile:
            Fn::GetOptionSetting:
              OptionName: IAMInstanceProfileARN
              DefaultValue: “arn:aws:iam:: 123456789123:instance-profile/SessionRole”
  • Pass the details of the DynamoDB table to the EC2 instances in your environment:
    files:
      “/var/app/sessiontable”:
        mode: “000444”
        content: |
          ` {“Ref” : “SessionTable” }`
          ` {“Ref” : “AWS::Region” }`
  • Add a CloudWatch Alarm so that you are alerted when requests to DynamoDB get throttled:
    SessionThrottledRequestsAlarm:
        Type: AWS::CloudWatch::Alarm
        Properties:
          AlarmDescription: { “Fn::Join” : [“”, [ { “Ref” : “AWSEBEnvironmentName” }, “: requests are being throttled.” ] ] }
          Namespace: AWS/DynamoDB
          MetricName: ThrottledRequests
          Dimensions:
            – Name: TableName
              Value: { “Ref” : “SessionTable” }
          Statistic: Sum
          Period: 300
          EvaluationPeriods: 1
          Threshold:
            Fn::GetOptionSetting:
              OptionName: SessionThrottledRequestsThreshold
              DefaultValue: 1
          ComparisonOperator: GreaterThanThreshold
          AlarmActions:
            – Ref: SessionAlarmTopic
          InsufficientDataActions:
            – Ref: SessionAlarmTopic
  • Make sure to include a require for the AWS SDK for PHP 2 in your composer.json file:
    “require” : {
                “aws/aws-sdk-php” : “*”
            }
  • Finally, add the following code to your PHP application to connect it to the configuration file:
    <?php

    // Include the SDK using the Composer autoloader
    require ‘vendor/autoload.php’;

    use Aws\DynamoDb\DynamoDbClient;

    // Grab the session table name and region from the configuration file
    list($tableName, $region) = file(__DIR__ . ‘/../sessiontable’);

    // Create a DynamoDB client and register the table as the session handler
    $dynamodb = DynamoDbClient::factory(array(‘region’ => $region));
    $handler = $dynamodb->registerSessionHandler(array(‘table_name’ => $tablename, ‘hash_key’ => ‘Index’));

    ?>

As you can see, you can use this new Elastic Beanstalk feature to set up and configure the AWS resources needed by your application.

— Jeff, with lots of help from Saad

EBS Snapshot Copy (Between Regions)

To make it even easier for you to build AWS applications that span regions, we’re introducing a new EBS Snapshot Copy feature today. You can now copy EBS snapshots between EC2 Regions.

Why Copy?

So, why would you want to copy an EBS Snapshot from one AWS Region to another? Here are some of the more common use cases:

  • Geographic Expansion – You want to be able to launch your application in a new Region.
  • Migration – You want to be able to migrate your application from one Region to another. 
  • Disaster Recovery – You want to back up your data and your log files across different geographical locations at regular intervals to minimize data loss and recovery time.

EBS Snapshot Copy simplifies each of these use cases by simplifying the copy process.

Ok, How do I Copy Snapshots?
You can now copy EBS Snapshots from one AWS Region to another. You can copy any accessible Snapshots that are in the “completed” status. This includes Snapshots that you created, Snapshots that were shared with you, and also Snapshots from the AWS Marketplace, VM Import/Export, and Storage Gateway. If you copy a Marketplace product to a new Region, you need to make sure that the product is supported in the destination Region.

You can initiate copies from the AWS Management Console or from the command line. You can also use the new CopySnapshot function from your own code. Here’s how you initiate a copy operation from the Console:

After you choose the Copy Snapshot operation, the Console will ask you where you would like to copy the snapshot:

While the copy is underway, you can watch the progress by switching to the destination Region:

What Else Do I Need to Know?
Here are a few important facts about this new feature:

  • The AWS Management Console shows the progress of a Snapshot copy in progress, you can check the percentage complete there.
  • You can initiate multiple Snapshot Copy commands simultaneously either by selecting and copying multiple Snapshots to the same region, or by copying a snapshot to multiple regions in parallel.  The in-progress copies do not affect the performance of the associated EBS Volumes.
  • The console-based interface is push-based; you log in to the source Region and tell the console where you’d like the Snapshot to end up. The API and the command line are, by contrast, pull-based and you must run them within the destination Region.

Pricing
You will be charged the regular data transfer prices when you copy snapshots between Regions. Once you have created a new Snapshot, you’ll pay the regular price to store it in the destination Region.

Coming Soon
As you can see, EBS Snapshot Copy, will let you simplify your geographic expansion, data center migration, and disaster recovery. As a next step, we will implement copying of AMIs (Amazon Machine Images) between Regions. You’ll be able to copy public and custom-created AMIs to the Region of your choice.

— Jeff;

AWS Expansion in Brazil – Elastic Beanstalk, Provisioned IOPS for EBS and RDS

We launched an AWS Region in Brazil almost a year ago, along with Portuguese and Spanish versions of the AWS Blog.

Today we are adding the following new AWS functionality in the Region:

AWS Elastic Beanstalk – You can now deploy and manage .NET, PHP, Python, Ruby, and Java applications in the AWS Cloud using AWS Elastic Beanstalk. With this new addition to our lineup in Brazil, you can get your application online more quickly and allow Elastic Beanstalk to handle system updates, scaling, monitoring, and lots more.

EBS Provisioned IOPS – You can now create EBS Provisioned IOPS volumes in Brazil, with up to 2000 IOPS per EBS volume. This will give you more control and the ability to create fast, responsive applications on AWS.

RDS Provisioned IOPS – You can now create RDS Database Instances with up to 10,000 Provisioned IOPS (for MySQL and Oracle) or 7,000 (SQL Server). This gives you the power to create fast, responsive database-driven applications on AWS. As part of today’s launch, this is also available in the Asia Pacific (Singapore) Region.

— Jeff;

PS[0] – Check out the AWS Products and Services by Region to find out which services are available in each AWS Region.

PS[1] – This post is also available in Portuguese and Spanish.

AWS Marketplace – Support for Windows Applications (ISVs Wanted)

Today we are opening up the AWS Markeplace to applications running on Microsoft Windows. We’re launching with an initial set of applications, with many more in the pipeline for release in the near future. If you are a Windows ISV (Independent Software Vendor), you should consider making your products available in the Marketplace. I’ll tell you why in just a minute.

What’s the Marketplace?
The AWS Marketplace makes it easy for you to find, compare, and immediately start using a wide variety of applications and technical services in the cloud. The applications you run are hosted on Amazon EC2 instances running Linux or Microsoft Windows, with 1-click deployment and billing integrated into your existing AWS account. You pay only for what you use, allowing you to eliminate on-going hardware and software costs when your needs change or your projects come to an end.

Now For Windows
We launched the AWS Marketplace earlier this year with support for applications and services running on Linux. Effective immediately, we now support applications that run on Microsoft Windows Server. We support a number of versions of Windows Server including Windows Server 2008, Windows Server 2008 R2, and Windows Server 2012.

You can choose to run Windows on the full spectrum of EC2 instance types, from micro instances up to our largest cluster-computing instances, and in any of our nine regions worldwide.  Windows is also available in the AWS Free Usage Tier, giving you the ability to test the environment for up to 750 hours per month without incurring charges, and then easily migrate to full-fledged environments when you are ready. 

Windows instances work seamlessly with existing AWS features like Amazon Elastic Block Store (EBS), Amazon CloudWatch, AWS CloudFormation, Amazon Virtual Private Cloud, AWS Elastic Beanstalk, Elastic Load Balancing, and Amazon Relational Database Service.  EC2 Windows instances are pre-configured for security and high-performance on AWS, and are set up for automatic patching and deployment.  All Windows EC2 instances come with basic support, and business- and enterprise-level support is available.

Windows Applications in the Marketplace
The following Windows applications are now available in the AWS Marketplace:

  • Parallels Plesk – A control panel for hosting and managing multiple web sites on a single server. 
  • Quest Toad – A database productivity tool for DBAs, developers, and analysts.
  • MicroStrategy – A comprehensive BI (Business Intelligence) platform.
  • SAP Afaria – Mobile device management.

ISVs Wanted
If you are a Windows ISV, we want you to list your software in the AWS Marketplace so that you can sell it to hundreds of thousands of active AWS customers located all over the world. Because the AWS Marketplace shortens the time between discovery and deployment to a matter of minutes, you’ll find that the sales cycle is dramatically shortened as well — decisions that once took days or weeks now take just minutes. Once your application is in the Marketplace, we’ll take care of all billing, collections, and disbursements for you so that you can focus on building your products and your business. You can now sell your traditional on-premises software in the cloud using a SaaS (Software-as-a-Service) model. Learn more about Selling on Amazon Marketplace.

— Jeff;

AWS Marketplace – Additional EC2 Operating System Support (FreeBSD, Debian, CentOS)

We’re working hard to make the AWS Marketplace even more flexible and to make sure that it contains the operating systems, tools, and other resources that you need. Today we are adding support for three new open source operating systems: FreeBSD, CentOS, and Debian. We are also making it easier for you to find software that runs on the operating system of your choice.

Expanded Operating System Support
You can now launch three new operating systems from within the AWS Marketplace:

FreeBSD is an advanced operating system for modern server, desktop and embedded computer platforms. FreeBSD provides advanced networking, impressive security features, and world class performance, and is used by some of the world’s busiest websites.

Debian is a popular and influential Linux distribution. The current stable release includes support for over 29,000 packages.

CentOS is a free, Enterprise-class Linux distribution derived from publicly available sources. CentOS conforms fully with the upstream vendor’s redistribution policy and aims to be 100% binary compatible with their offering.

These operating systems come directly from the Open Source community and are available at no charge other than the usual cost for the EC2 instances. You can find them in the Operating Systems section of the Marketplace.

Improved Searching
You can now search for software that’s running on the operating system of your choice:

I hope that you enjoy these new additions to the Marketplace.

— Jeff;

EBS Volume Status Checks

We’ve received some great feedback on the EC2 instance status checks that were released earlier this year. Our customers appreciate the fact that we can detect and report on how their EC2 instances are performing.

Today we are adding a new status check for EBS Provisioned IOPS volumes. As you know, you can now provision up to 2000 IOPS when you create a new EBS volume:

We are now making a new I/O Performance status check available for each Provisioned IOPS volume. The status check will tell you how well your volume is doing with respect to the number of IOPS that have been provisioned.  In the background, we measure the volume’s I/O performance each minute and determine if it is Normal, Degraded, Severely Degraded or Stalled. 

A volume’s performance may fall below normal if you are accessing data on it for the first time, making frequent snapshots at peak usage times, or accessing the volume from an EC2 instance that isn’t EBS-optimized.

You can view the status of your Provisioned IOPS volumes in the AWS Management Console:

You can also retrieve the status by calling the DescribeVolumeStatus function. To learn more about this feature, visit the Monitoring the Status of Your Volumes section of the EC2 documentation.

We are also publishing two new CloudWatch metrics for each volume at one minute intervals:

VolumeThroughputPercentage is the percentage of IOPS delivered out of the IOPS provisioned for an EBS volume.

VolumeConsumedReadWriteOps is the total amount of read and write operations consumed in the period. Provisioned IOPS volumes process your applications reads and writes in I/O block sizes of 16KB or less. Every increase in I/O size above 16KB will linearly increase the resources you need to achieve the same IOPS rate.

You can view these metrics in the AWS Management Console and you can access them through the CloudWatch APIs.

— Jeff;

 

Windows Server 2012 Now Available on AWS

Today we have an awesome guest post from a General Manager on the Amazon EC2 team. Enjoy!

— Jeff;


Before talking about Windows Server 2012, let me first introduce myself. My name is Tom Rizzo and Im the General Manager for the Windows team in AWS. I previously worked at Microsoft in a variety of business groups including Exchange Server, SQL Server and SharePoint Server. Im excited to be at Amazon helping AWS be the best place for customers to run Windows and Windows workloads.

Were trying something new on the AWS blog by introducing a series we call Windows Wednesdays to help you learn more about the work were doing to support Windows on AWS. While well try to post at least twice a month about some new feature, tidbit or something you may not know about Windows on AWS, I wont promise that we wont miss a Wednesday or two but well do our best!

With that said, we couldn’t wait for Wednesday for this announcement! AWS is excited to add Windows Server 2012 as an option for customers in addition to our existing Windows Server 2003 R2, 2008 and 2008 R2 offerings.  Windows Server 2012 provides a number of exciting enhancements including a Server Manager for multi-server setup, Powershell Improvements, Internet Information Services 8.0 and the .NET Framework 4.5.

What’s New in AWS for Windows Server?
Beyond whats new from Microsoft in terms of Windows Server 2012 functionality, here are some highlights specific to Windows Server 2012 on AWS.

Free Tier
First, you can use Windows Server, including Windows Server 2012, as part of the AWS Free Usage Tier. By using it, you get 750 hours of a Micro Instance per month for a full year!  Its a great way to start evaluating Windows Server 2012 and when youre ready to use it in production, just stop your instance and restart it on a bigger instance.  No migration and no messy moving around on different server hardware.  AWS makes it easy for you to grow the power of your instances with a few clicks of the mouse.

New Languages
With the global expansion of AWS, we want to make sure we meet the needs of our worldwide customers.  To that end, today were publishing 31 Windows Server 2012 AMIs which includes AMIs for 19 different languages and for Windows with SQL Server 2008 and 2008 R2 (Express, Web and Standard). The AMIs are available in every Region and can be used with any instance type.

AWS Elastic Beanstalk Support for Windows Server 2012
AWS Elastic Beanstalk allows you to focus on building your application, without having to worry about the provisioning and maintenance aspects of your applications. Elastic Beanstalk already supports Java, PHP, Python, Ruby, and Windows Server 2008 R2 based .NET applications. Starting today, it will also support Windows Server 2012 based .NET applications. You can conveniently deploy your applications from Visual Studio or the AWS Management Console.

.NET 4.5 and Visual Studio 2012 Support
To support building applications on Windows with .NET, AWS supports the new .NET framework shipped with Windows Server 2012 – .NET 4.5. With integrated Visual Studio support, including the new Visual Studio 2012, getting started with building AWS applications is as easy as creating an AWS project in Visual Studio as shown below..

Included with the Visual Studio tools is the AWS Explorer which allows you to see all of your AWS resources without leaving the Visual Studio environment.  In addition, you can deploy to AWS with just a few clicks and can decide whether you want to deploy to EC2 instances or use Elastic Beanstalk as the target for your applications.

Getting Started
To help you get started with Windows Server 2012, we put together a quick introduction video that steps you through the process of creating a Windows Server 2012 instance, expanding the size of the root volume, adding an EBS volume, and connecting to the instance.

If you are attending AWS re:Invent next week, please feel free to drop by the EC2 booth to learn more. We’ll also be presenting several sessions on Windows.

— Tom

 

AWS in Action – Behind the Scenes of a Presidential Campaign

Now that the 2012 US Presidential Election is over, I’d like to take you behind the scenes and tell you a bit about the technology that powered President Obama’s successful reelection campaign!

To set the stage, imagine setting up the technology infrastructure needed to power a dynamic, billion-dollar organization under strict time limits using volunteer labor, with traffic peaking for just one day, and then shutting everything down shortly thereafter.

The words “mission critical” definitely apply here. With the opportunity to lead the United States as the prize, the stakes were high.

Cloud Computing for the Win
The campaign’s technology team built, deployed, ran, and scaled up their applications on AWS. After the election, they backed it all up to Amazon S3 and scaled way, way down.

The campaign used AWS to avoid an IT investment that would have run into the tens of millions of dollars. Along the way they built and ran more than 200 applications on AWS, scaled to support millions of users. One of these apps, the campaign call tool, supported 7,000 concurrent users and placed over two million calls on the last four days of the campaign. Here’s a graph which depicts the growth in call volume in the days leading up to the election:

Data is Key
A database running on Amazon RDS, served as the primary registry of voter file information. This database integrated data from a number of sources including www.barackobama.com and donor information from the finance team) in order to give the campaign managers a dynamic, fully-integrated view of what was going on. Alongside this database, an analytics system running on EC2 Cluster Compute instances (cc2.8xl).  Another database cluster ran Leveldb on a trio of High-Memory Quadruple Extra Large instances.

This array of databases allowed campaign workers to target and segment prospective voters, shift marketing resources based on near real-time feedback on the effectiveness of certain ads, drive a donation system that collected over one billion dollars (the 30th largest ecommerce site in the world).

Complex Applications
The applications run by the Obama campaign are comparable in scope and complexity to those seen in the largest enterprises and data-rich startups. For example:

  • Massive data modeling using Vertica and Elastic MapReduce.
  • Multi-channel media management via TV, print, web, mobile, radio and email using dynamic production, targeting, retargeting, and multi-variant testing like you’d find in a sophisticated digital media agency.
  • Social coordination and collaboration of volunteers, donors, and supporters.
  • Massive transaction processing.
  • Voter abuse prevention and protection, including capture of incoming incidents and dispatch of volunteers.
  • A rich information delivery system for campaign news, polls, information on the issues, voter registration, and more.

The applications made use of virtually every AWS service including EC2, Route 53, SQS, DynamoDB, SES, RDS, VPC, EBS Provisioned IOPS, SNS, ElastiCache, Elastic Load Balancing, Auto Scaling, and CloudFront. They also took advantage of our team of Solution Architects and AWS Premium Support.

Learn More at re:Invent
We’ve assembled an impressive “Big Data and the US Presidential Campaign” panel for AWS re:Invent.  You’ll get to hear from the people who built and ran the applications described above:


Picture courtesy of @Trammel.

I am looking forward to learning even more about their use of technology to help win the election.

— Jeff;

Spotlight on Cost: TellApart

Part of our modus operandi at AWS is to help customers lower the costs of operating their infrastructure. When we are able to achieve better efficiencies in our own datacentres, we pass those savings back on to our customers. That can come in the form of price reductions, or in a variety of payment models.

With Amazon EC2, for example, you can choose to request compute resources as and when you need them, and pay a fixed on-demand price per hour. For more predictable workflows, you can pay a small up-front price for reserved capacity. This allows us to plan our own operations more efficiently, and so we’re able to pass those savings back to you in the form of a reduced hourly price. However, even with the combined reserved capacity and on-demand requests, there can still be unused capacity available in the AWS datacentres.

Rather than let this capacity go to waste, we offer it to customers at a variable price, on the EC2 Spot Market. You can name your own price for this under utilized capacity by placing bids, and achieve significant savings compared to on-demand or reserved capacity. Spot instances allow you to run time insensitive tasks at a lower cost, or to overclock your infrastructure, but adding more bang per unit cost.

To help illustrate how customers are using spot instances, I wanted to introduce you to some customers who have seen some steep cost reductions by integrated spot into their workflows. Today’s willing volunteer is ecommerce data legend, TellApart.

TellApart reduced costs by 75%

“At TellApart, our goal is to build a big data platform that enables retailers to unlock the power of their customer data”, said Julie Black, Director of Engineering at TellApart. “Every day were analyzing massive amounts of shopping data from hundreds of millions of users and building complex machine learning algorithms that empower retailers to identify their highest quality customers and deliver them perfectly targeted marketing.”

So how do they use spot instances?

“At the core of our system is our data processing pipeline. It serves as the backbone of our machine learning system and produces thousands of business metrics consumed daily by our clients”, continues Julie, “To make this all work, we use Amazon Elastic MapReduce to bring up a Hadoop cluster that can batch process log data generated by our distributed fleet of machines.”

“We use Spot Instances for all Hadoop Task Nodes for the flexibility and cost effectiveness we require. Through simple bidding optimizations on the Spot market, we have been able to cut our data processing costs more than 75%”.

Using spot instances with Hadoop and Amazon Elastic MapReduce is very straight forward: you can specify your bid price right in the AWS console when spinning up new clusters in just a few clicks. Easy money.

The Amazon EC2 Spotathon, 2012

If you’d like to shine a spotlight on what you do (or are planning to do) with the spot market, you’ll be interested in the very first Spotathon. There’s a grand prize of $2500 of AWS credit available to the winner who shows the most innovative, low cost use of spot instances in their application architecture.

That’s around 43,000 compute hours of the new m3.xlarge instances at the current spot price.

You can register for the Spotathon today. Looking forward to seeing how you’re using spot.

~ Matt