Category: Compute*


New Whitepaper: Mapping and GeoSpatial Analysis in the Cloud Using ArcGIS

Esri is one of the leaders in the Geographic Information Systems (GIS) industry and one of the largest privately held software companies focused on mapping and geospatial applications in the world with offices in more than 100 countries. Both public and private sector organizations use Esri technology to analyze and manage their geographic information and make better decisions uses range from planning cities and improving the quality of life for residents, to site selection, customer analytics, and streamlining logistics.

Esri and AWS have been working together since 2008 to bring the power of GIS to the masses. The AWS Partner Team recently attended the 2012 Esri International User Conference with over 14,000+ attendees, 300 exhibitors and a large number of ecosystem partners. A cloud computing theme dominated the conference.

Esri and AWS have co-authored a whitepaper, “Mapping and GeoSpatial Analysis Using ArcGIS“, to provide users who have interest in performing spatial analysis using their data with complimentary datasets. The paper discusses how users can publish and analyze imagery data (such as satellite imagery, or aerial imagery) and create and publish tile cache map services from spatially referenced data (such as data with x/y points, lines, polygons) in AWS using ArcGIS.

Download PDF: Mapping and GeoSpatial Analysis Using ArcGIS

ArcGIS_AWS
The paper focuses on imagery because that has been the most challenging data type to manage in the cloud, but the approaches discussed are general enough to apply to any type of data. It not only provides architecture guidance on how to scale ArcGIS servers in the cloud but also provides step-by-step guidance on publishing map services in the cloud. 

For more information on GeoApps in the AWS Cloud, see the presentation – The Cloud as a Platform for Geo below:

Looking forward to hearing your feedback!

Jinesh;

AWS Cost Allocation For Customer Bills

Growth Challenges
You probably know how it goes when you put AWS to work for your company. You start small — one Amazon S3 bucket for some backups, or one Amazon EC2 instance hosting a single web site or web application. Things work out well and before you know it, word of your success spreads to your team, and they start using it too. At some point the entire company jumps on board, and you become yet another AWS success story.

As your usage of AWS grows, you stop charging it to your personal credit card and create an account for your company. You use IAM to control access to the AWS resources created and referenced by each of the applications.

There’s just one catch — with all of those departments, developers, and applications making use of AWS from a single account, allocating costs to projects and to budgets is difficult because we didn’t give you the necessary information. Some of our customers have told us that this cost allocation process can consume several hours of their time each month.

Cost Allocation Via Tagging
Extending the existing EC2 tagging system (keys and values), we are launching a new cost allocation system to make it easy for you to tag your AWS resources and to access billing data that is broken down by tag (or tags).

With this release you can tag the following types of AWS resources for cost allocation purposes:

  • S3 buckets
  • EC2 Instances
  • EBS volumes
  • Reserved Instances
  • Spot Instance requests
  • VPN connections
  • Amazon RDS DB Instances
  • AWS CloudFormation Stacks

Here’s all that you need to do:

  1. Decide on Your Tagging Model – Typically, the key name identifies some axis that you care about and the key values identify the points along the axis. You could have a tag named Department, with values like Sales, Marketing, Development, QA, Engineering, and so forth. You could choose to align this with your existing accounting system. You can use multiple tags for cost allocation purposes, each of which represents an additional dimension of usage. If each department runs several AWS-powered applications (or stores lots of data in S3), you could add an Application tag, with the values representing all of the applications that are running on behalf of the department. You can use the tags to create your own custom hierarchy.
  2. Tag Your Resources – Apply the agreed-upon tags to your existing resources, and arrange to apply them to newly created resources as they appear. You can add up to ten fifty tags per resource. You can do this from the AWS Management Console, the service APIs, the command line, or through Auto Scaling:

    You can use CloudFormation to provision a set of related AWS resources and easily tag them.

  3. Tell AWS Which Tags Matter -Now you need to log in to the AWS Portal, sign up for billing reports, and tell the AWS billing system which tag keys are meaningful for cost allocation purposes by using the Manage Cost Allocation Report option:

    You can choose to include certain tags and to exclude others.

  4. Access Billing Data – The estimated billing data is generated multiple times per day and the month-end charges are generated within three days of the end of the month. You can access this data by enabling programmatic access and arranging for it to be delivered to your S3 bucket.

Data Processing
The Cost Allocation Report will contain one additional column for each of the tag keys that you selected in step 3. The corresponding tag value (if any) will be included in the appropriate column of the data:

In the Cost Allocation Report above, the relevant keys were Owner, Stack, Cost Center, Application, and Project. The column will be blank if the AWS resource doesn’t happen to have a value for the key. Data transfer and request charges are also included for tagged resources. In effect, these charges inherit the tags from the associated resource.

Once you have this data, you can feed it in to your own accounting system or you can slice and dice it any way you’d like for reporting or visualization purposes. For example, you could create a pivot table and aggregate the data along one or more dimensions:

In The Works
We are planning to make this feature even more useful over time. We want to give you the ability to tag more types of AWS resources and we are thinking about allowing you to use more than ten tags per resource. We also want to include the upfront fee for Reserved Instance purchases. Your suggestions for other features are always welcome too!

Jeff;

Announcing AWS Elastic Beanstalk support for Python, and seamless database integration

Its a good day to be a Python developer: AWS Elastic Beanstalk now supports Python applications! If youre not familiar with Elastic Beanstalk, its the easiest way to deploy and manage scalable PHP, Java, .NET, and now Python applications on AWS. You simply upload your application, and Elastic Beanstalk automatically handles all of the details associated with deployment including provisioning of Amazon EC2 instances, load balancing, auto scaling, and application health monitoring.

Elastic Beanstalk supports Python applications that run on the familiar Apache HTTP server and WSGI. In other words, you can run any Python applications, including your Django applications, or your Flask applications. Elastic Beanstalk supports a rich set of tools to help you develop faster. You can use eb and Git to quickly develop and deploy from the command line. You can also use the AWS Management Console to manage your application and configuration.

The Python release brings with it many platform improvements to help you get your application up and running more quickly and securely. Here are a few of the highlights below:

Integration with Amazon RDS

Amazon RDS makes it easy to set up, operate, and scale a relational database in the cloud, making it a great fit for scalable web applications running on Elastic Beanstalk. 

If your application requires a relational database, Elastic Beanstalk can create an Amazon RDS database instance to use with your application. The RDS database instance is automatically configured to communicate with the Amazon EC2 instances running your application.

Console-rds
A console screenshot showing RDS configuration options when launching a new
AWS Elastic Beanstalk environment.

Once the RDS database instance is provisioned, you can retrieve information about the database from your application using environment variables:

import os

 

if ‘RDS_HOSTNAME’ in os.environ:

    DATABASES = {

        ‘default’: {

            ‘ENGINE’: ‘django.db.backends.mysql’,

            ‘NAME’: os.environ[‘RDS_DB_NAME’],

            ‘USER’: os.environ[‘RDS_USER’],

            ‘PASSWORD’: os.environ[‘RDS_PASSWORD’],

            ‘HOST’: os.environ[‘RDS_HOSTNAME’],

            ‘PORT’: os.environ[‘RDS_PORT’],

        }

}

To learn more about using Amazon RDS with Elastic Beanstalk, visit Using Amazon RDS with Python in the Developer Guide.

 

Customize your Python Environment

You can customize the Python runtime for Elastic Beanstalk using a set of declarative text files within your application. If your application contains a requirements.txt in its top level directory, Elastic Beanstalk will automatically install the dependencies using pip.

Elastic Beanstalk is also introducing a new configuration mechanism that allows you to install packages from yum, run setup scripts, and set environment variables. You simply create a .ebextensions directory inside your application and add a python.config file in it. Elastic Beanstalk loads this configuration file and installs the yum packages, runs any scripts, and then sets environment variables. Here is a sample configuration file that syncs the database for a Django application:

commands:

    syncdb:

        command: “django-admin.py syncdb –noinput”

        leader_only: true

 

option_settings:

    “aws:elasticbeanstalk:application:python:environment”:

        DJANGO_SETTINGS_MODULE: “mysite.settings”

    “aws:elasticbeanstalk:container:python”:

        WSGIPath: “mysite/wsgi.py”

 

Snapshot your logs

To help debug problems, you can easily take a snapshot of your logs from the AWS Management console. Elastic Beanstalk aggregates the top 100 lines from many different logs, including the Apache error log, to help you squash those bugs.

Console-snapshot-logs

The snapshot is saved to S3 and is automatically deleted after 15 minutes. Elastic Beanstalk can also automatically rotate the log files to Amazon S3 on an hourly basis so you can analyze traffic patterns and identify issues. To learn more, visit Working with Logs in the Developer Guide.

 

Support for Django and Flask

Using the customization mechanism above, you can easily deploy and run your Django and Flask applications on Elastic Beanstalk.

For more information about using Python and Elastic Beanstalk, visit the Developer Guide.

We look forward to hearing about how you put these new features to use!


The AWS Report – Michael Wasser, Raveld

Earlier this summer I interviewed Michael Wasser, the founder of Raveld, located near Amazon’s Seattle headquarters. Raveld’s elevator pitch is “clear and concise cloud costs.” I enjoyed speaking with Michael to learn about his product and how it uses AWS:

Michael built Raveld after finding that his consulting customers were exceeding their cloud computing budgets due to a lack of insight into the resources they were using and what those resources were used for.

He runs the entire site on four EC2 instances. Interestingly enough, two of the instances (one front end and one back end) are acquired on the EC2 Spot market. The total cost of the four instances and an Elastic Load Balancer is less than $300 per month. Michael told me that “Without AWS, we wouldn’t be able to do this with such a shoestring budget.”

Watch the video to learn more!

Jeff;

PS – If you are interested in being on The AWS Report, send us a note (awsreport@amazon.com).

NASA and AWS – Curiosity has Landed!

If you are like me, you spent this past Sunday afternoon looking forward to the landing of the Curiosity rover on Mars. I was able to wrest the remote control away from my family and change to NASA TV in time to watch through the aptly named “seven minutes of terror” as Curiosity performed an intricate series of steps that resulted in a safe, on-target landing.

The AWS team is proud to have helped to bring the video to you and to other like-minded space geeks around the globe. Now that the (red) dust has cleared, we’ve published a new NASA success story with full information on their AWS setup.

We’ve been working with NASA for several years, providing support to a number of current and future missions. The data collection and image processing tasks for the earlier Spirit and Opportunity rovers runs on AWS, coordinated by the Amazon Simple Workflow service (read the case study to learn more).

For the Curiosity landing, NASA built a system to serve web content and streaming video to a worldwide audience. They made use of a number of AWS services including EC2, S3, SimpleDB, Route 53, CloudFront, the Relational Database Service, Simple Workflow, CloudFormation, and Elastic Load Balancing.

Now that Curiosity is ready to roll, the mission will use AWS to automate and accelerate the analysis of the images that it sends each day. This accelerated analysis will lead to better decision making and an increase in the amount of exploration that Curiosity can do each day.

Read our new NASA story to learn more.

The Curiosity rover includes a stereoscopic camera. In order to produce a finished image, each pair (left and right) of images must be warped to compensate for perspective, then stereo matched to each other, stitched together, and then tiled into a larger panorama. This complex workflow is difficult to express using a system based on message queues. Instead, the entire process was modeled and implemented using the Amazon Simple Workflow service and the AWS Flow Framework. As described in the case study, the Simple Workflow Service coordinates worker tasks that are spread across EC2 instances and JPL data centers. The worker tasks are able to participate in the workflow without regard to their location.

— Jeff;

EBS Provisioned IOPS – Some Interesting Resources

Our recent release of the EBS Provisioned IOPS feature (blog post, explanatory video, EBS home page) has met with a very warm reception. Developers all over the world are already making use of this important and powerful new EC2 feature. I would like to make you aware of some other new resources and blog posts to help you get the most from your EBS volumes.

 
  • The EC2 FAQ includes answers to a number of important performance and architecture questions about Provisioned IOPS.
  • The EC2 API tools have been updated and now support the creation of Provisioned IOPS volumes. The newest version of the ec2-create-volume tool supports the –type and –iops options. For example, the following command will create a 500 GB volume with 1000 Provisioned IOPS:
    $ ec2-create-volume –size 500 –availability-zone us-east-1b –type io1 –iops 1000
  • Eric Hammond has written a detailed migraton guide to show you how to convert a running EC2 instance to an EBS-Optimized EC2 instance with Provisioned IOPS volumes. It is a very handy post, and it also shows off the power of programmatic infrastructure.
  • I have been asked about the applicability of existing EC2 Reserved Instances to the new EBS-Optimized instances. Yes, they apply, and you pay only the additional hourly charge. Read our new FAQ entry to learn more.
  • I have also been asked about the availability of EBS-Optimized instances for more instance types. We intend to support other instance types based on demand. Please feel free to let us know what you need by posting a comment on this blog or in the EC2 forum.
  • The folks at CloudVertical have written a guide to understanding new AWS I/O options and costs.
  • The team at Stratalux wrote a very informative blog post, Putting Amazon’s Provisoned IOPS to the Test. Their conclusion:
    “Based upon our tests PIOPS definitely provides much needed and much sought after performance improvements over standard EBS volumes.  Im glad to see that Amazon has heeded the calls of its customers and developed a persistent storage solution optimized for database workloads.”

We have also put together a new guide to benchmarking provisioned IOPS volumes. The guide shows you how to set up and run high-quality, repeatable benchmarks on Linux and Windows using the fio, Oracle Orion, and SQLIO tools. The guide will walk you through the following steps:

  • Launching an EC2 instance.
  • Creating Provisioned IOPS EBS volumes.
  • Attaching the volumes to the instance.
  • Creating a RAID from the volumes.
  • Installing the appropriate benchmark tool.
  • Benchmarking the I/O performance of your volumes.
  • Deleting the volumnes and terminate the instance.

Since I like to try things for myself, I created six 100 GB volumes, each provisioned for 1000 IOPS:

Then I booted up an EBS-Optimized EC2 instance, built a RAID, and ran fio. Here’s what I saw in the AWS Management Console’s CloudWatch charts after the run. Each volume was delivering 1000 IOPS, as provisioned:

Here’s an excerpt from the results:

fio_test_file: ( groupid= 0, jobs= 32 ): err= 0: pid= 23549: Mon Aug   6 14:01: 14 2012
  read : io=123240MB, bw=94814KB /s, iops= 5925 , runt=1331000msec
    clat (usec ): min= 356 , max= 291546 , avg= 5391.52, stdev= 8448.68
     lat (usec ): min= 357 , max= 291547 , avg= 5392.91, stdev= 8448.68
    clat percentiles (usec ):
      |   1.00 th= [   418 ],   5.00 th= [   450 ], 10.00 th= [   478 ], 20.00 th= [   548 ],
      | 30.00 th= [   596 ], 40.00 th= [   668 ], 50.00 th= [   892 ], 60.00 th= [ 1160 ],
      | 70.00 th= [ 3152 ], 80.00 th= [ 10432 ], 90.00 th= [ 20864 ], 95.00 th= [ 26752 ],
      | 99.00 th= [ 29824 ], 99.50 th= [ 30336 ], 99.90 th= [ 31360 ], 99.95 th= [ 31872 ],
      | 99.99 th= [ 37120 ]

Read the benchmarking guide to learn more about running the benchmarks and interpreting the results.

— Jeff;

 

 

Build Mobile Applications with the Sybase Unwired Platform

Building mobile applications that are able to run on a variety of device types and operating systems is difficult. Building an enterprise application that connect to a variety of data sources is also tough. Combine the two, and you have a difficult challenge, and one that is very relevant in today’s world — the construction of mobile applications that span device types while securely accessing enterprise data.

SAP’s Sybase Unwired Platform (SUP) gives you the tools you need to do just that, and you can now do it on AWS.

To get started, visit the Sybase Unwired Platform Developer Center. You’ll learn how to create your own SUP instance on AWS, and how to access the SUP Mobile SDK that’s already pre-installed on the instance.

— Jeff;

Fast Forward – Provisioned IOPS for EBS Volumes

The I/O Imperative
As I noted earlier this month, modern web and mobile applications are highly I/O dependent, storing and retrieving lots of data in order to deliver a rich and personalized experience.

In order to give you the I/O performance and the flexibility that you need to build these types of applications, we’ve released several new offerings in the last few months:

  • For seamless, managed, scaling of NoSQL workloads, we recently introduced DynamoDB, an SSD-backed NoSQL database with read and write times measured in single-digit milliseconds, with very modest variance from request to request. DynamoDB makes it easy to scale up from 20 to 200,000 or more reads and writes per second and to scale back down again based on your application’s requirements. The response to DynamoDB has been incredible, and it has become (as Werner noted) our fastest-growing service.
  • Two weeks ago, we launched the first member of the EC2 High I/O family,  the h1.4xlarge instance type, to support time series analysis, NoSQL databases, and mobile and streaming applications requiring low latency access to storage systems delivering tens of thousands of IOPS.  The h1.4xlarge comes with 2 TB of SSD-backed storage on each instance.

We know that you want more options for your I/O intensive applications, and we’re happy to oblige.

Here You Go

What Are IOPS?

As you may know, the performance of a block storage device is commonly measured and quoted in a unit called IOPS, short for Input/Output Operations Per Second.

To put the numbers in this post into perspective, a disk drive spinning at 7,200 RPM can perform at 75 to 100 IOPS whereas a drive spinning at 15,000 RPM will deliver 175 to 210. The exact number will depend on a number of factors including the access pattern (random or sequential) and the amount of data transferred per read or write operation. We are focusing on improving the performance and consistency of database-backed applications that run on AWS by adding new EBS and EC2 options.

Here’s what we are announcing today:

  1. A new type of EBS volume called Provisioned IOPS that gives you the ability to dial in the level of performance that you need (currently up to 1,000 IOPS per volume, with more coming soon) . You can stripe (RAID 0) two or more volumes together in order to reach multiple thousands of IOPS.
  2. The ability to launch EBS-Optimized instances which feature dedicated throughput between these instances and EBS volumes.
 

EBS Provisioned IOPS
We released EBS in the summer of 2008. Since that time, our customers have very successfully used EBS to store the persistent data associated with their EC2 instances. We have found that there are certain workloads that require highly consistent IOPS, and others that require more IOPS on an absolute basis. Relational databases certainly qualify on both counts.

As a point of reference, a standard EBS volume will generally provide about 100 IOPS on average, with the ability to burst to hundreds of IOPS on a best-effort basis. Standard EBS volumes are great for applications with moderate or bursty I/O requirements as well as for boot volumes.

The new Provisioned IOPS EBS volume allows you to set the level of throughput that you need, and currently supports up to 1,000 IOPS (for 16K), with higher limits coming soon. For even higher performance, you can stripe multiple Provisioned IOPS volumes together, giving you the ability to deliver thousands of IOPS per logical volume to your EC2-powered application. These volumes deliver consistent performance and are well-suited to database storage, transaction processing, and other heavy random I/O loads. When attached to EBS-Optimized instances, these volumes are designed to deliver within 10% of their provisioned I/O performance 99.9% of the time.

You can create Provisioned IOPS EBS volumes from the AWS Management Console, the command line tools, or via the EC2 APIs. If you use the console, you need only select the Provisioned IOPS volume type and then enter the desired number of IOPS:

Provisioned IOPS volumes are priced at $0.125 per GB of allocated storage per month plus $0.10 per provisioned IOPS per month in US East (Northern Virginia); see the EBS page for more info. By default, each AWS account can create up to 20 TB of Provisioned IOPS volumes with a total of 10,000 IOPS. If you need more of either (or both), simply fill out this form.

You can create a provisioned equivalent of your existing EBS volume by suspending all I/O to your volume, creating a snapshot, and then creating a Provisioned IOPS volume using the snapshot as a starting point.

EBS-Optimized EC2 Instances
For maximum performance and to fully utilize the IOPS provisioned on an EBS volume, you can now request the launch of EBS Optimized EC2 instances. An EBS-Optimized instance is provisioned with dedicated throughput to EBS. The m1.large, m1.xlarge, and m2.4xlarge instance types are currently available as EBS-Optimized instances. m1.large instances can transfer data to and from EBS at a rate of 500 Mbit/second; m1.xlarge and m2.4xlarge instances can transfer data at a rate of 1000 Mbit/second. This is additional throughput, and doesn’t affect other general purpose network throughput already available on the instance.

There is an additional hourly charge for the EBS-Optimized instances: $0.025 for the m1.large and $0.05 for the m1.xlarge and m2.4xlarge instance types.

You can upgrade your EC2 instances to EBS-Optimized instances as follows:

  1. Shut down any applications that are running on the instance.
  2. Stop the instance.
  3. Modify the instance using the ec2-modify-instance-attribute command) and set the EBS-Optimized flag. Change the instance type to one of the supported instance types if necessary.
  4. Start the instances

And Here’s Arun
I spoke with Arun Sundaram, a Product Manager on the AWS Storage team, to learn more about these two features. Here’s what he had to say:

 And That’s That
These new features are available for you to use today. Give them a whirl, and let me know what you think!

— Jeff;

PS – The EBS team is hiring! If you are interested, send your resume to ebs-jobs@amazon.com . Open positions include Software Development Manager, Senior Software Development Engineer,  and Director of Product Management.

The AWS Report – Dave Peck, Co-Founder of Cloak

I recently interviewed Dave Peck, Co-Founder of Seattle-based Cloak. Designed for Macs, iPads, and iPhones, Cloak keeps you safe when you use public Wi-Fi networks in coffee shops, airports, hotels and conferences. It does this by creating an encrypted VPN connection between your device and and an Amazon EC2 instance run and managed by Cloak. The instance is dynamically chosen (using a technique that Dave described as “ping racing”) to minimize latency.

Here’s the interview:

Dave also talks about the false sense of security afforded by certain sites that make use of HTTPS only intermittently, and about how his product can be set up in one minute.

Cloak runs in all of the public AWS regions and they will automatically support new regions as they come online.

We’re currently booking AWS Report guests for late summer and early fall. Write us at awsreport at amazon.com if you are interested.

— Jeff;

 

EC2 Reserved Instances for Red Hat Enterprise Linux

You can now purchase EC2 Reserved Instances running Red Hat Enterprise Linux.

Reserved Instances lower costs by giving you the option to make a low, one-time payment to reserve compute capacity and receive a significant discount on the hourly charge for that instance. Reserved Instances are complementary to existing Red Hat Enterprise Linux On-Demand Instances and give you even more flexibility to reduce computing costs and gain access to a broad range of applications running upon a proven, dependable, and fully supported Linux distribution on the AWS Cloud.

If you want to use Reserved Instances with RHEL, you no longer need to perform additional steps to rebuild On-Demand RHEL AMIs before you can use them on Reserved Instances. Red Hat Enterprise Linux Reserved Instances are available in major versions both in 32-bit and 64-bit architectures, in all Regions except AWS GovCloud.

For technical information, check out the Getting Started Guide: Red Hat Enterprise Linux on Amazon EC2.

For pricing, consult the Amazon EC2 Running Red Hat Enterprise Linux page.

To make it even easier for you to get started, you can get a $50 AWS credit that you can use to launch EC2 instances running Red Hat Enterprise Linux.

 

Jeff;