AWS Official Blog

We’re adding four new checks to AWS Trusted Advisor. As you may know, AWS Trusted Advisor inspects your AWS environment and looks for ways to save money, increase performance & reliability, and to help close security gaps. Today’s checks are for Elastic Load Balancing, with a focus on security and fault tolerance.

Security Checks
The following new checks are designed to help you to improve the security profile of your Elastic Load Balancers:

ELB Listener Security – This check looks for load balancers that do not use recommended security configurations or protocols. It checks to see if the latest version of applicable security policies are in place and verifies that only recommended ciphers and protocols are used.

ELB Security Groups – This check looks for load balancers that do not have a security group, or that have a security group which allows access to ports that are not configured for the load balancer.

Fault Tolerance Checks
The following new checks are designed to help you to make your Elastic Load Balancing configuration more fault tolerant:

Cross-Zone Load Balancing – This check looks for load balancers that do not have cross-zone load balancing enabled. This feature makes it easier for you to deploy and manage applications that run across more than one Availability Zone.

ELB Connection Draining – This check looks for load balancers that do not have connection draining enabled. With this feature enabled, the load balancer will stop sending new requests to instances that are deregistering (in-flight requests will continue to be served).

Available Now
These new checks are available now and you can start to benefit from them today!

— Jeff;

A few months ago, I discussed 20 new features from Amazon Redshift, our petabyte-scale, fully managed data warehouse service that lets you get started for free and costs as little as $1,000 per TB per year.

The Amazon Redshift team has released over 100 new features since the launch, with a focus on price, performance, and ease of use. Customers are continuing to unlock powerful analytics using the service, as you can see from recent posts by IMS Health, Phillips, and GREE.

My colleague Tina Adams sent me a guest post to share some more Amazon Redshift updates. I’ll let her take over from here!

— Jeff;

I am happy to be able to announce two new Amazon Redshift features today!

The first, custom ODBC and JDBC drivers, now makes it easier and faster to connect to and query Amazon Redshift from your BI tool of choice. The second, Query Visualization in the Console, helps you optimize your queries to take full advantage of Amazon Redshift’s Massively Parallel Processing (MPP), columnar architecture.

Custom Amazon Redshift Drivers
We have launched custom JDBC and ODBC drivers optimized for use with Amazon Redshift, making them easier to use, more reliable, and more performant than drivers available on PostgreSQL.org.

Informatica, Microstrategy, Pentaho, Qlik, SAS, and Tableau will be supporting the new drivers with their BI and ETL solutions. Note that Amazon Redshift will continue to support the latest PostgreSQL ODBC drivers as well as JDBC 8.4-703, although JDBC 8.4-703 is no longer being updated. If you need to distribute these drivers to your customers or other third parties, please contact us at redshift-pm@amazon.com so that we can arrange an appropriate license to allow this.

Our JDBC driver features JDBC 4.1 and 4.0 support, up to a 35% performance gain over open source options, keep alive by default, and improved memory management. Specifically, you can set the number of rows to hold in memory and control memory consumption on a statement by statement basis.

Our ODBC driver is available for Linux, Windows, and Mac OS X. The driver features ODBC 3.8 support, better Unicode data and password handling, default keep alive, and a single-row mode that is more memory efficient than using DECLARE/FETCH. The driver is also backwards compatible with ODBC 2.x, supporting both Unicode and 64-bit applications. The driver includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit.

To connect to Amazon Redshift, find your Cluster Connection String by going to the Cluster tab detail view in the Amazon Redshift Console. For more information please see Connecting to a Cluster.

Query Visualization in the Console
The Console now helps you visualize the time Amazon Redshift spent processing different parts of your query, helping you optimize complex queries more quickly and easily. By going to the Actual tab of the new Query Execution Details section, you can view processing times tied to each stage or “plan node” (e.g. merge, sort, join) of a query execution plan.

This information is pulled from system tables and views, such as STL_EXPLAIN and SVL_QUERY_REPORT. In addition to identifying the parts of your query that took a long time to run, you can also see skew in execution times across your cluster. Plan nodes that caused an alert in the SVL_ALERT_EVENT_LOG system view will show a red exclamation, which you can click to see a recommended solution. For more information please see Analyzing Query Execution.

You can also click on each plan node to view the underlying steps and a detailed comparison of estimated and actual execution time:

Keep Following
There’s a lot more to come from Amazon Redshift, so stay tuned. To get the latest feature announcements, log in to the Amazon Redshift Forum and subscribe to the Amazon Redshift Announcements thread. You can also use the Developer Guide History and the Management Guide History to track product updates.

— Tina Adams, Senior Product Manager

After some fits and starts, I am now recording and producing the AWS Podcast on a regular basis. I am committed to releasing at least one new episode every week, with a stretch goal of two!

Last week I spent some a couple of days at the AWS Pop-up Loft in San Francisco!

With my trusty Zoom H5 recorder nearby, I sat down in the (relatively) quiet basement of the Loft and interviewed a total of six startups and AWS partners. I’ll be editing and publishing them as quickly as possible.

My first guest was Stefano Bellasio of Cloud Academy:

We talked about what Cloud Academy does, and how it relates to AWS.  We also discussed his background, his motivation for founding the company, and his experience coming to the US from Italy.

You can listen to the full interview to learn more. At the end of the interview, Stefano announced a special promotion for fans of the AWS Podcast. Simply visit http://promo.cloudacademy.com/ to receive a 30% discount on the Cloud Academy PRO Plan.

— Jeff;

Version 6.6 of Oracle Linux is now available in AWS Marketplace, with both hourly and annual pricing:

The 64-bit Unbreakable Enterprise Kernel was designed to provide scalability, reliability, and performance improvements for demanding enterprise applications, including Enterprise Oracle workloads.

This is a fully supported way to run Oracle Linux on AWS. Patches and updates are available from Oracle; 24×7 premium support is available through Orbitera.

The AMI (HVM) makes use of EBS volumes for storage and is recommended for use on the c3.large instance type (it can also be run on many other instance types).

To learn more, read the Oracle Linux Product Page and the Oracle Linux FAQ, then Launch Oracle Linux in the AWS Marketplace!

Oracle Linux is available today in the US East (Northern Virginia), US West (Oregon), US West (Northern California), Europe (Ireland), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), and South America (Brazil) regions. Software usage fees start at $0.06 / hour.

— Jeff;

AWS Cost Explorer is a spend management tool that lets you analyze, visualize and allocate your AWS spend to internal teams or projects. Using Cost Explorer, you can visualize your daily or monthly spend in easy-to-read bar and line charts or simple downloadable data tables. You can also choose to see your spend only for specific service types, linked account and tags and group the spend by both service types and linked accounts. Since the first launch of Cost Explorer in April 2014, thousands of customers have signed up to use Cost Explorer and have provided us with feedback about how easy and convenient it has become to analyze their AWS spend. They have also provided us with many suggestions for further improvements!

Group Spend By Tags
Last summer we added the ability to group your AWS spending by tag (this was a frequently requested feature, but one that I neglected to blog about at the time). Many of our customers attach tags to resources to track usage by internal departments or groups (such as setting the Environment tag to either Development or Production to allow separation of development and operational costs) and want to group the billing data in the Cost Explorer with these name/value pairs to easily perform cost allocation. To take advantage of this feature, you simply select the tag key that you would like to group your costs by (Environment):

Cost Explorer interface then displays costs broken down by all values applicable for that tag key in one report:

More Filtering and Grouping Dimensions
Today, based in large part on the response to last year’s launch, we are adding additional options for filtering and grouping your costs. In addition to the existing groupings (Service and Linked Account) you can now group by Availability Zone, Purchase Option (Reserved or Non-Reserved), and API Operation. You can also filter by multiple tag keys at the same time. You can use these new options to gain additional insights in to your AWS costs and the cost drivers behind them.

Here’s the newly updated Filter menu:

After you select a filtering option you can choose the value to show. For example, I can filter by API Operation and choose individual AWS API functions:

Or by Availability Zone:

Multiple filters can be used to further control the display. For example, I can look at costs for all development resources in a single Availability Zone:

Available Now
These features are available now and you can start using them today!

— Jeff;

Let’s take a quick look at what happened in AWS-land last week:

Upcoming Events

• Thursday, February 26 – Webinar -The Perfect Storm – Big Data Analytics in the Cloud with AWS – APN Partner Logi Analytics.
• Tuesday, March 3 – Webinar – Loading and Analyzing Behavioral Data in Amazon Redshift – APN Partner Segment.
• Thursday, March 5 – Webinar – Oracle EBS Migration from a Hosted, Third-Party Data Center to AWS – APN Partner Apps Associates.
• Wednesday, March 18 – Webinar – Learn how Echo360 Moved Workloads to AWS with Zadara Storage – Echo360 and APN Partner  Zadara Storage.
• AWS Summits.

Help Wanted

• Cloud Architects with Passion (Cloud Academy).
• AWS Careers.

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

— Jeff;

A recent Reddit thread discussed the topic of Keeping Your AWS Accounts Squeaky Clean. The discussion focused on ways to identify (and possibly terminate) AWS resources that were not properly tagged.  I had to keep my lips sealed as I read through the discussion because I knew that the features that I am about to describe were just about to launch!

We launched the Tag Editor and Resource Groups last December. The Tag Editor allows you to easily find and tag AWS resources; Resource Groups simplify the process of working on collections of AWS resources that share the same tags. You can invoke both of these helpful tools from the Console’s AWS menu:

Negative Tag Searching
Many organizations use tags to track resources for billing and ownership purposes. When this is the case, it is a good idea to make sure that all of the resources within an account are tagged in accordance with the organization’s requirements. You can use the Tag Editor’s new negative tag search to do this. You can now search for resources that do not have a desired tag. For example, suppose that I want to find all EC2 instances that do not have a Department tag. I choose the desired regions and resource type, choose the Department tag, and pick Not tagged from the menu. Then I click on the Find Resources button:

Empty Tag Searching
I can also search for tags that have an empty value by choosing Empty value from the menu:

Available Now
This feature is available now and you can start using it today.

— Jeff;

EC2 Container Service  (ECS) is a highly scalable, high performance container management service that supports Docker containers and allows you to easily run distributed applications on a managed cluster of Amazon EC2 instances.

My colleague Chris Barclay sent a guest post to spread the word about two additions to the service. As Chris explains below, you can use images stored in private Docker repositories. You can also store and share information between containers using data volumes from the host. Let’s see what Chris has to say!

— Jeff;

Use Images Stored in Private Docker Repositories
The Amazon ECS agent can now authenticate with Docker registries, including Docker Hub. Registry authentication lets you use Docker images from private repositories in your Task Definitions. Here’s how to set it up:

1. Create a private S3 object named ecs.config in an S3 bucket with your repository’s credentials (you can get the credentials from your .dockercfg file):

   ECS_ENGINE_AUTH_TYPE=dockercfg
   ECS_ENGINE_AUTH_DATA={"https://index.docker.io/v1/":{"auth":"YOUR_AUTH_CODE","email":"email@example.com"}}
   
   

2. Add a policy to the IAM role used by the Container Instances in your ECS cluster to provide access to the object created in step 1:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Action": [
           "s3:GetObject"
         ],
       "Sid": "Stmt0123456789",
       "Resource": [
         "arn:aws:s3:::YOUR_BUCKET/YOUR_OBJECT"
       ],
       "Effect": "Allow"
       }
     ]
   }
   

3. When launching an EC2 instance, in the Advanced Details drop-down paste the following script into the User data text box:

   #!/bin/bash
   yum install -y aws-cli
   aws s3 cp s3://YOUR_BUCKET/YOUR_OBJECT /etc/ecs/ecs.config
   

The container instances launched in step 3 can now pull private images referenced in an ECS Task Definition.

Mount Volumes in Containers
ECS Task Definitions now provide a way for containers to store and share information using data volumes. For data that should persist between tasks, such as a download cache, you can reference a location on the host as shown in the following volume definition:

"volumes": [
   {
    "name": "cache",
    "host": {
      "sourcePath": "/var/lib/MyApp/cache/"
    }
  } ]

You can then reference the volume by name and specify the path to mount the volume in the container definition:

"containerDefinitions":[
  {
    "name": "webserver",
     ...
    "mountPoints": [
    {
      "sourceVolume": "cache",
      "containerPath": "/usr/src/app/cache"
    } ]
    ...

If you don’t need your data to persist between task runs then you can specify an “empty” volume. By letting Docker manage the host storage, you don’t need to worry about creating or deleting the volume on the host; Docker creates a volume that persists for the lifetime of the task. Here is a Task Definition snippet that creates a volume that is managed by Docker:

"volumes": [
    {
      "name": "logs",
      "host": {}
    }

Docker also supports the ability to share volumes between containers. For example, you may take the logs your Apache server writes to /var/log/www and push them to a central repository using a cron job running in the Apache container. Another option is to create a backup job container with a backup daemon that references the shared logs volume using the volumesFrom attribute. Now the backup daemon can tar the logs in the shared volume periodically and store them in Amazon Simple Storage Service (S3). Here’s how you would reference a shared volume:

"containerDefinitions":[
  {
    "name": "backupManager",
     ...    
    "volumesFrom": [
        {
            "sourceContainer": "Apache",
            "readOnly": true
        }
      ]
    }

Available Now
These features are available now and you can start using them today (to borrow Jeff’s phrase). For more information, read the documentation on private Docker repositories and mounting volumes in containers.

— Chris Barclay, Principal Product Manager

As more companies are adopting cloud computing, they are also establishing DevOps practices within their organizations as a way to provide stable, secure, and predictable IT environments and increase IT efficiency.

When I talk to our customers they tell me that they are looking for IT professionals with cloud computing expertise, with a special focus on DevOps skills.  Last November we announced a new AWS Certification for the AWS Certified DevOps Engineer – Professional to help IT professionals validate their skills in the DevOps area and to allow employers to identify qualified candidates to lead DevOps initiatives.

Now Available
Today, we have released the exam publicly.  We have also awarded our first DevOps Engineer certifications to those who successfully completed the beta exam.

The AWS Certified DevOps Engineer – Professional certification is the next step in the path for AWS Certified Developers and SysOps Administrators.  It validates technical expertise in provisioning, operating, and managing distributed application systems on the AWS platform.  The exam is intended to identify individuals who are capable of implementing and managing systems which are highly available, scalable, and self-healing on the AWS platform. You must already be certified as an AWS Certified Developer – Associate or AWS Certified SysOps Administrator – Associate before you are eligible to take this exam.  Find the Exam Guide, a practice exam, and other resources at aws.amazon.com/certification.

Congratulations to everyone who has already earned their AWS Certified DevOps Engineer – Professional certifications!

— Jeff;

AWS Elastic Beanstalk simplifies the process of deploying and scaling Java, .NET, PHP, Node.js, Python, Ruby, Go, and Docker web applications and services.

Today we are making Elastic Beanstalk even more useful by adding three new features:

1. Easy cloning of environments.
2. 1-Click IAM role creation.
3. Support for periodic worker tasks.

Let’s take a look at each of these new features!

Easy Cloning of Environments
You can now use an existing Elastic Beanstalk environment as the basis for a new one by simply cloning and then editing it. Cloning an environment also creates copies of any AWS resources associated with it (it does not, however copy data stored in Amazon RDS to the clone).  As part of the cloning process you have the opportunity to update the Environment Name, Environment URL, Description and Platform values to reflect the new environment. You will use the same language and framework when you update the Platform and can choose a newer version of the same solution stack.

You can initiate the cloning process from the Elastic Beanstalk Console or from the Elastic Beanstalk command line, better known as eb (documentation). Here’s how you do it from the Console:

1-Click IAM Role Creation
IAM one-click role creation is a feature that various AWS services have adopted in order to streamline the process of creating IAM roles and granting those roles permissions. One-click role creation appears in Elastic Beanstalk as part of the workflow for creating new applications or launching new environments.

Creating roles with a single click is simple and straightforward, and does not require you to use the IAM Console. You simple create the role and configure the policy without leaving the Beanstalk workflow. You can even edit the policy directly.

Here’s what this feature looks like:

Periodic Worker Tasks
The Elastic Beanstalk Worker Tier is designed to host long-running processes such as reporting, analytics, and database cleanup. Workers are simply HTTP request handlers that are triggered by an Amazon Simple Queue Service (SQS) message.

You can now configure Elastic Beanstalk to send messages to a queue periodically. The message is delivered as an HTTP POST to a configurable URL on the local host; the HTTP header will contain the name of the periodic task.

Your Worker Tier can consist of a single EC2 instance or an auto-scaled, load-balanced set of EC2 instances.

In order to invoke periodic tasks, your application source bundle must include a cron.yaml file at the root level. The file must contain information about the periodic tasks you want to schedule. Specify this information using standard crontab syntax.

Available Now
These features are available now and you can start using them today!

— Jeff;