AWS Developer Blog

Release: AWS SDK for PHP – Version 2.5.0

by Jeremy Lindblom | on | in PHP | Permalink | Comments |  Share

We would like to announce the release of version 2.5.0 of the AWS SDK for PHP. This release adds support for audio profiles in Amazon Elastic Transcoder, geo restriction in Amazon CloudFront, and the recently announced China (Beijing) Region. Please see the official release notes or the CHANGELOG for a full list of changes.

Version 2.5.0 is a major release of the SDK, and contains some minor, backwards-incompatible changes that may affect the way you use some service clients in the SDK. Please read the UPGRADING.md document for details about what has changed and what you may need to do to update your code.

Install the SDK

Taste of JMX Using the AWS SDK for Java

by Hanson Char | on | in Java | Permalink | Comments |  Share

As you may know, starting from Java 5, the JMX technology is available out-of-the-box to monitor and manage the Java VM.  It seems natural to wonder what it would be like to configure and modify the behavior of the AWS Java SDK via JMX in real time.  Imagine, for example, every configuration option related to enabling SDK metrics can also be done via JMX.  The only difference is configuration via system properties is a one-off and static business, whereas configuration via JMX is dynamic, and can be done anytime on-the-fly in real time while the JVM is running!  Since version 1.6.7 of the AWS SDK for Java, you can do exactly that.

Why the need for JMX ?

Here are two use cases I can think of.

  1. You are trying to troubleshoot intermittent latency issues with a running JVM accessing AWS, but didn’t have metrics configured when the JVM was originally started.  Even though you could conceivably shut down, configure the necessary system properties, and then restart the JVM to enable automatic metric generation, you don’t have to!  Via JMX, you can dynamically turn on automatic metric generation and have the metrics uploaded to Amazon CloudWatch for as long as you want, and then have generation turned off afterwards if you desired.  (See sample JConsole screen shots below.)
  2. You are not entirely decided on whether you want to permanently enable metrics, or what the optimal metric configuration would be (for example, per-host-level metrics vs. per-JVM-level metrics).   Via JMX, you can experiment and play around with different options to see what is best for your applications.  Based on actual observation with different options, you can then decide on the best metric configuration and configure the metrics via system properties for long-term use.

To me, it’s simply fun and powerful to be able to manage and administer a fleet of JVMs dynamically on-the-fly, and especially so when remote scripting is involved.  JMX is an enabling technology, and what can be done with the power it offers is only limited by our imagination. 

Example: Metric Configuration via JConsole

Here are the steps to navigate to the Metric Administration MBean via JConsole:

  1. Fire up jconsole.
  2. Connect to a running JVM that accesses AWS via the AWS SDK for Java.
  3. Click the MBeans tab, and expand the name space

         com.amazonaws.management
  4. Click AwsSdkMetrics.
  5. Click Attributes.

Assuming you have no system properties configured for your JVM, you should see all the default values.  Something like:

jmx-attributes

Now, you can modify almost all the attribute values simply by clicking on the value fields. You then change the values directly in-place and press the Enter key.  Almost all changed values will take effect immediately (i.e. without the need to disable/re-enable metric generation).   The only exception is the value of CredentialFile which gets lazily applied as explained below. Here is a sample screenshot of what it looked like after I changed the Region value to "us-west-2", CredentialFile value to "/Users/hchar/.aws/awsTestAccount.properties" and JvmMetricName value to "MyJVM", but before enabling automatic metric generation:

jmx-disabled

To enable automatic metric generation, click Operations to find the enableDefaultMetrics button:

operations

Click the enableDefaultMetrics button to enable automatic metric generation and uploads to Amazon CloudWatch.  Once metric generation is enabled, you should see the values reflected on the Attributes tab:

jmx-enabled

What are the Limitations?

Compared to what you can do with system properties for metric configuration, there are absolutely no limitations!  In other words, for all the one-off metric configuration options that you can make using system properties, such as those described in the Metric Configuration blog, you can achieve the same effect using JMX, but dynamically in real time and as often as you want!

However, if you happened to modify the CredentialFile attribute value, which is the credential file used for uploading metrics to Amazon CloudWatch, you would need to disable and then re-enable the metrics (via invoking the disableMetrics and the enableDefaultMetrics operations) before the credentials would become effective.  This allows the underlying service client to re-initialize itself with the proper credentials.  In JConsole, you can do so by simply clicking the respective buttons on the MBean’s Operations tab.

Finally, have fun using JMX with the AWS SDK for Java, and Merry Christmas!

Ruby 2.0 on AWS OpsWorks

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

We are pleased to announce that AWS OpsWorks now supports Ruby 2.0. Simply select the Ruby version you want, your Rails stack – Passenger or Unicorn, the RubyGems version, and whether you want to use Bundler. Then deploy your app from your chosen repository – Git, Subversion, or bundles on S3. You can get started with a few clicks in the AWS Management console.

Eclipse Support for AWS Elastic Beanstalk Worker Environment Tiers

A web application is typically concerned with quickly sending responses to incoming queries from its users. This model works really well for things like rendering a web page based on a couple of database queries or validating some user input and storing it to a database. The user makes a request to the application to perform some work, the user’s browser waits while the application does the work, and then the application returns a response, which the user’s browser renders to inform him or her of the result of the action.

In some cases, however, servicing a user request requires some serious computation, such as compiling a month-end report based on multiple large scans over the database. The typical web application model gives a poor user experience in this case—the user’s browser simply “spins” with no feedback until the response is completely ready. In the meantime, the job is taking up resources on your front-end web servers, potentially leading to degraded experience for other users if their requests happen to land on the host running the complicated request.

The Solution: Worker Environment Tiers for Elastic Beanstalk

The solution to these problems is to offload the expensive computations to a back-end application tier asynchronously. The user receives a response from the front-end tier as soon as the work item is queued, indicating that the work is in progress. He or she can check back at his or her leisure to see the progress of the request and view the final result once it becomes available. And since the work is being done by a separate set of back-end hosts which are not serving normal user requests, there’s no chance that running this complicated job will negatively impact other customers.

Recently, the fine folks who make AWS Elastic Beanstalk introduced an exciting new feature called worker environment tiers that make it super easy to implement and deploy these kinds of asynchronous background-processing workers within your application. A worker environment tier accepts work requests from your front-end web environment tier via an Amazon SQS queue. All you have to do is write the business logic to be run when a work request is received, deploy it to an Elastic Beanstalk worker environment tier, then start writing asynchronous work requests to the environment’s queue. You can read some more in-depth information about worker environment tiers here.

The AWS Toolkit for Eclipse now includes support for Elastic Beanstalk worker environment tiers, so you can stand up and start deploying code to a worker environment tier with just a few quick clicks. This post will walk you through the process of setting up a new worker environment tier and deploying code to it, all without leaving the comfort of your favorite IDE!

Create a new worker tier project

The easiest way to get started with worker tiers is to create a new AWS Java Web Project. You can find this new project type under the AWS folder in the New Project dialog; it’s the same application type you may have used in the past to create a web application for Elastic Beanstalk. On the New AWS Java Web Project dialog, select the option to start from a Basic Amazon Elastic Beanstalk Worker Tier Application. This will create a basic Java EE Web Application that accepts work requests via an HTTP POST request to a servlet.

  • From the main menu bar, select File -> New -> Project …. Choose the AWS Java Web Project wizard.

The Eclipse New Project Wizard

 

  • Give your project a name and select the Basic Amazon Elastic Beanstalk Worker Tier Application option.

The New AWS Java Web Project Wizard

 

As with the Basic Java Web Application template, the build path for the newly-created project is preconfigured with the latest version of the AWS SDK for Java, so you’re already set up to interact with AWS services like Amazon S3 and Amazon DynamoDB as part of handling a work request. For example, you may want to have your worker write the result of its processing to Amazon S3 for later retrieval by the front-end tier, or use Amazon DynamoDB to track the progress of any work items that are currently being processed.

The example application demonstrates parsing a simple work request from the POST body, simulates doing some complicated work by sleeping for 10 seconds, then writes the “result” of its work to Amazon S3. Take a quick glance at the code to see how it works.

Create a new Elastic Beanstalk worker tier server

Next, we’ll create a new environment through the Eclipse Web Tools Platform’s Servers view. On the first page of the New Server wizard, select the AWS Elastic Beanstalk for Tomcat 7 server type from within the Amazon Web Services folder. After clicking Next, select the AWS region where your application will be hosted, choose Worker Environment from the Environment Type drop-down, and give your new application and environment names.

  • Right-click on the Servers view and select New -> Server.

 

  • Choose the AWS Elastic Beanstalk for Tomcat 7 option.

 

  • Choose a region, application name, environment name, and environment type.

 

On the next page of the wizard, you can modify other optional settings for the environment. If you have an existing SQS queue you would like your worker environment to read from, you can configure that here; otherwise, a new SQS queue will be automatically created for you.

You can also choose to associate an IAM role with your environment on this page, which is an easy way to give your application permission to access AWS resources such as S3. By default, a new role will be created for you that grants your environment permission to access your SQS queues and publish metrics into Amazon CloudWatch. Your environment won’t be able to function correctly if it doesn’t have these permissions, so if you pick a custom role here, make sure it includes those permissions as well.

  • Configure advanced settings for the environment.

 

On the final page, select your worker project to be deployed to the environment.

 

Now that everything is configured, right-click your new environment in the Servers view and start it. This process may take several minutes as Elastic Beanstalk provisions EC2 instances and deploys your application to them. Once the process has completed and your application is displayed as “Started”, your workers are ready to go!

  • Start your environment.

 

By default, your environment starts out with a single worker process running on a t1.micro EC2 instance, and will auto-scale up to as many as four instances if CPU usage on the workers is high. Double-clicking the environment in the Servers view will open up a configuration page where you can tweak these settings, along with many more.

Testing out your new worker

Unlike a traditional web application, your worker environment cannot be invoked directly from within your web browser to debug it. Instead, you need to send work requests to your environment via the SQS queue it is subscribed to. You can do this programmatically (as you will ultimately do from your front-end application):

 

AmazonSQS sqs = ...;

Sring workRequest =
    "{" +
    "  "bucket": "my-results-bucket"," +
    "  "key": "my-work-item-key"," +
    "  "message": "Hello, World"" +
    "}";

sqs.sendMessage(new SendMessageRequest()
    .withQueueUrl(MY_QUEUE_URL)
    .withMessageBody(workRequest));

 

For testing things out, you can also easily send messages to your application by clicking on the Queue URL in the Environment Resources tab of the server editor (available by double-clicking on the newly-created server in the Servers view). This will bring up a dialog allowing you to quickly send work requests to your environment via the SQS queue in order to test out how it handles them.

 

Conclusion

Now you’re all set up to use an Elastic Beanstalk worker tier in your application. Just fill in the appropriate code to handle the different kinds of asynchronous work requests your application requires, and with a couple mouse-clicks your updated code can be deployed out to a fleet of back-end workers running in the cloud. Nifty! Are you deploying code to Elastic Beanstalk (either worker tiers or traditional web server tiers) from within Eclipse? Let us know how it’s working in the comments below!

New Sample Simple Workflow

When you install the SDK from our website, many samples are installed inside Visual Studio, including the Express editions of Visual Studio. Look in the New Project Wizard, where you’ll find samples showing off many of the AWS services.

 

We recently added a new sample that shows off using Amazon Simple Workflow Service (SWF) with the .NET SDK. The sample is under AWS -> App Services section and is called AWS Simple Workflow Image Processing Sample. The sample shows how to use SWF to monitor images coming from S3 and to generate thumbnails of various sizes. In a real-world scenario, this would most likely be done with multiple processes monitoring SWF for decision and activity tasks. This sample is set up as WPF app hosting virtual consoles, each representing an individual process to make it easier to run the sample.

 

The virtual console on the top is the process that chooses an image to generate thumbnails for and starts the workflow execution.

// Snippet from StartWorkflowExecutionProcessor.cs that starts the workflow execution

swfClient.StartWorkflowExecution(new StartWorkflowExecutionRequest
{
    // Serialize input to a string
    Input = Utils.SerializeToJSON(input),
    //Unique identifier for the execution
    WorkflowId = DateTime.Now.Ticks.ToString(),
    Domain = Constants.ImageProcessingDomain,
    WorkflowType = new WorkflowType
    {
        Name = Constants.ImageProcessingWorkflow,
        Version = Constants.ImageProcessingWorkflowVersion
    }
});

 

The virtual console in the bottom left monitors SWF for decision tasks. When it gets a decision task, it looks at the workflow’s history and sees what activities have been completed to figure out which thumbnail hasn’t be created yet. If one of the thumbnail sizes hasn’t been created yet, it schedules an activity to create the next thumbnail sizes. If all the thumbnails have been created, it completes the workflow.

// Snippet from ImageProcessWorkflow.cs that polls for decision tasks and decides what decisions to make.

void PollAndDecide()
{
    this._console.WriteLine("Image Process Workflow Started");
    while (!_cancellationToken.IsCancellationRequested)
    {
        DecisionTask task = Poll();
        if (!string.IsNullOrEmpty(task.TaskToken))
        {
            // Create the next set of decisions based on the current state and
            // the execution history
            List decisions = Decide(task);

            // Complete the task with the new set of decisions
            CompleteTask(task.TaskToken, decisions);
        }
    }
}

 

The virtual console in the bottom right monitors SWF for activity tasks to perform. The activity task will have input from the decider process that tells what image to create a thumbnail for and what size of thumbnail.

// Snippet from ImageActivityWorker.cs showing the main loop for the worker that polls for tasks and processes them.

void PollAndProcessTasks()
{
    this._console.WriteLine("Image Activity Worker Started");
    while (!_cancellationToken.IsCancellationRequested)
    {
        ActivityTask task = Poll();
        if (!string.IsNullOrEmpty(task.TaskToken))
        {
            ActivityState activityState = ProcessTask(task.Input);
            CompleteTask(task.TaskToken, activityState);
        }
    }
}

 

Resource Condition Support in the AWS CloudFormation Editor

AWS CloudFormation recently added support for conditions that control whether resources are created or what value to set for properties on resources. The CloudFormation editor included with the AWS Toolkit for Visual Studio was updated to support conditions in version 1.6.1. If you have never used the CloudFormation editor, we have a screencast that gives a quick introduction to the editor.

Defining Conditions

To get started with conditions, you first need to define them.

In this example, there are 2 conditions defined. The first condition checks to see if the deployment will be a production deployment. The second condition checks to see if a new security group should be created.

Using Conditions to Control Resource Creation

For all resources defined in a template, you can set the Condition property. If the condition evaluates to true, then the resource is created with the CloudFormation stack that is the instantiation of the CloudFormation template.

This security group is created only if the CreateSecurityGroup condition evaluates to true, which occurs if no security group is passed in to the ExistingSecurityGroup parameter.

Using Conditions to Control Resource Properties

You can also use conditions to determine what value to set for a resource property.

Since the security group is going to be either created or set by the ExistingSecurityGroup parameter, the SecurityGroups property needs to have its value set conditionally depending on how the security group was created. Also, in this example, we are going to control the size of the EC2 instance depending on the deployment being a production deployment or not.

For more information about using conditions with CloudFormation, check out the AWS CloudFormation User Guide.

Using AWS CloudTrail in PHP – Part 2

by Jeremy Lindblom | on | in PHP | Permalink | Comments |  Share

This is part 2 of Using AWS CloudTrail in PHP. Part 1 demonstrated the basics of how to work with the CloudTrail service, including how to create a trail and turn logging on and off. Today, I want to show you how to read your log files and iterate over individual log records using the AWS SDK for PHP.

AWS CloudTrail log files

CloudTrail creates JSON-formatted log files containing your AWS API call history and stores them in the Amazon S3 bucket you choose. There is no API provided by CloudTrail for reading your log files, because the log files are stored in Amazon S3. Therefore, you can use the Amazon S3 client provided by the SDK to download and read your logs.

Your log files are stored in a predictable path within your bucket based on the account ID, region, and timestamp of the API calls. Each log file contains JSON-formatted data about the API call events, including the service, operation, region, time, user agent, and request and response data. You can see a full specification of the log record data on the CloudTrail Event Reference page of the CloudTrail documentation.

Log reading tools in the SDK

Even though it is a straightforward process to get your log files from Amazon S3, the SDK provides an easier way to do it from your PHP code. As of version 2.4.12 of the SDK, you can use the LogFileIterator, LogFileReader, and LogRecordIterator classes in the AwsCloudTrail namespace to read the log files generated by your trail.

  • LogFileIterator class – Allows you to iterate over the log files generated by a trail, and can be limited by a date range. Each item yielded by the iterator contains the bucket name and object key of the log file.
  • LogFileReader class – Allows you to read the log records of a log file identified by its bucket and key.
  • LogRecordIterator class – Allows you to iterate over log records from one or more log files, and uses the other two classes.

These classes add some extra conveniences over performing the Amazon S3 operations yourself, including:

  1. Automatically determining the paths to the log files based on your criteria.
  2. The ability to fetch log files or records from a specific date range.
  3. Automatically uncompressing the log files.
  4. Extracting the log records into useful data structures.

Instantiating the LogRecordIterator

You can instantiate the LogRecordIterator using one of the three provided factory methods. Which one you choose is determined by what data is available to your application.

  • LogRecordIterator::forTrail() – Use this if the name of the bucket containing your logs is not known.
  • LogRecordIterator::forBucket() – Use this if the bucket name is known.
  • LogRecordIterator::forFile() – Use this if retrieving records from a single file. The bucket name and object key are required.

If you already know what bucket contains your log files, then you can use the forBucket() method, which requires an instance of the Amazon S3 client, the bucket name, and an optional array of options.

use AwsCloudTrailLogRecordIterator;

$records = LogRecordIterator::forBucket($s3Client, 'YOUR_BUCKET_NAME', array(
    'start_date' => '-1 day',
    'log_region' => 'us-east-1',
));

Iterate over the LogRecordIterator instance allows you to get each log record one-by-one.

foreach ($records as $record) {
    // Print the operation, service name, and timestamp of the API call
    printf(
        "Called the %s operation on %s at %s.n",
        $record['eventName'],
        $record['eventSource'],
        $record['eventTime']
    );
}

NOTE: Each record is yielded as a Guzzle Collection object, which means it behaves like an array, but returns null for non-existent keys instead triggering an error. It also has methods like getPath() and getAll() that can be useful when working with the log record data.

A complete example

Let’s say that you want to look at all of your log records generated by the Amazon EC2 service during a specific week, and count how many times each Amazon EC2 operation was used. We’ll assume that the bucket name is not known, and that the trail was created via the AWS Management Console.

If you don’t know the name of the bucket, but you do know the name of the trail, then you can use the forTrail() factory method to instantiate the iterator. This method will use the CloudTrail client and the trail name to discover what bucket the trail uses for publishing log files. Trails created via the AWS Management Console are named "Default", so if you omit trail_name from the options array, "Default" will be used as the trail_name automatically.

$records = LogRecordIterator::forTrail($s3Client, $cloudTrailClient, array(
    'start_date' => '2013-12-08T00:00Z',
    'end_date'   => '2013-12-14T23:59Z',
));

The preceding code will give you an iterator that will yield all the log records for the week of December 8, 2013. To filter by the service, we can decorate the LogRecordIterator with an instance of PHP’s very own CallbackFilterIterator class.

$records = new CallbackFilterIterator($records, function ($record) {
    return (strpos($record['eventSource'], 'ec2') !== false);
});

NOTE: CallbackFilterIterator is available only in PHP 5.4+. However, Guzzle provides a similar class (GuzzleIteratorFilterIterator) for applications running on PHP 5.3.

At this point, it is trivial to count up the operations.

$opCounts = array();
foreach ($records as $record) {
    if (isset($opCounts[$record['eventName']])) {
        $opCounts[$record['eventName']]++;
    } else {
        $opCounts[$record['eventName']] = 1;
    }
}

print_r($opCounts);

There’s a Part 3, too

In the final part of Using AWS CloudTrail in PHP, I’ll show you how to set up CloudTrail to notify you of new log files via Amazon SNS. Then I’ll use the log reading tools from today’s post, combined with the SNS Message Validator class from the SDK, to show you how to read log files as soon as they are published.

AWS SDK for Ruby v1.30.0

by Loren Segal | on | in Ruby | Permalink | Comments |  Share

Yesterday afternoon, we released a new version of the AWS SDK for Ruby (aws-sdk RubyGem) version v1.30.0. This release:

You can view the release notes here.

Release: AWS SDK for PHP – Version 2.4.12

by Michael Dowling | on | in PHP | Permalink | Comments |  Share

We would like to announce the release of version 2.4.12 of the AWS SDK for PHP.

This release adds support for Amazon Kinesis, adds global secondary indexes support to the Amazon DynamoDB client, and provides several other client updates and fixes.

Install the Latest SDK