AWS Developer Blog

Customizing Windows Elastic Beanstalk Environments – Part 2

by Jim Flanagan | on | in .NET | Permalink | Comments |  Share

In the previous post in this series, we introduced the .ebextensions/*config file, and showed how you can use it to install packages, download files, run commands, and start services.

In this post, we’re going to dig a little bit into managing settings through this mechanism.

Writing configuration files

A common way to configure software systems is through text-based configuration. Elastic Beanstalk gives a couple of ways for us to write files in the filesystem that are not necessarily part of our web application, and which may live outside the web application directory. Files can be downloaded from a web-accessible place (such as an S3 bucket) or inlined in the .ebextensions/*.config file directly.

files:
   c:/MyApplicationSupport/main.conf:
     content: |
       <configuration>
         <environment>production</environment>
         <maxConnections>500</maxConnection>
         <defaultUser>guest</defultUser>
       <configuration>
   c:/MyApplicationSupport/auxilliary.conf:
     source: http://my-application-support/auxilliary.conf

The first file in the files: array shows an example of inlining.

Inlining can be handy for situations where you are adjusting content frequently—for example, during development when you are deploying your application more often. Using the source: key requires uploading any changes before deployment, so that’s a better method for more complex, or less volatile files.

Variable interpolation

Another benefit to inlining is that you can use the AWS CloudFormation intrinsic functions to interpolate information from the CloudFormation template associated with your Elastic Beanstalk environment into your configuration files.

Here are some examples of interpolation:

files:
   c:/cfn/environmentInfo.txt:
     content : |
       Environment Name: `{"Ref": "AWSEBEnvironmentName" }`
       Environment Id:   `{"Ref": "AWSEBEnvironmentId" }`
       Instance Type:    `{"Ref": "InstanceType" }`
       Stack Id:         `{"Ref": "AWS::StackId" }`
       Region:           `{"Ref": "AWS::Region" }`
       AMI Id:           `{"Fn::FindInMap": [ "AWSEBAWSRegionArch2AMI", { "Ref": "AWS::Region" }, { "Fn::FindInMap": [ "AWSEBAWSInstanceType2Arch", { "Ref": "InstanceType" }, "Arch" ]}]}`

For more ideas about what can be extracted from the CloudFormation template, you can inspect the template for your environment using the AWS Toolkit for Visual Studio. To do that, simply add a project of type AWS CloudFormation to your solution. When you create the project, you will be prompted with a dialog to choose a template source. Choose Create from existing AWS CloudFormation Stack, choose the correct account, region, and the stack associated with your environment, and then click Finish.

Uploading Archives to Amazon Glacier from PHP

by Jeremy Lindblom | on | in PHP | Permalink | Comments |  Share

You can easily upload your data archives to Amazon Glacier by using the Glacier client included in the AWS SDK for PHP. Similar to the Amazon S3 service, Amazon Glacier has an API for both single and multipart uploads. You can upload archives of up to 40,000 GB through the multipart operations. With the UploadArchive operation, you can upload archives of up to 4 GB in a single request; however, we recommended using the multipart operations for archives larger than 100 MB.

Before we look at how to use the specific operations, let’s create a client object to work with Amazon Glacier.

use AwsGlacierGlacierClient;

$client = GlacierClient::factory(array(
    'key'    => '[aws access key]',
    'secret' => '[aws secret key]',
    'region' => '[aws region]', // (e.g., us-west-2)
));

Uploading an archive in a single request

Now let’s upload some data to your Amazon Glacier vault. For the sake of this and other code samples in this blog post, I will assume that you have already created a vault and have stored the vault name in a variable called $vaultName. I’ll also assume that the archive data you are uploading is stored in a file and that the path to that file is stored in a variable called $filename. The following code demonstrates how to use the UploadArchive operation to upload an archive in a single request.

$result = $client->uploadArchive(array(
    'vaultName' => $vaultName,
    'body'      => fopen($filename, 'r'),
));
$archiveId = $result->get('archiveId');

In this case, the SDK does some additional work for you behind the scenes. In addition to the vault name and upload body, Amazon Glacier requires that you provide the account ID of the vault owner, a SHA-256 tree hash of the upload body, and a SHA-256 content hash of the entire payload. You can manually specify these parameters if needed, but the SDK will calculate them for you if you do not explicitly provide them.

For more details about the SHA-256 tree hash and SHA-256 content hash, see the Computing Checksums section in the Amazon Glacier Developer Guide. See the GlacierClient::uploadArchive API documentation for a list of all the parameters to the UploadArchive operation.

Uploading an archive in parts

Amazon Glacier also allows you to upload archives in parts, which you can do using the multipart operations: InitiateMultipartUpload, UploadMultipartPart, CompleteMultipartUpload, and AbortMultipartUpload. The multipart operations allow you to upload parts of your archive in any order and in parallel. Also, if one part of your archive fails to upload, you only need to reupload that one part, not the entire archive.

The AWS SDK for PHP provides two different techniques for doing multipart uploads with Amazon Glacier. First, you can use the multipart operations manually, which provides the most flexibility. Second, you can use the multipart upload abstraction which allows you to configure and create a transfer object that encapsulates the multipart operations. Let’s look at the multipart abstraction first.

Using the multipart upload abstraction

The easiest way to perform a multipart upload is to use the classes provided in the AwsGlacierModelMultipartUpload namespace. The classes provide an abstraction of the multipart uploading process. The main class you interact with is UploadBuilder. The following code uses the UploadBuilder to configure a multipart upload using a part size of 4 MB. The upload() method executes the uploads and returns the result of the CompleteMultipartUpload operation at the end of the upload process.

use AwsGlacierModelMultipartUploadUploadBuilder;

$uploader = UploadBuilder::newInstance()
    ->setClient($client)
    ->setSource($filename)
    ->setVaultName($vaultName)
    ->setPartSize(4 * 1024 * 1024)
    ->build();

$result = $uploader->upload();

$archiveId = $result->get('archiveId');

Using the UploadBuilder class, you can also configure the parts to be uploaded in parallel by using the setConcurrency() method.

$uploader = UploadBuilder::newInstance()
    ->setClient($client)
    ->setSource($filename)
    ->setVaultName($vaultName)
    ->setPartSize(4 * 1024 * 1024)
    ->setConcurrency(3) // Upload 3 at a time in parallel
    ->build();

If a problem occurs during the upload process, an AwsCommonExceptionMultipartUploadException is thrown, which has access to a TransferState object that represents the state of the upload.

try {
    $result = $uploader->upload();
    $archiveId = $result->get('archiveId');
} catch (AwsCommonExceptionMultipartUploadException $e) {
    // If the upload fails, get the state of the upload
    $state = $e->getState();
}

The TransferState object can be serialized so that the upload can be completed in a separate request if needed. To resume an upload using a TransferState object, you must use the resumeFrom() method of the UploadBuilder.

$resumedUploader = UploadBuilder::newInstance()
    ->setClient($client)
    ->setSource($filename)
    ->setVaultName($vaultName)
    ->resumeFrom($state)
    ->build();

$result = $resumedUploader->upload();

Using the multipart operations

For the most flexibility, you can manage all of the upload process yourself using the individual multipart operations. The following code sample shows how to initialize an upload, upload each of the parts one by one, and then complete the upload. It also uses the UploadPartGenerator class to help calculate the information about each part. UploadPartGenerator is not required to work with the multipart operations, but it does make it much easier, especially for calculating the checksums for each of the parts and the archive as a whole.

use AwsGlacierModelMultipartUploadUploadPartGenerator;

// Use helpers in the SDK to get information about each of the parts
$archiveData = fopen($filename, 'r');
$partSize = 4 * 1024 * 1024; // (i.e., 4 MB)
$parts = UploadPartGenerator::factory($archiveData, $partSize);

// Initiate the upload and get the upload ID
$result = $client->initiateMultipartUpload(array(
    'vaultName' => $vaultName,
    'partSize'  => $partSize,
));
$uploadId = $result->get('uploadId');

// Upload each part individually using data from the part generator
foreach ($parts as $part) {
    fseek($archiveData, $part->getOffset())
    $client->uploadMultipartPart(array(
        'vaultName'     => $vaultName,
        'uploadId'      => $uploadId,
        'body'          => fread($archiveData, $part->getSize()),
        'range'         => $part->getFormattedRange(),
        'checksum'      => $part->getChecksum(),
        'ContentSHA256' => $part->getContentHash(),
    ));
}

// Complete the upload by using data aggregated by the part generator
$result = $client->completeMultipartUpload(array(
    'vaultName'   => $vaultName,
    'uploadId'    => $uploadId,
    'archiveSize' => $parts->getArchiveSize(),
    'checksum'    => $parts->getRootChecksum(),
));
$archiveId = $result->get('archiveId');

fclose($archiveData);

For more information about the various multipart operations, see the API documentation for GlacierClient. You should also take a look at the API docs for the classes in the MultipartUpload namespace to become more familiar with the multipart abstraction. We hope that this post helps you work better with Amazon Glacier and take advantage of the low-cost, long-term storage it provides.

Customizing Windows Elastic Beanstalk Environments – Part 1

by Jim Flanagan | on | in .NET | Permalink | Comments |  Share

AWS Elastic Beanstalk recently announced support for customizing Windows environments with configuration files. Before this, the only way to customize a .NET container was to create a custom AMI in each region you wanted to run your application in.

Adding a configuration file to your application allows you to

  • install packages
  • write files in locations other than the application folder
  • execute commands, and run scripts
  • start services
  • set configuration options

    • for your application
    • for the Elastic Beanstalk environment.

Let’s walk through a simple example to show how it’s done.

Installing packages and starting custom services

Our hypothetical application MyApp relies on a third-party package called WebMagicThingy, which is packaged as an MSI. In addition, we have written a Windows service that periodically performs maintenance and cleanup operations on the host for our application. We want to install and start that service on each instance in our Elastic Beanstalk environment.

The initial step is to make our service code and the third-party package available on the web. We’ll put them in an S3 bucket called my-app-support.

my-app-support/WebMagicThingy.msi
my-app-support/MyAppJanitor.zip

Next, we’ll create a folder in our application called .ebextensions, and in that folder create a file called MyAppSupport.config. The .ebextensions folder can contain more than one file with a .config extension. You can either include these files in your project, or alternatively you can select All Files in the Project Folder for the Items to deploy option on the Package/Publish Web tab of the project properties pane to ensure that they get included in the deployment bundle.

The format of the configuration files is YAML. Visual Studio expects files with a .config extension to be XML files that conform to a specific schema, so it may be easier to create these files in an external editor, then include them in the project. Ours will look like this:

 packages:
   msi:
     WebMagicThingy: http://s3.amazonaws.com/my-app-support/WebMagicThingy.msi
 sources:
   c:/AppSupport/MyAppJanitor: http://s3.amazonaws.com/my-app-support/MyAppJanitor.zip
 commands:
   install-janitor:
     command: C:\Windows\Microsoft.NET\Framework\v4.0.30319\installutil MyAppJanitor.exe
     cwd: c:/AppSupport/MyAppJanitor
     waitForCompletion:0
 services:
   windows:
     MyAppJanitor:
       enabled: true
       ensureRunning: true
       commands: install-janitor

Each level of indentation is two spaces. Take care that your editor doesn’t replace consecutive spaces with tabs, or the file will not be interpreted correctly, and Elastic Beanstalk will stop the deployment with an error.

This configuration does four things on each instance in the Elastic Beanstalk environment:

  • Installs the MSI that we put in our S3 bucket earlier
  • Expands the MyAppJanitor.zip from the S3 bucket to the location C:AppSupportMyAppJanitor
  • Runs installutil.exe to install the MyAppJanitor service

    • The cwd: directory allows you to specify what folder the command is run in
    • If waitForCompletion is not specified for a command, the container will wait for 60 seconds by default.
  • Makes sure that the service is started

You may notice inconsistent use of path separators and escaping in various places in the file. Most directives can use forward-slash as the path separator, but invoking commands that are not in the PATH requires escaped backward-slash path separators.

In upcoming posts, we’ll explore other ways to customize Windows Elastic Beanstalk environments with the .ebextensions mechanism. In the meantime, you can explore the Elastic Beanstalk documentation on the topic and see what things you come up with.

Release: AWS SDK for Java 1.5.0

by Jason Fulghum | on | in Java | Permalink | Comments |  Share

We released version 1.5.0 of the AWS SDK for Java last night.  This release contains several exciting enhancements including:

  • Upgrading to the latest major version of Apache HttpClient
  • Support for the Closeable interface on Amazon S3 objects
  • Easier construction of requests that use map datatypes
  • Batching improvements for Amazon DynamoDB
  • Support for Amazon Elastic Transcoder’s latest API version, enabling support for frame rate options and watermarks. 

We’ll be digging into some of these features more in upcoming blog posts.

For more details, see the full release notes.

Release: AWS SDK for PHP 2.4.1

by Michael Dowling | on | in PHP | Permalink | Comments |  Share

We would like to announce the release of version 2.4.1 of the AWS SDK for PHP. This release adds support for setting watermark and max frame rate parameters in the Amazon Elastic Transcoder client and resolves issues with the Amazon S3, Amazon EC2, Amazon ElastiCache, AWS Elastic Beanstalk, Amazon EMR, and Amazon RDS clients.

Changelog

  • Added support for setting watermarks and max frame rates to the Amazon Elastic Transcoder client
  • Added MD5 validation to Amazon SQS ReceiveMessage operations
  • Added the AwsDynamoDbIteratorItemIterator class to make it easier to get items from the results of DynamoDB operations in a simpler form
  • Added support for the cr1.8xlarge EC2 instance type. Use AwsEc2EnumInstanceType::CR1_8XLARGE
  • Added support for the suppression list SES mailbox simulator. Use AwsSesEnumMailboxSimulator::SUPPRESSION_LIST
  • Fixed an issue with data formats throughout the SDK due to a regression. Dates are now sent over the wire with the correct format. This issue affected the Amazon EC2, Amazon ElastiCache, AWS Elastic Beanstalk, Amazon EMR, and Amazon RDS clients
  • Fixed an issue with the parameter serialization of the ImportInstance operation in the Amazon EC2 client
  • Fixed an issue with the Amazon S3 client where the RoutingRules.Redirect.HostName parameter of the PutBucketWebsite operation was erroneously marked as required
  • Fixed an issue with the Amazon S3 client where the DeleteObject operation was missing parameters
  • Fixed an issue with the Amazon S3 client where the Status parameter of the PutBucketVersioning operation did not properly support the "Suspended" value
  • Fixed an issue with the Amazon Glacier UploadPartGenerator class so that an exception is thrown if the provided body to upload is less than 1 byte

Install/Download the SDK

You can get the latest version of the SDK via:

Release v1.12.0

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

We just released v1.12.0 of the AWS SDK for Ruby  (aws-sdk gem).  This release includes the new aws-rb REPL that Loren bloged about. It also adds support for watermarks and max frame rates in Amazon Elastic Transcoder, resolves a number of issues, and it adds a few new configuration options.

We are slowly deprecating all of the service prefixed configuration options.  This release makes it easier as you can now group configuration options by the service.

# deprecated
AWS.config(:s3_region => 'us-west-2')

# new format
AWS.config(:s3 => {
  :region => 'us-west-2'
})

We are going to make it possible to set any configuration options per service as well. This will allow you to do things like specify a greater value for max retries for a single service.

You can read the release notes here.

Output Pagination with AWS Tools for PowerShell

by Pavel Safronov | on | in .NET | Permalink | Comments |  Share

Amongst the changes to the cmdlets in version 1.1 of the AWS Tools for Windows PowerShell are added support for both automatic and manual pagination of the output from services to the pipeline. Most of the time, you’ll probably want to use automatic paging to get all the data from a cmdlet, but on occasion you may prefer to control this yourself.

Automatic Pagination

Making use of automatic output pagination is simple—run the cmdlet with no paging parameters:

"mybucket" | Get-S3Object

The output from this to the pipeline will be zero or more S3Object instances. By default Amazon S3 returns a maximum of 1000 items per call, so the cmdlet keeps calling on your behalf and emitting each batch of results to the pipe until S3 signals that there is no more content.

Here’s another example of using automatic pagination in conjunction with the improved pipeline support, in this case to get the size (in GB in this example) of a bucket:

((Get-S3Object -BucketName mybucket).Size | Measure-Object -Sum).Sum / 1024 / 1024 / 1024

Automatic pagination doesn’t just work with Amazon S3:

Get-CFNStack | Get-CFNStackResources

In this example, Get-CFNStack enumerates all of your AWS CloudFormation stacks (in the current region set for the shell) and emits each to the downstream Get-CFNStackResources cmdlet to get additional resource information on the stack.

If you’re curious, or need to diagnose a problem, you can see the repeated calls that the cmdlets make to obtain a full set of results by inspecting the entries in the $AWSHistory shell variable (this is also new with version 1.1). You can also it to see how the cmdlets track the ‘next page’ marker for each service call on your behalf.

Controlling Pagination Yourself

Most of the time, automatic pagination can be used without further thought. However, there may be occasions when you are dealing with huge data sets in a memory-constrained environment. In these cases, you may elect to control the pagination yourself. The names of the parameters to use to control paging do vary by service but are usually called NextMarker, Marker, NextToken, and so on.

The cmdlets for S3 use parameters called MaxKeys and Marker to control the iteration (and NextMarker and IsTruncated fields in the service response to indicate where the next page of results can be fetched, if any). The first example in this post can be rewritten like this to fetch all the S3Object instances in batches of 50 at a time:

$nextMarker = $null
$keysPerPage = 50
do
{
	$objects = Get-S3Object "mybucketname" -KeyPrefix / -MaxKeys $keysPerPage -Marker $nextMarker
	$nextMarker = $AWSHistory.LastServiceResponse.NextMarker
        # do something with the batch of results in $objects here
} while ($nextMarker)

When you handle paging yourself, be sure to capture the ‘next marker’ token from the response straight away and before you make another call to AWS; otherwise, the response that $AWSHistory.LastServiceResponse points to may not be what you think it is.

Similar patterns can be followed for other services that support paginated result sets.

Parameter Aliases

As mentioned above, Amazon S3 cmdlets use MaxKeys and Marker parameters to expose pagination, mirroring the underlying service API. Other services use different names (NextToken for example, or MaxRecords). This can be difficult to remember, especially if you’re unfamiliar with the service. To help, we add aliases to the parameters so that you can expect a consistent naming scheme, regardless of the service. These aliases are NextToken and MaxItems. MaxItems, by the way, is the maximum number of items to emit to the pipeline, which may be more or less than the underlying service’s maximum per call.

Using aliases, these two commands are the same:

Get-S3Object mybucket -KeyPrefix / -Marker "token2" -MaxKeys 200
Get-S3Object mybucket -KeyPrefix / -NextToken "token2" -MaxItems 200

Summary

In this post, we covered how to make use of the automatic results pagination now available in the AWS PowerShell cmdlets and how to take control and perform the pagination yourself using parameters and aliases for a uniform experience across services. It also showed how to access additional information from the service responses being logged by the new $AWSHistory shell variable.

AWS Service Provider for Laravel 1.0.4

by Jeremy Lindblom | on | in PHP | Permalink | Comments |  Share

We would like to announce the availability of version 1.0.4 of the AWS Service Provider for Laravel. This version includes the AwsLaravelAwsFacade class which allows you to register an AWS facade in your Laravel 4 project, so you can retrieve clients in an easy and idiomatic way (e.g., $s3 = AWS::get('s3');).

We would also like to remind you about the other framework-specific packages that we currently support: the AWS SDK ZF2 Module and the AWS Service Provider for Silex. In the future, we will also post announcements on the blog when we update these packages.

Ruby 1.8 End of Life Plan

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

You have probably heard that Ruby 1.8.7 has officially reached it’s end of life. This makes it important for us to discuss what our plans will be for the AWS SDK for Ruby (aws-sdk gem) with regards to Ruby 1.8.

We currently support as far back as Ruby 1.8.7. There are now additional considerations with the passing of the end of life date. We would love to hear your feedback. There is an issue on our GitHub project where the discussion has already started. Please join in and share your input!

Instance Status Checks with the AWS SDK for .NET

by Wade Matveyenko | on | in .NET | Permalink | Comments |  Share

A question that we have heard from our customers is, "How do we get access to the Amazon EC2 instance status checks?" If you go the AWS Management Console, you can easily see those status checks displayed. The Amazon EC2 DescribeInstanceStatus API action returns the instance status for one or more EC2 instances. This code shows you how to make that call in the AWS SDK for .NET:

var instanceId = "yourInstanceIdHere";
var ec2Client = new AmazonEC2Client(RegionEndpoint.USWest2);
var statusRequest = new DescribeInstanceStatusRequest
{
    InstanceId = { instanceId }
};
var result = ec2Client.DescribeInstanceStatus(statusRequest).DescribeInstanceStatusResult;

This block of code returns an InstanceStatusResult. Inside of this object is an array of InstanceStatus objects. You get one ‘InstanceStatus’ for each instance that you asked for. In this example, since we asked for only one instance, there is only one element in that array. You access this object by retrieving result.InstanceStatus[0]; Status checks are separated into two types: system status checks and instance status checks. For more information about the types of status checks, see the EC2 documentation— Monitoring Instances with Status Checks. This code shows how to access the EC2 instance status checks and writes the output to the console:

//Get the instance status checks
Console.WriteLine("Instance Status = " +
    status.InstanceStatusDetail.Status);
Console.WriteLine("Instance Status Detail Name = " +
    status.InstanceStatusDetail.Detail[0].Name);
Console.WriteLine("Instance Status Detail Status = " +
    status.InstanceStatusDetail.Detail[0].Status;);

//Get the system status checks
Console.WriteLine("System Status = " +
    status.SystemStatusDetail.Status);
Console.WriteLine("System Status Detail Name = " +
    status.SystemStatusDetail.Detail[0].Name);
Console.WriteLine("System Status Detail Status = " +
    status.SystemStatusDetail.Detail[0].Status);

The output for this looks like:

Instance Status = ok
Instance Status Detail Name = reachability
Instance Status Detail Status = passed
System Status = ok
System Status Detail Name = reachability
System Status Detail Status = passed

The current status check API has only one check, so the Detail property contains a single element. The Name and Status properties contain the name of the check, currently only reachability, and the status of the check, which can contain passed | failed | insufficient-data. There is another property, ImpairedSince, which would contain a String of when the status check failed. If the status check was not a failure, this property will be empty. Now that you know how to use instance status checks, how will you use them? Let us know in the comments!