AWS Developer Blog

Streaming Amazon S3 Objects From a Web Server

by Michael Dowling | on | in PHP | Permalink | Comments |  Share

Have you ever needed a memory-efficient way to stream an Amazon S3 object directly from your web server to a browser? Perhaps your website has its own authorization system and you want to limit access to a file to only users who have purchased it. Or maybe you need to perform a specific action each time a file is accessed (e.g., add an image to a user’s "recently viewed" list).

Using PHP’s readfile function and the Amazon S3 stream wrapper provides a simple way to efficiently stream data from Amazon S3 to your users while proxying the bytes sent over the wire through a web server.

Register the Amazon S3 stream wrapper

First you need to create an Amazon S3 client:

use AwsS3S3Client;

$client = S3Client::factory(array(
    'key'    => '****',
    'secret' => '****'
));

Next you need to register the Amazon S3 stream wrapper:

$client->registerStreamWrapper();

Send the appropriate headers

Now you need to send the appropriate headers from the web server to the client downloading the file. You can specify completely custom headers to send to the client, including any relevant headers of the Amazon S3 object.

Here’s how you could retrieve the headers of a particular Amazon S3 object:

// Send a HEAD request to the object to get headers
$command = $client->getCommand('HeadObject', array(
    'Bucket' => 'my-bucket',
    'Key'    => 'my-images/php.gif'
));

$headers = $command->getResponse()->getHeaders();

Now that you’ve retrieved the headers of the Amazon S3 object, you can send the headers to the client that is downloading the object using PHP’s header function.

// Only forward along specific headers
$proxyHeaders = array('Last-Modified', 'ETag', 'Content-Type', 'Content-Disposition');

foreach ($proxyHeaders as $header) {
    if ($headers[$header]) {
        header("{$header}: {$headers[$header]}");
    }
}

Disable output buffering

When you use functions like echo or readfile, you might actually be writing to an output buffer. Using output buffering while streaming large files will unnecessarily consume a large amount of memory and reduce the performance of the download. You should ensure that output buffering is disabled before streaming the contents of the file.

// Stop output buffering
if (ob_get_level()) {
    ob_end_flush();
}

flush();

Send the data

Now you’re ready to stream the file using the Amazon S3 stream wrapper and the readfile function. The stream wrapper uses a syntax of "s3://[bucket]/[key]" where "[bucket]" is the name of an Amazon S3 bucket and "[key]" is the key of an object (which can contain additional "/" characters to emulate folder hierarchies).

readfile('s3://my-bucket/my-images/php.gif');

Caching

Our very simple approach to serving files from Amazon S3 does not take advantage of HTTP caching mechanisms. By implementing cache revalidation into your script, you can allow users to use a cached version of an object.

A few slight modifications to the script will allow your application to benefit from HTTP caching. By passing the ETag and Last-Modified headers from Amazon S3 to the browser, we are allowing the browser to know how to cache and revalidate the response. When a web browser has previously downloaded a file, a subsequent request to download the file will typically include cache validation headers (e.g., "If-Modified-Since", "If-None-Match"). By checking for these cache validation headers in the HTTP request sent to the PHP server, we can forward these headers along in the HEAD request sent to Amazon S3.

Here’s a complete example that will pass along cache-specific HTTP headers from the Amazon S3 object.

// Assuming the SDK was installed via Composer
require 'vendor/autoload.php';

use AwsS3S3Client;

// Create a client object
$client = S3Client::factory(array(
    'key'    => '****',
    'secret' => '****',
));

// Register the Amazon S3 stream wrapper
$client->registerStreamWrapper();

readObject($client, 'my-bucket', 'my-images/php.gif');

/**
 * Streams an object from Amazon S3 to the browser
 *
 * @param S3Client $client Client used to send requests
 * @param string   $bucket Bucket to access
 * @param string   $key    Object to stream
 */
function readObject(S3Client $client, $bucket, $key)
{
    // Begin building the options for the HeadObject request
    $options = array('Bucket' => $bucket, 'Key' => $key);

    // Check if the client sent the If-None-Match header
    if (isset($_SERVER['HTTP_IF_NONE_MATCH'])) {
        $options['IfNoneMatch'] = $_SERVER['HTTP_IF_NONE_MATCH'];
    }

    // Check if the client sent the If-Modified-Since header
    if (isset($_SERVER['HTTP_IF_MODIFIED_SINCE'])) {
        $options['IfModifiedSince'] = $_SERVER['HTTP_IF_MODIFIED_SINCE'];
    }

    // Create the HeadObject command
    $command = $client->getCommand('HeadObject', $options);

    try {
        $response = $command->getResponse();
    } catch (AwsS3ExceptionS3Exception $e) {
        // Handle 404 responses
        http_response_code(404);
        exit;
    }

    // Set the appropriate status code for the response (e.g., 200, 304)
    $statusCode = $response->getStatusCode();
    http_response_code($statusCode);

    // Let's carry some headers from the Amazon S3 object over to the web server
    $headers = $response->getHeaders();
    $proxyHeaders = array(
        'Last-Modified',
        'ETag',
        'Content-Type',
        'Content-Disposition'
    );

    foreach ($proxyHeaders as $header) {
        if ($headers[$header]) {
            header("{$header}: {$headers[$header]}");
        }
    }

    // Stop output buffering
    if (ob_get_level()) {
        ob_end_flush();
    }

    flush();

    // Only send the body if the file was not modified
    if ($statusCode == 200) {
        readfile("s3://{$bucket}/{$key}");
    }
}

Caveats

In most cases, this simple solution will work as expected. However, various software components are interacting with one another, and each component must be able to properly stream data in order to achieve optimal performance.

The PHP.net documentation for flush() provides some useful information to keep in mind when attempting to stream data from a web server to a browser:

Several servers, especially on Win32, will still buffer the output from your script until it terminates before transmitting the results to the browser. Server modules for Apache like mod_gzip may do buffering of their own that will cause flush() to not result in data being sent immediately to the client. Even the browser may buffer its input before displaying it. Netscape, for example, buffers text until it receives an end-of-line or the beginning of a tag, and it won’t render tables until the </table> tag of the outermost table is seen. Some versions of Microsoft Internet Explorer will only start to display the page after they have received 256 bytes of output, so you may need to send extra whitespace before flushing to get those browsers to display the page.

Release: AWS SDK for PHP 2.4.5

by Michael Dowling | on | in PHP | Permalink | Comments |  Share

We would like to announce the release of version 2.4.5 of the AWS SDK for PHP. This release adds support for using the Redis cache engine software with Amazon ElastiCache.

Changelog

  • Amazon ElastiCache now offers the Redis cache engine software, in addition to Memcached. Customers who currently use Redis can optionally "seed" a new ElastiCache Redis cache cluster with their existing data from a Redis snapshot file, easing migration to a managed ElastiCache environment. In addition, to support the Redis replication capabilities, the ElastiCache API now supports replication groups: Customers can create a replication group with a primary Redis cache node, and add one or more read replica nodes that automatically stay synchronized with cache data in the primary node. Read-intensive applications can be offloaded to a read replica, reducing the load on the primary node. Read replicas can also guard against data loss in the event of a primary cache node failure.
  • Added support for using the us-gov-west-1 region to the AWS CloudFormation client.

Install/Download the Latest SDK

AWS Service Provider for Laravel 1.1.0

by Jeremy Lindblom | on | in PHP | Permalink | Comments |  Share

We would like to announce the availability of version 1.1.0 of the AWS Service Provider for Laravel. This release updates the config handling logic of the service provider and provides a package-level configuration that can be published to your Laravel application via Artisan for easy customization.

Are there any other features you would like to see in the service provider? Please let us know on our GitHub issue tracker. Better yet, please consider submitting a pull request!

EC2Metadata

by Pavel Safronov | on | in .NET | Permalink | Comments |  Share

A few months ago we added a helper utility to the SDK called EC2Metadata. This is a class that provides convenient access to EC2 Instance Metada. The utility surfaces most instance data as static strings and some complex data as .NET structures. For instance, the following code sample illustrates how you can retrieve the current EC2 instance’s Id and network interfaces:

string instanceId = EC2Metadata.InstanceId;
Console.WriteLine("Current instance: {0}", instanceId);

var networkInstances = EC2Metadata.NetworkInterfaces;
foreach(var netInst in networkInstances)
{
    Console.WriteLine("Network Interface: Owner = {0}, MacAddress = {1}", netInst.OwnerId, netInst.MacAddress);
}

The utility also exposes methods to retrieve data that may not have yet been modeled in EC2Metadata. These are EC2Metadata.GetItems(string path) and EC2Metadata.GetData(string path). GetItems returns a collection of items for that source, while GetData returns the metadata for that path (if the path is invalid or the item doesn’t exist, GetData returns null). For example, to retrieve the current instance Id you can use the InstanceId property or, equivalently, you can use GetData:

string instanceId = EC2Metadata.GetData("/instance-id");
Console.WriteLine("Current instance: {0}", instanceId);

Similarly, you can use GetItems to retrieve the available nodes for a specific path:

// Retrieve nodes from the root, http://169.254.169.254/latest/meta-data/
var rootNodes = EC2Metadata.GetItems(string.Empty);
foreach(var item in rootNodes)
{
    Console.WriteLine(item);
}

Note: since instance metadata is accessible only from an EC2 instance, the SDK will throw an exception if you attempt to use this utility anywhere outside of an EC2 instance (for example, your desktop).

Uploading to Amazon S3 with HTTP POST using the AWS SDK for .NET

by Norm Johanson | on | in .NET | Permalink | Comments |  Share

Generally speaking, access to your Amazon S3 resources requires your AWS credentials, though there are situations where you would like to grant certain forms of limited access to other users. For example, to allow users temporary access to download a non-public object, you can generate a pre-signed URL.

Another common situation is where you want to give users the ability to upload multiple files over time to an S3 bucket, but you don’t want to make the bucket public. You might also want to set some limits on what type and/or size of files users can upload. For this case, S3 allows you to create an upload policy that describes what a third-party user is allowed to upload, sign that policy with your AWS credentials, then give the user the signed policy so that they can use it in combination with HTTP POST uploads to S3.

The AWS SDK for .NET comes with some utilities that make this easy.

Writing an Upload Policy

First, you need to create the upload policy, which is a JSON document that describes the limitations Amazon S3 will enforce on uploads. This policy is different from an Identity and Access Management policy.

Here is a sample upload policy that specifies

  • The S3 bucket must be the-s3-bucket-in-question
  • Object keys must begin with donny/uploads/
  • The S3 canned ACL must be private
  • Only text files can be uploaded
  • The POST must have an x-amz-meta-yourelement specified, but it can contain anything.
  • Uploaded files cannot be longer than a megabyte.
{"expiration": "2013-04-01T00:00:00Z",
  "conditions": [ 
    {"bucket": "the-s3-bucket-in-question"}, 
    ["starts-with", "$key", "donny/uploads/"],
    {"acl": "private"},
    ["eq", "$Content-Type", "text/plain"],
    ["starts-with", "x-amz-meta-yourelement", ""],
    ["content-length-range", 0, 1048576]
  ]
}

It’s a good idea to place as many limitations as you can on these policies. For example, make the expiration as short as reasonable, restrict separate users to separate key prefixes if using the same bucket, and constrain file sizes and types. For more information about policy construction, see the Amazon Simple Storage Service Developer Guide.

 

Signing a Policy

Once you have a policy, you can sign it with your credentials using the SDK.

using Amazon.S3.Util;
using Amazon.Runtime;

var myCredentials = new BasicAWSCredentials(ACCESS_KEY_ID, SECRET_ACCESS_KEY);
var signedPolicy = S3PostUploadSignedPolicy.GetSignedPolicy(policyString, myCredentials);

Ideally, the credentials used to sign the request would belong to an IAM user created for this purpose, and not your root account credentials. This allows you to further constrain access with IAM policies, and it also gives you an avenue to revoke the signed policy (by rotating the credentials of the IAM user).

In order to successfully sign POST upload policies, the IAM user permissions must allow the actions s3:PutObject and s3:PutObjectAcl.

Uploading an Object Using the Signed Policy

You can add this signed policy object to an S3PostUploadRequest.

var postRequest = new S3PostUploadRequest 
{
    Key = "donny/uploads/throwing_rocks.txt",
    Bucket = "the-s3-bucket-in-question",
    CannedACL = S3CannedACL.Private,
    InputStream = File.OpenRead("c:throwing_rocks.txt"),
    SignedPolicy = signedPolicy
};

postRequest.Metadata.Add("yourelement", myelement);

var response = AmazonS3Util.PostUpload(postRequest);

Keys added to the S3PostUploadRequest.Metadata dictionary will have the x-amz-meta- prefix added to them if it isn’t present. Also, you don’t always have to explicitly set the Content-Type if it can be inferred from the extension of the file or key.

Any errors returned by the service will result in an S3PostUploadException, which will contain an explanation of why the upload failed.

 

Exporting and Importing a Signed Policy

You can export the S3PostUploadSignedPolicy object to JSON or XML to be transferred to other users.

var policyJson = signedPolicy.ToJson();
var policyXml = signedPolicy.ToXml();

And the receiving user can re-create S3PostUploadSignedPolicy objects with serialized data.

var signedPolicy = S3PostUploadSignedPolicy.GetSignedPolicyFromJson(policyJson);
vat signedPolicy2 = S3PostUploadSignedPolicy.GetSignedPolicyFromXml(policyXML);

For more information about uploading objects to Amazon S3 with HTTP POST, including how to upload objects with a web browser, see the Amazon Simple Storage Service Developer Guide.

 

Release: AWS SDK for PHP 2.4.4

by Jeremy Lindblom | on | in PHP | Permalink | Comments |  Share

We would like to announce the release of version 2.4.4 of the
AWS SDK for PHP. This release updates the Amazon EC2 client to use the
2013-07-15 API version and fixes issues reported on the forums and GitHub.

Changelog

  • Added support for assigning a public IP address to a VPC instance at launch to the Amazon EC2 client
  • Updated the Amazon EC2 client to use the 2013-07-15 API version
  • Updated the Amazon SWF client to sign requests with Signature V4
  • Updated the Instance Metadata client to allow for higher and more customizable connection timeouts
  • Fixed an issue with the SDK where XML map structures were not being serialized correctly in some cases
  • Fixed issue #136 where a few of the new Amazon SNS mobile push operations were not working properly
  • Fixed an issue where the AWS STS AssumeRoleWithWebIdentity operation was requiring credentials and a signature
    unnecessarily
  • Fixed and issue with the S3Client::uploadDirectory method so that true key prefixes can be used
  • Updated the API docs to include sample code for each operation that indicates the parameter structure
  • Updated the API docs to include more information in the descriptions of operations and parameters
  • Added a page about Iterators to the user guide

Install/Download the Latest SDK

DynamoDB Session Store for Rack Applications

by Loren Segal | on | in Ruby | Permalink | Comments |  Share

Today we are announcing a new RubyGem that enables your Ruby on Rails or Rack-based applications to store session data inside of Amazon DynamoDB. The gem acts as a drop-in replacement for session stores inside of Rails and can also run as a Rack middleware for non-Rails apps. You can read more about how to install and configure the gem on the GitHub repository: aws/aws-sessionstore-dynamodb-ruby. If you want to get started right away, just add the gem to your Gemfile via:

gem 'aws-sessionstore-dynamodb', '~> 1.0'

For me, the best part of this gem is that it was the product of a summer internship project by one of our interns, Ruby Robinson. She did a great job ramping up on new skills and technologies, and ultimately managed to produce some super well-tested and idiomatic code in a very short period of time. Here’s Ruby in her own words:

Hello, my name is Ruby Robinson, and I was a summer intern with the AWS Ruby SDK team. My project was to create a RubyGem (aws-sessionstore-dynamodb) that allowed Rack applications to store sessions in Amazon DynamoDB.

I came into the internship knowing Java and, ironically, not knowing Ruby. It was an opportunity to learn something new and contribute to the community. After pouring myself through a series of books, tutorials, and blogs on Ruby, Ruby on Rails, and Rack, the gem emerged; with help from Loren and Trevor.

Along with creating the gem, I got to experience the Amazon engineering culture. It largely involves taking ownership of projects, innovation, and scalability. I got to meet with engineers who were solving problems at scales I had only heard of. With an Amazon internship, you are not told what to do; you are asked what you are going to do. As my technical knowledge grew, I was able to take ownership of my project and drive it to completion.

In the end I produced a gem with some cool features! The gem is a drop-in replacement for the default session store that gives you the persistence and scale of Amazon DynamoDB. So, what are you waiting for? Check out the gem today!

The experience of bringing a developer from another language into Ruby taught me quite a bit about all of the great things that our ecosystem provides us, and also shined a light on some of the things that are more confusing to newbies. In the end, it was extremely rewarding to watch someone become productive in the language in such a short period of time. I would recommend that everyone take the opportunity to teach new Rubyists the language, if that opportunity ever arises. I think it’s also important that we encourage new developers to become active in the community and write more open source code, since that’s what makes our ecosystem so strong. So, if you know of a new Rubyist in your area, invite them out to your local Ruby meetup or hackfest and encourage them to contribute to some of the projects. You never know, in a few years these might be the people writing and maintaining the library code you depend on every day.

And with that said, please check out our new Amazon DynamoDB Session Store for Rack applications and let us know what you think, either here, or on GitHub!

Provision an Amazon EC2 Instance with PHP

by Jeremy Lindblom | on | in PHP | Permalink | Comments |  Share

Amazon EC2 is a powerful AWS service that includes the ability to provision on-demand servers. While you can easily do this through the AWS Management Console, in this post, I want show you how to use the AWS SDK for PHP to do it programmatically by interacting with the Amazon EC2 API.

Let’s create a single PHP script, piece by piece, that uses the SDK to do the following:

  1. Create and configure an Amazon EC2 client.
  2. Create an EC2 key pair and store the private key.
  3. Create and configure an EC2 security group.
  4. Launch an EC2 instance of an Amazon Machine Image (AMI) and retrieve its public DNS name so we can access it via SSH.

Create an EC2 client

First, let’s bootstrap the SDK and create an EC2 client object. Make sure to replace the placeholder values in the following code with your AWS credentials and desired region.

<?php

require 'vendor/autoload.php';

use Aws\Ec2\Ec2Client;

$ec2Client = Ec2Client::factory(array(
    'key'    => '[aws access key]',
    'secret' => '[aws secret key]',
    'region' => '[aws region]' // (e.g., us-east-1)
));

Create a key pair

Next, we’ll create a key pair that will provide SSH access to our server once it is running. We need to create the key pair first so we can specify it when we launch the EC2 instance. Creating the key pair is simple.

// Create the key pair
$keyPairName = 'my-keypair';
$result = $ec2Client->createKeyPair(array(
    'KeyName' => $keyPairName
));

In order to use the key pair later, we will need to save the private key locally. We can do this by extracting the key material from the response and using some of PHP’s file handling functions to save it to a file. We also need to adjust the file permissions so that the key can be used for SSH access.

// Save the private key
$saveKeyLocation = getenv('HOME') . "/.ssh/{$keyPairName}.pem";
file_put_contents($saveKeyLocation, $result['keyMaterial']);

// Update the key's permissions so it can be used with SSH
chmod($saveKeyLocation, 0600);

Create and configure a security group

Next, let’s create and configure a security group which will allow the server to be accessed via HTTP (port 80) and SSH (port 22). By default, access to an EC2 instance is completely locked down. Security groups allow you to whitelist access to ports on an EC2 instance. Creating a security group requires only a name and description.

// Create the security group
$securityGroupName = 'my-security-group';
$result = $ec2Client->createSecurityGroup(array(
    'GroupName'   => $securityGroupName,
    'Description' => 'Basic web server security'
));

// Get the security group ID (optional)
$securityGroupId = $result->get('GroupId');

After creating the security group, you can then configure its rules. To open up ports 22 and 80 we will use the AuthorizeSecurityGroupIngress operation and specify the security group name.

// Set ingress rules for the security group
$ec2Client->authorizeSecurityGroupIngress(array(
    'GroupName'     => $securityGroupName,
    'IpPermissions' => array(
        array(
            'IpProtocol' => 'tcp',
            'FromPort'   => 80,
            'ToPort'     => 80,
            'IpRanges'   => array(
                array('CidrIp' => '0.0.0.0/0')
            ),
        ),
        array(
            'IpProtocol' => 'tcp',
            'FromPort'   => 22,
            'ToPort'     => 22,
            'IpRanges'   => array(
                array('CidrIp' => '0.0.0.0/0')
            ),
        )
    )
));

Note: In this simple example, we are granting all IP addresses access to these two ports, but in a production setting you should consider limiting the access to certain IP addresses or ranges as appropriate. Also, you may need to open additional ports for MySQL or HTTPS traffic.

Launch an instance

Now that we have a key pair and security group set up, we are ready to launch an EC2 instance (our server) with these settings. To launch an EC2 instance, you also need to specify the ImageId parameter, which is a reference to the AMI that the EC2 instance should be created from. In this example, we are going to use an Amazon Linux AMI. Use the EC2 RunInstances operation to launch the instance.

// Launch an instance with the key pair and security group
$result = $ec2Client->runInstances(array(
    'ImageId'        => 'ami-570f603e',
    'MinCount'       => 1,
    'MaxCount'       => 1,
    'InstanceType'   => 'm1.small',
    'KeyName'        => $keyPairName,
    'SecurityGroups' => array($securityGroupName),
));

From the result, we must get the ID of the instance. We do this using the getPath method available on the result object. This allows us to pull data out of the result that is deep within the result’s structure. The following line of code retrieves an array of instance IDs from the result. In this case, where we have launched only a single instance, the array contains only one value.

$instanceIds = $result->getPath('Instances/*/InstanceId');

Now that the launch has been triggered, we must wait for the instance to become available. The AWS SDK for PHP provides a feature called Waiters, which allow you to poll a resource until it is in a desired state. We will use the waitUntilInstanceRunning method of the EC2 client to wait until the instance that we have just launched is in the “Running” state.

// Wait until the instance is launched
$ec2Client->waitUntilInstanceRunning(array(
    'InstanceIds' => $instanceIds,
));

Once the instance is running, we can use the DescribeInstances operation to retrieve information about the instance, including its public DNS name. We’ll use the getPath method again on the result to extract the PublicDnsName value.

// Describe the now-running instance to get the public URL
$result = $ec2Client->describeInstances(array(
    'InstanceIds' => $instanceIds,
));
echo current($result->getPath('Reservations/*/Instances/*/PublicDnsName'));

Using the public DNS name and the private key that you downloaded, you can SSH into the server. You can do this (from Linux/Unix and Mac devices) by using the ssh command from your CLI.

ssh -i <path to key> ec2-user@<public dns name>

Once you are logged in, you can install software (e.g., yum install php) and deploy your application. Good work! Hopefully, this tutorial helps you with your next PHP-related DevOps project.

Amazon DynamoDB Session Manager for Apache Tomcat

Today we’re excited to talk about a brand new open source project on our GitHub page for managing Apache Tomcat sessions in Amazon DynamoDB!

DynamoDB’s fast and predictable performance characteristics make it a great match for handling session data. Plus, since it’s a fully-managed NoSQL database service, you avoid all the work of maintaining and operating a separate session store.

Using the DynamoDB Session Manager for Tomcat is easy. Just drop the library in the lib directory of your Tomcat installation and tell Tomcat you’re using a custom session manager in your context.xml configuration file:

<?xml version="1.0" encoding="UTF-8"?>
<Context>
    <WatchedResource>WEB-INF/web.xml</WatchedResource>
    <Manager className="com.amazonaws.services.dynamodb.sessionmanager.DynamoDBSessionManager"
             awsAccessKey="myAccessKey"
             awsSecretKey="mySecretKey"
             createIfNotExist="true" />
</Context>

The context.xml file above configures the session manager to store your sessions in DynamoDB, and uses the provided AWS security credentials to access DynamoDB. There are several other configuration options available, including many ways to provide your security credentials:

  • you can explicitly specify them (as shown above)
  • you can specify a properties file to load them from
  • you can rely on the DefaultAWSCredentialsProviderChain to load your credentials from environment variables, Java system properties, or IAM roles for Amazon EC2 instances

If you’re using the AWS Toolkit for Eclipse and deploying your application through AWS Elastic Beanstalk, then all you have to do is opt-in to using the DynamoDB Session Manager for Tomcat in the New AWS Java Web Project Wizard. Then when you deploy to AWS Elastic Beanstalk, all your sessions will be managed in DynamoDB.

For more details on using the session manager, check out the Session Manager section in the AWS SDK for Java Developer Guide. Or, if you really want to get into the details, check out the project on GitHub.

We’re excited to have the first version of the Amazon DynamoDB Session Manager for Apache Tomcat out there for customers to play with. What features do you want to see next? Let us know in the comments below!

Scripting your EC2 Windows fleet using Windows PowerShell and Windows Remote Management

by Steve Roberts | on | in .NET | Permalink | Comments |  Share

Today we have a guest post by one of our AWS Solutions Architects, James Saull, discussing how to take advantage of Windows PowerShell and Windows Remote Management (WinRM) to script your Windows fleet.

One of the advantages of using AWS is on-demand access to an elastic fleet of machines—continuously adjusting in response to demand and ranging, potentially, from zero machines to thousands. This presents a couple of challenges: within your infrastructure, how might you identify and run your script against a large and varying number of machines at the same time? In this post, we take a look at how to use EC2 tags for targeting and Windows Remote Management to simultaneously run PowerShell scripts.

Launching an AWS EC2 Windows instance from the console and connecting via RDP is a simple matter. You can even do it directly from within Visual Studio as recently documented here. From the RDP session, you might perform tasks such as updating the assets of an ASP.Net web application. If you had a second machine, you could open a second RDP session and repeat those tasks. Alternatively, if you are running in AWS VPC, you could avoid opening additional RDP sessions and just use PowerShell’s Enter-PSSession to the second machine. This does require that all instances are members of security groups that will allow Windows Remote Management traffic.

Below is an example of connecting to another host in a VPC and issuing a simple command (notice the date time stamps are different on the second host):

However, as the number of machines grows, you will quickly want the ability to issue a command once and have it run against the whole fleet simultaneously. To do this, we can use PowerShell’s Invoke-Command. Let’s take a look at how we might instruct a fleet of Windows EC2 hosts to all download the latest version of my web application assets from Amazon S3.

First, using EC2 tags, we will identify which machines are web servers, as only they should be downloading these files. The example below uses the cmdlets Get-EC2Instance and Read-S3Object, which are part of the AWS Tools for Windows PowerShell and are installed by default on AWS Windows Machine Images:

$privateIp = ((Get-EC2Instance -Region eu-west-1).RunningInstance `
            | Where-Object {
                $_.Tag.Count –gt 0 `
                –and $_.Tag.Key -eq  "Role" `
                -and $_.Tag.Value -match "WebServer"}).PrivateIpAddress 

Establish a session with each of the web servers:

$s = New-PSSession -ComputerName $privateIp 

Invoke the command that will now simultaneously run on each of the web servers:

Invoke-Command -Session $s -ScriptBlock {
    Read-S3Object   -BucketName mysourcebucket `
                    -KeyPrefix /path/towebassets/ `
                    -Directory z:webassets `
                    -Region eu-west-1 } 

This works well, but what if I want to run something that is individualized to the instance? There are many possible ways, but here is one example:

$scriptBlock = {
 param (
            [int] $clusterPosition , [int] $numberOfWebServers
        )
        "I am Web Server $clusterPosition out of $numberOfWebServers" | Out-File z:afile.txt
}

$position = 1
foreach($machine in $privateIp)
{
    Invoke-Command  -ComputerName $machine `
                    -ScriptBlock $scriptBlock `
                    -ArgumentList $position , ($PrivateIp.Length) `
                    -AsJob -JobName DoSomethingDifferent
    $position++
} 

Summary

This post showed how using EC2 tags can make scripting a fleet of instances via Windows Remote Management very convenient. We hope you find these tips helpful, and as always, let us know what other .NET or PowerShell information would be most valuable to you.