AWS Developer Blog

Migrating to Boto3

by Michael Dowling | on | in Python | Permalink | Comments |  Share

Boto3, the latest version of the AWS SDK for Python, was released earlier this year. Since its release, we’ve seen more and more customers migrating to the latest major version of Boto. Boto3 provides many significant improvements over Boto:

  • Faster availability of API updates and consistency in exposed interfaces
  • Collections that provide an iterable interface to a collection of resources, including batch actions that can be used to perform an API operation on an entire collection
  • Waiters that make it easier to poll resources for status changes
  • Data-driven resource abstractions that provide an object-oriented API while still allowing a rapid relase cadence with significantly reduced maintenance overhead

We understand that migrating to a new major version of a project can be a big undertaking. It can take significant developer time to migrate; it requires more testing to ensure the successful migration of your application; and it can often involve wrestling with your dependency graph to ensure everything is compatible.

Using Boto and Boto3 side-by-side

To make the process of migrating to Boto3 easier, we released Boto3 under the boto3 namespace so that you can use Boto and Boto3 in the same project without conflicts. This allows you to continue to use Boto for legacy applications Boto3 for new development.


pip install boto
pip install boto3


import boto3
import boto.ec2

# For new development, use boto3. In this case, S3
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
    print(bucket.name)

# Feel free to use boto for legacy code. In this case, EC2
conn = boto.ec2.connect_to_region('us-west-2')
conn.run_instances('ami-image-id')

If your legacy applications or individual application components are currently running without issue, you might not have much motiviation to migrate from Boto to Boto3. After all, if it ain’t broke, dont’ fix it. However, if you have new applications or applications that need to use newer services and service features, you are strongly encouraged to upgrade to Boto3. Boto3 is the future of Boto. It is where most of our development will be focused.

Anything holding you back?

We want to make Boto3 as good as it can be. Your input and feedback is crucial in helping us decide how to allocate developer time and which features to develop. We’d like to know if there’s something holding you back from migrating or using Boto3 for new application development. Maybe there’s a feature you relied on in Boto that is not present in Boto3. Or perhpas you find something confusing in the documentation or the way a client is used. Please open an issue on the Boto3 issue tracker and let us know. We’ll apply the appropriate issue labels to make it easier to find and +1 existing issues.

Ready to migrate?

Ready to migrate now? Our migration guide covers some of the high-level concepts to keep in mind. And again, feel free to use the the Boto3 issue tracker for questions or feature requests.

Welcome to the AWS Developer Blog for Python

by Jordon Phillips | on | in Python | Permalink | Comments |  Share

Hi everyone! Welcome to the AWS Developer Blog for Python. I’m Jordon Phillips, and I work on the AWS SDK for Python. This blog will be the place to go for information about:

  • Tips and tricks for using the AWS SDK for Python
  • New feature announcements
  • Deep dives into AWS SDK for Python features
  • Guest posts from AWS service teams

In the meantime, here are a few links to get you started:

We’re excited to get this blog started, and we hope to see you again real soon. Stay tuned!

Welcome to the AWS Developer Blog for Go

Hi, everyone! Welcome to the AWS Developer Blog for Go. I’m Jason Del Ponte, and I’m a developer on the AWS SDK for Go team. This blog will be the place to go for information about:

  • Tips and tricks for using the AWS SDK for Go
  • New feature announcements
  • Deep dives into the AWS SDK for Go
  • Guest posts from AWS service teams

In the meantime, we’ve created several resources to introduce you to the SDK:

We’re excited to see how you will use the SDK. Please share your experience, feedback, and questions. Stay tuned!

Tuning the AWS SDK for Java to Improve Resiliency

by Andrew Shore | on | in Java | Permalink | Comments |  Share

In this blog post we will discuss why it’s important to protect your application from downstream service failures, offer advice for tuning configuration options in the SDK to fit the needs of your application, and introduce new configuration options that can help you set stricter SLAs on service calls.

Service failures are inevitable. Even AWS services, which are highly available and fault-tolerant, can have periods of increased latency or error rates. When there are problems in one of your downstream dependencies, latency increases, retries start, and generally API calls take longer to complete, if they complete at all. This can tie up connections, preventing other threads from using them, congest your application’s thread pool, and hold onto valuable system resources for a call or connection that may ultimately be doomed. If the AWS SDK for Java is not tuned correctly, then a single service dependency (even one that may not be critical to your application) can end up browning out or taking down your entire application. We will discuss techniques you can use to safeguard your application and show you how to find data to tune the SDK with the right settings.

Gathering Metrics

The metrics system in the AWS SDK for Java has several predefined metrics that give you insight into the performance of each of your AWS service dependencies. Metric data can be aggregated at the service level or per individual API action. There are several ways to enable the metrics system. In this post, we will take a programmatic approach on application startup.

To enable the metrics system, add the following lines to the startup code of your application.

AwsSdkMetrics.enableDefaultMetrics();

AwsSdkMetrics.setCredentialProvider(credentialsProvider);

AwsSdkMetrics.setMetricNameSpace("AdvancedConfigBlogPost");

Note: The metrics system is geared toward longer-lived applications. It uploads metric data to Amazon CloudWatch at one-minute intervals. If you are writing a simple program or test case to test-drive this feature, it may terminate before the metrics system has a chance to upload anything. If you aren’t seeing metrics in your test program, try adding a sleep interval of a couple of minutes before terminating to allow metrics to be sent to CloudWatch.

For more information about the features of the metrics system and other ways to enable it, see this blog post.

Interpreting Metrics to tune the SDK

After you have enabled the metrics system, the metrics will appear in the CloudWatch console under the namespace you’ve defined (in the preceding example, AdvancedConfigBlogPost).

Let’s take a look at the metrics one by one to see how the data can help us tune the SDK.

HttpClientGetConnectionTime: Time, in milliseconds, for the underlying HTTP client library to get a connection.

  • Typically, the time it takes to establish a connection won’t vary in a service (that is, all APIs in the same service should have similar SLAs for establishing a connection). For this reason, it is valid to look at this metric aggregated across each AWS service.
  • Use this metric to determine a reasonable value for the connection timeout setting in ClientConfiguration.

    • The default value for this setting is 50 seconds, which is unreasonably high for most production applications, especially those hosted within AWS itself and making service calls to the same region. Connection latencies, on average, are on the order of milliseconds, not seconds.

HttpClientPoolAvailableCount: Number of idle persistent connections of the underlying HTTP client. This metric is collected from the respective PoolStats before the connection of a request is obtained.

  • A high number of idle connections is typical of applications that perform a batch of work at intervals. For example, consider an application that uploads all files in a directory to Amazon S3 every five minutes. When the application is uploading files, it’s creating several connections to S3 and then does nothing with the service for five minutes. The connections are left in the pool with nothing to do and will eventually become idle. If this is the case for your application, and there are constantly idle connections in the pool that aren’t serving a useful purpose, you can tune the connectionMaxIdleMillis setting and use the idle connection reaper (enabled by default) to more aggressively purge these connections from the pool.
  • Setting the connectionMaxIdleMillis too low can result in having to establish connections more frequently, which can outweigh the benefits of freeing up system resources from idle connections. Take caution before acting on the data from this metric.
  • If your application does have a bursty workload and you find that the cost of establishing connections is more damaging to performance than keeping idle connections, you can also increase the connectionMaxIdleMillis setting to allow the connections to persist between periods of work.

    • Note: The connectionMaxIdleMillis will be limited to the Keep-Alive time specified by the service. For example if you set connectionMaxIdleMillis to five minutes but the service only keeps connections alive for sixty seconds, the SDK will still discard connections after sixty seconds when they are no longer usable.

HttpClientPoolPendingCount: Number of connection requests being blocked while waiting for a free connection of the underlying HTTP client. This metric is collected from the respective PoolStats before the connection of a request is obtained

  • A high value for this metric can indicate a problem with your connection pool size or improper handling of service failures.
  • If your usage of the client exceeds the ability of the default connection pool setting to satisfy your request you can increase the size of the connection pool through this setting.
  • Connection contention can also occur when a service is experiencing increased latency or error rates and the SDK is not tuned properly to handle it. Connections can quickly be tied up waiting for a response from a faulty server or waiting for retries per the configured retry policy. Increasing the connection pool size in this case might only make things worse by allowing the application to hog more threads trying to communicate with a service in duress. If you suspect this may be the case, look at the other metrics to see how you can tune the SDK to handle situations like this in a more robust way.

HttpRequestTime: Number of milliseconds for a logical request/response round-trip to AWS. Captured on a per request-type level.

  • This metric records the time it takes for a single HTTP request to a service. This metric can be recorded multiple times per operation, depending on the retry policy in use.
  • We’ve recently added a new configuration setting that allows you to specify a timeout on each underlying HTTP request made by the client. The SLAs for requests between APIs or even per request can vary widely, so it’s important to use the provided metrics and consider the timeout setting carefully.

    • This new setting can be specified per request or for the entire client (through ClientConfiguration). Although it’s hard to set a reasonable timeout on the client, it makes sense to set a default timeout on the client and override it per request, where needed.
    • By default, this feature is disabled.
    • Request timeouts are only supported in Java 7 and later.

Using the DynamoDB client as an example, let’s look at how you can use this new feature.

ClientConfiguration clientConfig = new ClientConfiguration();
clientConfig.setRequestTimeout(20 * 1000);
AmazonDynamoDBClient ddb = new AmazonDynamoDBClient(credentialsProvider, clientConfig);
// Will inherit 20 second request timeout from client level setting
ddb.listTables();
// Request timeout overridden on the request level
ddb.listTables(new ListTablesRequest().withSdkRequestTimeout(10 * 1000));
// Turns off request timeout for this request
ddb.listTables(new ListTablesRequest().withSdkRequestTimeout(-1));

ClientExecuteTime: Total number of milliseconds for a request/response including the time to execute the request handlers, the round-trip to AWS, and the time to execute the response handlers. Captured on a per request-type level.

  • This metric includes any time spent executing retries per the configured retry policy in ClientConfiguration.
  • We have just launched a new feature that allows you to specify a timeout on the entire execution time, which matches up very closely to the ClientExecuteTime metric.

    • This new timeout configuration setting can be set for the entire client in ClientConfiguration or per request.
    • By default, this feature is disabled.
    • Client execution timeouts are only supported in Java 7 and later.

Using the DynamoDB client as an example, let’s look at how you would enable a client execution timeout.

ClientConfiguration clientConfig = new ClientConfiguration();
clientConfig.setClientExecutionTimeout(20 * 1000);
AmazonDynamoDBClient ddb = new AmazonDynamoDBClient(credentialsProvider, clientConfig);
// Will inherit 20 second client execution timeout from client level setting
ddb.listTables();
// Client Execution timeout overridden on the request level
ddb.listTables(new ListTablesRequest().withSdkClientExecutionTimeout(10 * 1000));
// Turns off client execution timeout for this request
ddb.listTables(new ListTablesRequest().withSdkClientExecutionTimeout(-1));

The new settings for request timeouts and client execution timeouts are complementary. Using them together is especially useful because you can use client execution timeouts to set harder limits on the API’s total SLA and use request timeouts to prevent one bad request from consuming too much of your total time to execute.

ClientConfiguration clientConfig = new ClientConfiguration();
clientConfig.setClientExecutionTimeout(20 * 1000);
clientConfig.setRequestTimeout(5 * 1000);
// Allow as many retries as possible until the client execution timeout expires
clientConfig.setMaxErrorRetry(Integer.MAX_VALUE);
AmazonDynamoDBClient ddb = new AmazonDynamoDBClient(credentialsProvider, clientConfig);
// Will inherit timeout settings from client configuration. Each HTTP request 
// is allowed 5 second to complete and the SDK will retry as many times as 
// possible (per the retry condition in the retry policy) within the 20 second 
// client execution timeout
ddb.listTables();

Conclusion

Configuring the SDK with aggressive timeouts and appropriately sized connection pools goes a long way toward protecting your application from downstream service failures, but it’s not the whole story. There are many techniques you can apply on top of the SDK to limit the negative effects of a dependency’s outage on your application. Hystrix is an open source library specifically designed to make fault-tolerance easier and your application even more resilient. To use Hystrix to its fullest potential, you’ll need some data to tune it to match your actual service SLAs in your environment. The metrics we discussed in this blog post can give you that information. Hystrix also has an embedded metrics system that can complement the SDKs metrics.

We would love your feedback on the configuration options and metrics provided by the SDK and what you would like to see in the future. Do we provide enough settings and hooks to allow you to tune your application for optimal performance? Do we provide too many settings and make configuring the SDK overwhelming? Does the SDK provide you with enough information to intelligently handle service failures?

 

 

 

Contributing to the AWS SDK for .NET

by Jim Flanagan | on | in .NET | Permalink | Comments |  Share

The AWS SDK for .NET is an open source project available on GitHub. This post is to help community developers navigate the SDK code base with an eye toward contributing features and fixes to the SDK.

Code Generation

The first gotcha for contributors is that major portions of the code are generated from models of the service APIs. In version 3 of the SDK, we reorganized the code base to make it obvious which code is generated. You’ll now find it under each service client folder in folders named "Generated." Similarly, handwritten code is now found in folders named "Custom." Most of the generated code is in partial classes to facilitate extending it without having changes get clobbered by the code generator.

The convention we use when adding extensions to generated partial classes is to place a file called Class.Extensions.cs under the Custom folder with the same hierarchy as the generated file.

The code generator can be found here. The models are under the ServiceModels folder. To add a client, add a model to the ServiceModels folder, update the _manifest.json file in the same folder, and run the generator. The customization files in the folder handle ways in which we can override the behavior of the generator, mostly to keep some consistency across older and newer services, as well as make adjustments to make the API more C#-friendly.

It is sometimes necessary to update the code generator to add a feature or fix an issue. Because changes to the generator may impact all existing services and require a lot of testing, these changes should not be undertaken lightly.

Platform Support

Another thing you may notice about the code base is that some files are under folders like _bcl, _bcl35, _bcl45, _mobile, or _async. This is how the SDK controls which files are included in platform-specific project files.

As an example, if you look at the AutoScaling client folder you will see the folders

Model
_bcl35
_bcl45
_mobile

The _bcl45 folder contains the Auto Scaling client and interface for version 4.5 of the AWS SDK for .NET. It differs from the 3.5 version in that it exposes Async/Await versions of the service APIs, where the 3.5 version exposes Begin/End for asynchrony. The Model folder contains code common to all platforms. For this reason, don’t use Visual Studio to add files to an SDK project. Instead, add the file to the appropriate file system location, and then reload the project. We try to use this subdirectory mechanism rather than #if directives in the code where possible.

Testing

It will be much easier to evaluate and provide feedback on contributions if they are accompanied by unit tests, which can be added to the UnitTests folder.

Sometimes it is good to include some integration tests that hit the service endpoint, too. Integration tests will, by necessity, create AWS resources under your account, so it’s possible you will incur some costs. Try to design integration tests that favor APIs that don’t create resources, are fast to run, and account for the eventual consistency of the APIs you’re testing.

Community Feature Requests

Many contributions are driven by the specific needs of a community member, but sometimes they’re driven simply by a desire to get involved. If you would like to get involved, we collect community feature requests in the FEATURE_REQUESTS.md at the top level of the repository.

Super-Charge Your AWS Command-Line Experience with aws-shell

by Peter Moon | on | in AWS CLI | Permalink | Comments |  Share
When we first started developing the AWS Command Line Interface (CLI) nearly three years ago, we had to figure out how to deliver a consistent command-line experience to the ever-expanding surface area of AWS APIs. We decided to auto-generate commands and options from the underlying models that describe AWS APIs. This strategy has enabled us to deliver timely support for new AWS services and API updates with a consistent style of commands.
We believe consistency and timeliness help our customers be productive on AWS. We also understand it is difficult to get familiar with and effectively use the thousands of commands and options available in the CLI. We are always looking for solutions to make the CLI as easy as possible to learn and use.
This search for better usability led us to create aws-shell, which we’re making public today at https://github.com/awslabs/aws-shell. The project, along with our plans to collaborate with Donne Martin, the author of SAWS, were announced during this year’s re:Invent talk, Automating AWS with the AWS CLI. Donne has made amazing progress in providing a great command line UI for AWS CLI users. It’s been a pleasant surprise and a validation of our feature ideas to see a community project take off and gain so much popularity in a short amount of time. We are excited and grateful to join forces with Donne and bring him on board as one of the maintainers of the aws-shell project.
Now let’s take a look at some of the key features available in aws-shell:
Interactive, fuzzy auto-completion of commands and options
Dynamic in-line documentation of commands and options
Auto-completion of resource identifiers (for example, Amazon EC2 instance IDs, Amazon SQS queue URLs, Amazon SNS topic names, and more)
Execute regular shell commands by piping or prefixing shell commands with ‘!’
Export all commands executed in the current session to your text editor (.edit special command)
Running the “.edit” command after executing some commands gives you all the commands in your default text editor

We’re excited to get aws-shell into your hands, and we look forward to your feedback. As always, you can find us in GitHub. Please share any questions and issues you have!

DynamoDB Document Model Manual Pagination

by Pavel Safronov | on | in .NET | Permalink | Comments |  Share

In version 3.1.1.2 of the DynamoDB .NET SDK package, we added pagination support to the Document Model. This feature allows you to use a pagination token returned by the API to paginate a set of Query or Scan results across sessions. Until now, it was not possible to resume pagination of Query or Scan results without retrieving the already encountered items. This post includes two simple examples of this new functionality.

The first example makes the initial Query call to retrieve all movies from the year 2012, and then saves the pagination token returned by the Search.PaginationToken property. The second example retrieves the PaginationToken and continues the same query. For these examples, we will assume we have functions (void SaveToken(string token) and string LoadToken()) to persist the pagination token across sessions. (If this were an ASP.NET application, the functions would use the session store to store the token, but this can be any similar environment where manual pagination is used.)

Initial query:

var client = new AmazonDynamoDBClient();
Table moviesTable = Table.LoadTable(client, "MoviesByYear");

// start initial query
var search = moviesTable.Query(new QueryOperationConfig
{
    Filter = new QueryFilter("Year", QueryOperator.Equal, 2012),
});

// retrieve one pages of items
List<Document> items = search.GetNextSet();

// get pagination token
string token = search.PaginationToken;

// persist the token in session data or something similar
SaveToken(token);

Resumed query:

var client = new AmazonDynamoDBClient();
Table moviesTable = Table.LoadTable(client, "MoviesByYear");

// load persisted token
string token = LoadToken();

// use token to resume query from last position
var search = moviesTable.Query(new QueryOperationConfig
{
    Filter = new QueryFilter("Year", QueryOperator.Equal, 2012),
    PaginationToken = token,
});
List<Document> items = search.GetNextSet();

// pagination token changed, persist new value
SaveToken(search.PaginationToken);

DataModel support

Although this functionality has not yet been added to the Object Persistence Model, it is possible to work around this limitation. In the following code sample, we can use the DocumentModel API to manually paginate our data, and then use DynamoDBContext to convert the retrieved Documents into .NET objects. Because we are using DynamoDBContext and don’t want to stray too far into the Document Model API, we’re going to use DynamoDBContext.GetTargetTable to avoid the manual construction of our Table instance.

// create DynamoDBContext object
var context = new DynamoDBContext(client);

// get the target table from the context
var moviesTable = context.GetTargetTable<Movie>();

// use token to resume query from last position
var search = moviesTable.Query(new QueryOperationConfig
{
    Filter = new QueryFilter("Year", QueryOperator.Equal, 2012),
    PaginationToken = token,
});
List<Document> items = search.GetNextSet();

// pagination token changed, persist new value
SaveToken(search.PaginationToken);

// convert page of Documents in .NET objects and enumerate over them
IEnumerable<Movie> movies = context.FromDocuments<Movie>(items);
foreach (var movie in movies)
    Log("{0} ({1})", movie.Title, movie.Year);

As you can see, even though we executed our Query using a Table object, we can continue working with familiar .NET classes while controlling the pagination of our data.

Installing Scheduled Tasks on EC2 Windows Instances Using EC2 Run

by Steve Roberts | on | in .NET | Permalink | Comments |  Share

Today’s guest post is the second part of the two part series by AWS Solutions Architect Russell Day. Part one can be found here.

In the previous post, we showed how to use the User data field to install Windows scheduled tasks automatically when Windows EC2 instances are launched. In this post, we will demonstrate how to do the same thing using the new EC2 Run Command, which provides a simple way to remotely execute PowerShell commands against EC2 instances.

Use the EC2 Run Command to Install scheduled tasks automatically

Just as we did in the previous post, we will demonstrate two methods for using the EC2 Run Command: the Amazon EC2 console and AWS Tools for PowerShell.

Use the EC2 console to execute the EC2 Run Command

  1. Complete steps 1 through 4 in the previous post.
  2. In the EC2 console, choose Commands.

  3. Choose the Run a command button.
  4. Under Command document, choose AWS-RunPowerShellScript.
  5. Select your target instances.
  6. Paste the PowerShell script from step 5 of the previous post into the Commands text box as shown.

  7. Leave all other fields at their defaults, and choose Run to invoke the PowerShell script on the target instances.
  8. You can monitor progress and view the output of the invoked scripts as shown.

Use PowerShell to Execute the EC2 Run Command

Alternatively, you can use PowerShell to invoke the EC2 Run Command as shown.

  1. If you have not already configured your PowerShell environment, follow these instructions to configure your PowerShell console to use the AWS Tools for Windows PowerShell.
  2. Save the PowerShell script from step 5 in the previous post as InstallWindowsTasks.ps1.

From a PowerShell session, simply replace ‘Instance-ID’ with the instance IDs of your target instances and provide the path to InstallWindowsTasks.ps1 as shown.

$runPSCommand=Send-SSMCommand 
    -InstanceId @('Instance-ID', 'Instance-ID') 
    -DocumentName AWS-RunPowerShellScript 
    -Parameter @{'commands'=
         @([System.IO.File]::ReadAllText("C:...InstallWindowsTasks.ps1"))}	

You can use the following commands to monitor the status.

Retrieve the command execution status:


Get-SSMCommand -CommandId $runPSCommand.CommandId

Retrieve the status of the command execution on a per-instance basis:


Get-SSMCommandInvocation -CommandId $runPSCommand.CommandId

Retrieve the command information with response data for an instance. (Be sure to replace Instance-ID)

Get-SSMCommandInvocation -CommandId $runPSCommand.CommandId `
      -Details $true -InstanceId Instance-ID | 
   select -ExpandProperty CommandPlugins

Summary

The EC2 Run Command simplifies remote management and customization of your EC2 instances. For more information, see the following resources:

http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/execute-remote-commands.html
https://aws.amazon.com/ec2/run-command/.

AWS SDK for .NET Refresh for ASP.NET 5

by Norm Johanson | on | in .NET | Permalink | Comments |  Share

Today we refreshed our ASP.NET 5 and CoreCLR support for the AWS SDK for .NET. This means we have pulled in all of the latest service updates, new services like AWS IoT, and enhancements from our stable 3.1 line of NuGet packages into new 3.2 beta versions of the SDK. Because there are a few remaining dependencies in our AWSSDK.Core package that are still in beta, we still need to keep our support in beta.

SDK Credentials Store

As part of CoreCLR support in the SDK, we have also enabled the SDK credentials store. The SDK credentials store is the encrypted storage for AWS credentials that you can manage using the AWS Explorer in Visual Studio. This means when you use the SDK on Windows and target the new CoreCLR runtime, the credential search pattern will be the same as the regular AWS SDK for .NET. On non-Windows platforms, we recommend using the shared credentials file.

Installing Scheduled Tasks on EC2 Windows Instances

by Steve Roberts | on | in .NET | Permalink | Comments |  Share

Today’s guest post is part one of a two part series by AWS Solutions Architect Russell Day.

Windows administrators and developers often use scheduled tasks to run programs or scripts on a recurring basis. In this post, we will demonstrate how to use the Amazon EC2 User data option to install scheduled tasks on Windows EC2 instances automatically at launch.

Using the user data field to specify scripts that will automatically configure instances is commonly referred to as bootstrapping. In this post, we will specify a PowerShell script in the user data field to install scheduled tasks when EC2 instances are launched. We will demonstrate two methods for launching EC2 instances: the EC2 console and AWS Tools for PowerShell.

Before we can get started, we need to export the scheduled tasks and store them in a location accessible to our EC2 instances. We will use Task Scheduler to export the scheduled tasks to XML and store them in an Amazon S3 bucket.

Export scheduled tasks.

In Task Scheduler, right-click on the scheduled tasks you want to export and install as XML files.

Create an S3 bucket to store the XML scheduled task definitions.

Use the S3 Console, CLI, or AWS Tools for Windows PowerShell to create an S3 bucket that will store the XML task definition files created in step 1.

Create manifest file(s).

The manifest file(s) contains the scheduled tasks you want to install on the target instances. Consider using a separate manifest file for each unique set of tasks (for example, ProductionServerTasks.xml, DevelopmentServerTasks.xml).

Modify and save the following XML to create your manifest file(s).

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <scheduledTasks>
    <task name="Daily Scheduled Task 1" source="ScheduledTask1.xml" />
    <task name="Daily Scheduled Task 2" source="ScheduledTask2.xml" />
    <task name="Daily Scheduled Task 3" source="ScheduledTask3.xml" />
    <task name="Daily Scheduled Task 4" source="ScheduledTask4.xml" />
  </scheduledTasks>
</configuration>

Upload the exported scheduled task definitions and manifest file(s) to S3.

Upload the scheduled tasks definitions created in step 1 and the manifest file(s) created in step 3 to the S3 bucket created in step 2.

Create a PowerShell script to download and install the scheduled tasks.

The following PowerShell script contains functions to download and install the scheduled tasks stored in our S3 bucket. Replace the $S3Bucket and $TaskManifest parameters with your S3 bucket name and manifest file name.

$VerbosePreference = "Continue";
$WorkingDirectory = "c:tasks";
$TaskManifest = "TaskManifest.xml";
$S3Bucket = "YourS3BucketName";
function Invoke-Functions
{
    Download-ScheduledTasks
    Install-ScheduledTasks
}
function Download-ScheduledTasks
{
    Read-S3Object `
        -BucketName $S3Bucket `
        -Key $TaskManifest `
        -File $WorkingDirectory$TaskManifest

    [xml]$cfg = gc $WorkingDirectory$TaskManifest;
    $cfg.configuration.scheduledtasks.task | 
        %{ 
           $task = $_;
           [string] $TaskFile = $task.source
           Read-S3Object `
                -BucketName $S3Bucket `
                -Key $task.source `
                -File "$WorkingDirectory$TaskFile" 
        }	
}

function Install-ScheduledTasks
{		
    [xml]$cfg = gc $WorkingDirectory$TaskManifest;
    $cfg.configuration.scheduledtasks.task | 
        %{
           $task = $_;
           [string] $TaskFile = $task.source
            Register-ScheduledTask `
                -Xml (get-content "$WorkingDirectory$TaskFile" | out-string) `
                -TaskName $task.name
        }
}

Invoke-Functions | Out-File "c:InstallTasksLog.txt" -Verbose;

Create an EC2 role to allow GetObject permissions to the S3 bucket.

Our PowerShell script uses the Read-S3Object PowerShell cmdlet to download the scheduled task definitions from S3. Therefore, we need to create an EC2 role that allows our EC2 instances to access our S3 bucket objects on our behalf.

Follow these steps to create the EC2 role.

  1. Open the IAM console.
  2. In the navigation pane, choose Policies.
  3. Choose Create Policy.
  4. Choose Create Your Own Policy, and use the following policy template. Replace [YourS3BucketName] with the name of your bucket.

  5. In the navigation pane, choose Roles.
  6. Choose Create Role.
  7. In the Role Name field, type a name for your role.
  8. Under AWS Service Roles, choose Amazon EC2, and then choose Select.
  9. On the Attach Policy page, choose the policy you created, and then choose Next Step.
  10. On the Review page, choose Create Role.

Use the EC2 Console to Launch EC2 Instance(s).

  1. Open the EC2 console, and choose Launch Instance.
  2. Choose your version of Microsoft Windows Server.
  3. Continue to Step: 3 Configure Instance Details.

    • For IAM Role, choose the EC2 role you just created.
    • In Advanced Details, paste the PowerShell script into the text box. Be sure to enclose it in tags as shown here.

  4. Complete the wizard steps and launch the Windows EC2 instance(s).
  5. After your instance(s) have been launched, you can verify the installation of your scheduled tasks.

Use AWS Tools for Windows PowerShell to Launch EC2 Instances.

In keeping with our theme of automation, you can use PowerShell to create the instances programmatically.

  1. If you have not already configured your PowerShell environment, follow these instructions to configure your PowerShell console to use the AWS Tools for Windows PowerShell.
  2. Save the PowerShell script that will download and install the scheduled tasks as InstallWindowsTasks.ps1.
  3. Save the following PowerShell script as a module named AWSHelper.psm1. This allows you to reuse it when you launch Windows EC2 instances in the future. Modify the following parameters with your environment resource values:

    # the key pair to associate with the instance(s)
    $KeyPairName
    # the EC2 instance(s) security group ID
    $SecurityGroupId
    # the subnet ID for the instance(s) after launch
    $SubnetId
    # the ARN of the EC2 role we created to allow access to our S3 bucket
    $InstanceProfile
    

     

    $VerbosePreference = "Continue";
    $scriptpath = $MyInvocation.MyCommand.Path;
    $moduledirectory = Split-Path $scriptpath;
    
    function ConvertTo-Base64($string) {
       $bytes = [System.Text.Encoding]: UTF8.GetBytes ($string);
       $encoded = [System.Convert]::ToBase64String($bytes); 
       return $encoded;
    }
    
    function New-WindowsEC2Instance
    {
      [CmdletBinding()]
      Param
      (                    
        [Parameter(Mandatory=$false)]
        [string] $InstanceType = "t2.micro",
        [Parameter(Mandatory=$false)]
        [string] $KeyPairName = "YourKeyPair", 
        [Parameter(Mandatory=$false)]
        [string] $SecurityGroupId = "sg-5xxxxxxx", 
        [Parameter(Mandatory=$false)]
        [string] $SubnetId = "subnet-1xxxxxxx",	
        [Parameter(Mandatory=$true)]
        [int32] $Count, 
        [Parameter(Mandatory=$false)]
        [string] $InstanceProfile ="EC2RoleARN",
        [Parameter(Mandatory=$false)]
        [string] $UserScript 
            = (Join-Path $script:moduledirectory "InstallWindowsTasks.ps1")
      )
      Process
      {
        $ami = Get-EC2ImageByName -Names 'WINDOWS_2012R2_BASE'
        $ImageID =  $ami[0].ImageId
        $UserData = "";
        if ($userScript -and (Test-Path $userScript))
        {
          $contents = "" + [System.IO.File]::ReadAllText($UserScript) + "";
    	  $filePath = gi $UserScript;
          $UserData = ConvertTo-Base64($contents);
        }
    
        $params = @{};
        $params.Add("ImageID", $ImageID);
        $params.Add("InstanceType", $InstanceType);
        $params.Add("KeyName", $KeyPairName); 
        $params.Add("MaxCount", $Count);
        $params.Add("MinCount", $Count);
        $params.Add("InstanceProfile_Arn", $InstanceProfile);
        $params.Add("SecurityGroupId", $SecurityGroupId); 
        $params.Add("SubnetId", $SubnetId);
        $params.Add("UserData", $UserData); 	
    
        $reservation = New-EC2Instance @params;
      }
    }
    
  4. To invoke the PowerShell code, import the AWSHelper.psm1 module, and then call the New-WindowsEC2Instance cmdlet as shown. Type the number of instances at the prompt.

Summary

The User data option provides a convenient way to automate the customization of your EC2 instances. For more information, see the following resources:

http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/UsingConfig_WinAMI.html#user-data-execution
http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/walkthrough-powershell.html