Microsoft MVP Marco Shaw demonstrates how to use Microsoft Windows PowerShell 2 and the AWS SDK for .NET together using examples for Amazon S3.
Amazon Web Services (AWS), first introduced in early 2006, is a cloud-based infrastructure that provides web services for companies of all sizes. AWS has a number of infrastructure services, including:
- Amazon Elastic Compute Cloud (Amazon EC2). A web service that provides resizable compute capacity in the cloud
- Amazon Simple Storage Service (Amazon S3). A simple web services interface that you can use to store and retrieve large amounts of data at any time from anywhere on the web
- Amazon CloudFront. A web service for content delivery
- Amazon SimpleDB. A web service for running queries on structured data in real time
- Amazon Simple Queue Service (Amazon SQS). A reliable, highly scalable, hosted queue for storing messages as they travel between computers
Note: For a complete list of services, see http://aws.amazon.com/products.
This article shows how to use Windows PowerShell version 2 scripting with AWSspecifically, with Amazon S3and the AWS software development kit (SDK) for Microsoft .NET. It provides a simple walkthrough of a few examples to lay the foundation for some exciting new possibilities. Some familiarity with Windows PowerShell scripting and Microsoft .NET Framework concepts would be helpful to fully understand all the concepts covered.
Get to Know Amazon S3
Amazon S3 is a cloud-based storage service that enables the uploading, downloading, and storage of data in the cloud. Amazon S3 can interact with other AWS services such as Amazon EC2 for a complete cloud-based solution.
There are three important concepts to understand when starting with Amazon S3:
- Objects. Objects are the fundamental entities stored in Amazon S3. They consist of data and metadata and can range in size from 1 byte to 5 GB. Think of an object as a simple file on any common operating system.
- Bucket. Objects are uploaded or stored in buckets. There is no limit to the number of objects that a bucket can hold. Think of a bucket as a simple folder on any common operating system.
- Key. A key is a unique identifier for an object within a bucket. Think of a key as the actual name of a regular file.
With the Amazon S3 service, there is a charge for storing objects in your buckets and for transferring objects in and out of these containers. Before you begin, be sure to sign up for Amazon S3 at http://aws.amazon.com/s3.
Get to Know Windows PowerShell
Windows PowerShell was first introduced publicly in 2003. After several years of development and testing, Microsoft formally released the scripting environment in November 2006 as a free download for all current operating systems at the time. In 2009, Windows PowerShell became part of the engineering standards (Common Engineering Criteria, or CEC) for all Microsoft server-based products.
Windows PowerShell version 2 was packaged with the Windows 7 and Windows Server 2008 R2, both released in October 2009. This time, not only was the environment bundled with the operating system, but it was considered part of the core operating system.
Windows PowerShell is built on the Microsoft .NET Framework 2.0. The Framework provides building blocks using an object-oriented approach to help programmers build applications for the Windows operating system. So, for anyone familiar with object-oriented programming concepts or who has used a language like C# to program, the learning curve for Windows PowerShell will be much easier to overcome.
You interact with the Windows PowerShell engine through a simple console. With Windows PowerShell version 2, an integrated development environment (IDE) is also provided (known as the Integrated Scripting Environment, or ISE). The Windows PowerShell console is shown in Figure 1.
Figure 1. The Windows PowerShell interface
The most basic building block in Windows PowerShell is called a cmdlet. Several cmdletsversion 2 has more than 200 core cmdletsare bundled with the core operating system. Cmdlets have a verbnoun syntax; an example of one that is relatively easy to grasp is
Get-Service, which provides a listing of all the services on the local system (see Figure 2).
Figure 2. Generating a list of the services on the local system
Learning Windows PowerShell is the subject of entire books, but for purposes of this article, here are the cmdlets used in the examples:
New-Object. This cmdlet creates an instance of a Microsoft .NET class. The class is basically provided as an argument when the cmdlet is invoked.
Format-Table. This cmdlet provides extra functionality in formatting how and what is displayed when Windows PowerShell commands are run.
Add-Type. This cmdlet is used to load Microsoft .NET assemblies into the current session.
Get-ChildItem. This cmdlet is used to list the items in a specific location. It is similar to the DOS command
If you're new to Windows PowerShell, the
Get-Command cmdlets are essential. You use
Get-Help to retrieve the built-in help information that comes with Windows PowerShell. It can be used simply as
Get-Help Get-Service, but it also accepts other switches that can provide complete documentation on all its options (
Get-Help Get-Service Full) or just the examples section (
Get-Help Get-Service Examples).
Get-Command cmdlet can be used to retrieve a listing of all the cmdlets (using
Get-Command alone) or cmdlets based on a particular verb or noun (for example,
If you're a more advanced user, you can use
Get-Member. The term member should look familiar to anyone who has been doing Microsoft .NET programming in either C# or in Microsoft Visual Basic.
One of the Windows PowerShell features that you'll be taking advantage of in this article is the fact that you can use most of the functionality the Microsoft .NET Framework provides. Even better, Windows PowerShell can actually load any Microsoft or third-party Microsoft .NET assemblies and usually make use of all of the features they provide.
The AWS SDK for .NET
Late in 2009, Amazon released the AWS SDK for Microsoft .NET (see "Learning More About AWS and Windows PowerShell"). This SDK allows Windows developers to create applications based on Microsoft .NET that use the AWS infrastructure services, including Amazon SS3, Amazon EC2, and Amazon SimpleDB. The SDK includes Microsoft Visual Studio project templates, the AWS .NET library, C# code samples, and documentation.
The following code can be run directly from a Windows PowerShell console or from the ISE. The ISE is installed by default on Windows 7, but you'll need to add it in Windows Server 2008 R2.
You can paste the code into a Windows PowerShell script (with a .ps1 extension), and then invoke it by a simple command like this:
Note: Even though a .ps1 extension is used and powershell.exe is in a
v1.0 sub-folder, if you're working on a Windows 7 or Windows Server 2008 R2 system, Windows PowerShell version 2 is actually installed. Also, by default, Windows PowerShell doesn't allow scripts to be run, so you may get an error when try. To enable the environment to run scripts, run the following command:
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned
Setting the execution policy does have security implications, but using the value
RemoteSigned is a good trade-off between usability and security.
Not only can Windows PowerShell run scripts, it can also be used interactively, so you can paste the code examples in this article directly into a Windows PowerShell console session and run them directly from there.
Note: When working with Windows PowerShell interactively, the console may display the characters
>> at the beginning of a line. These characters simply indicate that the Windows PowerShell parser believes that you have not fully completed your command and is expecting more. Simply issuing a Ctrl+C key sequence returns the prompt so you can retry.
Some Simple Examples
Now, let's dive into some actual examples and code. To run the example code as is, your system must have Windows PowerShell version 2 installed and the AWS SDK for .NET.
Begin by loading the AWS SDK assembly, defining the AWS access keys, and creating an AWS client, as shown in Listing 1.
Listing 1. Set up an AWS connection
Add-Type -Path "C:\Program Files (x86)\AWS SDK for .NET\bin\AWSSDK.dll" $secretKeyID="[secret-key-id]" $secretAccessKeyID="[secret-access-key-id]" $client=[Amazon.AWSClientFactory]::CreateAmazonS3Client($secretKeyID,$secretAccessKeyID)
Here's what the code does:
Add-Typecmdlet loads the Microsoft .NET assembly into the current session or console by providing the full path to the DLL file.
- Two variables are created that hold the user's AWS account information.
- A documented Microsoft .NET static method is used to create a client to specifically access the Amazon S3 service using the variables just created.
All of the following code listings require that you add the code in Listing 1 to the start of the listing before the code actually runs.
Retrieving a List of Buckets
One of the simplest examples you can run using the SDK is to retrieve a list of buckets for a particular account. Listing 2 produces a list of the buckets associated with the current account.
Listing 2. Invoking the ListBuckets method
The above code listing shows how to use a documented method to retrieve a listing of the existing buckets. Everything in Windows PowerShell is object based; as a result, the engine displays all the properties associated with the objects that are returned from the command, as shown in Figure 3.
Figure 3. A list of the buckets from a given account
Listing the Objects in an Amazon S3 Account
Even more interesting would be a listing of all the objects in the Amazon S3 storage account. For simplicity, the code to retrieve the objects in a particular bucket is shown in Listing 3.
Listing 3. Getting a list of objects
$request = New-Object TypeName Amazon.S3.Model.ListObjectsRequest $request.BucketName = "[bucket-name]" $response = $client.ListObjects($request) $response.S3Objects|Format-Table -AutoSize Property Key,Size
Here's what this code does:
- The first and second lines create a
ListObjectsRequestobject and set one of its properties so that a particular bucket is queried to retrieve the listing of objects it contains.
- The third and fourth lines call the
ListObjectsmethod by passing the previous object as a constructor argument and save the results into a variable that is then used to get a list of all the objects. The list of objects is passed (via a pipeline) to a special formatting cmdlet that resizes the output and displays only a subset of the available properties.
Figure 4 shows the results of running Listing 3. If the
Format-Table cmdlet were removed from the previous code listing, the display would look much different (not shown).
Figure 4. Generating a list of objects
Writing Data to Amazon S3
You've seen code to retrieve a list of buckets and a list of objects. But what if you wanted to actually write data to the Amazon S3 service? Listing 4 shows how to upload several files from a specific folder to the Amazon S3 service.
Listing 4. Writing objects to the service