Using Windows PowerShell with the AWS SDK for .NET

DEPRECATED

This article is deprecated. Although you can still use the .NET Framework methods discussed in this article, for best results, use the AWS PowerShell Tools cmdlets for managing Amazon S3. For more information, see Amazon S3 from the AWS Tools for Windows PowerShell.

DEPRECATED

This article is deprecated. Although you can still use the .NET Framework methods discussed in this article, for best results, use the AWS PowerShell Tools cmdlets for managing Amazon S3. For more information, see Amazon S3 from the AWS Tools for Windows PowerShell.

Microsoft MVP Marco Shaw demonstrates how to use Microsoft Windows PowerShell 2 and the AWS SDK for .NET together using examples for Amazon S3.

Amazon Web Services (AWS), first introduced in early 2006, is a cloud-based infrastructure that provides web services for companies of all sizes. AWS has a number of infrastructure services, including:

• Amazon Elastic Compute Cloud (Amazon EC2). A web service that provides resizable compute capacity in the cloud
• Amazon Simple Storage Service (Amazon S3). A simple web services interface that you can use to store and retrieve large amounts of data at any time from anywhere on the web
• Amazon CloudFront. A web service for content delivery
• Amazon SimpleDB. A web service for running queries on structured data in real time
• Amazon Simple Queue Service (Amazon SQS). A reliable, highly scalable, hosted queue for storing messages as they travel between computers

Note: For a complete list of services, see https://aws.amazon.com/products.

This article shows how to use Windows PowerShell version 2 scripting with AWS—specifically, with Amazon S3—and the AWS software development kit (SDK) for Microsoft .NET. It provides a simple walkthrough of a few examples to lay the foundation for some exciting new possibilities. Some familiarity with Windows PowerShell scripting and Microsoft .NET Framework concepts would be helpful to fully understand all the concepts covered.

Amazon S3 is a cloud-based storage service that enables the uploading, downloading, and storage of data in the cloud. Amazon S3 can interact with other AWS services such as Amazon EC2 for a complete cloud-based solution.

There are three important concepts to understand when starting with Amazon S3:

• Objects. Objects are the fundamental entities stored in Amazon S3. They consist of data and metadata and can range in size from 1 byte to 5 GB. Think of an object as a simple file on any common operating system.
• Bucket. Objects are uploaded or stored in buckets. There is no limit to the number of objects that a bucket can hold. Think of a bucket as a simple folder on any common operating system.
• Key. A key is a unique identifier for an object within a bucket. Think of a key as the actual name of a regular file.

With the Amazon S3 service, there is a charge for storing objects in your buckets and for transferring objects in and out of these containers. Before you begin, be sure to sign up for Amazon S3 at https://aws.amazon.com/s3.

Windows PowerShell was first introduced publicly in 2003. After several years of development and testing, Microsoft formally released the scripting environment in November 2006 as a free download for all current operating systems at the time. In 2009, Windows PowerShell became part of the engineering standards (Common Engineering Criteria, or CEC) for all Microsoft server-based products.

Windows PowerShell version 2 was packaged with the Windows 7 and Windows Server 2008 R2, both released in October 2009. This time, not only was the environment bundled with the operating system, but it was considered part of the core operating system.

Windows PowerShell is built on the Microsoft .NET Framework 2.0. The Framework provides building blocks using an object-oriented approach to help programmers build applications for the Windows operating system. So, for anyone familiar with object-oriented programming concepts or who has used a language like C# to program, the learning curve for Windows PowerShell will be much easier to overcome.

You interact with the Windows PowerShell engine through a simple console. With Windows PowerShell version 2, an integrated development environment (IDE) is also provided (known as the Integrated Scripting Environment, or ISE). The Windows PowerShell console is shown in Figure 1.

Figure 1. The Windows PowerShell interface

The most basic building block in Windows PowerShell is called a cmdlet. Several cmdlets—version 2 has more than 200 core cmdlets—are bundled with the core operating system. Cmdlets have a verb–noun syntax; an example of one that is relatively easy to grasp is Get-Service, which provides a listing of all the services on the local system (see Figure 2).

Figure 2. Generating a list of the services on the local system

Learning Windows PowerShell is the subject of entire books, but for purposes of this article, here are the cmdlets used in the examples:

• New-Object. This cmdlet creates an instance of a Microsoft .NET class. The class is basically provided as an argument when the cmdlet is invoked.
• Format-Table. This cmdlet provides extra functionality in formatting how and what is displayed when Windows PowerShell commands are run.
• Add-Type. This cmdlet is used to load Microsoft .NET assemblies into the current session.
• Get-ChildItem. This cmdlet is used to list the items in a specific location. It is similar to the DOS command dir.

If you're new to Windows PowerShell, the Get-Help and Get-Command cmdlets are essential. You use Get-Help to retrieve the built-in help information that comes with Windows PowerShell. It can be used simply as Get-Help Get-Service, but it also accepts other switches that can provide complete documentation on all its options (Get-Help Get-Service –Full) or just the examples section (Get-Help Get-Service –Examples).

The Get-Command cmdlet can be used to retrieve a listing of all the cmdlets (using Get-Command alone) or cmdlets based on a particular verb or noun (for example, Get-Command *service*).

If you're a more advanced user, you can use Get-Member. The term member should look familiar to anyone who has been doing Microsoft .NET programming in either C# or in Microsoft Visual Basic.

One of the Windows PowerShell features that you'll be taking advantage of in this article is the fact that you can use most of the functionality the Microsoft .NET Framework provides. Even better, Windows PowerShell can actually load any Microsoft or third-party Microsoft .NET assemblies and usually make use of all of the features they provide.

Late in 2009, Amazon released the AWS SDK for Microsoft .NET (see "Learning More About AWS and Windows PowerShell"). This SDK allows Windows developers to create applications based on Microsoft .NET that use the AWS infrastructure services, including Amazon SS3, Amazon EC2, and Amazon SimpleDB. The SDK includes Microsoft Visual Studio project templates, the AWS .NET library, C# code samples, and documentation.

The following code can be run directly from a Windows PowerShell console or from the ISE. The ISE is installed by default on Windows 7, but you'll need to add it in Windows Server 2008 R2.

You can paste the code into a Windows PowerShell script (with a .ps1 extension), and then invoke it by a simple command like this:

PS> ./my_script.ps1

Note: Even though a .ps1 extension is used and powershell.exe is in a v1.0 sub-folder, if you're working on a Windows 7 or Windows Server 2008 R2 system, Windows PowerShell version 2 is actually installed. Also, by default, Windows PowerShell doesn't allow scripts to be run, so you may get an error when try. To enable the environment to run scripts, run the following command:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned

Setting the execution policy does have security implications, but using the value RemoteSigned is a good trade-off between usability and security.

Not only can Windows PowerShell run scripts, it can also be used interactively, so you can paste the code examples in this article directly into a Windows PowerShell console session and run them directly from there.

Note: When working with Windows PowerShell interactively, the console may display the characters >> at the beginning of a line. These characters simply indicate that the Windows PowerShell parser believes that you have not fully completed your command and is expecting more. Simply issuing a Ctrl+C key sequence returns the prompt so you can retry.

Now, let's dive into some actual examples and code. To run the example code as is, your system must have Windows PowerShell version 2 installed and the AWS SDK for .NET.

Begin by loading the AWS SDK assembly, defining the AWS access keys, and creating an AWS client, as shown in Listing 1.

Listing 1. Set up an AWS connection

Add-Type -Path "C:\Program Files (x86)\AWS SDK for .NET\bin\AWSSDK.dll"

$secretKeyID="[secret-key-id]"
$secretAccessKeyID="[secret-access-key-id]"

$client=[Amazon.AWSClientFactory]::CreateAmazonS3Client($secretKeyID,$secretAccessKeyID)

Here's what the code does:

• The Add-Type cmdlet loads the Microsoft .NET assembly into the current session or console by providing the full path to the DLL file.
• Two variables are created that hold the user's AWS account information.
• A documented Microsoft .NET static method is used to create a client to specifically access the Amazon S3 service using the variables just created.

All of the following code listings require that you add the code in Listing 1 to the start of the listing before the code actually runs.

One of the simplest examples you can run using the SDK is to retrieve a list of buckets for a particular account. Listing 2 produces a list of the buckets associated with the current account.

Listing 2. Invoking the ListBuckets method

$client.ListBuckets()

The above code listing shows how to use a documented method to retrieve a listing of the existing buckets. Everything in Windows PowerShell is object based; as a result, the engine displays all the properties associated with the objects that are returned from the command, as shown in Figure 3.

Figure 3. A list of the buckets from a given account

Even more interesting would be a listing of all the objects in the Amazon S3 storage account. For simplicity, the code to retrieve the objects in a particular bucket is shown in Listing 3.

Listing 3. Getting a list of objects

$request = New-Object —TypeName Amazon.S3.Model.ListObjectsRequest
$request.BucketName = "[bucket-name]"
	
$response = $client.ListObjects($request)
$response.S3Objects|Format-Table -AutoSize —Property Key,Size

Here's what this code does:

• The first and second lines create a ListObjectsRequest object and set one of its properties so that a particular bucket is queried to retrieve the listing of objects it contains.
• The third and fourth lines call the ListObjects method by passing the previous object as a constructor argument and save the results into a variable that is then used to get a list of all the objects. The list of objects is passed (via a pipeline) to a special formatting cmdlet that resizes the output and displays only a subset of the available properties.

Figure 4 shows the results of running Listing 3. If the Format-Table cmdlet were removed from the previous code listing, the display would look much different (not shown).

Figure 4. Generating a list of objects

You've seen code to retrieve a list of buckets and a list of objects. But what if you wanted to actually write data to the Amazon S3 service? Listing 4 shows how to upload several files from a specific folder to the Amazon S3 service.

Listing 4. Writing objects to the service

$backup_directory="C:\directory_to_backup"

foreach($file in Get-ChildItem -Path $backup_directory){
  $request = New-Object -TypeName Amazon.S3.Model.PutObjectRequest
  [void]$request.WithFilePath($file.fullname)
  [void]$request.WithBucketName("[bucket-name]")
  [void]$request.WithKey($file.name)
  [void]$client.PutObject($request)
}

Here's what the above code does:

• Create a variable representing the directory that has the files you're looking to write to the service.
• A foreach loop is used to retrieve a listing of all the files in the directory.
• Inside the foreach loop, a PutObjecRequest object is created for each file, then methods are called to set some of the properties, such as the local path to the file, the bucket to write the object to, and the key or name of the object once written. The Windows PowerShell type accelerator [void] is used to delete any output provided when the methods are invoked.
• The final line in the foreach loop has the PubObject method of the AWS client to actually invoke the write process.

Running the above code will not result in output unless an error has occurred. You could easily add error checking and trapping to the above listing.

This article provided a quick overview of the new AWS SDK for Microsoft .NET and Windows PowerShell and also showed some simple yet practical examples of how you can use these technologies together to provide some exciting new opportunities.

Even developers who haven't used Windows PowerShell before may begin to see opportunities where the engine can help with things like research and development of code by being able to quickly string code together for proofs of concept or quick demonstrations.

This article highlights some aspects of working with AWS. Here are a few more resources to help you learn more:

• AWS web site. Learn more about each web service on the AWS web site.
• Developer Connection. The community web site for AWS developers includes forums on AWS, a Solutions Catalog for examples of what your peers have built, and more.
• Resource Center. Part of the Developer Connection web site, the Resource Center has links to tutorials, code samples, technical documentation, and other resources for building your application on AWS.
• AWS SDK for .NET. Get more information on the SDK, including sample code and complete application programming interface (API) documentation.
• Windows PowerShell version 2. Get Windows PowerShell for older operating systems.
• PowerShell team blog. Follow the Windows PowerShell development team at Microsoft.
• PowerShell Community Site. Get involved with the Windows PowerShell community.
• PowerShell scripts. Find sample scripts to get you started.

Marco Shaw is currently an independent contractor living in Atlantic Canada. He has been working in the IT industry for over 12 years. He was awarded a Microsoft Most Valuable Professional award for his contributions to the Windows PowerShell community in 2008, 2009, and again in 2010. Marco spoke at TechMentor in San Francisco in 2008, where he provided two popular sessions on Windows PowerShell. At TechDays in Canada (November 2009) he presented two sessions, including a packed room for a presentation on Windows Server 2008 R2 improvements. His recent author activities include writing Windows PowerShell content for a Windows Server 2008 book by Microsoft Press, a Windows PowerShell–related article on Microsoft System Center Operations Manager 2007 for TechNet Magazine, providing Windows PowerShell content for a Microsoft SQL Server 2008 book by SAMs and also for a revised edition of Microsoft System Center Operations Manager 2007 Unleashed by SAMs. He is the co-author of the second edition of PowerShell Unleashed, published by SAMs and released early in 2009.