Category: .NET


Using NuGet and Chocolatey package managers in AWS CloudFormation and AWS Elastic Beanstalk

by Jim Flanagan | on | in .NET | Permalink | Comments |  Share

In this guest post by AWS Solutions Architect Lee Atkinson, we are going to describe how you can take advantage of the NuGet and Chocolatey package managers inside your CloudFormation templates and Elastic Beanstalk applications.

AWS CloudFormation and AWS Elastic Beanstalk support the Microsoft Windows Installer for installing .msi files onto Microsoft Windows instances managed by those services. For details on how to do this for CloudFormation, see http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-init.html#aws-resource-init-packages, and for Elastic Beanstalk, see http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-windows-ec2.html#customize-containers-windows-format-packages.

NuGet (pronounced ‘New Get’) is a package manager for installing .NET development packages onto your development machine. It is available as a Microsoft Windows Visual Studio plugin as well as a standalone command line tool. Users can install packages from, and publish packages to, a central repository for packages located at http://www.nuget.org/.

Chocolatey NuGet builds on top of NuGet to provide a package manager for Microsoft Windows applications and describes itself as "a Machine Package Manager, somewhat like apt-get, but built with Windows in mind." It has a command line tool and a central repository located at http://chocolatey.org/.

AWS CloudFormation supports the downloading of files and execution of commands on EC2 instance creation using an application called ‘cfn-init.exe’ installed on instances running Microsoft Windows. We can leverage this functionality to install and execute both NuGet and Chocolatey. For more information on bootstrapping Microsoft Windows instances in CloudFormation, see http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-windows-stacks-bootstrapping.html.

Similarly, AWS Elastic Beanstalk supports the downloading of files and execution of commands on instance creation using container customization. We can use this functionality to install and execute both NuGet and Chocolatey. For more information on customizing Microsoft Windows containers in Elastic Beanstalk, see http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-windows-ec2.html.

Using NuGet in AWS CloudFormation

For installing NuGet packages on an EC2 instance using AWS CloudFormation, we can use the NuGet command line tool. First, we need to download the tool to the Microsoft Windows instance, and we can use the CloudFormation ‘file’ declaration. Then, to install NuGet packages, we can use a CloudFormation ‘command’ declaration.

Here’s an excerpt from an example AWS CloudFormation template to:

  1. Download NuGet.exe
  2. Install the JSON.NET NuGet package
  3. Install the Entity Framework NuGet package
"AWS::CloudFormation::Init": {
  "config": {
    "files" : {
      "c:/tools/nuget.exe" : {
        "source" : "https://nuget.org/nuget.exe"
      }
    },
    "commands" : {
      "1-create-myapp-folder" : {
        "command" : "if not exist c:\myapp mkdir c:\myapp",
        "waitAfterCompletion" : "0"
      },
      "2-install-json-net" : {
        "command" : "c:\tools\nuget install Newtonsoft.Json -NonInteractive -OutputDirectory c:\myapp",
        "waitAfterCompletion" : "0"
      },
      "3-install-entityframework" : {
        "command" : "c:\tools\nuget install EntityFramework -NonInteractive -OutputDirectory c:\myapp",
        "waitAfterCompletion" : "0"
      }
    }
  }
}

Using Chocolatey in AWS CloudFormation

Installing and using Chocolatey is similar to NuGet above, though the recommended way of installing Chocolatey is to execute a Microsoft Windows PowerShell script. As CloudFormation ‘command’ declarations are executed by cmd.exe, we need to execute PowerShell.exe and provide the install command to that.

The Chocolatey installer and the packages it installs may modify the machine’s PATH environment variable. This adds complexity since subsequent commands after these installations are executed in the same session, which does not have the updated PATH. To overcome this, we utilize a command file to set the session’s PATH to that of the machine before it executes our command.

Here’s an excerpt from an example AWS CloudFormation template to:

  1. Create a command file ‘ewmp.cmd’ to execute a command with the machine’s PATH
  2. Install Chocolatey
  3. Install Sublime Text 3
  4. Install Firefox
"AWS::CloudFormation::Init": {
  "config": {
    "files" : {
      "c:/tools/ewmp.cmd" : {
        "content": "@ECHO OFFnFOR /F "tokens=3,*" %%a IN ('REG QUERY "HKLM\System\CurrentControlSet\Control\Session Manager\Environment" /v PATH') DO PATH %%a%%bn%*"
      }
    },
    "commands" : {
      "1-install-chocolatey" : {
        "command" : "powershell -NoProfile -ExecutionPolicy unrestricted -Command "Invoke-Expression ((New-Object Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))""
      },
      "2-install-sublimetext" : {
        "command" : "c:\tools\ewmp choco install sublimetext3"
      },
      "3-install-firefox" : {
        "command" : "c:\tools\ewmp choco install firefox"
      }
    }
  }
}

Using NuGet and Chocolatey together in AWS CloudFormation

Another example for NuGet is when you are cloning a repository from a version control system that does not have the NuGet packages checked-in, which means those packages are missing from the clone. In this case, you can perform a NuGet Restore, which instructs NuGet to download the packages specified within the repository.

But we need to install git before we can clone—so we use Chocolatey!

Here’s an excerpt from an example AWS CloudFormation template to:

  1. Download NuGet.exe
  2. Create a command file ‘ewmp.cmd’ to execute a command with the machine’s PATH
  3. Install Chocolatey
  4. Install Git
  5. Clone a Git repository
  6. Restore NuGet packages defined in the repository’s solution file
"AWS::CloudFormation::Init": {
  "config": {
    "files" : {
      "c:/tools/nuget.exe" : {
        "source" : "https://nuget.org/nuget.exe"
      },
      "c:/tools/ewmp.cmd" : {
        "content": "@ECHO OFFnFOR /F "tokens=3,*" %%a IN ('REG QUERY "HKLM\System\CurrentControlSet\Control\Session Manager\Environment" /v PATH') DO PATH %%a%%bn%*"
      }
    },
    "commands" : {
      "1-install-chocolatey" : {
        "command" : "powershell -NoProfile -ExecutionPolicy unrestricted -Command "Invoke-Expression ((New-Object Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))"",
        "waitAfterCompletion" : "0"
      },
      "2-install-git" : {
        "command" : "c:\tools\ewmp choco install git",
        "waitAfterCompletion" : "0"
      },
      "3-create-myapp-folder" : {
        "command" : "if not exist c:\myapp mkdir c:\myapp",
        "waitAfterCompletion" : "0"
      },
      "4-clone-repo" : {
        "command" : "c:\tools\ewmp git clone git://github.com/aws/aws-sdk-net c:\myapp",
        "waitAfterCompletion" : "0"
      },
      "5-nuget-restore" : {
        "command" : "c:\tools\nuget restore c:\myapp\AWSSDK_DotNet.Mobile.sln",
        "waitAfterCompletion" : "0"
      }
    }
  }
}

Using NuGet and Chocolatey in AWS Elastic Beanstalk

The above examples can be translated into AWS Elastic Beanstalk config files to enable use of both NuGet and Chocolatey in Elastic Beanstalk. For Elastic Beanstalk, we create YAML .config files inside the .ebextensions folder of our source bundle.

Here’s an example .ebextensions config file to:

  1. Download NuGet.exe
  2. Install the JSON.NET NuGet package
  3. Install the Entity Framework NuGet package
files:
  c:/tools/nuget.exe:
    source: https://nuget.org/nuget.exe
commands:
  1-create-myapp-folder:
    command: if not exist c:myapp mkdir c:myapp
    waitAfterCompletion: 0
  2-install-json-net:
    command: c:toolsnuget install Newtonsoft.Json -NonInteractive -OutputDirectory c:myapp
    waitAfterCompletion: 0
  3-install-entityframework:
    command: c:toolsnuget install EntityFramework -NonInteractive -OutputDirectory c:myapp
    waitAfterCompletion: 0

Here’s an example .ebextensions config file to:

  1. Create a command file ‘ewmp.cmd’ to execute a command with the machine’s PATH
  2. Install Chocolatey
  3. Install Sublime Text 3
  4. Install Firefox
files:
  c:/tools/ewmp.cmd:
    content: |
      @ECHO OFF
      FOR /F "tokens=3,*" %%a IN ('REG QUERY "HKLMSystemCurrentControlSetControlSession ManagerEnvironment" /v PATH') DO PATH %%a%%b
      %*
commands:
  1-install-chocolatey:
    command: powershell -NoProfile -ExecutionPolicy unrestricted -Command "Invoke-Expression ((New-Object Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))"
  2-install-sublimetext:
    command: c:toolsewmp choco install sublimetext3
  3-install-firefox:
    command: c:toolsewmp choco install firefox

Here’s an example .ebextensions config file to:

  1. Download NuGet.exe
  2. Create a command file ‘ewmp.cmd’ to execute a command with the machine’s PATH
  3. Install Chocolatey
  4. Install Git
  5. Clone a Git repository
  6. Restore NuGet packages defined in the repository’s solution file
files:
  c:/tools/nuget.exe:
    source: https://nuget.org/nuget.exe
  c:/tools/ewmp.cmd:
    content: |
      @ECHO OFF
      FOR /F "tokens=3,*" %%a IN ('REG QUERY "HKLMSystemCurrentControlSetControlSession ManagerEnvironment" /v PATH') DO PATH %%a%%b
      %*
commands:
  1-install-chocolatey:
    command: powershell -NoProfile -ExecutionPolicy unrestricted -Command "Invoke-Expression ((New-Object Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))"
    waitAfterCompletion: 0
  2-install-git:
    command: c:toolsewmp choco install git
    waitAfterCompletion: 0
  3-create-myapp-folder:
    command: if not exist c:myapp mkdir c:myapp
    waitAfterCompletion: 0
  4-clone-repo:
    command: c:toolsewmp git clone git://github.com/aws/aws-sdk-net c:myapp
    waitAfterCompletion: 0
  5-nuget-restore:
    command: c:toolsnuget restore c:myappAWSSDK_DotNet.Mobile.sln
    waitAfterCompletion: 0

Summary

I hope this provides inspiration on how you can leverage both NuGet and Chocolatey to configure your Microsoft Windows instances managed by either AWS CloudFormation or AWS Elastic Beanstalk.

Mapping Cmdlets to AWS Service APIs

by Steve Roberts | on | in .NET | Permalink | Comments |  Share

The consistency of the standardized verb and naming scheme used by Windows PowerShell makes learning the basics of the shell relatively easy, but translating knowledge of an existing API to the standard names can be difficult at first. Starting with version 2.3.19, AWS Tools for Windows PowerShell contains a new cmdlet to help with discovery: Get-AWSCmdletName. This cmdlet accepts the name of an AWS service API and emits the names of cmdlets that invoke an API matching that name pattern. It can also accept an AWS CLI command line and give you back the corresponding PowerShell cmdlet—handy if you are converting an AWS CLI sample.

Discovering Service APIs

Running the PowerShell Get-Command cmdlet with verb and/or noun filtering only gets you so far in discovering the cmdlets that are available in a module. You as a user still need to make the mental leap to associate the verb and noun combination to a known service API. Sometimes this is obvious, sometimes not so much. To get the name of a cmdlet that invokes a known AWS service API is now as easy as:

PS C:> Get-AWSCmdletName -ApiOperation describeinstances

CmdletName              ServiceOperation
----------              ----------------
Get-EC2Instance         DescribeInstances
Get-OPSInstances        DescribeInstances

Note that the full name of the service, and the noun prefix, are displayed in additional columns that are not shown in these examples for brevity.

The parameter name -ApiOperation can be omitted to save typing. You can see from the output that the cmdlet has scanned all cmdlets contained in the AWS PowerShell module and output those that invoke a service API DescribeInstances regardless of the service.

If you know the service of interest, you can restrict the search using the optional -Service parameter:

PS C:> Get-AWSCmdletName describeinstances -Service ec2

CmdletName              ServiceOperation
----------              ----------------
Get-EC2Instance         DescribeInstances

The value supplied to the -Service parameter can be either the prefix code that is applied to the noun part of the name of cmdlets belonging to a service, or one or more words from the service name. For example, these two commands return the same output as the example above:

PS C:> Get-AWSCmdletName describeinstances -Service compute
PS C:> Get-AWSCmdletName describeinstances -Service "compute cloud"

Note that all searches are case insensitive.

If you know the exact name of the service API you are interested in, then you are good to go. But what if you want to find all cmdlets that have something to do with, say, security groups (based on the premise that the term ‘securitygroup’ forms part of the API name)? You might try this:

PS C:> Get-AWSCmdletName securitygroup

As you’ll see if you run the example, the cmdlet displays no output because there is no service API matching that name. What we need is a more flexible way to specify the pattern to match. You can do this by adding the -MatchWithRegex switch:

PS C:> Get-AWSCmdletName securitygroup -MatchWithRegex

CmdletName                              ServiceOperation
----------                              ----------------
Approve-ECCacheSecurityGroupIngress     AuthorizeCacheSecurityGroupIngress
Get-ECCacheSecurityGroup                DescribeCacheSecurityGroups
New-ECCacheSecurityGroup                CreateCacheSecurityGroup
Remove-ECCacheSecurityGroup             DeleteCacheSecurityGroup
Revoke-ECCacheSecurityGroupIngress      RevokeCacheSecurityGroupIngress
Get-EC2SecurityGroup                    DescribeSecurityGroups
Grant-EC2SecurityGroupEgress            AuthorizeSecurityGroupEgress
Grant-EC2SecurityGroupIngress           AuthorizeSecurityGroupIngress
New-EC2SecurityGroup                    CreateSecurityGroup
Remove-EC2SecurityGroup                 DeleteSecurityGroup
Revoke-EC2SecurityGroupEgress           RevokeSecurityGroupEgress
Revoke-EC2SecurityGroupIngress          RevokeSecurityGroupIngress
Join-ELBSecurityGroupToLoadBalancer     ApplySecurityGroupsToLoadBalancer
Enable-RDSDBSecurityGroupIngress        AuthorizeDBSecurityGroupIngress
Get-RDSDBSecurityGroup                  DescribeDBSecurityGroups
New-RDSDBSecurityGroup                  CreateDBSecurityGroup
Remove-RDSDBSecurityGroup               DeleteDBSecurityGroup
Revoke-RDSDBSecurityGroupIngress        RevokeDBSecurityGroupIngress
Approve-RSClusterSecurityGroupIngress   AuthorizeClusterSecurityGroupIngress
Get-RSClusterSecurityGroups             DescribeClusterSecurityGroups
New-RSClusterSecurityGroup              CreateClusterSecurityGroup
Remove-RSClusterSecurityGroup           DeleteClusterSecurityGroup
Revoke-RSClusterSecurityGroupIngress    RevokeClusterSecurityGroupIngress

As you can see its now easy to find all cmdlets that have something to do with a particular term, or object, across all services. When the -MatchWithRegex parameter is used the value of the -ApiOperation parameter is interpreted as a regular expression.

If we wanted to restrict the search to a specific service, we would just add the -Service parameter too, as shown earlier. The -Service parameter value always accepts a regular expression and is not affected by the -MatchWithRegex switch. When looking at the name of the owning service for a cmdlet, Get-AWSCmdletName automatically uses the -Service value as a regular expression, and if that does not yield a match, it then attempts to use the value in a simple text comparison on the service prefix that is used in cmdlet names to effectively namespace the cmdlets.

Translating from AWS CLI

The verb-noun naming standard of PowerShell is considered one of its strengths and one that we are pleased to support to give users a consistent experience. The AWS CLI follows more closely the AWS API naming conventions. Get-AWSCmdletName has one further ability and that is to be able to make a "best effort" at translating an AWS CLI command line to yield the corresponding AWS PowerShell cmdlet. This can be useful when translating a sample:

PS C:> Get-AWSCmdletName -AwsCliCommand "aws ec2 authorize-security-group-ingress"

CmdletName                           ServiceOperation
----------                           ----------------
Grant-EC2SecurityGroupIngress        AuthorizeSecurityGroupIngress

The supplied AWS CLI command is parsed to recover the service identifier and the operation name (which is stripped of any hyphens). You only need to specify enough of the command to allow the service and operation to be identified – the "aws" prefix in the parameter value can be omitted. Also, if you’ve pasted the parameter value from a sample and it contains any CLI options—identified by a ‘–‘ prefix— they are skipped.

Hopefully, you’ll find this new cmdlet useful in discovering and navigating the cmdlets available for working with AWS. Do you have an idea for something that would be useful for you and potentially others? Let us know in the comments!

Automatic Pagination of Responses in the AWS SDK for .NET (Preview)

by Jim Flanagan | on | in .NET | Permalink | Comments |  Share

As part of our recent preview release of Resource APIs for .NET we have exposed one of the underlying features in the low-level .NET SDK as well.

Many of the AWS APIs that return collections of items have a pagination interface. Rather than return all results at once, the service returns a certain number of results per request, and provides a token in the response to get the next "page" of results. In this way, you can chain requests together using the token to get as many results as you need.

Here’s what that looks like using the SDK for .NET to get all the IAM users for an account, 20 at a time:

ListUsersResponse response;
ListUsersRequest request = new ListUsersRequest { MaxItems = 20 };

do
{
    response = iam.ListUsers(request);
    ProcessUsers(response.Users);
    request.Marker = response.Marker;
}
while (response.IsTruncated);

In order to make the resource APIs feel more natural, we built in a mechanism that does something like the above code behind the scenes through an IEnumerable interface. Using the resource APIs, you can get the users like this:

var users = iam.GetUsers();
foreach (var user in users)
{
    Console.WriteLine("User: {0}", user.Name);
}

The first line does not result in a service call. No service calls will be made until your code starts iterating over IEnumerable, and subsequent calls will be made as needed under the covers.

This seemed useful to expose through the low-level API as well, so we added methods on some of the clients as part of the SDK Preview for the following clients:

  • Amazon.GlacierClient
  • Amazon.IdentityManagementServiceClient
  • Amazon.OpsWorksClient
  • Amazon.SimpleNotificationServiceClient

Using the paginators from the low-level request interface looks like this:

var users = client.ListUsersEnumerator(new ListUsersRequest { MaxItems = 20 });
foreach(var user in users)
{
    Console.WriteLine("User: {0}", user.Name);
}

As usual with IEnumerable, you will need to pay special attention when using LINQ and/or the System.Linq.Enumerable extension methods. Calling various extensions like .Count(), .Where(), or .Last() on one of the IEnumerables returned by these methods could result in multiple, unintended calls to the service. In those instances where you do need to use those methods, it can be a good idea to cache the IEnumerable returned for as long as possible.

Let us know if you find this facility useful. We look forward to hearing from you on GitHub and the AWS forums.

Upcoming Modularization of the AWS SDK for .NET

by Norm Johanson | on | in .NET | Permalink | Comments |  Share

Today, I would like to announce our plans to modularize the AWS SDK for .NET into individual assemblies and NuGet packages. This work will take us a few months to complete, but we recognize this will be a pretty big change to how developers see the SDK and want to give as much of a heads-up as we can. It is our intention to have as few breaking changes as possible, but a few will be unavoidable. For those changes, we plan on marking obsolete in the current version SDK as soon as possible. The most notable breaking change will be the removal of Amazon.AWSClientFactory because that class requires a reference to every service.

Why are we doing this?

When we first released the AWS SDK for .NET, there were 10 services and the total size of the SDK was about 600 KB. Today, the SDK has grown to support over 40 services and has grown to over 6 MB. We’ve heard from many of our users that they want a smaller SDK containing just the services they need. This is especially important for developers who are using our SDK for Windows Phone and Windows Store Apps.

Another reason we are doing this is the frequency of releases from AWS. If you take a look at our Nuget package, you can see we release the SDK nearly weekly, sometimes even more frequently. Our hope is that this change will allow developers to update their SDK only when the services they use are updated.

What happens next?

It will take a us a few months to update our build and release process. We’ll keep you updated as more information becomes available. Watch for methods being marked as obsolete, and move away from them as soon as possible. As we are doing all this refactoring, this is a perfect time for feedback from users of the SDK. If there are problems the SDK is not solving for you or things are hard to discover, let us know. You can give feedback either here, in our forums, or on GitHub.

Updated Amazon Cognito Credentials Provider

by Pavel Safronov | on | in .NET | Permalink | Comments |  Share

Amazon Cognito allows you to get temporary AWS credentials, so that you don’t have to distribute your own credentials with your application. Last year we added a Cognito credentials provider to the AWS SDK for .NET to simplify this process.

With the latest update to Cognito, we are now making it even easier to use Cognito with your application. Using the latest version of the SDK, you no longer need to specify IAM roles in your application if you have already associated the correct roles with your identity pool.

Below is an example of how you can construct and use the new credentials provider:

CognitoAWSCredentials credentials = new CognitoAWSCredentials(
    identityPoolId,   // identity pool id
    region);          // identity pool region

using (var s3Client = new AmazonS3Client(credentials))
{
    s3Client.ListBuckets();
}

Something to note is that even if you have associated roles with an identity pool, you can still specify IAM roles—even ones that are different from the roles configured on the identity pool—when creating these credentials. This gives you finer control over what resources and operations these credentials have access to.

Cross-Account IAM Roles in Windows PowerShell

by Brian Beach | on | in .NET | Permalink | Comments |  Share

As a company’s adoption of Amazon Web Services (AWS) grows, most customers adopt a multi-account strategy. Some customers choose to create an account for each application, while others create an account for each business unit or environment (development, testing, production). Whatever the strategy, there is often a use case that requires access to multiple accounts at once. This post examines cross-account access and the AssumeRole API, known as Use-STSRole in Windows PowerShell.

A role consists of a set of permissions that grant access to actions or resources in AWS. An application uses a role by calling the AssumeRole API function. The function returns a set of temporary credentials that the application can use in subsequent function calls. Cross-account roles allow an application in one account to assume a role (and act on resources) in another account.

One common example of cross-account access is maintaining a configuration management database (CMDB). Most large enterprise customers have a requirement that all servers, including EC2 instances, must be tracked in the CMDB. Example Corp., shown in Figure 1, has a Payer account and three linked accounts: Development, Testing, and Production.

Figure 1: Multiple Accounts Owned by a Single Customer

Note that linked accounts are not required to use cross-account roles, but they are often used together. You can use cross-account roles to access accounts that are not part of a consolidated billing relationship or between accounts owned by different companies. See the user guide to learn more about linked accounts and consolidated billing.

Scenario

Bob, a Windows administrator at Example Corp., is tasked with maintaining an inventory of all the instances in each account. Specifically, he needs to send a list of all EC2 instances in all accounts to the CMDB team each night. He plans to create a Windows PowerShell script to do this.

Bob could create an IAM user in each account and hard-code the credentials in the script. Though this would be simple, hard-coding credentials is not the most secure solution. The AWS best practice is to Use IAM roles. Bob is familiar with IAM roles for Amazon EC2 and wants to learn more about cross-account roles.

Bob plans to script the process shown in Figure 2. The CMDB script will run on an EC2 instance using the CMDBApplication role. For each account, the script will call Use-STSRole to retrieve a set of temporary credentials for the CMDBDiscovery role. The script will then iterate over each region and call Get-EC2Instance using the CMDBDiscovery credentials to access the appropriate account and list all of its instances.

Figure 2: CMDB Application and Supporting IAM Roles

Creating IAM Roles

Bob begins to build his solution by creating the IAM roles shown in Figure 3. The Windows PowerShell script will run on a new EC2 instance in the Payer account. Bob creates a CMDBApplication role in the Payer account. This role in used by the EC2Instance allowing the script to run without requiring IAM user credentials. In addition, Bob will create a CMDBDiscovery role in every account. The CMDBDiscovery role has permission to list (or “Discover”) the instances in that account.

Figure 3: CMDB Application and Supporting IAM Roles

Notice that Bob has created two roles in the Payer account: CMDBApplication and CMDBDiscovery. You may be asking why he needs a cross-account role in the same account as the application. Creating the CMDBDiscovery role in every account makes the code easier to write because all accounts are treated the same. Bob can treat the Payer account just like any of the other accounts without a special code branch.

Bob first creates the Amazon EC2 role, CMDBApplication, in the Payer account. This role will be used by the EC2 instance that runs the Windows PowerShell script. Bob signs in to the AWS Management Console for the Payer account and follows the instructions to create a new IAM Role for Amazon EC2 with the following custom policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["sts:AssumeRole"],
      "Resource": ["*"]
    }
  ]
}

Policy 1: Policy Definition for the CMDBApplication IAM Role

The CMDBApplication role grants a single permission, sts:AssumeRole, which allows the application to call the AssumeRole API to get temporary credentials for another account. Notice that Bob is following the best practice of Least Privilege and has assigned only one permission to the application.

Next, Bob creates a cross-account role called CMDBDiscovery in each of the accounts, including the Payer account. This role will be used to list the EC2 instances in that account. Bob signs in to the console for each account and follows the instructions to create a new IAM role for cross-account access. In the wizard, Bob supplies the account ID of the Payer account (111111111111 in our example) and specifies the following custom policy.

Note that when creating the role, there are two options. One provides access between accounts you own, and the other provides access from a third-party account. Third-party account roles include an external ID, which is outside the scope of this article. Bob chooses the first option because his company owns both accounts.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["ec2:DescribeInstances"],
      "Resource": ["*"]
    }
  ]
}

Policy 2: Policy Definition for the CMDBDiscovery IAM Role

Again, this policy follows the best practice of least privilege and assigns a single permission,ec2:DescribeInstances, which allows the caller to list the EC2 instances in the account.

Creating the CMDB Script

With the IAM roles created, Bob next launches a new EC2 instance in the Payer account. This instance will use the CMDBApplication role. When the instance is ready, Bob signs in and creates a Windows PowerShell script that will list the instances in each account and region.

The first part of the script, shown in Listing 1, lists the instances in a given region and account. Notice that in addition to the account number and region, the function expects a set of credentials. These credentials represent the CMDBDiscovery role and will be retrieved from the AssumeRole API in the second part of the script.

Function ListInstances {
    Param($Credentials, $Account, $Region)
          
    #List all instances in the region
    (Get-EC2Instance -Credential $Credentials -Region $Region).Instances | % {
        If($Instance = $_) {
  
            #If there are instances in this region return the desired attributes
            New-Object PSObject -Property @{
                Account = $Account
                Region = $Region
                InstanceId = $Instance.InstanceId
                Name = ($Instance.Tags | Where-Object {$_.Key -eq 'Name'}).Value
            }
        }
    }
}

Listing 1: Windows PowerShell Function to List EC2 Instances

The magic happens in the second part of the script, shown in Listing 2. We know that the script is running on the new EC2 instance using the CMDBApplication. Remember that the only thing this role can do is call the AssumeRole API. Therefore, we should expect to see a call to AssumeRole. The Windows PowerShell cmdlet that implements AssumeRole is Use-STSRole.

#List of accounts to check
$Accounts = @(111111111111, 222222222222, 333333333333, 444444444444)
  
#Iterate over each account
$Accounts | % {
    $Account = $_
    $RoleArn = "arn:aws:iam::${Account}:role/CMDBDiscovery"
  
    #Request temporary credentials for each account and create a credential object
    $Response = (Use-STSRole -Region $Region -RoleArn $RoleArn -RoleSessionName 'CMDB').Credentials
    $Credentials = New-AWSCredentials -AccessKey $Response.AccessKeyId -SecretKey $Response.SecretAccessKey -SessionToken $Response.SessionToken
  
    #Iterate over all regions and list instances
    Get-AWSRegion | % {
        ListInstances -Credential $Credentials -Account $Account -Region $_.Region
    }
  
} | ConvertTo-Csv

Listing 2: Windows PowerShell Script That Calls AssumeRole

Use-STSRole will retrieve temporary credentials for the IAM role specified in the ARN parameter. The ARN uses the following format, where “ROLE_NAME” is the role you created in “TARGET_ACCOUNT_NUMBER” (e.g. CMDBDiscovery).

arn:aws:iam::TARGET_ACCOUNT_NUMBER:role/ROLE_NAME

Use-STSRole will return an AccessKey, a SecretKey, and a SessionToken that can be used to access the account specified in the role ARN. The script uses this information to create a new credential object, which it passes to ListInstances. ListInstances uses the credential object to list EC2 instances in the account specified in the role ARN.

That’s all there is to it. Bob creates a scheduled task that executes this script each night and sends the results to the CMDB team. When the company adds additional accounts, Bob simply adds the CMDBDiscovery role to the new account and updates the account list in his script.

Summary

Cross-account roles are a valuable tool for large customers with multiple accounts. Roles provide temporary credentials that a user or application in one account can use to access resources in another account. These temporary credentials do not need to be stored or rotated, resulting in a secure and maintainable architecture.

Alternative Formatting for Metrics Data in .NET SDK Logs

by Jim Flanagan | on | in .NET | Permalink | Comments |  Share

The AWS SDK for .NET has had response logging and performance metrics logging since before version 2.0. We introduced SDK logging and metrics output in an earlier post. You might want to skim that as a refresher.

The metrics data is included in the logs in a human-readable format, but SDK users who aggregate, analyze, and report on this data have to implement their own parser to parse the data out of the logs, which takes time, and can be error prone. So, we’ve added an alternate format to emit the metrics data into the log as JSON.

If you need a format other than JSON, or if you only need to log a subset of the metrics, the SDK now also has a mechanism to add a custom formatter.

Switching to JSON is done through the App.config or Web.config. The SDK’s application configuration has changed a little bit since the aforementioned post; though, for the sake of backwards compatibility, the original configuration settings will still work. To use the JSON setting, however, you’ll have to adopt the new style of configuration, at least for the logging section.

The old style of configuration was a set of flat key-value pairs in the <appSettings> section, like this:

<appSettings>
    <add key="AWSLogging" value="SystemDiagnostics" />
    <add key="AWSLogMetrics" value="true" />
    <add key="AWSResponseLogging" value="OnError" />
</appSettings>

The new configuration uses a custom section for the SDK with a structured format, like this:

<configuration>
    <configSections>
        <section name="aws" type="Amazon.AWSSection, AWSSDK" />
    </configSections>
    ...
    <aws region="us-west-2">
        <logging logTo="SystemDiagnostics"
             logMetrics="true"
             logResponses="OnError"
             logMetricsFormat="JSON" />
    </aws>
    ...
</configuration>

You can see that this configuration selects the JSON formatting. The rest of the logging configuration, including selection of System.Diagnostics or Log4Net, is the same as specified in the introductory logging post.

Creating a custom formatter is easy too. First, you need to implement the Amazon.Runtime.IMetricsFormatter interface, specifying a single method that takes in an Amazon.Runtime.IRequestMetrics and returns a string. Here’s a trivial example that prints out a single metric for a request:

using Amazon.Runtime;
namespace MyLib.Util
{
    public class MyMetricsFormatter : IMetricsFormatter
    {
        public string FormatMetrics(IRequestMetrics metrics)
        {
            var fmt = string.Empty;
            if (metrics.Timings.ContainsKey(Metric.ResponseProcessingTime))
            {
                var timing = metrics.Timings[Metric.ResponseProcessingTime]
                    .FirstOrDefault();

                if (timing != null)
                    fmt = string.Format("ResponseProcessingTime (ms): {0}", 
                            timing.ElapsedTime.Milliseconds);
            }
            return fmt;
        }
    }
}

The IRequestMetrics has three dictionaries of metrics; Properties, Timings, and Counters. The keys for these dictionaries are defined in the Amazon.Runtime.Metric enum. The Properties and Timings dictionaries have lists as values, and the the Counters dictionary has long as values.

To use a custom formatter, use the logMetricsCustomFormatter configuration, specifying the type and assembly:

<aws region="us-west-2">
    <logging logTo="SystemDiagnostics"
         logMetrics="true"
         logResponses="OnError"
         logMetricsCustomFormater="MyLib.Util.MyMetricsFormatter, MyLib" />
</aws>

If you want to collect metrics for a subset of services or method calls, your custom formatter can inspect the Metrics.ServiceName and Metrics.MethodName items in the Properties dictionary. The default behavior can be accessed by calling ToString() on the passed in IRequestMetrics. Similarly, you can get the JSON by calling metrics.ToJSON().

Keep in mind that if you have metrics logging enabled and have specified a custom formatter your formatter will be called for every request, so keep it as simple as possible.

Support for Amazon SNS in the Preview Release of AWS Resource APIs for .NET

by Milind Gokarn | on | in .NET | Permalink | Comments |  Share

The latest addition to the AWS Resource APIs for .NET is Amazon Simple Notification Service (SNS). Amazon SNS is a web service that enables applications, end-users, and devices to instantly send and receive notifications. In this post, we’ll see how we can use the resource APIs to work with SNS and to publish messages.

Topics

The key concept in SNS is a topic. A topic is something that publishers send messages to and subscribers receive messages from. Let’s take a look at how we can create and use a topic.

using Amazon.SimpleNotificationService.Model;
using Amazon.SimpleNotificationService.Resources; // Namespace for SNS resource APIs

// Create an instance of the SNS service 
// You can also use the overload that accepts an instance of the service client.
var sns = new SimpleNotificationService();

// Create a new topic
var topic = sns.CreateTopic("testTopic");

// Check that the topic is now in the list of all topics
// To do this, we can retrieve a list of all topics and check that.
var exists = sns.GetTopics()
    .Any(t => t.Arn.Equals(topic.Arn));
Console.WriteLine("Topic exists = {0}", exists);

// Modify topic attributes
topic.SetAttributes("DisplayName", "Test Topic");

// Subscribe an email endpoint to the topic
topic.Subscribe("test@example.com", "email");

// Wait until the subscription has been confirmed by the endpoint
// WaitForSubscriptionConfirmation();

// Publish a message to the topic
topic.Publish("Test mesage");

// Delete the topic
topic.Delete();

// Check that the topic is no longer in the list of all topics
exists = sns.GetTopics()
    .Any(t => t.Arn.Equals(topic.Arn));
Console.WriteLine("Topic exists = {0}", exists);

As you can see, it’s easy to get started with and use the new Amazon SNS Resource APIs to work with the service.

Querying the Public IP Address Ranges for AWS

by Steve Roberts | on | in .NET | Permalink | Comments |  Share

A post on the AWS Official Blog last November noted that the authoritative public IP address ranges used by AWS could now be obtained from a JSON-format file. The same information can now be accessed easily from AWS Tools for Windows PowerShell with a new cmdlet, Get-AWSPublicIpAddressRange, without the need to parse JSON. This cmdlet was added in version 2.3.15.0.

When run with no parameters, the cmdlet outputs all of the address ranges to the pipeline:

PS C:> Get-AWSPublicIpAddressRange

IpPrefix                    Region             Service
--------                    ------             -------
50.19.0.0/16                us-east-1          AMAZON
54.239.98.0/24              us-east-1          AMAZON
...
50.19.0.0/16                us-east-1          EC2
75.101.128.0/17             us-east-1          EC2
...
205.251.192.0/21            GLOBAL             ROUTE53
54.232.40.64/26             sa-east-1          ROUTE53_HEALTHCHECKS
...
54.239.192.0/19             GLOBAL             CLOUDFRONT
204.246.176.0/20            GLOBAL             CLOUDFRONT
...

If you’re comfortable using the pipeline to filter output, this may be all you need, but the cmdlet is also able to filter output using the -ServiceKey and -Region parameters. For example you can get the address ranges for EC2 across all regions like this (the parameter value is case insensitive):

PS C:> Get-AWSPublicIpAddressRange -ServiceKey ec2

Similarly, you can get the address ranges used by AWS in a given region:

PS C:> Get-AWSPublicIpAddressRange -Region us-west-2

Both of these parameters accept string arrays and can be supplied together. This example shows how to get the address ranges for Amazon EC2 and Amazon Route53 health checks in both US West regions:

PS C:> Get-AWSPublicIpAddressRange -ServiceKey ec2,route53_healthchecks -Region us-west-1,us-west-2

IpPrefix                    Region              Service
--------                    ------              -------
184.72.0.0/18               us-west-1           EC2
54.215.0.0/16               us-west-1           EC2
...
54.214.0.0/16               us-west-2           EC2
54.245.0.0/16               us-west-2           EC2
...
54.241.32.64/26             us-west-1           ROUTE53_HEALTHCHECKS
54.245.168.0/26             us-west-2           ROUTE53_HEALTHCHECKS
54.244.52.192/26            us-west-2           ROUTE53_HEALTHCHECKS
54.183.255.128/26           us-west-1           ROUTE53_HEALTHCHECKS

As noted in the original post, this information can change several times per week. You can find the publication date and time of the current information using the -OutputPublicationDate switch. The returned value here is a DateTime object:

PS C:> Get-AWSPublicIpAddressRange -OutputPublicationDate

Monday, December 15, 2014 4:41:01 PM

The set of service keys may change over time (see AWS IP Address Ranges for current documentation on this information). The current set of keys in use in the file can be obtained using the -OutputServiceKeys switch:

PS C:> Get-AWSPublicIpAddressRange -OutputServiceKeys

AMAZON
EC2
ROUTE53
ROUTE53_HEALTHCHECKS
CLOUDFRONT

If you’ve read this far and are thinking that this would also be useful for your C#/.NET applications, then you’ll be glad to know it’s also exposed in the AWS SDK for .NET. See the AWSPublicIpAddressRanges class in the Amazon.Util namespace for more details.

We hope you find this new capability useful in your scripts. If you have ideas for other cmdlets that you would find useful, be sure to leave a comment!

Caching Amazon Cognito Identity IDs

by Norm Johanson | on | in .NET | Permalink | Comments |  Share

Amazon Cognito is a service that you can use to get AWS credentials to your mobile and desktop applications without embedding them in your code. A few months ago, we added a credentials provider for Cognito. In version 2.3.14 of the AWS SDK for .NET, we updated the credentials provider to support caching the identity ID that Cognito creates.

Caching IDs is really useful for mobile and desktop applications where you don’t want to require users to authenticate but need to remember the user for each run of the application. For example, if you have a game whose scores you want to store in Amazon S3, you can use the identity ID as the object key in S3. Then, in future runs of the game, you can use the identity ID to get the scores back from S3. To get the current identity ID, call the GetIdentityId method on the credentials provider. You can also use the identity ID in the AWS Identity and Access Management (IAM) role that Cognito is using to restrict access to only the current user’s score. Below is a policy that shows how to use the Cognito identity ID. In the policy, the variable ${cognito-identity.amazonaws.com:sub} is used. When the policy is evaluated, ${cognito-identity.amazonaws.com:sub} is replaced with the current user’s identity ID.

{
    "Version" : "2012-10-17",
    "Statement" : [
        {
            "Sid" : "1",
            "Effect" : "Allow",
            "Action" : [
                "mobileanalytics:PutEvents",
                "cognito-sync:*"
            ],
            "Resource" : "*"
        },
        {
            "Sid" : "2",
            "Effect" : "Allow",
            "Action" : ["s3:PutObject", "s3:GetObject"]
            "Resource" : "arn:aws:s3:::my-game-scores-bucket/scores/${cognito-identity.amazonaws.com:sub}.json"
        }
    ]
}

In the Windows Phone and Windows Store version of the SDK, caching is controlled by the IdentityIdCacheMode property on Amazon.CognitoIdentity.CognitoAWSCredentials. By default, this property is set to LocalSettings, which means the identity ID will be cached local to just the device. Windows.Storage.ApplicationData.Current.LocalSettings is used to cache the identity ID. It can also be set to RoamingSettings, which means the identity ID will be stored in Windows.Storage.ApplicationData.Current.RoamingSettings, and the Windows Runtime will sync data stored in this collection to other devices where the user is logged in. To turn off caching, set IdentityIdCacheMode to None.

To enable caching for the .NET 3.5 and 4.5 versions of the SDK, you need to extend the Amazon.CognitoIdentity.CognitoAWSCredentials class and implement the GetCachedIdentityId, CacheIdentityId, and ClearIdentityCache methods.