Tag: .NET


Retry Throttling

by Sattwik Pati | on | in .NET | Permalink | Comments |  Share

In this blog post, we discuss the existing request retry feature, and the new retry throttling feature that we have rolled out in the AWS SDK for .NET V3 from version 3.3.4.0 of the AWSSDK.Core package.

In request retry, client side requests are retried, and often succeed, in cases involving transient network or service issues. The advantage to you as a client is that you don’t have to experience noise resulting from these exceptions, and are saved the trouble of writing code that would retry these requests. The downside to this retry feature is that situations such as network connectivity or unavailability, in which all retried requests fail, leads to tying up the client application thread and fail-slow behavior. The client eventually ends up getting a service unavailable exception that could have been relayed earlier, without the retry loop. This delay in surfacing the exception hurts the client’s recovery times and prolongs the client side impact. We want to walk a middle ground where we provide the retry request feature but with some limiting constraints.

Retry throttling, like its name suggests, throttles retry attempts when a large number of retry requests are failing. Each time a retry request is made, an internal retry capacity pool is drained. Retry requests are no longer made if the capacity pool is depleted. Retry requests are attempted only when the client starts getting successful responses, which refills the client’s capacity pool. Retry throttling takes care of “retry storm” situations by entering a fail-fast mode, in which the exceptions are surfaced and the needless retry loop is skipped. Also, because retry throttling kicks in only when a large number of requests and their retry attempts fail, transient retries are unaffected by this feature.

The AWS SDK for Java has already introduced this feature to great effect. Their blog post contains the metrics that compare situations when throttling is enabled versus when it is not.

Disabling Retry Throttling

Retry throttling is enabled by default and can be disabled easily by changing the ThrottleRetries property to false on the config object. We demonstrate this in the following by using an AmazonS3Config object.

AmazonS3Config config = new AmazonS3Config();
config.ThrottleRetries = false; // Default value is true

As you can see, it’s easy to opt out of this feature. Retry throttling can improve the ability of the SDK to adapt to sub-optimal situations. Feel free to leave questions or comments below!

General Availability for .NET Core Support in the AWS SDK for .NET

by Norm Johanson | on | in .NET | Permalink | Comments |  Share

Today, we announce the general availability (GA) of our .NET Core support in the AWS SDK for .NET. Previously, we’ve supported .NET Core in our 3.2.x beta NuGet packages while maintaining our 3.1.x NuGet packages on our stable master branch with the frequent AWS service updates.

With the move to GA status for .NET Core, we’ve merged .NET Core support into the stable master branch and, going forward, will release version 3.3.x NuGet packages for the AWS SDK for .NET. We’ll add AWS service updates to our .NET Core version at the same time we add them to the rest of the .NET platforms we support, like .NET Framework 3.5 and 4.5. The SDK’s change of status also means our AWS Tools for PowerShell Core module (AWSPowerShell.NetCore) is at GA status, and its version bumps to 3.3.x to match the underlying SDK version.

This release is one more step in our continuing support for .NET Core on AWS. Other exciting .NET Core releases we’ve had this year include:

For help setting up and configuring the SDK for use with .NET Core, see our previous post on some of the extensions we added to take advantage of the new .NET Core frameworks.

We welcome your feedback. Check out our GitHub repository and let us know what you think of our .NET and .NET Core support.

Configuring AWS SDK with .NET Core

by Norm Johanson | on | in .NET | Permalink | Comments |  Share

One of the biggest changes in .NET Core is the removal of ConfigurationManager and the standard app.config and web.config files that were used ubiquitously with .NET Framework and ASP.NET applications. The AWS SDK for .NET used this configuration system to set things like AWS credentials and region so that you wouldn’t have to do this in code.

A new configuration system in .NET Core allows any type of input source from any location. Also, the configuration object isn’t a global singleton like the old ConfigurationManager was, so the AWS SDK for .NET doesn’t have access to read settings from it.

To make it easy to use the AWS SDK for .NET with .NET Core, we have released a new NuGet package called AWSSDK.Extensions.NETCore.Setup. Like many .NET Core libraries, it adds extension methods to the IConfiguration interface to make getting the AWS configuration seamless.

Using AWSSDK.Extensions.NETCore.Setup

If we create an ASP.NET Core MVC application in Visual Studio, the constructor for Startup.cs handles configuration by reading in various input sources, using the ConfigurationBuilder and setting the Configuration property to the built IConfiguration object.

public Startup(IHostingEnvironment env)
{
    var builder = new ConfigurationBuilder()
        .SetBasePath(env.ContentRootPath)
        .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
        .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
        .AddEnvironmentVariables();
    Configuration = builder.Build();
}

To use the Configuration object to get the AWS options, we first add the AWSSDK.Extensions.NETCore.Setup NuGet package. Then, we add our options to the configuration file. Notice one of the files added to the ConfigurationBuilder is called $"appsettings.{env.EnvironmentName}.json". If you look at the Debug tab in the project’s properties, you can see this file is set to Development. This works great for local testing because we can put our configuration in the appsettings.Development.json file, which is loaded only during local testing in Visual Studio. When we deploy to an Amazon EC2 instance the EnvironmentName will default to Production and this file will be ignored causing the AWS SDK for .NET to fall back to the IAM credentials and region configured for the EC2 instance.

Let’s add an appsettings.Development.json file to our project and supply our AWS settings.

{
  "AWS": {
    "Profile": "local-test-profile",
    "Region": "us-west-2"
  }
}

To get the AWS options set in the file, we call the extension method that is added to IConfiguration, GetAWSOptions. To construct a service client from these options, we call CreateServiceClient. The following example code shows how to create an S3 service client.

var options = Configuration.GetAWSOptions();
IAmazonS3 client = options.CreateServiceClient();

ASP.NET Core Dependency Injection

The AWSSDK.Extensions.NETCore.Setup NuGet package also integrates with a new dependency injection system in ASP.NET Core. The ConfigureServices method in Startup is where the MVC services are added. If the application is using Entity Framework, this is also where that is initialized.

public void ConfigureServices(IServiceCollection services)
{
    // Add framework services.
    services.AddMvc();
}

The AWSSDK.Extensions.NETCore.Setup NuGet package adds new extension methods to IServiceCollection that you can use to add AWS services to the dependency injection. The following code shows how we add the AWS options read from IConfiguration and add S3 and Amazon DynamoDB to our list of services.

public void ConfigureServices(IServiceCollection services)
{
    // Add framework services.
    services.AddMvc();
    services.AddDefaultAWSOptions(Configuration.GetAWSOptions());
    services.AddAWSService<IAmazonS3>();
    services.AddAWSService<IAmazonDynamoDB>();
}

Now, if our MVC controllers use either IAmazonS3 or IAmazonDynamoDB as parameters in their constructors, the dependency injection system passes those services in.

public class HomeController : Controller
{
    IAmazonS3 S3Client { get; set; }

    public HomeController(IAmazonS3 s3Client)
    {
        this.S3Client = s3Client;
    }

    ...

}

Summary

We hope this new AWSSDK.Extensions.NETCore.Setup NuGet package helps you get started with ASP.NET Core and AWS. Feel free to give us your feedback at our GitHub repository for the AWS SDK for .NET

Custom Elastic Beanstalk Application Deployments

by Norm Johanson | on | in .NET | Permalink | Comments |  Share

In the previous post, you learned how to use the new deployment manifest for the the Windows container in AWS Elastic Beanstalk to deploy a collection of ASP.NET Core and traditional ASP.NET applications. The deployment manifest supports a third deployment type, custom application deployment.

Custom application deployment is a powerful feature for advanced users who want to leverage the power of Elastic Beanstalk to create and manage their AWS resources and also have complete control over how their application is deployed. For a custom application deployment, you declare the PowerShell scripts for the three actions that Elastic Beanstalk performs: install, restart, and uninstall. Install is used when a deployment is initiated, restart is used when the RestartAppServer API is called (which can be done from either the toolkit or the web console), and uninstall is invoked on the previous deployment whenever a new deployment occurs.

For example, you might have an ASP.NET application that you want to deploy, and your documentation team has written a static website that they want to include with the deployment. You can do this by writing your deployment manifest as follows.

{
  "manifestVersion": 1,
  "deployments": {
 
    "msDeploy": [
      {
        "name": "app",
        "parameters": {
          "appBundle": "CoolApp.zip",
          "iisPath": "/"
        }
      }
    ],
    "custom": [
      {
        "name": "PowerShellDocs",
        "scripts": {
          "install": {
            "file": "install.ps1"
          },
          "restart": {
            "file": "restart.ps1"
          },
          "uninstall": {
            "file": "uninstall.ps1"
          }
        }
      }
    ]
  }
}

The scripts listed for each action are in the application bundle relative to the deployment manifest file. For this example, the application bundle will also contain a documentation.zip file that contains the static website from your documentation team.

The install.ps1 script extracts the .zip file and sets up the IIS path.

Add-Type -assembly "system.io.compression.filesystem"
[io.compression.zipfile]::ExtractToDirectory('./documentation.zip', 'c:inetpubwwwrootdocumentation')

C:WindowsSysNativeWindowsPowerShellv1.0powershell.exe -Command {New-WebApplication -Name documentation -PhysicalPath  c:inetpubwwwrootdocumentation -Force}

Because your application is running in IIS, the restart action will invoke an IIS reset.

iisreset /timeout:1

For uninstall scripts, it’s important to clean up all settings and files that were performed during the install stage so that when the new version is being installed, you can avoid any collision with the previous deployments. For this example, you need to remove the IIS application for the static website and remove the files.

C:WindowsSysNativeWindowsPowerShellv1.0powershell.exe -Command {Remove-WebApplication -Name documentation}

Remove-Item -Recurse -Force 'c:inetpubwwwrootdocumentation'

Using these scripts files and the documentation.zip file that are included in the application bundle, the deployment will deploy your ASP.NET application and then deploy the documentation site.

This example showed a simple deployment of a simple static website. By using the power of custom application deployment, you can deploy any type of application and let Elastic Beanstalk manage the AWS resources for the application.

Multiple Application Support for .NET and Elastic Beanstalk

by Norm Johanson | on | in .NET | Permalink | Comments |  Share

In the previous post we talked about the new deployment manifest you can use to deploy applications to AWS Elastic Beanstalk. You can now use the deployment manifest to deploy multiple applications to the same Elastic Beanstalk environment.

The deployment manifest supports ASP.NET Core web applications and msdeploy archives for traditional ASP.NET applications. Imagine a scenario in which you’ve written a new amazing application using ASP.NET Core for the front end and a Web API project for an extensions API. You also have an admin app that you wrote using traditional ASP.NET.

The toolkit’s deployment wizard focuses on deploying a single project. To take advantage of the multiple application deployment, you have to construct the application bundle by hand. To start, you need to write the manifest. For this example, write the manifest at the root of your solution.

The deployment section in the manifest has two children: an array of ASP.NET Core web applications to deploy, and an array of msdeploy archives to deploy. For each application, you set the IIS path and the location of the application’s bits relative to the manifest.

{
  "manifestVersion": 1,
  "deployments": {
 
    "aspNetCoreWeb": [
      {
        "name": "frontend",
        "parameters": {
          "appBundle": "./frontend",
          "iisPath": "/frontend"
        }
      },
      {
        "name": "ext-api",
        "parameters": {
          "appBundle": "./ext-api",
          "iisPath": "/ext-api"
        }
      }
    ],
    "msDeploy": [
      {
        "name": "admin",
        "parameters": {
          "appBundle": "AmazingAdmin.zip",
          "iisPath": "/admin"
        }
      }
    ]
  }
}

With the manifest written, you’ll use Windows PowerShell to create the application bundle and update an existing Elastic Beanstalk environment to run it. To get the full version of the Windows PowerShell script used in this example, right-click here. The script is written with the assumption that it will be run from the folder that contains your Visual Studio solution.

The first thing you need to do in the script is set up a workspace folder to create the application bundle.

$publishFolder = "c:temppublish"

$publishWorkspace = [System.IO.Path]::Combine($publishFolder, "workspace")
$appBundle = [System.IO.Path]::Combine($publishFolder, "app-bundle.zip")

If (Test-Path $publishWorkspace){
	Remove-Item $publishWorkspace -Confirm:$false -Force
}
If (Test-Path $appBundle){
	Remove-Item $appBundle -Confirm:$false -Force
}

Once the workspace is set up, you can get the front end ready. To do that use the dotnet CLI to publish the application.

Write-Host 'Publish the ASP.NET Core frontend'  
$publishFrontendFolder = [System.IO.Path]::Combine($publishWorkspace, "frontend")
dotnet publish .srcAmazingFrontendproject.json -o $publishFrontendFolder -c Release -f netcoreapp1.0

Notice that the subfolder "frontend" was used for the output folder matching the folder set in the manifest. Now let’s do the same for the Web API project.

Write-Host 'Publish the ASP.NET Core extensiblity API' 
$publishExtAPIFolder = [System.IO.Path]::Combine($publishWorkspace, "ext-api") 

dotnet publish .srcAmazingExtensibleAPIproject.json -o $publishExtAPIFolder -c Release -f netcoreapp1.0 

The admin site is a traditional ASP.NET application, so you can’t use the dotnet CLI. For this project, use msbuild, passing in the build target package to create the msdeploy archive. By default, the package target creates the msdeploy archive under the objReleasePackage folder, so you need to copy the archive to the publish workspace.

Write-Host 'Create msdeploy archive for admin site'

msbuild .srcAmazingAdminAmazingAdmin.csproj /t:package /p:Configuration=Release

Copy-Item .srcAmazingAdminobjReleasePackageAmazingAdmin.zip $publishWorkspace

To tell the Elastic Beanstalk environment what to do with all these applications, you copy the manifest from your solution to the publish workspace and then zip up the folder.

Write-Host 'Copy deployment manifest'
Copy-Item .aws-windows-deployment-manifest.json $publishWorkspace

Write-Host 'Zipping up publish workspace to create app bundle'
Add-Type -assembly "system.io.compression.filesystem"
[io.compression.zipfile]::CreateFromDirectory( $publishWorkspace, $appBundle)

Now that you have the application bundle, you can go to the web console and upload your archive to an Elastic Beanstalk environment. Or you can keep using Windows PowerShell and use the AWS PowerShell cmdlets to update the Elastic Beanstalk environment to the application bundle. Be sure you’ve set the current profile and region to the profile and region that has your Elastic Beanstalk environment by using the Set-AWSCredentials and Set-DefaultAWSRegion cmdlets.

Write-Host 'Write application bundle to S3'
# Determine S3 bucket to store application bundle
$s3Bucket = New-EBStorageLocation
Write-S3Object -BucketName $s3Bucket -File $appBundle


$applicationName = "ASPNETCoreOnAWS"
$environmentName = "ASPNETCoreOnAWS-dev"
$versionLabel = [System.DateTime]::Now.Ticks.ToString()

Write-Host 'Update Beanstalk environment for new application bundle'

New-EBApplicationVersion -ApplicationName $applicationName -VersionLabel $versionLabel -SourceBundle_S3Bucket $s3Bucket -SourceBundle_S3Key app-bundle.zip

Update-EBEnvironment -ApplicationName $applicationName -EnvironmentName $environmentName -VersionLabel $versionLabel

Now check the update status in either the Elastic Beanstalk environment status page or in the web console. Once complete, you can navigate to each of the applications you deployed at the IIS path set in the deployment manifest.

We hope you’re excited about the features we added to AWS Elastic Beanstalk for Windows and AWS Toolkit for Visual Studio. Visit our forums and let us know what you think of the new tooling and what else you would like to see us add.

Customizing ASP.NET Core Deployments

by Norm Johanson | on | in .NET | Permalink | Comments |  Share

In our previous post we announced support for deploying ASP.NET Core applications with AWS Elastic Beanstalk and the AWS Toolkit for Visual Studio. Today, we’ll talk about how deployment works and how you can customize it.

After you go through the deployment wizard in the AWS Toolkit for Visual Studio, the toolkit bundles the application and sends it to Elastic Beanstalk. When the toolkit creates the bundle, the first step is to use the new dotnet CLI and the publish command to prepare the application for publishing. The settings in the wizard pass the framework and configuration to the publish command. So if you selected Release for configuration and netcoreapp1.0 for the framework, the toolkit will execute the following command.

dotnet publish --configuration Release --framework netcoreapp1.0

When the publish command finishes, the toolkit writes the new deployment manifest into the publishing folder. The deployment manifest is a JSON file named aws-windows-deployment-manifest.json, which is read by the new tooling added to the 1.2 version of the Elastic Beanstalk Windows container to figure out how to deploy the application. For example, for an ASP.NET Core application that you want to deploy at the root of IIS, the toolkit generates a manifest file that looks like this.

{
  "manifestVersion": 1,
  "deployments": {
 
    "aspNetCoreWeb": [
      {
        "name": "app",
        "parameters": {
          "appBundle": ".",
          "iisPath": "/",
          "iisWebSite": "Default Web Site"
        }
      }
    ]
  }
}

The appBundle property indicates where the application bits are in relation to the manifest file. This property can point to either a directory or a ZIP archive. The iisPath and iisWebSite properties indicate where in IIS to host the application.

Declaring the Manifest

The toolkit only writes the manifest file if it doesn’t already exist in the publishing folder. If the file does exist, the toolkit updates the appBundle, iisPath, and iisWebSite properties in the first application listed in the aspNetCoreWeb section of the manifest. This allows you to add the aws-windows-deployment-manifest.json to your project and customize the manifest. To do this for an ASP.NET Core Web application in Visual Studio, add a new JSON file to the root of the project and name it aws-windows-deployment-manifest.json.

The manifest must be named aws-windows-deployment-manifest.json and it must be at the root of the project. The Elastic Beanstalk container looks for the manifest at the root and, if it finds it, will invoke the new deployment tooling. If the file doesn’t exist, the Elastic Beanstalk container falls back to the older deployment tooling, which assumes the archive is a msdeploy archive.

To ensure the dotnet CLI publish command includes the manifest, you must update the project.json file to include the manifest file in the include section that’s under publishOptions.

Customizing the Manifest File for Application Deployment

Now that you’ve declared the manifest so that it’s included in the app bundle, you can go beyond what the wizard supports to customize your application’s deployment. AWS has defined a JSON schema for aws-windows-deployment-manifest.json. When you installed the AWS Toolkit for Visual Studio, the setup registers deployment manifest schema.

If you look at the Schema box, you’ll see the schema is set to aws-windows-deployment-manifest-schema.json. When the schema is selected, Visual Studio provides IntelliSense while you’re editing the manifest.

For non Visual Studio users the deployment manifest schema can be accessed online here.

One customization you can do is to configure the IIS application pool that runs the application. The following example shows how you can define an IIS application pool that recycles the process every hour and assigns it to the application.

Additionally, the manifest can declare Windows PowerShell scripts to run before and after the install, restart, and uninstall actions. For example, the following manifest runs the Windows PowerShell script PostInstallSetup.ps1 to do more setup work after the ASP.NET Core application is deployed to IIS. Remember, just like the aws-windows-deployment-manifest.json file, be sure to add your scripts to the include section under publishOptions in the project.json file. If you don’t, the scripts won’t be included in the dotnet CLI publish command.

{
  "manifestVersion": 1,
  "deployments": {
    "aspNetCoreWeb": [
      {
        "name": "app",
        "scripts": {
          "postInstall": {
            "file": "SetupScripts/PostInstallSetup.ps1"
          }  
        }  
      }   
    ]
  }
}

What about ebextensions?

The Elastic Beanstalk ebextensions configuration files are still supported like all the other Elastic Beanstalk containers. To include them in an ASP.NET Core application, add the .ebextensions directory to the include section under publishOptions in the project.json file. For more information about ebextensions, check out the Elastic Beanstalk Developer Guide.

We hope this post gives you a better understanding of how deploying ASP.NET Core applications to Elastic Beanstalk works. In our next post, we’ll show you how to use the new deployment manifest file to deploy multiple applications to the same Elastic Beanstalk environment.

ASP.NET Core Support for Elastic Beanstalk

by Norm Johanson | on | in .NET | Permalink | Comments |  Share

Today, we release support for deploying ASP.NET Core applications to AWS by using AWS Elastic Beanstalk (Elastic Beanstalk) and the AWS Toolkit for Visual Studio. This Elastic Beanstalk release expands the support the service already offers for deploying applications, including traditional ASP.NET applications, in a variety of languages to AWS.

Let’s walk through the deployment experience. The AWS Toolkit for Visual Studio is the easiest way to get started deploying ASP.NET Core applications to Elastic Beanstalk. If you have used the toolkit before to deploy traditional ASP.NET applications, you’ll find the experience for ASP.NET Core to be very similar.

If you’re new to the toolkit, after you install it, the first thing you need to do is register your AWS credentials with it.

In Visual Studio, from the Views menu, choose AWS Explorer.

Click the Add button to add your AWS credentials.

To deploy an ASP.NET Core web application, right-click the project in the Solution Explorer, and choose Publish to AWS.

On the Publish to AWS Elastic Beanstalk page, you’ll choose to create an Elastic Beanstalk application. This is a logical representation of your application that will contain a collection of application versions and environments. The environments contain the actual AWS resources that will run an application version. Every time you deploy an application that creates an application version and points the environment to that version.

Next, set names for the application and its first environment. Each environment will have a unique CNAME associated with it that you can use to access your application when the deployment is complete.

On the AWS Options page, you configure the type of AWS resources to use for your application. For this example, we can leave the default values except for the Key pair section. Key pairs allow you to retrieve the Windows Administrator password so you can log into the machine. If you haven’t already created a key pair, you can select Create new key pair.

On the Permissions page, you assign AWS credentials to the Amazon EC2 instances running your application. This is important if your application is using the AWS SDK for .NET to access other AWS services. If you’re not using any other services from your application, you can leave this page at its default.

The Application Options page is the one that differs from deploying traditional ASP.NET Core applications. Here, you can specify the build configuration and framework used to package the application and also what IIS resource path to configure for the application.

Once that’s finished, click Next to review the settings. Then, click Deploy to begin the deployment process. After the application is packaged and uploaded to AWS, you can check the status of the Elastic Beanstalk environment by opening the environment status view from AWS Explorer.

Events are displayed in Status as the environment comes online. When everything is complete, the environment status changes to Environment is healthy. You can click the URL to view the site. Also from this page, you can pull the logs from the environment or remote desktop into the EC2 instances that are part of your Elastic Beanstalk environment.

The first deployment will take a little more time as it creates new AWS resources. As you iterate over your application, you can quickly redeploy by going back through the wizard or selecting Republish when you right-click the project.

Republish will package your application using the settings from the last time through the deployment wizard, and then upload the application bundle to the existing Elastic Beanstalk environment.

We hope you’re excited to try out our expanded ASP.NET Core support in AWS. In our next post, we’ll dive deeper into how the deployment works and how you can customize it.

Introducing AWS Tools for PowerShell Core Edition

by Steve Roberts | on | in .NET | Permalink | Comments |  Share

Today Microsoft announced PowerShell support on Windows, Linux and OS X platforms (read the blog announcement here). We’re pleased to also announce support for PowerShell on these new platforms with a new module, the AWS Tools for PowerShell Core Edition or "AWSPowerShell.NetCore" to give it its module name. Just like PowerShell itself, this new module can be used on Windows, Linux, and OS X platforms.

The AWSPowerShell.NetCore module is built on top of the .NET Core version of the AWS SDK for .NET, which is in beta while we finish up testing and port a few more features from the .NET Framework version of the AWS SDK for .NET. Note that updates to the new module for new service features may lag a little behind the sister AWS Tools for Windows PowerShell ("AWSPowerShell") module while the .NET Core version of the AWS SDK for .NET is in beta.

We hope to publish the new module to the PowerShell Gallery in a few days, after we finish final testing against the PowerShell libraries that Microsoft released today. We will update this blog post, together with a link to the new module on the gallery, once we have done so.

Update August 23rd 2016

The first beta release of the new module is now available on the PowerShell gallery. The version number is currently 3.2.7.0 to match the underlying AWS SDK for .NET product version. The services and service APIs supported in this release correspond to the 3.1.94.0 release of the AWSPowerShell module.

Some users are reporting issues with the Install-Module cmdlet built into PowerShell Core with errors related to semantic versioning (see GitHub Issue 202). Using the NuGet provider appears to resolve the issue currently. To install using this provider run this command, setting an appropriate destination folder (on Linux for example try -Destination ~/.local/share/powershell/Modules):

Install-Package -Name AWSPowerShell.NetCore -Source https://www.powershellgallery.com/api/v2/ -ProviderName NuGet -ExcludeVersion -Destination destfolder 

What Happens to AWS Tools for Windows PowerShell?

Nothing! We will continue to update this module just as we have being doing as new service features and new services are launched. The version number of this module will remain at 3.1.* for the time being (AWSPowerShell.NetCore is v3.2.* to match the .NET SDK Core beta, which also has a version of 3.2.*).

Once the .NET Core version of the AWS SDK for .NET exits beta status, so will the AWSPowerShell.NetCore module. At that point, both the AWSPowerShell and AWSPowerShell.NetCore modules will version bump to 3.3.*. Again, this corresponds to the underlying SDK product version. From then on, both modules will update in lockstep with the underlying SDK just as we have being doing with the AWSPowerShell module, with updates being pushed to the gallery.

One small difference is that we currently plan to distribute the AWSPowerShell.NetCore module only through the gallery. The AWSPowerShell module will continue to be distributed to the gallery and through a downloadable Windows Installer that also contains the SDK and AWS Toolkit for Visual Studio. If you would like the new module to also be included in this installer, please let us know in the comments.

Module Compatibility

There is a high degree of compatibility between the two modules.

All cmdlets that map to service APIs are present and function in the same way between the two modules, with the exception that cmdlets for the deprecated Amazon Elastic Compute Cloud (Amazon EC2) import APIs, Import-EC2Instance and Import-EC2Volume, are not present in AWSPowerShell.NetCore. If you currently use these cmdlets in the AWSPowerShell module, we encourage you to investigate their replacements, Import-EC2Image and Import-EC2Snapshot, because the new Amazon EC2 import APIs are faster and more convenient (see blog post for more detail).

A small number of cmdlets are currently still being ported to the new module and have been removed from the first release to the gallery. We hope to add these back as soon as we can, but for now, the following are the cmdlets that are currently unavailable in the AWSPowerShell.NetCore module.

Proxy cmdlets:

  • Set-AWSProxy
  • Clear-AWSProxy

Logging cmdlets:

  • Add-AWSLoggingListener
  • Remove-AWSLoggingListener
  • Set-AWSResponseLogging
  • Enable-AWSMetricsLogging
  • Disable-AWSMetricsLogging

SAML federated credentials cmdlets:

  • Set-AWSSamlEndpoint
  • Set-AWSSamlRoleProfile

Note that credential profiles holding regular AWS Access and Secret keys can be used, and you can also obtain credentials from Instance Profiles when running the new module on Amazon EC2 instances, even on Linux instances.

Credential Handling

All cmdlets accept AWS Access and Secret keys or the names of credential profiles when they run, the same as you use them today with the current AWSPowerShell module. When running on Windows, both modules have access to the AWS SDK for .NET credential store file (stored per-user AppDataLocalAWSToolkitRegisteredAccounts.json file). This file stores your keys in encrypted format and cannot be used on a different machine. This file is the first to be inspected for the AWSPowerShell module when looking for a credential profile, and is also where it stores new credential profiles. (The AWSPowerShell module does not currently support writing credentials to other files or locations.)

Both modules can also read profiles from the ini-format shared credentials file that is used by other AWS SDKs and the AWS CLI. On Windows, the default location for this file is ”’C:Users”userid”.awscredentials”’. On non-Windows platforms, it is at ”’~/.aws/credentials”’. The -ProfilesLocation parameter can be used to point to a non-default file name or file location.

The SDK credential store holds your credentials in encrypted form by using Windows crypto APIs. These APIs are not available on other platforms, so the AWSPowerShell.NetCore module uses the ini-format shared credentials file exclusively and also supports writing new credential profiles to the shared credential file. This support will be extended to the AWSPowerShell module in a future release.

These example commands that use the Set-AWSCredentials cmdlet show the options for handling credential profiles on Windows with either the AWSPowerShell or AWSPowerShell.NetCore modules:

# Writes a new (or updates existing) profile with name "myProfileName"
# in the encrypted SDK store file
Set-AWSCredentials -AccessKey akey -SecretKey skey -StoreAs myProfileName

# Checks the encrypted SDK credential store for the profile and then
# falls back to the shared credentials file in the default location
Set-AWSCredentials -ProfileName myProfileName

# Bypasses the encrypted SDK credential store and attempts to load the
# profile from the ini-format credentials file "mycredentials" in the
# folder C:MyCustomPath
Set-AWSCredentials -ProfileName myProfileName -ProfilesLocation C:MyCustomPathmycredentials

These examples show the behavior of the AWSPowerShell.NetCore module on Linux or OS X:

# Writes a new (or updates existing) profile with name "myProfileName"
# in the default shared credentials file ~/.aws/credentials
Set-AWSCredentials -AccessKey akey -SecretKey skey -StoreAs myProfileName

# Writes a new (or updates existing) profile with name "myProfileName"
# into an ini-format credentials file "~/mycustompath/mycredentials"
Set-AWSCredentials -AccessKey akey -SecretKey skey -StoreAs myProfileName -ProfilesLocation ~/mycustompath/mycredentials

# Reads the default shared credential file looking for the profile "myProfileName"
Set-AWSCredentials -ProfileName myProfileName

# Reads the specified credential file looking for the profile "myProfileName"
Set-AWSCredentials -ProfileName myProfileName -ProfilesLocation ~/mycustompath/mycredentials

Watch this space!

We’ll continue to update this post as new information becomes available and we finalize the publishing of the new module to the PowerShell Gallery. We hope you’re as excited about the ability to use PowerShell on platforms other than Windows as we are!

Argument Completion Support in AWS Tools for Windows PowerShell

by Steve Roberts | on | in .NET | Permalink | Comments |  Share

Version 3.1.93.0 of the AWS Tools for Windows PowerShell now includes support for tab completion of parameters that map to enumeration types in service APIs. Let’s look at how these types are implemented in the underlying AWS SDK for .NET and then see how this new support helps you at the Windows PowerShell command line or in script editors (like the PowerShell ISE) that support parameter IntelliSense.

Enumerations in the AWS SDK for .NET

You might expect the SDK to implement enumeration types used in service APIs as enum types but this isn’t the case. The SDK contains a ConstantClass base class from which it derives classes for service-specific enumeration types. These derived classes implement the permitted values for a service enumeration as a set of read-only static strings. For example, here’s a snippet of the InstanceType enumeration for Amazon Elastic Compute Cloud (Amazon EC2) instances (comments removed for brevity):

public class InstanceType : ConstantClass
{
    public static readonly InstanceType C1Medium = new InstanceType("c1.medium");
    public static readonly InstanceType C1Xlarge = new InstanceType("c1.xlarge");
    public static readonly InstanceType C32xlarge = new InstanceType("c3.2xlarge");
	...

    public InstanceType(string value)
           : base(value)
    {
    }
	...
}

In a typical SDK application, you would use the defined types (from Amazon EC2’s RunInstances API), like this:

var request = new RunInstancesRequest
{
    InstanceType = InstanceType.C1XLarge,
	...
};
var response = EC2Client.RunInstances(request);
...

In this way, the SDK’s enumerations are not very different from regular enum types but offer one very powerful capability over regular enums: When services update their enumeration values (for example when EC2 adds a new instance type) you do not need to update the version of the SDK your application is built against to use the new value! The new value won’t appear as a member of the enumeration class until you update your SDK but you can simply write code to use the value with whatever version you have. It just works:

var request = new RunInstancesRequest
{
    InstanceType = "new-instance-type-code"
	...
};
var response = EC2Client.RunInstances(request);
...

This ability to adopt new values also applies to the response data from the service. The SDK will simply unmarshal the response data and accept the new value. This is unlike what would happen with the use of real enum types that would throw an error. You are therefore insulated on both sides from services adding new enumeration values when you want or need to remain at a particular SDK version.

Using Service Enumerations from PowerShell

Let’s say we are working at a console and want to use New-EC2Instance (which maps to the RunInstances API):

PS C:> New-EC2Instance -InstanceType ???

As we noted, the underlying SDK does not use regular .NET enum types for the allowed values so there’s no data for the shell to run against in order to offer a suggestion list. Obviously this is a problem when you don’t know the permitted values but it’s also an issue when you know the value but not the casing. Windows PowerShell may be case-insensitive but some services require the casing shown in their enumerations for their API call to succeed.

Why Not Use ValidateSet?

One way to tackle this problem would be to use PowerShell’s ValidateSet attribution on parameters that map to service enumerations, but this has a shortcoming: validation! Using the example of the -InstanceType parameter again, should EC2 add a new type you would need to update your AWSPowerShell module in order to use the new value. Otherwise, the shell would reject your use of values not included in the ValidateSet attribution at the time we shipped the module. The ability to make use of new enumeration values without being forced to recompile the application with an updated SDK is very useful. It’s certainly a capability we wanted to extend to our Windows PowerShell module users.

What we want is behavior similar to these screenshots of Invoke-WebRequest but without locking users into requiring updates for new values. In the ISE, we get a pop-up menu of completions:

At the console when we press Ctrl+Space we get a list of completions to select from:

We can also use the Tab key to iterate through the completions one by one or we can enter a partial match and the resulting completions are filtered accordingly.

The Solution: Argument Completers

Support for custom argument completers was added in Windows PowerShell version 3. Custom argument completers allow cmdlet authors and users to register a script block to be called by the shell when a parameter is specified. This script block is responsible for returning a set of valid completions given the data at hand. The set of completions (if any) is displayed to the user in the same way as if the data were specified using ValidateSet attribution or through a regular .NET enum type.

Third-party modules like TabExpansionPlusPlus (formerly TabExpansion++) also contributed to this mechanism to give authors and users a convenient way to register the completers. Beginning in Windows PowerShell version 5, a new native cmdlet can perform the registration.

In version 3.1.93.0 of the AWSPowerShell module we have added a nested script module that implements argument completers across the supported AWS services. The data used by these completers to offer suggestion lists for parameters comes from the SDK enumeration classes at the time we build the module. The SDK’s data is created based on the service models when we build the SDK. The permitted values are therefore correctly cased for those services that are case-sensitive; no more guessing how a value should be expressed when at the command line.

Here’s an example of the InstanceType completer (shortened) for EC2:

$EC2_Completers = {
    param($commandName, $parameterName, $wordToComplete, $commandAst, $fakeBoundParameter)
    
    # to allow for same-name parameters of different ConstantClass-derived
    # types, check on command name concatenated with parameter name.
    switch ($("$commandName/$parameterName"))
	{	...
        # Amazon.EC2.InstanceType
        {
            ($_ -eq "Get-EC2ReservedInstancesOffering/InstanceType") -Or
            ($_ -eq "New-EC2Instance/InstanceType") -Or
            ($_ -eq "Request-EC2SpotInstance/LaunchSpecification_InstanceType")
        }
        {
            $v = "c1.medium","c1.xlarge",...,"t2.nano","t2.small",...
            break
        }
	...
	}
	
    # the standard code pattern for completers is to pipe through sort-object
    # after filtering against $wordToComplete, but our members are already sorted.
    $v |
        Where-Object { $_ -like "$wordToComplete*" } |
        ForEach-Object { New-Object System.Management.Automation.CompletionResult $_, $_, 'ParameterValue', $_ }
}        

When the AWSPowerShell module is loaded, the nested module is automatically imported and executed, registering all of the completer script blocks it contains. Completion support works with Windows PowerShell versions 3 and later. For Windows PowerShell version 5 or later, the module uses the native Register-ArgumentCompleter cmdlet. For earlier versions it determines if this cmdlet is available in your installed modules (this will be the case if you have TabExpansionPlusPlus installed). If the cmdlet cannot be found the shell’s completer table is updated directly (you’ll find several blog posts on how this is done if you search for Windows PowerShell argument completion).

The net effect of this is that when you are constructing a command at the console or writing a script you get a suggestion list for the values accepted by these parameters. No more hunting through documentation to determine the allowed values and their casing! As we required, the ISE displays the list immediately after you enter a space after the parameter name and partial content will filter the list:

In a console the Tab key will cycle through the available options. Pressing Ctrl+Space displays a pop-up selection list that you can cursor around. In both cases you can filter the display by typing in the partial content:

A Note About Script Security

To accompany the new completion script module we made one other significant change in the 3.1.93.0 release: to add an Authenticode signature to all script and module artifacts (in effect, all .psd1, .psm1, .ps1 and .ps1xml files contained in the module). A side-benefit of this is that the module is now compatible with where the execution policy for Windows PowerShell scripts is set to "AllSigned". More information on execution policies can be found in this TechNet article.

Wrap

We hope you enjoy the new support for parameter value completion and the ability to now use the module in environments that require the execution policy to be ‘AllSigned’. Happy scripting!

AWS SDK for .NET Status Update for .NET Core Support

by Norm Johanson | on | in .NET | Permalink | Comments |  Share

The AWS SDK for .NET has included support for the .NET Core platform in our NuGet 3.2 beta packages since last September. With our recent push of version 3.2.6-beta to NuGet, we’ve switched from netstandard 1.5 to 1.3 to increase the SDK’s compatibility with other libraries. This version also includes many of the high-level abstractions and utility methods that were previously only available on the 3.5 and 4.5 .NET Framework versions of the SDK.

After some final testing and performance optimization, we will move our .NET Core support out of beta and into our master branch. (We’re currently investigating differences in the way HTTP connections are handled between .NET Framework and .NET Core.) At that point, we will bump the version number of all NuGet packages for the SDK to 3.3. Then .NET Core support for future service updates will be provided with the regular .NET Framework version of the SDK.

Now is a great time to give us feedback on our .NET Core support. Feel free to use the 3.2 beta versions of the SDK from NuGet and leave your feedback on our GitHub repository.