Tag: .NET


Stripe Windows Ephemeral Disks at Launch

by Steve Roberts | on | in .NET | Permalink | Comments |  Share

Today we have another guest post by AWS Solutions Architect David Veith.

Amazon EC2 currently offers more than 20 current-generation instance types for your Windows operating system workloads. The root volume for Windows instances will always be a volume provided by the Amazon EBS service. Additional EBS drives can easily be added as desired.

Depending on the EC2 instance type selected, there will also be from zero to 24 instance-store volumes automatically available to the instance. Instance-store volumes provide temporary block-level storage to the instance. The data in an instance store persists only during the lifetime of its associated instance. Because of the temporary nature of instance-store volumes, they are often referred to as ””’ephemeral”’—not lasting, enduring, or permanent”.

Many workloads can benefit from this type of temporary block-level storage, and it’s important to mention that ephemeral volumes also come with no extra cost.

This blog post describes how the ephemeral volumes of any Windows EC2 instance can be detected at launch, and then automatically striped into one large OS volume. This is a common use case for many AWS customers.

Detecting Ephemeral vs. EBS Volumes

In order to build a striped volume consisting only of instance-store volumes, we first need a mechanism to distinguish the volume types (EBS or ephemeral) associated with the instance. The EC2 metadata service provides a mechanism to determine this.

The following PowerShell statement retrieves all the block devices of a running Windows EC2 instance it is executed upon:

$alldrives = (Invoke-WebRequest -Uri http://169.254.169.254/latest/meta-data/block-device-mapping/).Content

Here’s an example of the data returned from the metadata service for an M3.Xlarge instance (launched from the AWS Management Console) with one root EBS volume and two instance-store volumes:


Using the same instance type (M3.Xlarge), but this time launching the instance from an AWS CloudFormation script (or AWS command-line tools), the same code produces this output:

Why the difference?

When an instance is launched from the AWS Management Console, the console performs some additional steps to have the instance metadata reflect only the ephemeral drives that are actually present. In order for our code to handle both cases, we can query WMI to see if the OS actually sees the volume.

$disknumber = (Get-WmiObject -Class Win32_DiskDrive | where-object {$_.SCSITargetId -eq $scsiid}).Index
if ($disknumber -ne $null)

How EC2 Windows Maps Drives

Hopefully, you noticed in the code directly above that we queried WMI with the SCSI ID of each volume. Where did we get the SCSI ID?

To answer that question, we need to explain how EC2 Windows instances map block devices’ SCSI IDs in the operating system. The following table shows this:

For example, we can see that ‘xvdcb’ will always map to SCSI ID ’79’. We could build a lookup table that contains all the potential mount points and their corresponding SCSI IDs, but a more elegant approach is to use a simple algorithm based on ASCII arithmetic.

We know that all device mappings for Windows instances begin with the ‘xvd’ prefix. If we remove this prefix, we can use the remaining portion (‘cb’ in this example) to derive the correct SCSI ID.

'c' = ASCII 99
'b' = ASCII 98

(('c' - 97) * 26) + ('b' - 97) = 79

In the final PowerShell script below, this psuedo-code is implemented as the GetSCSI function.

The Complete Powershell Script

#################################################
#  Detect the Ephemeral drives and stripe them  
#################################################

# Be sure to choose a drive letter that will not already be assigned
$DriveLetterToAssign = "K:"

#################################################
#  Given a device (e.g. xvda), strip off 
# "xvd" and convert the remainder to the 
# appropriate SCSI ID   
#################################################
function GetSCSI {
	Param([string]$device)
	
        # remove xvd prefix
	$deviceSuffix = $device.substring(3)      

        if ($deviceSuffix.length -eq 1) {
		$scsi = (([int][char] $deviceSuffix[0]) - 97)
	}
	else {
		$scsi = (([int][char] $deviceSuffix[0]) - 96) *  26 
                            +  (([int][char] $deviceSuffix[1]) - 97)
	}

	return $scsi
}

#################################################
#  Main   
#################################################

# From metadata read the device list and grab only 
# the ephemeral volumes

$alldrives = (Invoke-WebRequest -Uri http://169.254.169.254/latest/meta-data/block-device-mapping/).Content
$ephemerals = $alldrives.Split(10) | where-object {$_ -like 'ephemeral*'} 

# Build a list of scsi ID's for the ephemeral volumes

$scsiarray = @()
foreach ($ephemeral in $ephemerals) {
	$device = (Invoke-WebRequest -Uri http://169.254.169.254/latest/meta-data/block-device-mapping/$ephemeral).Content
	$scsi = GetSCSI $device
	$scsiarray = $scsiarray + $scsi
}

# Convert the scsi ID's to OS drive numbers and set them up with diskpart

$diskarray = @()
foreach ($scsiid in $scsiarray) {
	$disknumber = (Get-WmiObject -Class Win32_DiskDrive | where-object {$_.SCSITargetId -eq $scsiid}).Index
	if ($disknumber -ne $null)
	{
		$diskarray += $disknumber
		$dpcommand = "select disk $disknumber
	                        select partition 1
	                        delete partition
	                        convert dynamic
	                        exit"
	    $dpcommand | diskpart
	}
}

# Build the stripe from the diskarray

$diskseries = $diskarray -join ','

if ($diskarray.count -gt 0) 
{
	if ($diskarray.count -eq 1) {
		$type = "simple"
	}
	else {
		$type = "stripe"
	}
		
	$dpcommand = "create volume $type disk=$diskseries
		         format fs=ntfs quick
                         assign letter=$DriveLetterToAssign
	                 exit"
	$dpcommand | diskpart
}
</powershell>

Extra Credit

In Windows Server 2012 R2, Microsoft introduced new PowerShell storage-management cmdlets that replace the need to use the diskpart utility in many cases. If you know your servers will be running only Windows Server 2012 R2, or later, you might want to use these newer Microsoft cmdlets. You can find more information on these cmdlets at http://technet.microsoft.com/en-us/library/hh848705.aspx.

Utilizing Amazon ElastiCache Auto Discovery in .NET Through Enyim

by Mason Schneider | on | in .NET | Permalink | Comments |  Share

Today, we released a new library, Amazon ElastiCache Cluster Configuration, that allows .NET applications to easily leverage ElastiCache features. This post explains why a programmer would want to use this library and offers a quick and easy way to try it yourself.

What is Memcached?

Memcached provides a way to easily avoid some of the latency that comes with using a database, and it can also help applications at scale by removing some of the strain that can be placed on databases. This is accomplished by having Memcached servers be an intermediary, in-memory cache that can return results much faster than a normal database. In a typical program flow, this is accomplished by requesting a key from a group of cache servers and, if a value is retrieved, no database query is needed. Below is a simple diagram of what this function normally looks like: Memcached Diagram

Why ElastiCache?

ElastiCache provides a way to dynamically add and remove Memcached servers inside of a cache cluster. All servers are completely managed, which means that when servers are added they are automatically configured for Memcached. They are also added to the cluster and, when they are deleted, the cluster is updated. This means you spend less time configuring Memcached servers and more time working on things that matter. Being able to add and remove these nodes dynamically also means your application can easily scale whenever necessary through the AWS Management Console or through one of the many AWS APIs.

Using ElastiCache in .NET

Many .NET developers leverage ElastiCache through the Enyim framework. Enyim provides a client that manages server connections as well as what server your cache data should be stored on. To be aware of the Memcached servers, the Enyim client is configured on instantiation with IPs and ports for all of the servers. When the server information changes, the client must be disposed and re-instantiated with the new server information. The re-instantiation of the client tends to be tedious and can also cause issues if you update your configuration incorrectly when nodes change. One feature of ElastiCache that helps avoid this issue is Auto Discovery. This feature allows clients to find out the cluster configuration through an endpoint URL. The endpoint URL is sort of like an alias that just points to one of the servers in the cluster. Each server holds information about the configuration such as how many times the configuration has changed and the hostname, ip, and port of each server in the cluster. For more information on how Auto Discovery works, visit this page.

ElastiCache Cluster Configuration

Although Auto Discovery is useful, it is not accessible through Enyim’s client because it is not something that is found in standard Memcached clusters. Currently, Amazon has released clients for Java and PHP that extend Memcached clients in order to take advantage of Auto Discovery. With today’s release of the ElastiCache Cluster Configuration library, any .NET application using framework 3.5 or higher can now take full advantage of this great feature. All that is required is to add the clusterclient section to your App.config or instantiate the configuration object through parameters. After that, pass it as the configuration for the Enyim MemcachedClient, and you now have a MemcachedClient functioning through ElastiCache Auto Discovery.

Here is a sample App.config that shows how to specify the Amazon ElastiCache cluster.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <configSections>
      <section name="clusterclient" type="Amazon.ElastiCacheCluster.ClusterConfigSettings, Amazon.ElastiCacheCluster" />
    </configSections>

    <clusterclient>
      <endpoint hostname="my-configuruation-url.cache.amazonaws.com" port="11211" />
      <poller intervalDelay="60000" />
    </clusterclient>

</configuration>

 

Try it now!

For this walkthrough, we assume you have an AWS account with the ability to control Amazon EC2 and ElastiCache services, internet access, and the AWS Tools for Windows PowerShell installed and configured with an API key and secret key.

Create AWS Services using PowerShell

Start a PowerShell instance as Administrator. Below, you can edit these variable values to change property names, and then copy and paste the rest.

$ec2SecurityGroupName = "myElastiCacheGroup"
$cacheGroupName = "myElastiCacheSecGroup"
$keyPairName = "myConfigKeyPair"
$cacheClusterName = "demoCluster"

To use ElastiCache, you must create an ElastiCache cluster and an EC2 instance to access it. First, we create a key pair and an EC2 security group, and then we create an EC2 instance based on those values.

$myConfigKeyPair = New-EC2KeyPair -KeyName $keyPairName
$myConfigKeyPair.KeyMaterial | Out-File -Encoding ascii C:$keyPairName.pem

New-EC2SecurityGroup -GroupName $ec2SecurityGroupName -GroupDescription "ElastiCache Config Demo"

$cidrBlocks = @("0.0.0.0/0")
$ipPermissions = New-Object Amazon.EC2.Model.IpPermission -Property @{IpProtocol = "tcp"; FromPort = 11211; ToPort = 11211; IpRanges = $cidrBlocks}
Grant-EC2SecurityGroupIngress -GroupName $ec2SecurityGroupName -IpPermissions $ipPermissions
$ipPermissions = New-Object Amazon.EC2.Model.IpPermission -Property @{IpProtocol = "tcp"; FromPort = 3389; ToPort = 3389; IpRanges = $cidrBlocks}
Grant-EC2SecurityGroupIngress -GroupName $ec2SecurityGroupName -IpPermissions $ipPermissions

$image = Get-EC2ImageByName -Names WINDOWS_2012R2_BASE
if($image -is [system.array]) {$image = $image[0]}

$reservation = New-EC2Instance -ImageId $image.ImageId -KeyName $keyPairName -SecurityGroups $ec2SecurityGroupName -InstanceType t1.micro

After that is complete, we create a new ElastiCache cluster with three nodes and add the EC2 security group to its policy.

New-ECCacheSecurityGroup -CacheSecurityGroupName $cacheGroupName -Description "Demo for ElastiCache Config"
$secGroup = Get-EC2SecurityGroup -GroupNames $ec2SecurityGroupName
Approve-ECCacheSecurityGroupIngress -CacheSecurityGroupName $cacheGroupName -EC2SecurityGroupName $ec2SecurityGroupName -EC2SecurityGroupOwnerId $secGroup.OwnerId

New-ECCacheCluster -CacheNodeType cache.t1.micro -CacheClusterId $cacheClusterName -CacheSecurityGroupNames $cacheGroupName -Engine memcached -EngineVersion 1.4.14 -NumCacheNodes 3 -Port 11211

Create the Application

To demonstrate how to use ElastiCache Cluster Configuration, we’ll make a quick console application. From the start page of Visual Studio 2010 or higher, click "New Project…", and in the new project dialog, create a new Visual C# Console Application named "Cluster Config Demo". After the project is created, in Solution Explorer, right-click on the "References" section, and in the drop-down menu, click "Manage NuGet Packages…". After that, we’re going to search for "ElastiCacheClusterConfig" and then install it into our current project. When you install this package, Enyim is also installed on the project. Now that we have the project configured, let’s write the code. First, add the packages to the code by pasting this code into the top of the file "Program.cs".

using Enyim.Caching;
using Amazon.ElastiCacheCluster;

Next, copy the code below and paste it into the Main function in the "Program.cs" file. This snippet creates an ElastiCacheClusterConfig object using the hostname and port specified in the parameters, and then defaults the rest of the settings. It then creates a MemcachedClient through the Enyim framework by passing in the ElastiCacheClusterConfig as an IMemcachedClientConfiguration. The program then attempts to store a value to the cache followed by trying to retrieve a value in the cache.

Console.WriteLine("Creating config...");
ElastiCacheClusterConfig config = new ElastiCacheClusterConfig("YOUR-URL-HERE", 11211);
    
Console.WriteLine("Creating client...");
MemcachedClient client = new MemcachedClient(config);
    
if (client.Store(Enyim.Caching.Memcached.StoreMode.Set, "Demo", "Hello World"))
{
    Console.WriteLine("Stored to cache successfully");
}
else
{
   Console.WriteLine("Did not store to cache successfully");
}

Object value;
if (client.TryGet("Demo", out value))
{
   Console.WriteLine("Got the value: " + (value as string));
}
else
{
   // Search Database if the get fails
   Console.WriteLine("Checking database because get failed");
}

Console.Read();

Be sure to replace "YOUR-URL-HERE" with the endpoint URL of your cluster. You can find this URL by setting your variables from earlier if you closed out of PowerShell, and then running the following:

(Get-ECCacheCluster -CacheCluster $cacheClusterName).ConfigurationEndpoint.Address

Now, go ahead and build your project by right-clicking on the project in Solution Explorer and clicking "Build". If you run this code on your local machine, it will throw an error because you can only connect to ElastiCache inside of an EC2 instance. That’s why we need to transfer it to the EC2 instance we created earlier.

Upload it to the EC2 instance and test it

There are many ways to access and upload your ElastiCache application to EC2 such as the Visual Studio Toolkit, opening PowerShell access remotely on the instance and downloading it from a URL, or using remote desktop into the instance. Today, we’ll use remote desktop just for simplicity, even though there are much better ways to do this in a development stack. Run the following cmdlet to open a remote desktop connection to the instance. If you closed PowerShell earlier, be sure to copy in the predefined variables. If login fails, which is caused by a changed registry value, simply copy the value from $pass, paste it in as the password, and then login.

$secGroup = Get-EC2SecurityGroup -GroupNames $ec2SecurityGroupName
$groupArray = @($secGroup.GroupId)
$filter_groupId= New-Object Amazon.EC2.Model.Filter -Property @{Name = "group-id"; Values = $groupArray}
$instances = (Get-EC2Instance -Filter $filter_groupId).Instances
$pass = Get-EC2PasswordData -InstanceId $instances.InstanceId -PemFile C:$keyPairName.pem
$dns = $instances.PublicDnsName
cmdkey /generic:$dns /user:administrator /pass:$pass
mstsc /v:$dns

If you would rather use the console, you can find the .pem file for the password in the base directory of C:. Now that we have our connection open, go ahead and copy the executable and .dlls we created earlier and paste them into the instance. Run it and you should see the following output:

Creating config...
Creating client...
Stored to cache successfully
Got the value: Hello World

Delete Demo Services

Once you’ve successfully run your application, you can then delete the resources we created from AWS using the cmdlets below. Note: Be sure to copy the variables from earlier if you’ve closed the PowerShell window before now.

$secGroup = Get-EC2SecurityGroup -GroupNames $ec2SecurityGroupName
$groupArray = @($secGroup.GroupId)
$filter_groupId= New-Object Amazon.EC2.Model.Filter -Property @{Name = "group-id"; Values = $groupArray}
$instances = (Get-EC2Instance -Filter $filter_groupId).Instances
Stop-EC2Instance -Instance $instances -Terminate -Force

Remove-EC2KeyPair -KeyName $keyPairName -Force

Remove-ECCacheCluster -CacheClusterId $cacheClusterName -Force

And delete the policies when both services have finished terminating the resources.

$secGroup = Get-EC2SecurityGroup -GroupNames $ec2SecurityGroupName
Remove-EC2SecurityGroup -GroupId $secGroup.GroupId -Force
Remove-ECCacheSecurityGroup -CacheSecurityGroupName $cacheGroupName -Force

And that’s it for using Amazon’s new ElastiCache Cluster Configuration library for .NET. If you’d like to find out more, visit this wiki or fork the code at our Github repository.

DynamoDB Series – Expressions

by Norm Johanson | on | in .NET | Permalink | Comments |  Share

For the final installment of our Amazon DynamoDB series, we are going to look at the new expression support. There are two types of expressions used by DynamoDB. First you can use expressions to update specific fields in an item. The other way is to use expressions on puts, updates, or deletes to prevent the operation from succeeding if the item in DynamoDB doesn’t meet the expression.

Update Expressions

Update expressions are great for atomic updates to attributes in an item in DynamoDB. For example, let’s say we add a player item to a DynamoDB table that records the number of games won or lost and the last time a game was played.

PutItemRequest putRequest = new PutItemRequest
{
    TableName = tableName,
    Item = new Dictionary<string, AttributeValue>
    {
        {"id", new AttributeValue{S = "1"}},
        {"name", new AttributeValue{S = "Norm"}},
        {"wins", new AttributeValue{N = "0"}},
        {"loses", new AttributeValue{N = "0"}}
    }
};

ddbClient.PutItem(putRequest);

When a player wins the game, we need to update the wins attribute and set the time the last game was played and who the opponent was. To do that, we could get the item and look up how many wins the player currently has and then update the wins with the current wins + 1. The tricky thing is what happens if there is an update to the item in between the get and the update. We can handle that by putting an ExpectedAttribute value on the update, which will cause the update to fail, and then we could retry the whole process.

Now, using expressions, we can increment the wins attribute without having to first read the value. Let’s look at the update call to see how that works.

UpdateItemRequest updateRequest = new UpdateItemRequest
{
    TableName = tableName,
    Key = new Dictionary<string, AttributeValue>
    {
        {"id", new AttributeValue{S = "1"}}
    },
    UpdateExpression = "ADD #a :increment SET #b = :date, #c = :opponent",
    ExpressionAttributeNames = new Dictionary<string, string>
    {
        {"#a", "wins"},
        {"#b", "last-played"},
        {"#c", "last-opponent"}
    },
    ExpressionAttributeValues = new Dictionary<string, AttributeValue>
    {
        {":increment", new AttributeValue{N = "1"}},
        {":date", new AttributeValue{S = DateTime.UtcNow.ToString("O")}},
        {":opponent", new AttributeValue{S = "Celeste"}}
    }
};

ddbClient.UpdateItem(updateRequest);

The TableName and Key properties are used to identify the item we want to update. The UpdateExpression property is the interesting property where we can see the expression that is run on the item. Let’s break this statement down by each token.

The ADD token is the command token, and for a numeric attribute it adds the specified value to the attribute. Next is the #a token, which is a variable. The ‘#’ means this variable will be replaced with an attribute name. :increment is another variable that is the value to be added to the attribute #a. All tokens that start with ‘:’ are variables that will have a value supplied in the update request.

SET is another command token. It means all the attributes following will have their value set. The #b variable will get its value from the :date variable, and #c will get is value from the :opponent variable.

It is also possible to remove an attribute using the REMOVE command token.

The ExpressionAttributeNames property is used to set all the attribute variables in the expression to the actual attributes we want to use. ExpressionAttributeValues property is used to set all the value variables to the values we want to use in the expression.

Once we invoke the update, DynamoDB guarantees that all the attributes in the expression will be updated at the same time without the worry of some other thread coming in and updating the item in the middle of the process. This also saves us from using up any of our read capacity to do the the update.

Check out the DynamoDB Developer Guide for more information on how to use update expressions.

Conditional Expressions

For Puts, Updates, and Deletes, a conditional expression can be set. If the expression evaluates to false, then a ConditionalCheckFailedException exception is thrown. On the low-level service client, this can be done using the ConditionExpression property. Conditional expressions can also be used on the Document Model API. To take a look how this is done, let’s first create a game document in our game table.

DateTime lastUpdated = DateTime.Now;
Table gameTable = Table.LoadTable(ddbClient, tableName, DynamoDBEntryConversion.V2);

Document game = new Document();
game["id"] = gameId;
game["players"] = new List<string>{"Norm", "Celeste"};
game["last-updated"] = lastUpdated;
gameTable.PutItem(game);

For the game’s logic, every time the game document is updated the last-updated attribute is checked to make sure it hasn’t changed since the document was retrieved and then updated to a new date. So first let’s get the document and update the winner.

Document game = gameTable.GetItem(gameId);

game["winner"] = "Norm";
game["last-updated"] = DateTime.Now;

To declare the conditional expression I need to create an Expression object.

var expr = new Expression();
expr.ExpressionStatement = "attribute_not_exists(#timestamp) or #timestamp = :timestamp";
expr.ExpressionAttributeNames["#timestamp"] = "last-updated";
expr.ExpressionAttributeValues[":timestamp"] = lastUpdated;

This expression evaluates to true if the last-updated attribute does not exist or is equal to the last retrieved timestamp. Then, to use the expression, assign it to the UpdateItemOperationConfig and pass it to the UpdateItem operation.

UpdateItemOperationConfig updateConfig = new UpdateItemOperationConfig
{
    ConditionalExpression = expr
};

try
{
    gameTable.UpdateItem(game, updateConfig);
}
catch(ConditionalCheckFailedException e)
{
    // Retry logic
}

To handle the expression evaluating to false, we need to catch the ConditionalCheckFailedException and call our retry logic. To avoid having to catch exceptions in our code, we can use the new "Try" methods added to the SDK, which return true or false depending on whether the write was successful. So the above code could be rewritten like this:

if(!gameTable.TryUpdateItem(game, updateConfig))
{
    // Retry logic
}

This same pattern can be used for Puts and Deletes. For more information about using conditional expressions, check out the Amazon DynammoDB Developer Guide.

Conclusion

We hope you have enjoyed our series on Amazon DynamoDB this week. Hopefully, you have learned some new tricks that you can use in your application. Let us know what you think either in the comments below or through our forums.

DynamoDB Series – Object Persistence Model

by Pavel Safronov | on | in .NET | Permalink | Comments |  Share

This week, we are running a series of five daily blog posts that will explain new DynamoDB changes and how they relate to the AWS SDK for .NET. This is the fourth blog post, and today we will be discussing the Object Persistence Model.

Object Persistence Model

The Object Persistence Model API provides a simple way to work with Plain Old CLR Objects (POCO), as the following examples illustrate.

First, let’s look at the POCO class definition. (Notice that the class is marked up with multiple Amazon DynamoDB* attributes. These are included for clarity, even though these attributes are now optional, and will be removed in the next sample.)

[DynamoDBTable("Products")]
public class Product
{
    [DynamoDBHashKey]
    public int Id { get; set; }
    [DynamoDBRangeKey]
    public string Name { get; set; }

    public List<string> Aliases { get; set; }
    public bool IsPublic { get; set; }
}

Next, we can create, store, load, and query DynamoDB, all while using our POCO.

var product = new Product
{
    Id = 1,
    Name = "CloudSpotter",
    Aliases = new List<string> { "Prod", "1.0" },
    IsPublic = true,
};
Context.Save(product);
var retrieved = Context.Load(2);
var products = Context.Query<Product>(1, QueryOperator.BeginsWith, "Cloud");

The addition of the DynamoDB data type M (a string-key map of arbitrary data) allows the Object Persistence Model API to store complex data types as attributes of a single DynamoDB item. (We covered the new DynamoDB types earlier this week. It might be a good idea for you to review this again.) To illustrate this, let’s consider the following example where our Product class may reference another class.

Here are the new class definitions we will be working with.

public class Product
{
    public int Id { get; set; }
    public string Name { get; set; }
    public List<string> Aliases { get; set; }
    public bool IsPublic { get; set; }
    public Dictionary<string, string> Map { get; set; }
    public Metadata Meta { get; set; }
}
public class Metadata
{
    public double InternalVersion { get; set; }
    HashSet<string> Developers { get; set; }
}

Notice that we are going to use Dictionary objects, which will also be stored as M data types. (The only limitations are that the key must be of type string, and the value must be a supported primitive type or a complex structure.)

Now we can instantiate and work with our objects as we normally would.

Product product = new Product
{
    Id = 1,
    Name = "CloudSpotter",
    Aliases = new List<string> { "Prod", "1.0" },
    IsPublic = true,
    Meta = new Metadata
    {
        InternalVersion = 1.2,
        Developers = new HashSet<string> { "Alan", "Franco" }
    },
    Map = new Dictionary<string, string>
    {
        { "a", "1" },
        { "b", "2" }
    }
};
Context.Save(product);
var retrieved = Context.Load(2);
var products = Context.Query<Product>(1, QueryOperator.BeginsWith, "Cloud");

As you can see, the new DynamoDB data types really expand the range of data that you can maintain and work with. Though, you do have to be careful that the objects you are creating do not end up having circular references, because the API will end up throwing an exception for these objects.

DynamoDB Series – Conversion Schemas

by Pavel Safronov | on | in .NET | Permalink | Comments |  Share

This week, we are running a series of five daily blog posts that will explain new DynamoDB changes and how they relate to the AWS SDK for .NET. This is the third blog post, and today we will be discussing conversion schemas.

Conversion Schemas

Document doc = new Document();
doc["Id"] = 1;
doc["Product"] = "DataWriter";
doc["Aliases"] = new List<string> { "Prod", "1.0" };
doc["IsPublic"] = true;
table.UpdateItem(doc);

As you have seen earlier this week and in this example, it is very easy to work with a Document object and use .NET primitives. But how is this data actually stored in Amazon DynamoDB? The latest version of DynamoDB has added support for—among other things—native support for booleans (BOOL type) and lists of arbitrary elements (L type). (The various new DynamoDB types are covered in the first post in this series.) So in the above sample, are we taking advantage of these new types? To address this question and to provide a simple mechanism to control how your data is stored, we have introduced the concept of conversion schemas.

Why conversion schemas?

The new DynamoDB features are a powerful addition to the .NET SDK, but adding them to the Document Model API presented a challenge. The API was already capable of writing booleans and lists of items to DynamoDB, so how do the new BOOL and L types fit in? Should all booleans now be stored as BOOL (current implementation stores booleans as N types, either 1 or 0) and should lists be stored as L instead of SS/NS/BS? Changing how data is stored with the new SDK would break existing applications (older code would not be aware of the new types and query/scan conditions depend on the current types), so we have provided conversion schemas so that you can control exactly how your data is stored in DynamoDB. Schema V1 will maintain the current functionality, while Schema V2 will allow you to use the new types.

V1

The default conversion approach that the Document Model uses is as follows:

  • Number types (byte, int, float, decimal, etc.) are converted to N
  • String and char are converted to S
  • Bool is converted to N (0=false, 1=true)
  • DateTime and Guid are converted to S
  • MemoryStream and byte[] are converted to B
  • List, HashSet, and array of numerics types are converted to NS
  • List, HashSet, and array of string-based types are converted to SS
  • List, HashSet, and array of binary-based types are converted to BS

This conversion approach is known as Conversion Schema V1. It is the default conversion that the Document Model API uses and is identical in functionality to the conversion used by the SDK prior to the 2.3.2 release. As you can see, this schema does not take full advantage of the new DynamoDB types BOOL or L: the attribute Aliases will be stored as a string set (SS), while the boolean attribute IsPublic will be stored as a numeric (N).

But what if you wanted to now store a boolean as BOOL instead of N and List<string> as L instead of SS? The simple way to do this would be to use Conversion Schema V2.

V2

Conversion Schema V2 differs from V1 in that boolean values are stored as BOOL types, Lists are stored as L, and HashSets are stored as sets (NS, SS, or BS, depending on the data). So if you use V2 schema to store the Document in our example, it will function identically from the perspective of the application, but the data stored in DynamoDB will be different. The V2 schema differs from V1 in the following ways:

  • Boolean values will be stored as BOOL instead of N.
  • Lists and arrays of numerics, string-based types, and binary-based types are converted to L type.
  • HashSets of numerics, string-based types, and binary-based types are converted to NS, SS, or BS, as appropriate.
  • Other types are not impacted.

Note that Conversion Schema V2 differs in how it stores List<T> vs. HashSet<T>: List<T> is stored as a DynamoDB List (L type), while HashSet<T> is stored as a DynamoDB Set (NS, SS, or BS type). So if we wanted to use schema V2 but keep the Tags attributed as a set, we could update the code in our example to use HashSet<string> instead of List<string>.

Using Conversion Schemas with the Document Model

Conversion schemas are set for a particular Table object. (This means that the same Document stored using different Table objects may result in different data being written to DynamoDB.) The following sample shows how to load two Table objects, one configured with schema V1 and the other with schema V2, using two different LoadTable approaches.

Table tableV1 = LoadTable(client, "SampleTable", DynamoDBEntryConversion.V1);
Table tableV2;
TryLoadTable(client, "SampleTable", DynamoDBEntryConversion.V2, out tableV2);

You may also load a Table object without specifying a conversion, in which case the Table will use either the default V1 conversion or the conversion that you specify in your app.config file, as shown in the following sample.

<configuration>
  <aws>
    <dynamoDB conversionSchema="V2" />
  </aws>
</configuration>

Avoiding Conversion

Conversion schemas are used to convert between .NET types and DynamoDB types. However, if you use classes that extend DynamoDBEntry (such as Primitive, DynamoDBBool or DynamoDBNull), conversion will not performed. So in cases where you want your data to be stored in a particular format irrespective of the conversion schema, you can use these types, as shown below.

var list = new DynamoDBList();
list.Add(1);
list.Add("Sam");
list.Add(new HashSet<string> { "Design", "Logo" });

doc["Bool"] = DynamoDBBool.True;
doc["Null"] = DynamoDBNull.Null;
doc["List"] = list;

Using Conversion Schemas with the Object Persistence Model

The changes to the Object Persistence Model API are very similar to the Document Model changes:

  • With Conversion Schema V1, booleans are stored as N, Lists and HashSets are stored as sets (NS, SS, or BS)
  • With Conversion Schema V2, booleans are stored as BOOL, Lists are stored as L, HashSets are stored as sets (NS, SS, or BS)

Similarly to the Document Model, a conversion schema is associated with a DynamoDBContext and can be explicitly specified in code or the app.config, and will default to V1 if not set. The following example shows how to configure a context with V2 conversion schema.

var config = new DynamoDBContextConfig
{
    Conversion = DynamoDBEntryConversion.V2
};
var contextV2 = new DynamoDBContext(client, config);

Tomorrow, we will take a deeper look into the Object Persistence Model API and how the new DynamoDB types allow you to work with complex classes.

DynamoDB Series – Document Model

by Norm Johanson | on | in .NET | Permalink | Comments |  Share

This week we are running a series of five daily blog posts that will explain new DynamoDB changes and how they relate to the .NET SDK. This is blog post number 2, and today we will be looking at the Document Model API.

Document Model

Yesterday, we learned about Amazon DynamoDB’s new data types such as lists and maps. Today, we are going to talk about how you can use the new data types in the Document Model API.

As a quick refresher on the Document Model API, it is an abstraction over the low-level service client and is found in the Amazon.DynamoDBv2.DocumentModel namespace. Here is an example for creating an item using the Document Model API.

var table = Table.LoadTable(ddbClient, tableName);
Document company = new Document();
company["id"] = "1";
company["name"] = "Dunder Mifflin";
company["industry"] = "paper";

table.PutItem(company);

The Document Model keeps things simple and easy. It takes care of all the data conversions and creating the underlying low-level request objects. Now if you want to take advantage of the new map data type, you can simply create a separate document object and assign it to the parent document. So in our example above, let’s give our paper company an address.

Document address = new Document();
address["street"] = "1725 Slough Avenue";
address["city"] = "Scranton";
address["state"] = "Pennsylvania";

Document company = new Document();
company["id"] = "1";
company["name"] = "Dunder Mifflin";
company["industry"] = "paper";
company["address"] = address;

table.PutItem(company);

To get the data back out, use the GetItem method from the table object passing in the key information for the item.

Document dunder = table.GetItem("1");

Now we can take advantage of the recursive capabilities of the list and map data types and add a list of employees to our branch office.

var address = dunder["address"].AsDocument();
address["employee"] = new List<string> { "Michael Scott", "Dwight Schrute", "Jim Halpert", "Pam Beesly" };
table.UpdateItem(dunder);

As I said, the Document Model translates its calls into the low-level service client. Here is how the Document Model translates the save of the Dunder Mifflin company into a call to the low-level service client.

var request = new PutItemRequest
{
    TableName = tableName,
    Item = new Dictionary<string, AttributeValue>
    {
        {"id", new AttributeValue{S = "1"}},
        {"name", new AttributeValue{S = "Dunder Mifflin"}},
        {"industry", new AttributeValue{S = "paper"}},
        {"address", new AttributeValue
            {M = new Dictionary<string, AttributeValue>
            {
                {"street", new AttributeValue{S = "1725 Slough Avenue"}},
                {"city", new AttributeValue{S = "Scranton"}},
                {"state", new AttributeValue{S = "Pennsylvania"}},
                {"employee", new AttributeValue
                {L = new List<AttributeValue>
                {
                    new AttributeValue{S = "Michael Scott"},
                    new AttributeValue{S = "Dwight Schrute"},
                    new AttributeValue{S = "Jim Halpert"},
                    new AttributeValue{S = "Pam Beesly"}
                }}
            }}
        }
    }}
};


ddbClient.PutItem(request);

Tomorrow, we are going to go deeper into how the Document Model handles converting types passed into a document into the underlying data types that DynamoDB supports.

DynamoDB Series Kickoff

by Pavel Safronov | on | in .NET | Permalink | Comments |  Share

Last week, Amazon DynamoDB added support for JSON document data structures. With this update, DynamoDB now supports nested data in the form of lists (L type) and maps (M type). Also part of this update was native support for booleans (BOOL type) and nulls (NULL type).

This week, we will be running a series of daily blog posts that will explain the new changes and how they relate to the AWS SDK for .NET, and we will see how you can take advantage of these new types to work with complex objects in all three .NET SDK DynamoDB APIs. In this, the first blog post of series, we will see how the low-level API has changed. In the following days, we will cover the Document Model, Conversion Schemas, Object Persistence Model, and finally Expressions.

New types

Until now, DynamoDB had only six data types:

  • Scalars N, S, and B that represent number, string, and binary data.
  • Sets NS, SS, and BS that represent number set, string set, and binary set.
    Sets have the limitation that the data they store has to be homogeneous (e.g., SS could only contain S elements) and unique (no two elements could be the same).

This release expands the possible data types with four new additions:

  • BOOL represents boolean data.
  • NULL represents null values.
  • L type represents a list of elements.
  • M type represents a string-to-element map.

The key point about L and M types is that they can contain any DynamoDB type. This allows you to create, for example, lists of maps of lists, which in turn can contain a mix of numbers, strings, bools, and nulls, or any other conceivable combination of attributes.

Low-level

The low-level API changes are straightforward: new DynamoDB types are now supported in all data calls. Here’s a sample that shows how both old and new types can be used in a PutItem call.

// Put item
client.PutItem("SampleTable", new Dictionary<string, AttributeValue>
{
    { "Id", new AttributeValue { N = "1" } },
    { "Product", new AttributeValue { S = "DataWriter" } },
    { "Aliases", new AttributeValue {
        SS = new List<string> { "Prod", "1.0" } } },
    { "IsPublic", new AttributeValue { BOOL = false } },
    { "Metadata", new AttributeValue {
        M = new Dictionary<string, AttributeValue>
        {
            { "InternalVersion", new AttributeValue { N = "1.2" } },
            { "Developers", new AttributeValue {
                SS = new List<string> { "Alan", "Franko" } } 
            },
            { "SampleInput", new AttributeValue {
                L = new List<AttributeValue>
                {
                    new AttributeValue { BOOL = true },
                    new AttributeValue { N =  "42" },
                    new AttributeValue { NULL = true },
                    new AttributeValue {
                        SS = new List<string> { "apple", "orange" } }
                } }
            }
        } }
    }
});

As you can see, the new M and L AttributeValue types may contain AttributeValues, allowing complex, nested data to be stored in a single DynamoDB record. In the above example, the item we just stored into DynamoDB will have an attribute of type M named "Metadata". This attribute will in turn contain three other attributes: N (number), SS (string set), and L (list). The list contains four more attributes, which in turn can be other M and L types, though in our example they are not.

Tomorrow, we will take a look at how the new additions can be used with the Document Model API.

Getting Ready for re:Invent

by Norm Johanson | on | in .NET | Permalink | Comments |  Share

AWS re:Invent 2014 is coming up fast. Steve and I are heading to Las Vegas and will be presenting a session where we discuss some of the latest features of the AWS SDK for .NET. We’ll also be hanging out in the Expo area at the AWS Developer Resources booth, so please drop by and say hi. We would love to hear how you use our .NET tooling.

It’s not too late to register for AWS re:Invent 2014! See you in Las Vegas!

Amazon Cognito Credentials Provider

by Norm Johanson | on | in .NET | Permalink | Comments |  Share

Amazon Cognito helps you create unique identifiers for your end users that are kept consistent across devices and platforms. Cognito also delivers temporary, limited-privilege credentials to your app to access AWS resources. With Amazon Cognito, your app can support unauthenticated guest users as well as users authenticated through a identity provider, such as Facebook, Google, Login with Amazon or with developer authenticated identity providers.

Version 2.3.1.0 of AWS SDK for .NET has added Amazon.CognitoIdentity.CognitoAWSCredentials, a credentials object that uses Cognito and the Security Token Service to retrieve credentials in order to make AWS calls.

The first step in setting up CognitoAWSCredentials is to create an ”identity pool”. (An identity pool is a store of user identity information specific to your account. The information is retrievable across client platforms, devices, and operating systems, so that if a user starts using the app on a phone and later switches to a tablet, the persisted app information is still available for that user.) You can create a new identity pool from the Amazon Cognito management console. If you are using the console, it will also provide you the other pieces of information we will need:

  • Your account number: this is a 12-digit number, such as 123456789012, that is unique to your account.
  • The unauthenticated role ARN: this is a role that unauthenticated users will assume. For instance, this role can provide read-only permissions to your data.
  • The authenticated role ARN: authenticated users will assume this role. This role can have more extensive permissions to your data.

 

Here’s a simple code sample illustrating how this information is used to set up CognitoAWSCredentials, which can then be used to make a call to Amazon S3 as an unauthenticated user.

CognitoAWSCredentials credentials = new CognitoAWSCredentials(
    accountId,        // account number
    identityPoolId,   // identity pool id
    unAuthRoleArn,    // role for unauthenticated users
    null,             // role for authenticated users, not set
    region);
using (var s3Client = new AmazonS3Client(credentials))
{
    s3Client.ListBuckets();
}

 

As you can see, we are able to make calls with just a minimum amount of data required to authenticate the user. User permissions are controlled by the role, so you are free to configure access as you see fit.

The next example shows how you can start using AWS as an unauthenticated user, then authenticate through Facebook and update the credentials to use Facebook credentials. Using this approach, you can grant different capabilities to authenticated users via the authenticated role. For instance, you might have a Windows Phone application that permits users to view content anonymously, but allows them to post if they are logged on with one or more of the configured providers.

CognitoAWSCredentials credentials = new CognitoAWSCredentials(
    accountId, identityPoolId,
    unAuthRoleArn,    // role for unauthenticated users
    authRoleArn,      // role for authenticated users
    region);
using (var s3Client = new AmazonS3Client(credentials))
{
    // Initial use will be unauthenticated
    s3Client.ListBuckets();
    
    // Authenticate user through Facebook
    string facebookToken = GetFacebookAuthToken();
    
    // Add Facebook login to credentials. This will clear the current AWS credentials
    // and new AWS credentials using the authenticated role will be retrieved.
    credentials.AddLogin("graph.facebook.com", facebookAccessToken);

    // This call will be performed with the authenticated role and credentials
    s3Client.ListBuckets();
}

This new credentials object provides even more functionality if used with the AmazonCognitoSyncClient that is part of the .NET SDK: if you are using both AmazonCognitoSyncClient and CognitoAWSCredentials, you don’t have to specify the IdentityPoolId and IdentityId properties when making calls with the AmazonCognitoSyncClient. These properties are automatically filled in from CognitoAWSCredentials. Our final example illustrates this, as well as an event that notifies us whenever the IdentityId for CognitoAWSCredentials changes. (The IdentityId can change in some cases, such as going from an unauthenticated user to an authenticated one.)

CognitoAWSCredentials credentials = GetCognitoAWSCredentials();

// Log identity changes
credentials.IdentityChangedEvent += (sender, args) =>
{
    Console.WriteLine("Identity changed: [{0}] => [{1}]", args.OldIdentityId, args.NewIdentityId);
};

using(var syncClient = new AmazonCognitoSyncClient(credentials))
{
    var result = syncClient.ListRecords(new ListRecordsRequest
    {
        DatasetName = datasetName
        // No need to specify these properties
        //IdentityId = "...",
        //IdentityPoolId = "..."        
    });
}

For more information on Amazon Cognito, including use-cases and sample policies, visit the official Amazon Cognito page or the Cognito section of the Mobile Development blog.

Object Lifecycles

by Pavel Safronov | on | in .NET | Permalink | Comments |  Share

When using the AWS SDK for .NET, you may find yourself wondering how to use various components of the SDK and, more importantly, how to properly dispose of resources once you are done with them. This blog post reviews the lifecycles of various SDK objects and the best practices for using them.

Lifecycles

There are three basic lifecycle concerns in the AWS SDK for .NET:

  • Thread safety – Some objects can be used across multiple threads without worrying about errors or data corruption.
  • Disposability – Some objects should be disposed of, either by calling .Dispose() on the object or with a using block.
  • Cacheability – Some objects should be cached or reused, usually because they are expensive to recreate.

Clients

The best-known aspect of the AWS SDK for .NET are the various service clients that you can use to interact with AWS. Client objects are thread safe, disposable, and can be reused. (Client objects are inexpensive, so you are not incurring a large overhead by constructing multiple instances, but it’s not a bad idea to create and reuse a client.)

Here’s a simple example of using an Amazon DynamoDB client to list all the tables in your account. Note that we wrap the client in a using block to make sure it is disposed of, either after the last line of the using block is executed or when an exception is thrown. The last point is why it is a good idea, whenever possible, to wrap disposable objects in a using block.

using(var client = new AmazonDynamoDBClient())
{
    var tableNames = client.ListTables().TableNames;
    Console.WriteLine("My DynamoDB tables: " + string.Join(", ", tableNames));
}

High-level objects

The SDK has a number of high-level abstractions built on top of the various service clients. These helper classes provide extra functionality, such as the Amazon S3 TransferUtility class, which automatically handles multi-part uploads, or the DynamoDBContext class, which allows you to store and load .NET objects in Amazon DynamoDB.

  • Amazon.DynamoDBv2.DocumentModel.Table
  • Amazon.DynamoDBv2.DataModel.DynamoDBContext
  • These classes are thread safe, disposable, and cacheable. Both Table and DynamoDBContext should be reused as much as possible, as these objects create and use caches that are populated either from DynamoDB or reflection, operations that could severely degrade performance if performed often (for instance, by recreating a table or a context for every operation).

  • Amazon.S3.Transfer.TransferUtility
  • Amazon.Glacier.Transfer.ArchiveTransferManager
  • These classes are thread safe and disposable. They can be treated like client objects.

Response objects

Most SDK response objects are simple classes, but in a few cases the response objects that you receive from a service call will be disposable. It is important to be aware of what these results are and to properly dispose of them.

The best-known example of this is GetObjectResponse. This is a response returned from an Amazon S3 GetObject call. It is disposable because it returns back to the caller a stream from Amazon S3 of the content for the object. Here’s an example of how this object should be used and disposed of.

MemoryStream ms = new MemoryStream();
using(var client = new Amazon.S3.AmazonS3Client())
{
    var request = new GetObjectRequest
    {
        BucketName = bucketName, Key = key
    };
    using(var response = client.GetObject(request))
    {
        response.ResponseStream.CopyTo(ms);
    }
}

As you can see, we wrapped both the client and the response in using blocks. This makes sure that we dispose of the underlying .NET web streams. (If these are not properly disposed of, you may eventually not be able to make any service calls at all.)

There are two disposable response objects like this in the SDK: Amazon.S3.Model.GetObjectResponse and Amazon.S3.Model.GetObjectTorrentResponse.