AWS Developer Blog

AWS SDK for Java Maven Modules

by Jason Fulghum | on | in Java | Permalink | Comments |  Share

We’re excited to announce that with version 1.9.0 of the AWS SDK for Java, we’ve switched to a modular Maven project structure. You can now selectively pick up which components of the SDK you want. For example, if your project uses only Amazon S3 and Amazon DynamoDB, you can configure the dependencies in your project’s pom.xml to pick up only those components, like this:

<dependencies>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-s3</artifactId>
        <version>1.9.0</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-dynamodb</artifactId>
        <version>1.9.0</version>
    </dependency>
</dependencies>

Notice that we’re picking up the same version for each of the two components we declare dependencies on. The individual components are versioned together, so you can easily pick compatible versions of different components that you know work together and have been tested together.

Just like before, if you want to pick up the entire SDK, with support for all AWS infrastructure services, you can configure your project’s pom.xml like this:

<dependencies>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk</artifactId>
        <version>1.9.0</version>
    </dependency>
</dependencies>

You can also still download the complete AWS SDK for Java as a .zip file.

This was the most requested issue on GitHub, and we were very excited to deliver it to customers!

Do you have other feature requests for the AWS SDK for Java?

Utilizing Amazon ElastiCache Auto Discovery in .NET Through Enyim

by Mason Schneider | on | in .NET | Permalink | Comments |  Share

Today, we released a new library, Amazon ElastiCache Cluster Configuration, that allows .NET applications to easily leverage ElastiCache features. This post explains why a programmer would want to use this library and offers a quick and easy way to try it yourself.

What is Memcached?

Memcached provides a way to easily avoid some of the latency that comes with using a database, and it can also help applications at scale by removing some of the strain that can be placed on databases. This is accomplished by having Memcached servers be an intermediary, in-memory cache that can return results much faster than a normal database. In a typical program flow, this is accomplished by requesting a key from a group of cache servers and, if a value is retrieved, no database query is needed. Below is a simple diagram of what this function normally looks like: Memcached Diagram

Why ElastiCache?

ElastiCache provides a way to dynamically add and remove Memcached servers inside of a cache cluster. All servers are completely managed, which means that when servers are added they are automatically configured for Memcached. They are also added to the cluster and, when they are deleted, the cluster is updated. This means you spend less time configuring Memcached servers and more time working on things that matter. Being able to add and remove these nodes dynamically also means your application can easily scale whenever necessary through the AWS Management Console or through one of the many AWS APIs.

Using ElastiCache in .NET

Many .NET developers leverage ElastiCache through the Enyim framework. Enyim provides a client that manages server connections as well as what server your cache data should be stored on. To be aware of the Memcached servers, the Enyim client is configured on instantiation with IPs and ports for all of the servers. When the server information changes, the client must be disposed and re-instantiated with the new server information. The re-instantiation of the client tends to be tedious and can also cause issues if you update your configuration incorrectly when nodes change. One feature of ElastiCache that helps avoid this issue is Auto Discovery. This feature allows clients to find out the cluster configuration through an endpoint URL. The endpoint URL is sort of like an alias that just points to one of the servers in the cluster. Each server holds information about the configuration such as how many times the configuration has changed and the hostname, ip, and port of each server in the cluster. For more information on how Auto Discovery works, visit this page.

ElastiCache Cluster Configuration

Although Auto Discovery is useful, it is not accessible through Enyim’s client because it is not something that is found in standard Memcached clusters. Currently, Amazon has released clients for Java and PHP that extend Memcached clients in order to take advantage of Auto Discovery. With today’s release of the ElastiCache Cluster Configuration library, any .NET application using framework 3.5 or higher can now take full advantage of this great feature. All that is required is to add the clusterclient section to your App.config or instantiate the configuration object through parameters. After that, pass it as the configuration for the Enyim MemcachedClient, and you now have a MemcachedClient functioning through ElastiCache Auto Discovery.

Here is a sample App.config that shows how to specify the Amazon ElastiCache cluster.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <configSections>
      <section name="clusterclient" type="Amazon.ElastiCacheCluster.ClusterConfigSettings, Amazon.ElastiCacheCluster" />
    </configSections>

    <clusterclient>
      <endpoint hostname="my-configuruation-url.cache.amazonaws.com" port="11211" />
      <poller intervalDelay="60000" />
    </clusterclient>

</configuration>

 

Try it now!

For this walkthrough, we assume you have an AWS account with the ability to control Amazon EC2 and ElastiCache services, internet access, and the AWS Tools for Windows PowerShell installed and configured with an API key and secret key.

Create AWS Services using PowerShell

Start a PowerShell instance as Administrator. Below, you can edit these variable values to change property names, and then copy and paste the rest.

$ec2SecurityGroupName = "myElastiCacheGroup"
$cacheGroupName = "myElastiCacheSecGroup"
$keyPairName = "myConfigKeyPair"
$cacheClusterName = "demoCluster"

To use ElastiCache, you must create an ElastiCache cluster and an EC2 instance to access it. First, we create a key pair and an EC2 security group, and then we create an EC2 instance based on those values.

$myConfigKeyPair = New-EC2KeyPair -KeyName $keyPairName
$myConfigKeyPair.KeyMaterial | Out-File -Encoding ascii C:$keyPairName.pem

New-EC2SecurityGroup -GroupName $ec2SecurityGroupName -GroupDescription "ElastiCache Config Demo"

$cidrBlocks = @("0.0.0.0/0")
$ipPermissions = New-Object Amazon.EC2.Model.IpPermission -Property @{IpProtocol = "tcp"; FromPort = 11211; ToPort = 11211; IpRanges = $cidrBlocks}
Grant-EC2SecurityGroupIngress -GroupName $ec2SecurityGroupName -IpPermissions $ipPermissions
$ipPermissions = New-Object Amazon.EC2.Model.IpPermission -Property @{IpProtocol = "tcp"; FromPort = 3389; ToPort = 3389; IpRanges = $cidrBlocks}
Grant-EC2SecurityGroupIngress -GroupName $ec2SecurityGroupName -IpPermissions $ipPermissions

$image = Get-EC2ImageByName -Names WINDOWS_2012R2_BASE
if($image -is [system.array]) {$image = $image[0]}

$reservation = New-EC2Instance -ImageId $image.ImageId -KeyName $keyPairName -SecurityGroups $ec2SecurityGroupName -InstanceType t1.micro

After that is complete, we create a new ElastiCache cluster with three nodes and add the EC2 security group to its policy.

New-ECCacheSecurityGroup -CacheSecurityGroupName $cacheGroupName -Description "Demo for ElastiCache Config"
$secGroup = Get-EC2SecurityGroup -GroupNames $ec2SecurityGroupName
Approve-ECCacheSecurityGroupIngress -CacheSecurityGroupName $cacheGroupName -EC2SecurityGroupName $ec2SecurityGroupName -EC2SecurityGroupOwnerId $secGroup.OwnerId

New-ECCacheCluster -CacheNodeType cache.t1.micro -CacheClusterId $cacheClusterName -CacheSecurityGroupNames $cacheGroupName -Engine memcached -EngineVersion 1.4.14 -NumCacheNodes 3 -Port 11211

Create the Application

To demonstrate how to use ElastiCache Cluster Configuration, we’ll make a quick console application. From the start page of Visual Studio 2010 or higher, click "New Project…", and in the new project dialog, create a new Visual C# Console Application named "Cluster Config Demo". After the project is created, in Solution Explorer, right-click on the "References" section, and in the drop-down menu, click "Manage NuGet Packages…". After that, we’re going to search for "ElastiCacheClusterConfig" and then install it into our current project. When you install this package, Enyim is also installed on the project. Now that we have the project configured, let’s write the code. First, add the packages to the code by pasting this code into the top of the file "Program.cs".

using Enyim.Caching;
using Amazon.ElastiCacheCluster;

Next, copy the code below and paste it into the Main function in the "Program.cs" file. This snippet creates an ElastiCacheClusterConfig object using the hostname and port specified in the parameters, and then defaults the rest of the settings. It then creates a MemcachedClient through the Enyim framework by passing in the ElastiCacheClusterConfig as an IMemcachedClientConfiguration. The program then attempts to store a value to the cache followed by trying to retrieve a value in the cache.

Console.WriteLine("Creating config...");
ElastiCacheClusterConfig config = new ElastiCacheClusterConfig("YOUR-URL-HERE", 11211);
    
Console.WriteLine("Creating client...");
MemcachedClient client = new MemcachedClient(config);
    
if (client.Store(Enyim.Caching.Memcached.StoreMode.Set, "Demo", "Hello World"))
{
    Console.WriteLine("Stored to cache successfully");
}
else
{
   Console.WriteLine("Did not store to cache successfully");
}

Object value;
if (client.TryGet("Demo", out value))
{
   Console.WriteLine("Got the value: " + (value as string));
}
else
{
   // Search Database if the get fails
   Console.WriteLine("Checking database because get failed");
}

Console.Read();

Be sure to replace "YOUR-URL-HERE" with the endpoint URL of your cluster. You can find this URL by setting your variables from earlier if you closed out of PowerShell, and then running the following:

(Get-ECCacheCluster -CacheCluster $cacheClusterName).ConfigurationEndpoint.Address

Now, go ahead and build your project by right-clicking on the project in Solution Explorer and clicking "Build". If you run this code on your local machine, it will throw an error because you can only connect to ElastiCache inside of an EC2 instance. That’s why we need to transfer it to the EC2 instance we created earlier.

Upload it to the EC2 instance and test it

There are many ways to access and upload your ElastiCache application to EC2 such as the Visual Studio Toolkit, opening PowerShell access remotely on the instance and downloading it from a URL, or using remote desktop into the instance. Today, we’ll use remote desktop just for simplicity, even though there are much better ways to do this in a development stack. Run the following cmdlet to open a remote desktop connection to the instance. If you closed PowerShell earlier, be sure to copy in the predefined variables. If login fails, which is caused by a changed registry value, simply copy the value from $pass, paste it in as the password, and then login.

$secGroup = Get-EC2SecurityGroup -GroupNames $ec2SecurityGroupName
$groupArray = @($secGroup.GroupId)
$filter_groupId= New-Object Amazon.EC2.Model.Filter -Property @{Name = "group-id"; Values = $groupArray}
$instances = (Get-EC2Instance -Filter $filter_groupId).Instances
$pass = Get-EC2PasswordData -InstanceId $instances.InstanceId -PemFile C:$keyPairName.pem
$dns = $instances.PublicDnsName
cmdkey /generic:$dns /user:administrator /pass:$pass
mstsc /v:$dns

If you would rather use the console, you can find the .pem file for the password in the base directory of C:. Now that we have our connection open, go ahead and copy the executable and .dlls we created earlier and paste them into the instance. Run it and you should see the following output:

Creating config...
Creating client...
Stored to cache successfully
Got the value: Hello World

Delete Demo Services

Once you’ve successfully run your application, you can then delete the resources we created from AWS using the cmdlets below. Note: Be sure to copy the variables from earlier if you’ve closed the PowerShell window before now.

$secGroup = Get-EC2SecurityGroup -GroupNames $ec2SecurityGroupName
$groupArray = @($secGroup.GroupId)
$filter_groupId= New-Object Amazon.EC2.Model.Filter -Property @{Name = "group-id"; Values = $groupArray}
$instances = (Get-EC2Instance -Filter $filter_groupId).Instances
Stop-EC2Instance -Instance $instances -Terminate -Force

Remove-EC2KeyPair -KeyName $keyPairName -Force

Remove-ECCacheCluster -CacheClusterId $cacheClusterName -Force

And delete the policies when both services have finished terminating the resources.

$secGroup = Get-EC2SecurityGroup -GroupNames $ec2SecurityGroupName
Remove-EC2SecurityGroup -GroupId $secGroup.GroupId -Force
Remove-ECCacheSecurityGroup -CacheSecurityGroupName $cacheGroupName -Force

And that’s it for using Amazon’s new ElastiCache Cluster Configuration library for .NET. If you’d like to find out more, visit this wiki or fork the code at our Github repository.

Client-side Encryption for Amazon DynamoDB

by Hanson Char | on | in Java | Permalink | Comments |  Share

We are thrilled to introduce one of the latest AWS Labs projects for enabling client-side encryption for Amazon DynamoDB in Java. This library is designed to support encryption and signing of your data when stored in Amazon DynamoDB.

A typical use of this library is when you are using DynamoDBMapper, where transparent encryption and signing of all objects serialized through the mapper can be enabled by configuring an AttributeEncryptor.

Getting Started

Suppose you have created (sample code) a DynamoDB table "MyStore", and want to store some Book objects. Let’s further suppose the security requirement involves classifying the attributes Title and Authors as sensitive information. Here is how the Book class might look like:

@DynamoDBTable(tableName="MyStore")
public class Book {
    private Integer id;
    private String title;
    private String ISBN;
    private Set bookAuthors;
    private String someProp;

    // Not encrypted because it is a hash key    
    @DynamoDBHashKey(attributeName="Id")  
    public Integer getId() { return id;}
    public void setId(Integer id) {this.id = id;}

    // Encrypted by default
    @DynamoDBAttribute(attributeName="Title")  
    public String getTitle() {return title; }
    public void setTitle(String title) { this.title = title; }

    // Specifically not encrypted
    @DoNotEncrypt
    @DynamoDBAttribute(attributeName="ISBN")  
    public String getISBN() { return ISBN; }
    public void setISBN(String ISBN) { this.ISBN = ISBN; }

    // Encrypted by default
    @DynamoDBAttribute(attributeName = "Authors")
    public Set<String> getBookAuthors() { return bookAuthors; }
    public void setBookAuthors(Set<String> bookAuthors) {
        this.bookAuthors = bookAuthors;
    }

    // Not encrypted nor signed
    @DoNotTouch
    public String getSomeProp() { return someProp;}
    public void setSomeProp(String someProp) {this.someProp = someProp;}
}

For a typical use case of DynamoDBMapper, you can easily save and retrieve a Book object to and from Amazon DynamoDB without encryption (nor signing). For example,

AmazonDynamoDBClient client = new AmazonDynamoDBClient(...);
DynamoDBMapper mapper = new DynamoDBMapper(client);
Book book = new Book();
book.setId(123);
book.setTitle("Secret Book Title ");
// ... etc. setting other properties

// Saves the book unencrypted to DynamoDB
mapper.save(book);

// Loads the book back from DynamoDB
Book bookTo = new Book();
bookTo.setId(123);
Book bookTo = mapper.load(bookTo);

To enable transparent encryption and signing, simply specify the necessary encryption material via an EncryptionMaterialsProvider. For example:

AmazonDynamoDBClient client = new AmazonDynamoDBClient(...);
SecretKey cek = ...;        // Content encrypting key
SecretKey macKey =  ...;    // Signing key
EncryptionMaterialsProvider provider = 
            new SymmetricStaticProvider(cek, macKey);
mapper = new DynamoDBMapper(client, DynamoDBMapperConfig.DEFAULT,
            new AttributeEncryptor(provider));
Book book = new Book();
book.setId(123);
book.setTitle("Secret Book Title ");
// ... etc. setting other properties

// Saves the book both encrypted and signed to DynamoDB
mapper.save(book);

// Loads the book both with signature verified and decrypted from DynamoDB
Book bookTo = new Book();
bookTo.setId(123);
bookTo = mapper.load(bookTo);

Note that by default all attributes except the primary keys are both encrypted and signed for maximum security. To selectively disable encryption, you can use the annotation @DoNotEncrypt as shown in the Book class above. To disable both encryption and signing, you can use the annotation @DoNotTouch.

There are a variety of existing EncryptionMaterialsProvider implementations that you can use to provide the encryption material, including KeyStoreMaterialsProvider which makes use of a Java keystore. Alternatively, you can also plug in your own custom implementation.

For more information, head over to aws-dynamodb-encryption-java, and give it a spin. Happy crypto, and may the power of security and Amazon DynamoDB be with you!

DynamoDB Series – Expressions

by Norm Johanson | on | in .NET | Permalink | Comments |  Share

For the final installment of our Amazon DynamoDB series, we are going to look at the new expression support. There are two types of expressions used by DynamoDB. First you can use expressions to update specific fields in an item. The other way is to use expressions on puts, updates, or deletes to prevent the operation from succeeding if the item in DynamoDB doesn’t meet the expression.

Update Expressions

Update expressions are great for atomic updates to attributes in an item in DynamoDB. For example, let’s say we add a player item to a DynamoDB table that records the number of games won or lost and the last time a game was played.

PutItemRequest putRequest = new PutItemRequest
{
    TableName = tableName,
    Item = new Dictionary<string, AttributeValue>
    {
        {"id", new AttributeValue{S = "1"}},
        {"name", new AttributeValue{S = "Norm"}},
        {"wins", new AttributeValue{N = "0"}},
        {"loses", new AttributeValue{N = "0"}}
    }
};

ddbClient.PutItem(putRequest);

When a player wins the game, we need to update the wins attribute and set the time the last game was played and who the opponent was. To do that, we could get the item and look up how many wins the player currently has and then update the wins with the current wins + 1. The tricky thing is what happens if there is an update to the item in between the get and the update. We can handle that by putting an ExpectedAttribute value on the update, which will cause the update to fail, and then we could retry the whole process.

Now, using expressions, we can increment the wins attribute without having to first read the value. Let’s look at the update call to see how that works.

UpdateItemRequest updateRequest = new UpdateItemRequest
{
    TableName = tableName,
    Key = new Dictionary<string, AttributeValue>
    {
        {"id", new AttributeValue{S = "1"}}
    },
    UpdateExpression = "ADD #a :increment SET #b = :date, #c = :opponent",
    ExpressionAttributeNames = new Dictionary<string, string>
    {
        {"#a", "wins"},
        {"#b", "last-played"},
        {"#c", "last-opponent"}
    },
    ExpressionAttributeValues = new Dictionary<string, AttributeValue>
    {
        {":increment", new AttributeValue{N = "1"}},
        {":date", new AttributeValue{S = DateTime.UtcNow.ToString("O")}},
        {":opponent", new AttributeValue{S = "Celeste"}}
    }
};

ddbClient.UpdateItem(updateRequest);

The TableName and Key properties are used to identify the item we want to update. The UpdateExpression property is the interesting property where we can see the expression that is run on the item. Let’s break this statement down by each token.

The ADD token is the command token, and for a numeric attribute it adds the specified value to the attribute. Next is the #a token, which is a variable. The ‘#’ means this variable will be replaced with an attribute name. :increment is another variable that is the value to be added to the attribute #a. All tokens that start with ‘:’ are variables that will have a value supplied in the update request.

SET is another command token. It means all the attributes following will have their value set. The #b variable will get its value from the :date variable, and #c will get is value from the :opponent variable.

It is also possible to remove an attribute using the REMOVE command token.

The ExpressionAttributeNames property is used to set all the attribute variables in the expression to the actual attributes we want to use. ExpressionAttributeValues property is used to set all the value variables to the values we want to use in the expression.

Once we invoke the update, DynamoDB guarantees that all the attributes in the expression will be updated at the same time without the worry of some other thread coming in and updating the item in the middle of the process. This also saves us from using up any of our read capacity to do the the update.

Check out the DynamoDB Developer Guide for more information on how to use update expressions.

Conditional Expressions

For Puts, Updates, and Deletes, a conditional expression can be set. If the expression evaluates to false, then a ConditionalCheckFailedException exception is thrown. On the low-level service client, this can be done using the ConditionExpression property. Conditional expressions can also be used on the Document Model API. To take a look how this is done, let’s first create a game document in our game table.

DateTime lastUpdated = DateTime.Now;
Table gameTable = Table.LoadTable(ddbClient, tableName, DynamoDBEntryConversion.V2);

Document game = new Document();
game["id"] = gameId;
game["players"] = new List<string>{"Norm", "Celeste"};
game["last-updated"] = lastUpdated;
gameTable.PutItem(game);

For the game’s logic, every time the game document is updated the last-updated attribute is checked to make sure it hasn’t changed since the document was retrieved and then updated to a new date. So first let’s get the document and update the winner.

Document game = gameTable.GetItem(gameId);

game["winner"] = "Norm";
game["last-updated"] = DateTime.Now;

To declare the conditional expression I need to create an Expression object.

var expr = new Expression();
expr.ExpressionStatement = "attribute_not_exists(#timestamp) or #timestamp = :timestamp";
expr.ExpressionAttributeNames["#timestamp"] = "last-updated";
expr.ExpressionAttributeValues[":timestamp"] = lastUpdated;

This expression evaluates to true if the last-updated attribute does not exist or is equal to the last retrieved timestamp. Then, to use the expression, assign it to the UpdateItemOperationConfig and pass it to the UpdateItem operation.

UpdateItemOperationConfig updateConfig = new UpdateItemOperationConfig
{
    ConditionalExpression = expr
};

try
{
    gameTable.UpdateItem(game, updateConfig);
}
catch(ConditionalCheckFailedException e)
{
    // Retry logic
}

To handle the expression evaluating to false, we need to catch the ConditionalCheckFailedException and call our retry logic. To avoid having to catch exceptions in our code, we can use the new "Try" methods added to the SDK, which return true or false depending on whether the write was successful. So the above code could be rewritten like this:

if(!gameTable.TryUpdateItem(game, updateConfig))
{
    // Retry logic
}

This same pattern can be used for Puts and Deletes. For more information about using conditional expressions, check out the Amazon DynammoDB Developer Guide.

Conclusion

We hope you have enjoyed our series on Amazon DynamoDB this week. Hopefully, you have learned some new tricks that you can use in your application. Let us know what you think either in the comments below or through our forums.

DynamoDB Series – Object Persistence Model

by Pavel Safronov | on | in .NET | Permalink | Comments |  Share

This week, we are running a series of five daily blog posts that will explain new DynamoDB changes and how they relate to the AWS SDK for .NET. This is the fourth blog post, and today we will be discussing the Object Persistence Model.

Object Persistence Model

The Object Persistence Model API provides a simple way to work with Plain Old CLR Objects (POCO), as the following examples illustrate.

First, let’s look at the POCO class definition. (Notice that the class is marked up with multiple Amazon DynamoDB* attributes. These are included for clarity, even though these attributes are now optional, and will be removed in the next sample.)

[DynamoDBTable("Products")]
public class Product
{
    [DynamoDBHashKey]
    public int Id { get; set; }
    [DynamoDBRangeKey]
    public string Name { get; set; }

    public List<string> Aliases { get; set; }
    public bool IsPublic { get; set; }
}

Next, we can create, store, load, and query DynamoDB, all while using our POCO.

var product = new Product
{
    Id = 1,
    Name = "CloudSpotter",
    Aliases = new List<string> { "Prod", "1.0" },
    IsPublic = true,
};
Context.Save(product);
var retrieved = Context.Load(2);
var products = Context.Query<Product>(1, QueryOperator.BeginsWith, "Cloud");

The addition of the DynamoDB data type M (a string-key map of arbitrary data) allows the Object Persistence Model API to store complex data types as attributes of a single DynamoDB item. (We covered the new DynamoDB types earlier this week. It might be a good idea for you to review this again.) To illustrate this, let’s consider the following example where our Product class may reference another class.

Here are the new class definitions we will be working with.

public class Product
{
    public int Id { get; set; }
    public string Name { get; set; }
    public List<string> Aliases { get; set; }
    public bool IsPublic { get; set; }
    public Dictionary<string, string> Map { get; set; }
    public Metadata Meta { get; set; }
}
public class Metadata
{
    public double InternalVersion { get; set; }
    HashSet<string> Developers { get; set; }
}

Notice that we are going to use Dictionary objects, which will also be stored as M data types. (The only limitations are that the key must be of type string, and the value must be a supported primitive type or a complex structure.)

Now we can instantiate and work with our objects as we normally would.

Product product = new Product
{
    Id = 1,
    Name = "CloudSpotter",
    Aliases = new List<string> { "Prod", "1.0" },
    IsPublic = true,
    Meta = new Metadata
    {
        InternalVersion = 1.2,
        Developers = new HashSet<string> { "Alan", "Franco" }
    },
    Map = new Dictionary<string, string>
    {
        { "a", "1" },
        { "b", "2" }
    }
};
Context.Save(product);
var retrieved = Context.Load(2);
var products = Context.Query<Product>(1, QueryOperator.BeginsWith, "Cloud");

As you can see, the new DynamoDB data types really expand the range of data that you can maintain and work with. Though, you do have to be careful that the objects you are creating do not end up having circular references, because the API will end up throwing an exception for these objects.

DynamoDB Series – Conversion Schemas

by Pavel Safronov | on | in .NET | Permalink | Comments |  Share

This week, we are running a series of five daily blog posts that will explain new DynamoDB changes and how they relate to the AWS SDK for .NET. This is the third blog post, and today we will be discussing conversion schemas.

Conversion Schemas

Document doc = new Document();
doc["Id"] = 1;
doc["Product"] = "DataWriter";
doc["Aliases"] = new List<string> { "Prod", "1.0" };
doc["IsPublic"] = true;
table.UpdateItem(doc);

As you have seen earlier this week and in this example, it is very easy to work with a Document object and use .NET primitives. But how is this data actually stored in Amazon DynamoDB? The latest version of DynamoDB has added support for—among other things—native support for booleans (BOOL type) and lists of arbitrary elements (L type). (The various new DynamoDB types are covered in the first post in this series.) So in the above sample, are we taking advantage of these new types? To address this question and to provide a simple mechanism to control how your data is stored, we have introduced the concept of conversion schemas.

Why conversion schemas?

The new DynamoDB features are a powerful addition to the .NET SDK, but adding them to the Document Model API presented a challenge. The API was already capable of writing booleans and lists of items to DynamoDB, so how do the new BOOL and L types fit in? Should all booleans now be stored as BOOL (current implementation stores booleans as N types, either 1 or 0) and should lists be stored as L instead of SS/NS/BS? Changing how data is stored with the new SDK would break existing applications (older code would not be aware of the new types and query/scan conditions depend on the current types), so we have provided conversion schemas so that you can control exactly how your data is stored in DynamoDB. Schema V1 will maintain the current functionality, while Schema V2 will allow you to use the new types.

V1

The default conversion approach that the Document Model uses is as follows:

  • Number types (byte, int, float, decimal, etc.) are converted to N
  • String and char are converted to S
  • Bool is converted to N (0=false, 1=true)
  • DateTime and Guid are converted to S
  • MemoryStream and byte[] are converted to B
  • List, HashSet, and array of numerics types are converted to NS
  • List, HashSet, and array of string-based types are converted to SS
  • List, HashSet, and array of binary-based types are converted to BS

This conversion approach is known as Conversion Schema V1. It is the default conversion that the Document Model API uses and is identical in functionality to the conversion used by the SDK prior to the 2.3.2 release. As you can see, this schema does not take full advantage of the new DynamoDB types BOOL or L: the attribute Aliases will be stored as a string set (SS), while the boolean attribute IsPublic will be stored as a numeric (N).

But what if you wanted to now store a boolean as BOOL instead of N and List<string> as L instead of SS? The simple way to do this would be to use Conversion Schema V2.

V2

Conversion Schema V2 differs from V1 in that boolean values are stored as BOOL types, Lists are stored as L, and HashSets are stored as sets (NS, SS, or BS, depending on the data). So if you use V2 schema to store the Document in our example, it will function identically from the perspective of the application, but the data stored in DynamoDB will be different. The V2 schema differs from V1 in the following ways:

  • Boolean values will be stored as BOOL instead of N.
  • Lists and arrays of numerics, string-based types, and binary-based types are converted to L type.
  • HashSets of numerics, string-based types, and binary-based types are converted to NS, SS, or BS, as appropriate.
  • Other types are not impacted.

Note that Conversion Schema V2 differs in how it stores List<T> vs. HashSet<T>: List<T> is stored as a DynamoDB List (L type), while HashSet<T> is stored as a DynamoDB Set (NS, SS, or BS type). So if we wanted to use schema V2 but keep the Tags attributed as a set, we could update the code in our example to use HashSet<string> instead of List<string>.

Using Conversion Schemas with the Document Model

Conversion schemas are set for a particular Table object. (This means that the same Document stored using different Table objects may result in different data being written to DynamoDB.) The following sample shows how to load two Table objects, one configured with schema V1 and the other with schema V2, using two different LoadTable approaches.

Table tableV1 = LoadTable(client, "SampleTable", DynamoDBEntryConversion.V1);
Table tableV2;
TryLoadTable(client, "SampleTable", DynamoDBEntryConversion.V2, out tableV2);

You may also load a Table object without specifying a conversion, in which case the Table will use either the default V1 conversion or the conversion that you specify in your app.config file, as shown in the following sample.

<configuration>
  <aws>
    <dynamoDB conversionSchema="V2" />
  </aws>
</configuration>

Avoiding Conversion

Conversion schemas are used to convert between .NET types and DynamoDB types. However, if you use classes that extend DynamoDBEntry (such as Primitive, DynamoDBBool or DynamoDBNull), conversion will not performed. So in cases where you want your data to be stored in a particular format irrespective of the conversion schema, you can use these types, as shown below.

var list = new DynamoDBList();
list.Add(1);
list.Add("Sam");
list.Add(new HashSet<string> { "Design", "Logo" });

doc["Bool"] = DynamoDBBool.True;
doc["Null"] = DynamoDBNull.Null;
doc["List"] = list;

Using Conversion Schemas with the Object Persistence Model

The changes to the Object Persistence Model API are very similar to the Document Model changes:

  • With Conversion Schema V1, booleans are stored as N, Lists and HashSets are stored as sets (NS, SS, or BS)
  • With Conversion Schema V2, booleans are stored as BOOL, Lists are stored as L, HashSets are stored as sets (NS, SS, or BS)

Similarly to the Document Model, a conversion schema is associated with a DynamoDBContext and can be explicitly specified in code or the app.config, and will default to V1 if not set. The following example shows how to configure a context with V2 conversion schema.

var config = new DynamoDBContextConfig
{
    Conversion = DynamoDBEntryConversion.V2
};
var contextV2 = new DynamoDBContext(client, config);

Tomorrow, we will take a deeper look into the Object Persistence Model API and how the new DynamoDB types allow you to work with complex classes.

Caching the Rails Asset Pipeline with Amazon CloudFront

by Alex Wood | on | in Ruby | Permalink | Comments |  Share

Amazon CloudFront is a content delivery web service. It integrates with other Amazon Web Services to give developers and businesses an easy way to distribute content to end users with low latency, high data transfer speeds, and no minimum usage commitments.

Ruby on Rails introduced the asset pipeline in version 3.1. The Rails asset pipeline provides a framework to concatenate and minify or compress JavaScript and CSS assets. It also adds the ability to write these assets in other languages and pre-processors such as CoffeeScript, Sass and ERB.

With CloudFront’s support for custom origin servers, and features of the Rails asset pipeline, building a CDN for your static assets is simple. In this blog post, we will show you how to set this up for your environment.

Do You Have Geographically Diverse Users?

Amazon CloudFront provides 52 (as of when this was written*) edge locations around the world. Your static content can be cached by these edge locations to reduce the latency of your web application. Additionally, this can reduce the load on your app servers, as it limits the number of times your app server needs to serve large static files.

* See the current list of edge locations here.

Prerequisites

You should be able to deploy your Ruby on Rails application to the Internet, and you should know the hostname or IP address for where your application is hosted. If you have followed along with the series and deployed our sample application on AWS OpsWorks, you can complete this tutorial. If not, consider trying out a deployment first.

Creating a CloudFront Distribution

First, we will create a new CloudFront distribution that uses our app as the custom origin. From the Amazon CloudFront console, click Create Distribution. Under “Web”, click Get Started.

Within this form, call the Origin ID “Rails App Server”, and for the Origin Domain Name, we will point to the URL of our Rails application. Here is how:

  • If you have a domain name (e.g., “www.example.com”), then use that.
  • If not, you should use as stable of a hostname as possible. For example, the hostname of your ELB instance, or at least an Elastic IP. For demonstration purposes, the public host name of your app server instance will also work.

If you’re using something other than a domain name, don’t worry, you can change the origin address later if you need to. All other options can be left at their default values, though you can turn on logging if you want. We aren’t going to talk about using your own domain for the CDN just yet. Once you have your origin options set, click Create Distribution.

Configuring the Ruby on Rails App to Use CloudFront

Using CloudFront as the asset Host for your static assets is truly a one line change.

In config/environments/production.rb:

config.action_controller.asset_host = ENV['CLOUDFRONT_ENDPOINT']

This tells Rails to use your CloudFront endpoint as the hostname for static assets. Your endpoint hostname will be specified in a host environment variable.

To pick up that change if you’re following along at home, go in to the OpsWorks console and edit your app:

  • Under “Application Source”, point to the cloudfront branch.
  • Add a new environment variable pair:
    • Key: CLOUDFRONT_ENDPOINT
    • Value: The URL of your CloudFront endpoint, available in the CloudFront console. For e.g., “lettersandnumbers.cloudfront.net”
    • You do not need to “Protect” this value.

Now, deploy your app! You do not need to run a database migration.

How It Works

While we wait for the deployment to complete, how does all of this work?

If you look at the page source of our application before adding the CloudFront CDN, you’ll see lines like this:

<link data-turbolinks-track="true" href="/assets/application-0f3bf7fe135e88baa2cb9deb7a660251.css" media="all" rel="stylesheet" />
<script data-turbolinks-track="true" src="/assets/application-2ab5007aba477451ae5c38028892fd78.js"></script>

Those lines are how the page is including your application.css and application.js files. In app/views/layouts/application.html.erb, they correspond to these lines:

<%= stylesheet_link_tag 'application', media: 'all', 'data-turbolinks-track' => true %>
<%= javascript_include_tag 'application', 'data-turbolinks-track' => true %>

In turn, these include statements source from app/assets/stylesheets/application.css.scss and app/assets/javascripts/application.js. If you run the command rake assets:precompile, these files will be compiled and a fingerprint will be added to the filename. For example, I ran rake assets:precompile and the following files were generated:

  • public/assets/application-f3fd37796ac920546df412f68b0d9820.js
  • public/assets/application-68a6279b040bd09341327b6c951d74bc.css

The fingerprinting is a big part of what makes all of this work so smoothly. Let’s take a look at the page source after our latest deployment:

<link data-turbolinks-track="true" href="http://lettersandnumbers.cloudfront.net/assets/application-bfe54945dee8eb9f51b20d52b93aa177.css" media="all" rel="stylesheet" />
<script data-turbolinks-track="true" src="http://lettersandnumbers.cloudfront.net/assets/application-4984ddfbabfbae63ef17d0c8dca28d6c.js"></script>

You can see that we are now sourcing our static assets from CloudFront, and that nothing broke in the process. You can also see the compiled assets with fingerprints added to the filenames. When we loaded the page, the stylesheet_link_tag and javascript_include_tag used our asset host as the host, adding the expected asset filenames to the end of the hostname. When CloudFront received the request, these assets did not exist in the cache, so it forwarded the request to the Rails server, which served the files to CloudFront, which cached the files and sent them to you, the requestor. Future requests would simply hit the CDN, see the file present, and serve it to you from the fastest edge node.

Because fingerprinting is included out of the box, we do not need to deal with cache invalidations. When the assets change, the fingerprint will change. When that happens, CloudFront will not have the new file, and it will make a request to the origin server to get it. Eventually, the old, unused files will expire. It just works.

Wrap-Up

In this post, we took a Ruby on Rails application and cached its static assets using Amazon CloudFront and the Ruby on Rails asset pipeline. We also discussed the broad strokes of how CloudFront and Rails work together to make this simple to do.

Have any questions, comments, or problems getting your application to cache static content with Amazon CloudFront? Suggestions for topics you would like to see next? Please let us know in the comments!

DynamoDB Series – Document Model

by Norm Johanson | on | in .NET | Permalink | Comments |  Share

This week we are running a series of five daily blog posts that will explain new DynamoDB changes and how they relate to the .NET SDK. This is blog post number 2, and today we will be looking at the Document Model API.

Document Model

Yesterday, we learned about Amazon DynamoDB’s new data types such as lists and maps. Today, we are going to talk about how you can use the new data types in the Document Model API.

As a quick refresher on the Document Model API, it is an abstraction over the low-level service client and is found in the Amazon.DynamoDBv2.DocumentModel namespace. Here is an example for creating an item using the Document Model API.

var table = Table.LoadTable(ddbClient, tableName);
Document company = new Document();
company["id"] = "1";
company["name"] = "Dunder Mifflin";
company["industry"] = "paper";

table.PutItem(company);

The Document Model keeps things simple and easy. It takes care of all the data conversions and creating the underlying low-level request objects. Now if you want to take advantage of the new map data type, you can simply create a separate document object and assign it to the parent document. So in our example above, let’s give our paper company an address.

Document address = new Document();
address["street"] = "1725 Slough Avenue";
address["city"] = "Scranton";
address["state"] = "Pennsylvania";

Document company = new Document();
company["id"] = "1";
company["name"] = "Dunder Mifflin";
company["industry"] = "paper";
company["address"] = address;

table.PutItem(company);

To get the data back out, use the GetItem method from the table object passing in the key information for the item.

Document dunder = table.GetItem("1");

Now we can take advantage of the recursive capabilities of the list and map data types and add a list of employees to our branch office.

var address = dunder["address"].AsDocument();
address["employee"] = new List<string> { "Michael Scott", "Dwight Schrute", "Jim Halpert", "Pam Beesly" };
table.UpdateItem(dunder);

As I said, the Document Model translates its calls into the low-level service client. Here is how the Document Model translates the save of the Dunder Mifflin company into a call to the low-level service client.

var request = new PutItemRequest
{
    TableName = tableName,
    Item = new Dictionary<string, AttributeValue>
    {
        {"id", new AttributeValue{S = "1"}},
        {"name", new AttributeValue{S = "Dunder Mifflin"}},
        {"industry", new AttributeValue{S = "paper"}},
        {"address", new AttributeValue
            {M = new Dictionary<string, AttributeValue>
            {
                {"street", new AttributeValue{S = "1725 Slough Avenue"}},
                {"city", new AttributeValue{S = "Scranton"}},
                {"state", new AttributeValue{S = "Pennsylvania"}},
                {"employee", new AttributeValue
                {L = new List<AttributeValue>
                {
                    new AttributeValue{S = "Michael Scott"},
                    new AttributeValue{S = "Dwight Schrute"},
                    new AttributeValue{S = "Jim Halpert"},
                    new AttributeValue{S = "Pam Beesly"}
                }}
            }}
        }
    }}
};


ddbClient.PutItem(request);

Tomorrow, we are going to go deeper into how the Document Model handles converting types passed into a document into the underlying data types that DynamoDB supports.

DynamoDB Series Kickoff

by Pavel Safronov | on | in .NET | Permalink | Comments |  Share

Last week, Amazon DynamoDB added support for JSON document data structures. With this update, DynamoDB now supports nested data in the form of lists (L type) and maps (M type). Also part of this update was native support for booleans (BOOL type) and nulls (NULL type).

This week, we will be running a series of daily blog posts that will explain the new changes and how they relate to the AWS SDK for .NET, and we will see how you can take advantage of these new types to work with complex objects in all three .NET SDK DynamoDB APIs. In this, the first blog post of series, we will see how the low-level API has changed. In the following days, we will cover the Document Model, Conversion Schemas, Object Persistence Model, and finally Expressions.

New types

Until now, DynamoDB had only six data types:

  • Scalars N, S, and B that represent number, string, and binary data.
  • Sets NS, SS, and BS that represent number set, string set, and binary set.
    Sets have the limitation that the data they store has to be homogeneous (e.g., SS could only contain S elements) and unique (no two elements could be the same).

This release expands the possible data types with four new additions:

  • BOOL represents boolean data.
  • NULL represents null values.
  • L type represents a list of elements.
  • M type represents a string-to-element map.

The key point about L and M types is that they can contain any DynamoDB type. This allows you to create, for example, lists of maps of lists, which in turn can contain a mix of numbers, strings, bools, and nulls, or any other conceivable combination of attributes.

Low-level

The low-level API changes are straightforward: new DynamoDB types are now supported in all data calls. Here’s a sample that shows how both old and new types can be used in a PutItem call.

// Put item
client.PutItem("SampleTable", new Dictionary<string, AttributeValue>
{
    { "Id", new AttributeValue { N = "1" } },
    { "Product", new AttributeValue { S = "DataWriter" } },
    { "Aliases", new AttributeValue {
        SS = new List<string> { "Prod", "1.0" } } },
    { "IsPublic", new AttributeValue { BOOL = false } },
    { "Metadata", new AttributeValue {
        M = new Dictionary<string, AttributeValue>
        {
            { "InternalVersion", new AttributeValue { N = "1.2" } },
            { "Developers", new AttributeValue {
                SS = new List<string> { "Alan", "Franko" } } 
            },
            { "SampleInput", new AttributeValue {
                L = new List<AttributeValue>
                {
                    new AttributeValue { BOOL = true },
                    new AttributeValue { N =  "42" },
                    new AttributeValue { NULL = true },
                    new AttributeValue {
                        SS = new List<string> { "apple", "orange" } }
                } }
            }
        } }
    }
});

As you can see, the new M and L AttributeValue types may contain AttributeValues, allowing complex, nested data to be stored in a single DynamoDB record. In the above example, the item we just stored into DynamoDB will have an attribute of type M named "Metadata". This attribute will in turn contain three other attributes: N (number), SS (string set), and L (list). The list contains four more attributes, which in turn can be other M and L types, though in our example they are not.

Tomorrow, we will take a look at how the new additions can be used with the Document Model API.

Introducing DynamoDB Document API (Part 2)

by Hanson Char | on | in Java | Permalink | Comments |  Share

In the previous blog, Introducing DynamoDB Document API (Part 1), we saw how to program against the DynamoDB Document API and produce code that is both easy to write and read.  But why is the API called the Document API, and how are JSON-style documents supported?

This perhaps can best be explained, well, with code! Using the same Game table from the previous blog, let’s start with a game object directly represented by a JSON document:

    {
        "Status" : "IN_PROGRESS",
        "GameId" : "abc",
        "Player1-Position" : 15,
        "Player2-Position" : 12
    }

First, we put this game to Amazon DynamoDB as a structured document:

AmazonDynamoDBClient client = new AmazonDynamoDBClient(...);
DynamoDB dynamodb = new DynamoDB(client);
Table table = dynamo.getTable("Game");
    
String json = "{"
                    + ""Status" : "IN_PROGRESS","
                    + ""GameId" : "abc","
                    + ""Player1-Position" : 15,"
                    + ""Player2-Position" : 12"
                    + "}"
                    ;
Item jsonItem = Item.fromJSON(json);
table.putItem(jsonItem);

Suppose we need to update the game, changing the status to "SUSPENDED", and adding 1 to the first player’s position, but only if both players’ positions are less than 20 and if the current status is "IN_PROGRESS":

UpdateItemOutcome outcome = table.updateItem(new UpdateItemSpec()
            .withReturnValues(ReturnValue.ALL_NEW)
            .withPrimaryKey("GameId", "abc")
            .withAttributeUpdate(
                new AttributeUpdate("Player1-Position").addNumeric(1), 
                new AttributeUpdate("Status").put("SUSPENDED"))
            .withExpected(
                new Expected("Player1-Position").lt(20),
                new Expected("Player2-Position").lt(20),
                new Expected("Status").eq("IN_PROGRESS"))
        );

Finally, let’s get back the updated document as JSON:


Item itemUpdated = outcome.getItem();
String jsonUpdated = itemUpdated.toJSONPretty();
System.out.println(jsonUpdated);

Here is the output in JSON:

    {
      "Status" : "SUSPENDED",
      "GameId" : "abc",
      "Player1-Position" : 16,
      "Player2-Position" : 12
    }

As you can see, saving JSON as a structured document in Amazon DynamoDB, or updating, retrieving and converting the document back into JSON is as easy as 1-2-3. :)  You can find more examples in the A-Z Document API quick-start folder at GitHub. Happy coding until next time!