AWS Developer Tools Blog

DynamoDB Series – Conversion Schemas

This week, we are running a series of five daily blog posts that will explain new DynamoDB changes and how they relate to the AWS SDK for .NET. This is the third blog post, and today we will be discussing conversion schemas.

Conversion Schemas

Document doc = new Document();
doc["Id"] = 1;
doc["Product"] = "DataWriter";
doc["Aliases"] = new List<string> { "Prod", "1.0" };
doc["IsPublic"] = true;
table.UpdateItem(doc);

As you have seen earlier this week and in this example, it is very easy to work with a Document object and use .NET primitives. But how is this data actually stored in Amazon DynamoDB? The latest version of DynamoDB has added support for—among other things—native support for booleans (BOOL type) and lists of arbitrary elements (L type). (The various new DynamoDB types are covered in the first post in this series.) So in the above sample, are we taking advantage of these new types? To address this question and to provide a simple mechanism to control how your data is stored, we have introduced the concept of conversion schemas.

Why conversion schemas?

The new DynamoDB features are a powerful addition to the .NET SDK, but adding them to the Document Model API presented a challenge. The API was already capable of writing booleans and lists of items to DynamoDB, so how do the new BOOL and L types fit in? Should all booleans now be stored as BOOL (current implementation stores booleans as N types, either 1 or 0) and should lists be stored as L instead of SS/NS/BS? Changing how data is stored with the new SDK would break existing applications (older code would not be aware of the new types and query/scan conditions depend on the current types), so we have provided conversion schemas so that you can control exactly how your data is stored in DynamoDB. Schema V1 will maintain the current functionality, while Schema V2 will allow you to use the new types.

V1

The default conversion approach that the Document Model uses is as follows:

  • Number types (byte, int, float, decimal, etc.) are converted to N
  • String and char are converted to S
  • Bool is converted to N (0=false, 1=true)
  • DateTime and Guid are converted to S
  • MemoryStream and byte[] are converted to B
  • List, HashSet, and array of numerics types are converted to NS
  • List, HashSet, and array of string-based types are converted to SS
  • List, HashSet, and array of binary-based types are converted to BS

This conversion approach is known as Conversion Schema V1. It is the default conversion that the Document Model API uses and is identical in functionality to the conversion used by the SDK prior to the 2.3.2 release. As you can see, this schema does not take full advantage of the new DynamoDB types BOOL or L: the attribute Aliases will be stored as a string set (SS), while the boolean attribute IsPublic will be stored as a numeric (N).

But what if you wanted to now store a boolean as BOOL instead of N and List<string> as L instead of SS? The simple way to do this would be to use Conversion Schema V2.

V2

Conversion Schema V2 differs from V1 in that boolean values are stored as BOOL types, Lists are stored as L, and HashSets are stored as sets (NS, SS, or BS, depending on the data). So if you use V2 schema to store the Document in our example, it will function identically from the perspective of the application, but the data stored in DynamoDB will be different. The V2 schema differs from V1 in the following ways:

  • Boolean values will be stored as BOOL instead of N.
  • Lists and arrays of numerics, string-based types, and binary-based types are converted to L type.
  • HashSets of numerics, string-based types, and binary-based types are converted to NS, SS, or BS, as appropriate.
  • Other types are not impacted.

Note that Conversion Schema V2 differs in how it stores List<T> vs. HashSet<T>: List<T> is stored as a DynamoDB List (L type), while HashSet<T> is stored as a DynamoDB Set (NS, SS, or BS type). So if we wanted to use schema V2 but keep the Tags attributed as a set, we could update the code in our example to use HashSet<string> instead of List<string>.

Using Conversion Schemas with the Document Model

Conversion schemas are set for a particular Table object. (This means that the same Document stored using different Table objects may result in different data being written to DynamoDB.) The following sample shows how to load two Table objects, one configured with schema V1 and the other with schema V2, using two different LoadTable approaches.

Table tableV1 = LoadTable(client, "SampleTable", DynamoDBEntryConversion.V1);
Table tableV2;
TryLoadTable(client, "SampleTable", DynamoDBEntryConversion.V2, out tableV2);

You may also load a Table object without specifying a conversion, in which case the Table will use either the default V1 conversion or the conversion that you specify in your app.config file, as shown in the following sample.

<configuration>
  <aws>
    <dynamoDB conversionSchema="V2" />
  </aws>
</configuration>

Avoiding Conversion

Conversion schemas are used to convert between .NET types and DynamoDB types. However, if you use classes that extend DynamoDBEntry (such as Primitive, DynamoDBBool or DynamoDBNull), conversion will not performed. So in cases where you want your data to be stored in a particular format irrespective of the conversion schema, you can use these types, as shown below.

var list = new DynamoDBList();
list.Add(1);
list.Add("Sam");
list.Add(new HashSet<string> { "Design", "Logo" });

doc["Bool"] = DynamoDBBool.True;
doc["Null"] = DynamoDBNull.Null;
doc["List"] = list;

Using Conversion Schemas with the Object Persistence Model

The changes to the Object Persistence Model API are very similar to the Document Model changes:

  • With Conversion Schema V1, booleans are stored as N, Lists and HashSets are stored as sets (NS, SS, or BS)
  • With Conversion Schema V2, booleans are stored as BOOL, Lists are stored as L, HashSets are stored as sets (NS, SS, or BS)

Similarly to the Document Model, a conversion schema is associated with a DynamoDBContext and can be explicitly specified in code or the app.config, and will default to V1 if not set. The following example shows how to configure a context with V2 conversion schema.

var config = new DynamoDBContextConfig
{
    Conversion = DynamoDBEntryConversion.V2
};
var contextV2 = new DynamoDBContext(client, config);

Tomorrow, we will take a deeper look into the Object Persistence Model API and how the new DynamoDB types allow you to work with complex classes.