Tag: aws-sdk-go


Introducing Amazon DynamoDB Expression Builder in the AWS SDK for Go

This post was authored by Hajime Hayano.

The v1.11.0 release of the AWS SDK for Go adds a new expression package that enables you to create Amazon DynamoDB Expressions using statically typed builders. The expression package abstracts away the low-level detail of using DynamoDB Expressions and simplifies the process of using DynamoDB Expressions in DynamoDB Operations. In this blog post, we explain how to use the expression package.

In earlier versions of the AWS SDK for Go, you had to explicitly declare the member fields of the DynamoDB Operation input structs, such as QueryInput and UpdateItemInput. That meant the syntax and rules of DynamoDB Expressions were up to you to figure out. The goal of the expression package is to create the formatted DynamoDB Expression strings under the hood thus simplifying the process of using DynamoDB Expressions. The following example shows the verbosity of writing DynamoDB Expressions by hand.

input := &dynamodb.ScanInput{
    ExpressionAttributeNames: map[string]*string{
        "#AT": aws.String("AlbumTitle"),
        "#ST": aws.String("SongTitle"),
    },
    ExpressionAttributeValues: map[string]*dynamodb.AttributeValue{
        ":a": {
            S: aws.String("No One You Know"),
        },
    },
    FilterExpression:     aws.String("Artist = :a"),
    ProjectionExpression: aws.String("#ST, #AT"),
    TableName:            aws.String("Music"),
}

Representing DynamoDB Expressions

DynamoDB Expressions are represented by static builder types in the expression package. These builders, like ConditionBuilder and UpdateBuilder, are created in the package using a builder pattern. The static typing of the builders allows compile-time checks on the syntax of the DynamoDB Expressions that are being created. The following example shows how to create a builder that represents a FilterExpression and a ProjectionExpression.

filt := expression.Name("Artist").Equal(expression.Value("No One You Know"))
// let :a be an ExpressionAttributeValue representing the string "No One You Know"
// equivalent FilterExpression: "Artist = :a"

proj := expression.NamesList(
    expression.Name("SongTitle"),
    expression.Name("AlbumTitle"),
)
// equivalent ProjectionExpression: "SongTitle, AlbumTitle"

In this example, the variable filt represents a FilterExpression. Notice that DynamoDB item attributes are represented using the function Name() and DynamoDB item values are similarly represented using the function Value(). In this context, the string "Artist" represents the name of the item attribute that we want to evaluate, and the string "No One You Know" represents the value we want to evaluate the item attribute against. You specify the relationship between the two operands by using the method Equal().

Similarly, the variable proj represents a ProjectionExpression. The list of item attribute names comprising the ProjectionExpression are specified as arguments to the function NamesList(). The expression package uses the type safety of Go and, if an item value is to be used as an argument to the function NamesList(), a compile time error is returned. The pattern of representing DynamoDB Expressions by indicating relationships between operands with functions is consistent throughout the whole expression package.

Creating an Expression

The Expression type is the core of the expression package. An Expression represents a collection of DynamoDB Expressions with getter methods, such as Condition() and Projection(), used to retrieve specific formatted DynamoDB Expression strings. The following example shows how to create an Expression.

filt := expression.Name("Artist").Equal(expression.Value("No One You Know"))
proj := expression.NamesList(
    expression.Name("SongTitle"),
    expression.Name("AlbumTitle"),
)

expr, err := expression.NewBuilder().
    WithFilter(filt).
    WithProjection(proj).
    Build()
if err != nil {
  fmt.Println(err)
}

In this example, the variable expr is an instance of an Expression type. An Expression is built using a builder pattern. First, a new Builder is initialized by the NewBuilder() function. Then, types representing DynamoDB Expressions are added to the Builder by the WithFilter() and WithProjection() methods. The Build() method returns an instance of an Expression and an error. The error is either an InvalidParameterError or an UnsetParameterError.

There is no limit to the number of different kinds of DynamoDB Expressions that you can add to the Builder, but adding the same type of DynamoDB Expression will overwrite the previous DynamoDB Expression. The following example shows a specific instance of this problem.

cond1 := expression.Name("foo").Equal(expression.Value(5))
cond2 := expression.Name("bar").Equal(expression.Value(6))
expr, err := expression.NewBuilder().
    WithCondition(cond1).
    WithCondition(cond2).
    Build()
if err != nil {
  fmt.Println(err)
}

This example shows that the second call of WithCondition() overwrites the first call.

Filling in the fields of a DynamoDB Scan API

The following example shows how to use an Expression to fill in the member fields of a DynamoDB Operation API.

filt := expression.Name("Artist").Equal(expression.Value("No One You Know"))
proj := expression.NamesList(
    expression.Name("SongTitle"),
    expression.Name("AlbumTitle"),
)
expr, err := expression.NewBuilder().
    WithFilter(filt).
    WithProjection(proj).
    Build()
if err != nil {
  fmt.Println(err)
}

input := &dynamodb.ScanInput{
  ExpressionAttributeNames:  expr.Names(),
  ExpressionAttributeValues: expr.Values(),
  FilterExpression:          expr.Filter(),
  ProjectionExpression:      expr.Projection(),
  TableName:                 aws.String("Music"),
}

In this example, the getter methods of the Expression type are used to get the formatted DynamoDB Expression strings. When using Expression, you must always assign the ExpressionAttributeNames and ExpressionAttributeValues member fields of the DynamoDB API because all item attribute names and values are aliased. That means that if the ExpressionAttributeNames and ExpressionAttributeValues members are not assigned with the corresponding Names() and Values() methods, the DynamoDB operation will run into a logic error.

If you need a starting point, check out the working example in the AWS SDK for Go.

Overall, the expression package makes using the DynamoDB Expressions clean and simple. The complicated syntax and rules of DynamoDB Expressions are abstracted away so you no longer have to worry about them!

Using Go 1.8’s Plugin for Credentials with the AWS SDK for Go

The v1.10.0 release of the AWS SDK for Go adds a new way to configure the SDK to retrieve AWS credentials. With this release, you can configure the SDK to retrieve AWS credentials from a Go plugin that is dynamically loaded during your application’s runtime. In this post, we explain how you can build a plugin and configure the SDK to use it. The SDK also includes a runnable example for you to try out the new plugin credential provider feature.

The SDK does takes advantage of the Go 1.8 plugin package, and associated build mode for Linux operating systems. The plugin package and associated build mode enable you to write components that can be loaded dynamically while your application runs. Plugins help you add functionality to your application while it’s running instead of only when the application is compiled.

The SDK’s plugincreds package enables you to use the plugins to retrieve AWS credentials. This package includes utilities to create a credentials Provider and Credentials loader.

Building a credential provider plugin

To use a plugin with the SDK, the SDK requires the plugin to export a function that returns two function pointers. The SDK uses these two returned function pointers to retrieve credentials and to determine if the credentials are expired. By default, the SDK expects the plugin to export the symbol named GetAWSSDKCredentialProvider for the getter function that returns the retrieve and isExpired function pointers.

The SDK requires the plugin’s getter function signature to match the following signature. If the getter function doesn’t match the signature, the SDK returns an error with the code ErrCodeInvalidSymbolError.

func() (RetrieveFn func() (key, secret, token string, err error), IsExpiredFn func() bool)

The SDK includes the NewCredentials helper function that looks up and validates the symbol, creating the SDK’s Credentials value automatically. You can use the returned Credentials value to configure a session or service client.

To use a custom symbol name, use the GetPluginProviderFnsByName function to look up the getter function from the plugin by name. This verifies that the symbol matches the expected signature. It also gets the credential provider’s retrieve and isExpired function pointers by calling the getter function. The retrieve and isExpired function pointers are returned. The SDK requires both function pointers to be valid and not nil.

Here is an example of a plugin that provides credential retrieve and expired functions to the application that loaded the plugin.

package main

func main() {}

// Build: go build -o plugin.so -buildmode=plugin plugin.go
func init() {
	// Initialize a mock credential provider with mock values. In a real-world usage
	// the provider's Retrieve method could reach out to the source of credentials
	// and return the credentials there, instead of this mock credential provider that statically
	// sets the credential values.
	myCredProvider = provider{"key","secret","token"}
}

// GetAWSSDKCredentialProvider is the symbol the SDK will look up and use to
// get the credential provider's retrieve and isExpired functions.
func GetAWSSDKCredentialProvider() (func() (key, secret, token string, err error), func() bool) {
	return myCredProvider.Retrieve,	myCredProvider.IsExpired
}

// Mock implementation of a type that retrieves credentials and
// returns if they are expired.
type provider struct {
	key, secret, token string
}

// Return the credentials that were previously set into the provider value.
func (p provider) Retrieve() (key, secret, token string, err error) {
	return p.key, p.secret, p.token, nil
}

func (p *provider) IsExpired() bool {
	return false;
}

Once you’ve written the code for your plugin, you can build it as a plugin file that can be loaded dynamically into your application with the -buildmode=plugin build flag.

go build -o myCredPlugin.so -buildmode=plugin plugin.go

You can find an example you can start from in the SDK’s plugincreds example.

Using a credential provider plugin

Once you’ve built your plugin, you can configure the SDK to retrieve credentials using it. The SDK makes this easy with the plugincreds package’s NewCredential function. This function takes a Plugin pointer value and looks up the expected credentials provider getter functions. See the plugincreds package for errors that can be returned.

The following example shows you how an application can open a Go plugin dynamically at runtime, and configure the SDK to use the plugin to retrieve AWS credentials.

// In your application code, open the plugin using its file name. This loads
// the plugin into memory, executing the plugin's main package init function.
p, err := plugin.Open(pluginFilename)
if err != nil {
	return nil, errors.Wrapf(err, "failed to open plugin, %s", pluginFilename)
}

// NewCredentials looks up the symbol from the plugin and configures the Credentials
// value that can be used to configure a session or service client.
//
// You can share the Credentials value and credentials, across many session and service clients 
// concurrently and safely.
creds, err := plugincreds.NewCredentials(p)
if err != nil {
	return nil, errors.Wrapf(err, "failed to load plugin credentials provider, %s", pluginFilename)
}

// Configure a session to use the credentials sourced from the plugin that is loaded.
sess := session.Must(session.NewSession(&aws.Config{
	Credentials: creds,
}))

// Return the configured session so it can be used to create service clients.
return sess, nil

You can find a usable example of this in the SDK’s plugincreds example.

Putting it all together

With this configuration, you can deploy your plugin and application independently to the platforms that your application will run on. Loading plugins dynamically allows you to separate your application from where your AWS credentials are retrieved. This practice allows your application to be more flexible when working with multiple environments. This technique is particularly useful for CLI applications where users of the CLI need to provide custom ways of retrieving credentials.

Let us know how you use the credentials plugin in your applications.

Context Pattern added to the AWS SDK for Go

The AWS SDK for Go v1.8.0 release adds support for the API operation request functional options, and the Context pattern. Both of these features were high demand requests from our users. Request options allow you to easily configure and augment how the SDK makes API operation requests to AWS services. The SDK’s support for the Context pattern allows your application take advantage of cancellation, timeouts, and Context Values on requests.  The new request options and Context pattern give your application even more control over SDK’s request execution and handling.

Request Options

Request Options are functional arguments that you pass in to the SDK’s API operation methods. These enable you to configure the request in line with functional options. Functional options are a pattern you can use to configure an operation via passed-in functions or closures in line with the method call.

For example, you can configure the Amazon S3 API operation PutObject to log debug information about the request directly, without impacting the other API operations used by your application.

// Log this API operation only. 
resp, err := svc.PutObjectWithContext(ctx, params, request.WithLogLevel(aws.LogDebug))

This pattern is also helpful when you want your application to inject request handlers into the request. This allows you to do so in line with the API operation method call.

resp, err := svc.PutObjectWithContext(ctx, params, func(r *request.Request) {
	start := time.Now()
	r.Handlers.Complete.PushBack(func(req *request.Request) {
		fmt.Println("request %s took %s to complete", req.RequestID, time.Since(start))
	})
})

All of the SDK’s new service client methods that have a WithContext suffix support these request options. You can also apply request options to the SDK’s standard Request directly with the ApplyOptions method.

API Operations with Context

All of the new methods of the SDK’s API operations that have a WithContext suffix take a ContextValue. This value must be non-nil. Context allows your application to control API operation request cancellation. This means you can now easily institute request timeouts based on the Context pattern. Go introduced the Context pattern in the experimental package golang.org/x/net/context, and it was later added to the Go standard library in Go 1.7. For backward compatibility with previous Go versions, the SDK created the Context interface type in the github.com/aws/aws-sdk-go/aws package. The SDK’s Context type is compatible with Context from both golang.org/x/net/context and the Go 1.7 standard library Context package.

Here is an example of how to use a Context to cancel uploading an object to Amazon S3. If the put doesn’t complete within the timeout passed in, the API operation is canceled. When a Context is canceled, the SDK returns the CanceledErrorCode error code. A working version of this example can be found in the SDK.

sess := session.Must(session.NewSession())
svc := s3.New(sess)

// Create a context with a timeout that will abort the upload if it takes 
// more than the passed in timeout.
ctx := context.Background()
var cancelFn func()
if timeout > 0 {
	ctx, cancelFn = context.WithTimeout(ctx, timeout)
}
// Ensure the context is canceled to prevent leaking.
// See context package for more information, https://golang.org/pkg/context/
defer cancelFn()

// Uploads the object to S3. The Context will interrupt the request if the 
// timeout expires.
_, err := svc.PutObjectWithContext(ctx, &s3.PutObjectInput{
	Bucket: aws.String(bucket),
	Key:    aws.String(key),
	Body:   body,
})
if err != nil {
	if aerr, ok := err.(awserr.Error); ok && aerr.Code() == request.CanceledErrorCode {
		// If the SDK can determine the request or retry delay was canceled
		// by a context the CanceledErrorCode error code will be returned.
		fmt.Println("request's context canceled,", err)
	}
	return err
}

API Operation Waiters

Waiters were expanded to include support for request Context and waiter options. The new WaiterOption type defines functional options that are used to configure the waiter’s functionality.

For example, the WithWaiterDelay allows you to provide your own function that returns how long the waiter will wait before checking the waiter’s resource state again. This is helpful when you want to configure an exponential backoff, or longer retry delays with ConstantWaiterDelay.

The example below highlights this by configuring the WaitUntilBucketExists method to use a 30-second delay between checks to determine if the bucket exists.

svc := s3.New(sess)
ctx := contex.Background()

_, err := svc.CreateBucketWithContext(ctx, &s3.CreateBucketInput{
	Bucket: aws.String("myBucket"),
})
if err != nil {
	return fmt.Errorf("failed to create bucket, %v", err)
}

err := svc.WaitUntilBucketExistsWithContext(ctx,
	&s3.HeadBucket{
		Bucket: aws.String("myBucket"),
	},
	request.WithWaiterDelay(request.ConstantWaiterDelay(30 * time.Second)),
)
if err != nil {
	return fmt.Errorf("failed to wait for bucket exists, %v", err)
}

fmt.Println("bucket created")

API Operation Paginators

Paginators were also expanded to add support for Context and request options. Configuring request options for pagination applies the options to each new Request that the SDK creates to retrieve the next page. By extending the Pages API methods to include Context and request options the SDK gives you control over how the SDK will make each page request, and cancellation of the pagination.

svc := s3.New(sess)
ctx := context.Background()

err := svc.ListObjectsPagesWithContext(ctx,
	&s3.ListObjectsInput{
		Bucket: aws.String("myBucket"),
		Prefix: aws.String("some/key/prefix"),
		MaxKeys: aws.Int64(100),
	},
	func(page *s3.ListObjectsOutput, lastPage bool) bool {
		fmt.Println("Received", len(page.Contents), "objects in page")
		for _, obj := range page.Contents {
			fmt.Println("Key:", aws.StringValue(obj.Key))
		}
		return true
	},
)
if err != nil {
	return fmt.Errorf("failed to create bucket, %v", err)
}

API Operation Pagination without Callbacks

In addition to the Pages API operations, you can use the new Pagination type in the github.com/aws/aws-sdk-go/aws/request package. This type enables you to control the iterations of pages directly. This is helpful when you do not want to use callbacks for paginating AWS operations. This new type allows you to treat pagination similar to the Go stdlib bufio package’s Scanner type to iterate through pages with a for loop. You can also use this pattern with the Context pattern by calling Request.SetContext on each request in the NewRequest function.

svc := s3.New(sess)

params := s3.ListObjectsInput{
	Bucket: aws.String("myBucket"),
	Prefix: aws.String("some/key/prefix"),
	MaxKeys: aws.Int64(100),
}
ctx := context.Background()

p := request.Pagination{
	NewRequest: func() (*request.Request, error) {
		req, _ := svc.ListObjectsRequest(&params)
		req.SetContext(ctx)
		return req, nil
	},
}

for p.Next(){
	page := p.Page().(*s3.ListObjectsOutput)
	
	fmt.Println("Received", len(page.Contents), "objects in page")
	for _, obj := range page.Contents {
		fmt.Println("Key:", aws.StringValue(obj.Key))
	}
}

return p.Err()

Wrap Up

The addition of Context and request options expands the capabilities of the AWS SDK for Go, giving your applications the tools needed to implement request lifecycle and configuration with the SDK. Let us know your experiences using the new Context pattern and request options features.