Tag: golang


AWS SDK for Go 2.0 Developer Preview

We’re pleased to announce the Developer Preview release of the AWS SDK for Go 2.0. Many aspects of the SDK have been refactored based on your feedback, with a strong focus on performance, consistency, discoverability, and ease of use. The Developer Preview is here for you to provide feedback, and influence the direction of the AWS SDK for Go 2.0 before its production-ready, general availability launch. Tell us what you like, and what you don’t like. Your feedback matters to us. Find details at the bottom of this post on how to give feedback and contribute.

You can safely use the AWS SDK for Go 2.0 in parallel with the 1.x SDK, with both SDKs coexisting in the same Go application. We won’t drop support for the 1.0 SDK any time soon. We know there are a lot of customers depending on the 1.x SDK, and we will continue to support them. As we get closer to general availability for 2.0, we’ll share a more detailed plan about how we’ll support the 1.x SDK.

Getting started

Let’s walk through setting up an application with the 2.0 SDK. We’ll build a simple Go application using the 2.0 SDK to make a service API request. For this walkthrough, you’ll need to use a minimum of Go 1.9.

  1. Create a new Go file for the example application. We’ll name ours main.go in a new directory, awssdkgov2-example, in our Go Workspace.

    mkdir -p $(go env GOPATH)/src/awssdkgov2-example
    cd $(go env GOPATH)/src/awssdkgov2-example
    

    All of the following steps assume you will execute them from the awssdkgov2-example directory.

  2. To get the AWS SDK for Go 2.0, you can go get the SDK manually or use a dependency management tool such as Dep. You can manually get the 2.0 SDK with go get github.com/aws/aws-sdk-go-v2.

  3. In your favorite editor create a main.go file, and add the following code. This code loads the SDK’s default configuration, and creates a new Amazon DynamoDB client. See the external package for more details on how to customize the way the SDK’s default configuration is loaded.

    package main
    
    import (
        "github.com/aws/aws-sdk-go-v2/aws"
        "github.com/aws/aws-sdk-go-v2/aws/endpoints"
        "github.com/aws/aws-sdk-go-v2/aws/external"
        "github.com/aws/aws-sdk-go-v2/service/dynamodb"
    )
    
    func main() {
        // Using the SDK's default configuration, loading additional config
        // and credentials values from the environment variables, shared
        // credentials, and shared configuration files
        cfg, err := external.LoadDefaultAWSConfig()
        if err != nil {
            panic("unable to load SDK config, " + err.Error())
        }
    
        // Set the AWS Region that the service clients should use
        cfg.Region = endpoints.UsWest2RegionID
    
        // Using the Config value, create the DynamoDB client
        svc := dynamodb.New(cfg)
    }
    
  4. Build the service API request, send it, and process the response.

    // Build the request with its input parameters
    req := svc.DescribeTableRequest(&dynamodb.DescribeTableInput{
        TableName: aws.String("myTable"),
    })
    
    // Send the request, and get the response or error back
    resp, err := req.Send()
    if err != nil {
        panic("failed to describe table, "+err.Error())
    }
    
    fmt.Println("Response", resp)
    

What has changed?

Our focus for the 2.0 SDK is to improve the SDK’s development experience and performance, make the SDK easy to extend, and add new features. The changes made in the Developer Preview target the major pain points of configuring the SDK and using AWS service API calls. Check out our 2.0 repo for details on pending changes that are in development and designs we’re discussing.

The following are some of the larger changes included in the AWS SDK for Go 2.0 Developer Preview.

SDK configuration

The 2.0 SDK simplifies how you configure the SDK’s service clients by using a single Config type. This simplifies the Session and Config type interaction from the 1.x SDK. In addition, we’ve moved the service-specific configuration flags to the individual service client types. This reduces confusion about where service clients will use configuration members.

We added the external package to provide the utilities for you to use external configuration sources to populate the SDK’s Config type. External sources include environmental variables, shared credentials file (~/.aws/credentials), and shared config file (~/.aws/config). By default, the 2.0 SDK will now automatically load configuration values from the shared config file. The external package also provides you with the tools to customize how the external sources are loaded and used to populate the Config type.

You can even customize external configuration sources to include your own custom sources, for example, JSON files or a custom file format.

Use LoadDefaultAWSConfig in the external package to create the default Config value, and load configuration values from the external configuration sources.

cfg, err := external.LoadDefaultAWSConfig()

To specify the shared configuration profile load used in code, use the WithSharedConfigProfile helper passed into LoadDefaultAWSConfig with the profile name to use.

cfg, err := external.LoadDefaultAWSConfig(
	external.WithSharedConfigProfile("gopher")
)

Once a Config value is returned by LoadDefaultAWSConfig, you can set or override configuration values by setting the fields on the Config struct, such as Region.

cfg.Region = endpoints.UsWest2RegionID

Use the cfg value to provide the loaded configuration to new AWS service clients.

svc := dynamodb.New(cfg)

API request methods

The 2.0 SDK consolidates several ways to make an API call, providing a single request constructor for each API call. This means that your application will create an API request from input parameters, then send it. This enables you to optionally modify and configure how the request will be sent. This includes, but isn’t limited to, modifications such as setting the Context per request, adding request handlers, and enabling logging.

Each API request method is suffixed with Request and returns a typed value for the specific API request.

As an example, to use the Amazon Simple Storage Service GetObject API, the signature of the method is:

func (c *S3) GetObjectRequest(*s3.GetObjectInput) *s3.GetObjectRequest

To use the GetObject API, we pass in the input parameters to the method, just like we would with the 1.x SDK. The 2.0 SDK’s method will initialize a GetObjectRequest value that we can then use to send our request to Amazon S3.

req := svc.GetObjectRequest(params)

// Optionally, set the context or other configuration for the request to use
req.SetContext(ctx)

// Send the request and get the response
resp, err := req.Send()

API enumerations

The 2.0 SDK uses typed enumeration values for all API enumeration fields. This change provides you with the type safety that you and your IDE can leverage to discover which enumeration values are valid for particular fields. Typed enumeration values also provide a stronger type safety story for your application than using string literals directly. The 2.0 SDK uses string aliases for each enumeration type. The SDK also generates typed constants for each enumeration value. This change removes the need for enumeration API fields to be pointers, as a zero-value enumeration always means the field isn’t set.

For example, the Amazon Simple Storage Service PutObject API has a field, ACL ObjectCannedACL. An ObjectCannedACL string alias is defined within the s3 package, and each enumeration value is also defined as a typed constant. In this example, we want to use the typed enumeration values to set an uploaded object to have an ACL of public-read. The constant that the SDK provides for this enumeration value is ObjectCannedACLPublicRead.

svc.PutObjectRequest(&s3.PutObjectInput{
	Bucket: aws.String("myBucket"),
	Key:    aws.String("myKey"),
	ACL:    s3.ObjectCannedACLPublicRead,
	Body:   body,
})

API slice and map elements

The 2.0 SDK removes the need to convert slice and map elements from values to pointers for API calls. This will reduce the overhead of needing to use fields with a type, such as []string, in API calls. The 1.x SDK’s pattern of using pointer types for all slice and map elements was a significant pain point for users, requiring them to convert between the types. The 2.0 SDK does away with the pointer types for slices and maps, using value types instead.

The following example shows how value types for the Amazon Simple Queue Service AddPermission API’s AWSAccountIds and Actions member slices are set.

svc := sqs.New(cfg)

svc.AddPermission(&sqs.AddPermissionInput{
	AWSAcountIds: []string{
		"123456789",
	},
	Actions: []string{
		"SendMessage",
	},

	Label:    aws.String("MessageSender"),
	QueueUrl: aws.String(queueURL)
})

SDK’s use of pointers

We listened to your feedback on the SDK’s significant use of pointers across the surface of the SDK. The Config type was refactored to remove the use of pointers for both Config value and its member fields. Enumeration pointers were removed, and replaced with typed constants. Slice and map elements were also updated to be value types instead of pointers.

The API parameter fields are still pointer types in the 2.0 SDK. We looked at patterns to remove these pointers, changing them to values, but chose not to make a change to the field types. All of the patterns we investigated either didn’t solve the problem fully or imposed an increased risk of unintended API call outcome. For example, unknowingly making an API call with a field type’s zero value required parameters. The 2.0 SDK does include setter’s and utility functions for all API parameter types to provide a workaround to using pointers directly.

Please share your thoughts in GitHub issues or our Gitter channel. We want to hear your feedback.

Giving feedback and contributing

The 2.0 SDK will use GitHub issues to track feature requests and issues with the 2.0 repo. In addition, we’ll use GitHub projects to track large tasks spanning multiple pull requests, such as refactoring the SDK’s internal request lifecycle. You can provide feedback to us in several ways.

GitHub issues. To provide feedback using GitHub issues on the 2.0 repo. This is the preferred mechanism to give feedback so that other users can engage in the conversation, +1 issues, etc. Issues you open will be evaluated, and included in our roadmap for the GA launch.

Gitter channel. For more informal discussions or general feedback, check out our Gitter channel for the 2.0 repo. The Gitter channel is also a great place to ask general questions, and find help to get started with the 2.0 SDK Developer Preview.

Contributing. You can open pull requests for fixes or additions to the AWS SDK for Go 2.0 Developer Preview release. All pull requests must be submitted under the Apache 2.0 license and will be reviewed by an SDK team member before being merged in. Accompanying unit tests, where possible, are appreciated.

Context Pattern added to the AWS SDK for Go

The AWS SDK for Go v1.8.0 release adds support for the API operation request functional options, and the Context pattern. Both of these features were high demand requests from our users. Request options allow you to easily configure and augment how the SDK makes API operation requests to AWS services. The SDK’s support for the Context pattern allows your application take advantage of cancellation, timeouts, and Context Values on requests.  The new request options and Context pattern give your application even more control over SDK’s request execution and handling.

Request Options

Request Options are functional arguments that you pass in to the SDK’s API operation methods. These enable you to configure the request in line with functional options. Functional options are a pattern you can use to configure an operation via passed-in functions or closures in line with the method call.

For example, you can configure the Amazon S3 API operation PutObject to log debug information about the request directly, without impacting the other API operations used by your application.

// Log this API operation only. 
resp, err := svc.PutObjectWithContext(ctx, params, request.WithLogLevel(aws.LogDebug))

This pattern is also helpful when you want your application to inject request handlers into the request. This allows you to do so in line with the API operation method call.

resp, err := svc.PutObjectWithContext(ctx, params, func(r *request.Request) {
	start := time.Now()
	r.Handlers.Complete.PushBack(func(req *request.Request) {
		fmt.Println("request %s took %s to complete", req.RequestID, time.Since(start))
	})
})

All of the SDK’s new service client methods that have a WithContext suffix support these request options. You can also apply request options to the SDK’s standard Request directly with the ApplyOptions method.

API Operations with Context

All of the new methods of the SDK’s API operations that have a WithContext suffix take a ContextValue. This value must be non-nil. Context allows your application to control API operation request cancellation. This means you can now easily institute request timeouts based on the Context pattern. Go introduced the Context pattern in the experimental package golang.org/x/net/context, and it was later added to the Go standard library in Go 1.7. For backward compatibility with previous Go versions, the SDK created the Context interface type in the github.com/aws/aws-sdk-go/aws package. The SDK’s Context type is compatible with Context from both golang.org/x/net/context and the Go 1.7 standard library Context package.

Here is an example of how to use a Context to cancel uploading an object to Amazon S3. If the put doesn’t complete within the timeout passed in, the API operation is canceled. When a Context is canceled, the SDK returns the CanceledErrorCode error code. A working version of this example can be found in the SDK.

sess := session.Must(session.NewSession())
svc := s3.New(sess)

// Create a context with a timeout that will abort the upload if it takes 
// more than the passed in timeout.
ctx := context.Background()
var cancelFn func()
if timeout > 0 {
	ctx, cancelFn = context.WithTimeout(ctx, timeout)
}
// Ensure the context is canceled to prevent leaking.
// See context package for more information, https://golang.org/pkg/context/
defer cancelFn()

// Uploads the object to S3. The Context will interrupt the request if the 
// timeout expires.
_, err := svc.PutObjectWithContext(ctx, &s3.PutObjectInput{
	Bucket: aws.String(bucket),
	Key:    aws.String(key),
	Body:   body,
})
if err != nil {
	if aerr, ok := err.(awserr.Error); ok && aerr.Code() == request.CanceledErrorCode {
		// If the SDK can determine the request or retry delay was canceled
		// by a context the CanceledErrorCode error code will be returned.
		fmt.Println("request's context canceled,", err)
	}
	return err
}

API Operation Waiters

Waiters were expanded to include support for request Context and waiter options. The new WaiterOption type defines functional options that are used to configure the waiter’s functionality.

For example, the WithWaiterDelay allows you to provide your own function that returns how long the waiter will wait before checking the waiter’s resource state again. This is helpful when you want to configure an exponential backoff, or longer retry delays with ConstantWaiterDelay.

The example below highlights this by configuring the WaitUntilBucketExists method to use a 30-second delay between checks to determine if the bucket exists.

svc := s3.New(sess)
ctx := contex.Background()

_, err := svc.CreateBucketWithContext(ctx, &s3.CreateBucketInput{
	Bucket: aws.String("myBucket"),
})
if err != nil {
	return fmt.Errorf("failed to create bucket, %v", err)
}

err := svc.WaitUntilBucketExistsWithContext(ctx,
	&s3.HeadBucket{
		Bucket: aws.String("myBucket"),
	},
	request.WithWaiterDelay(request.ConstantWaiterDelay(30 * time.Second)),
)
if err != nil {
	return fmt.Errorf("failed to wait for bucket exists, %v", err)
}

fmt.Println("bucket created")

API Operation Paginators

Paginators were also expanded to add support for Context and request options. Configuring request options for pagination applies the options to each new Request that the SDK creates to retrieve the next page. By extending the Pages API methods to include Context and request options the SDK gives you control over how the SDK will make each page request, and cancellation of the pagination.

svc := s3.New(sess)
ctx := context.Background()

err := svc.ListObjectsPagesWithContext(ctx,
	&s3.ListObjectsInput{
		Bucket: aws.String("myBucket"),
		Prefix: aws.String("some/key/prefix"),
		MaxKeys: aws.Int64(100),
	},
	func(page *s3.ListObjectsOutput, lastPage bool) bool {
		fmt.Println("Received", len(page.Contents), "objects in page")
		for _, obj := range page.Contents {
			fmt.Println("Key:", aws.StringValue(obj.Key))
		}
		return true
	},
)
if err != nil {
	return fmt.Errorf("failed to create bucket, %v", err)
}

API Operation Pagination without Callbacks

In addition to the Pages API operations, you can use the new Pagination type in the github.com/aws/aws-sdk-go/aws/request package. This type enables you to control the iterations of pages directly. This is helpful when you do not want to use callbacks for paginating AWS operations. This new type allows you to treat pagination similar to the Go stdlib bufio package’s Scanner type to iterate through pages with a for loop. You can also use this pattern with the Context pattern by calling Request.SetContext on each request in the NewRequest function.

svc := s3.New(sess)

params := s3.ListObjectsInput{
	Bucket: aws.String("myBucket"),
	Prefix: aws.String("some/key/prefix"),
	MaxKeys: aws.Int64(100),
}
ctx := context.Background()

p := request.Pagination{
	NewRequest: func() (*request.Request, error) {
		req, _ := svc.ListObjectsRequest(&params)
		req.SetContext(ctx)
		return req, nil
	},
}

for p.Next(){
	page := p.Page().(*s3.ListObjectsOutput)
	
	fmt.Println("Received", len(page.Contents), "objects in page")
	for _, obj := range page.Contents {
		fmt.Println("Key:", aws.StringValue(obj.Key))
	}
}

return p.Err()

Wrap Up

The addition of Context and request options expands the capabilities of the AWS SDK for Go, giving your applications the tools needed to implement request lifecycle and configuration with the SDK. Let us know your experiences using the new Context pattern and request options features.

Using Custom Request Handlers

Requests are the core of the Go SDK. They handle the network request to a specific service. When you call a service method in the AWS SDK for Go, the SDK creates a request object that contains the input parameters and request lifecycle handlers. The logic involved in each step of a request’s lifecycle, such as Build, Validate, or Send, is handled by its corresponding list of handlers. For example, when the request’s Send method is called, it will iterate through and call every function in the request’s Send handler list. For the full list of handlers, see this API reference page. These handlers  shape and mold  what your request does on the way to AWS and what the response does on the way out.

Customizability is an important aspect of request handlers. By customizing its handlers, you can modify a request’s lifecycle for various use cases. You can do this for an individual request object returned by DoFooRequest client methods or, more interestingly, for an entire service client object, which will affect every request processed by the client.

An example use case would be gathering additional information on retried requests. In this case, a simple addition to the Retry handler would provide information required for further debugging. Logging is another use case. For example, a logger function can be added to log every request, or a benchmark function can be added to profile requests. Let’s see how you can add a custom logger to different handler lists.


func customLogger(req *request.Request) {
	 log.Printf(Op: %s\nParams: %v\nTime:      %s\nRetryCount: %d\n\n",
	 req.Operation.Name, req.Params, req.Time, req.RetryCount
	)
}


svc.Handlers.Send.PushBack(customLogger)
svc.Handlers.Unmarshal.PushBack(func(req *request.Request) {
log.Println(“Unmarshal handler”)
customLogger(req)
})

We added a customLogger request handler to the Sign and Send handlers and a closure function in the Unmarshal handler. The only requirement for this function is that it takes a request.Request pointer. The customLogger, which was added to the Send handler, will log the operation’s name, the parameters passed, time executed, and the retry count. The anonymous function added to Unmarshal will log when the Unmarshal handler was executed and then call the customLogger.

These functions are being added to the end of the handler list. During the request’s Send step, it will iterate through the list and execute the functions. Having custom loggers can be useful when developing and debugging applications because they provide more visibility into each API operation’s request/response layers.

In addition to logging, sometimes benchmarking or profiling requests are important. This will give you an idea of what to expect when hitting your production environments.


dt := time.Time{}
svc.Handlers.Send.PushFront(func(req *request.Request) {
	 dt = time.Now()
})
svc.Handlers.Send.PushBack(func(req *request.Request) {
	 fmt.Println("SENDING", time.Now().Sub(dt))
})
// Output:
// SENDING 201.007364ms

Here we push one function to the beginning of the Send’s list and then compute the delta time in the handler added to the end of the Send’s list to measure how long the API request took to send and receive a response.

As we have shown in these code samples, request handlers allow you to easily customize the SDK’s behavior and add new functionality. If you have used handlers to augment the SDK’s request/response functionality, please let us know in the comments!