I am excited to announce today’s stable release of version 2 of the AWS SDK for Ruby. It is available now as the aws-sdk gem on RubyGems.

Features

Version 2 of the AWS SDK for Ruby, the aws-sdk gem, provides a number of powerful features for developers including:

• Support for 40+ services
• Resource interfaces for 7 services
• Client response stubs
• Waiters
• Response paging
• Improved parameter validation
• Flexible gem dependencies
• Structured response data
• Improved API reference documentation with examples
• Aws.memoize no longer needed
• Intelligent credential management
• Extensible via plugins and custom request handlers

Upgrading

Version 2 of the AWS SDK for Ruby uses a different namespace, making it possible to use version 1 and version 2 in the same application.

# Gemfile
gem 'aws-sdk', '~> 2'
gem 'aws-sdk-v1'

# code
require 'aws-sdk-v1'
require 'aws-sdk'

ec2_v1 = AWS::EC2.new # v1
ec2_v2 = Aws::EC2::Resource.new # v2

This allows you to start using the version 2 SDK today without changing existing code.

Feedback

Please share your questions, comments, issues, etc. with us on GitHub. You can also catch us in our Gitter channel.

We plan to release version 2 of the AWS SDK for Ruby next week. We will remove the preview flag from the 2.0 version of aws-sdk.

Specify Your Version Dependencies

The AWS SDK for Ruby uses semantic versioning. Updates within version 1 are backwards compatible.

Version 2 of the aws-sdk gem is not backwards compatible.

If you depend on the aws-sdk gem today, and do not specify the major version, please add this now. If not, you may run into issues when you bundle update.

# Gemfile
gem 'aws-sdk', '< 2.0'

# gemspec
spec.add_dependency('aws-sdk', '< 2.0')

NameError: uninitialized constant AWS

If you receive this error, you likely have a dependency on aws-sdk and have updated so that you now have version 2 installed. Version 2 uses a different module name, so it does not define AWS.

To resolve this issue, specify your version dependency as instructed above.

Using Both Versions

The following diagram shows how the version 1 and version 2 gems are organized.

The aws-sdk gem is empty, and simply requires version 1 or version 2 specific gems. This allows you to install version 1 and version 2 in the same application.

Option A, for existing users

# Gemfile
gem 'aws-sdk', '~> 1'
gem 'aws-sdk-resources', '~> 2'

# in code
require 'aws-sdk'
require 'aws-sdk-resources'

Option B, for new users

# Gemfile
gem 'aws-sdk-v1'
gem 'aws-sdk', '~> 2'

# in code
require 'aws-sdk-v1'
require 'aws-sdk'

Attention Library Authors

If you maintain a gem that has a dependency on version 1 of aws-sdk, I strongly recommend that you replace it with a dependency on aws-sdk-v1. This allows end users to require version 2 of aws-sdk.

Please report any issues you have on our GitHub repository.

We recently added client response stubs to the aws-sdk-core gem. Response stubbing disables network traffic and causes a client to return fake or stubbed data.

# no API calls are made
s3 = Aws::S3::Client.new(stub_responses: true)
s3.list_buckets.buckets.map(&:name)
#=> []

Custom Response Data

By default, stubbed responses return empty lists, empty maps, and placeholder scalars. These empty responses can be useful at times, but often you want to control the data returned.

s3.stub_responses(:list_buckets, buckets:[{name:'aws-sdk'}])
s3.list_buckets.buckets.map(&:name)
#=> ['aws-sdk']

Safe Stubbing

One of the common risks when writing tests with stub data is that the stub doesn’t match the shape of the actual response. You risk coding against stubs that provide methods that won’t exist out of your tests.

We resolve this issue by validating your stub data hash against the model of the API. An ArgumentError is raised when calling #stub_responses with invalid data.

s3.stub_resposnes(:list_buckets, buckets:['aws-sdk'])
#=> raises ArgumentError, "expected params[:buckets][0] to be a hash"

Stubbing Multiple Calls

By calling #stub_responses with an operation name and stub data, the client will serve that data for each call. You can specify multiple responses, and they will be used in sequence.

s3.stub_responses(:list_buckets, 
  { buckets:[{name:'aws-sdk'}] },
  { buckets:[{name:'aws-sdk', 'aws-sdk-2'}] }
)

s3.list_buckets.buckets.map(&:name)
#=> ['aws-sdk']

s3.list_buckets.buckets.map(&:name)
#=> ['aws-sdk', 'aws-sdk-2']

Stubbing Errors

In addition to stubbing response data, you can configure errors to raise. You can specify a service error by name, or you can provide an error object or class to raise.

# everything is broken
s3.stub_responses(:head_bucket, 
  'NotFound'
  Timeout::Error,
  RuntimeError.new('oops')
)

s3.head_bucket(bucket:'aws-sdk')
# raises Aws::S3::Errors::NotFound

s3.head_bucket(bucket:'aws-sdk')
# raises a new Timeout::Error

s3.head_bucket(bucket:'aws-sdk')
# raises RuntimeError.new('oops')

You can mix stubbed response data and errors. This approach is great when you want to test how well your code recovers from errors. 

Stubbing All Clients

The default config can be used to enable client stubbing globally. This can be very useful when writing tests to enable stubbing in your test helper once.

# stub everything
Aws.config[:stub_responses] = true

Give it a try and let us know what you think.

We’ve added a feature called Waiters to the v2 AWS SDK for Ruby, and I am pretty excited about it. A waiter is a simple abstraction around the pattern of polling an AWS API until a desired state is reached.

Basic Usage

This simple example shows how to use waiters to block until a particular EC2 instance is running:

ec2 = Aws::EC2::Client.new
ec2.wait_until(:instance_running, instance_ids:['i-12345678'])

Waiters will not wait indefinitely and can fail. Each waiter has a default polling interval and maximum number of attempts to make. If a waiter encounters an unexpected error or fails to reach the desired condition in time it will raise an error:

begin
  ec2.wait_until(:instance_running, instance_ids:['i-12345678'])
resuce Aws::Waiters::Errors::WaiterFailed
  # oops
end

Configuration

You can modify the default interval and wait time between attempts by passing a block.

# this will wait upto ~ one hour
ec2.wait_until(:instance_running, instance_ids:['i-12345678']) do |w|

  # seconds between each attempt
  w.interval = 15

  # maximum number of polling attempts before giving up
  w.max_attempts = 240

end

Callbacks

In addition to interval and maximum attempts, you can configure callbacks to trigger before each attempt polling attempt and before sleeping between attempts.

ec2.wait_until(:instance_running, instance_ids:['i-12345678']) do |w|

  w.before_attempt do |n|
    # n - the number of attempts made
  end

  w.before_wait do |n, resp|
    # n - the number of attempts made
    # resp -the client response from the previous attempt
  end

end

You can throw :success or :failure from these callbacks to stop the waiter immediately. You can use this to write you own delay and back-off logic.

Here I am using a callback to perform exponential back-off between polling attempts:

ec2.wait_until(:instance_running, instance_ids:['i-12345678']) do |w|
  w.interval = 0 # disable normal sleep
  w.before_wait do |n, resp|
    sleep(n ** 2)
  end
end

This example gives up after one hour.

ec2.wait_until(:instance_running, instance_ids:['i-12345678']) do |w|
  one_hour_later = Time.now + 3600
  w.before_wait do |n, resp|
    throw :failure, 'waited too long' if Time.now > one_hour_later
  end
end

Waiters and Resources, Looking Ahead

You may have noticed that some waiters have already been exposed to the resource classes.

ec2 = Aws::EC2::Resource.new
instance = ec2.instance('i-12345678')
instance.stop
instance.wait_until_stopped
puts instance.id + ' is stopped'

In addition to connecting more waiters and resources, I’m excited to look into batch waiters. Imagine the following use case:

instances = ec2.create_instances(min_count: 5, ...)
instances.wait_until_running
puts "the following new instances are now running:n"
puts instances.map(&:id)

Documentation

Waiters are documented in the Ruby SDK API reference. Each service client documents the #wait_until method and provides a list of available waiter names. Here are links to the Aws::EC2::Client waiter methods:

• Aws::EC2::Client#wait_until

• Aws::EC2::Client#waiter_names

Give waiters a try and let us know what you think!

With the recent 2.0 stable release of the aws-sdk-core gem, we started publishing preview releases of aws-sdk-resources. Until the preview status is released, you will need to use the –pre flag to install this gem:

gem install aws-sdk-resources --pre

In bundler, you should give the full version:

# update the version as needed
gem 'aws-sdk-resources', version: '2.0.1.pre'

Usage

Each service module has a Client class that provides a 1-to-1 mapping of the service API. Each service module now also has a Resource class that provides an object-oriented interface to work with.

Each resource object wraps a service client.

s3 = Aws::S3::Resource.new
s3.client
#=> #<Aws::S3::Client>

Given a service resource object you can start exploring related resources. Lets start with buckets in Amazon S3:

# enumerate all of my buckets
s3.buckets.map(&:name)
#=> ['aws-sdk', ...]

# get one bucket
bucket = s3.buckets.first
#=> #<Aws::S3::Bucket name="aws-sdk">

If you know the name of a bucket, you can construct a bucket resource without making an API request.

bucket = s3.bucket('aws-sdk')

# constructors are also available
bucket = Aws::S3::Bucket.new('aws-sdk')
bucket = Aws::S3::Bucket.new(name: 'aws-sdk')

In each of the three previous examples, an instance of Aws::S3::Bucket is returned. This is a lightweight reference to an actual bucket that might exist in Amazon S3. When you reference a resource, no API calls are made until you operate on the resource.

Here I will use the bucket reference to delete the bucket.

bucket.delete

You can use a resource to reference other resources. In the next exmple, I use the bucket object to reference an object in the bucket by its key.
Again, no API calls are made until I invoke an operation such as #put or #delete.

obj = bucket.object('hello.txt')
obj.put(body:'Hello World!')
obj.delete

Resource Data

Resources have one or more identifiers, and data. To construct a resource, you only need the identifiers. A resource can load itself using its identifiers.

Constructing a resource object from its identifiers will never make an API call.

obj = s3.bucket('aws-sdk').object('key') # no API call made

# calling #data loads an object, returning a structure
obj.data.etag
#=> "ed076287532e86365e841e92bfc50d8c"

# same as obj.data.etag
obj.etag
#=> "ed076287532e86365e841e92bfc50d8c"

Resources will never update internal data until you call #reload. Use #reload if you need to poll a resource attribute for a change.

# force the resource to refresh data, returning self
obj.reload.last_updated_at

Resource Associations

Most resources types are associated with one or more different resources. For example, an Aws::S3::Bucket object bucket has many objects, a website configuration, an ACL, etc.

Each association is documented on the resource class. The API documentation will specify what API call is being made. If the association is plural, it will document when multiple calls are made.

When working with plural associations, such as bucket that has many objects, resources are automatically paginated. This makes it simple to lazily enumerate all objects.

bucket = s3.bucket('aws-sdk')

# enumerate **all** objects in a bucket, objects are fetched
# in batches of 1K until every object has been yielded
bucket.objects.each do |obj|
  puts "#{obj.key} => #{obj.etag}"
end

# filter objects with a prefix
bucket.objects(prefix:'/tmp/').map(&:key)

Some APIs support operating on resources in batches. When possible,
the SDK will provide batch actions.

# gets and deletes objects in batches of 1K, sweet!
bucket.objects(prefix:'/tmp/').delete

Resource Waiters

Some resources have associated waiters. These allow you to poll until the resource enters a desired state.

instance = Aws::EC2::Instance.new('i-12345678')
instance.stop
instance.wait_until_stopped
puts instance.id + ' is stopped'

Whats Next?

The resource interface has a lot of unfinished features. Some of the things we are working on include:

• Adding #exists? methods to all resource objects
• Consistent tagging interfaces
• Batch waiters
• More service coverage with resource definitions

We would love to hear your feedback. Resources are available now in the preview release of the aws-sdk-resources gem and in the master branch of GitHub.

Happy coding!

Version 2 of the AWS SDK for Ruby is available now as a preview release. If you use Bundler with some standard best-practices, you should be unaffected by the v2 release of the aws-sdk gem. This blog post highlights a few things you might want to be aware of.

Installing V2 Preview Release

V2 of the AWS SDK for Ruby is available now as a preview release. To install v2, use –pre:

$ gem install aws-sdk --pre

If you are using bundler, you must specify the full version until the preview status is removed.

gem 'aws-sdk', '2.0.0.pre'

Lock your Dependencies

The V2 Ruby SDK is not backwards compatible with the V1 Ruby SDK. If you have a bundler dependency on aws-sdk and you do not specify a version, you will run into problems with the 2.0 final is released.
To ensure you are unaffected by the major version bump, ensure you specify a version dependency in your Gemfile:

gem 'aws-sdk', '< 2.0'

Alternatively, you can change your gem dependency from aws-sdk to aws-sdk-v1

gem 'aws-sdk-v1'

The AWS SDK for Ruby follows semver. This allows users to update within the same major version with confidence that there are not backwards incompatible changes. If there are, they will be treated as bugs.

Use Both Version in One Application

The V1 and V2 Ruby SDKs use different namespaces. You may only load one version of a single gem. We publish the v1 Ruby SDK as a separate gem now to allow users to load both versions. Additionally, the v2 SDK uses a different root namespace to avoid conflicts.

# in your Gemfile
gem 'aws-sdk-v1'
gem 'aws-sdk', '2.0.0.pre'

And then in your application:

require 'aws-sdk-v1'
require 'aws-sdk'

# v1 uses the AWS module, v2 uses the Aws module
s3_v1 = AWS::S3::Client.new
s3_v2 = Aws::S3::Client.new

Links of Interest

• Documentation
• Issues
• Forums
• License

Happy coding, and as always, feedback is welcomed!

In version 1 of the AWS SDK for Ruby provides a 1-to-1 client class for each AWS service. For many services it also provides a resource-oriented interface. These resource objects use the client to provide a more natural object-oriented experience when working with AWS APIs.

We are busy working resource interfaces for the v2 Ruby SDK.

Resource Interfaces

The following examples use version 1 of the aws-sdk gem. This first example uses the 1-to-1 client to terminate running instances:

ec2 = AWS::EC2::Client.new
resp = ec2.describe_instances
resp[:reservations].each do |reservation|
  reservation[:instances].each do |instance|
    if instance[:state][:name] == 'running'
      ec2.terminate_instances(instance_ids:[instance[:instance_id]])
    end
  end
end

This example uses the resource abstraction to start instances in the stopped state:

ec2 = AWS::EC2.new
ec2.instances.each do |instance|
  instance.start if instance.status == :stopped
end

Resources for Version 2

We have a lot of lessons learned from our v1 resource interfaces. We are busy working on the v2 abstraction. Here are some of the major changes from v1 to v2.

Memoization Interfaces Removed

The version 1 resource abstraction was very chatty by default. It did not memoize any resource attributes and a user could unknowingly trigger a large number of API requests. As a workaround, users could use memoization blocks around sections of their code.

In version 2, all resources objects will hold onto their data/state until you explicitly call a method to reload the resource. We are working hard to make it very obvious when calling a method on a resource object will generate an API request over the network.

Less Hand-Written Code and More API Coverage

The version 1 SDK has hand-coded resource and collection classes. In version 2, our goal is to extend the service API descriptions that power our clients with resource definitions. These definitions will be consumed to generate our resource classes.

Using resource definitions helps eliminate a significant amount of hand written code, ensures interfaces are consistent, and makes it easier for users to contribute resource abstractions.

We also plan to provide extension points to resources to allow for custom logic and more powerful helpers.

Resource Waiters

It is a common pattern to operate on a resource and then wait for the change to take effect. Waiting typically requires making an API request, asserting some value has changed and optionally waiting and trying again. Waiting for a resource to enter a certain state can be tricky. You need to deal with terminal cases, failures, transient errors, etc.

Our goal is to provide waiter definitions and attach them to our resource interfaces. For example:

# create a new table in Amazon DynamoDB
table = dynamodb.table('my-table')
table.update(provisioned_throughput: { 
  read_capcity_units: 1000
})

# wait for the table to be ready
table.wait_for(:status, 'ACTIVE')

In a follow up blog post, I will be introducing the resources branch of the SDK that is available today on GitHub. Please take a look and feedback is always welcome!

We’ve been busy working on version 2 of the AWS SDK for Ruby. One of the features we added recently was response paging.

Paging in the Version 1 Ruby SDK

In version 1 of the Ruby SDK provides collection classes for many AWS resources. These collections are enumerable objects that yield resource objects.

iam = AWS::IAM.new
user_collection = iam.users
user_collection.each |user|
  puts user.name
end

A collection in version 1 sends a request to enumerate resources. If the response indicates that more data is available, then the collection will continue sending requests to enumerate all resources.

If you want to enumerate a resource that is not modeled in the version 1 SDK, then you need to drop down to the client abstraction and deal with paging on your own.

iam = AWS::IAM.new
options = { max_items: 2 }
begin
  response = iam.client.list_users(options)
  response[:users].each do |user|
    puts user[:user_name]
  end
  options[:marker] = response[:marker]
end while options[:marker]

Response Paging in Version 2 Ruby SDK

One of our main goals of the version 2 Ruby SDK is to improve the experience of users accessing AWS from the client abstractions. Version 2 does not provide resource abstractions yet, but it does provide full response paging from the client interface.

Here is the example above re-written using the version 2 Ruby SDK:

iam = Aws::IAM.new
iam.list_users.each do |response|
  puts response.users.map(&:user_name)
end

Each AWS operation now returns a pageable response object. This object is enumerable. Calling #each on a Aws::PagableResponse object yields the response and any follow up responses.

There are a few other helper methods that make it easy to control response paging:

resp.last_page? #=> false
resp.next_page? #=> true

# get the next page, raises an error if this is the last page
resp = resp.next_page

# gets each response in a loop
resp = resp.next_page until resp.last_page?

Resource Enumeration in the Version 2 Ruby SDK

You will notice the response paging examples don’t address enumerating individual resource objects. We are busy implementing a resource abstraction for the version 2 Ruby SDK. The v2 resources will be enumerable in a method similar to v1. It will however be built on top of client response paging.

Watch the GitHub respository and this blog for more information on resource abstractions in the version 2 Ruby SDK.

We recently published v2.0.0.rc12 of the aws-sdk-core gem (https://github.com/aws/aws-sdk-core-ruby). This release merges the long-running normalized branch onto master.

Upgrading Notes

Please note, when updating to rc12, you may need to make some minor code changes. These are summarized below:

• Service modules now have a Client class, these should be used to construct API clients:

  # deprecated, will be removed for 2.0.0 final
  s3 = Aws::S3.new
  
  # preferred
  s3 = Aws::S3::Client.new
  

  Looking forward to the resources update, this will ensure we have a suitable namespace for the new resource classes. Look for more information in a follow up blog post.

• The Amazon SimpleDB client class has been renamed from Aws::SDB to Aws::SimpleDB. This also affects the short name used in configuration:

  # old configuration key
  Aws.config[:sdb] = { ... }
  
  # new key
  Aws.config[:simpledb] = { ... }
  

• The :raw_json configuration option has been renamed to :simple_json. This is used for services that use the JSON protocol.

Less Visible Changes

If you have written plugins for the aws-sdk-core gem, there are a few other changes to the internals you need to be aware of.

• Seahorse::Model has received significant updates, especially the API model format. This new format is much more flexible than the denormalized format used previously. Additionally, the AWS API models are now consumed as-is without translation. See the API reference for more information.

• Seahorse::Client::Http::Request#endpoint is now URI::HTTPS or URI::HTTP object. The custom Endpoint class has been removed in favor of these objects provided by the Ruby standard library.

• Seahorse::Client::HandlerList#add no longer accepts instance objects and requires a handler class that can be constructed.

The AWS SDK for Ruby provides a few methods for getting objects out of Amazon S3. This blog post focuses on using the v2 Ruby SDK (the aws-sdk-core gem) to download objects from Amazon S3.

Downloading Objects into Memory

For small objects, it can be useful to get an object and have it available in your Ruby processes. If you do not specify a :target for the download, the entire object is loaded into memory into a StringIO object.

s3 = Aws::S3::Client.new
resp = s3.get_object(bucket:'bucket-name', key:'object-key')

resp.body
#=> #<StringIO ...> 

resp.body.read
#=> '...'

Call #read or #string on the StringIO to get the body as a String object.

Downloading to a File or IO Object

When downloading large objects from Amazon S3, you typically want to stream the object directly to a file on disk. This avoids loading the entire object into memory. You can specify the :target for any AWS operation as an IO object.

File.open('filename', 'wb') do |file|
  reap = s3.get_object({ bucket:'bucket-name', key:'object-key' }, target: file)
end

The #get_object method still returns a response object, but the #body member of the response will be the file object given as the :target instead of a StringIO object.

You can specify the target as String or Pathname, and the Ruby SDK will create the file for you.

resp = s3.get_object({ bucket:'bucket-name', key:'object-key' }, target: '/path/to/file')

Using Blocks

You can also use a block for downloading objects. When you pass a block to #get_object, chunks of data are yielded as they are read off the socket.

File.open('filename', 'wb') do |file|
  s3.get_object(bucket: 'bucket-name', key:'object-key') do |chunk|
    file.write(chunk)
  end
end

Please note, when using blocks to downloading objects, the Ruby SDK will NOT retry failed requests after the first chunk of data has been yielded. Doing so could cause file corruption on the client end by starting over mid-stream. For this reason, I recommend using one of the preceding methods for specifying the target file path or IO object.

Retries

The Ruby SDK retries failed requests up to 3 times by default. You can override the default using :retry_limit. Setting this value to 0 disables all retries.

If the Ruby SDK encounters a network error after the download has started, it attempts to retry request. It first checks to see if the IO target responds to #truncate. If it does not, the SDK disables retries.

If you prefer to disable this default behavior, you can either use the block mode or set :retry_limit to 0 for your S3 client.

Range GETs

For very large objects, consider using the :range option and download the object in parts. Currently there are no helper methods for this in the Ruby SDK, but if you are interested in submitting something, we accept pull requests!

Happy downloading.