Tag: ruby

Locking Dependency Versions with Bundler

by Loren Segal | on | in Ruby | Permalink | Comments |  Share

When writing a Ruby application or library that makes use of third-party dependencies, it is always best to keep track of those dependency versions. This is an easy thing to do with Bundler, but not everybody does it. We think you should, so here are some quick tips on how to get started.

Locking to Major Versions

Bundler works by providing a Gemfile that contains the list of third-party dependencies for your application or library. This list can specify library dependencies by a specific (or fuzzy) version, but it does not require the version field to be set.

Even though the version field is optional, we recommend that it should always be set to at least a "major version" of your dependencies. If those libraries use Semantic Versioning (SemVer), this means locking to all "1.x", "2.x", "3.x", or other major releases. You can do this in Bundler by using the fuzzy version check syntax:

gem 'some_dependency', '~> 1.0'

This locks the some_dependency gem to any version of 1.x, from 1.0 all the way to (but not including) 2.0. Without this check, you risk breaking your application build (or downstream consumers or your library) when the dependency releases a new major version. Instead, with this simple constraint, your application or library will not automatically pick up a new major version, and the risk of your code randomly breaking due to third-party changes should decrease greatly.

If you are a library developer and are not already following Semantic Versioning, we recommend that you read up on these versioning conventions and consider following them, as it makes your library much more reliable for downstream consumers. Your users will thank you.

Getting Specific

If you want to provide a more fine-grained constraint than a major version, it is possible to do so with the same fuzzy version check syntax as above. This might be necessary when using libraries that do not follow Semantic Versioning conventions. The only syntactic difference is that you also must specify the minor version that you want to lock into.

For example, to lock to any patchlevel release in a 1.5 minor version of a library, you can provide the following constraint:

gem 'some_dependency', '~> 1.5.0'

Note the extra ".0" suffix, which tracks the dependency through all 1.5.x releases. Without the ".0" suffix, the constraint would refer to any 1.x release that is greater than or equal to 1.5.

Notes for the AWS SDK for Ruby

The good news is that the AWS SDK for Ruby follows Semantic Versioning, which means we do not make backward-incompatible changes within a major release. To ensure that such changes will not make their way downstream to your application or library, it is best to always lock your version of the Ruby SDK to a major release. Since we are currently in our 1.x major version, you can do this with Bundler by specifying the aws-sdk dependency as follows:

gem 'aws-sdk', '~> 1.0'

This way you will not accidentally receive any backward-incompatible changes should we ever release a new major version of the Ruby SDK.

In Your Gem Specification

If you release your library and maintain a separate .gemspec file, you can (and should) use the same constraint syntax there too. You can see the RubyGems Specification Reference for more details, but in short, you simply need to list the dependency as:

spec.add_runtime_dependency 'some_dependency', '~> 1.0'

This will provide the same major version constraint that Bundler does for anybody who runs gem install yourgem.

Finishing Up

Specifying major versions for third-party dependencies in your Gemfile or .gemspec file is easy, and we should all be doing it. At the very least, providing a major version helps ensure that your library is more resistant to backward-incompatible changes coming from third-party code, and saves your downstream users from ending up with those breaking changes. As the community starts to make use of more third-party libraries in a single application or gem, it’s much more important to stay on top of dependency management and avoid these kinds of failures.

Stubbing AWS Responses

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

I come across questions frequently about how to test an application that uses the AWS SDK for Ruby (aws-sdk gem). Testing an application that makes use of an external service is always tricky. One technique is to stub the client-level responses returned by the SDK.


Calling the AWS.stub! method in your ruby process configures the client classes (e.g. AWS::EC2::Client) to stub their responses when called. They stop short of making a live HTTP request and instead return an empty response.


instance_ids = AWS::EC2.new.instances.map(&:id)
instance_ids #=> will always be empty in this example, not HTTP request made

Under the covers, the code example is constructing an AWS::EC2::Client object and is calling the #describe_instances method. AWS.stub! causes the client to return an empty response that looks like a normal response with a few differences:

  • Lists are returned as empty arrays
  • Maps are returned as empty hashes
  • Numeric values are always zero
  • Dates are returned as now

Localized Stubbing

Calling AWS.stub! is the same as calling AWS.config(:stub_requests => true). You can use this configuration option with any constructor that accepts configuration.

stub_ec2 = AWS::EC2.new(:stub_requests => true)
real_ec2 = AWS::EC2.new

Customizing the Responses

In addition to getting empty responses, you can access the stubbed responses and populate them with fake data.


ec2 = AWS::EC2::Client.new
resp = ec2.stub_for(:describe_instances)
resp.data[:reservation_set] = [...]

# now calling ec2.describe_instances will return my fake data
#=> { :reservation_set => [...] } 

There are two methods you can use here:

  • #stub_for(operation_name)
  • #new_stub_for(operation_name)

The first method, #stub_for, returns the same stubbed response every time. It is the default response for that operation for that client object (not shared between instances). The second method, #new_stub_for, generates a new response. This is useful if you need to stub the client to return different data from multiple calls. This is common for paged responses.

Not a Mock

Unfortunately, this approach stubs responses, but it does not mock AWS services. If I used a stubbed AWS::DynamoDB::Client and call #put_item, I will not be able to get the data back. There are a number of third-party libraries that attempt to provide local service mocks. These can be helpful when you are trying to run local tests without hitting the network.

Logging HTTP Wire Traces

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

In a previous previous blog post, I wrote about how to log requests generated by the AWS SDK for Ruby (aws-sdk gem). While this can be a valuable tool in seeing how your code translates into requests to AWS, it doesn’t do everything. What if you think the SDK is serializing your request incorrectly? Sometimes anything short of a HTTP wire trace just isn’t enough.

That problem is easily solved:

ddb = AWS::DynamoDB.new(:http_wire_trace => true)
#=> 'aws-sdk-test'

This will send the following to your configured logger:

opening connection to dynamodb.us-east-1.amazonaws.com...
<- "POST / HTTP/1.1rnContent-Type: application/x-amz-json-1.0rnX-Amz-Target: DynamoDB_20111205.ListTablesrnContent-Length: 11rnUser-Agent: aws-sdk-ruby/1.8.5 ruby/1.9.3 x86_64-darwin11.4.2rnHost: dynamodb.us-east-1.amazonaws.comrnX-Amz-Date: 20130315T163624ZrnX-Amz-Content-Sha256: 55522f708dcfebccb7bd3e8d0001a53ecaf2beca9ca801f1e9161e24215faa99rnAuthorization: AWS4-HMAC-SHA256 Credential=AKIAJUNH63P3WCTAYHFA/20130315/us-east-1/dynamodb/aws4_request, SignedHeaders=content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-target, Signature=8a5a5082afb33542eacd3c9429a5efaa80194b6e8390d4592da53f117d1d6d0drnAccept: */*rnrn"
<- "{"Limit":1}"
-> "HTTP/1.1 200 OKrn"
-> "x-amz-crc32: 349808839rn"
-> "Content-Type: application/x-amz-json-1.0rn"
-> "Content-Length: 71rn"
-> "Date: Fri, 15 Mar 2013 16:36:25 GMTrn"
-> "rn"
reading 71 bytes...
-> "{"LastEvaluatedTableName":"aws-sdk-test","TableNames":["aws-sdk-test"]}"
read 71 bytes
Conn keep-alive

If you have not configured a logger, this output will be sent to $stdout (very helpful when you are using IRB). You can also enable HTTP wire logging globally:

AWS.config(:logger => Logger.new($stderr), :http_wire_trace => true)


Contributing to the AWS SDK for Ruby

by Loren Segal | on | in Ruby | Permalink | Comments |  Share

We love getting contributions from the community to the AWS SDK for Ruby. Whether it be added features, fixed bugs, or just extra optimizations, submitting a pull request helps make the SDK better for all of our users. Since we started the project, the SDK has seen over 60 contributors providing everything from a one line typo fix to a 500+ line high level service abstraction. In the past month alone we have merged over 20 pull requests, and we continue to see more every day. We not only want to keep seeing these pull requests, we want to see more of them, so if you have been submitting pull requests we want you to know that you should keep them coming!

However, if you haven’t yet submitted a pull request, today is a great day to start. We know that contributing code to a large project can occasionally seem daunting, so let’s talk about some of the very easy things you can do to make it easier for us to evaluate your patch and possibly merge it into the SDK.

How to Contribute on GitHub

If you would like to contribute to the SDK you can do so on our GitHub project at https://github.com/aws/aws-sdk-ruby. From there you can fork the repository and submit a pull request of your changes. GitHub has tools and guides to make the technical portions of this process as easy as possible, including desktop applications for Microsoft Windows and Mac OS X to sync your code to and from the site. You should visit GitHub’s help pages to get more information about submitting pull requests.

Contributing Documentation

Contributing documentation and fixing typos is one of the easiest ways to get started as a contributor to the SDK. Fortunately, GitHub makes editing files for small text changes very easy thanks to its inline file editor. If you find a typo or missing documentation, you can navigate to the file in GitHub and click the "Edit" button at the top to quickly bring up the file editor:

Contributing documentation part 1

You can now edit the file and write a descriptive comment about the change you’ve made. You should also make sure to "Preview" the document and ensure that it still looks okay in GitHub.

Editing a file in GitHub

If you do not already have a fork of the repository, GitHub will automatically fork the project to a new repository that you can use to submit a pull request from.

Submitting a pull request

This opens a pull request against the aws/aws-sdk-ruby project and begins the process of allowing us to evaluate the changes.

Contributing Code

To contribute code, you first need to clone your own fork of the aws-sdk-ruby Git repository. Once you have cloned the repository, use the Bundler bundle install command to install all development dependencies.

You should then run tests to ensure that there are no issues with the project before you get started. To run tests, run the rake command.

After you verify that the tests pass, you can make your changes. Please make sure to include new tests if you change or add behavior to the SDK. You should then run tests and ensure that they all pass before submitting your pull request, as this makes the process of merging your commits hassle-free.

Once you have added code, tests, and all the tests pass, you can commit your code with descriptive messages and push your changes to your forked repository. You can then log onto GitHub and click the "Pull Request" button at the top of your fork of the SDK.

Submitting a pull request from a forked repository

Submitting the pull request follows the same process as for documentation. Once you submit the pull request, we can evaluate the code and discuss any adjustments, if necessary.

Accepting Contributions!

As I said before, we love getting contributions from the community. If you have ideas about how the AWS SDK for Ruby can be improved, we definitely appreciate the help. Feature suggestions are always encouraged, but providing working code with tests in the form of a pull request will usually mean that we can get the changes into the SDK much faster. Plus, you get credit for making the SDK a better product, not just for yourself, but for all of your fellow Rubyists!

So go ahead, check out the repository on GitHub, and start submitting patches.

AWS SDK for Ruby Release v1.8.3

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

We just published version 1.8.3 of the AWS SDK for Ruby (aws-sdk gem).  This release adds support for AWS OpsWorks and resolves a number of customer reported issues. 

require 'aws-sdk'

opsworks = AWS::OpsWorks.new
resp = opsworks.client.describe_stacks
resp #=> { :stacks => [] }

You can view the AWS::OpsWorks::Client API documentation here. Take it for a spin and leave some feedback!

Logging Requests

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

The AWS SDK for Ruby (aws-sdk gem) has some pretty cool logging features. I find them particularly helpful when I need to debug something. I generally jump into an IRB session that has a logger pre-wired for me and then start sending requests.

Configuring a Logger

To get log messages from the aws-sdk gem, you need to configure a logger. The easiest way is to create a Logger (from Ruby’s standard lib) and pass it to AWS.config.

require 'aws-sdk'
require 'logger'

AWS.config(:logger => Logger.new($stdout))

# make a request
s3 = AWS::S3.new

# log output sent to standard out
I, [2013-02-14T09:49:12.856086 #31922]  INFO -- : [AWS S3 200 0.194491 0 retries] head_object(:bucket_name=>"aws-sdk",:key=>"key")

By default, requests are logged with a level of :info. You can override the default log level with AWS.config.

AWS.config(:log_level => :debug)

Log Formatters

The default log message contain the following information:

  • The service class name (e.g. ‘S3’)
  • The HTTP response status code (e.g. 200)
  • The total time taken in seconds
  • The number of retries
  • A summary of the client method called

Similar to how you can configure :logger and :log_level, you can register a custom log formatter via :log_formatter. Log formatters accept a AWS::Core::Response object and then return a formatted log message. The built-in AWS::Core::LogFormatter class has support for simple pattern replacements.

pattern = '[REQUEST :http_status_code] :service :operation :duration'
formatter = AWS::Core::LogFormatter.new(pattern)

AWS::S3(:log_formatter => formatter).new.buckets.first

# log output
I, [2013-02-14T09:49:12.856086 #31922]  INFO -- : [REQUEST :http_status_code] S3 list_buckets 0.542574

Canned Log Formatters

You can choose from a handful of ready-to-use log formatters you can choose from, including:

  • AWS::Core::LogFormatter.default
  • AWS::Core::LogFormatter.short
  • AWS::Core::LogFormatter.debug
  • AWS::Core::LogFormatter.colored

Just pass one of these to AWS.config and start making requests.

AWS.config(:log_formatter => AWS::Core::LogFormatter.colored)

Logging in Rails

If you require the aws-sdk gem inside a Rails application, then the Ruby SDK automatically wire itself up to the Rails.logger. You are free to still configure a different logger or to change the log level or formatter.

Fetch Object Data and Metadata from Amazon S3 (in a Single Call)

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

I came across an excellent question earlier this week on our support forums. The question was essentially, "How can I fetch object data and metadata from Amazon S3 in a single call?"

This is fair question, also one I did not have a good answer to. Amazon S3 returns both object data and metadata in a single GET object response, while AWS::S3::S3Object#readdoes not. But why?

How It Used to Be

Here is an example of how to get data from an object in S3.

obj = s3.buckets['my-bucket'].objects['key']
data = obj.read

Notice the #read method is returning the object data. This leaves no good place to return the metadata (returning multiple values from a function in ruby is generally frowned upon). In this case, the aws-sdk gem was getting the data and metadata from S3, but it was discarding the metadata.

The Best of Both Worlds

Last year we added support for streaming reads to AWS::S3::S3Object#read. If you pass a block to #read, then the data is yielded in chunks to the block.

File.open('filename', 'wb') do |file|
  obj.read do |chunk|

Perfect! Since the #read method is yielding data in chunks, its return value becomes unused. This allowed me to patch the #read method to return the object metadata instead of nil.

resp = obj.read do |chunk|

resp #=> {:meta => {"foo" => "bar"}, :restore_in_progress => false, :content_type=>"text/plain", :etag=>""37b51d194a7513e45b56f6524f2d51f2"", :last_modified => 2013-02-06 12:54:39 -0800, :content_length => 94512, :data => nil}

You can checkout the new feature on our GitHub master branch now. This will be part of our next release.

If you see an issue with the AWS SDK for Ruby (aws-sdk gem), please, post an issue on our GitHub issue tracker!

IAM Roles for Amazon EC2 Instances (Credential Management Part 4)

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

This is the fourth and final part (part 1, part 2, part 3) in a series on how to securely manage your AWS access credentials.

This week I am focusing on using AWS Identity and Access Management (IAM) roles for Amazon EC2 instances with the AWS SDK for Ruby (aws-sdk). Simply put, IAM roles for EC2 instances remove the need to try to bootstrap your instances with credentials. Example:

# look ma! No credentials (when run from EC2)!
require 'aws-sdk'

You can stop worrying about how to get credentials onto a box that Auto Scaling just spun-up on your behalf. Your cron scripts on instances no longer have to search around for credentials on disk or worry about when they get rotated.

One of the best features of IAM roles for EC2 instances is the credentials on EC2 are auto-rotated! Instances started with an IAM instance profile will get temporary credentials deployed on a regular basis to the EC2 instance metadata service.

My favorite feature? The aws-sdk gem just works when running on an instance with an IAM profile.

How it Works

  • You create an IAM instance profile
  • Start one or more instances, with the instance profile
  • Upon instance boot, session credentials will be available on your instance(s)
  • Credentials are rotated regularly (before they expire)

You can reuse an instance profile as many times as you like. The aws-sdk gem will automatically attempt to load credentials from the metadata service when no other credentials are provided.

Create an Instance Profile with the aws-sdk Gem

An instance profile consists of a role. The role consists of a policy. Each of these has a name. I am going to use the aws-sdk to create a sample profile starting with the policy.

For this example, I’ll be building an instance profile that has limited permissions. I only want applications on EC2 to be able to read from Amazon S3 (list and get buckets/objects). First, I need to build the policy.

require 'aws-sdk'

AWS.config(:access_key_id => '...', :secret_access_key => '…')

# the role, policy and profile all have names, pick something descriptive
role_name = 's3-read-only'
policy_name = 's3-read-only'
profile_name = 's3-read-only'

# required so that Amazon EC2 can generate session credentials on your behalf
assume_role_policy_document = '{"Version":"2008-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":["ec2.amazonaws.com"]},"Action":["sts:AssumeRole"]}]}'

# build a custom policy    
policy = AWS::IAM::Policy.new
policy.allow(:actions => ["s3:Get*","s3:List*"], :resources => '*')

Now that I have a policy built, I am going to build a role and add the policy to the role. I am going to use the policy and role names I chose above.

iam = AWS::IAM.new

# create the role
  :role_name => role_name,
  :assume_role_policy_document => assume_role_policy_document)

# add the policy to role
  :role_name => role_name,
  :policy_name => policy_name,
  :policy_document => policy.to_json)

Last step is to create a profile for your role.

resp = iam.client.create_instance_profile(
  :instance_profile_name => instance_profile_name)

# this may be handy later
profile_arn = resp[:instance_profile][:arn]

  :instance_profile_name => instance_profile_name,
  :role_name => role_name)

Using an IAM Instance Profile

You can now use the instance profile name (or ARN we captured above) to run instances with your special profile.

# you can use the profile name or ARN as the :iam_instance_profile option
ec2 = AWS::EC2.new
ec2.instances.create(:image_id => "ami-12345678", :iam_instance_profile => profile_name)

Thats it! Your new instance will boot with session credentials available that the aws-sdk can consume with zero configuration. Happy computing!

Credential Providers (Credential Management Part 3)

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

In part 1 of this series, I wrote about how to configure your access credentials with the AWS SDK for Ruby (aws-sdk gem). In part 2 we learned how to rotate your access credentials using the aws-sdk gem.

This week we explore credential providers and how they can help you keep your secrets safe and fresh.

Credential Providers

A credential provider is an object that responds to the following methods:

  • #access_key_id
  • #secret_access_key
  • #session_token
  • #refresh

Internally, the aws-sdk gem uses a chain of credential providers to load credentials from various locations including:

  • AWS.config
  • ENV (from multiple key prefixes)
  • EC2 instance metadata service

You can fully customize this behavior by configuring your own custom credential provider, as follows.

AWS.config(:credential_provider => CustomCredentialProvider.new)

Why Use a Custom Credential Provider?

In the previous post in this series we discussed rotating credentials. It can be painful to build logic that restarts processes or applications that are using stale or soon-to-be removed/expired credentials.

If your application uses a custom credential provider, the application does not need to be restarted. The SDK automatically calls #refresh on the credential provider when it receives a response from AWS that indicates its credentials are expired.

In-house hosted applications and utility scripts can use a custom provider to load credentials from a simple web service that vends current credentials. This web service could even go as far as vending session credentials that auto-expire and are limited to specific AWS operations. This greatly reduces exposure if these credentials are ever leaked.

Build a Custom Credential Provider

Here is a really simple custom credential provider that makes a HTTPS request to https://internal.domain/ and expects a JSON response of credentials.

require 'net/https'

class CustomCredentialProvider

  include AWS::Core::CredentialProviders::Provider


  def get_credentials
      http = Net::HTTP.new('internal.domain', 443)
      http.use_ssl = true
      http.verify_mode = OpenSSL::SSL::VERIFY_PEER
      response = http.request(Net::HTTP::Get.new('/'))

      # symbolize keys to :access_key_id, :secret_access_key
      if response.code == '200'
        JSON.load(response.body).inject({}) {|h,(k,v)| h.merge(k.to_sym => v) }


The include statement in the example above does much of the heavy lifting. It defines all of the public methods required for a credential provider. It also caches the credentials until #refresh is called. We only have to define #get_credentials and return a hash with symbol keys (or an empty hash if we fail).

You can make this example more robust if you:

  • Set network timeouts
  • Rescue Exceptions raised by HTTP that do not extend StandardError
  • Add basic retry logic to handle transient network errors

In the next (and last) post in this series I will explore how the aws-sdk gem uses the EC2 metadata service.

Rotating Credentials (Credential Management Part 2)

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

In a previous blog post I wrote about ways to securely configure your AWS access credentials when using the aws-sdk gem. This week I want to talk about a security best practice, credential rotation.

Did you know that AWS recommends that you rotate your access keys every 90 days?

Even if you are very careful with your AWS access credentials, you may find yourself in a situation where someone has gained access to your secrets. If you build your applications with a regular key rotation solution, then an out-of-bounds replacement of keys can be painless. In the heat of the moment when you are scrambling to replace compromised keys, this can be a life saver.

Rotating Credentials

The process for rotating credentials boils down to the following steps:

  • Generate new keys
  • Securely distribute keys to your applications
  • Ensure the applications refresh their keys
  • Disable the old access keys
  • Ensure everything still works
  • Delete the old access keys

For best effect, you should automate this process. If you have to do it by hand, the process will be much more error prone and you will likely do it less often. You can use the aws-sdk gem to do much of the work for you.

This simple example demonstrates how to generate a new key pair, disable old keys and then eventually delete the old keys. I inserted placeholders for where you should distribute your new keys and refresh your applications with the new keys.

iam = AWS::IAM.new

# create new set of access credentials
new_keys = iam.access_keys.create

# you should persist the new key pair somewhere secure to start with
new_keys.id # access key id
new_keys.secret # secret access key

## deploy the new keys to your applications now, make
## sure they pick up the new keys

# deactivate the old keys
old_keys = iam.access_keys['AKID12346789…'] # old access key id

## the old keys still exist, they are temporarily disabled, use
## this time to test your applications to ensure they are working

# if you are confident your applications are using the new keys
# you can then safely delete the old key pair

How you distribute your keys and refresh your application is going to be very specific to your own needs. Just be certain to test your applications before you delete your disabled keys. You can not restore them once they have been deleted.

For the next post in this series, I will write about credential providers and how the aws-sdk makes it easy for your applications to pick up new credentials without restarts or downtime. This can be very useful when you are rotating credentials.