Category: Ruby


Ruby SDK Version 2 and Memoization

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

Version 1 of the AWS SDK for Ruby (aws-sdk gem) provides a higher-level abstraction for working with AWS resources. These resources can be used to get information about AWS objects and operate on them as well. For example:

AWS.ec2.instances.each do |i|
  puts i.instance_id + ' => ' + i.status.to_s
end

The problem with the example above is if you do not enable memoization, it will make n + 1 requests to Amazon EC2. The first request retries a list of instances, and then it makes one more request per instance to fetch the status. An experienced AWS developer will know that all of this information can be retrieved in a single call to DescribeInstances with Amazon EC2.

To address this issue, the v1 Ruby SDK introduced a feature called memoization which allows the SDK to used cached values inside a block.

# now only one request is made
AWS.memoize do
  AWS.ec2.instances.each do |i|
    puts i.instance_id + ' => ' + i.status.to_s
  end
end 

For more background information you can read this blog post.

Memoization is Not Obvious

If you are unaware of this feature, your code will still execute, and will produce a correct response. Unfortunately, it is also likely that a large number of unnecessary requests will be made. These requests can significantly increase the execution time and can result in an account getting throttled. This usually results in a frustrating search for why extra requests are being made, which finally leads to #memoize.

Memoization Removed in V2

In version 2 of the Ruby SDK (aws-sdk-core gem), there is no Aws.memoize method. Instead we chose a design that removes the need to provide this hook.

  • Response objects have a #data attribute. Accessing response data will never trigger another request.
  • The upcoming higher-level resource abstractions are also going to provide access to the data for a resource and an explicit #refresh! method that will refresh the data on demand.

If you haven’t had a chance to take the version 2 Ruby SDK for a spin, I encourage you to try it out. It installs via a separate gem (aws-sdk-core) and uses a separate namespace (Aws vs. AWS) so that you can use it in the same process.

Also, please share any feedback or ideas!

Parameter Validation

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

One of my favorite features of version 2 of the AWS SDK for Ruby (aws-sdk-core gem) is the new parameter validation system. One of the challenges in using an API library is understanding how to specify request parameters. During development stages it is common to make mistakes and to generate errors.

Using the version 1 Ruby SDK (aws-sdk gem), I might want to use the ‘2012-08-10’ Amazon DynamoDB API version to create a table. Without knowing the all of the required parameters I might try something like this:

ddb = AWS::DynamoDB::Client.new(api_version: '2012-08-10')
ddb.create_table(
  table: 'name',
  provisioned_throughput: { read_capacity_units: 'Over 9000!' }
)
#=> raises AWS::Core::OptionGrammar::FormatError: expected integer value for key read_capacity_units

Oops! It’s easy enough to correct that error, but there are more validation errors waiting to raise.

Version 2 Validation

Version 2 of the Ruby SDK aggregates all validation errors and raises them in a single pass.

ddb = Aws::DynamoDB.new(api_version: '2012-08-10')
ddb.create_table(
  table: 'name',
  provisioned_throughput: { read_capacity_units: 'Over 9000!' }
)

# raises the following:

ArgumentError: parameter validator found 6 errors:
  - missing required parameter params[:attribute_definitions]
  - missing required parameter params[:table_name]
  - missing required parameter params[:key_schema]
  - unexpected value at params[:table]
  - missing required parameter params[:provisioned_throughput][:write_capacity_units]
  - expected params[:provisioned_throughput][:read_capacity_units] to be an integer

Each error gives the context as well as information on what needs to be corrected before the request can be sent. I can now correct all of the validation errors in a single pass.

Formatting Examples

Even better, the V2 API documentation gives formatting examples of how to call each operation. Here is a link to the create table method docs with an example.

We’ve been getting a lot of great customer feedback on our GitHub project for the version 2 Ruby SDK. Keep it coming! We would love to hear what you think is missing.

Flexible Gem Dependencies

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

Version 1 of the AWS SDK for Ruby (aws-sdk gem) depends on Nokogiri and JSON. While these are robust and performant libraries, there are multiple reasons why a developer may not want to be locked into these dependencies:

  • I might want to use pure Ruby solutions that do not require native extensions for maximum portability.
  • I might want to use Oj/Ox for performance reasons
  • I might require more flexibility in choosing the version of dependent gem.

MultiJSON and MultiXML

In version 2 of the Ruby SDK (aws-sdk-core gem), we depend on the multi_json and multi_xml gems. By default these gems use the fastest available XML and JSON libraries available. If you have not explicitly installed any libraries, then those available in the Ruby standard library will be used. These are REXML and JSON.

I very much enjoy Oj and Ox. They are fast XML and JSON parsers/serializers. They are written in c, but have no external dependencies. You can enable these libraries by adding them to your Gemfile as follows:

gem 'oj'
gem 'ox'

MultiJSON and MultiXML will choose these by default. The same approach works with Nokogiri. Just add gem 'nokogiri' to your Bundler Gemfile and you are set.

We are working hard to ensure the SDK depends on as few gems as possible and to ensure it works on as many platforms as possible. If you have any feedback or suggestions on the version 2 Ruby SDK, please let us know!

Using RSpec 3

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

Using RSpec 3

I have been a long time user of RSpec and many of the Ruby projects I work with use RSpec as the primary testing framework. It provides an expressive specification DSL. As you may know, RSpec 3 is currently in the works. I have blogged a few times recently about using MiniTest. I decided I should also give RSpec some love.

RSpec 3 Changes

You can find an excellent summary of the changes and planned changes for RSpec 3 over here. Some of the primary changes:

  • No support for Ruby 1.8.6. You should strongly consider upgrading any projects using Ruby 1.8 to 2.0+.
  • Less magic by default, and RSpec 3 now provides a zero monkey patching mode.

Upgrading to RSpec 3 required very few changes for me. The primary difference is how I assert expectations. I now use the new expect helpers:

# using should helpers
obj_under_test.attribute.should eq('some-value')
lambda { obj_under_test.something_invalid }.should raise_error(...)

# using expect
expect(obj_under_test.attribute).to eq('some-value')
expect { obj_under_test.something_invalid }.to raise_error(...)

Using RSpec 3 Now

To try out the pre-release version of RSpec 3, get started by adding the following to your project’s Gemfile:

gem 'rspec', github: 'rspec/rspec'
gem 'rspec-core', github: 'rspec/rspec-core'
gem 'rspec-mocks', github: 'rspec/rspec-mocks'
gem 'rspec-expectations', github: 'rspec/rspec-expectations'
gem 'rspec-support', github: 'rspec/rspec-support'

I have been using RSpec 3 for a few months now and have found it to be very stable and enjoyable to work with. Happy Testing!

Ruby 2.0 on AWS OpsWorks

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

We are pleased to announce that AWS OpsWorks now supports Ruby 2.0. Simply select the Ruby version you want, your Rails stack – Passenger or Unicorn, the RubyGems version, and whether you want to use Bundler. Then deploy your app from your chosen repository – Git, Subversion, or bundles on S3. You can get started with a few clicks in the AWS Management console.

AWS SDK for Ruby v1.30.0

by Loren Segal | on | in Ruby | Permalink | Comments |  Share

Yesterday afternoon, we released a new version of the AWS SDK for Ruby (aws-sdk RubyGem) version v1.30.0. This release:

You can view the release notes here.

Using SimpleCov with Multiple Test Suites

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

It can be helpful to generate coverage reports when testing software. While coverage reports do not guarantee well tested software, they can highlight were test coverage is lacking. This is especially true for legacy, or un-tested projects.

Recently I ran into a situation where I wanted to generate a coverage report, but the project used multiple test frameworks. One framework is used for unit testing, the other for integration testing. Fortunately, SimpleCov makes it easy to merge coverage reports.

Basic SimpleCov Usage

To use SimpleCov, you normally require simplecov and then call Simplecov.start with configuration options. Here is a typical configuration from an RSpec test helper:

require 'simplecov'
SimpleCov.start do
  # configure SimpleCov
end

require 'rspec'
require 'my-library'

With multiple test frameworks, you might find yourself wanting to duplicate this configuration code. Instead, move the shared SimpleCov configuration to a .simplecov file. This file is parsed by Ruby. I like to conditionally run SimpleCov when testing based on an environment variable. Here is an example:

# in .simplecov
if ENV['COVERAGE']
  SimpleCov.start do
    # configure SimpleCov
  end
end

With this file, my test helpers are reduced to:

require 'simplecov'
require 'rspec'
require 'my-library'

Lastly, when using Rake, I like to make it possible to generate coverage reports for unit tests, integration tests, or both. I accomplish this by grouping my test tasks:

task 'test:unit' do
   # ...
end

desc 'Runs integration tests'
task 'test:integration' do
   # ...
end

desc 'Runs unit and integration tests'
task 'test' => ['test:unit', 'test:integration']

desc 'Generate coverage report'
task 'coverage' do
  ENV['COVERAGE'] = true
  rm_rf "coverage/"
  task = Rake::Task['test']
  task.reenable
  task.invoke
end

Good luck and have fun testing!

Running Your Minitest Unit Test Suite

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

I have blogged a few times recently about Minitest. With Minitest you need to chose how you will execute your tests. When using other tools, like Rspec, there is a bundled test runner.

$ rspec
............

Finished in 0.03324 seconds
12 examples, 0 failures

Minitest does not provide a test runner as a command line script. One common workaround is to use minitest/autorun.

# inside test/my_class_test.rb
require 'minitest/autorun'

class MyClassTest < Minitest::Test
  ...
end

Now you cn execute your tests using the ruby command:

$ ruby test/my_class_test.rb

Minitest uses very little Ruby magic, but this is one case it indulges. minitest/autorun uses #at_exit to execute tests. This makes it possible for you to specify many test files and have them all run at the end.

Instead of supplying a list of test files at the command line, I prefer to setup a rake task that executes my test files. Here is a simple Rake task file that creates two tasks. The first task executes my unit tests. The second task executes my unit tests while also generating a coverage report.

# inside tasks/test.rake
require 'rake/testtask'

Rake::TestTask.new do |t|
  t.libs.push 'test'
  t.pattern = 'test/**/*_test.rb'
  t.warning = true
  t.verbose = true
end

task :default => :test

desc 'Generates a coverage report'
task :coverage do
  ENV['COVERAGE'] = 'true'
  Rake::Task['test'].execute
end

Next, I place the following line at the top of each test file. This allows me to still run the test files from the command line, while loading any shared testing helpers.

require 'test_helper'

Finally my test/test_helper.rb file looks like this:

if ENV['COVERAGE']
  require 'simplecov'
  SimpleCov.start do
    add_filter 'test'
    command_name 'Mintest'
  end
end

require 'minitest/autorun'
require 'my-library'

Running rake from the command line will now run my unit tests. rake coverage will generate a coverage report. This setup is simple and the tests run fast. I hope this helps.

Have fun and keep on Testing!

AWS re:Invent 2013 Talk Now Available

by Loren Segal | on | in Ruby | Permalink | Comments |  Share

This week, talks from AWS re:Invent 2013 started to become available through YouTube and SlideShare. If you were at re:Invent this year, you may have seen Trevor and I give a talk on the new AWS SDK for Ruby V2. If you missed it, or if you just want to check it out again, the talk is linked below and ready for viewing.

View Slides on SlideShare

V2 is Moving Forward

We are still working on V2 of the SDK. In fact, we just recently polished and committed pagination support on our GitHub repository.

More Resources for Frontend JavaScript Developers

Finally, if you happen to do any JavaScript development in the frontend of your web applications (and who doesn’t?), I gave a talk highlighting our new AWS SDK for JavaScript, which is now available directly in browsers and mobile devices. If you are interested, you can see the talk for that below:

View Slides on SlideShare

From Minitest::Spec to Minitest::Test

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

In a previous blog post, I introduced Minitest from the perspective of RSpec. Some Minitest users prefer to avoid the specification style of Minitest::Spec. Instead they use Minitest::Test. It’s closer to the metal and uses a more vanilla Ruby syntax.

Here is an example spec file using Minitest::Spec:

require 'spec_helper'

describe MyClass do
  describe '#some_method' do
    it 'returns a string' do
      MyClass.new.some_method.must_be_kind_of(String)
    end
  end
end

Converting this to use Minitest::Test looks like:

require 'test_helper'

class MyClassTest < Minitest::Test
  def test_some_method_returns_a_string
    assert_kind_of String, MyClass.new.some_method
  end
end

Some key differences:

  • Assertions are instance methods provided by the test class. Instead of calling magic methods added to Object, you pass the object under test into the assertion. Example:

    value.must_be_kind_of(String)
    

    becomes:

    assert_kind_of(String, value) 
    
  • There is no DSL for defining test cases. The it method is removed. Instead, all methods prefixed with test_ are executed as test cases.

  • Nesting describe blocks is a useful technique for grouping specs. You can still do this, but you have to nest test classes.

    class MyClassTest < Minitest::Test
        class SubClassTest < Minitest::Test
        ...
      end
    end
    

I don’t feel as strongly as others about using vanilla Minitest::Test over Minitest::Spec. I personally find the specs easier to read, but that may be due to my experience with RSpec. You may have a different experience based on your testing background.

Happy Testing!