Tag: Testing


Client Response Stubs

by Trevor Rowe | on | in Ruby | | Comments

We recently added client response stubs to the aws-sdk-core gem. Response stubbing disables network traffic and causes a client to return fake or stubbed data.

# no API calls are made
s3 = Aws::S3::Client.new(stub_responses: true)
s3.list_buckets.buckets.map(&:name)
#=> []

Custom Response Data

By default, stubbed responses return empty lists, empty maps, and placeholder scalars. These empty responses can be useful at times, but often you want to control the data returned.

s3.stub_responses(:list_buckets, buckets:[{name:'aws-sdk'}])
s3.list_buckets.buckets.map(&:name)
#=> ['aws-sdk']

Safe Stubbing

One of the common risks when writing tests with stub data is that the stub doesn’t match the shape of the actual response. You risk coding against stubs that provide methods that won’t exist out of your tests.

We resolve this issue by validating your stub data hash against the model of the API. An ArgumentError is raised when calling #stub_responses with invalid data.

s3.stub_resposnes(:list_buckets, buckets:['aws-sdk'])
#=> raises ArgumentError, "expected params[:buckets][0] to be a hash"

Stubbing Multiple Calls

By calling #stub_responses with an operation name and stub data, the client will serve that data for each call. You can specify multiple responses, and they will be used in sequence.

s3.stub_responses(:list_buckets, 
  { buckets:[{name:'aws-sdk'}] },
  { buckets:[{name:'aws-sdk', 'aws-sdk-2'}] }
)

s3.list_buckets.buckets.map(&:name)
#=> ['aws-sdk']

s3.list_buckets.buckets.map(&:name)
#=> ['aws-sdk', 'aws-sdk-2']

Stubbing Errors

In addition to stubbing response data, you can configure errors to raise. You can specify a service error by name, or you can provide an error object or class to raise.

# everything is broken
s3.stub_responses(:head_bucket, 
  'NotFound'
  Timeout::Error,
  RuntimeError.new('oops')
)

s3.head_bucket(bucket:'aws-sdk')
# raises Aws::S3::Errors::NotFound

s3.head_bucket(bucket:'aws-sdk')
# raises a new Timeout::Error

s3.head_bucket(bucket:'aws-sdk')
# raises RuntimeError.new('oops')

You can mix stubbed response data and errors. This approach is great when you want to test how well your code recovers from errors. 

Stubbing All Clients

The default config can be used to enable client stubbing globally. This can be very useful when writing tests to enable stubbing in your test helper once.

# stub everything
Aws.config[:stub_responses] = true

Give it a try and let us know what you think.

Using RSpec 3

by Trevor Rowe | on | in Ruby | | Comments

Using RSpec 3

I have been a long time user of RSpec and many of the Ruby projects I work with use RSpec as the primary testing framework. It provides an expressive specification DSL. As you may know, RSpec 3 is currently in the works. I have blogged a few times recently about using MiniTest. I decided I should also give RSpec some love.

RSpec 3 Changes

You can find an excellent summary of the changes and planned changes for RSpec 3 over here. Some of the primary changes:

  • No support for Ruby 1.8.6. You should strongly consider upgrading any projects using Ruby 1.8 to 2.0+.
  • Less magic by default, and RSpec 3 now provides a zero monkey patching mode.

Upgrading to RSpec 3 required very few changes for me. The primary difference is how I assert expectations. I now use the new expect helpers:

# using should helpers
obj_under_test.attribute.should eq('some-value')
lambda { obj_under_test.something_invalid }.should raise_error(...)

# using expect
expect(obj_under_test.attribute).to eq('some-value')
expect { obj_under_test.something_invalid }.to raise_error(...)

Using RSpec 3 Now

To try out the pre-release version of RSpec 3, get started by adding the following to your project’s Gemfile:

gem 'rspec', github: 'rspec/rspec'
gem 'rspec-core', github: 'rspec/rspec-core'
gem 'rspec-mocks', github: 'rspec/rspec-mocks'
gem 'rspec-expectations', github: 'rspec/rspec-expectations'
gem 'rspec-support', github: 'rspec/rspec-support'

I have been using RSpec 3 for a few months now and have found it to be very stable and enjoyable to work with. Happy Testing!

Using SimpleCov with Multiple Test Suites

by Trevor Rowe | on | in Ruby | | Comments

It can be helpful to generate coverage reports when testing software. While coverage reports do not guarantee well tested software, they can highlight were test coverage is lacking. This is especially true for legacy, or un-tested projects.

Recently I ran into a situation where I wanted to generate a coverage report, but the project used multiple test frameworks. One framework is used for unit testing, the other for integration testing. Fortunately, SimpleCov makes it easy to merge coverage reports.

Basic SimpleCov Usage

To use SimpleCov, you normally require simplecov and then call Simplecov.start with configuration options. Here is a typical configuration from an RSpec test helper:

require 'simplecov'
SimpleCov.start do
  # configure SimpleCov
end

require 'rspec'
require 'my-library'

With multiple test frameworks, you might find yourself wanting to duplicate this configuration code. Instead, move the shared SimpleCov configuration to a .simplecov file. This file is parsed by Ruby. I like to conditionally run SimpleCov when testing based on an environment variable. Here is an example:

# in .simplecov
if ENV['COVERAGE']
  SimpleCov.start do
    # configure SimpleCov
  end
end

With this file, my test helpers are reduced to:

require 'simplecov'
require 'rspec'
require 'my-library'

Lastly, when using Rake, I like to make it possible to generate coverage reports for unit tests, integration tests, or both. I accomplish this by grouping my test tasks:

task 'test:unit' do
   # ...
end

desc 'Runs integration tests'
task 'test:integration' do
   # ...
end

desc 'Runs unit and integration tests'
task 'test' => ['test:unit', 'test:integration']

desc 'Generate coverage report'
task 'coverage' do
  ENV['COVERAGE'] = true
  rm_rf "coverage/"
  task = Rake::Task['test']
  task.reenable
  task.invoke
end

Good luck and have fun testing!

Running Your Minitest Unit Test Suite

by Trevor Rowe | on | in Ruby | | Comments

I have blogged a few times recently about Minitest. With Minitest you need to chose how you will execute your tests. When using other tools, like Rspec, there is a bundled test runner.

$ rspec
............

Finished in 0.03324 seconds
12 examples, 0 failures

Minitest does not provide a test runner as a command line script. One common workaround is to use minitest/autorun.

# inside test/my_class_test.rb
require 'minitest/autorun'

class MyClassTest < Minitest::Test
  ...
end

Now you cn execute your tests using the ruby command:

$ ruby test/my_class_test.rb

Minitest uses very little Ruby magic, but this is one case it indulges. minitest/autorun uses #at_exit to execute tests. This makes it possible for you to specify many test files and have them all run at the end.

Instead of supplying a list of test files at the command line, I prefer to setup a rake task that executes my test files. Here is a simple Rake task file that creates two tasks. The first task executes my unit tests. The second task executes my unit tests while also generating a coverage report.

# inside tasks/test.rake
require 'rake/testtask'

Rake::TestTask.new do |t|
  t.libs.push 'test'
  t.pattern = 'test/**/*_test.rb'
  t.warning = true
  t.verbose = true
end

task :default => :test

desc 'Generates a coverage report'
task :coverage do
  ENV['COVERAGE'] = 'true'
  Rake::Task['test'].execute
end

Next, I place the following line at the top of each test file. This allows me to still run the test files from the command line, while loading any shared testing helpers.

require 'test_helper'

Finally my test/test_helper.rb file looks like this:

if ENV['COVERAGE']
  require 'simplecov'
  SimpleCov.start do
    add_filter 'test'
    command_name 'Mintest'
  end
end

require 'minitest/autorun'
require 'my-library'

Running rake from the command line will now run my unit tests. rake coverage will generate a coverage report. This setup is simple and the tests run fast. I hope this helps.

Have fun and keep on Testing!

From Minitest::Spec to Minitest::Test

by Trevor Rowe | on | in Ruby | | Comments

In a previous blog post, I introduced Minitest from the perspective of RSpec. Some Minitest users prefer to avoid the specification style of Minitest::Spec. Instead they use Minitest::Test. It’s closer to the metal and uses a more vanilla Ruby syntax.

Here is an example spec file using Minitest::Spec:

require 'spec_helper'

describe MyClass do
  describe '#some_method' do
    it 'returns a string' do
      MyClass.new.some_method.must_be_kind_of(String)
    end
  end
end

Converting this to use Minitest::Test looks like:

require 'test_helper'

class MyClassTest < Minitest::Test
  def test_some_method_returns_a_string
    assert_kind_of String, MyClass.new.some_method
  end
end

Some key differences:

  • Assertions are instance methods provided by the test class. Instead of calling magic methods added to Object, you pass the object under test into the assertion. Example:

    value.must_be_kind_of(String)
    

    becomes:

    assert_kind_of(String, value) 
    
  • There is no DSL for defining test cases. The it method is removed. Instead, all methods prefixed with test_ are executed as test cases.

  • Nesting describe blocks is a useful technique for grouping specs. You can still do this, but you have to nest test classes.

    class MyClassTest < Minitest::Test
        class SubClassTest < Minitest::Test
        ...
      end
    end
    

I don’t feel as strongly as others about using vanilla Minitest::Test over Minitest::Spec. I personally find the specs easier to read, but that may be due to my experience with RSpec. You may have a different experience based on your testing background.

Happy Testing!

From RSpec to Minitest

by Trevor Rowe | on | in Ruby | | Comments

One of my favorite aspects of working with Ruby is how natural it is to write tests for. The Ruby community does an excellent job of encouraging authors to produce well tested code. There is a plethora of well supported tools to choose from. I like to joke that new Ruby developers write Micro test frameworks instead of "Hello World!".

Much of the Ruby code I have maintained uses RSpec. Lately I have been spending some time with Minitest. You may have worked with Minitest — it ships as part of the Ruby standard library.

Why bother learning another testing framework when your current tool suits your needs? My answer is why not? It’s always good to expand your horizons. I have found learning a new testing tool expands my ability to write good tests. I pick up new patterns and the context switch forces to me to question my standard testing approaches. As a result, I tend to write better tests.

Minitest::Spec

Minitest::Spec does a great job of bridging the gap between RSpec-style specifications and Minitest-style unit tests.

Here is an example RSpec test file:

require 'spec_helper'

describe MyClass do
  describe '#some_method' do
    it 'returns a a string' do
      MyClass.new.should be_a(String)
    end
  end
end

And the same thing using Minitest::Spec:

require 'test_helper'

describe MyClass do
  describe '#some_method' do
    it 'returns a string' do
      MyClass.new.some_method.must_be_kind_of(String)
    end
  end
end

Matchers

The primary difference above is how you make assertions. RSpec-style should matchers can be converted to Minitest expectations with ease. The table below gives a few examples.

RSpec Matcher Minitest Matcher
obj.should be(value) obj.must_be(value)
obj.should be_empty obj.must_be_empty
obj.should be(nil) obj.must_be_nil
obj.should equal(value) obj.must_equal(value)
lambda { … }.should raise_error(ErrorClass) lambda { … }.must_raise(ErrorClass)

See the Minitest api documentation for more expectations.

Mocks

Mocks (and stubs) are where the two testing libraries differ the most. RSpec provides doubles; Minitest provides a Mock class.

@mock = Minitest::Mock.new

You can set expectations about what messages the mock should receive. You name the method to expect, what to return from the mock and then optionally the arguments the method should receive. Given I need a mock for a user, and I expect the user’s delete method will be called, I could do the following:

user = Minitest::Mock.new
user.expect(:delete, true) # returns true, expects no args

UserDestoyer.new.delete_user(user)

assert user.verify

Calling #verify is necessary for the mock to enforce the expectations. RSpec makes this a little easier, but its not a huge adjustment.

Stubs

Stubs are pretty straight forward. Unlike RSpec, the stub lasts only till the end of the block. You also cannot stub methods that don’t exist yet.

Time.stub :now, Time.at(0) do
  assert obj_under_test.stale?
end

Summary

Minitest works well and I’m impressed by how fast it run tests. Coming from RSpec you may find yourself missing features. Instead of trying to find exact replacements, consider using plain old Ruby solutions instead. You may find you write better tests. I still enjoy working with RSpec and appreciate it for its strengths, but you should also consider giving Minitest a spin.

Static Service Client Facades

by Jeremy Lindblom | on | in PHP | | Comments

Version 2.4 of the AWS SDK for PHP adds the ability to enable and use static client facades. These "facades" provide an easy, static interface to service clients available in the service builder. For example, when working with a normal client instance, you might have code that looks like the following:

// Get the configured S3 client from the service builder
$s3 = $aws->get('s3');

// Execute the CreateBucket command using the S3 client
$s3->createBucket(array('Bucket' => 'your-new-bucket-name'));

With client facades enabled, you can also accomplish this with the following code:

// Execute the CreateBucket command using the S3 client
S3::createBucket(array('Bucket' => 'your-new-bucket-name'));

Enabling and using client facades

To enable static client facades to be used in your application, you must use the AwsCommonAws::enableFacades method when you setup the service builder.

// Include the Composer autoloader
require 'vendor/autoload.php';

// Instantiate the SDK service builder with my config and enable facades
$aws = Aws::factory('/path/to/my_config.php')->enableFacades();

This will setup the client facades and alias them into the global namespace. After that, you can use them anywhere to have more simple and expressive code for interacting with AWS services.

// List current buckets
echo "Current Buckets:n";
foreach (S3::getListBucketsIterator() as $bucket) {
    echo "{$bucket['Name']}n";
}

$args = array('Bucket' => 'your-new-bucket-name');
$file = '/path/to/the/file/to/upload.jpg';

// Create a new bucket and wait until it is available for uploads
S3::createBucket($args) and S3::waitUntilBucketExists($args);
echo "nCreated a new bucket: {$args['Bucket']}.n";

// Upload a file to the new bucket
$result = S3::putObject($args + array(
    'Key'  => basename($file),
    'Body' => fopen($file, 'r'),
));
echo "nCreated a new object: {$result['ObjectURL']}n";

You can also mount the facades into a namespace other than the global namespace. For example, if you want to make the client facades available in the "Services" namespace, you can do the following:

Aws::factory('/path/to/my_config.php')->enableFacades('Services');

$result = ServicesDynamoDb::listTables();

Why use client facades?

The use of static client facades is completely optional. We included this feature in the SDK in order to appeal to PHP developers who prefer static notation or who are familiar with PHP frameworks like CodeIgnitor, Laravel, or Kohana where this style of method invocation is common.

Though using static client facades has little real benefit over using client instances, it can make your code more concise and prevent you from having to inject the service builder or client instance into the context where you need the client object. This can make your code easier to write and understand. Whether or not you should use the client facades is purely a matter of preference.

How client facades work in the AWS SDK for PHP is similar to how facades work in the Laravel 4 Framework. Even though you are calling static classes, all of the method calls are proxied to method calls on actual client instances—the ones stored in the service builder. This means that the usage of the clients via the client facades can still be mocked in your unit tests, which removes one of the general disadvantages to using static classes in object-oriented programming. For information about how to test code that uses client facades, please see the Testing Code that Uses Client Facades section of the AWS SDK for PHP User Guide.

Though we are happy to offer this new feature, we we don’t expect you to change all of your code to use the static client facades. We are simply offering it as an alternative that may be more convenient or familiar to you. We still recommend using client instances as you have in the past and support the use of dependency injection. Be sure to let us know in the comments if you like this new feature and if you plan on using it.

Stubbing AWS Responses

by Trevor Rowe | on | in Ruby | | Comments

I come across questions frequently about how to test an application that uses the AWS SDK for Ruby (aws-sdk gem). Testing an application that makes use of an external service is always tricky. One technique is to stub the client-level responses returned by the SDK.

AWS.stub!

Calling the AWS.stub! method in your ruby process configures the client classes (e.g. AWS::EC2::Client) to stub their responses when called. They stop short of making a live HTTP request and instead return an empty response.

AWS.stub!

instance_ids = AWS::EC2.new.instances.map(&:id)
instance_ids #=> will always be empty in this example, not HTTP request made

Under the covers, the code example is constructing an AWS::EC2::Client object and is calling the #describe_instances method. AWS.stub! causes the client to return an empty response that looks like a normal response with a few differences:

  • Lists are returned as empty arrays
  • Maps are returned as empty hashes
  • Numeric values are always zero
  • Dates are returned as now

Localized Stubbing

Calling AWS.stub! is the same as calling AWS.config(:stub_requests => true). You can use this configuration option with any constructor that accepts configuration.

stub_ec2 = AWS::EC2.new(:stub_requests => true)
real_ec2 = AWS::EC2.new

Customizing the Responses

In addition to getting empty responses, you can access the stubbed responses and populate them with fake data.

AWS.stub!

ec2 = AWS::EC2::Client.new
resp = ec2.stub_for(:describe_instances)
resp.data[:reservation_set] = [...]

# now calling ec2.describe_instances will return my fake data
ec2.describe_instances
#=> { :reservation_set => [...] } 

There are two methods you can use here:

  • #stub_for(operation_name)
  • #new_stub_for(operation_name)

The first method, #stub_for, returns the same stubbed response every time. It is the default response for that operation for that client object (not shared between instances). The second method, #new_stub_for, generates a new response. This is useful if you need to stub the client to return different data from multiple calls. This is common for paged responses.

Not a Mock

Unfortunately, this approach stubs responses, but it does not mock AWS services. If I used a stubbed AWS::DynamoDB::Client and call #put_item, I will not be able to get the data back. There are a number of third-party libraries that attempt to provide local service mocks. These can be helpful when you are trying to run local tests without hitting the network.

Faster AWS Tests with VCR

by Loren Segal | on | in Ruby | | Comments

Performing integration testing against a live AWS service can often be inefficient since you are forced to make requests across the network for every test. Fortunately, there are tools that you can use to help reduce the amount of network traffic your tests require and make them run faster in the process. We will look at how to do this with the VCR gem available for Ruby.

What is VCR?

VCR is a library that is able to record HTTP network traffic over the wire and then replay those requests locally. Since recording occurs only once, all subsequent executions of your tests do not need to hit the network, and will likely be much faster.

Installing and Using VCR

To install VCR, you need to install two gems: the VCR gem, and the FakeWeb HTTP mocking library:

gem install vcr fakeweb

To use VCR, you simply configure the library with the path on disk to store fixture data (the recorded network traffic), as well as the HTTP mocking library to use (we use fakeweb as the HTTP mocking library):

require 'vcr'

VCR.configure do |c|
  c.cassette_library_dir = 'fixtures/vcr_cassettes'
  c.hook_into :fakeweb
end

After you’ve configured VCR, you can use it with a simple VCR.use_cassette method that wraps and records a block of arbitrary network code:

VCR.use_cassette('some_cassette') do
  # Perform some network transfer
  Net::HTTP.get_response(URI('http://example.com/'))
end

On subsequent executions of this code, VCR uses the data loaded from the fixtures on disk located in fixtures/vcr_cassettes/some_cassette.yaml.

Testing AWS with VCR

Now that you know how to configure and use VCR, it’s easy to plug this library into your existing tests that use the AWS SDK for Ruby to record and replay requests over the network. In our case, we will use the RSpec testing framework, but you can use any testing framework you like.

Test Setup

The top of your test file or helper file requires a bit of configuration. For one, you need to configure VCR (as shown previously), but you also need to configure RSpec to wrap all of the test blocks inside of the VCR.use_cassette method. If you are using RSpec for your testing, you can add the following helper/configuration code to any test suite to get VCR to automatically record and replay your tests for improved speed:

require 'rspec'
require 'aws-sdk'
require 'vcr'

VCR.configure do |c|
  c.cassette_library_dir = 'fixtures/vcr_cassettes'
  c.hook_into :fakeweb
end

# Fixes a missing attribute in Fakeweb's stubbed HTTP client
class FakeWeb::StubSocket; attr_accessor :read_timeout end

RSpec.configure do |c|
  c.around(:each) do |example|
    VCR.use_cassette(example.metadata[:full_description]) do
      example.run
    end
  end
end

Test Example

Now that you’ve set up your test environment, let’s see how it affects your tests. Say, for example, you had a test that uploaded a bunch of files to Amazon S3 and tried to read them back out. The test might look something like this:

describe 'Uploading files to S3' do
  let(:s3) { AWS::S3.new }
  let(:bucket) { s3.buckets['test_s3_upload'] }
  before { s3.buckets.create(bucket.name) }

  it 'uploads multiple files to S3 and reads them out' do
    # Uploads items
    25.times do |i|
      bucket.objects["file#{i}.txt"].write("DATA")
    end

    # Reads items back
    25.times do |i|
      bucket.objects["file#{i}.txt"].read.should == "DATA"
    end
  end
end

Depending on latency, running this test the first time (when VCR is recording) could take anywhere from 5 to 15 seconds. For instance:

$ rspec test_s3.rb 
.

Finished in 14.18 seconds
1 example, 0 failures

But VCR has now recorded our data inside fixtures/vcr_cassettes, and your subsequent execution will be much faster:

$ rspec test_s3.rb
.

Finished in 0.48604 seconds
1 example, 0 failures

Using VCR, this test just went from taking 14 seconds down to 0.5 seconds— MUCH faster. This is because VCR loaded the server-side response from a local file on disk instead of sending the request over the wire.

Clear Your Fixture Cache Often

Although VCR can make your tests much faster, you do lose out on some of the accuracy provided by testing directly against live AWS services. VCR is useful to temporarily cache test results in fast development cycles, but it should not be a complete replacement to full-on integration testing.

One easy way to use the power of a library like VCR and still have the accuracy of full integration testing is to delete your fixtures/vcr_cassettes cache often. You may even want to ignore it from your repository in order to emphasize that the data is, in fact, temporary. That way you will still have fast tests most of the time but still get up-to-date responses from the service at regular intervals during development.

Conclusion

Testing with VCR can make your tests much faster and reduce your costs when testing against AWS services. Best of all, adding this performance optimization is almost fully transparent to your tests, uses actual data from the AWS services, and is simpler than manually creating fixtures that mock the server responses yourself. Try it out and see how much time you can shave off of your tests!