Category: Ruby


Faster AWS Tests with VCR

by Loren Segal | on | in Ruby | Permalink | Comments |  Share

Performing integration testing against a live AWS service can often be inefficient since you are forced to make requests across the network for every test. Fortunately, there are tools that you can use to help reduce the amount of network traffic your tests require and make them run faster in the process. We will look at how to do this with the VCR gem available for Ruby.

What is VCR?

VCR is a library that is able to record HTTP network traffic over the wire and then replay those requests locally. Since recording occurs only once, all subsequent executions of your tests do not need to hit the network, and will likely be much faster.

Installing and Using VCR

To install VCR, you need to install two gems: the VCR gem, and the FakeWeb HTTP mocking library:

gem install vcr fakeweb

To use VCR, you simply configure the library with the path on disk to store fixture data (the recorded network traffic), as well as the HTTP mocking library to use (we use fakeweb as the HTTP mocking library):

require 'vcr'

VCR.configure do |c|
  c.cassette_library_dir = 'fixtures/vcr_cassettes'
  c.hook_into :fakeweb
end

After you’ve configured VCR, you can use it with a simple VCR.use_cassette method that wraps and records a block of arbitrary network code:

VCR.use_cassette('some_cassette') do
  # Perform some network transfer
  Net::HTTP.get_response(URI('http://example.com/'))
end

On subsequent executions of this code, VCR uses the data loaded from the fixtures on disk located in fixtures/vcr_cassettes/some_cassette.yaml.

Testing AWS with VCR

Now that you know how to configure and use VCR, it’s easy to plug this library into your existing tests that use the AWS SDK for Ruby to record and replay requests over the network. In our case, we will use the RSpec testing framework, but you can use any testing framework you like.

Test Setup

The top of your test file or helper file requires a bit of configuration. For one, you need to configure VCR (as shown previously), but you also need to configure RSpec to wrap all of the test blocks inside of the VCR.use_cassette method. If you are using RSpec for your testing, you can add the following helper/configuration code to any test suite to get VCR to automatically record and replay your tests for improved speed:

require 'rspec'
require 'aws-sdk'
require 'vcr'

VCR.configure do |c|
  c.cassette_library_dir = 'fixtures/vcr_cassettes'
  c.hook_into :fakeweb
end

# Fixes a missing attribute in Fakeweb's stubbed HTTP client
class FakeWeb::StubSocket; attr_accessor :read_timeout end

RSpec.configure do |c|
  c.around(:each) do |example|
    VCR.use_cassette(example.metadata[:full_description]) do
      example.run
    end
  end
end

Test Example

Now that you’ve set up your test environment, let’s see how it affects your tests. Say, for example, you had a test that uploaded a bunch of files to Amazon S3 and tried to read them back out. The test might look something like this:

describe 'Uploading files to S3' do
  let(:s3) { AWS::S3.new }
  let(:bucket) { s3.buckets['test_s3_upload'] }
  before { s3.buckets.create(bucket.name) }

  it 'uploads multiple files to S3 and reads them out' do
    # Uploads items
    25.times do |i|
      bucket.objects["file#{i}.txt"].write("DATA")
    end

    # Reads items back
    25.times do |i|
      bucket.objects["file#{i}.txt"].read.should == "DATA"
    end
  end
end

Depending on latency, running this test the first time (when VCR is recording) could take anywhere from 5 to 15 seconds. For instance:

$ rspec test_s3.rb 
.

Finished in 14.18 seconds
1 example, 0 failures

But VCR has now recorded our data inside fixtures/vcr_cassettes, and your subsequent execution will be much faster:

$ rspec test_s3.rb
.

Finished in 0.48604 seconds
1 example, 0 failures

Using VCR, this test just went from taking 14 seconds down to 0.5 seconds— MUCH faster. This is because VCR loaded the server-side response from a local file on disk instead of sending the request over the wire.

Clear Your Fixture Cache Often

Although VCR can make your tests much faster, you do lose out on some of the accuracy provided by testing directly against live AWS services. VCR is useful to temporarily cache test results in fast development cycles, but it should not be a complete replacement to full-on integration testing.

One easy way to use the power of a library like VCR and still have the accuracy of full integration testing is to delete your fixtures/vcr_cassettes cache often. You may even want to ignore it from your repository in order to emphasize that the data is, in fact, temporary. That way you will still have fast tests most of the time but still get up-to-date responses from the service at regular intervals during development.

Conclusion

Testing with VCR can make your tests much faster and reduce your costs when testing against AWS services. Best of all, adding this performance optimization is almost fully transparent to your tests, uses actual data from the AWS services, and is simpler than manually creating fixtures that mock the server responses yourself. Try it out and see how much time you can shave off of your tests!

Credential Management – Part 1

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

When using AWS, it is important to keep your access credentials secure. It can be challenging to make your credentials available to your application securely. The AWS SDK for Ruby provides a number of helpful interfaces for configuring your credentials that help you keep your secrets safe.

This blog post focuses on securely configuring the aws-sdk gem with your credentials. I will follow up with additional blog posts about rotating credentials, and using roles for instances.

Credential Configuration

The aws-sdk gem requires that you provide your access credentials before making a request. The gem tries to locate your credentials in a number of default locations. If it is unable to find your credentials, it raises an error. The locations it searches are:

  • AWS.config
  • ENV
  • EC2 instance metadata
  • Rails Configuration (RAILS_ROOT/config/aws.yml)

AWS.config

You can use AWS.config to statically configure your credentials for all AWS requests. Simply call AWS.config with your :access_key_id and :secret_access_key (you may also provide a :session_token).

require 'aws-sdk'

AWS.config(:access_key_id => '...', :secret_access_key => '...')

Alternatively, you may provide credentials directly to the service interfaces. This can be helpful if you are working with multiple sets of credentials for multiple AWS accounts (or IAM users).

s3 = AWS::S3.new(:access_key_id => '...', :secret_access_key => '...')    

As tempting as it might be, you should never put access credentials in source code. This makes your secrets available to anyone with access to your source code. It also creates a maintenance burden when you need to rotate your credentials. There are many alternatives, including loading credentials from a configuration file that is not tracked with source control.

require 'aws-sdk'
require 'yaml'

# configure aws-sdk gem from a yaml configuration file
AWS.config(YAML.load_file(path_to_configuration_file))

In the preceding example, the YAML file might look like this:

:access_key_id: 'YOUR_ACCESS_KEY_ID'
:secret_access_key: 'YOUR_SECRET_ACCESS_KEY'

ENV

You can alternatively provide credentials to your application via ENV. By default, the aws-sdk gem searches ENV using two different prefixes for your keys (‘AWS’ and ‘AMAZON’).

ENV['AWS_ACCESS_KEY_ID']
ENV['AWS_SECRET_ACCESS_KEY']
ENV['AWS_SESSION_TOKEN']

EC2 Instance Metadata

The aws-sdk gem supports loading credentials from the instance metadata service on Amazon EC2 instances. If you have started your instance using an AWS IAM instance profile, this will just work, no configuration required. I will dive into this deeper in a followup blog post.

Credential Loading in Rails (RAILS_ROOT/config/aws.yml)

If you are using the aws-sdk gem in a Rails application, the gem attempts to load credentials from RAILS_ROOT/config/aws.yml. This file is ERB-parsed and then loaded by YAML. This file should be formatted in the same way as RAILS_ROOT/config/database.yml. It should have one top-level entry for the Rails.env you are running.

# RAILS_ROOT/config/aws.yml
development:
  access_key_id: DEV_ACCESS_KEY_ID
  secret_access_key: DEV_ACCESS_KEY_ID

If this file contains your secrets in plain text and does not use ERB to load them securely, you should not track this file with source control.

You can now read part two in this series: Rotating Credentials.

Ruby at re:Invent 2012

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

Loren and I will be attending AWS re:Invent in Las Vegas next week. I will be presenting a session, and we will both be answering questions in the Developer Lounge, and assisting with the AWS re:Invent Code Challenges.

Stop by to say "Hi", get your questions answered, or participate in a Code Challenge. We love to meet developers, get feedback, and to talk Ruby.

Getting Productive with the AWS SDK for Ruby, Wednesday, November 28 from 4:30 p.m. – 5:20 p.m.

Join us in the Venetian, Room 3001A. You will learn about how to get the most of the aws-sdk gem, and we will be building a sample application live that uses three different AWS services.

See you in Las Vegas!

Deploying Ruby Applications to AWS Elastic Beanstalk with Git

by Loren Segal | on | in Ruby | Permalink | Comments |  Share

With the release of Ruby container support on AWS Elastic Beanstalk, it is now possible to deploy Rails (or any Rack-based) applications onto the AWS infrastructure in just a few easy steps. The container is backed by Amazon EC2 and automatically provisions the machines to use Phusion Passenger (with nginx under the hood). You can run your own local database with your application, or better yet, you can have Elastic Beanstalk automatically set you up with a MySQL database backed by Amazon Relational Database Service (RDS). All of the machines sit behind a load balancer with auto-scaling support automatically configured so you don’t even have to think about ops. Scaling is built-in.

But enough with the awesome features. Let’s deploy an app!

Setting up the application

Before we can deploy an application to Elastic Beanstalk, we first have to create one. Let’s generate an empty Rails skeleton and set it up against a Git repository. We will be using the repository later to deploy our app.

$ rails new fooapp
$ cd fooapp
$ git init && git add -A && git commit -m "Initial commit"

Initializing our Elastic Beanstalk environment

At this point we have an application, but we have not yet configured it to deploy to Elastic Beanstalk. In order to do that, we need to install the AWS Elastic Beanstalk command line tool. This package provides us a binary, eb, which we will use to manage and deploy our applications.

Note that you do need to have Python (2.7 or 3.0) installed in order to use the CLI tools. If you are on OS X or a modern Linux OS, you should already have this installed. If not, visit Python’s website for download information. In order to verify if you have Python on your machine, you can type python --version

We can now initialize our application with the CLI tool by typing eb init. The tool will ask us for some information about our deployment such as our AWS credentials, the deployment region, and the solution stack we want to deploy. For our case, we want to use the "64bit Amazon Linux running Ruby 1.9.3" stack. The tool also asks if we want to setup an RDS database instance for our environment. Let’s answer "no" on that one for now.

$ eb init
To get your AWS Access Key ID and Secret Access Key, 
  visit "https://aws-portal.amazon.com/gp/aws/securityCredentials".
Enter your AWS Access Key ID: ****
Enter your AWS Secret Access Key: ****
Select an AWS Elastic Beanstalk service region.
Available service regions are:
2) US West (Oregon)
...
Select:  (1 to 6): 2
Enter an AWS Elastic Beanstalk application name (auto-generated value is "fooapp"): 
Enter an AWS Elastic Beanstalk environment name (auto-generated value is "fooapp-env"): 
Select a solution stack.
Available solution stacks are:
...
10) 64bit Amazon Linux running Ruby 1.9.3
Select:  (1 to 10): 10
Create an RDS DB Instance? [y/n]: n         

Note that this command will create a .elasticbeanstalk directory in our project and add it to our .gitignore. This directory contains the private environment configuration data provided above, as well as other customized settings. We then commit this change to our .gitignore with the following command:

$ git add .gitignore && git commit -m "Ignore .elasticbeanstalk from Git"

You can see more examples of setting up a project with the eb tool on the Amazon Web Services Blog.

Starting the environment

Now that our environment is configured, we simply need to start it. Starting the environment can take a couple of minutes. During this time, Elastic Beanstalk is spawning a set of EC2 instances, setting up a load balancer, and establishing health monitoring. Fortunately we only have to do this once per environment. Once Elastic Beanstalk is done, we can start the environment with the following command:

$ eb start
Starting application "fooapp".
Waiting for environment "fooapp-env" to launch.
... 
Application is available at "fooapp-env-djr9ywimiu.elasticbeanstalk.com".

After a little bit of waiting, our environment should be live. We can use the environment in a web browser with the URL provided above:

AWS Elastic Beanstalk Ruby Container Welcome Page

You can now also type eb status to see the status of your environment:

$ eb status
URL : fooapp-env-djr9ywimiu.elasticbeanstalk.com
Status  : Ready
Health  : Green

Deploying with Git

The eb init command we ran above actually did some nifty things behind the scenes. The most notable thing is that it setup a Git subcommand named aws.push that can push our code to the environment. This means that whenever we need to deploy our app, we just have to type git aws.push. Since our environment is running, and our app is ready, let’s do that right now!

$ git aws.push

If everything succeeds, you should have initiated a deploy to your environment. Typing eb status should now tell us that a deploy is in progress:

$ eb status
URL : fooapp-env-djr9ywimiu.elasticbeanstalk.com
Status  : Updating
Health  : Grey

Eventually, the status will be green, and we will be able to access our new Rails application:

Rails Welcome Page

Notice that gems listed in our Gemfile were automatically installed, assets were precompiled, and, if we had any, database migrations would have been run too. Migrations and asset compilation happen by default for any Rails 3.x application.

All we have to do now is update our application, commit to our Git repository, and execute the command git aws.push to push our new changes to deploy new versions of the app. Migrations will be run, and our assets will get updated.

Customizing your environment with .ebextensions

To run commands and drop environment variables onto your system via your app during a deployment, you can add a special .config file to the .ebextensions directory in your project root. The config file is formatted as YAML, and is documented on the Customizing and Configuring AWS Elastic Beanstalk Environments documentation page.

For reference, here is a sample .ebextensions/redmine.config that runs some extra hooks for installing Redmine on Elastic Beanstalk. Note that the environment variables are not necessary; BUNDLE_WITHOUT and RACK_ENV already default to the values below. We’re just setting them as an example of some config we might want to perform.

# Sample configuration for Redmine Rails installation

option_settings:
  - option_name: BUNDLE_WITHOUT
    value: "test:development"
  - option_name: RACK_ENV
    value: production

packages:
  yum:
    git: []

container_commands:
  01_secret:
    command: rake generate_secret_token

Skipping migrations and asset compilation

I mentioned you can set environment variables using the option_settings key in the .ebextensions/*.config files. Two specific variables you might want to set are:

  • RAILS_SKIP_MIGRATIONS — set this to true if you don’t want Elastic Beanstalk to run migrations on your behalf. You may want to manage migrations yourself.
  • RAILS_SKIP_ASSET_COMPILATION — set this to true if you don’t use asset compilation in your Rails app. Note, however, that your Elastic Beanstalk container is already configured with a JS runtime to properly handle asset compilation, you do not have to do anything fancy like install the therubyracer gem. It should just work.

Setting up Amazon RDS

The final-but-very-optional step for using the Ruby solution stack in Elastic Beanstalk is to hook up an RDS Instance to your application. If you are running any kind of real world application, you most likely have some database. If your application uses a relational database, you can take advantage of Amazon RDS for access to a MySQL (or Oracle) database.

Fortunately, the setup for this is extremely simple. Remember when we said "no" to creating an RDS DB Instance when we ran eb init? Well, we can initialize another application in the exact same way, but instead of no, we can say yes. We will be prompted to provide a password for this database, but everything else will be directly configured by Elastic Beanstalk on our behalf.

How do we configure it with our Rails app, you ask? Easy. All of the DB information will be passed to our application as environment variables. In Rails, this means we can setup our application to use RDS with a simple database.yml snippet:

production:
  adapter: mysql2
  encoding: utf8
  database: <%= ENV['RDS_DB_NAME'] %>
  username: <%= ENV['RDS_USERNAME'] %>
  password: <%= ENV['RDS_PASSWORD'] %>
  host: <%= ENV['RDS_HOSTNAME'] %>
  port: <%= ENV['RDS_PORT'] %>

The only other thing we have to do is add mysql2 to our Gemfile. It would look something like:

gem 'mysql2'

And RDS should now be fully configured for our Rails application.

Testing locally with Passenger

Because we use Passenger in your container, it’s really easy to test your deployment locally in a very similar webserver environment. If you don’t already have passenger installed on your local machine, you can set it up and initialize your server with:

$ gem install passenger
$ passenger start

A few quick notes if you’ve never deployed to a Passenger server before:

  • Passenger relies on a config.ru Rack file in your root application directory by convention.
  • Passenger also relies on a public/ directory for your static assets located in your application root.

Passenger needs both of these paths to exist in order to detect your Rack/Rails application. In a Rails 3.x app, these paths already exist, but if you are rolling your own low-level Rack application, you need to make sure that they are created. Once they have been setup, your application is ready to be tested as if it were deployed to Elastic Beanstalk. You can consult the Phusion Passenger documentation for more information about how to run your application on this snappy webserver.

Conclusion

So that’s it. We just went through the nuts and bolts of deploying a Ruby application to AWS Elastic Beanstalk. Although there is a lot here to read through, the steps themselves are pretty simple. In short, you can deploy an application with a few commands:

  1. Run eb init on your Git-backed application
  2. Run eb start to load up your environment
  3. Run git aws.push to deploy your code to Elastic Beanstalk
  4. Profit!

Setting up RDS is just a matter of using the configuration values provided to your application through standard environment variables. And of course if you need to dig deeper and customize settings, you can do so by adding a .ebextensions/config.yml file.

There is of course plenty you can do with an Elastic Beanstalk application, and there are plenty more things to talk about with the Ruby solution stack, but hopefully this helps you get a Rack-based application up and running in just a few minutes!

New Blog, New Org, and New Issue Reporting!

by Loren Segal | on | in Ruby | Permalink | Comments |  Share

Hey everyone!

My name is Loren Segal, and I will be working with Trevor Rowe on our shiny new blog here at Amazon Web Services. We both work on the AWS SDK for Ruby, and over the coming weeks, months, and years, we will be using this spot to post new updates, tips, and tricks about the Ruby SDK. We hope these posts will be informative and shed some light on what’s coming down the pipe, but we also want to get the word out about all the cool things we already support. We’re really excited about sharing all this new information with you.

And with that, we already have new announcements to make: we are changing our GitHub organization name and enabling GitHub Issues in our new organization for easier issue and feature tracking in the Ruby SDK!

A better GitHub organization name

We decided to move our GitHub amazonwebservices organization to the much shorter and easier-to-remember aws organization. With that change, we’ve also renamed the project from "aws-sdk-for-ruby" to a simpler "aws-sdk-ruby". We will be following this naming convention across our other projects to make things easier for our collaborators.

As of this blog post, we will no longer be developing against our old repository, but we will leave this old repository around so that users can still access the code. Note, however, that in about a week, we replace the contents of the repository with a README guiding our new users to the proper repository location.

Remember to update your Git remotes and Gemfiles!

If you’re currently pointing your remotes to our old organization, now is the time to update the location to the new repository. You can do so by typing the following command in your local working copy (assuming the remote is origin):

$ git remote set-url origin git://github.com/aws/aws-sdk-ruby

If you are pointing your Gemfiles to our Git repository, you will also want to update those by pointing to the new repo:

gem 'aws-sdk', :git => "git://github.com/aws/aws-sdk-ruby.git"

Note that for Gemfiles, we would much rather you point to the gem released on RubyGems.org, though!

GitHub Issues are enabled!

Finally, with our newly renamed organization, we’ve decided to enable GitHub Issues on the new aws-sdk-ruby project. You can now post bug reports, submit pull requests, and track development progress right on our issues page.

Since we’re on the topic, here are a couple of notes about submitting new issues:

Submitting pull requests: include tests

We love when users contribute code to the SDK! Really! If you’ve found a bug in the SDK and you’ve tracked down the issue and have a local working patch, submitting a pull request will help us get the fix out to our users much faster than just submitting a bug report and greedily keeping that awesome bugfix to yourself!

However, if you do plan on submitting pull requests, please try to include tests for your new code. The better the tests, the more easily we can guarantee that your patch will continue to work, and the quicker we can pull it in. Now that we are using Travis-CI for continuous integration, GitHub will happily tell us that your changes will not break the build.

Submitting bug reports: include reproduction steps

If you are having issues with the SDK not working properly, we want to hear about it and fix it. GitHub Issues now allows you to post your problems directly on our GitHub repository. If you do report an issue, please make sure to include reliable reproduction steps, as well as much extra relevant information as possible. Please make sure you do not post your AWS credentials in bug reports, however! We will never ask you for this information— that is your data.

Shiny new things!

We’re really happy about these new changes. The new blog will be here to provide new insight on what we’re up to on the Ruby team, and hopefully the new GitHub re-organization will make it much easier to contribute code and submit bugs on the AWS SDK for Ruby! Go check out our new org right now:

http://github.com/aws/aws-sdk-ruby