Category: Ruby


Announcing the aws-sdk-rails Gem

by Alex Wood | on | in Ruby | Permalink | Comments |  Share

With the release of V2 of the AWS SDK for Ruby, we’ve received customer feedback asking for support for the Ruby on Rails integration features provided by V1 of the SDK.

Today, we’re excited to announce the release of the aws-sdk-rails gem, available now via RubyGems and, of course, on GitHub.

To get started, add the aws-sdk-rails gem to your Gemfile:

gem 'aws-sdk-rails', '~> 1.0'

ActionMailer and Amazon Simple Email Service (SES)

The gem will automatically configure Rails to include an :aws_sdk delivery method for ActionMailer, that uses Amazon SES as a backend. It is simple to configure Rails to use this delivery method:

# config/application.rb
config.action_mailer.delivery_method = :aws_sdk

The aws-sdk-rails gem will use the AWS SDK for Ruby V2’s SES client automatically for any mail delivery event.

Logging

The gem will automatically wire the AWS SDK for Ruby’s logger to use Rails.logger by default.

You can customize the SDK log level and an optional log formatter in a config initializer:

# config/initializers/aws-sdk.rb
# log level defaults to :info
Aws.config[:log_level] = :debug

It is important to understand that all SDK log messages are logged at the same log level. Why is this important? When you set the Rails log level, you’re muting all log messages below that log level. So, if you want to, for example, only see SDK log messages in development, you might set the SDK log level to :debug as shown above, and set the Rails logger to show debug in development.

Credentials

The AWS SDK for Ruby will attempt locate credentials by searching the following locations:

  • ENV['AWS_ACCESS_KEY_ID'] and ENV['AWS_SECRET_ACCESS_KEY']
  • The shared credentials ini file at ~/.aws/credentials
  • From an instance profile when running on Amazon EC2

If you need to manually configure credentials, you should add them to your initializer:

# config/initializers/aws-sdk.rb
Aws.config[:credentials] = Aws::Credentials.new(access_key, secret)

Learn more about credentials in the AWS SDK for Ruby V2.

Never commit your credentials to source control. Besides being a security risk, it makes it very difficult to rotate your credentials.

Available Now

The aws-sdk-rails gem is available now.

As always, we’d love to hear your feedback, and welcome any Issues or Pull Requests at the aws-sdk-rails GitHub repo.

Amazon DynamoDB Document API in Ruby (Part 3 – Update Expressions)

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

As we showed in previous posts, it’s easy to put JSON items into Amazon DynamoDB, retrieve specific attributes with projection expressions, and fetch only data that meet some criteria with condition expressions. Now, let’s take a look at how we can conditionally modify existing items with Update Expressions. (Note: this code uses the same ProductCatalog table we used in Parts 1 and 2).

In the following examples, we use the following helper method to perform conditional updates. It performs the UpdateItem operation with return_values set to return the old item. We also use the GetItem operation so the method can return both the old and new items for us to compare. (If the update condition in the request is not met, then the method sets the returned old item to nil.)

def do_update_item(key_id, update_exp, condition_exp, exp_attribute_values)
  begin
    old_result = @dynamodb.update_item(
      :update_expression => update_exp,
      :condition_expression => condition_exp,
      :expression_attribute_values => exp_attribute_values,
      :table_name => "ProductCatalog",
      :key => { :Id => key_id },
      :return_values => "ALL_OLD",
    ).data.attributes
  rescue Aws::DynamoDB::Errors::ConditionalCheckFailedException
    old_result = nil
    puts "Condition not met"
  end

  new_result = @dynamodb.get_item(
    :table_name => "ProductCatalog", :key => { :Id => key_id },
    :consistent_read => true
  ).data.item  

  return old_result, new_result
end

Using Conditional Update Expressions

Updates in DynamoDB are atomic. This allows applications to concurrently update items without worrying about conflicts occurring. For example, the following code demonstrates maintaining a MAX value in DynamoDB with a conditional update using SET. Note that, because DynamoDB is schema-less, we don’t need to define the HighestRating attribute beforehand. Instead, we create it on the first call.

# storing a "max" value with conditional SET
# SET attribute if doesn't exist, otherwise SET if stored highest rating < this rating
def update_highest_rating(rating)
  do_update_item(303,
    "SET HighestRating = :val",
    "attribute_not_exists(HighestRating) OR HighestRating < :val",
    {
      ":val" => rating
    }
  )
end

# multiple threads trying to SET highest value (ranging from 0 to 10)
threads = []
(0..10).to_a.shuffle.each { |i|
  # some number of "Condition not met" depending on shuffled order
  puts i
  threads[i] = Thread.new {
    update_highest_rating(i)
  }
}
threads.each {|t| t.join}

# fetch the item and examine the HighestRating stored
puts "Max = #{@dynamodb.get_item(
  :table_name => "ProductCatalog", :key => { :Id => 303 }
).data.item["HighestRating"].to_i}"   # Max = 10

We can also use update expressions to atomically maintain a count and add to a set:

# ADD to intialize/increment and add to set
threads = []
20.times do |i|
  threads[i] = Thread.new {
    do_update_item(303,
      "ADD TimesViewed :val, Tags :was_here",
      nil, # no condition expression
      {
        # Each of the 20 threads increments by 1
        ":val" => 1,

        # Each thread adds to the tag set
        # Note: type must match stored attribute's type
        ":was_here" => Set.new(["#Thread#{i}WasHere"])
      }
    )
  }
end
threads.each {|t| t.join}

# fetch the item and examine the TimesViewed attribute
item = @dynamodb.get_item(
  :table_name => "ProductCatalog", :key => { :Id => 303 }
).data.item

puts "TimesViewed = #{item["TimesViewed"].to_i}"
# TimesViewed = 20

puts "Tags = #{item["Tags"].inspect}"
# Tags = #<Set: {"#Mars", "#MarsCuriosity", "#StillRoving", ..each thread was here...}>

Similarly, we can decrement the count and remove from the set to undo our previous operations.

# Undo the views and set adding that we just performed
threads = []
20.times do |i|
  threads[i] = Thread.new {
    do_update_item(303,
      "ADD TimesViewed :val DELETE Tags :was_here",
      nil,  # no condition expression
      {
        # Each of the 20 threads decrements by 1
        ":val" => -1,

        # Each thread removes from the tag set
        # Note: type must match stored attribute's type
        ":was_here" => Set.new(["#Thread#{i}WasHere"])
      }
    )
  }
end
threads.each {|t| t.join}

# fetch the item and examine the TimesViewed attribute
item = @dynamodb.get_item(
  :table_name => "ProductCatalog", :key => { :Id => 303 }
).data.item

puts "TimesViewed = #{item["TimesViewed"].to_i}"
# TimesViewed = 0

puts "Tags = #{item["Tags"].inspect}"
# Tags = #<Set: {"#Mars", "#MarsCuriosity", "#StillRoving"}>

We can also use the REMOVE keyword to delete attributes, such as the HighestRating and TimesViewed attributes we added in the previous code.

# removing attributes from items
old_and_new = do_update_item(303,
  "REMOVE HighestRating, TimesViewed",
  nil,  # no condition expression
  nil,  # no attribute expression values
)

puts "OLD HighestRating is nil ? #{old_and_new[0]["HighestRating"] == nil}"
#=> false

puts "OLD TimesViewed is nil ? #{old_and_new[0]["TimesViewed"] == nil}"
#=> false

puts "NEW HighestRating is nil ? #{old_and_new[1]["HighestRating"] == nil}"
#=> true

puts "NEW TimesViewed is nil ? #{old_and_new[1]["TimesViewed"] == nil}"
#=> true

Conclusion

We hope this series was helpful in demonstrating expressions and how they allow you to interact with DynamoDB more flexibly than before. We’re always interested in hearing what developers would like to see in the future, so let us know what you think in the comments or through our forums!

Amazon DynamoDB Document API in Ruby (Part 2 – Condition Expressions)

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

As we showed in the previous post, it’s easy to put JSON items into Amazon DynamoDB and retrieve specific attributes with projection expressions. Condition Expressions provide a more flexible and SQL-like way to retrieve only the items you want from DynamoDB. First, let’s put a few more items into DynamoDB using a BatchWriteItem operation. (Note: this code uses the same ProductCatalog table we used in Part 1)

# add some more items
@dynamodb.batch_write_item(
  :request_items => {
    "ProductCatalog" => [

      {:put_request => { :item => {
        Id: 300,
        Title: "Sojourner",
        Description: "Mars Pathfinder robotic Mars rover",
        Price: BigDecimal.new("2.65e8"),
        LaunchDate: {
          M: 12, D: 4, Y: 1996
        },
        LostCommunicationDate: {
          M: 9, D: 27, Y: 1997
        },
        Features: {
          Rover: true,
        },
        NumberInStock: 10,
        OrdersPlaced: 3,
        Tags: ["#Mars", "#InStarTrekSeason4", "#InRedPlant2000", "#LostComms"],
      }}},

      {:put_request => { :item => {
        Id: 301,
        Title: "Spirit",
        Description: "Mars Exploration Rover – A",
        Price: BigDecimal.new("4.1e8"),
        LaunchDate: {
          M: 6, D: 10, Y: 2003
        },
        LostCommunicationDate: {
          M: 3, D: 22, Y: 2010
        },
        Features: {
          Rover: true,
        },
        NumberInStock: 10,
        OrdersPlaced: 5,
        Tags: Set.new(["#Mars", "#StuckOnMars", "#LostComms"]),
      }}},

      {:put_request => { :item => {
        Id: 302,
        Title: "Opportunity",
        Description: "Mars Exploration Rover – B",
        Price: BigDecimal.new("4.1e8"),
        LaunchDate: {
          M: 7, D: 7, Y: 2003
        },
        LostCommunicationDate: nil,
        Features: {
          Rover: true,
        },
        NumberInStock: 10,
        OrdersPlaced: 10,
        Tags: Set.new(["#Mars", "#StillRoving"]),
      }}},

      {:put_request => { :item => {
        Id: 303,
        Title: "Curiosity",
        Description: "car-sized robotic rover",
        Price: BigDecimal.new("2.5e9"),
        LaunchDate: {
          M: 11, D: 26, Y: 2011
        },
        LostCommunicationDate: nil,
        Features: {
          Rover: true,
          RoboticArm: true,
        },
        NumberInStock: 0,
        OrdersPlaced: 30,
        Tags: Set.new(["#Mars", "#MarsCuriosity", "#StillRoving"]),
      }}},

    ]
  }
)

Using Condition Expressions

We could also use condition expressions on the results of Query, but since we’re using a simple data model (only have hash key on product Id), we demonstrate this with scans. We use the following helper method to perform the scan and format the product titles returned:

def do_scan(filter_exp, exp_attribute_values)
  result = @dynamodb.scan(
    :expression_attribute_values => exp_attribute_values,
    :filter_expression => filter_exp,   # Condition Expressions are supplied through the FilterExpression parameter
    :projection_expression => "Title",
    :table_name => "ProductCatalog"
  ).data.items

  # format all retrieved titles into a single line
  return "scan retrieved: #{(result.map { |item| item["Title"] }).join(", ")}"
end

Let’s look at some example expressions and the results they return from our current ProductCatalog table:

# All products that don't have a launch month of November (11)
puts do_scan(
  "LaunchDate.M <> :m",
  {
    ":m" => 11
  }
)
# scan retrieved: 20-Bicycle 205, Opportunity, Spirit, Sojourner


# All rover products that don't have a launch month of November
puts do_scan(
  "attribute_exists(Features.Rover) AND LaunchDate.M <> :m",
  {
    ":m" => 11,
  }
)
# scan retrieved: Opportunity, Spirit, Sojourner


# Non-rovers
puts do_scan(
  "attribute_not_exists(Features.Rover)",
  nil
)
# scan retrieved: 20-Bicycle 205


# mid-range rovers or inexpensive products
puts do_scan(
  "(Price BETWEEN :low AND :high) OR Price < :verylow",
  {
    ":verylow" => BigDecimal.new("1e8"),
    ":low" => BigDecimal.new("3e8"),
    ":high" => BigDecimal.new("5e8")
  }
)
# scan retrieved: 20-Bicycle 205, Opportunity, Spirit


# within-Item referencing: more orders placed than in stock
puts do_scan(
  "OrdersPlaced > NumberInStock",
  nil
)
# scan retrieved: Curiosity


# string prefixing
puts do_scan(
  "begins_with(Title, :s)",
  {
    ":s" => "S",
  }
)
# scan retrieved: Spirit, Sojourner


# contains
puts do_scan(
  "contains(Tags, :tag1) AND contains(Tags, :tag2)",
  {
    ":tag1" => "#StuckOnMars",
    ":tag2" => "#LostComms",
  }
)
# scan retrieved: Spirit


# contains (Note: "Tags" is a list for Sojourner)
puts do_scan(
  "contains(Tags, :tag1)",
  {
    ":tag1" => "#LostComms",
  }
)
# scan retrieved: Spirit, Sojourner


# in operator
puts do_scan(
  "Id in (:id1, :id2)",
  {
    ":id1" => 302,
    ":id2" => 303,
  }
)
# scan retrieved: Curiosity, Opportunity


# equivalently, with parentheses
puts do_scan(
  "(Id = :id1) OR (Id = :id2)",
  {
    ":id1" => 302,
    ":id2" => 303,
  }
)
# scan retrieved: Curiosity, Opportunity

Next Steps

As you can see, condition expressions enable you to write more concise code to retrieve data. They also provide querying capabilities unavailable with the original access model such as within-Item references and more flexible conditions with parentheses. In an upcoming blog post, we’ll take a closer look at how we can update existing data through update expressions.

Amazon DynamoDB Document API in Ruby (Part 1 – Projection Expressions)

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

Amazon DynamoDB launched JSON Document Support along with several improvements to the DynamoDB API. This post is part of a series where we’ll explore these features in more depth with the AWS SDK for Ruby V2. In particular, this post focuses on putting items into DynamoDB using the Ruby SDK and controlling the data we get back with projection expressions. At the end of the post, we also provide some helpful information for getting started with DynamoDB Local.

Putting JSON data into DynamoDB

DynamoDB now supports the following new data types: Maps, Lists, Booleans, and Nulls. Suppose we have a DynamoDB table for products with a hash key on an "Id" attribute. It’s easy to store such data into DynamoDB with native Ruby types:

# put a JSON item
item = {
  Id: 205, # hash key
  Title: "20-Bicycle 205",
  Description: "205 description",
  BicycleType: "Hybrid",
  Brand: "Brand-Company C",
  Price: 500,
  Gender: "B",
  Color: Set.new(["Red", "Black"]),
  ProductCategory: "Bike",
  InStock: true,
  QuantityOnHand: nil,
  NumberSold: BigDecimal.new("1E4"),
  RelatedItems: [
    341, 
    472, 
    649
  ],
  Pictures: { # JSON Map of views to url String
    FrontView: "http://example.com/products/205_front.jpg", 
    RearView: "http://example.com/products/205_rear.jpg",
    SideView: "http://example.com/products/205_left_side.jpg",
  },
  ProductReviews: { # JSON Map of stars to List of review Strings
    FiveStar: [
      "Excellent! Can't recommend it highly enough!  Buy it!",
      "Do yourself a favor and buy this."
    ],
    OneStar: [
      "Terrible product!  Do not buy this."
    ]
  }
}
dynamodb.put_item(:table_name => "ProductCatalog", :item => item)

Getting data from DynamoDB using projection expressions

Since DynamoDB now supports more interesting data types, we’ve also added projection expressions and expression attribute names to make it easier to retrieve only the attributes we want:

# get only the attributes we want with projection expressions
item = dynamodb.get_item(
  :table_name => "ProductCatalog",

  # Get the item with Id == 205
  :key => {
    :Id => 205
  },

  # for less typing, use expression attribute names to substitute
  # "ProductReviews" with "#pr" and "RelatedItems" with "#ri"
  :expression_attribute_names => {
    "#pr" => "ProductReviews",
    "#ri" => "RelatedItems",
  },

  # get Price, Color, FiveStar reviews, 0th and 2nd related items
  :projection_expression => "Price, Color, #pr.FiveStar, #ri[0], #ri[2], 
    #pr.NoStar, #ri[4]" # try projecting non-existent attributes too
).data.item

puts item["Price"].to_i
# 500

puts item["Color"].inspect
# #<Set: {"Black", "Red"}>

puts item["ProductReviews"]["FiveStar"][0]
# Excellent! Can't recommend it highly enough!  Buy it!

puts item["ProductReviews"]["FiveStar"][1]
# Do yourself a favor and buy this.

puts item["ProductReviews"]["OneStar"].inspect
# nil (because we only projected FiveStar reviews)

puts item["ProductReviews"]["NoStar"].inspect
# nil (because no NoStar reviews)

puts item["RelatedItems"]
# 0.341E3   (0th element)
# 0.649E3   (2nd element)

puts item["RelatedItems"].size
# 2 (non-existent 4th element not present)

Next Steps

As you can see, it’s easy to put and get items in DynamoDB with the AWS SDK for Ruby. In upcoming blog posts, we’ll take a closer look at expressions for filtering and updating data.

Feel free to get started on DynamoDB Local with the following code (note that it uses the credentials file approach for specifying AWS credentials):

#! /usr/bin/ruby

require "set"
require "bigdecimal"
require "aws-sdk-core"

# Configure SDK

# use credentials file at .aws/credentials
Aws.config[:credentials] = Aws::SharedCredentials.new
Aws.config[:region] = "us-west-2"

# point to DynamoDB Local, comment out this line to use real DynamoDB
Aws.config[:dynamodb] = { endpoint: "http://localhost:8000" }

dynamodb = Aws::DynamoDB::Client.new

## Create the table if it doesn't exist
begin
  dynamodb.describe_table(:table_name => "ProductCatalog")
rescue Aws::DynamoDB::Errors::ResourceNotFoundException
  dynamodb.create_table(
    :table_name => "ProductCatalog",
    :attribute_definitions => [
      {
        :attribute_name => :Id,
        :attribute_type => :N
      }
    ],
    :key_schema => [
      {
        :attribute_name => :Id,
        :key_type => :HASH
      }
    ],
    :provisioned_throughput => {
      :read_capacity_units => 1,
      :write_capacity_units => 1,
    }
  )

  # wait for table to be created
  puts "waiting for table to be created..."
  dynamodb.wait_until(:table_exists, table_name: "ProductCatalog")
  puts "table created!"
end

Announcing V2 of the AWS SDK for Ruby

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

I am excited to announce today’s stable release of version 2 of the AWS SDK for Ruby. It is available now as the aws-sdk gem on RubyGems.

Features

Version 2 of the AWS SDK for Ruby, the aws-sdk gem, provides a number of powerful features for developers including:

Upgrading

Version 2 of the AWS SDK for Ruby uses a different namespace, making it possible to use version 1 and version 2 in the same application.

# Gemfile
gem 'aws-sdk', '~> 2'
gem 'aws-sdk-v1'

# code
require 'aws-sdk-v1'
require 'aws-sdk'

ec2_v1 = AWS::EC2.new # v1
ec2_v2 = Aws::EC2::Resource.new # v2

This allows you to start using the version 2 SDK today without changing existing code.

Feedback

Please share your questions, comments, issues, etc. with us on GitHub. You can also catch us in our Gitter channel.

Upcoming Stable Release of AWS SDK for Ruby – Version 2

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

We plan to release version 2 of the AWS SDK for Ruby next week. We will remove the preview flag from the 2.0 version of aws-sdk.

Specify Your Version Dependencies

The AWS SDK for Ruby uses semantic versioning. Updates within version 1 are backwards compatible.

Version 2 of the aws-sdk gem is not backwards compatible.

If you depend on the aws-sdk gem today, and do not specify the major version, please add this now. If not, you may run into issues when you bundle update.

# Gemfile
gem 'aws-sdk', '< 2.0'

# gemspec
spec.add_dependency('aws-sdk', '< 2.0')

NameError: uninitialized constant AWS

If you receive this error, you likely have a dependency on aws-sdk and have updated so that you now have version 2 installed. Version 2 uses a different module name, so it does not define AWS.

To resolve this issue, specify your version dependency as instructed above.

Using Both Versions

The following diagram shows how the version 1 and version 2 gems are organized.

The aws-sdk gem is empty, and simply requires version 1 or version 2 specific gems. This allows you to install version 1 and version 2 in the same application.

Option A, for existing users

# Gemfile
gem 'aws-sdk', '~> 1'
gem 'aws-sdk-resources', '~> 2'

# in code
require 'aws-sdk'
require 'aws-sdk-resources'

Option B, for new users

# Gemfile
gem 'aws-sdk-v1'
gem 'aws-sdk', '~> 2'

# in code
require 'aws-sdk-v1'
require 'aws-sdk'

Attention Library Authors

If you maintain a gem that has a dependency on version 1 of aws-sdk, I strongly recommend that you replace it with a dependency on aws-sdk-v1. This allows end users to require version 2 of aws-sdk.

Please report any issues you have on our GitHub repository.

Using Amazon RDS with Ruby on Rails and AWS OpsWorks

by Alex Wood | on | in Ruby | Permalink | Comments |  Share

Earlier in this blog series, we showed you how to deploy a Ruby on Rails application to Amazon Web Services using AWS OpsWorks. In that example, we used an OpsWorks-managed MySQL database run on an Amazon EC2 instance. One common piece of feedback on that post was a desire to see how you can set up your stack with Amazon RDS. Today, we are going to show you how.

Prerequisites

We are going to assume you’re familiar with our earlier post on how to deploy with OpsWorks in general. For this tutorial you can take one of the following approaches:

  • Take your existing stack from following along with that post, delete your database and MySQL layers, then follow this tutorial.
  • Clone your existing stack (in case you don’t want to lose your work), create a new app server instance, delete the new stack’s MySQL layer, then follow along.
  • Start from a brand new stack, and go between both tutorials, replacing the DB steps from the previous tutorial with the DB steps here. If you’re going to go that route, we would recommend reading this tutorial first to understand the differences in approach.

Whichever approach you choose, you can be up and running in just a few minutes!

Create an RDS Instance

AWS OpsWorks will not create Amazon RDS instances on your behalf. You will need to create an instance and link it to OpsWorks.

  1. Open up the RDS console, navigate to Instances, and click Launch DB Instances.
  2. To be able to use our Rails example code unaltered, choose the MySQL engine (you can choose another engine if you like, see the next section for details).
  3. On the next screen, you can choose either Multi-AZ deployments or single-AZ, but if you want to stay within the RDS Free Usage Tier, then you should not choose a multi-AZ deployment.
  4. Make sure that you keep track of your Master Username, Master Password, and the Database Name from the rest of the form.
  5. Within Configure Advanced Settings, make sure you set your VPC Security Groups to include AWS-OpsWorks-DB-Master-Server and AWS-OpsWorks-Rails-App-Server. You can set Publicly Accessible to "No" as well.
  6. Once you’ve completed the forms, click Launch DB Instance. It may take a few minutes for your instance to launch, so now is not a bad time for tea or coffee.

Don’t Want MySQL?

You do not need to use MySQL as your database backend, that is just what we have chosen for this example. Want to use a different database backend? Just do the following:

  1. Replace the mysql2 gem with the adapter gem appropriate to your DB engine selection.
  2. Make sure you also use your adapter selection in your custom deployment JSON, found in your stack settings if you used our exact tutorial steps.
  3. Of course, select that database engine when creating your RDS instance.

Create an RDS Layer

Now that you have an RDS Instance, you need to register it with OpsWorks.

  1. Navigate to the OpsWorks console, to the Layers section.
  2. Click + Layer, and select the RDS tab.
  3. Your RDS instance should be on this screen to select. Select it, then enter your User and Password from the RDS instance creation step.
  4. Click Register with Stack.

This registers your database with OpsWorks, and next you need to connect your app.

Attach Your RDS Instance to Your App

  1. From the Apps section, click on Edit for TodoApp (or whatever app you are developing for).
  2. Under the Data Sources section, select RDS.
  3. It should auto-select your instance; select it if not. Then, fill in your Database name from the RDS instance creation process.
  4. Click Save to save your changes.

Note that this should work for any of the topic branches we’ve made for TodoApp, as your choice of database host is transparent to your app so long as you have the correct database adapter gem in your Gemfile.

Deploy the App

To use your new database, simply run an app deployment, making sure you select the Migrate database option so that Rails can set up your database.

Once that’s done, navigate to your app server host and play around with your app. You’re now running on RDS!

AWS re:Invent 2014 Ruby Recap

by Alex Wood | on | in Ruby | Permalink | Comments |  Share

Last week, we had a great time meeting with AWS customers using Ruby at AWS re:Invent! We appreciate the feedback we received, and the discussions that we had with you.

AWS SDK for Ruby Presentation Notes

At AWS re:Invent this year I took many of you on a tour of version 2 of the AWS SDK for Ruby. We were thrilled to have such a great audience, and we had a great time meeting with many of you both before and after the talk.

In the presentation, we talked about the architecture and new features available in version 2 of the AWS SDK for Ruby, including resources, pagination, and waiters. We also walked through an end-to-end example using Ruby on Rails.

If you didn’t get a chance to make it to my talk, I encourage you to check it out. If you did, you still might find it worthwhile to code along with our Rails example. You can skip ahead to that here if you like.

The presentation also had a lot of links that you might want to check out. For your convenience, we’ve compiled them here:

See You Next Year

AWS re:Invent is October 6 – 9, 2015, once again at The Venetian in Las Vegas, NV. Let us know what you’d like to see next on Twitter @awsforruby and we hope to see you there!

Client Response Stubs

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

We recently added client response stubs to the aws-sdk-core gem. Response stubbing disables network traffic and causes a client to return fake or stubbed data.

# no API calls are made
s3 = Aws::S3::Client.new(stub_responses: true)
s3.list_buckets.buckets.map(&:name)
#=> []

Custom Response Data

By default, stubbed responses return empty lists, empty maps, and placeholder scalars. These empty responses can be useful at times, but often you want to control the data returned.

s3.stub_responses(:list_buckets, buckets:[{name:'aws-sdk'}])
s3.list_buckets.buckets.map(&:name)
#=> ['aws-sdk']

Safe Stubbing

One of the common risks when writing tests with stub data is that the stub doesn’t match the shape of the actual response. You risk coding against stubs that provide methods that won’t exist out of your tests.

We resolve this issue by validating your stub data hash against the model of the API. An ArgumentError is raised when calling #stub_responses with invalid data.

s3.stub_resposnes(:list_buckets, buckets:['aws-sdk'])
#=> raises ArgumentError, "expected params[:buckets][0] to be a hash"

Stubbing Multiple Calls

By calling #stub_responses with an operation name and stub data, the client will serve that data for each call. You can specify multiple responses, and they will be used in sequence.

s3.stub_responses(:list_buckets, 
  { buckets:[{name:'aws-sdk'}] },
  { buckets:[{name:'aws-sdk', 'aws-sdk-2'}] }
)

s3.list_buckets.buckets.map(&:name)
#=> ['aws-sdk']

s3.list_buckets.buckets.map(&:name)
#=> ['aws-sdk', 'aws-sdk-2']

Stubbing Errors

In addition to stubbing response data, you can configure errors to raise. You can specify a service error by name, or you can provide an error object or class to raise.

# everything is broken
s3.stub_responses(:head_bucket, 
  'NotFound'
  Timeout::Error,
  RuntimeError.new('oops')
)

s3.head_bucket(bucket:'aws-sdk')
# raises Aws::S3::Errors::NotFound

s3.head_bucket(bucket:'aws-sdk')
# raises a new Timeout::Error

s3.head_bucket(bucket:'aws-sdk')
# raises RuntimeError.new('oops')

You can mix stubbed response data and errors. This approach is great when you want to test how well your code recovers from errors. 

Stubbing All Clients

The default config can be used to enable client stubbing globally. This can be very useful when writing tests to enable stubbing in your test helper once.

# stub everything
Aws.config[:stub_responses] = true

Give it a try and let us know what you think.

Waiters

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

We’ve added a feature called Waiters to the v2 AWS SDK for Ruby, and I am pretty excited about it. A waiter is a simple abstraction around the pattern of polling an AWS API until a desired state is reached.

Basic Usage

This simple example shows how to use waiters to block until a particular EC2 instance is running:

ec2 = Aws::EC2::Client.new
ec2.wait_until(:instance_running, instance_ids:['i-12345678'])

Waiters will not wait indefinitely and can fail. Each waiter has a default polling interval and maximum number of attempts to make. If a waiter encounters an unexpected error or fails to reach the desired condition in time it will raise an error:

begin
  ec2.wait_until(:instance_running, instance_ids:['i-12345678'])
resuce Aws::Waiters::Errors::WaiterFailed
  # oops
end

Configuration

You can modify the default interval and wait time between attempts by passing a block.

# this will wait upto ~ one hour
ec2.wait_until(:instance_running, instance_ids:['i-12345678']) do |w|

  # seconds between each attempt
  w.interval = 15

  # maximum number of polling attempts before giving up
  w.max_attempts = 240

end

Callbacks

In addition to interval and maximum attempts, you can configure callbacks to trigger before each attempt polling attempt and before sleeping between attempts.

ec2.wait_until(:instance_running, instance_ids:['i-12345678']) do |w|

  w.before_attempt do |n|
    # n - the number of attempts made
  end

  w.before_wait do |n, resp|
    # n - the number of attempts made
    # resp -the client response from the previous attempt
  end

end

You can throw :success or :failure from these callbacks to stop the waiter immediately. You can use this to write you own delay and back-off logic.

Here I am using a callback to perform exponential back-off between polling attempts:

ec2.wait_until(:instance_running, instance_ids:['i-12345678']) do |w|
  w.interval = 0 # disable normal sleep
  w.before_wait do |n, resp|
    sleep(n ** 2)
  end
end

This example gives up after one hour.

ec2.wait_until(:instance_running, instance_ids:['i-12345678']) do |w|
  one_hour_later = Time.now + 3600
  w.before_wait do |n, resp|
    throw :failure, 'waited too long' if Time.now > one_hour_later
  end
end

Waiters and Resources, Looking Ahead

You may have noticed that some waiters have already been exposed to the resource classes.

ec2 = Aws::EC2::Resource.new
instance = ec2.instance('i-12345678')
instance.stop
instance.wait_until_stopped
puts instance.id + ' is stopped'

In addition to connecting more waiters and resources, I’m excited to look into batch waiters. Imagine the following use case:

instances = ec2.create_instances(min_count: 5, ...)
instances.wait_until_running
puts "the following new instances are now running:n"
puts instances.map(&:id)

Documentation

Waiters are documented in the Ruby SDK API reference. Each service client documents the #wait_until method and provides a list of available waiter names. Here are links to the Aws::EC2::Client waiter methods:

Give waiters a try and let us know what you think!