Tag: ruby


Polling Messages from a Amazon SQS Queue

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

We’ve recently added a utility class to the AWS SDK for Ruby that makes it easy to poll an Amazon SQS queue for messages.

poller = Aws::SQS::QueuePoller.new(queue_url)

poller.poll do |msg|
  puts msg.body
end

Messages are automatically deleted from the queue at the end of the block. This tool supports receiving and deleting messages in batches, long-polling, client-side tracking of stats, and more.

Long Polling

By default, messages are received using long polling. This method will force a default :wait_time_seconds of 20 seconds. If you prefer to use the queue default wait time, then pass a nil value for :wait_time_seconds.

# disables 20 second default, use queue ReceiveMessageWaitTimeSeconds attribute
poller.poll(wait_time_seconds:nil) do |msg|
  # ...
end

When disabling :wait_time_seconds by passing nil, you must ensure the queue ReceiveMessageWaitTimeSeconds attribute is set to a non zero value, or you will be short polling. This will trigger significantly more API calls.

Batch Receiving Messages

You can specify a maximum number of messages to receive with each polling attempt via :max_number_of_messages. When this is set to a positive value, greater than 1, the block will receive an array of messages, instead of a single message.

# receives and yields up to 10 messages at a time
poller.poll(max_number_of_messages:10) do |messages|
  messages.each do |msg|
    # ...
  end
end

The maximum value for :max_number_of_messages is enforced by Amazon SQS.

Visibility Timeouts

When receiving messages, you have a fixed amount of time to process and delete each message before it is added back into the queue. This is the visibility timeout. By default, the queue’s VisibilityTimeout attribute is used. You can provide an alternative visibility timeout when polling.

# override queue visibility timeout
poller.poll(visibility_timeout:10) do |msg|
  # do work ...
end

You can reset the visibility timeout of a single message by calling #change_message_visibility. This is useful when you need more time to finish processing the message.

poller.poll do |msg|

  # do work ...

  # need more time for processing
  poller.change_message_visibility(msg, 60)

  # finish work ...

end

If you change the visibility timeout of a message to zero, it will return to the queue immediately.

Deleting Messages

Messages are deleted from the queue when the block returns normally.

poller.poll do |msg|
  # do work
end # messages deleted here

You can skip message deletion by passing skip_delete: true. This allows you to manually delete the messages using {#deletemessage}, or {#deletemessages}.

# single message
poller.poll(skip_delete: true) do |msg|
  poller.delete_message(msg) # if successful
end

# message batch
poller.poll(skip_delete: true, max_number_of_messages:10) do |messages|
  poller.delete_messages(messages)
end

Another way to manage message deletion is to throw :skip_delete from the poll block. You can use this to choose when a message, or message batch is deleted on an individual basis:

poller.poll do |msg|
  begin
    # do work
  rescue
    # unexpected error occurred while processing messages,
    # log it, and skip delete so it can be re-processed later
    throw :skip_delete
  end
end

Terminating the Polling Loop

By default, polling will continue indefinitely. You can stop the poller by providing an idle timeout or by throwing :stop_polling from the {#before_request} callback.

:idle_timeout

This is a configurable, maximum number of seconds to wait for a new message before the polling loop exists. By default, there is no idle timeout.

# stops polling after a minute of no received messages
poller.poll(idle_timeout: 60) do |msg|
  # ...
end

:stop_polling

If you want more fine-grained control, you can configure a before request callback to trigger before each long poll. Throwing :stop_polling from this callback will cause the poller to exit normally without making the next request.

# stop after processing 100 messages
poller.before_request do |stats|
  throw :stop_polling if stats.receive_message_count >= 100
end

poller.poll do |msg|
  # do work ...
end

Tracking Progress

The poller will automatically track a few statistics client-side in a PollerStats object. You can access the poller stats three ways:

  • The first block argument of {#before_request}
  • The second block argument of {#poll}.
  • The return value from {#poll}.

Here are examples of accessing the statistics.

  • Configure a {#before_request} callback.

    poller.before_reqeust do |stats|
      logger.info("requests: #{stats.request_count}")
      logger.info("messages: #{stats.received_message_count}")
      logger.info("last-timestamp: #{stats.last_message_received_at}")
    end
  • Accept a second argument in the poll block, for example:

    poller.poll do |msg, stats|
      logger.info("requests: #{stats.request_count}")
      logger.info("messages: #{stats.received_message_count}")
      logger.info("last-timestamp: #{stats.last_message_received_at}")
    end
  • Return value:

    stats = poller.poll(idle_timeout:10) do |msg|
      # do work ...
    end
    logger.info("requests: #{stats.request_count}")
    logger.info("messages: #{stats.received_message_count}")
    logger.info("last-timestamp: #{stats.last_message_received_at}")

Feedback

Let us know what you think about the new queue poller. Join the conversation in our Gitter channel or open a GitHub issue.

Announcing the aws-sdk-rails Gem

by Alex Wood | on | in Ruby | Permalink | Comments |  Share

With the release of V2 of the AWS SDK for Ruby, we’ve received customer feedback asking for support for the Ruby on Rails integration features provided by V1 of the SDK.

Today, we’re excited to announce the release of the aws-sdk-rails gem, available now via RubyGems and, of course, on GitHub.

To get started, add the aws-sdk-rails gem to your Gemfile:

gem 'aws-sdk-rails', '~> 1.0'

ActionMailer and Amazon Simple Email Service (SES)

The gem will automatically configure Rails to include an :aws_sdk delivery method for ActionMailer, that uses Amazon SES as a backend. It is simple to configure Rails to use this delivery method:

# config/application.rb
config.action_mailer.delivery_method = :aws_sdk

The aws-sdk-rails gem will use the AWS SDK for Ruby V2’s SES client automatically for any mail delivery event.

Logging

The gem will automatically wire the AWS SDK for Ruby’s logger to use Rails.logger by default.

You can customize the SDK log level and an optional log formatter in a config initializer:

# config/initializers/aws-sdk.rb
# log level defaults to :info
Aws.config[:log_level] = :debug

It is important to understand that all SDK log messages are logged at the same log level. Why is this important? When you set the Rails log level, you’re muting all log messages below that log level. So, if you want to, for example, only see SDK log messages in development, you might set the SDK log level to :debug as shown above, and set the Rails logger to show debug in development.

Credentials

The AWS SDK for Ruby will attempt locate credentials by searching the following locations:

  • ENV['AWS_ACCESS_KEY_ID'] and ENV['AWS_SECRET_ACCESS_KEY']
  • The shared credentials ini file at ~/.aws/credentials
  • From an instance profile when running on Amazon EC2

If you need to manually configure credentials, you should add them to your initializer:

# config/initializers/aws-sdk.rb
Aws.config[:credentials] = Aws::Credentials.new(access_key, secret)

Learn more about credentials in the AWS SDK for Ruby V2.

Never commit your credentials to source control. Besides being a security risk, it makes it very difficult to rotate your credentials.

Available Now

The aws-sdk-rails gem is available now.

As always, we’d love to hear your feedback, and welcome any Issues or Pull Requests at the aws-sdk-rails GitHub repo.

Amazon DynamoDB Document API in Ruby (Part 3 – Update Expressions)

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

As we showed in previous posts, it’s easy to put JSON items into Amazon DynamoDB, retrieve specific attributes with projection expressions, and fetch only data that meet some criteria with condition expressions. Now, let’s take a look at how we can conditionally modify existing items with Update Expressions. (Note: this code uses the same ProductCatalog table we used in Parts 1 and 2).

In the following examples, we use the following helper method to perform conditional updates. It performs the UpdateItem operation with return_values set to return the old item. We also use the GetItem operation so the method can return both the old and new items for us to compare. (If the update condition in the request is not met, then the method sets the returned old item to nil.)

def do_update_item(key_id, update_exp, condition_exp, exp_attribute_values)
  begin
    old_result = @dynamodb.update_item(
      :update_expression => update_exp,
      :condition_expression => condition_exp,
      :expression_attribute_values => exp_attribute_values,
      :table_name => "ProductCatalog",
      :key => { :Id => key_id },
      :return_values => "ALL_OLD",
    ).data.attributes
  rescue Aws::DynamoDB::Errors::ConditionalCheckFailedException
    old_result = nil
    puts "Condition not met"
  end

  new_result = @dynamodb.get_item(
    :table_name => "ProductCatalog", :key => { :Id => key_id },
    :consistent_read => true
  ).data.item  

  return old_result, new_result
end

Using Conditional Update Expressions

Updates in DynamoDB are atomic. This allows applications to concurrently update items without worrying about conflicts occurring. For example, the following code demonstrates maintaining a MAX value in DynamoDB with a conditional update using SET. Note that, because DynamoDB is schema-less, we don’t need to define the HighestRating attribute beforehand. Instead, we create it on the first call.

# storing a "max" value with conditional SET
# SET attribute if doesn't exist, otherwise SET if stored highest rating < this rating
def update_highest_rating(rating)
  do_update_item(303,
    "SET HighestRating = :val",
    "attribute_not_exists(HighestRating) OR HighestRating < :val",
    {
      ":val" => rating
    }
  )
end

# multiple threads trying to SET highest value (ranging from 0 to 10)
threads = []
(0..10).to_a.shuffle.each { |i|
  # some number of "Condition not met" depending on shuffled order
  puts i
  threads[i] = Thread.new {
    update_highest_rating(i)
  }
}
threads.each {|t| t.join}

# fetch the item and examine the HighestRating stored
puts "Max = #{@dynamodb.get_item(
  :table_name => "ProductCatalog", :key => { :Id => 303 }
).data.item["HighestRating"].to_i}"   # Max = 10

We can also use update expressions to atomically maintain a count and add to a set:

# ADD to intialize/increment and add to set
threads = []
20.times do |i|
  threads[i] = Thread.new {
    do_update_item(303,
      "ADD TimesViewed :val, Tags :was_here",
      nil, # no condition expression
      {
        # Each of the 20 threads increments by 1
        ":val" => 1,

        # Each thread adds to the tag set
        # Note: type must match stored attribute's type
        ":was_here" => Set.new(["#Thread#{i}WasHere"])
      }
    )
  }
end
threads.each {|t| t.join}

# fetch the item and examine the TimesViewed attribute
item = @dynamodb.get_item(
  :table_name => "ProductCatalog", :key => { :Id => 303 }
).data.item

puts "TimesViewed = #{item["TimesViewed"].to_i}"
# TimesViewed = 20

puts "Tags = #{item["Tags"].inspect}"
# Tags = #<Set: {"#Mars", "#MarsCuriosity", "#StillRoving", ..each thread was here...}>

Similarly, we can decrement the count and remove from the set to undo our previous operations.

# Undo the views and set adding that we just performed
threads = []
20.times do |i|
  threads[i] = Thread.new {
    do_update_item(303,
      "ADD TimesViewed :val DELETE Tags :was_here",
      nil,  # no condition expression
      {
        # Each of the 20 threads decrements by 1
        ":val" => -1,

        # Each thread removes from the tag set
        # Note: type must match stored attribute's type
        ":was_here" => Set.new(["#Thread#{i}WasHere"])
      }
    )
  }
end
threads.each {|t| t.join}

# fetch the item and examine the TimesViewed attribute
item = @dynamodb.get_item(
  :table_name => "ProductCatalog", :key => { :Id => 303 }
).data.item

puts "TimesViewed = #{item["TimesViewed"].to_i}"
# TimesViewed = 0

puts "Tags = #{item["Tags"].inspect}"
# Tags = #<Set: {"#Mars", "#MarsCuriosity", "#StillRoving"}>

We can also use the REMOVE keyword to delete attributes, such as the HighestRating and TimesViewed attributes we added in the previous code.

# removing attributes from items
old_and_new = do_update_item(303,
  "REMOVE HighestRating, TimesViewed",
  nil,  # no condition expression
  nil,  # no attribute expression values
)

puts "OLD HighestRating is nil ? #{old_and_new[0]["HighestRating"] == nil}"
#=> false

puts "OLD TimesViewed is nil ? #{old_and_new[0]["TimesViewed"] == nil}"
#=> false

puts "NEW HighestRating is nil ? #{old_and_new[1]["HighestRating"] == nil}"
#=> true

puts "NEW TimesViewed is nil ? #{old_and_new[1]["TimesViewed"] == nil}"
#=> true

Conclusion

We hope this series was helpful in demonstrating expressions and how they allow you to interact with DynamoDB more flexibly than before. We’re always interested in hearing what developers would like to see in the future, so let us know what you think in the comments or through our forums!

Amazon DynamoDB Document API in Ruby (Part 2 – Condition Expressions)

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

As we showed in the previous post, it’s easy to put JSON items into Amazon DynamoDB and retrieve specific attributes with projection expressions. Condition Expressions provide a more flexible and SQL-like way to retrieve only the items you want from DynamoDB. First, let’s put a few more items into DynamoDB using a BatchWriteItem operation. (Note: this code uses the same ProductCatalog table we used in Part 1)

# add some more items
@dynamodb.batch_write_item(
  :request_items => {
    "ProductCatalog" => [

      {:put_request => { :item => {
        Id: 300,
        Title: "Sojourner",
        Description: "Mars Pathfinder robotic Mars rover",
        Price: BigDecimal.new("2.65e8"),
        LaunchDate: {
          M: 12, D: 4, Y: 1996
        },
        LostCommunicationDate: {
          M: 9, D: 27, Y: 1997
        },
        Features: {
          Rover: true,
        },
        NumberInStock: 10,
        OrdersPlaced: 3,
        Tags: ["#Mars", "#InStarTrekSeason4", "#InRedPlant2000", "#LostComms"],
      }}},

      {:put_request => { :item => {
        Id: 301,
        Title: "Spirit",
        Description: "Mars Exploration Rover – A",
        Price: BigDecimal.new("4.1e8"),
        LaunchDate: {
          M: 6, D: 10, Y: 2003
        },
        LostCommunicationDate: {
          M: 3, D: 22, Y: 2010
        },
        Features: {
          Rover: true,
        },
        NumberInStock: 10,
        OrdersPlaced: 5,
        Tags: Set.new(["#Mars", "#StuckOnMars", "#LostComms"]),
      }}},

      {:put_request => { :item => {
        Id: 302,
        Title: "Opportunity",
        Description: "Mars Exploration Rover – B",
        Price: BigDecimal.new("4.1e8"),
        LaunchDate: {
          M: 7, D: 7, Y: 2003
        },
        LostCommunicationDate: nil,
        Features: {
          Rover: true,
        },
        NumberInStock: 10,
        OrdersPlaced: 10,
        Tags: Set.new(["#Mars", "#StillRoving"]),
      }}},

      {:put_request => { :item => {
        Id: 303,
        Title: "Curiosity",
        Description: "car-sized robotic rover",
        Price: BigDecimal.new("2.5e9"),
        LaunchDate: {
          M: 11, D: 26, Y: 2011
        },
        LostCommunicationDate: nil,
        Features: {
          Rover: true,
          RoboticArm: true,
        },
        NumberInStock: 0,
        OrdersPlaced: 30,
        Tags: Set.new(["#Mars", "#MarsCuriosity", "#StillRoving"]),
      }}},

    ]
  }
)

Using Condition Expressions

We could also use condition expressions on the results of Query, but since we’re using a simple data model (only have hash key on product Id), we demonstrate this with scans. We use the following helper method to perform the scan and format the product titles returned:

def do_scan(filter_exp, exp_attribute_values)
  result = @dynamodb.scan(
    :expression_attribute_values => exp_attribute_values,
    :filter_expression => filter_exp,   # Condition Expressions are supplied through the FilterExpression parameter
    :projection_expression => "Title",
    :table_name => "ProductCatalog"
  ).data.items

  # format all retrieved titles into a single line
  return "scan retrieved: #{(result.map { |item| item["Title"] }).join(", ")}"
end

Let’s look at some example expressions and the results they return from our current ProductCatalog table:

# All products that don't have a launch month of November (11)
puts do_scan(
  "LaunchDate.M <> :m",
  {
    ":m" => 11
  }
)
# scan retrieved: 20-Bicycle 205, Opportunity, Spirit, Sojourner


# All rover products that don't have a launch month of November
puts do_scan(
  "attribute_exists(Features.Rover) AND LaunchDate.M <> :m",
  {
    ":m" => 11,
  }
)
# scan retrieved: Opportunity, Spirit, Sojourner


# Non-rovers
puts do_scan(
  "attribute_not_exists(Features.Rover)",
  nil
)
# scan retrieved: 20-Bicycle 205


# mid-range rovers or inexpensive products
puts do_scan(
  "(Price BETWEEN :low AND :high) OR Price < :verylow",
  {
    ":verylow" => BigDecimal.new("1e8"),
    ":low" => BigDecimal.new("3e8"),
    ":high" => BigDecimal.new("5e8")
  }
)
# scan retrieved: 20-Bicycle 205, Opportunity, Spirit


# within-Item referencing: more orders placed than in stock
puts do_scan(
  "OrdersPlaced > NumberInStock",
  nil
)
# scan retrieved: Curiosity


# string prefixing
puts do_scan(
  "begins_with(Title, :s)",
  {
    ":s" => "S",
  }
)
# scan retrieved: Spirit, Sojourner


# contains
puts do_scan(
  "contains(Tags, :tag1) AND contains(Tags, :tag2)",
  {
    ":tag1" => "#StuckOnMars",
    ":tag2" => "#LostComms",
  }
)
# scan retrieved: Spirit


# contains (Note: "Tags" is a list for Sojourner)
puts do_scan(
  "contains(Tags, :tag1)",
  {
    ":tag1" => "#LostComms",
  }
)
# scan retrieved: Spirit, Sojourner


# in operator
puts do_scan(
  "Id in (:id1, :id2)",
  {
    ":id1" => 302,
    ":id2" => 303,
  }
)
# scan retrieved: Curiosity, Opportunity


# equivalently, with parentheses
puts do_scan(
  "(Id = :id1) OR (Id = :id2)",
  {
    ":id1" => 302,
    ":id2" => 303,
  }
)
# scan retrieved: Curiosity, Opportunity

Next Steps

As you can see, condition expressions enable you to write more concise code to retrieve data. They also provide querying capabilities unavailable with the original access model such as within-Item references and more flexible conditions with parentheses. In an upcoming blog post, we’ll take a closer look at how we can update existing data through update expressions.

Amazon DynamoDB Document API in Ruby (Part 1 – Projection Expressions)

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

Amazon DynamoDB launched JSON Document Support along with several improvements to the DynamoDB API. This post is part of a series where we’ll explore these features in more depth with the AWS SDK for Ruby V2. In particular, this post focuses on putting items into DynamoDB using the Ruby SDK and controlling the data we get back with projection expressions. At the end of the post, we also provide some helpful information for getting started with DynamoDB Local.

Putting JSON data into DynamoDB

DynamoDB now supports the following new data types: Maps, Lists, Booleans, and Nulls. Suppose we have a DynamoDB table for products with a hash key on an "Id" attribute. It’s easy to store such data into DynamoDB with native Ruby types:

# put a JSON item
item = {
  Id: 205, # hash key
  Title: "20-Bicycle 205",
  Description: "205 description",
  BicycleType: "Hybrid",
  Brand: "Brand-Company C",
  Price: 500,
  Gender: "B",
  Color: Set.new(["Red", "Black"]),
  ProductCategory: "Bike",
  InStock: true,
  QuantityOnHand: nil,
  NumberSold: BigDecimal.new("1E4"),
  RelatedItems: [
    341, 
    472, 
    649
  ],
  Pictures: { # JSON Map of views to url String
    FrontView: "http://example.com/products/205_front.jpg", 
    RearView: "http://example.com/products/205_rear.jpg",
    SideView: "http://example.com/products/205_left_side.jpg",
  },
  ProductReviews: { # JSON Map of stars to List of review Strings
    FiveStar: [
      "Excellent! Can't recommend it highly enough!  Buy it!",
      "Do yourself a favor and buy this."
    ],
    OneStar: [
      "Terrible product!  Do not buy this."
    ]
  }
}
dynamodb.put_item(:table_name => "ProductCatalog", :item => item)

Getting data from DynamoDB using projection expressions

Since DynamoDB now supports more interesting data types, we’ve also added projection expressions and expression attribute names to make it easier to retrieve only the attributes we want:

# get only the attributes we want with projection expressions
item = dynamodb.get_item(
  :table_name => "ProductCatalog",

  # Get the item with Id == 205
  :key => {
    :Id => 205
  },

  # for less typing, use expression attribute names to substitute
  # "ProductReviews" with "#pr" and "RelatedItems" with "#ri"
  :expression_attribute_names => {
    "#pr" => "ProductReviews",
    "#ri" => "RelatedItems",
  },

  # get Price, Color, FiveStar reviews, 0th and 2nd related items
  :projection_expression => "Price, Color, #pr.FiveStar, #ri[0], #ri[2], 
    #pr.NoStar, #ri[4]" # try projecting non-existent attributes too
).data.item

puts item["Price"].to_i
# 500

puts item["Color"].inspect
# #<Set: {"Black", "Red"}>

puts item["ProductReviews"]["FiveStar"][0]
# Excellent! Can't recommend it highly enough!  Buy it!

puts item["ProductReviews"]["FiveStar"][1]
# Do yourself a favor and buy this.

puts item["ProductReviews"]["OneStar"].inspect
# nil (because we only projected FiveStar reviews)

puts item["ProductReviews"]["NoStar"].inspect
# nil (because no NoStar reviews)

puts item["RelatedItems"]
# 0.341E3   (0th element)
# 0.649E3   (2nd element)

puts item["RelatedItems"].size
# 2 (non-existent 4th element not present)

Next Steps

As you can see, it’s easy to put and get items in DynamoDB with the AWS SDK for Ruby. In upcoming blog posts, we’ll take a closer look at expressions for filtering and updating data.

Feel free to get started on DynamoDB Local with the following code (note that it uses the credentials file approach for specifying AWS credentials):

#! /usr/bin/ruby

require "set"
require "bigdecimal"
require "aws-sdk-core"

# Configure SDK

# use credentials file at .aws/credentials
Aws.config[:credentials] = Aws::SharedCredentials.new
Aws.config[:region] = "us-west-2"

# point to DynamoDB Local, comment out this line to use real DynamoDB
Aws.config[:dynamodb] = { endpoint: "http://localhost:8000" }

dynamodb = Aws::DynamoDB::Client.new

## Create the table if it doesn't exist
begin
  dynamodb.describe_table(:table_name => "ProductCatalog")
rescue Aws::DynamoDB::Errors::ResourceNotFoundException
  dynamodb.create_table(
    :table_name => "ProductCatalog",
    :attribute_definitions => [
      {
        :attribute_name => :Id,
        :attribute_type => :N
      }
    ],
    :key_schema => [
      {
        :attribute_name => :Id,
        :key_type => :HASH
      }
    ],
    :provisioned_throughput => {
      :read_capacity_units => 1,
      :write_capacity_units => 1,
    }
  )

  # wait for table to be created
  puts "waiting for table to be created..."
  dynamodb.wait_until(:table_exists, table_name: "ProductCatalog")
  puts "table created!"
end

Announcing V2 of the AWS SDK for Ruby

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

I am excited to announce today’s stable release of version 2 of the AWS SDK for Ruby. It is available now as the aws-sdk gem on RubyGems.

Features

Version 2 of the AWS SDK for Ruby, the aws-sdk gem, provides a number of powerful features for developers including:

Upgrading

Version 2 of the AWS SDK for Ruby uses a different namespace, making it possible to use version 1 and version 2 in the same application.

# Gemfile
gem 'aws-sdk', '~> 2'
gem 'aws-sdk-v1'

# code
require 'aws-sdk-v1'
require 'aws-sdk'

ec2_v1 = AWS::EC2.new # v1
ec2_v2 = Aws::EC2::Resource.new # v2

This allows you to start using the version 2 SDK today without changing existing code.

Feedback

Please share your questions, comments, issues, etc. with us on GitHub. You can also catch us in our Gitter channel.

Upcoming Stable Release of AWS SDK for Ruby – Version 2

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

We plan to release version 2 of the AWS SDK for Ruby next week. We will remove the preview flag from the 2.0 version of aws-sdk.

Specify Your Version Dependencies

The AWS SDK for Ruby uses semantic versioning. Updates within version 1 are backwards compatible.

Version 2 of the aws-sdk gem is not backwards compatible.

If you depend on the aws-sdk gem today, and do not specify the major version, please add this now. If not, you may run into issues when you bundle update.

# Gemfile
gem 'aws-sdk', '< 2.0'

# gemspec
spec.add_dependency('aws-sdk', '< 2.0')

NameError: uninitialized constant AWS

If you receive this error, you likely have a dependency on aws-sdk and have updated so that you now have version 2 installed. Version 2 uses a different module name, so it does not define AWS.

To resolve this issue, specify your version dependency as instructed above.

Using Both Versions

The following diagram shows how the version 1 and version 2 gems are organized.

The aws-sdk gem is empty, and simply requires version 1 or version 2 specific gems. This allows you to install version 1 and version 2 in the same application.

Option A, for existing users

# Gemfile
gem 'aws-sdk', '~> 1'
gem 'aws-sdk-resources', '~> 2'

# in code
require 'aws-sdk'
require 'aws-sdk-resources'

Option B, for new users

# Gemfile
gem 'aws-sdk-v1'
gem 'aws-sdk', '~> 2'

# in code
require 'aws-sdk-v1'
require 'aws-sdk'

Attention Library Authors

If you maintain a gem that has a dependency on version 1 of aws-sdk, I strongly recommend that you replace it with a dependency on aws-sdk-v1. This allows end users to require version 2 of aws-sdk.

Please report any issues you have on our GitHub repository.

Using Amazon RDS with Ruby on Rails and AWS OpsWorks

by Alex Wood | on | in Ruby | Permalink | Comments |  Share

Earlier in this blog series, we showed you how to deploy a Ruby on Rails application to Amazon Web Services using AWS OpsWorks. In that example, we used an OpsWorks-managed MySQL database run on an Amazon EC2 instance. One common piece of feedback on that post was a desire to see how you can set up your stack with Amazon RDS. Today, we are going to show you how.

Prerequisites

We are going to assume you’re familiar with our earlier post on how to deploy with OpsWorks in general. For this tutorial you can take one of the following approaches:

  • Take your existing stack from following along with that post, delete your database and MySQL layers, then follow this tutorial.
  • Clone your existing stack (in case you don’t want to lose your work), create a new app server instance, delete the new stack’s MySQL layer, then follow along.
  • Start from a brand new stack, and go between both tutorials, replacing the DB steps from the previous tutorial with the DB steps here. If you’re going to go that route, we would recommend reading this tutorial first to understand the differences in approach.

Whichever approach you choose, you can be up and running in just a few minutes!

Create an RDS Instance

AWS OpsWorks will not create Amazon RDS instances on your behalf. You will need to create an instance and link it to OpsWorks.

  1. Open up the RDS console, navigate to Instances, and click Launch DB Instances.
  2. To be able to use our Rails example code unaltered, choose the MySQL engine (you can choose another engine if you like, see the next section for details).
  3. On the next screen, you can choose either Multi-AZ deployments or single-AZ, but if you want to stay within the RDS Free Usage Tier, then you should not choose a multi-AZ deployment.
  4. Make sure that you keep track of your Master Username, Master Password, and the Database Name from the rest of the form.
  5. Within Configure Advanced Settings, make sure you set your VPC Security Groups to include AWS-OpsWorks-DB-Master-Server and AWS-OpsWorks-Rails-App-Server. You can set Publicly Accessible to "No" as well.
  6. Once you’ve completed the forms, click Launch DB Instance. It may take a few minutes for your instance to launch, so now is not a bad time for tea or coffee.

Don’t Want MySQL?

You do not need to use MySQL as your database backend, that is just what we have chosen for this example. Want to use a different database backend? Just do the following:

  1. Replace the mysql2 gem with the adapter gem appropriate to your DB engine selection.
  2. Make sure you also use your adapter selection in your custom deployment JSON, found in your stack settings if you used our exact tutorial steps.
  3. Of course, select that database engine when creating your RDS instance.

Create an RDS Layer

Now that you have an RDS Instance, you need to register it with OpsWorks.

  1. Navigate to the OpsWorks console, to the Layers section.
  2. Click + Layer, and select the RDS tab.
  3. Your RDS instance should be on this screen to select. Select it, then enter your User and Password from the RDS instance creation step.
  4. Click Register with Stack.

This registers your database with OpsWorks, and next you need to connect your app.

Attach Your RDS Instance to Your App

  1. From the Apps section, click on Edit for TodoApp (or whatever app you are developing for).
  2. Under the Data Sources section, select RDS.
  3. It should auto-select your instance; select it if not. Then, fill in your Database name from the RDS instance creation process.
  4. Click Save to save your changes.

Note that this should work for any of the topic branches we’ve made for TodoApp, as your choice of database host is transparent to your app so long as you have the correct database adapter gem in your Gemfile.

Deploy the App

To use your new database, simply run an app deployment, making sure you select the Migrate database option so that Rails can set up your database.

Once that’s done, navigate to your app server host and play around with your app. You’re now running on RDS!

AWS re:Invent 2014 Ruby Recap

by Alex Wood | on | in Ruby | Permalink | Comments |  Share

Last week, we had a great time meeting with AWS customers using Ruby at AWS re:Invent! We appreciate the feedback we received, and the discussions that we had with you.

AWS SDK for Ruby Presentation Notes

At AWS re:Invent this year I took many of you on a tour of version 2 of the AWS SDK for Ruby. We were thrilled to have such a great audience, and we had a great time meeting with many of you both before and after the talk.

In the presentation, we talked about the architecture and new features available in version 2 of the AWS SDK for Ruby, including resources, pagination, and waiters. We also walked through an end-to-end example using Ruby on Rails.

If you didn’t get a chance to make it to my talk, I encourage you to check it out. If you did, you still might find it worthwhile to code along with our Rails example. You can skip ahead to that here if you like.

The presentation also had a lot of links that you might want to check out. For your convenience, we’ve compiled them here:

See You Next Year

AWS re:Invent is October 6 – 9, 2015, once again at The Venetian in Las Vegas, NV. Let us know what you’d like to see next on Twitter @awsforruby and we hope to see you there!

Client Response Stubs

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

We recently added client response stubs to the aws-sdk-core gem. Response stubbing disables network traffic and causes a client to return fake or stubbed data.

# no API calls are made
s3 = Aws::S3::Client.new(stub_responses: true)
s3.list_buckets.buckets.map(&:name)
#=> []

Custom Response Data

By default, stubbed responses return empty lists, empty maps, and placeholder scalars. These empty responses can be useful at times, but often you want to control the data returned.

s3.stub_responses(:list_buckets, buckets:[{name:'aws-sdk'}])
s3.list_buckets.buckets.map(&:name)
#=> ['aws-sdk']

Safe Stubbing

One of the common risks when writing tests with stub data is that the stub doesn’t match the shape of the actual response. You risk coding against stubs that provide methods that won’t exist out of your tests.

We resolve this issue by validating your stub data hash against the model of the API. An ArgumentError is raised when calling #stub_responses with invalid data.

s3.stub_resposnes(:list_buckets, buckets:['aws-sdk'])
#=> raises ArgumentError, "expected params[:buckets][0] to be a hash"

Stubbing Multiple Calls

By calling #stub_responses with an operation name and stub data, the client will serve that data for each call. You can specify multiple responses, and they will be used in sequence.

s3.stub_responses(:list_buckets, 
  { buckets:[{name:'aws-sdk'}] },
  { buckets:[{name:'aws-sdk', 'aws-sdk-2'}] }
)

s3.list_buckets.buckets.map(&:name)
#=> ['aws-sdk']

s3.list_buckets.buckets.map(&:name)
#=> ['aws-sdk', 'aws-sdk-2']

Stubbing Errors

In addition to stubbing response data, you can configure errors to raise. You can specify a service error by name, or you can provide an error object or class to raise.

# everything is broken
s3.stub_responses(:head_bucket, 
  'NotFound'
  Timeout::Error,
  RuntimeError.new('oops')
)

s3.head_bucket(bucket:'aws-sdk')
# raises Aws::S3::Errors::NotFound

s3.head_bucket(bucket:'aws-sdk')
# raises a new Timeout::Error

s3.head_bucket(bucket:'aws-sdk')
# raises RuntimeError.new('oops')

You can mix stubbed response data and errors. This approach is great when you want to test how well your code recovers from errors. 

Stubbing All Clients

The default config can be used to enable client stubbing globally. This can be very useful when writing tests to enable stubbing in your test helper once.

# stub everything
Aws.config[:stub_responses] = true

Give it a try and let us know what you think.