Tag: security

Using Client-Side Encryption for S3 in the AWS SDK for Ruby

by Alex Wood | on | in Ruby | Permalink | Comments |  Share

What is client-side encryption, and why might I want to use it?

If you wish to store sensitive data in Amazon S3 with the AWS SDK for Ruby, you have several ways of managing the safety and security of the data. One good practice is to use HTTPS whenever possible to protect your data in transit. Another is to use S3’s built in server-side encryption to protect your data at rest. In this post, we highlight yet another option, client-side encryption.

Client-side encryption is a little more involved than server-side encryption, since you bring your own security credentials. But, it has the added benefit that data never exists in an unencrypted state outside of your execution environment.

How do I use client-side encryption with the AWS SDK for Ruby?

The SDK for Ruby does most of the heavy lifting for you when using client-side encryption for your S3 objects. When performing read and write operations on S3, you can specify various options in an option hash passed in to the S3Object#write and S3Object#read methods.

One of these options is :encryption_key, which accepts either an RSA key (for asymmetric encryption), or a string (for symmetric encryption). The SDK for Ruby then uses your key to encrypt an auto-generated AES key, which is used to encrypt and decrypt the payload of your message. The encrypted form of your auto-generated key is stored with the headers of your object in S3.

Here is a short example you can try to experiment with client-side encryption yourself:

require 'aws-sdk'
require 'openssl'

# Set your own bucket/key/data values
bucket = 's3-bucket'
key = 's3-object-key'
data = 'secret message'

# Creates a string key - store this!
symmetric_key = OpenSSL::Cipher::AES256.new(:CBC).random_key

options = { :encryption_key => symmetric_key }
s3_object = AWS.s3.buckets[bucket].objects[key]

# Writing an encrypted object to S3
s3_object.write(data, options)

# Reading the object from S3 and decrypting
puts s3_object.read(options)

There are a couple practical matters you should consider. One is that if you lose the key used to encrypt the object, you will be unable to decrypt your contents. You should securely store your key (e.g., as a file or using a separate key management system) and load it when needed for writing or reading objects. Additionally, encryption and decryption on your objects does bring with it some performance overhead, so you should use it only when needed (this overhead varies depending on the size and type of key used).

You can read more about the encryption choices available to you with the AWS SDK for Ruby in our API documentation. You can also read more about general best practices for security in AWS by following the AWS Security Blog. As you consider the choices available for securing your data, we hope you find them effective and simple to use.

IAM Roles for Amazon EC2 Instances (Credential Management Part 4)

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

This is the fourth and final part (part 1, part 2, part 3) in a series on how to securely manage your AWS access credentials.

This week I am focusing on using AWS Identity and Access Management (IAM) roles for Amazon EC2 instances with the AWS SDK for Ruby (aws-sdk). Simply put, IAM roles for EC2 instances remove the need to try to bootstrap your instances with credentials. Example:

# look ma! No credentials (when run from EC2)!
require 'aws-sdk'

You can stop worrying about how to get credentials onto a box that Auto Scaling just spun-up on your behalf. Your cron scripts on instances no longer have to search around for credentials on disk or worry about when they get rotated.

One of the best features of IAM roles for EC2 instances is the credentials on EC2 are auto-rotated! Instances started with an IAM instance profile will get temporary credentials deployed on a regular basis to the EC2 instance metadata service.

My favorite feature? The aws-sdk gem just works when running on an instance with an IAM profile.

How it Works

  • You create an IAM instance profile
  • Start one or more instances, with the instance profile
  • Upon instance boot, session credentials will be available on your instance(s)
  • Credentials are rotated regularly (before they expire)

You can reuse an instance profile as many times as you like. The aws-sdk gem will automatically attempt to load credentials from the metadata service when no other credentials are provided.

Create an Instance Profile with the aws-sdk Gem

An instance profile consists of a role. The role consists of a policy. Each of these has a name. I am going to use the aws-sdk to create a sample profile starting with the policy.

For this example, I’ll be building an instance profile that has limited permissions. I only want applications on EC2 to be able to read from Amazon S3 (list and get buckets/objects). First, I need to build the policy.

require 'aws-sdk'

AWS.config(:access_key_id => '...', :secret_access_key => '…')

# the role, policy and profile all have names, pick something descriptive
role_name = 's3-read-only'
policy_name = 's3-read-only'
profile_name = 's3-read-only'

# required so that Amazon EC2 can generate session credentials on your behalf
assume_role_policy_document = '{"Version":"2008-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":["ec2.amazonaws.com"]},"Action":["sts:AssumeRole"]}]}'

# build a custom policy    
policy = AWS::IAM::Policy.new
policy.allow(:actions => ["s3:Get*","s3:List*"], :resources => '*')

Now that I have a policy built, I am going to build a role and add the policy to the role. I am going to use the policy and role names I chose above.

iam = AWS::IAM.new

# create the role
  :role_name => role_name,
  :assume_role_policy_document => assume_role_policy_document)

# add the policy to role
  :role_name => role_name,
  :policy_name => policy_name,
  :policy_document => policy.to_json)

Last step is to create a profile for your role.

resp = iam.client.create_instance_profile(
  :instance_profile_name => instance_profile_name)

# this may be handy later
profile_arn = resp[:instance_profile][:arn]

  :instance_profile_name => instance_profile_name,
  :role_name => role_name)

Using an IAM Instance Profile

You can now use the instance profile name (or ARN we captured above) to run instances with your special profile.

# you can use the profile name or ARN as the :iam_instance_profile option
ec2 = AWS::EC2.new
ec2.instances.create(:image_id => "ami-12345678", :iam_instance_profile => profile_name)

Thats it! Your new instance will boot with session credentials available that the aws-sdk can consume with zero configuration. Happy computing!

Credential Providers (Credential Management Part 3)

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

In part 1 of this series, I wrote about how to configure your access credentials with the AWS SDK for Ruby (aws-sdk gem). In part 2 we learned how to rotate your access credentials using the aws-sdk gem.

This week we explore credential providers and how they can help you keep your secrets safe and fresh.

Credential Providers

A credential provider is an object that responds to the following methods:

  • #access_key_id
  • #secret_access_key
  • #session_token
  • #refresh

Internally, the aws-sdk gem uses a chain of credential providers to load credentials from various locations including:

  • AWS.config
  • ENV (from multiple key prefixes)
  • EC2 instance metadata service

You can fully customize this behavior by configuring your own custom credential provider, as follows.

AWS.config(:credential_provider => CustomCredentialProvider.new)

Why Use a Custom Credential Provider?

In the previous post in this series we discussed rotating credentials. It can be painful to build logic that restarts processes or applications that are using stale or soon-to-be removed/expired credentials.

If your application uses a custom credential provider, the application does not need to be restarted. The SDK automatically calls #refresh on the credential provider when it receives a response from AWS that indicates its credentials are expired.

In-house hosted applications and utility scripts can use a custom provider to load credentials from a simple web service that vends current credentials. This web service could even go as far as vending session credentials that auto-expire and are limited to specific AWS operations. This greatly reduces exposure if these credentials are ever leaked.

Build a Custom Credential Provider

Here is a really simple custom credential provider that makes a HTTPS request to https://internal.domain/ and expects a JSON response of credentials.

require 'net/https'

class CustomCredentialProvider

  include AWS::Core::CredentialProviders::Provider


  def get_credentials
      http = Net::HTTP.new('internal.domain', 443)
      http.use_ssl = true
      http.verify_mode = OpenSSL::SSL::VERIFY_PEER
      response = http.request(Net::HTTP::Get.new('/'))

      # symbolize keys to :access_key_id, :secret_access_key
      if response.code == '200'
        JSON.load(response.body).inject({}) {|h,(k,v)| h.merge(k.to_sym => v) }


The include statement in the example above does much of the heavy lifting. It defines all of the public methods required for a credential provider. It also caches the credentials until #refresh is called. We only have to define #get_credentials and return a hash with symbol keys (or an empty hash if we fail).

You can make this example more robust if you:

  • Set network timeouts
  • Rescue Exceptions raised by HTTP that do not extend StandardError
  • Add basic retry logic to handle transient network errors

In the next (and last) post in this series I will explore how the aws-sdk gem uses the EC2 metadata service.

Rotating Credentials (Credential Management Part 2)

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

In a previous blog post I wrote about ways to securely configure your AWS access credentials when using the aws-sdk gem. This week I want to talk about a security best practice, credential rotation.

Did you know that AWS recommends that you rotate your access keys every 90 days?

Even if you are very careful with your AWS access credentials, you may find yourself in a situation where someone has gained access to your secrets. If you build your applications with a regular key rotation solution, then an out-of-bounds replacement of keys can be painless. In the heat of the moment when you are scrambling to replace compromised keys, this can be a life saver.

Rotating Credentials

The process for rotating credentials boils down to the following steps:

  • Generate new keys
  • Securely distribute keys to your applications
  • Ensure the applications refresh their keys
  • Disable the old access keys
  • Ensure everything still works
  • Delete the old access keys

For best effect, you should automate this process. If you have to do it by hand, the process will be much more error prone and you will likely do it less often. You can use the aws-sdk gem to do much of the work for you.

This simple example demonstrates how to generate a new key pair, disable old keys and then eventually delete the old keys. I inserted placeholders for where you should distribute your new keys and refresh your applications with the new keys.

iam = AWS::IAM.new

# create new set of access credentials
new_keys = iam.access_keys.create

# you should persist the new key pair somewhere secure to start with
new_keys.id # access key id
new_keys.secret # secret access key

## deploy the new keys to your applications now, make
## sure they pick up the new keys

# deactivate the old keys
old_keys = iam.access_keys['AKID12346789…'] # old access key id

## the old keys still exist, they are temporarily disabled, use
## this time to test your applications to ensure they are working

# if you are confident your applications are using the new keys
# you can then safely delete the old key pair

How you distribute your keys and refresh your application is going to be very specific to your own needs. Just be certain to test your applications before you delete your disabled keys. You can not restore them once they have been deleted.

For the next post in this series, I will write about credential providers and how the aws-sdk makes it easy for your applications to pick up new credentials without restarts or downtime. This can be very useful when you are rotating credentials.

Credential Management – Part 1

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

When using AWS, it is important to keep your access credentials secure. It can be challenging to make your credentials available to your application securely. The AWS SDK for Ruby provides a number of helpful interfaces for configuring your credentials that help you keep your secrets safe.

This blog post focuses on securely configuring the aws-sdk gem with your credentials. I will follow up with additional blog posts about rotating credentials, and using roles for instances.

Credential Configuration

The aws-sdk gem requires that you provide your access credentials before making a request. The gem tries to locate your credentials in a number of default locations. If it is unable to find your credentials, it raises an error. The locations it searches are:

  • AWS.config
  • ENV
  • EC2 instance metadata
  • Rails Configuration (RAILS_ROOT/config/aws.yml)


You can use AWS.config to statically configure your credentials for all AWS requests. Simply call AWS.config with your :access_key_id and :secret_access_key (you may also provide a :session_token).

require 'aws-sdk'

AWS.config(:access_key_id => '...', :secret_access_key => '...')

Alternatively, you may provide credentials directly to the service interfaces. This can be helpful if you are working with multiple sets of credentials for multiple AWS accounts (or IAM users).

s3 = AWS::S3.new(:access_key_id => '...', :secret_access_key => '...')    

As tempting as it might be, you should never put access credentials in source code. This makes your secrets available to anyone with access to your source code. It also creates a maintenance burden when you need to rotate your credentials. There are many alternatives, including loading credentials from a configuration file that is not tracked with source control.

require 'aws-sdk'
require 'yaml'

# configure aws-sdk gem from a yaml configuration file

In the preceding example, the YAML file might look like this:

:access_key_id: 'YOUR_ACCESS_KEY_ID'
:secret_access_key: 'YOUR_SECRET_ACCESS_KEY'


You can alternatively provide credentials to your application via ENV. By default, the aws-sdk gem searches ENV using two different prefixes for your keys (‘AWS’ and ‘AMAZON’).


EC2 Instance Metadata

The aws-sdk gem supports loading credentials from the instance metadata service on Amazon EC2 instances. If you have started your instance using an AWS IAM instance profile, this will just work, no configuration required. I will dive into this deeper in a followup blog post.

Credential Loading in Rails (RAILS_ROOT/config/aws.yml)

If you are using the aws-sdk gem in a Rails application, the gem attempts to load credentials from RAILS_ROOT/config/aws.yml. This file is ERB-parsed and then loaded by YAML. This file should be formatted in the same way as RAILS_ROOT/config/database.yml. It should have one top-level entry for the Rails.env you are running.

# RAILS_ROOT/config/aws.yml
  access_key_id: DEV_ACCESS_KEY_ID
  secret_access_key: DEV_ACCESS_KEY_ID

If this file contains your secrets in plain text and does not use ERB to load them securely, you should not track this file with source control.

You can now read part two in this series: Rotating Credentials.