AWS Developer Blog

Credential Management – Part 1

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

When using AWS, it is important to keep your access credentials secure. It can be challenging to make your credentials available to your application securely. The AWS SDK for Ruby provides a number of helpful interfaces for configuring your credentials that help you keep your secrets safe.

This blog post focuses on securely configuring the aws-sdk gem with your credentials. I will follow up with additional blog posts about rotating credentials, and using roles for instances.

Credential Configuration

The aws-sdk gem requires that you provide your access credentials before making a request. The gem tries to locate your credentials in a number of default locations. If it is unable to find your credentials, it raises an error. The locations it searches are:

  • AWS.config
  • ENV
  • EC2 instance metadata
  • Rails Configuration (RAILS_ROOT/config/aws.yml)

AWS.config

You can use AWS.config to statically configure your credentials for all AWS requests. Simply call AWS.config with your :access_key_id and :secret_access_key (you may also provide a :session_token).

require 'aws-sdk'

AWS.config(:access_key_id => '...', :secret_access_key => '...')

Alternatively, you may provide credentials directly to the service interfaces. This can be helpful if you are working with multiple sets of credentials for multiple AWS accounts (or IAM users).

s3 = AWS::S3.new(:access_key_id => '...', :secret_access_key => '...')    

As tempting as it might be, you should never put access credentials in source code. This makes your secrets available to anyone with access to your source code. It also creates a maintenance burden when you need to rotate your credentials. There are many alternatives, including loading credentials from a configuration file that is not tracked with source control.

require 'aws-sdk'
require 'yaml'

# configure aws-sdk gem from a yaml configuration file
AWS.config(YAML.load_file(path_to_configuration_file))

In the preceding example, the YAML file might look like this:

:access_key_id: 'YOUR_ACCESS_KEY_ID'
:secret_access_key: 'YOUR_SECRET_ACCESS_KEY'

ENV

You can alternatively provide credentials to your application via ENV. By default, the aws-sdk gem searches ENV using two different prefixes for your keys (‘AWS’ and ‘AMAZON’).

ENV['AWS_ACCESS_KEY_ID']
ENV['AWS_SECRET_ACCESS_KEY']
ENV['AWS_SESSION_TOKEN']

EC2 Instance Metadata

The aws-sdk gem supports loading credentials from the instance metadata service on Amazon EC2 instances. If you have started your instance using an AWS IAM instance profile, this will just work, no configuration required. I will dive into this deeper in a followup blog post.

Credential Loading in Rails (RAILS_ROOT/config/aws.yml)

If you are using the aws-sdk gem in a Rails application, the gem attempts to load credentials from RAILS_ROOT/config/aws.yml. This file is ERB-parsed and then loaded by YAML. This file should be formatted in the same way as RAILS_ROOT/config/database.yml. It should have one top-level entry for the Rails.env you are running.

# RAILS_ROOT/config/aws.yml
development:
  access_key_id: DEV_ACCESS_KEY_ID
  secret_access_key: DEV_ACCESS_KEY_ID

If this file contains your secrets in plain text and does not use ERB to load them securely, you should not track this file with source control.

You can now read part two in this series: Rotating Credentials.

Connecting to Amazon RDS Databases from Eclipse

When developing any application that uses a database, it’s incredibly important to have good tools at hand for working with the data in your database. Eclipse has great tools for working with relational data, and if you’re using the Amazon Relational Database Service (Amazon RDS) to manage your database, then configuring a connection to your database with the AWS Toolkit for Eclipse only takes a few clicks.

Before going any further, make sure you have the AWS Toolkit for Eclipse and the Eclipse Data Tools Platform (DTP) installed. The easiest way to ensure you have Eclipse DTP installed is to start with the Eclipse Java EE distribution. Once you’re running the Eclipse JEE distribution, you can install the AWS Toolkit for Eclipse through the Eclipse Marketplace, or by pointing the Eclipse Update Manager directly to the AWS Eclipse update site at http://aws.amazon.com/eclipse/ .

If you don’t have an Amazon RDS Database Instance running yet, you can launch one with just a few clicks in the AWS Management Console.

Once you’ve got a DB Instance running, open the AWS Explorer view in the AWS Toolkit for Eclipse. Make sure you’re using the correct account and AWS region, and you should see your DB Instance listed in the AWS Explorer view.

Once you find your DB Instance in the AWS Explorer view, right-click and select Connect....

From here, you need to specify the password to use when connecting to your database. You might also have to specify where the JDBC driver library is for your database type. The MySQL JDBC driver is distributed with the plug-in, but other databases have less permissive licenses and require you to download the JDBC driver from the vendor’s website and tell the plug-in where the jar is.

Once you’ve created the connection inside Eclipse to your DB Instance, you can use all the tools in Eclipse’s Data Tools Platform to work with your data. You can use the SQL Query Editor to run queries on your database, review your schema, and make changes using the DataSource Explorer, or just browse and edit your data in a spreadsheet editor.

AWS re:Invent Wrap Up

We’re all back in the office after a great week at re:Invent! We enjoyed meeting so many customers and hearing feedback from you on how you’re using the AWS services, SDKs, and tools.

If you weren’t able to attend AWS re:Invent this year, you can still check out lots of great video content online:

The technical sessions are being uploaded, too, including the sessions Zach and I presented:

There’s lots more, all available on the AmazonWebServices channel on YouTube.

What did you enjoy most about AWS re:Invent? You can find a survey for the whole conference, as well as individual surveys for each technical session you attended on the AWS re:Invent Sessions page. Surveys will be open until the end of the week (December 7th), so hurry and get your feedback in!

The AWS SDK for Java at re:Invent

The first ever AWS conference, re:Invent, has sold out! If you are one of the lucky people with a conference pass, we hope that you will come to hear Jason Fulghum and me speak about advanced features of the AWS SDK for Java and the AWS Toolkit for Eclipse. On Wednesday morning after the keynote, I’m giving a presentation and live demo of the AWS Toolkit for Eclipse’s features. It’s called Develop, Deploy, and Debug with Eclipse and the AWS SDK for Java:

The AWS SDK for Java and the AWS Toolkit for Eclipse enable developers to easily manage AWS resources, quickly build web scale Java applications that interact with AWS services, and deploy those applications to the AWS platform. In this session, learn what functionality the AWS SDK for Java and the AWS Toolkit for Eclipse provide, see common usage scenarios with the AWS SDK for Java, and discover how to use the management, deployment, and debugging capabilities in the AWS Toolkit for Eclipse.

Then on Thursday afternoon, Jason Fulghum will be sharing our best tips and tricks for getting the most out of the AWS SDK for Java in his presentation, Being Productive with the AWS SDK for Java:

The AWS SDK for Java includes several higher level APIs that make working with AWS simpler. Learn more about higher level APIs such as TransferManager, which allows you to easily manage asynchronous uploads and downloads from Amazon Simple Storage Service (Amazon S3); the AWS DynamoDB Object Persistence Layer, which allows you to annotate your Java classes to specify how the SDK should map them to AWS DynamoDB tables when they are saved and loaded; and other higher level APIs such as the Amazon Simple Email Service (Amazon SES) JavaMail provider.

We hope to see you at re:Invent! But if you can’t make it this year, don’t worry — we’ll be sharing the tips in our talks, and more, right here on the AWS Java Blog in the months to come.

Ruby at re:Invent 2012

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

Loren and I will be attending AWS re:Invent in Las Vegas next week. I will be presenting a session, and we will both be answering questions in the Developer Lounge, and assisting with the AWS re:Invent Code Challenges.

Stop by to say "Hi", get your questions answered, or participate in a Code Challenge. We love to meet developers, get feedback, and to talk Ruby.

Getting Productive with the AWS SDK for Ruby, Wednesday, November 28 from 4:30 p.m. – 5:20 p.m.

Join us in the Venetian, Room 3001A. You will learn about how to get the most of the aws-sdk gem, and we will be building a sample application live that uses three different AWS services.

See you in Las Vegas!

Deploying Ruby Applications to AWS Elastic Beanstalk with Git

by Loren Segal | on | in Ruby | Permalink | Comments |  Share

With the release of Ruby container support on AWS Elastic Beanstalk, it is now possible to deploy Rails (or any Rack-based) applications onto the AWS infrastructure in just a few easy steps. The container is backed by Amazon EC2 and automatically provisions the machines to use Phusion Passenger (with nginx under the hood). You can run your own local database with your application, or better yet, you can have Elastic Beanstalk automatically set you up with a MySQL database backed by Amazon Relational Database Service (RDS). All of the machines sit behind a load balancer with auto-scaling support automatically configured so you don’t even have to think about ops. Scaling is built-in.

But enough with the awesome features. Let’s deploy an app!

Setting up the application

Before we can deploy an application to Elastic Beanstalk, we first have to create one. Let’s generate an empty Rails skeleton and set it up against a Git repository. We will be using the repository later to deploy our app.

$ rails new fooapp
$ cd fooapp
$ git init && git add -A && git commit -m "Initial commit"

Initializing our Elastic Beanstalk environment

At this point we have an application, but we have not yet configured it to deploy to Elastic Beanstalk. In order to do that, we need to install the AWS Elastic Beanstalk command line tool. This package provides us a binary, eb, which we will use to manage and deploy our applications.

Note that you do need to have Python (2.7 or 3.0) installed in order to use the CLI tools. If you are on OS X or a modern Linux OS, you should already have this installed. If not, visit Python’s website for download information. In order to verify if you have Python on your machine, you can type python --version

We can now initialize our application with the CLI tool by typing eb init. The tool will ask us for some information about our deployment such as our AWS credentials, the deployment region, and the solution stack we want to deploy. For our case, we want to use the "64bit Amazon Linux running Ruby 1.9.3" stack. The tool also asks if we want to setup an RDS database instance for our environment. Let’s answer "no" on that one for now.

$ eb init
To get your AWS Access Key ID and Secret Access Key, 
  visit "https://aws-portal.amazon.com/gp/aws/securityCredentials".
Enter your AWS Access Key ID: ****
Enter your AWS Secret Access Key: ****
Select an AWS Elastic Beanstalk service region.
Available service regions are:
2) US West (Oregon)
...
Select:  (1 to 6): 2
Enter an AWS Elastic Beanstalk application name (auto-generated value is "fooapp"): 
Enter an AWS Elastic Beanstalk environment name (auto-generated value is "fooapp-env"): 
Select a solution stack.
Available solution stacks are:
...
10) 64bit Amazon Linux running Ruby 1.9.3
Select:  (1 to 10): 10
Create an RDS DB Instance? [y/n]: n         

Note that this command will create a .elasticbeanstalk directory in our project and add it to our .gitignore. This directory contains the private environment configuration data provided above, as well as other customized settings. We then commit this change to our .gitignore with the following command:

$ git add .gitignore && git commit -m "Ignore .elasticbeanstalk from Git"

You can see more examples of setting up a project with the eb tool on the Amazon Web Services Blog.

Starting the environment

Now that our environment is configured, we simply need to start it. Starting the environment can take a couple of minutes. During this time, Elastic Beanstalk is spawning a set of EC2 instances, setting up a load balancer, and establishing health monitoring. Fortunately we only have to do this once per environment. Once Elastic Beanstalk is done, we can start the environment with the following command:

$ eb start
Starting application "fooapp".
Waiting for environment "fooapp-env" to launch.
... 
Application is available at "fooapp-env-djr9ywimiu.elasticbeanstalk.com".

After a little bit of waiting, our environment should be live. We can use the environment in a web browser with the URL provided above:

AWS Elastic Beanstalk Ruby Container Welcome Page

You can now also type eb status to see the status of your environment:

$ eb status
URL : fooapp-env-djr9ywimiu.elasticbeanstalk.com
Status  : Ready
Health  : Green

Deploying with Git

The eb init command we ran above actually did some nifty things behind the scenes. The most notable thing is that it setup a Git subcommand named aws.push that can push our code to the environment. This means that whenever we need to deploy our app, we just have to type git aws.push. Since our environment is running, and our app is ready, let’s do that right now!

$ git aws.push

If everything succeeds, you should have initiated a deploy to your environment. Typing eb status should now tell us that a deploy is in progress:

$ eb status
URL : fooapp-env-djr9ywimiu.elasticbeanstalk.com
Status  : Updating
Health  : Grey

Eventually, the status will be green, and we will be able to access our new Rails application:

Rails Welcome Page

Notice that gems listed in our Gemfile were automatically installed, assets were precompiled, and, if we had any, database migrations would have been run too. Migrations and asset compilation happen by default for any Rails 3.x application.

All we have to do now is update our application, commit to our Git repository, and execute the command git aws.push to push our new changes to deploy new versions of the app. Migrations will be run, and our assets will get updated.

Customizing your environment with .ebextensions

To run commands and drop environment variables onto your system via your app during a deployment, you can add a special .config file to the .ebextensions directory in your project root. The config file is formatted as YAML, and is documented on the Customizing and Configuring AWS Elastic Beanstalk Environments documentation page.

For reference, here is a sample .ebextensions/redmine.config that runs some extra hooks for installing Redmine on Elastic Beanstalk. Note that the environment variables are not necessary; BUNDLE_WITHOUT and RACK_ENV already default to the values below. We’re just setting them as an example of some config we might want to perform.

# Sample configuration for Redmine Rails installation

option_settings:
  - option_name: BUNDLE_WITHOUT
    value: "test:development"
  - option_name: RACK_ENV
    value: production

packages:
  yum:
    git: []

container_commands:
  01_secret:
    command: rake generate_secret_token

Skipping migrations and asset compilation

I mentioned you can set environment variables using the option_settings key in the .ebextensions/*.config files. Two specific variables you might want to set are:

  • RAILS_SKIP_MIGRATIONS — set this to true if you don’t want Elastic Beanstalk to run migrations on your behalf. You may want to manage migrations yourself.
  • RAILS_SKIP_ASSET_COMPILATION — set this to true if you don’t use asset compilation in your Rails app. Note, however, that your Elastic Beanstalk container is already configured with a JS runtime to properly handle asset compilation, you do not have to do anything fancy like install the therubyracer gem. It should just work.

Setting up Amazon RDS

The final-but-very-optional step for using the Ruby solution stack in Elastic Beanstalk is to hook up an RDS Instance to your application. If you are running any kind of real world application, you most likely have some database. If your application uses a relational database, you can take advantage of Amazon RDS for access to a MySQL (or Oracle) database.

Fortunately, the setup for this is extremely simple. Remember when we said "no" to creating an RDS DB Instance when we ran eb init? Well, we can initialize another application in the exact same way, but instead of no, we can say yes. We will be prompted to provide a password for this database, but everything else will be directly configured by Elastic Beanstalk on our behalf.

How do we configure it with our Rails app, you ask? Easy. All of the DB information will be passed to our application as environment variables. In Rails, this means we can setup our application to use RDS with a simple database.yml snippet:

production:
  adapter: mysql2
  encoding: utf8
  database: <%= ENV['RDS_DB_NAME'] %>
  username: <%= ENV['RDS_USERNAME'] %>
  password: <%= ENV['RDS_PASSWORD'] %>
  host: <%= ENV['RDS_HOSTNAME'] %>
  port: <%= ENV['RDS_PORT'] %>

The only other thing we have to do is add mysql2 to our Gemfile. It would look something like:

gem 'mysql2'

And RDS should now be fully configured for our Rails application.

Testing locally with Passenger

Because we use Passenger in your container, it’s really easy to test your deployment locally in a very similar webserver environment. If you don’t already have passenger installed on your local machine, you can set it up and initialize your server with:

$ gem install passenger
$ passenger start

A few quick notes if you’ve never deployed to a Passenger server before:

  • Passenger relies on a config.ru Rack file in your root application directory by convention.
  • Passenger also relies on a public/ directory for your static assets located in your application root.

Passenger needs both of these paths to exist in order to detect your Rack/Rails application. In a Rails 3.x app, these paths already exist, but if you are rolling your own low-level Rack application, you need to make sure that they are created. Once they have been setup, your application is ready to be tested as if it were deployed to Elastic Beanstalk. You can consult the Phusion Passenger documentation for more information about how to run your application on this snappy webserver.

Conclusion

So that’s it. We just went through the nuts and bolts of deploying a Ruby application to AWS Elastic Beanstalk. Although there is a lot here to read through, the steps themselves are pretty simple. In short, you can deploy an application with a few commands:

  1. Run eb init on your Git-backed application
  2. Run eb start to load up your environment
  3. Run git aws.push to deploy your code to Elastic Beanstalk
  4. Profit!

Setting up RDS is just a matter of using the configuration values provided to your application through standard environment variables. And of course if you need to dig deeper and customize settings, you can do so by adding a .ebextensions/config.yml file.

There is of course plenty you can do with an Elastic Beanstalk application, and there are plenty more things to talk about with the Ruby solution stack, but hopefully this helps you get a Rack-based application up and running in just a few minutes!

Storing Java objects in Amazon DynamoDB tables

by zachmu | on | in Java | Permalink | Comments |  Share

The AWS SDK for Java makes it easy to store objects in Amazon DynamoDB and get them back out again, all without having to write the code to transform your objects into table items and vice versa. All you need to do is annotate your domain classes in a few places and the SDK will handle the work of getting objects into and out of the database. For example, here’s a minimal User class we want to store in DynamoDB:

@DynamoDBTable(tableName = "users")
public class User {
 
    private Integer id;
    private Set<String> friends;
    private String status;
 
    @DynamoDBHashKey
    public Integer getId() { return id; }
    public void setId(Integer id) { this.id = id; }
 
    @DynamoDBAttribute
    public Set<String> getFriends() { return friends; }
    public void setFriends(Set<String> friends) { this.friends = friends; }
 
    @DynamoDBAttribute
    public String getStatus() { return status; }
    public void setStatus(String status) { this.status = status; }
}

The DynamoDBMapper utility class makes it simple to store instances of this class in DynamoDB.

AmazonDynamoDB dynamo = new AmazonDynamoDBClient(awsCredentials);
DynamoDBMapper mapper = new DynamoDBMapper(dynamo);
 
// save a new item
mapper.save(newUser);
 
// update the item
newUser.setStatus("active");
newUser.getFriends().remove("Jeremy");
mapper.save(newUser);
 
// delete the item
mapper.delete(newUser);

DynamoDBMapper can also handle query and scan operations with automatic result pagination, optimistic locking, automatic hash key generation, custom marshaling logic, and much more. For more detail, see this article.

New Blog, New Org, and New Issue Reporting!

by Loren Segal | on | in Ruby | Permalink | Comments |  Share

Hey everyone!

My name is Loren Segal, and I will be working with Trevor Rowe on our shiny new blog here at Amazon Web Services. We both work on the AWS SDK for Ruby, and over the coming weeks, months, and years, we will be using this spot to post new updates, tips, and tricks about the Ruby SDK. We hope these posts will be informative and shed some light on what’s coming down the pipe, but we also want to get the word out about all the cool things we already support. We’re really excited about sharing all this new information with you.

And with that, we already have new announcements to make: we are changing our GitHub organization name and enabling GitHub Issues in our new organization for easier issue and feature tracking in the Ruby SDK!

A better GitHub organization name

We decided to move our GitHub amazonwebservices organization to the much shorter and easier-to-remember aws organization. With that change, we’ve also renamed the project from "aws-sdk-for-ruby" to a simpler "aws-sdk-ruby". We will be following this naming convention across our other projects to make things easier for our collaborators.

As of this blog post, we will no longer be developing against our old repository, but we will leave this old repository around so that users can still access the code. Note, however, that in about a week, we replace the contents of the repository with a README guiding our new users to the proper repository location.

Remember to update your Git remotes and Gemfiles!

If you’re currently pointing your remotes to our old organization, now is the time to update the location to the new repository. You can do so by typing the following command in your local working copy (assuming the remote is origin):

$ git remote set-url origin git://github.com/aws/aws-sdk-ruby

If you are pointing your Gemfiles to our Git repository, you will also want to update those by pointing to the new repo:

gem 'aws-sdk', :git => "git://github.com/aws/aws-sdk-ruby.git"

Note that for Gemfiles, we would much rather you point to the gem released on RubyGems.org, though!

GitHub Issues are enabled!

Finally, with our newly renamed organization, we’ve decided to enable GitHub Issues on the new aws-sdk-ruby project. You can now post bug reports, submit pull requests, and track development progress right on our issues page.

Since we’re on the topic, here are a couple of notes about submitting new issues:

Submitting pull requests: include tests

We love when users contribute code to the SDK! Really! If you’ve found a bug in the SDK and you’ve tracked down the issue and have a local working patch, submitting a pull request will help us get the fix out to our users much faster than just submitting a bug report and greedily keeping that awesome bugfix to yourself!

However, if you do plan on submitting pull requests, please try to include tests for your new code. The better the tests, the more easily we can guarantee that your patch will continue to work, and the quicker we can pull it in. Now that we are using Travis-CI for continuous integration, GitHub will happily tell us that your changes will not break the build.

Submitting bug reports: include reproduction steps

If you are having issues with the SDK not working properly, we want to hear about it and fix it. GitHub Issues now allows you to post your problems directly on our GitHub repository. If you do report an issue, please make sure to include reliable reproduction steps, as well as much extra relevant information as possible. Please make sure you do not post your AWS credentials in bug reports, however! We will never ask you for this information— that is your data.

Shiny new things!

We’re really happy about these new changes. The new blog will be here to provide new insight on what we’re up to on the Ruby team, and hopefully the new GitHub re-organization will make it much easier to contribute code and submit bugs on the AWS SDK for Ruby! Go check out our new org right now:

http://github.com/aws/aws-sdk-ruby

Welcome to the AWS SDKs and Tools Java blog

Welcome to the AWS SDKs and Tools Java blog. We’re glad to see you here!

This blog features information for AWS Java developers, including:

  • Demonstrations of new features in the AWS SDK for Java
  • Announcements of new features in the AWS Toolkit for Eclipse
  • Best practices for using the SDK
  • Important AWS product announcements for Java developers
  • Lots of code samples

You’ll see content from many of our team members, including developers from the SDKs and Tools team, as well as Java developers from service teams.

We hope you come back often to visit or subscribe to our blog using the RSS feed button at the top of the page. If you’d like us to cover any specific topics, please let us know and we’ll do our best.