PAWS Marketplace - A Rails Marketplace Application in the Cloud

Articles & Tutorials>Ruby>PAWS Marketplace A Rails Marketplace Application in the Cloud
The PAWS Marketplace is a proof-of-concept Ruby on Rails web application that runs entirely on Amazon Web Services. The application’s web site allows anyone with an Amazon Payments account to buy or sell digital products, while the marketplace’s owner earns commissions for the sales it performs. In addition to serving as an example marketplace implementation, this application is interesting because it uses the SimpleDB service to store its data, instead of a traditional relational database.

Details

Submitted By: James Murty
AWS Products Used: Amazon EC2, Amazon FPS, Amazon S3, Amazon SimpleDB
Language(s): Ruby
Created On: May 29, 2008 7:35 AM GMT
Last Updated: September 21, 2008 9:58 PM GMT

by James Murty, creator of the JetS3t Java S3 library and author of Programming Amazon Web Services: S3, EC2, SQS, FPS, and SimpleDB (O’Reilly, 2008)

Introduction

Amazon Web Services (AWS) provides a set of infrastructure services that can replace or augment the physical infrastructure you normally need to run online applications. You can use these services to build applications that scale massively and cheaply, while minimizing the burden of buying and maintaining your own IT infrastructure. Each AWS service offers different capabilities that you can mix-and-match according to your application’s requirements.

This article introduces the PAWS Marketplace, a proof-of-concept Ruby on Rails web application that runs entirely on Amazon Web Services. The application’s web site allows anyone with an Amazon Payments account to buy or sell digital products, while the marketplace’s owner earns commissions for the sales it performs. In addition to serving as an example marketplace implementation, this application is interesting because it uses the SimpleDB service to store its data, instead of a traditional relational database.

The PAWS Marketplace application uses four AWS infrastructure services:

  • Flexible Payments Service (FPS) to transfer funds between the marketplace participants. The application runs in the FPS Sandbox environment so no real money is involved.

  • SimpleDB to store, query, and retrieve application data. The PAWS Marketplace code includes tools that make SimpleDB items accessible through an ActiveResource-like interface.

  • Simple Storage Service (S3) to store and distribute the files sold by vendors in the marketplace. Because the application keeps files in S3, it has virtually unlimited storage space and bandwidth.

  • Elastic Compute Cloud (EC2) to provide a virtual server instance that will run the application on the Ruby on Rails framework.

In this article I will show you how to obtain the PAWS Marketplace application and run it yourself. I will also briefly highlight the application’s more interesting features, and indicate where they are implemented in the code so you can see how they work.

This article is too brief to describe the application in detail, so I have created a PAWS Marketplace Screencast video that demonstrates how to use the marketplace web site.

Disclaimer

The PAWS Marketplace is a proof-of-concept application intended for demonstration purposes only. The code is not suitable for use in a real live application. Feel free to examine, adapt, and improve the example code, but do not rely on it.

Requirements

To run the PAWS Marketplace application, you will need to have an AWS developer account and to be signed up for the FPS, S3, EC2, and SimpleDB services. You can sign up for an AWS developer account and the required services at the Amazon Web Services home page. There may be a waiting period when you sign up for the SimpleDB service because it is still under limited beta release.

The PAWS Marketplace code is available as a zipped archive from my web site: http://s3.jamesmurty.com/PAWS_Marketplace_Demo.zip. The archive includes the web site application and a Rails vendor plugin module. The plugin provides tools to interact with AWS services, and to use SimpleDB as the application’s database. The code is designed to work with Ruby version 1.8.6 and Rails version 2.0.2.

Start an EC2 Server Instance

To run the application, you will need an EC2 instance that has the Ruby on Rails framework properly installed and configured. Rather than installing all the required software from scratch, we will launch an instance based on a public Amazon Machine Image (AMI) that is pre-prepared with almost everything we need: Fedora Core 8 i386 with Ruby on Rails 2, Mongrel, Nginx, MySQL (ami-a93adfc0).

Launch a new instance from this AMI using your favorite EC2 client tool. Make sure you launch the instance within a security group that allows secure shell connections on TCP port 22, and remember to provide the name of your own EC2 key pair so the instance can authenticate you.

ec2-run-instances ami-a93adfc0 --group default --key my-key -n 1

Once the instance has started, use ssh or a similar secure shell client program to log in. In this article, I will use the port forwarding feature of ssh to securely relay traffic to TCP port 3000 on the EC2 instance. Port forwarding allows you to access the PAWS Marketplace web site without the risk of making it publicly accessible over the Internet. Alternately, you could configure an EC2 security group to allow direct connections to port 3000 on your instance, but be very careful not to make the application publicly accessible because this would pose a massive security risk.

Here is the ssh command to connect to an EC2 instance with the public DNS address ec2-75-101-208-112.compute-1.amazonaws.com, and to securely forward traffic to the instance’s TCP port 3000:

ssh -i my-key.pem -L3000:127.0.0.1:3000 root@ec2-75-101-208-112.amazonaws.com

Once you have logged in to your instance, confirm that it contains the required versions of Ruby and Rails:

[~]: ruby -v
ruby 1.8.6 (2007-09-24 patchlevel 111) [i386-linux]

[~]: rails -v
Rails 2.0.2

Install and Configure the Marketplace Application

The PAWS Marketplace application requires a non-standard Ruby library to generate Universally Unique IDentifiers (UUIDs). You will need to manually install the uuid gem library:

[~]: gem install uuid

To install the PAWS Marketplace application, download and unzip the archive file:

[~]: wget http://s3.jamesmurty.com/PAWS_Marketplace_Demo.zip
[~]: unzip PAWS_Marketplace_Demo.zip

To configure the application, edit the settings in the development section of the configuration file PAWS_Marketplace_Demo/config/paws_config.yml. You can configure the following settings:

  • aws_access_key - Your AWS Access Key
  • aws_secret_key - Your AWS Secret Key
  • s3_bucket - The name of the S3 bucket that will store the files sold through the marketplace, for example: marketplace-bucket. You do not need to create this bucket in advance, but you must choose a bucket name that is unique in S3.
  • domain - The name of the SimpleDB domain in which the application’s data will be stored, for example: marketplace-domain. You do not need to create this domain in advance.
  • memcached_servers (Optional) - A list of one or more hosts running the memcached caching server, for example: 127.0.0.1:11211. Using memcached servers can improve the performance of SimpleDB requests, but you will need to install the memcached daemon and the Ruby MemCache client library — tasks that are beyond the scope of this article.
    Ignore this setting unless you have experience using memcached.

Once you have edited the configuration file, you can perform commands in the Rails console to test your settings and experiment with the SimpleDB tools included with the PAWS Marketplace application. Run the Rails console by changing to the PAWS_Marketplace_Demo directory and running the script/console command:

[~]: cd PAWS_Marketplace_Demo
[~]: script/console 
Loading development environment (Rails 2.0.2)
>>

The PawsHelper Module

To begin with, let’s examine the PawsHelper module that is included with the application in the paws vendor plugin (vendor/plugins/paws/lib/PawsHelper.rb). This module contains a full set of AWS service clients, and provides an easy way to access the APIs of Amazon’s EC2, S3, SQS, FPS and SimpleDB (SDB) services.

To list the buckets in your S3 account, you can use the module’s S3 client:

>> PawsHelper::S3.list_buckets

Similarly, you can use the EC2 client to list your virtual machine instances:

>> PawsHelper::EC2.describe_instances

I’m sure you get the idea. The code for these service client implementations is discussed in detail in the book Programming Amazon Web Services.

SimpleDbResource: An ActiveResource Implementation for SimpleDB

Ruby on Rails applications use model objects to represent information stored in a database. These model objects include features that make it easy to create, update, store, and find your data, without you being forced to interact directly with the database.

Because Rails applications generally use relational databases to store information, the Rails framework includes fantastic support for these databases in the form of the ActiveRecord model class. However, Amazon’s SimpleDB behaves very differently from a relational database and the standard ActiveRecord class will not work with the service. To my knowledge there are not yet any ActiveRecord implementations designed specifically for SimpleDB.

To allow the PAWS Marketplace to store information in SimpleDB, the application includes an ActiveResource implementation called SimpleDbResource (vendor/paws/lib/SimpleDbResource.rb) with enough basic data management features to do everything the application needs. The SimpleDbResource model class is nowhere near as powerful as the Rails ActiveRecord class and it lacks the more sophisticated features, such as support for relationship macros and inheritance, but it does understand most validation macros and serves as a workable substitute.

The SimpleDbResource class is not the only ActiveResource implementation available for SimpleDB. I am aware of two projects that provide tools to help integrate SimpleDB into Rails applications: AWS SDB Proxy plugin and RightScale AWS gems. SimpleDbResource was inspired and informed by these other projects, so I am indebted to their creators.

Introducing the Example Model

The best way to find out how to use model objects based on SimpleDbResource is to experiment with them. For this purpose the PAWS Marketplace application includes an Example model class (app/model/example.rb) that is never used by the web site. You can play with this model to your heart’s content without affecting the real application.

Before you experiment with the Example model, return to the Rails console and make sure that the SimpleDB domain name you entered in the paws_config.yml file actually exists:

>> PawsHelper::SDB.list_domains.include?(SimpleDbResource::DOMAIN) # => false

>> PawsHelper::SDB.create_domain(SimpleDbResource::DOMAIN)

Now you can create a new Example model to store information in SimpleDB.

>> tim = Example.create(:name => 'Timothy')
=> #<Example:0x256c424
    @current=true, 
    @attributes={"_resource"=>"example", 
                 "id"=>"411b4e90-e429-012a-83c4-0016cb9d98ec", 
                 "_created_timestamp"=>Fri Apr 24 03:58:05 UTC 2008,
                 "_updated_timestamp"=>Fri Apr 24 03:58:05 UTC 2008,
                 "name"=>"Timothy"},
    @errors=#<ActiveRecord::Errors:0x256c3ac @errors={} ...>>>

In the output above, you can see three key variables that are present in all SimpleDbResource model objects:

  • current - A boolean flag that indicates whether the object’s data is up-to-date in SimpleDB.
  • attributes - A hash of attribute names and values that are saved to a unique item in SimpleDB.
  • errors - An ActiveRecord::Errors object that keeps track of data validation errors, so the web site can provide feedback when a user enters invalid information.

The attributes variable contains the information that is actually stored in the SimpleDB service. In addition to any custom data included in a model object, all SimpleDbResource objects include four special attributes which are used for record-keeping:

  • id - A unique identifier for the item. The PAWS Marketplace application uses UUIDs as identifiers because SimpleDB cannot provide identifier values that are guaranteed to be unique.
  • _resource - The name of the model class represented by the SimpleDB item. By including the class name in the SimpleDB item’s attributes, it is possible to store multiple model types in a single domain.
  • _created_timestamp - The UTC time when the data item was first saved in SimpleDB.
  • _updated_timestamp - The UTC time when the data item was last updated in SimpleDB.

Saving and Updating Model Data

The SimpleDB service does not enforce any predefined data schema. This means that you can add to, remove, or update a model’s attributes whenever you like, provided your attribute names and values are no longer than the service’s 1,024 byte limit.

You can set the model’s attributes using hash syntax or dynamic method names:

>> tim[:FavoriteFoods] = 'pizza'
>> tim.age = 14
>> tim['name'] = 'Tim'

To read data from the model, you can use hash lookups or dynamic method names. You can also obtain a cloned copy of all the attributes as a hash:

>> tim.FavoriteFoods  # => "pizza"
>> tim['age']  # => 14
>> tim[:name]  # => "Tim"

>> tim.attributes
=> {"_resource"=>"example", "id"=>"411b4e90-e429-012a-83c4-0016cb9d98ec",
    "name"=>"Tim", "age"=>14,  "FavoriteFoods"=>"pizza",
    . . .

Because we have changed the attribute values in our tim model object since we first created it, the object will no longer be in synch with the SimpleDB service and its current variable will be set to false. The save method updates an item in SimpleDB when its attributes have changed.

>> tim.current  # => false
>> tim.save  # => true

# The save method does nothing if the model is current.
>> tim.current  # => true
>> tim.save  # => false

One of the advantages of SimpleDB, compared to a standard relational database, is the service’s ability to store multiple values for a single attribute. To store multiple attribute values, you specify the values as an array:

>> tim.FavoriteFoods = ['pizza', 'ice cream']
>> tim.save

To delete an attribute from a model, you set the attribute’s value to nil:

>> tim.test = 'SomeValue'
>> tim.attributes.has_key? 'test'  # => true

>> tim.test = nil
>> tim.attributes.has_key? 'test'  # => false

Performing Queries

The SimpleDbResource class includes features that make it easy to search for information in SimpleDB. Create a second Example item so you have more than one item to query.

>> Example.create(:name => 'James', :age => 30, 
                  :FavoriteFoods => ['pizza','pancakes'])

You can search for and retrieve items from SimpleDB using the find class method. To find all the Example model items currently stored in SimpleDB, use the find method with the :all parameter and no constraints:

>> Example.find(:all).size  # => 2

To find all the items with attributes matching a given value, include the attribute’s name and value as constraints:

>> Example.find(:all, :name => 'Tim').size  # => 1

When you perform searches for a given attribute value, the service also searches multi-value attributes for a match. The following queries use the multi-value attribute FavoriteFoods to show that both Tim and James like pizza, but only Tim likes ice cream:

>> Example.find(:all, :FavoriteFoods => 'pizza').size  # => 2

>> Example.find(:all, :FavoriteFoods => 'ice cream').size  # => 1

To search using comparison operators other that equals (=), you provide an array with two entries as the attribute value. The first entry in the array is a SimpleDB operator, while the second entry is the query’s constraint value:

>> Example.find(:all, :age => ['>=', 10]).size  # => 2

>> Example.find(:all, :age => ['<', 20]).size  # => 1

Notice that the queries above use integer comparisons to search within SimpleDB, despite the fact the service only stores textual data and is not able to perform integer comparisons itself. The SimpleDbResource class uses the SimpleDB.rb client to interact with the service, and takes advantage of this implementation’s data encoding and decoding features. This means that Float, Integer, Time, and boolean values are automatically encoded before they are stored in SimpleDB, in such a way that the natural and lexicographical ordering will be the same.

Although this feature is very convenient, it does not solve all data conversion problems because it only allows you to compare values of the same type. For example, you cannot compare integer and float values even though they are both numeric types:

>> Example.find(:all, :age => 14).size  # => 1
>> Example.find(:all, :age => 14.0).size  # => 0

>> Example.find(:all, :age => ['<', 20.0]).size  # => 0

If you are familiar with SimpleDB and its query language, you can provide your own query statements to the find method as a string parameter. Your query statement will be sent directly to the service, once a prefix constraint has been added to identify the model class. For example, the following command will perform the query: ['_resource' = 'example'] intersection ['name' starts-with 'J' and 'name' != 'James'].

>> Example.find(:all, "['name' starts-with 'J' and 'name' != 'James']")

To retrieve only a single item from SimpleDB, you can use the symbols :one or :first as the first argument to the find method. If your query constraints give multiple results, the method will only return the first one. Be careful though, as the “first” result will not be based on any particular ordering.

>> Example.find(:first, :age => ['>', 10])
=> #<Example:0x125f818
     @attributes={"name"=>"James", "age"=>30,
     . . .

Deleting Items

To delete an item from SimpleDB, you use the model’s delete method. You can invoke this method on a class, in which case it will delete an item with the given identifier:

>> Example.delete(tim.id)  # => true

>> Example.exists(tim.id)  # => false

Alternately, you can call the delete method on a particular model object:

>> example = Example.find(:first)

>> example.delete  # => true

Running the PAWS Marketplace

Once you have configured the PAWS Marketplace application and tested your settings in the Rails console, you are ready to run the actual marketplace web site. Quit the Rails console on your EC2 instance by pressing Control-D, then run the Rails server script script/server to start the web site.

[~]: script/server
=> Booting Mongrel (use 'script/server webrick' to force WEBrick)
=> Rails application starting on http://0.0.0.0:3000
=> Ctrl-C to shutdown server

If you used ssh port forwarding as discussed above, you should now be able to load the web site from the local URL http://localhost:3000/. If you did not use port forwarding, you will need to point your web browser to port 3000 on your instance’s public address, for example: http://ec2-75-101-208-112.compute-1.amazonaws.com:3000.

The first thing you will see when you visit the web site is a set of error messages, because the application needs to create a number of AWS resources to do its work. Simply click on the link below each error message and follow the instructions to create the required resources.

Watch the PAWS Marketplace Screencast video to see a walk-though of the whole process, from launching an EC2 instance and installing the application, to running the web site and using the marketplace to buy and sell files.

Behind the Scenes

In this section I will discuss how the PAWS Marketplace application works behind the scenes.

User Roles

The PAWS Marketplace application allows web site visitors to assume one of three different roles: marketeer, vendor or customer. Visitors log in to the marketplace as a specific user by providing username and password credentials, and may then perform tasks appropriate to that user role. Visitors can switch between roles by logging out of one user account and logging in as a different user.

The Marketeer user is the administrator of the marketplace. This user can:

  • Manage user accounts by adding, editing or deleting User model items in SimpleDB.
  • View all financial transactions that have occurred in the marketplace, including product sales, refunds, and vendor commission payments.
  • Perform refunds on behalf of vendors.

Vendor users sell products through the marketplace, and pay the marketplace owner a 10% commission fee for each sale. Any web site visitor can create a vendor account, and after installing the required FPS tokens the visitor will be able to add his own products to the marketplace. Vendors can:

  • Create, edit, and delete Product model items to describe a product and set its price in the marketplace.
  • Sell digital artifacts by uploading a file to S3 and linking the uploaded file to a product.
  • View their financial transactions, including the sales they have made, refunds they have performed, and the commission fees they have paid to the marketplace.
  • Perform refunds.

Customer users buy products through the marketplace. Any web site visitor can create a customer account which will allow him or her to:

  • Buy products that vendors have made available in the marketplace.
  • View their financial transactions, including their purchases and any refunds they have received.

FPS Tokens and Transactions

The PAWS Marketplace application performs financial transactions in the sandbox environment of the Flexible Payments Service (FPS). The application uses the Transaction model class to store detailed information about each transaction in SimpleDB. For money to be transferred from one marketplace user to another, each user must have his or her own Amazon Payments account, and must install FPS tokens to authorize the transaction.

As the owner of the marketplace application, you need to install Caller and Recipient tokens in your own Amazon Payments developer account. The Caller token permits the application to transfer funds between the marketplace participants, and the Recipient token permits the application to receive commission payments from vendors. These tokens are created by the methods install_caller_token and install_caller_recipient_token in app/controllers/home_controller.rb.

Vendor users must have an Amazon Payments Business account to receive payments through the marketplace, and must install Recipient/Refund and Sender tokens in their account. The combined Recipient and Refund tokens allow a vendor to receive payments from customers and to refund these payments. The Sender token is a multi-use authorization that allows the marketplace to extract sale commission fees from a vendor up to a total of $500.

Marketplace customer users need only a standard Amazon Payments account and are not required to install any FPS tokens in advance. When a customer clicks the “Buy” link for a product, he will be directed to approve a single-use Sender token to authorize the payment.

Third-party web site visitors, such as vendors and customers, will be directed to the FPS Co-Branded UI pipeline (CBUI) to install FPS tokens in their Amazon Payments accounts. These users are sent to the CBUI pipeline using specially crafted links generated by the methods uri_for_recipient_and_refund_tokens, uri_for_multi_use_sender_token, and uri_for_single_use_sender in app/helpers/application_helper.rb. When a user completes the pipeline, he or she is redirected back to the marketplace web site with a Result URL that describes the outcome of the pipeline process. The pipeline result for vendor users is interpreted by the interpret_installed_tokens method in app/controllers/products_controller.rb, while the result for customers is interpreted by the pay method in app/controllers/transaction_controller.rb.

When the PAWS Marketplace application performs a transaction to transfer funds from one user to another, the bulk of the work is handled by the perform_purchase or refund methods in app/controllers/transaction_controller.rb.

File Storage and Distribution with S3

Vendors make a product available in the marketplace by defining a name, description, and price for the product, and then uploading the file that will be sold. A vendor can optionally include a preview image for each product.

The PAWS Marketplace stores all product files as private objects in your S3 bucket, and it also distributes these files directly from the S3 service. The application uses two mechanisms to allow vendors and customers to interact with your S3 account: authorized HTML upload forms, and signed download URLs.

The application generates S3 POST HTML forms that allow vendors to upload files directly into your S3 account. Uploaded files can be up to 5MB in size, and each file is saved to a virtual “subdirectory” specific to that vendor. The POST forms are generated by the s3_upload_form_for_vendor method in app/helpers/products_helper.rb, and the result URLs are interpreted by the upload method in app/controllers/products_controller.rb.

When a customer purchases a product, the application generates a signed URL that will allow the customer to download the file as often as he or she likes for one week. This signed URL is provided as a “Download” link on the customer’s transactions page. The application also generates signed URLs for product preview images so they can be displayed on the web site. The signed URLs are generated by the uri_for_download method in app/helpers/application_helper.rb.

Application Data Model

The data model used by the PAWS Marketplace application is fairly simple, but at the same time it demonstrates some interesting differences from the data model you would employ if the application was based on a relational database instead of SimpleDB.

The main difference is that the PAWS Marketplace crams as much information as possible into each item stored in SimpleDB, rather than normalizing information into multiple tables as you would in an SQL-based database. This approach minimizes the number of HTTP requests necessary to retrieve information, and reduces the number of data relationships the application needs to manage. The schema-less nature of SimpleDB makes this approach feasible as it is easy to add or change item attributes as needed. However, for real applications I would recommend a more considered approach, because it can be very risky to make ad-hoc changes to your data model.

The marketplace application uses three model classes from the app/models directory: User, Product, and Transaction. It is worth looking at these class files because they demonstrate how to use the macros supported by the SimpleDbResource class, such as validation macros.

Conclusion

The PAWS Marketplace example shows that you can build fairly sophisticated applications based entirely on Amazon’s infrastructure services. I hope that it will serve as an interesting demonstration of just how far you can push your use of Amazon Web Services, as well as providing sample code to help you build your own applications.

Additional Resources

©2014, Amazon Web Services, Inc. or its affiliates. All rights reserved.