Building an Amazon S3 Browser with Grails

Don Denoncourt shows you how to add scalable and reliable storage to your Grails-based web application with Amazon S3 and the AWS SDK for Java.


Submitted By: Craig@AWS
AWS Products Used: Amazon S3
Language(s): Java
Created On: July 26, 2010


By Don Denoncourt, AWS Community Developer

In this article, you develop a Grails-based Web application with which you can browse, add, delete, upload, and download your Amazon Simple Storage Service (Amazon S3) objects. Along the way, you get an introduction to Grails and the Amazon Web Services (AWS) software development kit (SDK) for Java. In 20 or 30 minutes, you'll have the browser for Amazon S3 running on your system:

Generating the Browser for Amazon S3 with Grails

Grails is an amazing framework that provides rapid application development (RAD) for Web applications. One of the cool features of Grails is its ability to generate the infrastructure for an entire application in seconds. Then, after creating a simple domain class, Grails can generate the artifacts required to provide Web-based maintenance for that domain.

You can download the latest Grails release from Grails - Downloads, then extract the .zip file to a local directory (I typically use /opt/grails on UNIX and C:\opt\grails on Windows). Grails requires Java SDK 5.0 or later. I usually extract my Java downloads to /opt/java on UNIX and C:\opt\Java on Windows.

With Grails and Java installed, you set up set up GRAILS_HOME and JAVA_HOME environment variables so that they point to their prospective installation directory. Then, modify the PATH environment variable to reference the Grails and Java bin directories. (See Grails - Installation for more information on installation.)

To create the browser for Amazon S3, you'll cheat a bit by asking the Grails framework to generate the application infrastructure complete with controller classes and fully styled Web pages. To do that, perform the following steps:

  1. Use Grails to generate the maintenance infrastructure, including controller classes and HTML pages.
  2. Create two domain classes—Bucket and BucketObject—that roughly approximate basic information about Amazon S3 buckets and bucket objects.
  3. Use the Grails embedded test server to review the functionality of the generated application.
  4. Replace the Grails-generated object retrieval, update, and delete logic with application programming interface (API) calls to methods from the classes in the AWS SDK for Java.
  5. Test the browser for Amazon S3 with live Amazon S3 buckets.

Note: You can really cheat by downloading the completed version of the Grails application. You will have to modify the AwsCredentials.properties file, change your current directory (in a command window) to the extracted location, and then follow the instructions at Using the Amazon S3 Browser.

Generate the Infrastructure

Begin by asking Grails to generate the infrastructure for the application. Open a shell (Windows command window or UNIX shell), and navigate to a directory in which you want the application's root directory to be generated. Type the following Grails command:

grails create-app s3browser

Grails creates the complete infrastructure for a Spring-based Web application. At this point, you could test the application, but it does little more than throw up an index page.

Create the Domain Classes

Your next step is to add a couple of domain classes. You could create the files yourself, but let Grails do the work, instead. In your shell window, change your directory to the newly generated s3browser directory, and use the Grails create command to generate domain objects called Bucket and BucketObject.

Note: The Grails project is Eclipse ready in that you can import it directly into an Eclipse workspace. NetBeans and IntelliJ also have excellent support for Grails development.

cd s3browser
grails create-domain-class Bucket
grails create-domain-class BucketObject

Note: When the Grails generator prompts you with a warning about packages, type y for "yes." You want to keep things simple for now.

Grails generates the skeletons for those classes in the grails-app/domain directory. Grails also generates unit tests, but you need not concern yourself with those yet.

class Bucket {
    static constraints = {
    }
}
class BucketObject {
    static constraints = {
    }
}

Using your favorite text editor, modify the domain classes Bucket.groovy:

class Bucket {
	String id
	String owner
	static hasMany = [bucketObjects:BucketObject]
    static mapping = {
        id generator:'assigned', column:'name', type:'string'
    }
    static constraints = {
    }
}

and BucketObject.groovy:

class BucketObject {
	String id
	Date	lastModified
	String contentType
	int	contentLength
	static belongsTo = [bucket:Bucket]
    static mapping = {
        id generator:'assigned', column:'key', type:'string'
    }
    static constraints = {
    }
}

Now, generate the maintenance application by typing the following code in your shell window:

grails generate-all Bucket 
grails generate-all BucketObject 

Test the Application's Functionality

The application is ready to test, but you have to have something to display. So, add some dummy objects into the Grails start-up script. Edit the grails-app/conf/BootStrap.groovy file as follows:

class BootStrap {
  def init = { servletContext ->
    def bucketObject1 = new BucketObject(
              lastModified:new Date(), 
    contentType:'text/plain',contentLength:12)
    bucketObject1.id = 'key1'
    bucketObject1.save()
    def bucketObject2 = 
    new BucketObject(lastModified:new Date(), 
    contentType:'text/plain',contentLength:12)
    bucketObject2.id = 'key2'
    bucketObject2.save()
    
    def bucket = new Bucket(id:'one', owner:'me')
    bucket.id = 'one'
    bucket.save()
    bucket.addToBucketObjects(bucketObject1)
    bucket.addToBucketObjects(bucketObject2)
    bucket = new Bucket(id:'one', owner:'me')
    bucket.id = 'two'
    bucket.save()
  }
  def destroy = {}
} 

Now, you can test the application. Again, from the command line, run the Grails test server:

grails run-app

Then, test the generated application from your browser:

Click the BucketController link to access your bucket list:

In the ID column, click one to display a list of bucket objects:

Working with the Generated Code

Now that you've let Grails set up the infrastructure for your browser for Amazon S3, you can make a few changes to get the application to work with live Amazon S3 data. But, before getting into that, let's review the artifacts that Grails generated to manage Bucket objects:

The important artifacts are the two Groovy classes under /grails-app/controller, the two classes under /grails-app/domain, and the Groovy Server Pages (GSP) under /grails-app/views/bucket and /bucketObject.

REST Easy with Grails

Grails uses a RESTful Model–View–Controller (MVC) strategy. For example, take the URL https://localhost:8080/s3browser/bucket/list. The component parts of the URL are interpreted as follows:

  • /s3browser. The good old context root
  • /bucket. By Grails convention, directs the request to BucketController for handling
  • /list. By Grails convention, specifies the name of the action BucketController is to invoke

The BucketController consists of a set of actions that respond to HTML form actions. The actions are implemented as Groovy closures. If you are not familiar with closures, you can think of them as methods.

The following table describes the actions of BucketController.groovy:

Action Description
list Build a list of Bucket objects from the relational database with built-in pagination.
create Instance a Bucket object from the relational database to be displayed for update.
save Perform validation on Bucket object attributes, and persist the changes to the database.
show Retrieve a Bucket object from the relational database for display.
edit Retrieve a Bucket object from the relational database for update.
update Update a Bucket row in the relational database from HTTP request parameters.
delete Delete a Bucket row in the relational database.

The following table describes the GSP that Grails generates:

File Description
grails-app/controllers/BucketController.groovy Handles create, read, update, and delete operations as well as list generation on Bucket objects
grails-app/views/bucket/create.gsp Displays a new Bucket object in edit mode
grails-app/views/bucket/edit.gsp Displays an existing Bucket object in edit mode
grails-app/views/bucket/list.gsp Displays a paginated list of Bucket objects
grails-app/views/bucket/show.gsp Displays a Bucket object with options to edit or delete

You may have noticed that the GSP file names matched the names of the controller's actions. That's Convention-Over-Configuration (CoF) at work. A Grails controller action, unless explicitly specified with render(view:'gspName'), will build the page whose name matches the controller's action name.

Grails also generated, for maintenance of BucketObject objects, a BucketObjectController.groovy file and its associated create, edit, list, and show GSP elements.

The AWS SDK for Java

The first step in modifying the generated code to work with Amazon S3 objects is to download the AWS SDK for Java. After extracting the AWS SDK, copy each of the following Java archive (JAR) files to your s3browser/lib directory:

  • aws-java-sdk-1.0.003.jar
  • commons-codec-1.3.jar
  • commons-httpclient-3.0.1.jar
  • commons-logging-1.1.1.jar
  • jackson-core-asl-1.4.3.jar
  • saxon9he.jar

In your s3browser/src/java directory, create a file called AwsCredentials.properties, and add your AWS access key and secret key:

accessKey=
secretKey=

The BucketController

The Grails-generated actions of the BucketController use Grails Object Relational Mapping (GORM) calls to create, read, update, and delete rows from a relational database. You'll continue to use the same actions, but you'll replace the use of the GORM API with the AWS SDK for Java.

Replace the entire contents of grails-app/controllers/BucketController.groovy with the following code:

import com.amazonaws.auth.PropertiesCredentials
import com.amazonaws.services.s3.AmazonS3
import com.amazonaws.services.s3.AmazonS3Client

class BucketController {

  private static AmazonS3 s3;  
  
  def beforeInterceptor = {
    if (!s3) {
      s3 = new AmazonS3Client(new PropertiesCredentials(
         this.class.getResourceAsStream("AwsCredentials.properties")));
    }
  }
  def list = { [buckets:s3.listBuckets()] }
  def create = { } // show GSP
  def save = {
    if (!s3.doesBucketExist(params.bucketName)) {
      s3.createBucket(params.bucketName);
      flash.message = "bucket ${params.bucketName} created"
    } else {
      flash.message = "bucket ${params.bucketName} already exists"
    }
    redirect(action: "list")
  }
  def show = { 
    redirect(controller:'bucketController', action: "list") 
  }
  def edit = {render "edit implemented"  }
  def update = {render "update not implemented" }
  def delete = {
    try {
    	s3.deleteBucket(params.bucketName)
        flash.message = "Bucket ${params.bucketName} deleted"
    } catch (Exception e) {flash.message = e }
    redirect(action: "list")
  }
}

The private static AmazonS3 s3 object that is instanced in beforeInterceptor is the controller's hook to your Amazon S3 objects. Amazon knows which Amazon S3 buckets you own from the access and secret keys you placed in the AwsCredentials.properties file. The beforeInterceptor closure does what you might guess: Before any of the action closures of the BucketObject are invoked, the beforeInterceptor's code is executed.

The following table describes the Amazon S3 functionality that replaces the Grails-generated GORM code in the BucketController's actions:

Action Description
list This terse closure simply creates a hash map with a value retrieved from a call to AmazonS3.listBuckets(). That map is then automatically passed to list.gsp for rendering.
create This noop closure is simply a placeholder. RESTful requests to the create action are forwarded to create.gsp. Subsequent HTTP posts from create.gsp are handled by the save action.
save This action double-checks for the bucket's existence before creating a new bucket. An HTTP redirect is made to the list action to rebuild the list of buckets. Note the flash.message string, if it exists, will be displayed on the list page.
show The contents of an Amazon S3 bucket is the list of bucket objects that it contains. The BucketObjectController knows how to do that, so a redirect is made to its list action.
edit This action is not implemented. Amazon S3 objects are not edited, they are replaced.
update This action is not implemented. Amazon S3 objects are not updated, they are replaced.
delete Delete a Bucket object using the AmazonS3.deleteBucket method.

In your grails-app/view/bucket directory, delete the edit.gsp and show.gsp files, and then, in grails-app/views/bucket/list.gsp, replace the contents of the

division with the following:


  
idCreate Date
${bucket.name} ${bucket.creationDate}

Then, delete the

division in list.gsp.

Next, replace the entire contents of create.gsp with the following:



${flash.message}

Create Amazon S3 Bucket

Name:

The Bucket Object Controller

Now, you're ready to move on to the slightly more complex operations of dealing with Amazon S3 Bucket objects.

Replace the entire contents of grails-app/controllers/BucketObjectController.groovy with the following:

import com.amazonaws.auth.PropertiesCredentials
import com.amazonaws.services.s3.AmazonS3
import com.amazonaws.services.s3.AmazonS3Client
import com.amazonaws.services.s3.model.GetObjectRequest
import com.amazonaws.services.s3.model.ObjectMetadata
import com.amazonaws.services.s3.model.ListObjectsRequest

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;


class BucketObjectController {
  private static AmazonS3 s3
  private static PropertiesCredentials awsCreds
  
  def beforeInterceptor = {
    if (!s3) {
      awsCreds = new PropertiesCredentials(
      this.class.getResourceAsStream("AwsCredentials.properties"))
      s3 = new AmazonS3Client(awsCreds);
    }
  }
  def list = {
    params.max = Math.min(params.max ? params.int('max') : 5, 100)
    def bucketObjects
    if (flash.bucketObjects && params.page == 'next') {
      bucketObjects = s3.listNextBatchOfObjects(flash.bucketObjects)
      flash.page += 1
    } else {
      def lstObjReq = new ListObjectsRequest()
      lstObjReq.setBucketName(params.id)
      lstObjReq.setMaxKeys(params.max)
      bucketObjects = s3.listObjects(lstObjReq)
      flash.page = bucketObjects?.isTruncated()?1:0
    }
    if (bucketObjects?.isTruncated() ) {
      flash.bucketObjects = bucketObjects 
    } else {
      flash.bucketObjects = null
    }
    def bucketObjectsSummaries = bucketObjects?.objectSummaries
    def metadataList = []
    bucketObjectsSummaries.each {bos ->
      metadataList << 
           s3.getObjectMetadata(bucketObjects.bucketName, bos.key).metadata
    }
    [bucketName:params.id, 
     bucketObjectsSummaries:bucketObjectsSummaries, 
     metadataList:metadataList]
  }

  def create = {
    def now = new Date()+1
    def policy = 
     """{"expiration": "${now.format('yyyy-MM-dd')}T${now.format( 'hh:mm:ss')}Z", "conditions":
      [ 
       {"bucket": "$params.id"}, 
       ["starts-with", "\$key", ""],
       {"acl": "private"},
       {"success_action_redirect": "${g.resource(dir: '/',
        absolute:'true')}bucketObject/list/${params.id}"},
       ["starts-with", "\$Content-Type", ""],
       ["content-length-range", 0, 1048576]
     ]
    }""".replaceAll("\n","").replaceAll("\r","")
     .getBytes("UTF-8").encodeBase64().toString()
    Mac hmac = Mac.getInstance("HmacSHA1")
    hmac.init(new SecretKeySpec(awsCreds.getAWSSecretKey()
                .getBytes("UTF-8"), "HmacSHA1"));
    String signature = hmac.doFinal(policy.getBytes("UTF-8"))
                              .encodeBase64().toString()
    [policy:policy, signature:signature, params:params,
     awsAccessKeyId:awsCreds.AWSAccessKeyId]
  }

  def save = {render "save not implented" }

  def show = {
    params.max = params.max ? params.int('max') : 10000
    params.offset = params.offset ? params.int('offset'):0
    
    ObjectMetadata meta = s3.getObjectMetadata(params.bucketName, params.key)
    GetObjectRequest getObjReq = new GetObjectRequest(params.bucketName, params.key)
    getObjReq.withRange(params.offset, params.max+params.offset)

    [text:s3.getObject(getObjReq).objectContent.text,meta:meta.metadata, params:params]
  }
  def download = {  
    response.contentType = "application/octet-stream"
    response.outputStream <<
         s3.getObject(params.bucketName, params.key).objectContent
  }
  def edit = {render "edit not implemented"  }
  def update = {render "update not implented" }
  def delete = {
    s3.deleteObject(params.bucketName, params.key)
    flash.message = "bucket: $params.bucketName key: $params.key deleted."
    redirect(action: "list", id:params.bucketName)
  }
}

You will be modifying the create, list, and show GSPs, but go ahead and delete edit.gsp. You won't need the edit page, as you don't edit Amazon S3 objects: You can only replace the contents of an existing Amazon S3 Bucket object.

Amazon S3 Bucket Object List Processing

The list action of BucketObjectController uses several of the overloaded Amazon S3 listObjects methods. But because a bucket could contain innumerable objects, pagination is required. The Amazon S3 version of the list action could not easily use the standard Grails pagination facilities. The ListObjectsRequest parameter to the Amazon S3 listObjects(ListObjectsRequest listObjectsRequest) method has attributes for bucket name and the maximum number of keys (of bucket objects) you want returned. BucketObjectController's list closure sets the maximum key value to the default of 5 (you probably want to change that to a more reasonable number, like 10 or 20). The listObjects method returns an ObjectListing object that has an isTruncated() method. With BucketObjectController's list action, if the returned ObjectListing isTruncated() is true, the list action stores the ObjectListing to the Grails flash scope.

The existance of flash.bucketObjects is then used to predicate the Next button in list.gsp. And, when the list action responds to a Next or First button click, flash.bucketObjects is used to predicate the use of listNextBatchOfObjects(ObjectListing previousObjectListing). Also, so users know where they are in the list, a page number is kept in flash and displayed on the pagination line of list.gsp.

Note: Objects with the Grails' flash scope are retained for two HTTP request cycles.

The Bucket Object's List Page

In grails-app/views/bucketObject/list.gsp, replace the

and
divisions with the following:

id last modifiedcontent typecontent lengthDownload
${bucketObj.key} ${bucketObj.lastModified} ${metadataList[i]['Content-Type']} ${metadataList[i]['Content-Length']} download
<% if (flash.page) { %>
First Page: ${flash.page} <% if (flash.bucketObjects) { %> Next <% } %>
<% } %>

One last change to list.gsp: So that the BucketObjectController's list action knows the bucket name to list, add the following id attribute setting to the tag that is inside