AWS SDK for Java Tips and Tricks

Articles & Tutorials>Amazon RDS>AWS SDK for Java Tips and Tricks
This article describes some helpful tips and tricks for developing applications on the AWS SDK for Java.

Details

Submitted By: Craig@AWS
AWS Products Used: Amazon EC2, Amazon Elastic MapReduce, Amazon RDS, Amazon S3, Amazon SQS, Amazon SimpleDB
Language(s): Java
Created On: March 21, 2010 5:38 PM GMT
Last Updated: September 3, 2010 5:55 PM GMT

Using the SDK, you can build solutions for Amazon Simple Storage Service (Amazon S3), Amazon Elastic Compute Cloud (Amazon EC2), Amazon SimpleDB, and more. The AWS SDK for Java includes the AWS Java library, code samples, and reference documentation, but what about information on how to use the Java library? This article addresses some important tips for developing software with the AWS SDK for Java. The primary areas we'll address are:

  • Client configuration
  • Logging control
  • Exception handling

See our Java Development forum if you have additional questions or comments about developing applications on the AWS SDK for Java.

Client Configuration

The AWS SDK for Java allows you to change the default client configuration, which is helpful when you want to:

  • Change the region endpoint
  • Connect to the Internet through proxy
  • Tweak HTTP transport settings, such as connection timeout and request retries.
  • Specify TCP socket buffer size hints

AWS Region Selection

Regions allow you to access AWS services that physically reside in a specific geographic region. This can be useful both for redundancy and to keep your data and applications running close to where you and your users will access them.

Each client can be pointed to a specific endpoint through the setEndpoint(String endpointUrl) method.

For example, to point the AWS EC2 Java client at the EU West region, use the following code:

AmazonEC2 ec2 = new  AmazonEC2(myCredentials);
ec2.setEndpoint("https://eu-west-1.ec2.amazonaws.com");

Be aware that regions are logically isolated from each other, so for example, you won't be able to access US East resources when communicating with the EU West endpoint.

See Available Region Endpoints for the AWS SDKs for a list of valid endpoints.

Proxy Configuration

When constructing a client object, you can pass in an optional com.amazonaws.ClientConfiguration object to customize the client's configuration.

If you're connecting to the Internet through a proxy server, you'll need to configure your proxy server settings (proxy host, port and username/password) through the ClientConfiguration object.

HTTP Transport Configuration

Several HTTP transport options can be configured through the com.amazonaws.ClientConfiguration object. Default values will suffice for the majority of users, but users who want more control can configure:

  • Socket timeout
  • Connection timeout
  • Maximum retry attempts for retry-able errors
  • Maximum open HTTP connections

TCP Socket Buffer Size Hints

Advanced users who want to tune low-level TCP parameters can additionally set TCP buffer size hints through the ClientConfiguration object. The majority of users will never need to tweak these values, but they are provided for advanced users.

Optimal TCP buffer sizes for an application are highly dependent on network and OS configuration and capabilities. For example, most modern operating systems provide auto-tuning logic for TCP buffer sizes, which can have a big impact on performance for TCP connections that are held open long enough for the auto-tuning to optimize buffer sizes.

Large buffer sizes (ex: 2MB) allow the OS to buffer more data in memory without requiring the remote server to acknowledge receipt of that information, so can be particularly useful when the network has high latency.

This is only a hint, and the OS may choose not to honor it. When using this option, users should always check the operating system's configured limits and defaults. Most OS's have a maximum TCP buffer size limit configured, and won't let you go beyond that limit unless you explicitly raise the max TCP buffer size limit.

Many resources available to help with configuring TCP buffer sizes and operating system specific TCP settings, including:

Logging Control

The AWS SDK for Java uses Apache Commons Logging to log diagnostic messages. Apache Commons Logging provides a generic interface to logging libraries so that the user can plug in any compatible logging library. AWS recommends using Apache log4j as the logging library underneath Commons Logging and the configuration examples below are all log4j.properties files.

Service-Specific Errors and Warnings

AWS recommends that you always leave the "com.amazonaws" logger hierarchy set to "WARN" in order to catch any important messages from the client libraries. For example, if the Amazon S3 client detects that your application hasn't properly closed an InputStream and could be leaking resources, it will report it through a warning message to the logs. This will also ensure that messages are logged if the client has any problems handling requests or responses.

The following log4j.properties file sets the rootLogger to WARN, which will cause warning and error messages from all loggers in the "com.amazonaws" hierarchy to be included. Alternatively, you can explicitly set the com.amazonaws logger to WARN.

log4j.rootLogger=WARN, A1
log4j.appender.A1=org.apache.log4j.ConsoleAppender
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%d [%t] %-5p %c -  %m%n
# Or you can explicitly enable WARN and ERROR messages for  the AWS Java clients
log4j.logger.com.amazonaws=WARN

Request/Response Summary Logging

Every request to an AWS service generates a unique AWS request ID that is useful if you run into an issue with how an AWS service is handling a request. AWS request IDs are accessible programmatically through Exception objects in the SDK for any failed service call, and can also be reported through the INFO log level in the "com.amazonaws.request" logger.

The following log4j.properties file enables a summary of requests and responses, including AWS request IDs.

log4j.rootLogger=WARN, A1
log4j.appender.A1=org.apache.log4j.ConsoleAppender
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%d [%t] %-5p %c -  %m%n
# Turn on INFO logging in com.amazonaws.request to log
# a summary of requests/responses with AWS request IDs
log4j.logger.com.amazonaws.request=INFO

Here is an example of the log output:

2009-12-17 09:53:04,269 [main] INFO com.amazonaws.request - Sending Request: POST https://rds.amazonaws.com / Parameters: (MaxRecords: 20, Action: DescribeEngineDefaultParameters, SignatureMethod: HmacSHA256, AWSAccessKeyId: AKIAJUHAG74EVMS3UVZA, Version: 2009-10-16, SignatureVersion: 2, Engine: mysql5.1, Timestamp: 2009-12-17T17:53:04.267Z, Signature: 4ydexGGkC77PovHhbfzAMA1H0nDnqIQxG9q+Yq3uw5s=, )
2009-12-17 09:53:04,464 [main] INFO com.amazonaws.request - Received successful response: 200, AWS Request ID: 06c12a39-eb35-11de-ae07-adb69edbb1e4
2009-12-17 09:53:04,469 [main] INFO com.amazonaws.request - Sending Request: POST https://rds.amazonaws.com / Parameters: (ResetAllParameters: true, Action: ResetDBParameterGroup, SignatureMethod: HmacSHA256, DBParameterGroupName: java-integ-test-param-group-1261072381023, AWSAccessKeyId: AKIAJUHAG74EVMS3UVZA, Version: 2009-10-16, SignatureVersion: 2, Timestamp: 2009-12-17T17:53:04.467Z, Signature: 9WcgfPwTobvLVcpyhbrdN7P7l3uH0oviYQ4yZ+TQjsQ=, )
2009-12-17 09:53:04,646 [main] INFO com.amazonaws.request - Received successful response: 200, AWS Request ID: 06e071cb-eb35-11de-81f7-01604e1b25ff

Verbose Wire Logging

In some cases, it may be useful to see the exact requests and responses being sent and received by the AWS SDK for Java. This logging should not be enabled in production systems since writing out large requests (ex: a file being uploaded to Amazon S3) or responses can significantly slow down an application. If you really need access to this information, you can temporarily enable it through Apache HttpClient's logger. Enabling the DEBUG level on the "httpclient.wire" logger enables logging for all request and response data.

The following log4j.properties file turns on full wire logging in Apache HttpClient and should only be turned on temporarily since it can have a significant performance impact on your application.

log4j.rootLogger=WARN, A1
log4j.appender.A1=org.apache.log4j.ConsoleAppender
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%d [%t] %-5p %c -  %m%n
# Log all HTTP content (headers, parameters, content, etc)  for
# all requests and responses. Use caution with this since it can
# be very expensive to log such verbose data!
log4j.logger.httpclient.wire=DEBUG

Exception Handling

Understanding how and when the AWS SDK for Java throws exceptions is important in order to build high quality applications using the SDK. The following sections describe the different cases of exceptions that are thrown by the SDK and how to handle them appropriately.

Why Unchecked Exceptions?

The AWS Java SDK uses runtime (or unchecked) exceptions instead of checked exceptions for a few reasons:

  • To allow developers fine grained control over the errors they want to handle without forcing them to handle exceptional cases they aren't concerned about (and making their code overly verbose)
  • To prevent scalability issues inherent with checked exceptions in large applications

In general, checked exceptions work well on small scales, but can become troublesome as applications grow and become more complex.

For more information on the ongoing discussion in the Java community around the use of checked and unchecked exceptions, see the following references:

AmazonServiceException (and Subclasses)

This is the main type of exception that you'll need to deal with when using the AWS SDK for Java. This exception represents an error response from an AWS service. For example, if you request to terminate an Amazon EC2 instance that doesn't exist, EC2 will return an error response and all the details of that error response will be included in the thrown AmazonServiceException. For some cases, a subclass of AmazonServiceException will be thrown to allow developers fine grained control over handling error cases through catch blocks.

When you encounter an AmazonServiceException, you know that your request was successfully sent to the AWS service, but could not be successfully processed either because of errors in the request's parameters or because of issues on the service side.

AmazonServiceException has several useful fields in it, including:

  • The returned HTTP status code
  • The returned AWS error code
  • The detailed error message from the service
  • The AWS request ID for the failed request

AmazonServiceException also includes a field that describes whether the failed request was the caller's fault (i.e. a request with illegal values) or the AWS service's fault (i.e. an internal service error).

AmazonClientException

This exception indicates that a problem occurred inside the Java client code, either while trying to send a request to AWS or while trying to parse a response from AWS. AmazonClientExceptions are more severe than AmazonServiceExceptions and indicate a major problem that is preventing the client from being able to make service calls to AWS services. For example, the AWS Java SDK will throw an AmazonClientException if no network connection is available when you try to call an operation on one of the clients.

IllegalArgumentException

When calling operations if you pass an illegal argument, the AWS SDK for Java will throw an IllegalArgumentException. For example, if you call an operation and pass a null value in for one of the required parameters, the SDK will throw an IllegalArgumentException describing the illegal argument.

Using Access Control Policies

AWS access control policies allow you to specify fine grained access controls on your AWS resources. You can allow or deny access to your AWS resources based on:

  • what '''resource''' is being accessed
  • who is accessing the resource (i.e. the principal)
  • what action is being taken on the resource
  • a variety of conditions including date restrictions, IP address restrictions, etc.

Access control policies are a collection of statements. Each statement takes the form: "A has permission to do B to C where D applies".

  • A is the principal - the AWS account that is making a request to access or modify one of your AWS resources.
  • B is the action - the way in which your AWS resource is being accessed or modified, such as sending a message to an Amazon SQS queue, or storing an object in an Amazon S3 bucket.
  • C is the resource - your AWS entity that the principal wants to access, such as an Amazon SQS queue, or an object stored in Amazon S3.
  • D is the set of conditions - optional constraints that specify when to allow or deny access for the principal to access your resource. Many expressive conditions are available, some specific to each service. For example you can use date conditions to allow access to your resources only after or before a specific time.

Amazon S3 Example

The following example demonstrates a policy that allows anyone access to read all the objects in a bucket, but restricts access to uploading objects to that bucket to two specific AWS accounts (in addition to the bucket owner's account).

Statement allowPublicReadStatement = new Statement(Effect.Allow)
         .withPrincipals(Principal.AllUsers)
         .withActions(S3Actions.GetObject)
         .withResources(new S3ObjectResource(myBucketName, "*"));
Statement allowRestrictedWriteStatement = new Statement(Effect.Allow)
         .withPrincipals(new Principal("123456789"), new Principal("876543210"))
         .withActions(S3Actions.PutObject)
         .withResources(new S3ObjectResource(myBucketName, "*"));

Policy policy = new Policy()
         .withStatements(allowPublicReadStatement, allowRestrictedWriteStatement);

AmazonS3 s3 = new AmazonS3Client(myAwsCredentials);
s3.setBucketPolicy(myBucketName, policy.toJson());

Amazon SQS Example

One common use of policies is to authorize an Amazon SQS queue to receive messages from an Amazon SNS topic.

/*
 * This policy allows an SNS topic to send messages to an SQS queue.
 * You can find your SNS topic's ARN through the SNS getTopicAttributes operation.
 */
Policy policy = new Policy().withStatements(
        new Statement(Effect.Allow)
            .withPrincipals(Principal.AllUsers)
            .withActions(SQSActions.SendMessage)
            .withConditions(ConditionFactory.newSourceArnCondition(myTopicArn)));

Map queueAttributes = new HashMap();
queueAttributes.put(QueueAttributeName.Policy.toString(), policy.toJson());

AmazonSQS sqs = new AmazonSQSClient(myAwsCredentials);
sqs.setQueueAttributes(new SetQueueAttributesRequest(myQueueUrl, queueAttributes));

Amazon SNS Example

Some services offer additional conditions that can be used in policies. Amazon SNS provides conditions for allowing or denying subscriptions to SNS topics based on the protocol ''(ex: email, HTTP, HTTPS, SQS)'' and endpoint ''(ex: email address, URL, SQS ARN)'' of the request to subscribe to a topic.

/*
 * This SNS condition allows you to restrict subscriptions to an Amazon SNS topic
 * based on the requested endpoint (email address, SQS queue ARN, etc) used when
 * someone tries to subscribe to your SNS topic.
 */
Condition endpointCondition = 
        SNSConditionFactory.newEndpointCondition("*@mycompany.com");

Policy policy = new Policy().withStatements(
        new Statement(Effect.Allow)
            .withPrincipals(Principal.AllUsers)
            .withActions(SNSActions.Subscribe)
            .withConditions(endpointCondition));

AmazonSNS sns = new AmazonSNSClient(myAwsCredentials);
sns.setTopicAttributes(
        new SetTopicAttributesRequest(myTopicArn, "Policy", policy.toJson()));

Additional Resources

Some additional resources to help you build your application include:

©2014, Amazon Web Services, Inc. or its affiliates. All rights reserved.