Tag: Waiters


Context Pattern added to the AWS SDK for Go

The AWS SDK for Go v1.8.0 release adds support for the API operation request functional options, and the Context pattern. Both of these features were high demand requests from our users. Request options allow you to easily configure and augment how the SDK makes API operation requests to AWS services. The SDK’s support for the Context pattern allows your application take advantage of cancellation, timeouts, and Context Values on requests.  The new request options and Context pattern give your application even more control over SDK’s request execution and handling.

Request Options

Request Options are functional arguments that you pass in to the SDK’s API operation methods. These enable you to configure the request in line with functional options. Functional options are a pattern you can use to configure an operation via passed-in functions or closures in line with the method call.

For example, you can configure the Amazon S3 API operation PutObject to log debug information about the request directly, without impacting the other API operations used by your application.

// Log this API operation only. 
resp, err := svc.PutObjectWithContext(ctx, params, request.WithLogLevel(aws.LogDebug))

This pattern is also helpful when you want your application to inject request handlers into the request. This allows you to do so in line with the API operation method call.

resp, err := svc.PutObjectWithContext(ctx, params, func(r *request.Request) {
	start := time.Now()
	r.Handlers.Complete.PushBack(func(req *request.Request) {
		fmt.Println("request %s took %s to complete", req.RequestID, time.Since(start))
	})
})

All of the SDK’s new service client methods that have a WithContext suffix support these request options. You can also apply request options to the SDK’s standard Request directly with the ApplyOptions method.

API Operations with Context

All of the new methods of the SDK’s API operations that have a WithContext suffix take a ContextValue. This value must be non-nil. Context allows your application to control API operation request cancellation. This means you can now easily institute request timeouts based on the Context pattern. Go introduced the Context pattern in the experimental package golang.org/x/net/context, and it was later added to the Go standard library in Go 1.7. For backward compatibility with previous Go versions, the SDK created the Context interface type in the github.com/aws/aws-sdk-go/aws package. The SDK’s Context type is compatible with Context from both golang.org/x/net/context and the Go 1.7 standard library Context package.

Here is an example of how to use a Context to cancel uploading an object to Amazon S3. If the put doesn’t complete within the timeout passed in, the API operation is canceled. When a Context is canceled, the SDK returns the CanceledErrorCode error code. A working version of this example can be found in the SDK.

sess := session.Must(session.NewSession())
svc := s3.New(sess)

// Create a context with a timeout that will abort the upload if it takes 
// more than the passed in timeout.
ctx := context.Background()
var cancelFn func()
if timeout > 0 {
	ctx, cancelFn = context.WithTimeout(ctx, timeout)
}
// Ensure the context is canceled to prevent leaking.
// See context package for more information, https://golang.org/pkg/context/
defer cancelFn()

// Uploads the object to S3. The Context will interrupt the request if the 
// timeout expires.
_, err := svc.PutObjectWithContext(ctx, &s3.PutObjectInput{
	Bucket: aws.String(bucket),
	Key:    aws.String(key),
	Body:   body,
})
if err != nil {
	if aerr, ok := err.(awserr.Error); ok && aerr.Code() == request.CanceledErrorCode {
		// If the SDK can determine the request or retry delay was canceled
		// by a context the CanceledErrorCode error code will be returned.
		fmt.Println("request's context canceled,", err)
	}
	return err
}

API Operation Waiters

Waiters were expanded to include support for request Context and waiter options. The new WaiterOption type defines functional options that are used to configure the waiter’s functionality.

For example, the WithWaiterDelay allows you to provide your own function that returns how long the waiter will wait before checking the waiter’s resource state again. This is helpful when you want to configure an exponential backoff, or longer retry delays with ConstantWaiterDelay.

The example below highlights this by configuring the WaitUntilBucketExists method to use a 30-second delay between checks to determine if the bucket exists.

svc := s3.New(sess)
ctx := contex.Background()

_, err := svc.CreateBucketWithContext(ctx, &s3.CreateBucketInput{
	Bucket: aws.String("myBucket"),
})
if err != nil {
	return fmt.Errorf("failed to create bucket, %v", err)
}

err := svc.WaitUntilBucketExistsWithContext(ctx,
	&s3.HeadBucket{
		Bucket: aws.String("myBucket"),
	},
	request.WithWaiterDelay(request.ConstantWaiterDelay(30 * time.Second)),
)
if err != nil {
	return fmt.Errorf("failed to wait for bucket exists, %v", err)
}

fmt.Println("bucket created")

API Operation Paginators

Paginators were also expanded to add support for Context and request options. Configuring request options for pagination applies the options to each new Request that the SDK creates to retrieve the next page. By extending the Pages API methods to include Context and request options the SDK gives you control over how the SDK will make each page request, and cancellation of the pagination.

svc := s3.New(sess)
ctx := context.Background()

err := svc.ListObjectsPagesWithContext(ctx,
	&s3.ListObjectsInput{
		Bucket: aws.String("myBucket"),
		Prefix: aws.String("some/key/prefix"),
		MaxKeys: aws.Int64(100),
	},
	func(page *s3.ListObjectsOutput, lastPage bool) bool {
		fmt.Println("Received", len(page.Contents), "objects in page")
		for _, obj := range page.Contents {
			fmt.Println("Key:", aws.StringValue(obj.Key))
		}
		return true
	},
)
if err != nil {
	return fmt.Errorf("failed to create bucket, %v", err)
}

API Operation Pagination without Callbacks

In addition to the Pages API operations, you can use the new Pagination type in the github.com/aws/aws-sdk-go/aws/request package. This type enables you to control the iterations of pages directly. This is helpful when you do not want to use callbacks for paginating AWS operations. This new type allows you to treat pagination similar to the Go stdlib bufio package’s Scanner type to iterate through pages with a for loop. You can also use this pattern with the Context pattern by calling Request.SetContext on each request in the NewRequest function.

svc := s3.New(sess)

params := s3.ListObjectsInput{
	Bucket: aws.String("myBucket"),
	Prefix: aws.String("some/key/prefix"),
	MaxKeys: aws.Int64(100),
}
ctx := context.Background()

p := request.Pagination{
	NewRequest: func() (*request.Request, error) {
		req, _ := svc.ListObjectsRequest(&params)
		req.SetContext(ctx)
		return req, nil
	},
}

for p.Next(){
	page := p.Page().(*s3.ListObjectsOutput)
	
	fmt.Println("Received", len(page.Contents), "objects in page")
	for _, obj := range page.Contents {
		fmt.Println("Key:", aws.StringValue(obj.Key))
	}
}

return p.Err()

Wrap Up

The addition of Context and request options expands the capabilities of the AWS SDK for Go, giving your applications the tools needed to implement request lifecycle and configuration with the SDK. Let us know your experiences using the new Context pattern and request options features.

Waiters

by Trevor Rowe | on | in Ruby | Permalink | Comments |  Share

We’ve added a feature called Waiters to the v2 AWS SDK for Ruby, and I am pretty excited about it. A waiter is a simple abstraction around the pattern of polling an AWS API until a desired state is reached.

Basic Usage

This simple example shows how to use waiters to block until a particular EC2 instance is running:

ec2 = Aws::EC2::Client.new
ec2.wait_until(:instance_running, instance_ids:['i-12345678'])

Waiters will not wait indefinitely and can fail. Each waiter has a default polling interval and maximum number of attempts to make. If a waiter encounters an unexpected error or fails to reach the desired condition in time it will raise an error:

begin
  ec2.wait_until(:instance_running, instance_ids:['i-12345678'])
resuce Aws::Waiters::Errors::WaiterFailed
  # oops
end

Configuration

You can modify the default interval and wait time between attempts by passing a block.

# this will wait upto ~ one hour
ec2.wait_until(:instance_running, instance_ids:['i-12345678']) do |w|

  # seconds between each attempt
  w.interval = 15

  # maximum number of polling attempts before giving up
  w.max_attempts = 240

end

Callbacks

In addition to interval and maximum attempts, you can configure callbacks to trigger before each attempt polling attempt and before sleeping between attempts.

ec2.wait_until(:instance_running, instance_ids:['i-12345678']) do |w|

  w.before_attempt do |n|
    # n - the number of attempts made
  end

  w.before_wait do |n, resp|
    # n - the number of attempts made
    # resp -the client response from the previous attempt
  end

end

You can throw :success or :failure from these callbacks to stop the waiter immediately. You can use this to write you own delay and back-off logic.

Here I am using a callback to perform exponential back-off between polling attempts:

ec2.wait_until(:instance_running, instance_ids:['i-12345678']) do |w|
  w.interval = 0 # disable normal sleep
  w.before_wait do |n, resp|
    sleep(n ** 2)
  end
end

This example gives up after one hour.

ec2.wait_until(:instance_running, instance_ids:['i-12345678']) do |w|
  one_hour_later = Time.now + 3600
  w.before_wait do |n, resp|
    throw :failure, 'waited too long' if Time.now > one_hour_later
  end
end

Waiters and Resources, Looking Ahead

You may have noticed that some waiters have already been exposed to the resource classes.

ec2 = Aws::EC2::Resource.new
instance = ec2.instance('i-12345678')
instance.stop
instance.wait_until_stopped
puts instance.id + ' is stopped'

In addition to connecting more waiters and resources, I’m excited to look into batch waiters. Imagine the following use case:

instances = ec2.create_instances(min_count: 5, ...)
instances.wait_until_running
puts "the following new instances are now running:n"
puts instances.map(&:id)

Documentation

Waiters are documented in the Ruby SDK API reference. Each service client documents the #wait_until method and provides a list of available waiter names. Here are links to the Aws::EC2::Client waiter methods:

Give waiters a try and let us know what you think!

Provision an Amazon EC2 Instance with PHP

by Jeremy Lindblom | on | in PHP | Permalink | Comments |  Share

Amazon EC2 is a powerful AWS service that includes the ability to provision on-demand servers. While you can easily do this through the AWS Management Console, in this post, I want show you how to use the AWS SDK for PHP to do it programmatically by interacting with the Amazon EC2 API.

Let’s create a single PHP script, piece by piece, that uses the SDK to do the following:

  1. Create and configure an Amazon EC2 client.
  2. Create an EC2 key pair and store the private key.
  3. Create and configure an EC2 security group.
  4. Launch an EC2 instance of an Amazon Machine Image (AMI) and retrieve its public DNS name so we can access it via SSH.

Create an EC2 client

First, let’s bootstrap the SDK and create an EC2 client object. Make sure to replace the placeholder values in the following code with your AWS credentials and desired region.

<?php

require 'vendor/autoload.php';

use Aws\Ec2\Ec2Client;

$ec2Client = Ec2Client::factory(array(
    'key'    => '[aws access key]',
    'secret' => '[aws secret key]',
    'region' => '[aws region]' // (e.g., us-east-1)
));

Create a key pair

Next, we’ll create a key pair that will provide SSH access to our server once it is running. We need to create the key pair first so we can specify it when we launch the EC2 instance. Creating the key pair is simple.

// Create the key pair
$keyPairName = 'my-keypair';
$result = $ec2Client->createKeyPair(array(
    'KeyName' => $keyPairName
));

In order to use the key pair later, we will need to save the private key locally. We can do this by extracting the key material from the response and using some of PHP’s file handling functions to save it to a file. We also need to adjust the file permissions so that the key can be used for SSH access.

// Save the private key
$saveKeyLocation = getenv('HOME') . "/.ssh/{$keyPairName}.pem";
file_put_contents($saveKeyLocation, $result['keyMaterial']);

// Update the key's permissions so it can be used with SSH
chmod($saveKeyLocation, 0600);

Create and configure a security group

Next, let’s create and configure a security group which will allow the server to be accessed via HTTP (port 80) and SSH (port 22). By default, access to an EC2 instance is completely locked down. Security groups allow you to whitelist access to ports on an EC2 instance. Creating a security group requires only a name and description.

// Create the security group
$securityGroupName = 'my-security-group';
$result = $ec2Client->createSecurityGroup(array(
    'GroupName'   => $securityGroupName,
    'Description' => 'Basic web server security'
));

// Get the security group ID (optional)
$securityGroupId = $result->get('GroupId');

After creating the security group, you can then configure its rules. To open up ports 22 and 80 we will use the AuthorizeSecurityGroupIngress operation and specify the security group name.

// Set ingress rules for the security group
$ec2Client->authorizeSecurityGroupIngress(array(
    'GroupName'     => $securityGroupName,
    'IpPermissions' => array(
        array(
            'IpProtocol' => 'tcp',
            'FromPort'   => 80,
            'ToPort'     => 80,
            'IpRanges'   => array(
                array('CidrIp' => '0.0.0.0/0')
            ),
        ),
        array(
            'IpProtocol' => 'tcp',
            'FromPort'   => 22,
            'ToPort'     => 22,
            'IpRanges'   => array(
                array('CidrIp' => '0.0.0.0/0')
            ),
        )
    )
));

Note: In this simple example, we are granting all IP addresses access to these two ports, but in a production setting you should consider limiting the access to certain IP addresses or ranges as appropriate. Also, you may need to open additional ports for MySQL or HTTPS traffic.

Launch an instance

Now that we have a key pair and security group set up, we are ready to launch an EC2 instance (our server) with these settings. To launch an EC2 instance, you also need to specify the ImageId parameter, which is a reference to the AMI that the EC2 instance should be created from. In this example, we are going to use an Amazon Linux AMI. Use the EC2 RunInstances operation to launch the instance.

// Launch an instance with the key pair and security group
$result = $ec2Client->runInstances(array(
    'ImageId'        => 'ami-570f603e',
    'MinCount'       => 1,
    'MaxCount'       => 1,
    'InstanceType'   => 'm1.small',
    'KeyName'        => $keyPairName,
    'SecurityGroups' => array($securityGroupName),
));

From the result, we must get the ID of the instance. We do this using the getPath method available on the result object. This allows us to pull data out of the result that is deep within the result’s structure. The following line of code retrieves an array of instance IDs from the result. In this case, where we have launched only a single instance, the array contains only one value.

$instanceIds = $result->getPath('Instances/*/InstanceId');

Now that the launch has been triggered, we must wait for the instance to become available. The AWS SDK for PHP provides a feature called Waiters, which allow you to poll a resource until it is in a desired state. We will use the waitUntilInstanceRunning method of the EC2 client to wait until the instance that we have just launched is in the “Running” state.

// Wait until the instance is launched
$ec2Client->waitUntilInstanceRunning(array(
    'InstanceIds' => $instanceIds,
));

Once the instance is running, we can use the DescribeInstances operation to retrieve information about the instance, including its public DNS name. We’ll use the getPath method again on the result to extract the PublicDnsName value.

// Describe the now-running instance to get the public URL
$result = $ec2Client->describeInstances(array(
    'InstanceIds' => $instanceIds,
));
echo current($result->getPath('Reservations/*/Instances/*/PublicDnsName'));

Using the public DNS name and the private key that you downloaded, you can SSH into the server. You can do this (from Linux/Unix and Mac devices) by using the ssh command from your CLI.

ssh -i <path to key> ec2-user@<public dns name>

Once you are logged in, you can install software (e.g., yum install php) and deploy your application. Good work! Hopefully, this tutorial helps you with your next PHP-related DevOps project.