Tag: guzzle


Concurrency in Version 3 of the AWS SDK for PHP

by Jonathan Eskew | on | in PHP | Permalink | Comments |  Share

From executing API commands in the background to grouping commands and waiters into concurrent pools, version 3 of the AWS SDK for PHP lets you write asynchronous code that blocks only when you need it to. In this blog post, I’ll show you how to take advantage of some of the SDK’s concurrency abstractions, including promises, command pools, and waiters.

Promises

The AWS SDK for PHP makes extensive use of promises internally, relying on the Guzzle Promises library. Each operation defined on an API supported by the SDK has an asynchronous counterpart that can be invoked by tacking Async to the name of the operation. An asynchronous operation will immediately return a promise that will be fulfilled with the result of the operation or rejected with an error:

$s3Client = new AwsS3S3Client([ 
    ‘region’ => ‘us-west-2’,
    ‘version’ => ‘2006-03-01’,
]);

// This call will block until a response is obtained
$buckets = $s3Client->listBuckets();

// This call will not block
$promise = $s3Client->listBucketsAsync();

// You can provide callbacks to invoke when the request finishes
$promise->then(
    function (AwsResultInterface $buckets) {
        echo ‘My buckets are: ‘
             . implode(‘, ‘, $buckets->search(‘Buckets[].Name’));
    },
    function ($reason) {
        echo "The promise was rejected with {$reason}";
    }
);

The Guzzle promises returned by asynchronous operations have a wait method that you can call if you need to block until an operation is completed. In fact, synchronous operations like $s3Client->listBuckets() are calling asynchronous operations under the hood and then waiting on the result.

Where promises really shine, though, is in groupings. Let’s say you need to upload ten files to an Amazon S3 bucket. Rather than simply loop over the files and upload them one by one, you can create ten promises and then wait for those ten requests to be complete:

$filesToUpload = [
    ‘/path/to/file/1.ext’,
    ‘/path/to/file/2.ext’,
    …
    ‘/path/to/file/10.ext’,
];
$promises = [];
foreach ($filesToUpload as $path) {
    $promises []= $s3Client->putObjectAsync([
        ‘Bucket’ => $bucketName,
        ‘Key’ => basename($path),
        ‘SourceFile’ => $path,
    ]);
}
// Construct a promise that will be fulfilled when all
// of its constituent promises are fulfilled
$allPromise = GuzzleHttpPromiseall($promises);
$allPromise->wait();

Rather than taking ten times as long as uploading a single file, the asynchronous code above will perform the uploads concurrently. For more information about promises, see the AWS SDK for PHP User Guide.

Waiters

Some AWS operations are naturally asynchronous (for example, those in which a successful response means that a process has been started, but is not necessarily complete). Provisioning Amazon EC2 instances or S3 buckets are gppd examples. If you were starting a project that required three S3 buckets and an Amazon ElastiCache cluster, you might start out by provisioning those resources programmatically:

$sdk = new AwsSdk([‘region’ => ‘us-west-2’, ‘version’ => ‘latest’]);
$elasticacheClient = $sdk->get(‘elasticache’);
$s3Client = $sdk->get(‘s3’);
$promises = [];
for ($i = 0; $i < 3; $i++) {
    $promises []= $s3Client->createBucket([
        ‘Bucket’ => “my-bucket-$i”,
    ]);
}
$cacheClusterId = uniqid(‘cache’);
$promises []= $elasticacheClient->createCacheCluster([
    ‘CacheClusterId’ => $cacheClusterId,
]);
$metaPromise = GuzzleHttpPromiseall($promises);
$metaPromise->wait();

Waiting on the $metaPromise will block only until all of the requests sent by the createBucket and createCacheCluster operations have been completed.You would need to use a waiter to block until those resources are available. For example, you can wait on a single bucket with an S3Client’s waitUntil method:

$s3Client->waitUntil(‘BucketExists’, [‘Bucket’ => $bucketName]);
Like operations, waiters can also return promises, allowing you to compose meta-waiters from individual waiters:
$waiterPromises = [];
for ($i = 0; $i < 3; $i++) {
    // Create a waiter
    $waiter = $s3Client->getWaiter(‘BucketExists’, [
        ‘Bucket’ => “my-bucket-$i”,
    ]);
    // Initiate the waiter and retrieve a promise.
    $waiterPromises []= $waiter->promise();
}
$waiterPromises []= $elasticacheClient
    ->getWaiter(‘CacheClusterAvailable’, [
        ‘CacheClusterId’ => $cacheClusterId,
    ])
    ->promise();
// Composer a higher-level promise from the individual waiter promises
$metaWaiterPromise = GuzzleHttpPromiseall($waiterPromises);
// Block until all waiters have completed
$metaWaiterPromise->wait();

Command Pools

The SDK also allows you to use command pools to fine-tune the way in which a series of operations are performed concurrentl. Command pools are created with a client object and an iterable list of commands, which can be created by calling getCommand with an operation name on any SDK client:

// Create an S3Client
$s3Client = new AwsS3S3Client([
    ‘region’ => ‘us-west-2’,
    ‘version’ => ‘latest’,
]);

// Create a list of commands
$commands = [
    $s3Client->getCommand(‘ListObjects’, [‘Bucket’ => ‘bucket1’]),
    $s3Client->getCommand(‘ListObjects’, [‘Bucket’ => ‘bucket2’]),
    $s3Client->getCommand(‘ListObjects’, [‘Bucket’ => ‘bucket3’]),
];

// Create a command pool
$pool = new AwsCommandPool($s3Client, $commands);

// Begin asynchronous execution of the commands
$promise = $pool->promise();

// Force the pool to complete synchronously
$promise->wait();

How is this different from gathering promises from individual operations, such as by calling $s3Client->listObjectsAsync(…)? One key difference is that no action is taken until you call $pool->promise(), whereas requests are dispatched immediately when you call $s3Client->listObjectsAsync(…). The CommandPool defers the initiation of calls until you explicitly tell it to do so. In addition, by default, a command pool will limit concurrency to 25 operations at a time by default, This simultaneous operation limit can be tuned according to your project’s needs. For a more complex example see CommandPool in the AWS SDK for PHP User Guide.

With promises, waiters, and command pools, version 3 of the AWS SDK for PHP makes it easy to write asynchronous or concurrent code! We welcome your feedback.

Guzzle 4 and the AWS SDK

by Jeremy Lindblom | on | in PHP | Permalink | Comments |  Share

Since Guzzle 4 was released in March (and even before then), we’ve received several requests for us to update the AWS SDK for PHP to use Guzzle 4. Earlier this month, we tweeted about it too and received some pretty positive feedback about the idea. We wanted to take some time to talk about what upgrading Guzzle would mean for the SDK and solicit your feedback.

The SDK relies heavily on Guzzle

If you didn’t already know, the AWS SDK for PHP relies quite heavily on version 3 of Guzzle. The AWS service clients extend from the Guzzle service clients, and we have formatted the entire set of AWS APIs into Guzzle "service descriptions". Roughly 80 percent of what the SDK does is done with Guzzle. We say all this because we want you to understand that updating the SDK to use Guzzle 4 is potentially a big change.

What does Guzzle 4 offer?

We’ve had several requests for Guzzle 4 support, and we agree that it would be great. But what exactly does Guzzle 4 offer — besides it being the new "hotness" — that makes it worth the effort?

We could mention a few things about the code itself: it’s cleaner, it’s better designed, and it has simpler and smaller interfaces. While those are certainly good things, they’re not strong enough reasons to change the SDK. However, Guzzle 4 also includes some notable improvements and new features, including:

  • It’s up to 30 percent faster and consumes less memory than Guzzle 3 when sending requests serially.
  • It no longer requires cURL, but still uses cURL by default, if available.
  • It supports swappable HTTP adapters, which enables you to provide custom adapters. For example, this opens up the possibility for a non-blocking, asynchronous adapter using ReactPHP.
  • It has improved cURL support, including faster and easier handling of parallel requests using a rolling queue approach instead of batching.

These updates would provide great benefits to SDK users, and would allow even more flexible and efficient communications with AWS services.

Guzzle 4 has already been adopted by Drupal, Laravel, Goutte, and other projects. I expect it to be adopted by even more during the rest of this year, and as some of the supplementary Guzzle packages reach stable releases. We definitely want users of the AWS SDK for PHP to be able to use the SDK alongside these other packages without causing conflicts or bloat.

Consequences of updating to Guzzle 4

Because the AWS SDK relies so heavily on Guzzle, the changes to Guzzle will require changes to the SDK.

In Guzzle 4, many things have changed. Classes have been renamed or removed, including classes that are used by the current SDK and SDK users. A few notable examples include the removal of the GuzzleBatch and GuzzleIterator namespaces, and how GuzzleHttpEntityBody has been changed and moved to GuzzleHttpStreamStream.

The event system of Guzzle 4 has also changed significantly. Guzzle has moved away from the Symfony Event Dispatcher, and is now using its own event system, which is pretty nice. This affects any event listeners and subscribers you may have written for Guzzle 3 or the SDK, because they will need a little tweaking to work in Guzzle 4.

Another big change in Guzzle 4 is that it requires PHP 5.4 (or higher). Using Guzzle 4 would mean that the SDK would also require PHP 5.4+.

Most of the changes in Guzzle 4 wouldn’t directly affect SDK users, but there are a few, like the ones just mentioned, that might. Because of this, if the SDK adopted Guzzle 4, it would require a new major version of the SDK: a Version 3.

What are your thoughts?

We think that updating the SDK to use Guzzle 4 is the best thing for the SDK and SDK users. Now that you know the benefits and the consequences, we want to hear from you. Do you have any questions or concerns? What other feedback or ideas do you have? Please join our discussion on GitHub or leave a comment below.

Receiving Amazon SNS Messages in PHP

by Jeremy Lindblom | on | in PHP | Permalink | Comments |  Share

The following post details how to use version 2 of the AWS SDK for PHP to receive and validate HTTP(S) messages from Amazon SNS. For a guide on how to do so with version 3 of the SDK, please see our updated post.

Handling inbound Amazon SNS notification messages with PHP is simple. In this post, I’ll show you how to retrieve data from incoming messages, and verify that the messages are coming from Amazon SNS.

A Little About Amazon SNS

Amazon Simple Notification Service (Amazon SNS) is a fast, fully-managed, push messaging service. Amazon SNS can deliver messages to email, mobile devices (i.e., SMS; iOS, Android and FireOS push notifications), Amazon SQS queues, and HTTP/HTTPS endpoints.

With Amazon SNS, you can setup topics to publish custom messages to subscribed endpoints. However, SNS messages are used by many of the other AWS services to communicate information asynchronously about your AWS resources. Some examples include:

  • Configuring Amazon Glacier to notify you when a retrieval job is complete.
  • Configuring AWS CloudTrail to notify you when a new log file has been written.
  • Configuring Amazon Elastic Transcoder to notify you when a transcoding job changes status (e.g., from "Progressing" to "Complete")

Though you can certainly subscribe your email address to receive SNS messages from service events like these, your inbox would fill up rather quickly. There is great power, however, in being able to subscribe an HTTP/HTTPS endpoint to receive the messages. This allows you to program webhooks for your applications to easily respond to various events.

Receiving a Message

In order for an HTTP/HTTPS endpoint to receive messages, you must subscribe the endpoint to an SNS topic. Before you do that, you need to create and deploy a script to the endpoint to process the messages.

Here is a naïvely simple PHP script that can read a posted SNS message.

<?php

// Fetch the raw POST body containing the message
$postBody = file_get_contents('php://input');

// JSON decode the body to an array of message data
$message = json_decode($postBody, true);
if ($message) {
    // Do something with the data
    echo $message['Message'];
}

The AWS SDK for PHP has an SNS Message class for representing an SNS message. It encapsulates the preceding code, and also validates the structure of the message data.

<?php

// Include Composer autoloader
require 'path/to/vendor/autoload.php';

// Create a message object from the POST body
$message = AwsSnsMessageValidatorMessage::fromRawPostData();
echo $message->get('Message');

Amazon SNS sends different types of messages including SubscriptionConfirmation, Notification, and UnsubscribeConfirmation. The formats of these messages are described on the Appendix: Message and JSON Formats section of the Amazon SNS Developer Guide.

Confirming a Subscription to a Topic

In order to handle a SubscriptionConfirmation message, we need to add some code that actually does something with the message. SubscriptionConfirmation messages provide a URL that you can use to confirm the subscription. We’ll use a Guzzle HTTP client to send a GET request to the URL.

$message = AwsSnsMessageValidatorMessage::fromRawPostData();

// Create a Guzzle client and send a request to the SubscribeURL
$client = new GuzzleHttpClient();
$client->get($message->get('SubscribeURL'))->send();

Verifying a SNS Message’s Signature

Messages from Amazon SNS are signed. It’s a good practice to verify the signature and ensure that a message was actually sent from Amazon SNS before performing actions as a result of the message. The SDK includes a MessageValidator class for validating the message, but you must have the OpenSSL PHP extension installed to use it.

use AwsSnsMessageValidatorMessage;
use AwsSnsMessageValidatorMessageValidator;

$message = Message::fromRawPostData();

// Validate the message
$validator = new MessageValidator();
$validator->validate($message);

Handling Notifications

Let’s put it all together and add some extra code for handling both SubscriptionConfirmation and Notification messages.

<?php

require 'path/to/vendor/autoload.php';

use AwsSnsMessageValidatorMessage;
use AwsSnsMessageValidatorMessageValidator;
use GuzzleHttpClient;

// Make sure the request is POST
if ($_SERVER['REQUEST_METHOD'] !== 'POST') {
    http_response_code(405);
    die;
}

try {
    // Create a message from the post data and validate its signature
    $message = Message::fromRawPostData();
    $validator = new MessageValidator();
    $validator->validate($message);
} catch (Exception $e) {
    // Pretend we're not here if the message is invalid
    http_response_code(404);
    die;
}

if ($message->get('Type') === 'SubscriptionConfirmation') {
    // Send a request to the SubscribeURL to complete subscription
    (new Client)->get($message->get('SubscribeURL'))->send();
} elseif ($message->get('Type') === 'Notification') {
    // Do something with the notification
    save_message_to_database($message);
}

Conclusion

As you can see, receiving, verifying and handling Amazon SNS messages is simple. Setting up your application to receive SNS messages will allow you to create applications that can handle asynchronous communication from AWS services and other parts of your application.

EDIT: My next blog post is a follow up to this one, and describes how you can test your Amazon SNS webhooks locally.

Wire Logging in the AWS SDK for PHP

by Jeremy Lindblom | on | in PHP | Permalink | Comments |  Share

One of the features of the AWS SDK for PHP that I often recommend to customers is the LogPlugin, that can be used to do wire logging. It is one of the many plugins included with Guzzle, which is the underlying HTTP library used by the SDK. Guzzle’s LogPlugin includes a default configuration that will output the content of the requests and responses sent over the wire to AWS. You can use it to help debug requests or just learn more about how the AWS APIs work.

Adding the LogPlugin to any client in the SDK is simple. The following shows how to set it up.

$logPlugin = GuzzlePluginLogLogPlugin::getDebugPlugin();
$client->addSubscriber($logPlugin);

The output generated by LogPlugin for a single request looks similar to the following text (this request was for executing an Amazon S3 ListBuckets operation).

# Request:
GET / HTTP/1.1
Host: s3.amazonaws.com
User-Agent: aws-sdk-php2/2.4.6 Guzzle/3.7.3 curl/7.25.0 PHP/5.3.27
Date: Fri, 27 Sep 2013 15:53:10 +0000
Authorization: AWS AKIAEXAMPLEEXAMPLE:eEXAMPLEEsREXAMPLEWEFo=

# Response:
HTTP/1.1 200 OK
x-amz-id-2: EXAMPLE4j/v8onDxyeuFaQFsNvN66EXAMPLE30KQLfq0T6sVcLxj
x-amz-request-id: 4F3EXAMPLEE14
Date: Fri, 27 Sep 2013 15:53:09 GMT
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3

<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">[...]</ListAllMyBucketsResult>

This is the output generated using the default configuration. You can configure the LogPlugin to customize the behavior, format, and location of what is logged. It’s also possible to integrate with third-party logging libraries like Monolog. For more information, see the section about the wire logger in the AWS SDK for PHP User Guide.