Tag: logs


PHP application logging with Amazon CloudWatch Logs and Monolog

by Joseph Fontes | on | in PHP | Permalink | Comments |  Share

Logging and information debugging can be approached from a multitude of different angles. Whether you use an application framework or coding from scratch it’s always comforting to have familiar components and tools across different projects. In our examples today, I am going to enable Amazon CloudWatch Logs logging with a PHP application. To accomplish this, I wanted to use an existing solution that is both already popular and well used, and that is standards compliant. For these reasons, we are going to use the open source log library, PHP Monolog (https://github.com/Seldaek/monolog).

PHP Monolog

For those who work with a new PHP application, framework, or service, one of the technology choices that appears more frequently across solutions is the use of Monolog for application logging. PHP Monolog is a standards-compliant PHP library that enables developers to send logs to various destination types including, databases, files, sockets, and different services. Although PHP Monolog predates the standards for PHP logging defined in PSR-3, it does implement the PSR-3 interface and standards. This makes Monolog compliant with the common interface for logging libraries. Using Monolog with CloudWatch Logs creates a PSR-3 compatible logging solution. Monolog is available for use with a number of different applications and frameworks such as Laravel, Symfony, CakePHP, and many others. Our example today is about using PHP Monolog to send information to CloudWatch Logs for the purpose of application logging and to build a structure and process that enables the use of our application data with CloudWatch alarms and notifications. This enables us to use logs from our application for cross-service actions such as with Amazon EC2 Auto Scaling decisions.

Amazon CloudWatch Logs

As a customer-driven organization, AWS is constantly building and releasing significant features and services requested by AWS customers and partners. One of those services that we highlight today is Amazon CloudWatch Logs. CloudWatch Logs enables you to store log file information from applications, operating systems and instances, AWS services, and various other sources. An earlier blog post highlighted the use of CloudWatch Logs with various programming examples.

Notice in the blog post that there is a PHP example that uses CloudWatch Logs to store an entry from an application. You can use this example and extend it as a standalone solution to provide logging to CloudWatch Logs from within your application. With our examples, we’ll enhance this opportunity by leveraging PHP Monolog.

Implementing Monolog

To begin using Monolog, we install the necessary libraries with the use of Composer (https://getcomposer.org/). The instructions below install the AWS SDK for PHP, PHP Monolog, and an add-on to Monolog that enables logging to CloudWatch Logs.

curl -sS https://getcomposer.org/installer | php
php composer.phar require aws/aws-sdk-php
php composer.phar require monolog/monolog
php composer.phar require maxbanton/cwh:^1.0

Alternatively, you can copy the following entry to the composer.json file and install it via the php composer.phar install command.

{
    "minimum-stability": "stable",
    "require": {
        "aws/aws-sdk-php": "^3.24",
        "aws/aws-php-sns-message-validator": "^1.1",
        "monolog/monolog": "^1.21",
        "maxbanton/cwh": "^1.0"
    }
}

Local logging

Now that PHP Monolog is available for use, we can test the implementation. We start with an example of logging to a single file.

require "vendor/autoload.php";

use Monolog\Logger;
use Monolog\Formatter\LineFormatter;
use Monolog\Handler\StreamHandler;

$logFile = "testapp_local.log";

$logger = new Logger('TestApp01');
$formatter = new LineFormatter(null, null, false, true);
$infoHandler = new StreamHandler(__DIR__."/".$logFile, Logger::INFO);
$infoHandler->setFormatter($formatter);
$logger->pushHandler($infoHandler);
$logger->info('Initial test of application logging.');

In the previous example, we start by requiring the composer libraries we installed earlier. The new Logger line sets the channel name as “TestApp01”. The next line creates a new LineFormatter that removes brackets around unused log items. The next line establishes the destination as the file name we identified, testapp_local.log, and associates that with the INFO log level. Next, we apply the format to our stream handler. Then we add the stream handler with the updated format to the handler list. Finally, a new message is logged with the log level of INFO. For information about log levels and different handlers, see the Monolog GitHub page and IETF RFC 5424 and PSR-3.

We can now view the contents of the log file to ensure functionality:

Syslog logging

Now that we are able to write a simple log entry to a local file, our next example uses the system Syslog to log events.

$logger = new Logger($appName);

$localFormatter = new LineFormatter(null, null, false, true);
$syslogFormatter = new LineFormatter("%channel%: %level_name%: %message% %context% %extra%",null,false,true);

$infoHandler = new StreamHandler(__DIR__."/".$logFile, Logger::INFO);
$infoHandler->setFormatter($localFormatter);

$warnHandler = new SyslogHandler($appName, $facility, Logger::WARNING);
$warnHandler->setFormatter($syslogFormatter);

$logger->pushHandler($warnHandler);
$logger->pushHandler($infoHandler);

$logger->info('Test of PHP application logging.');
$logger->warn('Test of the warning system logging.');

Here we can see that the format of the syslog messages has been changed with the value, $syslogFormatter. Because syslog provides a date/time with each log entry, we don’t need to include these values in our log text. The syslog facility is set to local0 with all WARNING messages sent to syslog with the INFO level messages and WARNING level messages logged to our local file. You can find additional information about Syslog facilities and log levels on the Syslog Wikipedia page.

Logging to CloudWatch Logs

Now that you’ve seen the basic use of Monolog, let’s send some logs over to CloudWatch Logs. We can use the Amazon Web Services CloudWatch Logs Handler for Monolog library to integrate Monolog with CloudWatch Logs. In our example, an authentication application produces log information.

use Aws\CloudWatchLogs\CloudWatchLogsClient;
use Maxbanton\Cwh\Handler\CloudWatch;
use Monolog\Logger;
use Monolog\Formatter\LineFormatter;
use Monolog\Handler\StreamHandler;
use Monolog\Handler\SyslogHandler;

$logFile = "testapp_local.log";
$appName = "TestApp01";
$facility = "local0";

// Get instance ID:
$url = "http://169.254.169.254/latest/meta-data/instance-id";
$instanceId = file_get_contents($url);

$cwClient = new CloudWatchLogsClient($awsCredentials);
// Log group name, will be created if none
$cwGroupName = 'php-app-logs';
// Log stream name, will be created if none
$cwStreamNameInstance = $instanceId;
// Instance ID as log stream name
$cwStreamNameApp = "TestAuthenticationApp";
// Days to keep logs, 14 by default
$cwRetentionDays = 90;

$cwHandlerInstanceNotice = new CloudWatch($cwClient, $cwGroupName, $cwStreamNameInstance, $cwRetentionDays, 10000, [ 'application' => 'php-testapp01' ],Logger::NOTICE);
$cwHandlerInstanceError = new CloudWatch($cwClient, $cwGroupName, $cwStreamNameInstance, $cwRetentionDays, 10000, [ 'application' => 'php-testapp01' ],Logger::ERROR);
$cwHandlerAppNotice = new CloudWatch($cwClient, $cwGroupName, $cwStreamNameApp, $cwRetentionDays, 10000, [ 'application' => 'php-testapp01' ],Logger::NOTICE);

$logger = new Logger('PHP Logging');

$formatter = new LineFormatter(null, null, false, true);
$syslogFormatter = new LineFormatter("%channel%: %level_name%: %message% %context% %extra%",null,false,true);
$infoHandler = new StreamHandler(__DIR__."/".$logFile, Logger::INFO);
$infoHandler->setFormatter($formatter);

$warnHandler = new SyslogHandler($appName, $facility, Logger::WARNING);
$warnHandler->setFormatter($syslogFormatter);

$cwHandlerInstanceNotice->setFormatter($formatter);
$cwHandlerInstanceError->setFormatter($formatter);
$cwHandlerAppNotice->setFormatter($formatter);

$logger->pushHandler($warnHandler);
$logger->pushHandler($infoHandler);
$logger->pushHandler($cwHandlerInstanceNotice);
$logger->pushHandler($cwHandlerInstanceError);
$logger->pushHandler($cwHandlerAppNotice);

$logger->info('Initial test of application logging.');
$logger->warn('Test of the warning system logging.');
$logger->notice('Application Auth Event: ',[ 'function'=>'login-action','result'=>'login-success' ]);
$logger->notice('Application Auth Event: ',[ 'function'=>'login-action','result'=>'login-failure' ]);
$logger->error('Application ERROR: System Error');

In this example, application authentication events are passed as a PHP array and presented in CloudWatch Logs as JSON. The events with a result of login-success and login-failure are sent to both the log stream associated with the instance ID and to the log stream associated with the application name.

 

Using these different stream locations, we can create metrics and alarms at either a per-instance level or per-application level. Let’s assume that we want to create a metric for total number of users logged into our application over the past five minutes. Select your event group and then choose Create Metric Filter.

On the next page, we can create our filter and test in the same window. For the filter data, we use the JSON string from the log entry. Enter the following string to extract all the successful logins.

{ $.result = login-success }

Below, we can see the filter details. I updated the Filter Name to a value that’s easy to identify. The Metric Namespace now has a value associated with the application name and the metric name reflects the number of login-success values.

 

We could now create an alarm to send a notification or perform some action (such as an Amazon EC2 scaling decision), based on this information being received via CloudWatch Logs.

With these values, we would receive an alert each time there were more than 50 successful logins within a five-minute period.

Laravel logging

Monolog is used as the logging solution for a number of PHP applications and frameworks, including, the popular Laravel PHP framework. In this example, we’ll show the use of Monolog with CloudWatch Logs within Laravel. Our first step is to find out the current log settings for our Laravel application. If you open config/app.php within your application root, you see various log settings. By default, Laravel is set to log to a single log file using the baseline log level of debug.

Next, we add the AWS SDK for PHP as a service provider within Laravel using instructions and examples from here.

You also want to add the Monolog library for CloudWatch Logs to the composer.json file for inclusion in the application, as shown.

You now need to extend the current Laravel Monolog configuration with your custom configuration. You can find additional information about this step on the Laravel Error and Logging page. The following is an example of this addition to the bootstrap/app.php file.

use Maxbanton\Cwh\Handler\CloudWatch;

$app->configureMonologUsing( function($monolog) {

    $cwClient = App::make('aws')->createClient('CloudWatchLogs');
    $cwGroupName = env('AWS_CWL_GROUP', 'laravel-app-logs');
    $cwStreamNameApp = env('AWS_CWL_APP', 'laravel-app-name');
    $cwTagName = env('AWS_CWL_TAG_NAME', 'application');
    $cwTagValue = env('AWS_CWL_TAG_VALUE', 'laravel-testapp01');
    $cwRetentionDays = 90;
    $cwHandlerApp = new CloudWatch($cwClient, $cwGroupName, $cwStreamNameApp, $cwRetentionDays, 10000, [ $cwTagName => $cwTagValue ] );

    $monolog->pushHandler($cwHandlerApp);
});

For testing purposes, we add a logging call to a test route in routes/web.php.

Route::get('/test', function () {
    Log::warning('Clicking on test link!!!');
    return view('test');
});

When the test route is invoked, the logs now show in CloudWatch Logs.

Conclusion

In our examples, we’ve shown how to use PHP Monolog to log to a local file, syslog, and CloudWatch Logs. We have also demonstrated the integration of Monolog with CloudWatch Logs within a popular PHP application framework. Finally, we’ve shown how to create CloudWatch Logs metric filters and apply those to CloudWatch Alarms that make the data from the logs actionable with notifications, as well as scaling decisions. CloudWatch Logs provides a central logging capability for your PHP applications and, combined with Monolog, ensures the availability of the library for use within established projects and custom engagements.

Using AWS CloudTrail in PHP – Part 2

by Jeremy Lindblom | on | in PHP | Permalink | Comments |  Share

This is part 2 of Using AWS CloudTrail in PHP. Part 1 demonstrated the basics of how to work with the CloudTrail service, including how to create a trail and turn logging on and off. Today, I want to show you how to read your log files and iterate over individual log records using the AWS SDK for PHP.

AWS CloudTrail log files

CloudTrail creates JSON-formatted log files containing your AWS API call history and stores them in the Amazon S3 bucket you choose. There is no API provided by CloudTrail for reading your log files, because the log files are stored in Amazon S3. Therefore, you can use the Amazon S3 client provided by the SDK to download and read your logs.

Your log files are stored in a predictable path within your bucket based on the account ID, region, and timestamp of the API calls. Each log file contains JSON-formatted data about the API call events, including the service, operation, region, time, user agent, and request and response data. You can see a full specification of the log record data on the CloudTrail Event Reference page of the CloudTrail documentation.

Log reading tools in the SDK

Even though it is a straightforward process to get your log files from Amazon S3, the SDK provides an easier way to do it from your PHP code. As of version 2.4.12 of the SDK, you can use the LogFileIterator, LogFileReader, and LogRecordIterator classes in the AwsCloudTrail namespace to read the log files generated by your trail.

  • LogFileIterator class – Allows you to iterate over the log files generated by a trail, and can be limited by a date range. Each item yielded by the iterator contains the bucket name and object key of the log file.
  • LogFileReader class – Allows you to read the log records of a log file identified by its bucket and key.
  • LogRecordIterator class – Allows you to iterate over log records from one or more log files, and uses the other two classes.

These classes add some extra conveniences over performing the Amazon S3 operations yourself, including:

  1. Automatically determining the paths to the log files based on your criteria.
  2. The ability to fetch log files or records from a specific date range.
  3. Automatically uncompressing the log files.
  4. Extracting the log records into useful data structures.

Instantiating the LogRecordIterator

You can instantiate the LogRecordIterator using one of the three provided factory methods. Which one you choose is determined by what data is available to your application.

  • LogRecordIterator::forTrail() – Use this if the name of the bucket containing your logs is not known.
  • LogRecordIterator::forBucket() – Use this if the bucket name is known.
  • LogRecordIterator::forFile() – Use this if retrieving records from a single file. The bucket name and object key are required.

If you already know what bucket contains your log files, then you can use the forBucket() method, which requires an instance of the Amazon S3 client, the bucket name, and an optional array of options.

use AwsCloudTrailLogRecordIterator;

$records = LogRecordIterator::forBucket($s3Client, 'YOUR_BUCKET_NAME', array(
    'start_date' => '-1 day',
    'log_region' => 'us-east-1',
));

Iterate over the LogRecordIterator instance allows you to get each log record one-by-one.

foreach ($records as $record) {
    // Print the operation, service name, and timestamp of the API call
    printf(
        "Called the %s operation on %s at %s.n",
        $record['eventName'],
        $record['eventSource'],
        $record['eventTime']
    );
}

NOTE: Each record is yielded as a Guzzle Collection object, which means it behaves like an array, but returns null for non-existent keys instead triggering an error. It also has methods like getPath() and getAll() that can be useful when working with the log record data.

A complete example

Let’s say that you want to look at all of your log records generated by the Amazon EC2 service during a specific week, and count how many times each Amazon EC2 operation was used. We’ll assume that the bucket name is not known, and that the trail was created via the AWS Management Console.

If you don’t know the name of the bucket, but you do know the name of the trail, then you can use the forTrail() factory method to instantiate the iterator. This method will use the CloudTrail client and the trail name to discover what bucket the trail uses for publishing log files. Trails created via the AWS Management Console are named "Default", so if you omit trail_name from the options array, "Default" will be used as the trail_name automatically.

$records = LogRecordIterator::forTrail($s3Client, $cloudTrailClient, array(
    'start_date' => '2013-12-08T00:00Z',
    'end_date'   => '2013-12-14T23:59Z',
));

The preceding code will give you an iterator that will yield all the log records for the week of December 8, 2013. To filter by the service, we can decorate the LogRecordIterator with an instance of PHP’s very own CallbackFilterIterator class.

$records = new CallbackFilterIterator($records, function ($record) {
    return (strpos($record['eventSource'], 'ec2') !== false);
});

NOTE: CallbackFilterIterator is available only in PHP 5.4+. However, Guzzle provides a similar class (GuzzleIteratorFilterIterator) for applications running on PHP 5.3.

At this point, it is trivial to count up the operations.

$opCounts = array();
foreach ($records as $record) {
    if (isset($opCounts[$record['eventName']])) {
        $opCounts[$record['eventName']]++;
    } else {
        $opCounts[$record['eventName']] = 1;
    }
}

print_r($opCounts);

There’s a Part 3, too

In the final part of Using AWS CloudTrail in PHP, I’ll show you how to set up CloudTrail to notify you of new log files via Amazon SNS. Then I’ll use the log reading tools from today’s post, combined with the SNS Message Validator class from the SDK, to show you how to read log files as soon as they are published.

Using AWS CloudTrail in PHP – Part 1

by Jeremy Lindblom | on | in PHP | Permalink | Comments |  Share

AWS CloudTrail is a new service that was announced at AWS re:Invent 2013.

CloudTrail provides a history of AWS API calls for your account, delivered as log files to one of your Amazon S3 buckets. The AWS API call history includes API calls made via the AWS Management Console, AWS SDKs, command line interface, and higher-level AWS services like AWS CloudFormation. Using CloudTrail can help you with security analysis, resource change tracking, and compliance auditing.

Today, I want to show you how to create a trail and start logging API calls using the AWS SDK for PHP. The CloudTrail client is available as of version 2.4.10 of the SDK.

Creating a trail for logging

The easiest way to create a trail is through the AWS Management Console (see Creating and Updating Your Trail), but if you need to create a trail through your PHP code (e.g., automation), you can use the SDK.

Setting up the log file destination

CloudTrail creates JSON-formatted log files containing your AWS API call history and stores them in the Amazon S3 bucket you choose. Before you set up your trail, you must first set up an Amazon S3 bucket with an appropriate bucket policy.

First, create an Amazon S3 client object (e.g., $s3Client).

Creating the Amazon S3 bucket

Use the Amazon S3 client to create a bucket. (Remember, bucket names must be globally unique.)

$bucket = 'YOUR_BUCKET_NAME';

$s3Client->createBucket(array(
    'Bucket' => $bucket
));

$s3Client->waitUntilBucketExists(array(
    'Bucket' => $bucket
));

Creating the bucket policy

Once the bucket is available, you need to create a bucket policy. This policy should grant the the CloudTrail service the access it needs to upload log files into your bucket. The CloudTrail documentation has an example of a bucket policy that we will use in the next code example. You will need to substitute a few of your own values into the example policy including:

  • Bucket Name: The name of the Amazon S3 bucket where your log files should be delivered.
  • Account Number: This is your AWS account ID, which is the 12-digit number found on the Account Identifiers section of the AWS Security Credentials page.
  • Log File Prefix: An optional key prefix you specify when you create a trail that is prepended to the object keys of your log files.

The following code prepares the policy document and applies the policy to the bucket.

$prefix = 'YOUR_LOG_FILE_PREFIX';
$account = 'YOUR_AWS_ACCOUNT_ID';
$policy = <<<POLICY
"Version": "2012-10-17",
"Statement": [
  {
    "Sid": "AWSCloudTrailAclCheck20131101",
    "Effect": "Allow",
    "Principal": {
      "AWS":[
        "arn:aws:iam::086441151436:root",
        "arn:aws:iam::113285607260:root"
      ]
    },
    "Action": "s3:GetBucketAcl",
    "Resource": "arn:aws:s3:::{$bucket}"
  },
  {
    "Sid": "AWSCloudTrailWrite20131101",
    "Effect": "Allow",
    "Principal": {
      "AWS": [
        "arn:aws:iam::086441151436:root",
        "arn:aws:iam::113285607260:root"
      ]
    },
    "Action": "s3:PutObject",
    "Resource": "arn:aws:s3:::{$bucket}/{$prefix}/AWSLogs/{$account}/*",
    "Condition": {
      "StringEquals": {
        "s3:x-amz-acl": "bucket-owner-full-control"
      }
    }
  }
]
POLICY;

$s3Client->putBucketPolicy(array(
    'Bucket' => $bucket,
    'Policy' => $policy,
));

Creating the trail

Now that the bucket has been set up, you can create a trail. Instantiate a CloudTrail client object, then use the createTrail() method of the client to create the trail.

use AwsCloudTrailCloudTrailClient;

$cloudTrailClient = CloudTrailClient::factory(array(
    'key'    => 'YOUR_AWS_ACCESS_KEY_ID',
    'secret' => 'YOUR_AWS_SECRET_KEY',
    'region' => 'us-east-1', // or us-west-2
));

$trailName = 'YOUR_TRAIL_NAME';
$cloudTrailClient->createTrail(array(
    'Name'         => $trailName,
    'S3BucketName' => $bucket,
));

NOTE: Currently, the CloudTrail service only allows for 1 trail at a time.

Start logging

After creating a trail, you can use the SDK to turn on logging via the startLogging() method.

$cloudTrailClient->startLogging(array(
    'Name' => $trailName
));

Your log files are published to your bucket approximately every 5 minutes and contain JSON-formatted data about your AWS API calls. Log files written to your bucket will persist forever by default. However, you can alter your bucket’s lifecycle rules to automatically delete files after a certain retention period or archive them to Amazon Glacier.

Turning it off

If you want to turn off logging, you can use the stopLogging() method.

$cloudTrailClient->stopLogging(array(
    'Name' => $trailName
));

Disabling logging does not delete your trail or log files. You can resume logging by calling the startLogging() method.

In some cases (e.g., during testing) you may want to remove your trail and log files completely. You can delete your trail and bucket using the SDK as well.

Deleting the trail

To delete a trail, use the deleteTrail() method.

$cloudTrailClient->deleteTrail(array(
    'Name' => $trailName
));

Deleting your log files and bucket

To delete the log files and your bucket, you can use the Amazon S3 client.

// Delete all the files in the bucket
$s3Client->clearBucket($bucket);

// Delete the bucket
$s3Client->deleteBucket(array(
    'Bucket' => $bucket
));

Look for Part 2

In the next part of Using AWS CloudTrail in PHP, I’ll show you how you can read your log files and iterate over individual log records using the SDK.

In the meantime, check out the AWS CloudTrail User Guide to learn more about the service.