Tag: cloudwatch


Writing Custom Metrics to Amazon CloudWatch Using the AWS SDK for Java

by Sascha Moellering | on | in Java | Permalink | Comments |  Share

Metrics measure the performance of your system. Several AWS services provide free metrics, such as the CPU usage of an Amazon EC2 instance. You can create Amazon CloudWatch alarms based on metrics and send Amazon SNS messages when the alarm state changes. You can use this mechanism to implement elastic scaling if the message is sent to an Auto Scaling group to change the desired capacity of the group. For many workloads, metrics like CPU usage are sufficient. However, from time to time, workloads have specific requirements and need a more complex metric to scale efficiently. It’s possible to publish your own metrics to CloudWatch, known as custom metrics, by using the AWS CLI, an API, or the CloudWatch collectd plugin. In this blog post, we’ll show you a more complex example of using the capabilities of the AWS SDK for Java to implement a framework integration to publish framework-related custom metrics to CloudWatch.

Integrating Vert.x and Amazon CloudWatch

Vert.x is an event-driven, reactive, nonblocking, and polyglot framework to implement microservices. It runs on the Java virtual machine (JVM) by using the low-level IO library Netty. You can write applications in Java, JavaScript, Groovy, Ruby, and Ceylon. The framework offers a simple and scalable actor-like concurrency model: Vert.x calls handlers by using a thread known as an event loop. To use this model, you have to write code known as verticles. Those verticles share certain similarities with actors in the Actor Model, and to use them, you have to implement the `Verticle` interface.
The following example shows a basic verticle implementation.

public class SimpleVerticle extends AbstractVerticle {
      // Method is called when the verticle is deployed
      public void start() {
      }

      // Optional method, called when verticle is undeployed
      public void stop() {
      }
}

Verticles communicate with each other using a single event bus. Those messages are sent on the event bus to a specific address, and verticles can register to this address by using handlers. In our example, we use the default event bus address cloudwatch.metrics. Then we register this address to consume all messages and push this data into CloudWatch.

With only a few exceptions, none of the APIs in Vert.x block the calling thread. Similar to Node.js, Vert.x uses the reactor pattern. However, in contrast to Node.js, Vert.x uses several event loops. Unfortunately, not all APIs in the Java ecosystem are written asynchronously, for example, the JDBC API. Vert.x offers a possibility to run this, blocking APIs without blocking the event loop. These special verticles are called worker verticles. You don’t execute worker verticles by using the standard Vert.x event loops, but by using a dedicated thread from a worker pool. Basically, this means that worker verticles don’t block the event loop.

If you start writing low-latency applications, you can reach a certain point where internal metrics of frameworks are required for further optimization. By default, Vert.x doesn’t record any metrics, but offers a Service Provider Interface (SPI) that you can implement to get more information about the behavior of Vert.x internals. The interface that you have to implement is described in the API documentation.
Vert.x provides an in-depth look into the framework by offering metrics for the following:

  • Datagram/UDP
  • Vert.x event bus
  • HTTP client
  • HTTP server
  • TCP client
  • TCP server
  • Pools used by Vert.x, such as execute blocking or worker verticle

To receive metrics from Vert.x, for example, HTTP server metrics, you have to implement the `HttpServerMetrics` interface and the following method from the `VertxMetrics` interface :

HttpServerMetrics<?, ?, ?> createMetrics(HttpServer httpServer, SocketAddress address, HttpServerOptions serverOptions);

The following code snippet shows a typical implementation of `HttpServerMetrics`.

private final LongAdder processingTime = new LongAdder();
    private final LongAdder requestCount = new LongAdder();
    private final LongAdder requests = new LongAdder();
    private final SocketAddress localAddress;
    private final HttpServerMetricsSupplier httpServerMetricsSupplier;

    public  HttpServerMetricsImpl(SocketAddress localAddress, HttpServerMetricsSupplier httpServerMetricsSupplier) {
        this.localAddress = localAddress;
        this.httpServerMetricsSupplier = httpServerMetricsSupplier;
        httpServerMetricsSupplier.register(this);
    }

    @Override
    public void responseEnd(Long nanoStart, HttpServerResponse response) {
        long requestProcessingTime = System.nanoTime() - nanoStart;
        processingTime.add(requestProcessingTime);
        requestCount.increment();
        requests.decrement();
    }

In this example, the `responseEnd` method is called if an HTTP server response has ended. The processing time of the request is calculated, the number of requests is incremented, and the number of current requests is decremented. Now we have to send the data we collected to CloudWatch.
To collect metrics data and send it to CloudWatch, we need to implement the `MetricSupplier` interface and override the `collect()` method. Each metric value is represented by an object of type `CloudWatchDataPoint`. This data point class is a simple POJO containing the name of the metric, the value, the timestamp of collection, and a CloudWatch StandardUnit. The `StandardUnit` enumeration represents the unit of the data point in CloudWatch (e.g., Bytes). After collecting a list of data points, the `Sender` class pushes the data to CloudWatch. To connect to CloudWatch, the Sender class uses the AWS SDK for Java and the `DefaultAWSCredentialsProviderChain`. This enables you to use Vert.x-CloudWatch SPI on an EC2 instance, as well as on your local development workstation.

    public Sender(Vertx vertx, VertxCloudwatchOptions options, Context context) {
        this.vertx = vertx;

        // Configuring the CloudWatch client
        // AWS credentials provider chain that looks for credentials in this order:
        //      - Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (RECOMMENDED since they are recognized by all the AWS SDKs and CLI except for .NET), or AWS_ACCESS_KEY and AWS_SECRET_KEY (only recognized by the SDK for Java)
        //      - Java System Properties - aws.accessKeyId and aws.secretKey
        //      - Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI
        //      - Instance profile credentials delivered through the Amazon EC2 metadata service
        this.cloudWatchClient = initCloudWatchClient(options.getCloudwatchRegion());
        this.namespace = options.getNamespace();
        this.instanceId = options.getInstanceId();

        batchSize = options.getBatchSize();
        batchDelay = NANOSECONDS.convert(options.getBatchDelay(), SECONDS);
        queue = new ArrayList<>(batchSize);
        sendTime = System.nanoTime();

        context.runOnContext(aVoid -> timerId = vertx.setPeriodic(MILLISECONDS.convert(batchDelay, NANOSECONDS), this::flushIfIdle));
    }

    ...

    private void send(List<CloudWatchDataPoint> dataPoints) {
        List<MetricDatum> cwData = toCloudwatchData(dataPoints);
        PutMetricDataRequest metricDataRequest = new PutMetricDataRequest();
        metricDataRequest.setMetricData(cwData);
        metricDataRequest.setNamespace(this.namespace);
        Future future = cloudWatchClient.putMetricDataAsync(metricDataRequest);
        sendTime = System.nanoTime();

        try {
            future.get();
        } catch (Exception exc) {
            LOG.error(exc);
        }
    }

    private List<MetricDatum> toCloudwatchData(List<CloudWatchDataPoint> dataPoints) {
        List<MetricDatum> metrics = new ArrayList<>();

        dataPoints.forEach(metric -> {

            MetricDatum point = new MetricDatum();

            point.setTimestamp(new Date(metric.getTimestamp()));
            point.setValue((double) metric.getValue());
            point.setMetricName(metric.getName());
            point.setUnit(metric.getStandardUnit());
            List<Dimension> dimensionList = new ArrayList<>();
            dimensionList.add(new Dimension().withName("InstanceId").withValue(this.instanceId));

            point.setDimensions(dimensionList);
            metrics.add(point);
        });

        return metrics;
    }

To use the CloudWatch Vert.x SPI implementation, we have to set the necessary metrics options. In our case, we want to use the CloudWatch namespace `Vertx/CloudWatch`. Let’s assume that the application runs on an EC2 instance. In this case, the CloudWatch SPI automatically detects the region that the EC2 instance is running in and the instance ID. This information is determined by using the EC2MetadataUtils-class.
After setting the metrics options, we initiate a Vert.x instance and create a simple HTTP server on port 8080 that returns “Hello Vert.x!” in plain text. The SPI automatically detects that an HTTP server is created and collects HTTP server-related metrics such as the number of HTTP connections, the number of bytes sent, and a set of other metrics.
In addition to that, we want to send the consumed memory of the JVM to CloudWatch. This custom metric isn’t collected by the SPI, so we have to calculate the consumed memory by using the Runtime-class. A timer sends this data as a JSON message every five seconds over the event bus to the CloudWatch SPI. The SPI collects the data and sends it to CloudWatch.

    VertxOptions options = new VertxOptions().setMetricsOptions(
                new VertxCloudwatchOptions()
                        .setEnabled(true)
                        .setMetricsBridgeEnabled(true)
                        .setBatchSize(10)
                        .setBatchDelay(30)
                        .setNamespace("Vertx/Cloudwatch"));
    vertx = Vertx.vertx(options);

    // Creating HTTP server for metrics
    HttpServer server = vertx.createHttpServer();

    server.requestHandler(request -> {

        // This handler is called for each request that arrives on the server
        HttpServerResponse response = request.response();
        response.putHeader("content-type", "text/plain");

        // Write to the response and end it
        response.end("Hello Vert.x!");
    });

    vertx.setPeriodic(5000, id -> {
        long usedMem = this.getUsedMemory();
        JsonObject message = new JsonObject()
                .put("metricName", "JVMMemory")
                .put("unit", StandardUnit.Megabytes.toString())
                .put("value", usedMem);

        vertx.eventBus().publish("cloudwatch.metrics", message);
    });

    server.listen(8080);

The following figure shows metrics such as the number of HTTP connections, the number of requests, the amount of bytes sent, and the consumed memory displayed as a graph in CloudWatch.

Vert.x metrics

Note that a custom metric is defined as the unique combination of metric name and dimensions associated with the metric. Custom metrics are priced based on monthly usage per metric. See CloudWatch pricing for details.

Summary

In this blog post we created a Vert.x SPI implementation to write framework metrics to CloudWatch. We used the capabilities of the AWS SDK for Java not only for the communication with CloudWatch, but also to get insights about the instance and the region using EC2 metadata. We hope we’ve given you ideas for creating your own applications and framework integrations by using the AWS SDK for Java. Feel free to share your ideas and thoughts in the comments below!

 

PHP application logging with Amazon CloudWatch Logs and Monolog

by Joseph Fontes | on | in PHP | Permalink | Comments |  Share

Logging and information debugging can be approached from a multitude of different angles. Whether you use an application framework or coding from scratch it’s always comforting to have familiar components and tools across different projects. In our examples today, I am going to enable Amazon CloudWatch Logs logging with a PHP application. To accomplish this, I wanted to use an existing solution that is both already popular and well used, and that is standards compliant. For these reasons, we are going to use the open source log library, PHP Monolog (https://github.com/Seldaek/monolog).

PHP Monolog

For those who work with a new PHP application, framework, or service, one of the technology choices that appears more frequently across solutions is the use of Monolog for application logging. PHP Monolog is a standards-compliant PHP library that enables developers to send logs to various destination types including, databases, files, sockets, and different services. Although PHP Monolog predates the standards for PHP logging defined in PSR-3, it does implement the PSR-3 interface and standards. This makes Monolog compliant with the common interface for logging libraries. Using Monolog with CloudWatch Logs creates a PSR-3 compatible logging solution. Monolog is available for use with a number of different applications and frameworks such as Laravel, Symfony, CakePHP, and many others. Our example today is about using PHP Monolog to send information to CloudWatch Logs for the purpose of application logging and to build a structure and process that enables the use of our application data with CloudWatch alarms and notifications. This enables us to use logs from our application for cross-service actions such as with Amazon EC2 Auto Scaling decisions.

Amazon CloudWatch Logs

As a customer-driven organization, AWS is constantly building and releasing significant features and services requested by AWS customers and partners. One of those services that we highlight today is Amazon CloudWatch Logs. CloudWatch Logs enables you to store log file information from applications, operating systems and instances, AWS services, and various other sources. An earlier blog post highlighted the use of CloudWatch Logs with various programming examples.

Notice in the blog post that there is a PHP example that uses CloudWatch Logs to store an entry from an application. You can use this example and extend it as a standalone solution to provide logging to CloudWatch Logs from within your application. With our examples, we’ll enhance this opportunity by leveraging PHP Monolog.

Implementing Monolog

To begin using Monolog, we install the necessary libraries with the use of Composer (https://getcomposer.org/). The instructions below install the AWS SDK for PHP, PHP Monolog, and an add-on to Monolog that enables logging to CloudWatch Logs.

curl -sS https://getcomposer.org/installer | php
php composer.phar require aws/aws-sdk-php
php composer.phar require monolog/monolog
php composer.phar require maxbanton/cwh:^1.0

Alternatively, you can copy the following entry to the composer.json file and install it via the php composer.phar install command.

{
    "minimum-stability": "stable",
    "require": {
        "aws/aws-sdk-php": "^3.24",
        "aws/aws-php-sns-message-validator": "^1.1",
        "monolog/monolog": "^1.21",
        "maxbanton/cwh": "^1.0"
    }
}

Local logging

Now that PHP Monolog is available for use, we can test the implementation. We start with an example of logging to a single file.

require "vendor/autoload.php";

use Monolog\Logger;
use Monolog\Formatter\LineFormatter;
use Monolog\Handler\StreamHandler;

$logFile = "testapp_local.log";

$logger = new Logger('TestApp01');
$formatter = new LineFormatter(null, null, false, true);
$infoHandler = new StreamHandler(__DIR__."/".$logFile, Logger::INFO);
$infoHandler->setFormatter($formatter);
$logger->pushHandler($infoHandler);
$logger->info('Initial test of application logging.');

In the previous example, we start by requiring the composer libraries we installed earlier. The new Logger line sets the channel name as “TestApp01”. The next line creates a new LineFormatter that removes brackets around unused log items. The next line establishes the destination as the file name we identified, testapp_local.log, and associates that with the INFO log level. Next, we apply the format to our stream handler. Then we add the stream handler with the updated format to the handler list. Finally, a new message is logged with the log level of INFO. For information about log levels and different handlers, see the Monolog GitHub page and IETF RFC 5424 and PSR-3.

We can now view the contents of the log file to ensure functionality:

Syslog logging

Now that we are able to write a simple log entry to a local file, our next example uses the system Syslog to log events.

$logger = new Logger($appName);

$localFormatter = new LineFormatter(null, null, false, true);
$syslogFormatter = new LineFormatter("%channel%: %level_name%: %message% %context% %extra%",null,false,true);

$infoHandler = new StreamHandler(__DIR__."/".$logFile, Logger::INFO);
$infoHandler->setFormatter($localFormatter);

$warnHandler = new SyslogHandler($appName, $facility, Logger::WARNING);
$warnHandler->setFormatter($syslogFormatter);

$logger->pushHandler($warnHandler);
$logger->pushHandler($infoHandler);

$logger->info('Test of PHP application logging.');
$logger->warn('Test of the warning system logging.');

Here we can see that the format of the syslog messages has been changed with the value, $syslogFormatter. Because syslog provides a date/time with each log entry, we don’t need to include these values in our log text. The syslog facility is set to local0 with all WARNING messages sent to syslog with the INFO level messages and WARNING level messages logged to our local file. You can find additional information about Syslog facilities and log levels on the Syslog Wikipedia page.

Logging to CloudWatch Logs

Now that you’ve seen the basic use of Monolog, let’s send some logs over to CloudWatch Logs. We can use the Amazon Web Services CloudWatch Logs Handler for Monolog library to integrate Monolog with CloudWatch Logs. In our example, an authentication application produces log information.

use Aws\CloudWatchLogs\CloudWatchLogsClient;
use Maxbanton\Cwh\Handler\CloudWatch;
use Monolog\Logger;
use Monolog\Formatter\LineFormatter;
use Monolog\Handler\StreamHandler;
use Monolog\Handler\SyslogHandler;

$logFile = "testapp_local.log";
$appName = "TestApp01";
$facility = "local0";

// Get instance ID:
$url = "http://169.254.169.254/latest/meta-data/instance-id";
$instanceId = file_get_contents($url);

$cwClient = new CloudWatchLogsClient($awsCredentials);
// Log group name, will be created if none
$cwGroupName = 'php-app-logs';
// Log stream name, will be created if none
$cwStreamNameInstance = $instanceId;
// Instance ID as log stream name
$cwStreamNameApp = "TestAuthenticationApp";
// Days to keep logs, 14 by default
$cwRetentionDays = 90;

$cwHandlerInstanceNotice = new CloudWatch($cwClient, $cwGroupName, $cwStreamNameInstance, $cwRetentionDays, 10000, [ 'application' => 'php-testapp01' ],Logger::NOTICE);
$cwHandlerInstanceError = new CloudWatch($cwClient, $cwGroupName, $cwStreamNameInstance, $cwRetentionDays, 10000, [ 'application' => 'php-testapp01' ],Logger::ERROR);
$cwHandlerAppNotice = new CloudWatch($cwClient, $cwGroupName, $cwStreamNameApp, $cwRetentionDays, 10000, [ 'application' => 'php-testapp01' ],Logger::NOTICE);

$logger = new Logger('PHP Logging');

$formatter = new LineFormatter(null, null, false, true);
$syslogFormatter = new LineFormatter("%channel%: %level_name%: %message% %context% %extra%",null,false,true);
$infoHandler = new StreamHandler(__DIR__."/".$logFile, Logger::INFO);
$infoHandler->setFormatter($formatter);

$warnHandler = new SyslogHandler($appName, $facility, Logger::WARNING);
$warnHandler->setFormatter($syslogFormatter);

$cwHandlerInstanceNotice->setFormatter($formatter);
$cwHandlerInstanceError->setFormatter($formatter);
$cwHandlerAppNotice->setFormatter($formatter);

$logger->pushHandler($warnHandler);
$logger->pushHandler($infoHandler);
$logger->pushHandler($cwHandlerInstanceNotice);
$logger->pushHandler($cwHandlerInstanceError);
$logger->pushHandler($cwHandlerAppNotice);

$logger->info('Initial test of application logging.');
$logger->warn('Test of the warning system logging.');
$logger->notice('Application Auth Event: ',[ 'function'=>'login-action','result'=>'login-success' ]);
$logger->notice('Application Auth Event: ',[ 'function'=>'login-action','result'=>'login-failure' ]);
$logger->error('Application ERROR: System Error');

In this example, application authentication events are passed as a PHP array and presented in CloudWatch Logs as JSON. The events with a result of login-success and login-failure are sent to both the log stream associated with the instance ID and to the log stream associated with the application name.

 

Using these different stream locations, we can create metrics and alarms at either a per-instance level or per-application level. Let’s assume that we want to create a metric for total number of users logged into our application over the past five minutes. Select your event group and then choose Create Metric Filter.

On the next page, we can create our filter and test in the same window. For the filter data, we use the JSON string from the log entry. Enter the following string to extract all the successful logins.

{ $.result = login-success }

Below, we can see the filter details. I updated the Filter Name to a value that’s easy to identify. The Metric Namespace now has a value associated with the application name and the metric name reflects the number of login-success values.

 

We could now create an alarm to send a notification or perform some action (such as an Amazon EC2 scaling decision), based on this information being received via CloudWatch Logs.

With these values, we would receive an alert each time there were more than 50 successful logins within a five-minute period.

Laravel logging

Monolog is used as the logging solution for a number of PHP applications and frameworks, including, the popular Laravel PHP framework. In this example, we’ll show the use of Monolog with CloudWatch Logs within Laravel. Our first step is to find out the current log settings for our Laravel application. If you open config/app.php within your application root, you see various log settings. By default, Laravel is set to log to a single log file using the baseline log level of debug.

Next, we add the AWS SDK for PHP as a service provider within Laravel using instructions and examples from here.

You also want to add the Monolog library for CloudWatch Logs to the composer.json file for inclusion in the application, as shown.

You now need to extend the current Laravel Monolog configuration with your custom configuration. You can find additional information about this step on the Laravel Error and Logging page. The following is an example of this addition to the bootstrap/app.php file.

use Maxbanton\Cwh\Handler\CloudWatch;

$app->configureMonologUsing( function($monolog) {

    $cwClient = App::make('aws')->createClient('CloudWatchLogs');
    $cwGroupName = env('AWS_CWL_GROUP', 'laravel-app-logs');
    $cwStreamNameApp = env('AWS_CWL_APP', 'laravel-app-name');
    $cwTagName = env('AWS_CWL_TAG_NAME', 'application');
    $cwTagValue = env('AWS_CWL_TAG_VALUE', 'laravel-testapp01');
    $cwRetentionDays = 90;
    $cwHandlerApp = new CloudWatch($cwClient, $cwGroupName, $cwStreamNameApp, $cwRetentionDays, 10000, [ $cwTagName => $cwTagValue ] );

    $monolog->pushHandler($cwHandlerApp);
});

For testing purposes, we add a logging call to a test route in routes/web.php.

Route::get('/test', function () {
    Log::warning('Clicking on test link!!!');
    return view('test');
});

When the test route is invoked, the logs now show in CloudWatch Logs.

Conclusion

In our examples, we’ve shown how to use PHP Monolog to log to a local file, syslog, and CloudWatch Logs. We have also demonstrated the integration of Monolog with CloudWatch Logs within a popular PHP application framework. Finally, we’ve shown how to create CloudWatch Logs metric filters and apply those to CloudWatch Alarms that make the data from the logs actionable with notifications, as well as scaling decisions. CloudWatch Logs provides a central logging capability for your PHP applications and, combined with Monolog, ensures the availability of the library for use within established projects and custom engagements.