Tracking website metrics in real-time can provide insight into who visits your website, where they come from, and what content they view. You can track time series data on visitor counts, page views, time spent on site, channels driving traffic, actions visitors take, and other custom metrics. This insight can help you react quickly to changing conditions.

To help customers more easily track website activity, Amazon Web Services offers the Real-Time Web Analytics with Kinesis Data Analytics solution, a reference implementation that automatically provisions the services necessary to track and visualize website clickstream data in real-time. This solution is designed to provide a framework for analyzing and visualizing metrics, allowing you to focus on adding new metrics rather than managing the underlying infrastructure.

This webpage provides best practices for implementing a website analytics solution, as well as an overview of the Real-Time Web Analytics with Kinesis Data Analytics solution.

Before you implement a real-time website analytics solution, determine the key metrics you want to track and what you want to do with those metrics. Consider solutions that provide the flexibility to easily add new metrics, are easy to deploy and use, and can scale to handle a large number of events. Keeping these general principles in mind, consider the following best practices for monitoring website usage on the AWS Cloud:

  • Choose a solution that integrates with your existing data warehouses so you can track end-to-end conversions, including signups and purchases.
  • Consider a solution that provides highly-scalable, reliable, and low-latency data processing and storage.
  • Implement a solution that provides visibility across all operating environments. The solution should integrate with both on-premises and AWS workloads.
  • Use a solution that complements your company’s existing processes and skill sets to ensure that you can manage and modify your solution to meet future requirements. This will reduce operational complexity for developers and operators.

AWS offers a solution that uses beacon web servers to log requests from a user’s web browser, Amazon Kinesis Data Firehose to capture website clickstream data, Amazon Kinesis Data Analytics to compute metrics in real-time, and Amazon Simple Storage Service (Amazon S3) and Amazon DynamoDB to durably store metric data. The solution also features a dashboard that visualizes your account activity in real-time. The diagram below presents the architecture you can deploy in minutes using the solution's implementation guide and accompanying AWS CloudFormation template.

  1. Web beacon servers log requests from a user’s web browser and send the data to a Kinesis Data Firehose delivery stream.
  2. The delivery stream archives the events in an Amazon S3 bucket and sends the data to a Kinesis Data Analytics application for processing.
  3. Once the data is processed, it is sent to Kinesis Data Streams. An AWS Lambda function reads data from the stream and sends the data in real-time to an Amazon DynamoDB table to be stored.
  4. The solution also creates an Amazon Cognito user pool, an Amazon S3 bucket, and real-time dashboard to securely read and display the account activity stored in the DynamoDB table.
Deploy Solution
Implementation Guide

What you'll accomplish:

Deploy Real-Time Web Analytics with Kinesis Data Analytics using AWS CloudFormation. The AWS CloudFormation template will automatically launch and configure the components necessary to track website usage in real-time.

Automatically deploy a real-time dashboard that displays web usage metrics. You can customize the dashboard to include any metrics.

Launch a VPC that follows AWS best practices for security and availability.

What you'll need before starting:

An AWS account: You will need an AWS account to begin provisioning resources. Sign up for AWS.

Skill level: This solution is intended for IT infrastructure architects, administrators, and DevOps professionals who have practical experience architecting on the AWS Cloud.

Real-time visualizations

Real-Time Web Analytics with Kinesis Data Analytics enables you to track website activity in real-time. Visualize web usage metrics including events per hour, visitor count, user agents, abnormal events, aggregate event count, referrers, and recent events. You can also customize this solution to include additional metrics.


Beacon server performance metrics

This solution provides an optional dashboard you can use to monitor the performance of your beacon web servers with Amazon CloudWatch. The dashboard displays custom operational metrics for the servers, including the number of healthy servers, the average processed network packets, aggregate requests, 5XX errors, and Amazon DynamoDB throughput capacity and throttling.


Q: What metrics are displayed on the dashboard?

The solution dashboard displays a default set of metrics including events per hour, visitor count, user agents, abnormal events, aggregate event count, referrers, and recent events.  You can also customize the dashboard to include custom metrics. For more information, see the implementation guide.

Q: Can I deploy the web beacon servers in my VPC?

Yes. This solution includes a template that enables you to launch the solution’s web beacon servers in your existing VPC. For more information, see the implementation guide.

Q: Can I deploy this solution in any AWS Region?

This solution uses the Kinesis Data Firehose and Kinesis Data Analytics services, which are currently available in specific AWS Regions only. Therefore, you must deploy this solution in an AWS Region that supports these services. However, once deployed, this solution monitors all regions for events. For more information, see AWS service availability by region.

Need more resources to get started with AWS? Visit the Getting Started Resource Center to find tutorials, projects and videos to get started with AWS.

Tell us what you think