Collect, parse, transform, and stream Windows events, logs, and metrics using Amazon Kinesis Agent for Microsoft Windows

A complete data pipeline that includes Amazon Kinesis Agent for Microsoft Windows (KA4W) can help you analyze and monitor the performance, security, and availability of Windows-based services. You can build near-real-time dashboards and alarms for your Windows services. You can also use visualization and business intelligence tools such as Amazon Athena, Kibana, Amazon QuickSight, and Amazon CloudWatch to rapidly locate, diagnose, and resolve operational and security issues.

KA4W eliminates cloud-based log processing by parsing and transforming logs into standard formats such as JSON. These formats can then be immediately consumed by the visualization and business intelligence tools in the data pipeline.

Here are a few words from one of our customers summarizing their experience with KA4W:

“Autodesk builds software tools and services that help our customers design and make anything. These tools and services must be reliable so they can get their job done. A fundamental part of any trusted cloud connected software is observability and this starts with collecting log data. The new Amazon Kinesis Agent for Microsoft Windows simplifies workflows for streaming logs by eliminating the need for complicated interconnected systems and tools. The agent is easy to set-up, configure and update. The Amazon Kinesis Agent for Microsoft Windows significantly reduces the complexity of log collection and management”

– Ben Cochran, sr director, Cloud Engineering, Autodesk

In this post, we review how the new Kinesis Agent for Windows enables streaming analytics use cases related to Windows applications, servers, and workstations. We also show you how to get started with the new agent. By using KA4W to push real-time data into Amazon Kinesis services, you can solve a wide range of operational issues, including the following:

Monitoring of Dynamic Host Configuration Protocol (DHCP) servers for identifying IP lease refusals in case of exhausted scopes
Monitoring of Microsoft Exchange servers for identifying top email senders, mail-storm situations, and heavy load conditions
Monitoring of web-based application and Internet Information Services (IIS) logs for performance, availability, and security issues
Monitoring of domain controllers for Active Directory and security issues
Enabling security intelligence platforms to ingest log files for forensics and penetration testing

Overview of Kinesis Agent for Windows

Amazon Kinesis Agent for Microsoft Windows (KA4W) is a configurable and extensible agent. It runs on Windows laptops, desktop computers, and servers, either on-premises or in the AWS Cloud. KA4W efficiently and reliably gathers, parses, transforms, and streams logs, events, and metrics to various AWS services, including Kinesis Data Streams, Kinesis Data Firehose, CloudWatch, and CloudWatch Logs.

Kinesis Agent for Windows provides built-in parsers that simplify the processing of logs from common Windows services such as Exchange, Active Directory, DHCP, Microsoft SharePoint, and Windows security logs. It solves many technical and operational challenges for streaming logs, events, and metrics to AWS services, including the following:

Handling large volume of log files, processing and transforming logs and events from many different sources and formats in near-real time
Handling different kinds of log rotation approaches and accessing log files even when those logs files are locked by log writers
Reducing data transfer and storage costs by filtering unnecessary data before delivering to AWS services
Adding context to the collected data that enables precise analysis and rapid resolution of operational and security issues
Providing data about the health of the agent itself, which confirms the accuracy and completeness of the data collected and streamed

The following diagram illustrates some of the ways you can build custom, real-time data pipelines by using Kinesis Agent for Windows and stream-processing frameworks.

Amazon Kinesis Agent for Microsoft Windows includes an array of plugins. By configuring these plugins, you can customize KA4W to satisfy most requirements for the collection, transformation, and near-real-time delivery of logs, events, and metrics. You can even create your own plugin if you have custom requirements. Plugins are categorized by sources, pipes, and sinks.

Sources are the plugins that gather various Windows logs, events, and metrics. KA4W comes with multiple built-in source plugins, including the following:

DirectorySource
ExchangeLogSource
3SVCLogSource
UlsSource (SharePoint)
WindowsEventLogSource
WindowsETWEventSource
WindowsPerformanceCounterSource

For more information about these sources, see Source Declarations in the Kinesis Agent for Windows User Guide.

Pipes connect sources and sinks. You can use pipes to filter unnecessary data to improve data transfer and reduce storage and API usage cost. Filters improve data quality and provide an effective way to restrict the amount of data for analysis. For details on configuring pipes, see Pipe Declarations in the Kinesis Agent for Windows User Guide.

Sinks are the plugins that stream logs, event, and metrics data to different AWS services. Kinesis Agent for Windows comes with multiple built-in sink plugins, such as KinesisStream, KinesisFirehose, CloudWatch, and CloudWatchLogs. Kinesis Agent for Windows needs to authenticate with AWS services to send data. For details on sinks and security configuration, see Sink Declarations in the Kinesis Agent for Windows User Guide.

Monitoring a web server example

Suppose that you have a web-based application and want to monitor the underlying web server. To do so, follow these steps to build the data pipeline that is required to move Windows events and web server logs to Kinesis Data Firehose and Amazon S3 for analytics:

Install Kinesis Agent for Windows
Create the streams
Set the permissions
Configure Kinesis Agent for Windows
Start Kinesis Agent for Windows
View the ingested logs

Install Kinesis Agent for Windows

Go to the Amazon Kinesis Agent for Microsoft Windows download page, and follow the instructions to download the agent. For example, you can run the following command in an elevated PowerShell command prompt:

Invoke-Expression ((New-Object System.Net.WebClient).DownloadString('https://s3-us-west-2.amazonaws.com/kinesis-agent-windows/downloads/InstallKinesisAgent.ps1'))

Create streams

Create two Kinesis Data Firehose delivery streams named EventLogStream and W3SVCLogStream. Configure both streams to deliver data to Amazon Simple Storage Service (Amazon S3). To expedite log delivery to Amazon S3, reduce the delivery stream buffer size to 1 MB and the buffer interval to 60 seconds. This results in more frequent writes to Amazon S3.

Set permissions

If your host is an Amazon EC2 instance, the security configuration is simpler, and you only need to grant your Amazon EC2 instance role PutRecordBatch permission to the streams created. A sample IAM (security) policy looks like the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "firehose:PutRecordBatch"
            ],
            "Resource": [
                "arn:aws:firehose:region:account-id:deliverystream/delivery-stream-name"
            ]
        }
    ]
}

Configure Kinesis Agent for Windows

For this example, replace the content of “C:\Program Files\Amazon\AWSKinesisTap\appsettings.json” with the following:

{
    "Sources": [
        {
            "Id": "ApplicationLog",
            "SourceType": "WindowsEventLogSource",
            "LogName": "Application"
        },
        {
            "Id": "SecurityLog",
            "SourceType": "WindowsEventLogSource",
            "LogName": "Security"
        },
        {
            "Id": "SystemLog",
            "SourceType": "WindowsEventLogSource",
            "LogName": "System"
        },
        {
            "Id": "W3SVCLog1",
            "SourceType": "W3SVCLogSource",
            "Directory": "C:\\inetpub\\logs\\LogFiles\\W3SVC1",
            "FileNameFilter": "*.log",
            "TimeZoneKind": "UTC"
        }
    ],
    "Sinks": [
        {
            "Id": "EventLogSink",
            "SinkType": "KinesisFirehose",
            "StreamName": "EventLogStream",
            "Format": "json"
        },
        {
            "Id": "W3SVCLogSink",
            "SinkType": "KinesisFirehose",
            "Region": "us-west-2",
            "StreamName": "W3SVCLogStream",
            "Format": "json"
        }
    ],
    "Pipes": [
        {
            "Id": "ApplicationLogToFirehose",
            "SourceRef": "ApplicationLog",
            "SinkRef": "EventLogSink"
        },
        {
            "Id": "SecurityLogToFirehose",
            "SourceRef": "SecurityLog",
            "SinkRef": "EventLogSink"
        },
        {
            "Id": "SystemLogToFirehose",
            "SourceRef": "SystemLog",
            "SinkRef": "EventLogSink"
        },
        {
            "Id": "W3SVCLog1ToKinesisStream",
            "SourceRef": "W3SVCLog1",
            "SinkRef": "W3SVCLogSink"
        }
    ]
}

In the preceding configuration file, there are four sources. The first three are for different Windows Event Logs, and the last one is for IIS (W3SVC) logs. Typically IIS writes logs to the C:\inetpub\logs\LogFiles\W3SVC1 directory.

There are two KinesisFirehose sinks: EventLogSink sends data to EventLogStream, and W3SVCLogSink sends data to W3SVCLogStream. For both sinks, you set the Format key-value pair to json. This instructs the agent to send logs in JSON-formatted data to Kinesis Data Firehose.

The first three pipes connect the first three sources to the EventLogSink, and the last pipe connects the last source to the W3SVCLogSink.

In this example, authentication and authorization are based on the instance profile that is associated with the Amazon EC2 instance that is running Kinesis Agent for Windows. There are other approaches to authentication and authorization. For more information, see “Sink Security Configuration” in Sink Declarations in the Kinesis Agent for Windows User Guide.

Start Kinesis Agent for Windows

Note: During the development of Kinesis Agent for Windows, the internal name was “AWSKinesisTap”. To maintain backward compatibility, we have maintained this terminology within the agent’s configuration.

There are a few ways to start the agent:

Start the agent from the services applet.
Start the agent from an elevated command prompt and run net start AWSKinesisTap.
Start the agent from an elevated PowerShell command prompt and run Start-Service -Name AWSKinesisTap.

If Kinesis Agent for Windows does not start, check the application event log. If the agent starts, you can find the logs in “C:\Program Data\Amazon\AWSKinesisTap\logs”.

If you run into difficulty, see Troubleshooting in the Kinesis Agent for Windows User Guide.

View the log data

Here is a quick and easy way to view the ingested data in Amazon S3:

Sign in to the AWS Management Console, and open the Amazon S3 console. Go to the S3 bucket that the Kinesis Data Firehose delivery streams are streaming to, and choose an object.
Choose the Select from
Under File format, choose JSON, and then choose Show file preview.

The following shows an example for a Windows Event Log:

The Amazon Kinesis Agent for Microsoft Windows User Guide contains a tutorial that demonstrates a powerful scenario using Amazon Athena to query the data in Amazon S3.

Cost

Amazon Kinesis Agent for Microsoft Windows is free to use. However, you pay for the AWS resources that interact with your complete data pipeline, such as Kinesis Data Streams, Kinesis Data Firehose, and Amazon S3.

Performance

Amazon Kinesis Agent for Microsoft Windows consumes a minimal amount of system resources. Memory and CPU usage can vary based on the amount of data streaming to AWS services and the virtual or physical hardware configuration of the machines hosting Kinesis Agent for Windows.

Summary

Amazon Kinesis Agent for Microsoft Windows streams logs, events, and metrics to AWS services. It is a key part of constructing an efficient, reliable, and cost-effective data pipeline for discovering, preventing, and resolving complex operational and security issues with Windows desktop and server machines. The example in this post demonstrates the simplicity of configuring a custom-tailored solution for gathering and streaming operational data from a web server host. Kinesis Agent for Windows allows you to choose the right AWS services for your scenarios to construct data pipelines and gain deeper insight into your operational challenges.

Going further

Kinesis Agent for Windows is very flexible and has many additional features. Here are a few suggestions for next steps:

Review the many configuration examples in the Kinesis Agent for Windows User Guide.
Learn about DirectorySource, a flexible source plugin that parses many types of text-based logs stored in the Windows file system. For details on supported parsers, see Source Declarations.
View the configuration file, which has variables support to allow the agent to get information from the environment variables and Amazon EC2 metadata.
Decorate the logs using TextDecoration and ObjectDecoration.
Configure Kinesis Agent for Windows to update itself and its configuration file. For more information, see Configuring Automatic Updates.

Additional resources

About the authors

Harvir Singh is a Software Development Manager for Amazon Kinesis Agent for Microsoft Windows.

Li Chen is a Senior Software Development Engineer for Amazon Kinesis Agent for Microsoft Windows.

Bonnie Feinberg is a Senior Software Development Engineer for Amazon Kinesis Agent for Microsoft Windows.

AWS Big Data Blog