How do I add Amazon EC2 metadata when pushing logs to Kinesis using Kinesis Agent?

Last updated: 2020-05-12

I'm trying to send logs from Amazon Elastic Compute Cloud (Amazon EC2) to Amazon Kinesis using Amazon Kinesis Agent. How do I append the Amazon EC2 metadata to each log line?

Short Description

You can append the EC2 metadata to each log line by doing the following:

1.    Install and set up Kinesis Agent on either a Linux or Windows platform.

2.    Update your configuration settings to display EC2 metadata.

3.    Verify that you have the required AWS Identity and Access Management (IAM) permissions.

Note: You need IAM permissions to retrieve EC2 metadata and to publish data to Amazon Kinesis Data Firehose.

Resolution

On a Linux platform

To use Kinesis Agent to send logs to Kinesis Data Firehose, perform the following steps:

1.    Download and install the agent.

2.    Create an IAM role with the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "firehose:PutRecord",
        "firehose:PutRecordBatch"
      ],
      "Resource": [
        "<KFH ARN>"
      ]
    },
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:DescribeInstanceAttribute",
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeInstanceStatus"
      ],
      "Resource": "*"
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": "cloudwatch:PutMetricData",
      "Resource": "*"
    }
  ]
}

3.    Attach the newly created IAM role to the EC2 instances where you installed Kinesis Agent. For more information about assigning an existing IAM role, see How do I assign an existing IAM role to an EC2 instance?

4.    Edit the /etc/aws-kinesis/agent.json file:

{
  "cloudwatch.emitMetrics": true,
  "kinesis.endpoint": "",
  "firehose.endpoint": "firehose.us-east-1.amazonaws.com",
  
  "flows": [
    {
      "filePattern": "/tmp/app.log*",
      "deliveryStream": "yourdeliverystream",
      "partitionKeyOption": "RANDOM",
      "dataProcessingOptions": [
	      {
                    "optionName": "LOGTOJSON",
                    "logFormat": "COMMONAPACHELOG"
              },
	      {
	        "optionName": "ADDEC2METADATA",
		"logFormat": "COMMONAPACHELOG"
	      }
      ]
    }
  ]
}

In this example, the /etc/aws-kinesis/agent.json file processes the COMMONAPACHELOG log file format. Note that if your log file is in a different format, then the dataProcessingOptions setting must be updated to your log file format. For more information about agents and processing options, see Use the Agent to Preprocess Data.

Important: Add ADDEC2METADATA for the optionName field to be sure that the EC2 metadata is appended to each log line. By default, the Kinesis Agent appends the following parameters in EC2 metadata: privateIp, availabilityZone, instanceId, instanceType, accountId, amiId, region, and metadataTimestamp.

5.    Configure and start the agent. The agent now runs as a system service in the background. It continuously monitors the specified files, sending data to the specified delivery stream. The agent activity is logged in the /var/log/aws-kinesis-agent/aws-kinesis-agent.log file like in this example output:

{
        "host": "157.92.12.106",
        "ident": null,
        "authuser": null,
        "datetime": "31/Aug/1995:20:50:31 -0400",
        "request": "GET /history/astp/astp-spacecraft.txt HTTP/1.0",
        "response": "200",
        "bytes": "440",
        "privateIp": "X.X.X.X",
        "availabilityZone": "us-east-1c",
        "instanceId": "i-01bxxxxxxxxxx43a0",
        "instanceType": "t2.xlarge",
        "accountId": "585xxxxxx740",
        "amiId": "ami-0fc61db8544a617ed",
        "region": "us-east-1",
        "metadataTimestamp": "2020-04-20T02:28:40+0000"
    }

On a Windows platform

To use Amazon Kinesis Tap Agent to send data to Data Firehose, perform the following steps:

1.    Install Kinesis Agent for Windows.

2.    Create an IAM role with the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "firehose:DeleteDeliveryStream",
        "firehose:PutRecord",
        "firehose:PutRecordBatch",
        "firehose:UpdateDestination"
      ],
      "Resource": [
        "<KFH ARN>"
      ]
    },
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:DescribeInstanceAttribute",
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeInstanceStatus"
      ],
      "Resource": "*"
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": "cloudwatch:PutMetricData",
      "Resource": "*"
    }
  ]
}

3.    Attach the newly created IAM role to the EC2 instances where you installed Kinesis Tap Agent.

4.    Open the C:\Program Files\Amazon\AWSKinesisTap\appsettings.json file:

{
    "Sources": [
        {
            "Id": "W3SVCLog1",
            "SourceType": "W3SVCLogSource",
            "Directory": "C:\\inetpub\\logs\\LogFiles\\W3SVC1",
            "FileNameFilter": "*.log",
            "TimeZoneKind": "UTC"
        }
    ],
    "Sinks": [
        {
            "Id": "W3SVCLogSink",
            "SinkType": "KinesisFirehose",
            "Region": "eu-central-1",
            "StreamName": " W3SVCLogStream",
	"Format": "json",
	"ObjectDecoration": "instance_id={instance_id};hostname={hostname};ec2:local-hostname={ec2:local-hostname};computername={computername};env:computername={env:computername};timestamp:yyyyMMdd={timestamp:yyyyMMdd}"
        }		
	
    ],
    "Pipes": [
        {
            "Id": "W3SVCLog1ToKinesisStream",
            "SourceRef": "W3SVCLog1",
            "SinkRef": "W3SVCLogSink"
        }
    ]
}

Important: Add "ObjectDecoration": "instance_id={instance_id};hostname={hostname};ec2:local-hostname={ec2:local-hostname};computername={computername};env:computername={env:computername};timestamp:yyyyMMdd={timestamp:yyyyMMdd}" to your sink to be sure that EC2 metadata is appended to each log line. Kinesis Tap Agent appends the following parameters as EC2 metadata: instance_id, hostname, ec2:local-hostname, computername, env:computername, and timestamp:yyyyMMdd. If you don't want to display all of these parameters, then specify the parameters that you want to append.

For more information about configuration options, see Configuring Amazon Kinesis Agent for Microsoft Windows.

5.    Configure and start Kinesis Agent for Windows to start Kinesis Tap Agent. The following output should appear in each log line:

{
    "EventId": 7036,
    "Description": "The WinHTTP Web Proxy Auto-Discovery Service service entered the stopped state.",
    "LevelDisplayName": "Informational",
    "LogName": "System",
    "MachineName": "EC2AMAZ-GLL60A7",
    "ProviderName": "Service Control Manager",
    "TimeCreated": "2020-04-20T06:02:51.5847181Z",
    "Index": 34427,
    "UserName": null,
    "Keywords": "Classic",
    "instance_id": "i-0183xxxxxxxxxx4b7",
    "hostname": "ip-x-x-x-x.ec2.internal",
    "ec2:local-hostname": "ip-x-x-x-x.ec2.internal",
    "computername": "EC2AMAZ-GLL60A7",
    "env:computername": "EC2AMAZ-GLL60A7",
    "timestamp:yyyyMMdd": "20200420"
}