How do I create, configure, and troubleshoot a subscription filter to Kinesis using the CloudWatch console?

Short description

Use a subscription filter to send CloudWatch logs in near real-time to the same account or to cross-account Kinesis or Amazon Kinesis Data Firehose destinations. The CloudWatch Logs console supports the destination and setup configuration.

For information on how to use filter pattern syntax to configure a subscription filter, see Filter and pattern syntax.

Resolution

Subscription configuration for a Kinesis data stream in the same or current account

Note: The AWS Region for the CloudWatch log group and the Kinesis destination must be the same.

Before you create your subscription, complete the following actions:

• If you didn't create create a Kinesis stream, then create one.
• Create an AWS Identify and Access Management (IAM) role that has the appropriate trust policy and permissions.

To create a custom IAM role and role policy, complete the following steps:

1.    Open the IAM console with your user that has administrator permissions.

2.    In the navigation pane, choose Policies.

3.    In the content pane, choose Create policy.

4.    Enter the following role permissions policy document into the JSON tab. Replace REGION, ACCOUNT_ID, AND STREAM_NAME with your own details:

{
  "Statement": \[{
    "Effect": "Allow",
    "Action": "kinesis:PutRecord",
    "Resource": "arn:aws:kinesis:REGION:ACCOUNT\_ID:stream/STREAM\_NAME"
  }\]
}

5.    Open the IAM console.

6.    In the navigation pane, choose Roles, and then choose Create role.

7.    Choose the Custom trust policy role type.

8.    In the Custom trust policy section, enter or paste the custom trust policy for the role. See the following example trust policy:

{
  "Statement": {
    "Effect": "Allow",
    "Principal": {
      "Service": "logs.region.amazonaws.com"
    },
    "Action": "sts:AssumeRole",
    "Condition": {
      "StringLike": {
        "aws:SourceArn": "arn:aws:logs:REGION:ACCOUNT\_ID:\*"
      }
    }
  }
}

9.    Choose Next.

10.    Select the custom IAM policy that you created in step 4.

11.    Choose Next, and then choose Create a Role.

After you create the Kinesis stream and the IAM role, create the subscription filter:

1.    Open the CloudWatch console.

2.    Choose Log Group.

3.    Choose Action, Subscription Filters.

4.    To configure the destination, choose Create Kinesis Subscription Filter.

5.    Choose Current Account.

6.    Select your Kinesis data stream from the dropdown list.

7.    Select the IAM role that you created.

8.    Choose the Distribution method:
By Log Stream: This verifies that downstream consumers can aggregate log events by log stream, but might be less efficient. This method might also incur higher streaming costs because it requires more shards.
Random: This distributes the load across Kinesis stream shards, but downstream consumers can't aggregate log events by log stream.

9.    Configure Log format and filters:
Select the log format. The format can be Amazon VPC Flow Logs, AWS CloudTrail, or AWS Lambda for logs that are published by Amazon VPC, CloudTrail, or Lambda. Or, you can choose JSON, Space delimited, or Other, depending on your incoming log events.
Define the filter pattern in the Subscription filter pattern section.
Enter a name for your subscription filter.

10.    Verify your pattern with the existing log event data.

11.    After verifying, choose Start Streaming.

12.    (Optional) Validate the flow of log events to verify that your data stream works.

Configuration for a cross-account Kinesis data stream destination

Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.

You can deliver CloudWatch Logs events to Kinesis data streams in different AWS accounts and Regions. To do this, set up cross-account log data sharing with subscriptions and specify the Region.

In the following example, CloudWatch logs in the us-east-1 Region are delivered to another AWS user's Kinesis data stream in us-west-2 . The log data recipient's account ID is 222222222222, and the log data sender's account ID is 111111111111.

Create a destination data stream in the recipient account 222222222222

Create a destination data stream in Kinesis in the data recipient's account with an IAM role and trust policy.

Create a subscription filter in the source account 111111111111

To create the subscription filter, complete the following steps:

1.    Select the log group.

2.    Choose Action, Subscription Filters.

3.    To select the destination, choose Create, and then select Create Kinesis Subscription Filter.

4.    Choose Another Account.

5.    For a cross-account Kinesis or Kinesis data Firehose destination, provide the Destination ARN.

6.    Choose the Distribution method:
By Log Stream: This verifies that downstream consumers can aggregate log events by log stream, but might be less efficient. This method might also incur higher streaming costs because it requires more shards.
Random: This distributes the load across Kinesis stream shards, but downstream consumers can't aggregate log events by log stream.

7.    Configure Log format and filters:
Select the log format. The format can be Amazon VPC Flow Logs, CloudTrail, or AWS Lambda for logs that are published by Amazon VPC, CloudTrail, or Lambda. Or, you can choose JSON, Space delimited, or Other, depending on your incoming log events.
Define the filter pattern in the Subscription filter pattern section.
Enter a name of your subscription filter.

8.    Verify your pattern with the existing log event data.

9.    After verifying, choose Start Streaming.

10.    (Optional) Validate the flow log events to verify that your data stream works.

Troubleshooting

• Make sure that your Kinesis stream is in the Active state. You can view the stream on the Kinesis console or use the DescribeStream API call.
• Verify that the CloudWatch log group and Kinesis data stream Regions are the same.
• Make sure that there is an IAM role that has trust permissions for logs.yourregion.amazonaws.com and allows the permission kinesis:putrecords.
• Verify that the Regions and resources in the IAM policy are correct.
• Make sure that you didn't select Kinesis Firehose when configuring a subscription filter for Kinesis data stream.
• After you start streaming, check the subscription filter metrics to confirm that the filter pattern is valid and matches the incoming log events. Review the following metrics: ForwardedBytes: The volume of log events in compressed bytes forwarded to the subscription destination. ForwardedLogEvents: The number of log events forwarded to the subscription destination.
• Check that there aren't errors when streaming log events to your destination. Review the following metrics: DeliveryErrors: The number of log events that CloudWatch Logs received an error for when forwarding data to the subscription destination. DeliveryThrottling: The number of log events that indicate CloudWatch Logs was throttled when forwarding data to the subscription destination.
• If you have a dedicated Kinesis stream, then check the stream's metrics to confirm functionality.
• For issues with cross-account logging, see Troubleshooting your CloudWatch cross-account setup.