How do I troubleshoot CloudWatch Logs so that it streams to my Amazon ES domain?

Last updated: 2020-10-06

I'm unable to stream my Amazon CloudWatch Logs to my Amazon Elasticsearch Service (Amazon ES) domain. How do I troubleshoot this issue?

Resolution

I'm unable to stream multiple CloudWatch log groups to the same Amazon ES domain

By default, Amazon CloudWatch creates only one AWS Lambda function for each Amazon ES domain. If you set up multiple log groups to index data into an Amazon ES domain, all the multiple log groups invoke the same Lambda function. When the first log group invokes a Lambda function, the invocation creates an index and a type field in the Amazon ES domain.

For Elasticsearch versions 6.0 or later, you can have only one mapping type. The same rule applies to Elasticsearch versions 5.x to 6.x. For more information about Elasticsearch mapping types, see What are mapping types? on the Elasticsearch website.

When other log groups try to invoke the same Lambda function, the invocation fails with the following error message:

"reason": "Rejecting mapping update to [<index_name>] as the final mapping would have more than 1 type: [log-group-1, log-group-2]”

To resolve this issue, update your Lambda function with the following syntax:

var indexName = [
        'cwl-' + payload.logGroup.toLowerCase().split('/').join('-') + '-' + timestamp.getUTCFullYear(),
        ('0' + (timestamp.getUTCMonth() + 1)).slice(-2),
        ('0' + timestamp.getUTCDate()).slice(-2) 
        ].join('.');

This syntax creates multiple indices for the different log groups that are streaming into your Amazon ES domain. Then, save the updated Lambda function to create separate indices for the multiple log groups that are streaming into your Amazon ES domain.

I'm unable to stream to a VPC-based Amazon ES domain in the same AWS account

Important: Before streaming the CloudWatch log groups to your VPC-based Amazon ES domain, be sure to update your AWS Identity and Access Management (IAM) role policy. The IAM role attached to the corresponding Lambda function must have the AWSLambdaVPCAccessExecutionRole policy attached to it.

Here's an AWSLambdaVPCAccessExecutionRole policy in JSON format:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "ec2:CreateNetworkInterface",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DeleteNetworkInterface"
            ],
            "Resource": "*"
        }
    ]
}

Note: This managed policy enables the Lambda function to write the CloudWatch log group to the Elasticsearch cluster in the VPC.

After you've attached the policy to your Lambda function, begin streaming the logs to your Amazon ES domain in the VPC.

I'm unable to stream my CloudWatch log group to an Amazon ES domain when fine-grained access control is enabled

If you stream your CloudWatch Logs to an Amazon ES domain with fine-grained access control, you might encounter the following permissions error:

"{\"statusCode\":403,\"responseBody\":{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"no permissions for [indices:data/write/bulk] and User [name=arn:aws:iam::123456789101:role/lambda_elasticsearch_execution, roles=[arn:aws:iam::123456789101:role/lambda_elasticsearch_execution], requestedTenant=null]\"}],\"type\":\"security_exception\",\"reason\":\"no permissions for [indices:data/write/bulk] and User [name=arn:aws:iam::123456789101:role/lambda_elasticsearch_execution, roles=[arn:aws:iam::123456789101:role/lambda_elasticsearch_execution], requestedTenant=null]\"},\"status\":403}}"

If you receive this error message from your Lambda function logs, then it indicates that the role mapping is incomplete.

Note: By default, Amazon ES creates an AWS Lambda function for you. 

To resolve the error message, perform the following steps:

1.    Open Kibana. You can find a link to Kibana in the domain summary of your Amazon ES console.

2.    On the left navigation pane, choose the lock icon.

3.    Select Role mappings.

4.    Choose all_access and security_manager as your roles.

Note: The all_access role provides access only to your Elasticsearch cluster. Based on your use case, you can also add fine-grained access control to your Elasticsearch cluster.

5.    Edit the mapping for all_access.

6.    For Backend Role, add the Lambda function's execution role and choose Submit. Your logs should now stream to your Amazon ES domain.

For more information about role mapping, see Mapping roles to users.

I'm getting a cluster_block_exception error

Cluster block exceptions are caused by the following:

  • Lack of free storage space
  • Excessive JVM memory pressure

For more information about troubleshooting cluster block exceptions, see ClusterBlockException.


Did this article help?


Do you need billing or technical support?