How do I troubleshoot CloudWatch Logs so that it streams to my Amazon OpenSearch Service domain?

Last updated: 2021-07-30

I'm unable to stream my Amazon CloudWatch Logs to my Amazon OpenSearch Service (successor to Amazon Elasticsearch Service) domain. How do I troubleshoot this issue?

Resolution

I'm unable to stream multiple CloudWatch log groups to the same OpenSearch Service domain

By default, Amazon CloudWatch creates only one AWS Lambda function for each OpenSearch Service domain. If you set up multiple log groups to index data into your domain, all the multiple log groups invoke the same Lambda function. When the first log group invokes a Lambda function, the invocation creates an index and a type field in your domain.

For Elasticsearch versions 5.0 or later, you can have only one mapping type. The same rule applies to Elasticsearch versions 5.x to 6.x. For more information about Elasticsearch mapping types, see What are mapping types? on the Elasticsearch website.

When other log groups try to invoke the same Lambda function, the invocation fails with the following error message:

"reason": "Rejecting mapping update to [<index_name>] as the final mapping would have more than 1 type: [log-group-1, log-group-2]”

To resolve this issue, update your Lambda function with the following syntax:

var indexName = [
        'cwl-' + payload.logGroup.toLowerCase().split('/').join('-') + '-' + timestamp.getUTCFullYear(),
        ('0' + (timestamp.getUTCMonth() + 1)).slice(-2),
        ('0' + timestamp.getUTCDate()).slice(-2) 
        ].join('.');

This syntax creates multiple indices for the different log groups that are streaming into your OpenSearch Service domain.

Then, save the updated Lambda function to create separate indices for the multiple log groups that are streaming into your domain.

I'm unable to stream to a VPC-based OpenSearch Service domain in the same AWS account

Important: Before streaming the CloudWatch log groups to your VPC-based OpenSearch Service domain, make sure to update your AWS Identity and Access Management (IAM) role policy. The IAM role attached to the corresponding Lambda function must have the AWSLambdaVPCAccessExecutionRole policy attached to it.

Here's an AWSLambdaVPCAccessExecutionRole policy in JSON format:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "ec2:CreateNetworkInterface",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DeleteNetworkInterface"
            ],
            "Resource": "*"
        }
    ]
}

Note: This managed policy enables the Lambda function to write the CloudWatch log group to your cluster in the VPC.

After you attach the policy to your Lambda function, you can begin streaming the logs to your OpenSearch Service domain in the VPC.

I'm unable to stream my CloudWatch log group to an OpenSearch Service domain when fine-grained access control is enabled

If you stream your CloudWatch Logs to an OpenSearch Service domain with fine-grained access control, you might encounter the following permissions error:

"{\"statusCode\":403,\"responseBody\":{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"no permissions for [indices:data/write/bulk] and User [name=arn:aws:iam::123456789101:role/lambda_elasticsearch_execution, roles=[arn:aws:iam::123456789101:role/lambda_elasticsearch_execution], requestedTenant=null]\"}],\"type\":\"security_exception\",\"reason\":\"no permissions for [indices:data/write/bulk] and User [name=arn:aws:iam::123456789101:role/lambda_elasticsearch_execution, roles=[arn:aws:iam::123456789101:role/lambda_elasticsearch_execution], requestedTenant=null]\"},\"status\":403}}"

If you receive this error message from your Lambda function logs, then the role mapping is incomplete.

Note: By default, OpenSearch Service creates an AWS Lambda function for you.

To resolve the error message, perform the following steps:

1.    Open OpenSearch Dashboards. You can find a link to OpenSearch Dashboards in the domain summary of your OpenSearch Service console.

2.    On the left navigation pane, choose the lock icon.

3.    Select Role mappings.

4.    Choose all_access and security_manager as your roles.

Note: The all_access role provides access only to your cluster. Based on your use case, you can also add fine-grained access control to your cluster.

5.    Edit the mapping for all_access.

6.    For Backend Role, add the Lambda function's execution role and choose Submit. Your logs should now stream to your OpenSearch Service domain.

For more information about role mapping, see Mapping roles to users.

My Amazon CloudWatch logs aren't being delivered to my OpenSearch Service domain

If you stream CloudWatch Logs (using the default AWS Lambda function) to your OpenSearch Service domain, you might encounter the following indexing error:

"errorMessage": "{\"statusCode\":200,\"responseBody\":{\"took\":42,\"errors\":true}}",

Note: By default, AWS Lambda errors are returned as 200 OK responses.

To troubleshoot this error message, perform the following steps:

1.    Open the default AWS Lambda function.

2.    Find the following line of code: 

"var logFailedResponses = true;"

3.    Update the var logFailedResponses value to "false". This update provides additional information for any new indexing requests using the AWS Lambda function. You can use the additional information to identify the cause of your indexing issue.

I'm getting a cluster_block_exception error

Cluster block exceptions are caused by the following:

  • Lack of free storage space
  • Excessive JVM memory pressure

For more information about troubleshooting cluster block exceptions, see ClusterBlockException.


Did this article help?


Do you need billing or technical support?