How do I troubleshoot CloudWatch Logs so that it streams to my Amazon OpenSearch Service domain?

Last updated: 2022-12-13

I'm unable to stream my Amazon CloudWatch Logs to my Amazon OpenSearch Service domain. How do I troubleshoot this issue?

Resolution

I'm unable to stream multiple CloudWatch log groups to the same OpenSearch Service domain

By default, Amazon CloudWatch creates only one AWS Lambda function for each OpenSearch Service domain. If you set up multiple log groups to index data into your domain, then all the multiple log groups invoke the same Lambda function. When the first log group invokes a Lambda function, the invocation creates an index and a type field in your domain.

Indices that are created in Amazon OpenSearch Service 6.0.0 or later can contain only a single mapping type. Indices that are created in 5.x with multiple mapping types continue to function as before in OpenSearch Service 6.x. For more information about OpenSearch Service mapping types deprecation, see Removal of mapping types on the Elastic website.

When other log groups try to invoke the same Lambda function, the invocation fails with the following error message:

"reason": "Rejecting mapping update to [ ] as the final mapping would have more than 1 type: [log-group-1, log-group-2]”

To resolve this issue, first update your Lambda function with the following syntax:

var indexName = [
     'cwl-' + payload.logGroup.toLowerCase().split('/').join('-') + '-' + timestamp.getUTCFullYear(),
     ('0' + (timestamp.getUTCMonth() + 1)).slice(-2),
     ('0' + timestamp.getUTCDate()).slice(-2) 
     ].join('.');

This syntax creates multiple indices for the different log groups that are streaming into your OpenSearch Service domain.

Then, save the updated Lambda function to create separate indices for the multiple log groups that are streaming into your domain.

I'm unable to stream to a VPC-based OpenSearch Service domain in the same AWS account

Important: Before streaming the CloudWatch log groups to your VPC-based OpenSearch Service domain, make sure to update your AWS Identity and Access Management (IAM) role policy. The IAM role attached to the corresponding Lambda function must have the AWSLambdaVPCAccessExecutionRole policy attached to it.

Here's an AWSLambdaVPCAccessExecutionRole policy in JSON format:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "ec2:CreateNetworkInterface",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DeleteNetworkInterface"
      ],
      "Resource": "*"
    }
  ]
}

Note: This managed policy enables the Lambda function to write the CloudWatch log group to your cluster in the VPC.

After you attach the policy to your Lambda function, you can begin streaming the logs to your OpenSearch Service domain in the VPC.

I'm unable to stream my CloudWatch log group to an OpenSearch Service domain when fine-grained access control is enabled

If you stream your CloudWatch Logs to an OpenSearch Service domain with fine-grained access control, you might encounter the following permissions error:

"{\"statusCode\":403,\"responseBody\":{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"no permissions for [indices:data/write/bulk] and User [name=arn:aws:iam::123456789101:role/lambda_opensearch_execution, roles=[arn:aws:iam::123456789101:role/lambda_opensearch_execution], requestedTenant=null]\"}],\"type\":\"security_exception\",\"reason\":\"no permissions for [indices:data/write/bulk] and User [name=arn:aws:iam::123456789101:role/lambda_opensearch_execution, roles=[arn:aws:iam::123456789101:role/lambda_opensearch_execution], requestedTenant=null]\"},\"status\":403}}"

If you receive this error message from your Lambda function logs, then the role mapping is incomplete.

Note: By default, OpenSearch Service creates an AWS Lambda function for you.

OpenSearch Service domains running 7.9 and later (including OpenSearch version 1.x)

To resolve the error message, perform the following steps:

1.    Open OpenSearch Dashboards. You can find a link to OpenSearch Dashboards in the domain summary of your OpenSearch Service console.

2.    From the navigation pane, choose Security.

3.    Choose Roles.

4.    Choose the all_access role.

5.    Choose the Mapped users tab.

6.    On the Mapped users dialog page, choose Manage mapping.

7.    Under Backend roles, enter the Lambda function execute role ARN.

8.    Choose Map. Your logs should now stream to your OpenSearch Service domain.

For more information about role mapping, see Mapping roles to users.

OpenSearch Service domains running 7.8 and earlier

To resolve the error message, perform the following steps:

1.    Open OpenSearch Dashboards. You can find a link to OpenSearch Dashboards in the domain summary of your OpenSearch Service console.

2.    On the left navigation pane, choose the lock icon.

3.    Select Role mappings.

4.    Choose all_access and security_manager as your roles.

Note: The all_access role provides access only to your cluster. Based on your use case, you can also add fine-grained access control to your cluster.

5.    Edit the mapping for all_access.

6.    For Backend Role, add the Lambda function's execution role and choose Submit. Your logs should now stream to your OpenSearch Service domain.

My CloudWatch logs aren't being delivered to my OpenSearch Service domain

When you stream CloudWatch logs (using the default AWS Lambda function) to your OpenSearch Service domain, you might encounter the following indexing error:

"errorMessage": "{\"statusCode\":200,\"responseBody\":{\"took\":42,\"errors\":true}}",

Note: By default, AWS Lambda errors are returned as 200 OK responses.

To troubleshoot this error message, perform the following steps:

1.    Open the default AWS Lambda function.

2.    Find the following line of code:

"var logFailedResponses = false;"

3.    Update the var logFailedResponses value to "true." This update provides additional information for any new indexing requests using the AWS Lambda function. You can use the additional information to identify the cause of your indexing issue.

I'm getting a cluster_block_exception error

Cluster block exceptions are caused by the following:

  • Lack of free storage space
  • Excessive JVM memory pressure

For more information about troubleshooting cluster block exceptions, see ClusterBlockException.

My CloudWatch subscription filter fails to send data to my cluster through a default Lambda function (OpenSearch Service 2.0 and later)

You might get an error if you have a CloudWatch subscription filter to send logs to Amazon OpenSearch Service 2.x using the default Lambda function. If the subscription filter fails to ingest the logs and you see the following error, then the error is caused by a deactivated parameter:

"{\"statusCode\":400,\"responseBody\":{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"},\"status\":400}}"

In OpenSearch Service versions 2.0 and later, the _type parameter is removed from API endpoints. To resolve this error, you must also remove the parameter from your Lambda function's code.

1.    Open the AWS Lambda console.

2.    Select the default Lambda function for your subscription filter.

3.    See the code for your function under Code source.

4.    Find the transform function that's defined in the code. Within this function, the data is converted into the JSON indexing format for OpenSearch Service. The first lines of this code look similar to the following:

function transform(payload) {
    if (payload.messageType === 'CONTROL_MESSAGE') {
        return null;
    }

    var bulkRequestBody = '';

    payload.logEvents.forEach(function(logEvent) {
                var timestamp = new Date(1 * logEvent.timestamp);

5.    Under the transform function, find the _type parameter. In most cases, this is will be in line 79. Remove or comment out the line of code that adds the _type parameter. After removal, your code looks similar to the following:

var action = {
    "index": {}
};
action.index._index = indexName;
//action.index._type = payload.logGroup;
action.index._id = logEvent.id;

bulkRequestBody += [

You can now send indexing requests successfully.


Did this article help?


Do you need billing or technical support?