How do I stream data from CloudWatch Logs to a VPC-based Amazon OpenSearch Service cluster in a different account?

Last updated: 2021-07-23

I'm trying to stream data from Amazon CloudWatch Logs to an Amazon OpenSearch Service cluster using a virtual private cloud (VPC) in another account. However, I receive an "Enter a valid Amazon OpenSearch Service Endpoint" error message. How do I resolve this error?

Short description

Note: Amazon OpenSearch Service is the successor to Amazon Elasticsearch Service (Amazon ES).

To stream data from CloudWatch Logs to an OpenSearch Service cluster in another account, perform the following steps:

1.    Set up CloudWatch Logs in Account A.

2.    Configure AWS Lambda in Account A.

3.    Configure Amazon Virtual Private Cloud (Amazon VPC) peering between accounts.

Resolution

Set up CloudWatch Logs in Account A

1.    Open the CloudWatch Logs console in Account A and select your log group.

2.    Choose Actions.

3.    Choose the Create OpenSearch subscription filter.

4.    For the Select Account option, select This account.

5.    For the OpenSearch Service cluster dropdown list, choose an existing cluster for Account A.

6.    Choose the Lambda IAM Execution Role that has permissions to make calls to the selected OpenSearch Service cluster.

7.    Attach the AWSLambdaVPCAccessExecutionRole policy to your role.

8.    In Configure log format and filters, select your Log Format and Subscription Filter Pattern.

9.    Choose Next.

10.    Enter the Subscription filter name and choose Start Streaming. For more information about streaming, Streaming CloudWatch Logs data to Amazon OpenSearch Service.

Configure Lambda in Account A

1.    In Account A, open the Lambda console.

2.    Select your Lambda function.

3.    In the function code, update the endpoint variable of the OpenSearch Service cluster in Account B. This update allows the Lambda function to send data to the OpenSearch Service domain in Account B.

Note: You can paste the public DNS name for Account B. Make sure to remove "https://" from the URL endpoint.

4.    Choose Configuration.

5.    Choose VPC.

6.    Under VPC, choose Edit.

7.    Select your VPC, subnets, and security groups.

Note: This selection makes sure that the Lambda function runs inside a VPC, using VPC routing to send data back to the OpenSearch Service domain. For more information about Amazon Virtual Private Cloud (Amazon VPC) configurations, see Configuring a Lambda function to access resources in a VPC.

8.    Choose Save.

Configure VPC peering between accounts

1.    Open the Amazon VPC console in Account A and Account B.

Note: Be sure that your VPC doesn't have overlapping CIDR blocks.

2.    Create a VPC peering session between the two custom VPCs (Lambda and OpenSearch Service). This VPC peering session allows Lambda to send data to your OpenSearch Service domain. For more information about VPC peering connections, see Creating and accepting a VPC peering connection.

3.    Update the route table for both VPCs. For more information about route tables, see Updating your route tables for a VPC peering connection.

4.    In Account A, go to Security Groups.

5.    Select the security group assigned to the subnet where Lambda is set up.

6.    Add the inbound rule to allow traffic from the OpenSearch Service subnets.

7.    In Account B, select the security group assigned to the subnet where OpenSearch Service is set up.

8.    Add the inbound rule to allow traffic from the Lambda subnets.

9.    In Account B, open the OpenSearch Service console.

10.    Choose Actions.

11.    Choose modify access policy, and then append the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
       
    "AWS": "arn:aws:iam::<AWS Account A>:role/<Lambda Execution Role>"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1: ::<AWS
    Account B>:domain/<OpenSearch Domain Name>/*"
    }
  ]
}

This policy allows OpenSearch Service to make calls from the Lambda function's execution role.

12.    Check the Error count and success rate metric in the Lambda console. This metric verifies whether logs are successfully delivered to OpenSearch Service.

13.    Check the Indexing rate metric in OpenSearch Service to confirm whether the data was sent. CloudWatch Logs now streams across both accounts in your Amazon VPC.


Did this article help?


Do you need billing or technical support?