How do I stream data from CloudWatch Logs to a VPC-based Amazon Elasticsearch Service cluster in a different account?

Last updated: 2020-06-16

I'm trying to stream data from Amazon CloudWatch Logs to an Amazon Elasticsearch Service (Amazon ES) cluster using virtual private cloud (VPC) in another account. However, I receive an "Enter a valid Amazon Elasticsearch Service Endpoint" error message. How do I resolve this error?

Short Description

To stream data from CloudWatch Logs to an Elasticsearch cluster in another account, perform the following steps:

1.    Set up CloudWatch Logs in Account A.

2.    Configure AWS Lambda in Account A.

3.    Configure VPC peering between accounts.

Resolution

Set up CloudWatch Logs in Account A

1.    Open the CloudWatch Logs console in Account A and select your log group.

2.    Choose the Stream to Amazon Elasticsearch Service action.

3.    For the Select Account option, select This account.

4.    For the Elasticsearch cluster drop-down, choose an existing cluster for Account A.

5.    Choose the Lambda IAM Execution Role that has permissions to make calls to the selected Elasticsearch cluster.

6.    Attach the AWSLambdaVPCAccessExecutionRole policy to your role and choose Next.

7.    Select your Log Format and Subscription Filter Pattern.

8.    Choose Next.

9.    Enter your Lambda Function Name and choose Start Streaming. For more information about streaming, see Streaming CloudWatch Logs data to Amazon Elasticsearch Service.

Configure Lambda in Account A

1.    In Account A, open the Lambda console.

2.    Select your Lambda function.

3.    In the function code, update the endpoint variable of the Elasticsearch cluster in Account B. This update allows the Lambda function to send data to the Amazon ES domain in Account B.

Note: You can paste the public DNS name for Account B. Be sure to remove "https://" from the URL endpoint.

4.    Choose Custom VPC.

5.    Choose Save. This selection ensures that the Lambda function runs inside a VPC, using VPC routing to send data back to the Amazon ES domain. For more information about Amazon Virtual Private Cloud (Amazon VPC) configurations, see Configuring a Lambda function to access resources in a VPC.

Configure VPC peering between accounts

1.    Open the Amazon VPC console in Account A and Account B.

Note: Be sure that your VPC doesn't have overlapping CIDR blocks.

2.    Create a VPC peering session between the two custom VPCs (Lambda and Amazon ES). This VPC peering session allows Lambda to send data to your Amazon ES domain. For more information about VPC peering connections, see Creating and accepting a VPC peering connection.

3.     Update the route table for both VPCs. For more information about route tables, see Updating your Route tables for a VPC peering connection.

4.    In Account A, go to Security Groups.

5.    Select the security group assigned to the subnet where Lambda is set up.

6.    Add the inbound rule to allow traffic from the Amazon ES subnets.

7.    In Account B, select the security group assigned to the subnet where Amazon ES is set up.

8.    Add the inbound rule to allow traffic from the Lambda subnets.

9.    In Account B, open the Amazon ES console.

10.    Choose Actions.

11.    Choose modify access policy, and then append the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
       
    "AWS": "arn:aws:iam::<AWS Account A>:role/<Lambda Execution Role>"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1: ::<AWS
    Account B>:domain/<ES Domain Name>/*"
    }
  ]
}

This policy allows Amazon ES to make calls from the Lambda function's execution role.

12. Check the Error count and success rate metric in the Lambda console. This metric verifies whether logs are successfully delivered to Amazon ES.

13. Check the Indexing rate metric in Amazon ES to confirm whether the data was sent. CloudWatch Logs now streams across both accounts in your Amazon VPC.


Did this article help you?

Anything we could improve?


Need more help?