Hybrid Networking using VPC Endpoints (AWS PrivateLink) and Amazon CloudWatch for Financial Services

Amazon CloudWatch offers a centralized service to collect monitoring and operational data in the form of logs, metrics, and events. This provides a unified view of AWS resources, applications, and services that run on AWS and on-premises servers. When you have Amazon CloudWatch agents running on-premises, the default behavior is to export the collected metrics to CloudWatch public endpoints which can be reached over the internet or Direct Connect Public VIF. This presents a challenge to many enterprises who are bound by their compliance policies to have private connectivity to AWS services. This is especially critical to customers in the financial services industry who hold sensitive data. With AWS PrivateLink, you can prevent your sensitive data from traversing the Internet and maintain compliance with regulations such as HIPAA, EU/US Privacy Shield, and PCI. You can leverage VPC Interface endpoints, powered by AWS PrivateLink, to extend the private connectivity all the way to your on-premises network using Direct Connect Private VIF or VPN. Your traffic is not exposed to the internet, eliminating important threat vectors.

This blog post demonstrates how you can use VPC Endpoint to export metrics from CloudWatch agents running on on-premises servers to Amazon CloudWatch via a private connection. We’ll also secure the endpoint using security-groups and endpoint policy. Let’s get started…

Prerequisites

• An AWS account
• Private connectivity through Direct Connect and/or Site-to-Site VPN to on-premises/DC
• Setup of CloudWatch agent with respect to system-level metrics on your on-premises servers

Solution Overview

Walkthrough

To Implement the solution, you need to complete the following steps for which I have provided step-by-step guidance below:

1. Create CloudWatch Interface endpoint from AWS VPC console in us-east-1 region
2. Verify private network connectivity between AWS and on-premises using DX/VPN
3. Setup Route53 resolver inbound endpoints to resolve DNS queries from on-premises
4. Set VPC Interface endpoint with endpoint policies, security group, and network ACL on endpoint ENI
5. Verify system-level metrics being delivered to CloudWatch using VPC endpoint

Create VPC Interface Endpoint for CloudWatch Monitoring service

Open Amazon VPC console and navigate to Endpoints and create an Endpoint.

Select AWS services and search for “com.amazonaws.us-east-1.monitoring”

Select the VPC in which you want to create the endpoint and optionally you can also configure tags.

Once you select the VPC, go ahead and select the subnets where you want the endpoint ENIs to be created. As a best practice, you should always select multiple subnets for high availability.

Select the check box for ‘Enable DNS name’. This feature associates a private hosted zone with the VPC that contains a record set. This enables you to leverage Amazon’s private network connectivity to reach the service. You can do so while making requests to the service’s default public endpoint DNS name.

Finally, attach a security-group to the endpoint ENIs. You can also create new security-group to restrict access based on your requirements. Later in the blog, I will customize the security group to allow access from on-premises subnet.

Check and verify the DNS entries and ENIs that were created for the endpoint. As I had selected the ‘Enable DNS name’ during endpoint creation, it created and associated a private hosted zone with the specified VPC. You can find out the DNS entries for your endpoint interface by either selecting VPC endpoint in the VPC console or using the below cli command. After this you can simply run nslookup on monitoring.us-east-1.amazonaws.com or custom DNS entry for endpoint from an ec2 instance.

aws ec2 describe-vpc-endpoints --filter Name=vpc-endpoint-id,Values=vpce-123456789abcdef --query 'VpcEndpoints[*].{DNS:DnsEntries[*].DnsName}' --output text

DNS	vpce-0123456789abcdef-nk86lk5o.monitoring.us-east-1.vpce.amazonaws.com
DNS	vpce-0123456789abcdef-nk86lk5o-us-east-1b.monitoring.us-east-1.vpce.amazonaws.com
DNS	vpce-0123456789abcdef-nk86lk5o-us-east-1c.monitoring.us-east-1.vpce.amazonaws.com
DNS	vpce-0123456789abcdef-nk86lk5o-us-east-1d.monitoring.us-east-1.vpce.amazonaws.com
DNS	vpce-0123456789abcdef-nk86lk5o-us-east-1e.monitoring.us-east-1.vpce.amazonaws.com
DNS	vpce-0123456789abcdef-nk86lk5o-us-east-1f.monitoring.us-east-1.vpce.amazonaws.com
DNS	vpce-0123456789abcdef-nk86lk5o-us-east-1a.monitoring.us-east-1.vpce.amazonaws.com
DNS	monitoring.us-east-1.amazonaws.com

[ec2-user@ip-10-100-0-37 ~]$ nslookup monitoring.us-east-1.amazonaws.com
...
Non-authoritative answer:
Name:	monitoring.us-east-1.amazonaws.com
Address: 10.100.4.146
Name:	monitoring.us-east-1.amazonaws.com
Address: 10.100.5.39
Name:	monitoring.us-east-1.amazonaws.com
Address: 10.100.0.101
Name:	monitoring.us-east-1.amazonaws.com
Address: 10.100.1.6
Name:	monitoring.us-east-1.amazonaws.com
Address: 10.100.2.241
Name:	monitoring.us-east-1.amazonaws.com
Address: 10.100.3.194

Note the private DNS name (monitoring.us-east-1.amazonaws.com) which is listed in the above command. You will need this when you setup conditional forwarding on your on-premises DNS resolver. It is always a best practice to be very specific with conditional forwarding as you may want other AWS services over public access.

If you wish to setup the CloudWatch VPC endpoint using CloudFormation for automation purposes, you can also leverage the CloudFormation template below. The CloudFormation template creates a CloudWatch Monitoring VPC endpoint and sets up security-groups on endpoint ENIs. This restricts usage to network subnets you specify and applies an endpoint policy which further restricts the endpoint usage to a specified VPC.

AWSTemplateFormatVersion: '2010-09-09'
Description: 'CloudWatch Monitoring VPCEndpoint'
Metadata:
  'AWS::CloudFormation::Interface':
    ParameterGroups:
    - Label:
        default: 'VPC Parameters'
      Parameters:
      - VPC
      - OnPremiseNet
      - PrivateSubnets

Parameters:
  VPC:
    Description: 'VPC'
    Type: 'AWS::EC2::VPC::Id'

  OnPremiseNet:
    Description: 'Please enter the IP range (CIDR notation) for OnPremise-Net'
    Type: String

  PrivateSubnets:
    Description: 'Select the subnets for Interface Endpoint ENIs'
    Type: 'List<AWS::EC2::Subnet::Id>'

Resources:
  SecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupDescription: !Ref 'AWS::StackName'
      GroupName: !Ref 'AWS::StackName'
      VpcId: !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          ToPort: 443
          FromPort: 443
          CidrIp: !Ref 'OnPremiseNet'

  Endpoint:
    Type: 'AWS::EC2::VPCEndpoint'
    Properties:
      PrivateDnsEnabled: true
      SecurityGroupIds: [!Ref SecurityGroup]
      ServiceName: !Sub 'com.amazonaws.${AWS::Region}.monitoring'
      SubnetIds: !Ref PrivateSubnets
      VpcEndpointType: Interface
      VpcId: !Ref VPC
      PolicyDocument: '{
        "Version":"2012-10-17",
        "Statement": [{
          "Sid": "Restrict Usage to PutMetric Action",
          "Effect": "Allow",
          "Principal": "*",
          "Action": ["cloudwatch:PutMetricData"],
          "Resource": "*"
          }]
      }'

Outputs:
  StackName:
    Description: 'Stack name.'
    Value: !Sub '${AWS::StackName}'
  Endpoint:
    Description: 'The VPC endpoint to a service.'
    Value: !Ref Endpoint
    Export:
      Name: !Sub '${AWS::StackName}-Endpoint'
  VPCEndpointDNS:

Connectivity between On-Premises and AWS VPC

In this blog we are not going into detail about how to set up Private VIF over Direct Connect or VPN. If you need assistance with DX and VPN setup, please follow our documentations on creating VIFs over DX and setting up VPN. Once you have connectivity up and running, make sure you can communicate with resources in your VPC. For this demo, I tested the connectivity from my on-premises router to an EC2 instance running in the VPC in which the CloudWatch VPC endpoint is deployed.

OnPremise-Router#show ip route 10.100.0.0 longer-prefixes  | i 10.100
S        10.100.0.0/24 is directly connected, Tunnel1
S        10.100.1.0/24 is directly connected, Tunnel1
S        10.100.2.0/24 is directly connected, Tunnel1
S        10.100.3.0/24 is directly connected, Tunnel1
S        10.100.4.0/24 is directly connected, Tunnel1
S        10.100.5.0/24 is directly connected, Tunnel1

OnPremise-Router#ping 10.100.0.37 source 10.200.2.203
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.100.0.37, timeout is 2 seconds:
Packet sent with a source address of 10.200.2.203
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 10/11/12 ms

Setting up Route53 Resolver Inbound Endpoint

Now that you have CloudWatch VPC endpoint setup in your VPC, you will still need a way to communicate with it from your on-premises network. This is where we need to setup Route53 inbound resolver so that DNS queries for private hosted zones forwarded by on-premises DNS resolver can be resolved. You will need to setup conditional forwarding on your on-premises DNS resolver to forward all queries for amazonaws.com to Route53 inbound endpoint IPs. In the following section, I’ll show you how to set up the Route53 endpoint.

1. Navigate to the Route53 console and to the Resolver section then select inbound endpoints.
2. When you create the resolver endpoint, select the VPC you used when creating CloudWatch VPC endpoint.
3. During the resolver endpoint creation, you will need to provide 2 AZs/Subnets so that Route53 can assign 1 IP for each subnet for high availability.

As you already have routing setup between your VPC and on-premises network, you should be able to reach Route53 resolver endpoint’s IPs. You can verify this by running a quick dig command. If you need further assistance, you can also follow our support blog which provides step by step instructions on setting up Route 53 inbound resolvers and testing your configuration.

Dig (Domain Information Groper) is a command line utility that performs DNS lookup by querying name servers and displaying the result to you.

dig @10.100.1.177 monitoring.us-east-1.amazonaws.com
.....
;; QUESTION SECTION:
;monitoring.us-east-1.amazonaws.com. IN	A

;; ANSWER SECTION:
monitoring.us-east-1.amazonaws.com. 60 IN A	10.100.2.241
monitoring.us-east-1.amazonaws.com. 60 IN A	10.100.4.146
monitoring.us-east-1.amazonaws.com. 60 IN A	10.100.0.101
monitoring.us-east-1.amazonaws.com. 60 IN A	10.100.3.194
monitoring.us-east-1.amazonaws.com. 60 IN A	10.100.5.39
monitoring.us-east-1.amazonaws.com. 60 IN A	10.100.1.6

Secure your VPC Endpoint

To secure the endpoint, use the endpoint policy. I used the below policy to restrict access to the VPC which has DX and VPN deployed. Please note: You can also restrict access to an AWS account ID. For more information on endpoint policies, please check out: Controlling access to services with VPC endpoints

{
    "Statement": [
        {
            "Sid": "Restrict usage to my VPC",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "CloudWatch:PutMetricData"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:sourceVpc": "vpc-1234567890abcdef"
                }
            }
        }

    ]
}

To further restrict access to your on-premises network you can use security-groups, which is applied to the endpoint ENI. With this, you can ensure that networks and resources that are allowed to access endpoint are able to. In this demo, I have only allowed access to 10.200.2.0/24, which is my on-premises subnet.

Verify that On-Premises Server Metrics Show-up in CloudWatch

Before checking the CloudWatch metric console, I quickly ran a tcpdump on one of my on-premises servers to verify that the CloudWatch agent was sending the metric data to the CloudWatch VPC endpoint. From the tcpdump snippet, you can see that server is querying Route53 resolver endpoint (10.100.1.77) for monitoring.us-east-1.amazonaws.com. Route53 resolver responds back with ‘A’ records (A 10.100.5.39, A 10.100.3.194, A 10.100.4.146, A 10.100.0.101, A 10.100.2.241, A 10.100.1.6) for CloudWatch VPC endpoint. Once the DNS resolution is complete, CloudWatch agent forwards the metric data to the CloudWatch VPC endpoint private IPs.

22:36:37.070714 IP ip-10-200-2-111.37364 > ip-10-100-1-177.domain: 38463+ AAAA? monitoring.us-east-1.amazonaws.com. (52)
22:36:37.070930 IP ip-10-200-2-111.55929 > ip-10-100-1-177.domain: 11267+ A? monitoring.us-east-1.amazonaws.com. (52)
22:36:37.074819 IP ip-10-100-1-177.domain > ip-10-200-2-111.37364: 38463 0/1/0 (136)
22:36:37.076165 IP ip-10-100-1-177.domain > ip-10-200-2-111.55929: 11267 6/0/0 A 10.100.5.39, A 10.100.3.194, A 10.100.4.146, A 10.100.0.101, A 10.100.2.241, A 10.100.1.6 (148)
22:36:37.076378 IP ip-10-200-2-111.41284 > ip-10-100-5-39.https: Flags [S], seq 83924387, win 26883, options [mss 8961,sackOK,TS val 111389825 ecr 0,nop,wscale 7], length 0
22:36:37.076453 IP ip-10-200-2-111.34390 > ip-10-100-1-177.domain: 54039+ PTR? 39.5.100.10.in-addr.arpa. (42)
22:36:37.078666 IP ip-10-100-1-177.domain > ip-10-200-2-111.34390: 54039 1/0/0 PTR ip-10-100-5-39. (83)
22:36:37.079026 IP ip-10-100-5-39.https > ip-10-200-2-111.41284: Flags [S.], seq 364925350, ack 83924388, win 8190, options [mss 1360,nop,wscale 6,nop,nop,sackOK], length 0
22:36:37.079036 IP ip-10-200-2-111.41284 > ip-10-100-5-39.https: Flags [.], ack 1, win 211, length 0
22:36:37.079268 IP ip-10-200-2-111.41284 > ip-10-100-5-39.https: Flags [P.], seq 1:305, ack 1, win 211, length 304
22:36:37.081659 IP ip-10-100-5-39.https > ip-10-200-2-111.41284: Flags [.], ack 305, win 252, length 0
22:36:37.081864 IP ip-10-100-5-39.https > ip-10-200-2-111.41284: Flags [.], ack 305, win 424, length 0

Finally, we can check in the CloudWatch Console for on-premises server metrics, under the Custom Namespaces section. You can view it by navigating to CloudWatch console -> Metrics -> All Metrics.

Conclusion

This blog post illustrated how you can privately send CloudWatch metrics from your on-premises network to CloudWatch, using VPC endpoint. It explained how customers (especially in the financial services industry) with critical data can secure their VPC endpoint by using endpoint policy and then further restrict the access to on-premises networks using security-groups applied to endpoint ENIs. AWS PrivateLink provides you with granular control over how your applications connect to AWS services without needing to establish connectivity over the public internet and thus complying with your company’s compliance and security policies. You can leverage this architecture for other AWS Services that integrate with AWS PrivateLink as well. The core building blocks of the solution remain the same.

You can learn more about AWS PrivateLink in our service page here.