Networking & Content Delivery
Scale traffic using multiple Interface Endpoints
AWS PrivateLink is a networking service that is used to connect to AWS services, your internal services, and third-party Software as a Service (SaaS) services–all over the private, secure, and scalable AWS network.
AWS PrivateLink has two sides to it:
- Service provider: Responsible for offering the service. The service provider creates an Amazon Virtual Private Cloud (VPC) Endpoint Service (service) using Network Load Balancer (NLB).
- Service consumer: Consumes one or more services offered by service provider. To consume the service, the consumer creates an interface VPC endpoint (endpoint), using the service name provided by the service provider.
Security combined with simplicity has made AWS PrivateLink a go-to mechanism for secure consumption of services. It connects services across different accounts and Amazon VPCs, with no need for firewall rules, path definitions, or route tables. There is no need to configure an internet gateway, VPC peering connection, or manage VPC Classless Inter-Domain Routing (CIDRs).
That said, as per interface endpoint properties and limitations, each interface endpoint currently supports a bandwidth of up to 10 Gbps per Availability Zone (AZ) and bursts of up to 40 Gbps per AZ. For use cases where you need higher throughput, such as connecting to services like Amazon Kinesis Data Streams, you might require sustained throughput of greater than 10 Gbps per Availability Zone (AZ).
In this blog, I will demonstrate a solution to support higher bandwidth requirements when consuming a service over AWS PrivateLink.
This solution is based on horizontally scaling interface endpoints by creating multiple endpoints for the same service. It uses Amazon Route 53 – a highly available and scalable cloud Domain Name System (DNS) web service – to distribute the traffic across these multiple endpoints.
In the workflow that follows (Figure 1), I am using Kinesis Data Streams as an example service.
- Step 1: For a given service, create two (or more) interface endpoints with their Private DNS attribute (PrivateDnsEnabled) set to false. This is shown in Figure 2.
- Step 2: (2a) – Create a Private Hosted Zone (PHZ). For example: kinesis.us-west-2.amazonaws.com
- Step2: (2b) – associate PHZ with the Consumer VPC.
- Step 3: For the PHZ created in Step 2, create weighted ALIAS A records pointing to Regional DNS names of the interface endpoints created in Step 1.
An example, shown in Figure 4: Create Weighted ALIAS A Records, I created two ‘alias to VPC endpoint records’ such that traffic is distributed equally across these two records. This allows you to distribute traffic across two interface endpoints and in turn achieve desired sustained throughput.
Note: Make sure that Evaluate target health is enabled (toggled to Yes).
- To design scalable architecture, you should have good understanding of your traffic requirements – total required capacity, and how it is being dispersed/consumed.
- AWS PrivateLink managed Private DNS does not work with multiple endpoints. You must create your own Private Hosted Zone to use Private DNS with multiple endpoints, by following the steps mentioned in High-level workflow.
- As traffic scales on the consumer side, the VPC endpoint service on the service provider side also must scale.
- For AWS owned and operated services, AWS manages the service provider setup.
- For partner/independent software vendor (ISV) owned and operated services, partner/ISV manages the service provider setup.
- For shared services that you own and operate, you will be responsible for managing, and in turn, scaling service provider setup.
Monitoring interface endpoints consumption allows you to either scale up or down to suit your business requirements. You can use VPC Flow Logs to monitor interface endpoint elastic network interface (ENI) utilization.
Here are the steps to setup interface endpoint monitoring:
- Step 1: Configure flow logs for each interface endpoint ENI that must be monitored. One-min aggregation interval is recommended for better granularity. Destination should be Amazon CloudWatch Logs, and choose the desired log group and IAM role. Custom format is required with at least the following fields (you can select more fields to view additional metadata if necessary):
Figures 5a and 5b show how to configure a VPC Flow Log for an ENI.
- Step 2: Using CloudWatch insights for each ENI, create the following queries. Make sure to use the correct parse format in case extra fields were added into the flow logs.
The following screenshots (Figures 6, 7 and 8) show how this looks when the query text is added in the console.
fields @ingestionTime, @message, @timestamp, @logStream | parse @message "* * * *" as interface, packets, bytes, tcpflags | filter interface="eni-0dfebe5f446e64a66" -- REPLACE WITH YOUR VPC ENDPOINT ENI ID | stats sum(bytes)*8/60 as avg_bps by bin(1m) as time | sort time desc
fields @ingestionTime, @message, @timestamp, @logStream | parse @message "* * * *" as bytes, interface, packets, tcpflags | filter interface="eni-0dfebe5f446e64a66" -- REPLACE WITH YOUR VPC ENDPOINT ENI ID | stats sum(packets)/60 as avg_pps by bin(1m) as time | sort time desc
New Connections/s Query:
fields @ingestionTime, @message, @timestamp, @logStream | parse @message "* * * *" as interface, packets, bytes, tcpflags | filter interface="eni-0dfebe5f446e64a66" and tcpflags=2 -- REPLACE WITH YOUR VPC ENDPOINT ENI ID | stats sum(packets)/60 as cps by bin(1m) as time, interface | sort time desc
- Step 3: Add the queries to the CloudWatch dashboard. This is shown in the following screenshots (Figures 9a and 9b).
Repeat to add each query/insight to dashboard. The following screenshot (Figure 10) shows a completed CloudWatch Dashboard.
- Step 4: Repeat steps for each ENI that must be monitored. Make sure to adjust the graph name and ENI ID in the queries.
To cleanup your environment and ensure you don’t incur any unwanted charges, delete your interface endpoints, private hosted zone, and any other related resources that you might have created while going through this blog.
AWS PrivateLink is a networking service that allows you to connect to AWS services, to shared services owned by customer in other VPCs, or to third-party SaaS services, privately and securely over the AWS network. In this blog, I showed you how you can achieve sustained bandwidth of greater than 10 Gbps/AZ by creating multiple interface endpoints and using Amazon Route 53 to distribute traffic across those endpoints. I also illustrated how to monitor interface endpoint ENI using VPC Flow Logs.