External Connectivity to Amazon VPC Lattice

In this blog post, we discuss how to connect on-premises and external services to Amazon VPC Lattice. We will go over architectural considerations and requirements for connecting services from trusted locations (on-premises), non-trusted locations (3^rd party), and across AWS Regions. Then, we provide a solution that builds out a capability to facilitate this using AWS Elastic Load Balancer (ELB) and a fleet of Serverless web proxies.

Ready?

Background

Amazon VPC Lattice (VPC Lattice) is an application networking service that consistently connects, monitors, and secures communications between your services; removing much of the complexity associated with the planning and implementation of application-to-application communications. Inside an Amazon Virtual Private Cloud (VPC), the way that you connect your applications using VPC Lattice is straight-forward: As a VPC Lattice service owner, you create representations of your compute endpoints such as AWS Lambda, Amazon EKS, an Auto Scaling group an Application Load Balancer or even an IP target in an Account and VPC that you control. You can then make these VPC Lattice services available inside an Amazon VPC Service Network (Service Network) by associating them, or by sharing them using Resource Access Manager (RAM), with other AWS Accounts. Consumers of a VPC Lattice service need only to have their respective VPCs associated with a Service Network where the VPC Lattice Service is published. Once the Service Network is associated with a consumer’s VPC, discovery of the published VPC Lattice service is made possible by Amazon Route53 (R53). Inside the VPC, VPC Lattice handles all the connectivity and discovery requirements for you. For detailed guidance on how you can use VPC Lattice with your compute workloads in a simple and consistent way, check out the following: Build secure multi-account multi-VPC connectivity for your applications with Amazon VPC Lattice.

One obvious question here is ‘why can’t you access a VPC Lattice service outside of the VPC?’ This is related to the discovery and connectivity process. When you create a VPC Lattice service, you are given a DNS name that represents it. This name is globally unique and externally resolvable – that part is simple. However, from outside of the VPC, the DNS name resolves to a series of IP addresses in the 169.254.171.x/24 range (within the IPv4 link-local range 169.254/16 defined in RFC3927) and the fd00:ec2:80::/64 range (within the IPv6 unique local range fc00::/7 defined in RFC4193). Link-local and unique local addresses are not globally routable and are intended for devices that are either connected to the same physical (or logical) link or within a more limited area/site. When a consumer in a VPC resolves a VPC Lattice service to a link-local address, packets put on the wire to that address are routed to an ingress endpoint for VPC Lattice. In the destination VPC, the inverse happens, and VPC Lattice makes the necessary connections to the specific target groups via an egress endpoint.

Figure 1, VPC to VPC service consumption with Amazon VPC Lattice, illustrates this:

Figure 1, VPC to VPC service consumption with Amazon VPC Lattice

The thing to understand is that unless your consumer is deployed within a VPC that has been associated with the Service Network, whilst DNS resolution may be possible, connections will not be.

With that out of the way, what are some of the architectural considerations and requirements to make this work?

Architectural considerations

Discovery and Targets

VPC Lattice service names are globally unique and externally resolvable and each resolve to a series of link-local addresses. Whilst this DNS name can be used for the connectivity, it’s more likely that you will wish to use your own DNS names. VPC Lattice supports custom domain names for exactly this. As well as cleaning up the naming from the auto-generated namespace, using custom DNS names represents an opportunity to manipulate the resolution process through additional DNS layers. Take, for example, this VPC Lattice service name:

• myservice-02628e33.7d67968.vpc-lattice-svcs.us-west-2.on.aws

For external consumers, this DNS name resolves to a series of addresses in the 169.254.171.x/24 range (for consumers in a VPC, a single zonal IP response will be provided). If we masquerade behind a domain name within our control, perhaps by using a CNAME record (RFC1034), then we can create a more friendly and memorable entry point for the service:

• example.com

This manipulation in the resolution is helpful if, for example, we want to route external consumers to an ELB perhaps to an AWS Network Load Balancer (NLB) which could be used to connect to the Service Network.

External resolution for that might look something like this:

• example.com (CNAME)
  • nlb-long-name.elb.us-west-2.amazonaws.com
    • myservice-02628e33.7d67968.vpc-lattice-svcs.us-west-2.on.aws

In this way, external consumers use a friendly name that they can connect to, that name represents the name for the NLB, which then makes connections to the Service Network (effectively becoming a reverse proxy) and crucially, from within the VPC!

Consumers within the VPC don’t need to use a proxy, as they can connect directly to the Service Network, but they still may wish to continue using the CNAME. This then becomes a classical split-horizon DNS configuration when you use DNS to influence routing decisions, dependent on where your consumers or resolvers are located!

Internal resolution for that might look something like this:

• example.com (CNAME)
  • myservice-02628e33aa0cd333c.7d67968.vpc-lattice-svcs.us-west-2.on.aws

Layer 4 or 7?

When considering how to proxy external consumer connections into the VPC, one line of thought is how much interaction the proxy should have on the traffic. If you proxy the connections at layer 4 (transport layer) then you don’t need to know anything about the higher-level protocols that ride on that transport (TLS/HTTP) which means that you don’t need to manage any sort of state or encryption material – the proxy is simply routing connections, rewriting IP headers and forwarding the traffic on your behalf. If you choose to proxy at a higher level, say, layer 7 (application layer) then your proxy will need to handle HTTP negotiations as well as header management and origination of connections to the VPC Lattice services. For unencrypted workloads this is relatively trivial, however when TLS is required, your proxy will need to have the requisite Private Keys in order to decrypt traffic. The AWS NLB operates at layer 4 when using a TCP/UDP listener configuration, Application Load Balancer (ALB) is a layer 7 aware load balancer.

Traditional ingress methods?

Should I use multiple ingress patterns for each particular AWS service in my VPC Lattice deployment or should I build something custom to centralise this?

Often this is going to be a trade-off between cost, complexity, management, and features. Knowing that many of the compute services that will probably exist within your VPC Lattice environment will already have established ingress patterns is a pause for consideration – for instance – if you wanted to make an AWS Lambda (Lambda) function available to the outside world, you could use an ALB or Amazon API Gateway, for an Amazon EC2 instance perhaps just a load balancer – these are all valid patterns but you’d likely end up with many such ingress patterns to reach your services (especially as you grow). Much of VPC Lattice’s value is its ability to interconnect workloads independently of VPC boundaries, as well as simplified management and sharing of services.

Choosing to centralise ingress into the Service Network means that you can craft a single VPC that has a public and private entry-point from which you can reach all the VPC Lattice services, but it is by no means the only option available!

Native Services or Custom?

In order to use an ELB with VPC Lattice, IP targets must be used so there is a requirement to perform a DNS query operation, per AZ, per VPC Lattice service so that all possible endpoints are known. There is no native feature for this discovery or any such API call that you can use, so save for doing this manually, it could be achieved using some automation with AWS Lambda and supporting services for storing and tracking endpoints. When querying, note not all VPC Lattice services correspond to the same zonal IP addresses in the 169.254.171/24 or fd00:ec2:80::/64 ranges, this means that if the resolved IP addresses differ, additional ELBs (if using NLB) or more target groups (if using ALB and host-header based matching) would be required – this adds cost and complexity to the solution.

With regards to a purely native capability, there are a lot of steps to consider. A possible compromise here is to use a combination of an ELB for both public and private (hybrid) access and a layer 7 aware proxy fleet that sits behind it. This proxy fleet can provide a layer 4 TCP stream engine for TLS and a layer 7 HTTP engine for unencrypted traffic whilst dynamically discovering the corresponding link-local addressing for each VPC Lattice service based on either host headers or sni fields – this removes the need to deploy a discovery and mapping function.

Solution Overview

In the previous section, we discussed building an ingress solution based on an ELB, targeting a fleet of proxies. We have chosen NLB as the load balancer for this solution due to its high throughput characteristics and flexibility on protocol and feature support. ALB is another option and this load balancer can provide traffic inspection using services such as Web Application Firewall (WAF). As noted previously, this requires consideration of http negotiations and TLS certificate management. For the proxy layer, we have opted to use AWS Fargate (Fargate) serverless tasks running open-source NGINX, powered by Amazon Elastic Container Services (ECS).

The solution is published in aws-samples, where you can find more information about all the design considerations, a deeper explanation of the proxy fleet functionality, and detailed instructions about its deployment.

External Access

Figure 2, Ingress VPC Lattice service consumption – Internet connectivity, illustrates how traffic flows from external applications to VPC Lattice services when using this solution:

Figure 2, Ingress VPC Lattice service consumption – Internet connectivity

1. We have a R53 public hosted zone with our custom domain name (example.com) pointing to DNS domain name of the public NLB via a CNAME record.
2. Requests to our VPC Lattice service’s domain name from users outside AWS will be routed to the NLB and load balanced to the proxy fleet.
3. The proxy solution will perform a domain name look-up (example.com). A CNAME record in a Private Hosted Zone (PHZ) that points to the auto-generated VPC Lattice service name will ensure that the response is a zonal link-local address.
4. The request will be forwarded to the Service Network, which will redirect the request to the appropriate backend.

By adjusting the DNS resolution, we can provide external users with the consumption of VPC Lattice services. In this way, we can simplify network communication whilst exposing multiple services.

Hybrid Access

When working in hybrid scenarios, the pattern is similar. Two small distinctions of note:

• The NLB in the ingress VPC now can be an internal load balancer
• All DNS requests can be to private endpoint addresses

Figure 3, Ingress VPC Lattice service consumption – Hybrid connectivity, illustrates this.

Figure 3, Ingress VPC Lattice service consumption – Hybrid connectivity

1. Users in on-premises locations will query the domain name (example.com) in their local DNS servers. These servers will forward the DNS requests to a R53 inbound endpoint located in a VPC with a private hosted zone associated that translates this domain name to the NLB domain name (CNAME record).
2. Via a hybrid connection, the traffic will be redirected to the NLB that will load balance it to the proxy fleet.

Steps 3 and 4 remain unchanged from the external connectivity scenario.

This scenario follows the same principles as before with the Internet connectivity. DNS provides a different resolution depending the location of the DNS request. Take into account that you need to provide hybrid connectivity between your on-premises locations and both the ingress VPC and the VPC where you have the R53 inbound endpoint.

Consuming VPC Lattice services in other AWS Regions

Another use case that the ingress pattern permits is the consumption of VPC Lattice services from other AWS Regions. Service Networks are regional constructs, which means that consumers in different AWS Regions where this Service Network is located need this proxy service to jump to a “local” VPC.

Figure 4, Inter-AWS Region VPC Lattice service consumption, illustrates this.

Figure 4, Inter-AWS Region Lattice Service consumption

1. The consumer needs to resolve the Service domain name (example.com) to the remote NLB domain name. As this consumer is located within AWS, a private hosted zone containing this CNAME record can be directly associated to the VPC where the consumer resides.
2. Connectivity between the VPC in the remote AWS Region and the ingress VPC is needed, either by using VPC peering, AWS Transit Gateway (inter-Region peering), or AWS Cloud WAN.

Steps 3 and 4 remain unchanged from the external connectivity scenario.

Implementation guide

This solution is deployed in two parts:

First, deploy the baseline stack using the pipeline-stack.yml template in any AWS Region where you are publishing Amazon VPC Lattice services. More succinctly, you must deploy this stack as many times as you have distinct Service Networks in a region.

Once complete, your base solution is deployed, as illustrated in Figure 5, Base Solution.

Figure 5, Base Solution

After the baseline stack has been deployed, your CodePipeline will be waiting for you to release it. You are required to ‘Enable transition’ from the SourceStage stage to the BuildStage.

Figure 6, Pipeline Transition, Illustrates this:

Figure 6, Pipeline Transition

After you enable this transition, the pipeline will build the Fargate infrastructure and deploy the load balancers and containers.

Once complete, your ECS solution is deployed, as illustrated in Figure 7, ECS Fargate Solution

Figure 7, ECS Fargate Solution

Following this, you can now associate the ingress VPC with the Service Network you want.

In order to access your VPC Lattice services, you will need to configure DNS resolution as follows:

1. Create a private hosted zone and CNAME record to map each unique custom domain name to the VPC Lattice service generated domain name and attach this to the ingress VPC.
   • [public access] in addition, you will need a public hosted zone with a CNAME record pointing to the public NLB DNS name.
   • [hybrid access] In addition, you will need a private hosted zone attached to your hybrid DNS VPC or inter-Region VPC with a CNAME record pointing to the internal NLB’s DNS name.

A simple curl operation against your custom DNS name should allow communication with your VPC Lattice services – the following example uses the SigV4 signer switch as part of curl.

curl https://yourvpclatticeservice.name --aws-sigv4 “aws:amz:%region%:vpc-lattice-svcs" --user “$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" --header “x-amz-security-token:$AWS_SESSION_TOKEN" --header “x-amz-content-sha256:UNSIGNED-PAYLOAD"

(this is required if you have Auth Policies defined on either your Service Network of Service)

You can find an example that deploys a Service Network and a VPC Lattice service for you to test the solution. The example also creates the CNAME records explained above, but it does not create any R53 hosted zone.

Clean Up

Clean-up of this solution is straight-forward. First, start by removing the stack that was created by the AWS CodePipeline – this can be identified in the AWS CloudFormation console with the name %basestackname%-%accountid%-ecs. Once this has been removed, you can remove the parent stack that built the base solution. 

NOTE The ECR repo and the S3 bucket will remain and should be removed manually.

Solution Considerations

This solution provides layer 3 IP control using a combination of an ipcontrol.conf file, loaded by the NGINX configuration and the Proxy Protocol feature of NLB. The default settings permit all RFC1918 addresses. Be sure to update this file to permit/deny access from additional addresses. Moreover, you can use Network Access Control Lists (NACLs) in the subnets where you place the NLBs and protect them directly using Security Groups. For SigV4 signing, you must have access to AWS security credentials. For clients that reside outside of AWS, consider utilizing IAM Roles Anywhere in order to get these credentials.

Conclusion

In this blog post, we have reviewed the architecture considerations for building an ingress proxy solution for Amazon VPC Lattice. We proposed and provided a fully automated solution using NLB, targeting a fleet of lightweight serverless proxies on AWS Fargate. This wasn’t an exhaustive list of all the possible ingress solutions that you could use to achieve this.

You can check this solution in the following aws-samples GitHub repository.

Visit the Amazon VPC Lattice product page, documentation, and pricing page for additional information.

About the Authors