Build secure multi-account multi-VPC connectivity for your applications with Amazon VPC Lattice

Introduction

In this blog post, we will discuss how you can use Amazon VPC Lattice to connect your services securely, and monitor communication flows, in a simple and consistent way across instances, containers, and serverless, in a multi-account and multi-Virtual Private Cloud (VPC) environment. We’ll define the new constructs VPC Lattice leverages to enable application networking, explore key design considerations, and walk you through a step-by-step setup of the service.

Customers are modernizing their applications from large tightly coupled monoliths into modular components, called microservices, because it helps them build and deploy software faster. This move to microservices has sped up the development process, but also resulted in single applications now containing many more individual components that have to communicate with one another. To enable these new communication patterns, developers must build custom application code to handle networking tasks, such as service discovery, traffic routing between individual services, authorizing access, and providing detailed metrics and logs to assist with troubleshooting, which can be complicated and time consuming. Alternatively, developers can implement an architectural pattern called a service mesh, that places a proxy alongside each service, to offload some of these tasks from the application code. However, this extra software developers have to operate and maintain also adds complexity. In either approach, developers still have to manage network connectivity, security, and monitoring between service dependencies, which requires an understanding of networking concepts that are often outside their expertise. Larger organizations usually have dedicated network or cloud administrator teams to manage network connectivity across multiple accounts and VPCs, and security teams to ensure that proper authentication, authorization, and encryption are enforced. This strategy can lead to a disjointed experience between admins and developers. Admins have to configure and maintain connectivity and security at the network layer, without having visibility into application dependencies and access patterns. In parallel, whether writing custom code or using a service mesh, developers need to reason about how the underlying network architecture affects their business logic.

Amazon VPC Lattice is a fully managed end-to-end application networking service that is purpose built to help solve these challenges. It simplifies the onboarding experience for developers by removing the need to implement custom application code, or run additional proxies next to every workload, while maintaining the tools and controls admins require to audit and secure their environment. VPC Lattice works universally across Elastic Compute Cloud (EC2) instances, containers, and serverless compute, automatically handling service connectivity across VPCs. It provides native service discovery, and a broad set of application layer proxy and load balancing functionalities, to support common deployment patterns like blue-green and canary style roll-outs. VPC Lattice also facilitates an improved security posture with reliable authentication and context-specific authorization, powered by AWS Identity and Access Management (IAM), allowing customers to implement a defense in depth strategy with rich traffic controls, and end-to-end observability.

Before we dive into setting up Amazon VPC Lattice, let us first define the core components.

Amazon VPC Lattice Components

• Service: A VPC Lattice service represents a customer-owned application that can extend across all compute options–EC2 instances, containers, and serverless. VPC Lattice services consist of listeners (protocol and port number), routing rules that allow you to control application flows (for example, path, method, header-based, or weighted routing), and one or more target groups, which define your application infrastructure. Once you create a VPC Lattice service, you can choose to share it with accounts within or outside of your AWS Organization, using AWS Resource Access Manager (RAM).
• Service network: A VPC Lattice service network is a new logical grouping mechanism that simplifies how you can enable connectivity across VPCs or accounts and apply common security policies for service-to-service communication. You can associate VPC Lattice services and VPCs with service networks to facilitate connectivity. You can create a service network in an account and share it with accounts within or outside of your AWS Organization using AWS RAM.
• Service directory: A service directory is a list of all VPC Lattice services you create locally within an AWS account, together with any VPC Lattice services shared with your account using AWS RAM. You can associate VPC Lattice services in your service directory with one or more VPC Lattice service networks.
• Auth policies: VPC Lattice auth policies are IAM policy documents that you attach to service networks or services to control whether a specified principal has access to a group of services or to a specific service. You can attach an auth policy to each service network or service that you want to control access to. VPC Lattice auth policies are AWS IAM resource policies. They provide request-level authentication and context specific authorization for service access requests.

Getting started with VPC Lattice

Let’s explore some Amazon VPC Lattice design considerations, and how it brings together developers and admins to meet the connectivity, security, and application management needs of a common service-oriented deployment platform.

Administrative model and resource ownership — who does what?

Developers, or service owners, can create VPC Lattice services in their AWS accounts, fully controlling their listeners, routing rules, and target groups. They can also configure their service-level auth policies by defining IAM resource policies to control access granularly from other services in the environment. Then, they can share VPC Lattice services with other accounts, and associate them with VPC Lattice service networks. Finally, they can configure monitoring and observability for their VPC Lattice services, using service access logs and target group metrics.

Network or infrastructure admins can create and manage VPC Lattice service networks, configure service network level auth policies, and monitoring. Admins can choose to configure auth policies in a more generic, coarse-grained manner, to enforce foundational security controls. For example, as a network admin, you can write a coarse-grained policy that allows only authenticated requests from principals in your AWS Organization through the service network, and give service owners fine-grained control over their service-specific auth policies. Admins can associate VPCs and VPC Lattice services with one or more service networks. Alternatively, depending on the chosen operational model, admins can give service owners control to associate their VPCs and services by sharing VPC Lattice service networks using AWS RAM.

Service discovery

Amazon VPC Lattice leverages a DNS-based service discovery model. Each VPC Lattice service, when created, receives a unique global DNS name which is managed by VPC Lattice in Amazon Route 53 public hosted zones. If you prefer to use your own domain names, you can reference the VPC Lattice managed domain names as CNAME or Alias records in Route 53. VPC Lattice manages SSL/TLS certificates for HTTPS services, or you can use your custom certificates from AWS Certificate Manager (ACM). You can find more details about configuring HTTPS VPC Lattice services and managing custom DNS names in the documentation.

VPC Lattice services, services networks, and clients

In a multi-account, multi-VPC environment, you may deploy one or more service networks to support your AWS account structure and administrative model. Each VPC can be associated with one service network, while each VPC Lattice service can be associated with multiple service networks, so you have the flexibility of meeting various needs and levels of access segregation. For example, you may have production and development service networks, and organization-wide shared VPC Lattice services associated with service networks, to allow consistent client access. Alternatively, for sandbox environments, each VPC can have its own service network, with a limited set of associated services and fine-grained auth policies.

VPC Lattice is a Regional AWS service, therefore clients and services are deployed in VPCs in AWS accounts within the same AWS Region. Clients of VPC Lattice services can be deployed across all available compute platforms, for example EC2 instances, Amazon ECS, Amazon EKS, AWS Fargate, or AWS Lambda. Similarly, you can combine multiple compute types for the same VPC Lattice service, which can help during migration scenarios. Once an Amazon VPC is associated with a service network, the resources deployed in that VPC can access VPC Lattice services associated with the service network, provided the security controls allow it. Note that VPCs hosting VPC Lattice services don’t need to be associated with a service network for communication to be established.

Security controls and defense in depth

You can create a defense in depth strategy by leveraging VPC Lattice Auth policies, in addition to existing network layer controls, such as security groups and network access control lists. When you associate a VPC to a service network, you can apply a security group on the association, and use security group referencing within the VPC to control network level access granularly. For example, for certain VPC resources, you can allow access to the associated service network only, or you can deny access to the service network altogether.

Additionally, VPC Lattice enables you to configure and manage application-level security controls consistently. Both VPC Lattice service and service network auth policies provide strong and reliable authentication and context-specific authorization. VPC Lattice auth policies are IAM policy documents that you attach to your service networks or services, to control whether a specified principal (service) has access to another specific service or group of services. VPC Lattice auth policies use the same syntax as IAM resource policies, and they rely on the following elements: Principal (i.e. an IAM entity), Effect (i.e. Allow or Deny), Action (i.e. vpc-lattice-services:Invoke), Resource (i.e. services affected by the action), and optional Conditions (i.e. when your policy will be in effect). Once a principal (service) authenticates (or signs) a request to a service, that request has to be allowed by the security groups configuration at a network level, the service network auth policies and the service auth policies.

Network connectivity with VPC Lattice

Amazon VPC Lattice enables you to create consistent and secure service-to-service communication across multiple VPCs and accounts, without requiring additional connectivity to be provided by services like VPC peering, AWS PrivateLink or AWS Transit Gateway. To achieve this, VPC Lattice operates in the link-local address space in IPv4 and Unique Local Address space in IPv6, outside of the IP addresses you assign to your VPCs. The DNS records for VPC Lattice services resolve to IP addresses form 169.254.0.0/16 for IPv4, and fc00::/7 for IPv6. The VPC association with the service network provides connectivity to the link-local and unique local IP space. VPC Lattice maintains a managed prefix list for IPv4 and for IPv6 in each account. You can reference these prefix lists in the security group on the VPC association with the service network, and in the security groups associated with your VPC resources. An important outcome of VPC Lattice using link-local IPv4 and IPv6 addresses is that you can use it to provide connectivity between services that may have overlapping IPv4 addresses. You can use VPC Lattice for connectivity between clients supporting only IPv4 addresses and services deployed on IPv6, or vice-versa.

Monitoring and observability

Amazon VPC Lattice provides granular access logs for service-to-service interactions. You can choose to send these logs either to an Amazon S3 bucket, Amazon Kinesis Data Firehose, or to an Amazon CloudWatch log group. Developers can choose to save and view the access logs specific for their services, and admins can save and view the access logs at the service-network level. This provides a mechanism for both admins and service owners to get the required level of logging detail for their needs. We show the snapshots of the logs towards the end of step-by-step walkthrough section below.

Now that we reviewed some of the design considerations, let’s get started with setting up Amazon VPC Lattice in our test environment.

Prerequisites

For the following sections, we assume you are familiar with creating and using AWS IAM policies and with fundamental AWS networking services, such as Amazon VPCs and Amazon Route 53 DNS configurations for private and public hosted zones. We also assume that you are familiar with deploying services and applications using compute platforms such as Amazon EKS, EC2, and AWS Lambda functions. We won’t focus on the detailed steps for deploying infrastructure and services, so we assume you will deploy the following setup in your test environment. Our setup makes use of four services that need to be pre-deployed:

• Two container-based services, App1 and App2, deployed across two separate Amazon EKS clusters, in two separate accounts and VPCs. Here are the detailed instructions on how to set up your own EKS clusters, and the automation templates to deploy the two services, each in its own EKS cluster. We chose the same overlapping IPv4 CIDR block (10.1.0.0/16) for the two VPCs hosting App1 and App2 (VPC-1 and VPC-2), to highlight how you can use VPC Lattice even if you have overlapping IP space within your environment.
• One Lambda-based service, App3, deployed in a third account and VPC. Here are the detailed instruction for deploying an AWS Lambda function in a VPC, and App3 Python code. Note that it’s not a requirement to deploy Lambda in a VPC to use it with VPC Lattice.
• An EC2-based service, App4, deployed using an EC2 Auto Scaling group, in the same account as App3. Here are the detailed instruction for setting up an EC2 Auto Scaling group, and App4 deployment code.

Figure 1 below details the applications deployed across VPCs and accounts, in the test environment in the us-west-2 AWS Region:

Figure 1: VPC Lattice test environment in the us-west-2 AWS Region

We’ll use an infrastructure-admin account 4 for creating a service network, and for performing operational or debugging related tasks, such as exploring VPC Lattice service network access logs. All accounts are part of the same AWS Organization, although this is not a requirement for the setup.

Amazon VPC Lattice setup

We assume you have set up the VPCs and the corresponding services shown in Figure 1 above. In this section, we’ll discuss how to provide secure connectivity and observability for all four services using VPC Lattice.

Service communication flows

Before we configure VPC Lattice, let’s define access patterns for the four services in the environment through a service connectivity matrix. This is an important step in architecting secure connectivity with VPC Lattice, given that associating a VPC to a service network enables clients in the VPC to access associated services, while associating services to a service network allows them to be accessed by clients in associated VPCs. So, for us to know which VPCs and services we need to associate with our service network, and how to configure auth policies, we need a service connectivity matrix. Figure 2 below highlights the access patterns for the four apps:

Figure 2: Services communication matrix

• App1 and App2 need to be able to communicate with each other bidirectionally
• App1, App2 and App4 need to be able to access App3

Let’s now walk through the step by step set up of Amazon VPC Lattice:

Step 1 – Create service network in the infrastructure account (admin)

First step is to create a VPC Lattice service network, named lattice-sn, in the infrastructure account (account 4).

Figure 3: VPC Lattice service network lattice-sn

The auth policies for the service network allow only authenticated traffic originated from IAM principals that belong to the AWS Organization. We will apply a more granular auth policy when configuring VPC Lattice services. Figure 4 below shows the service network lattice-sn overview and auth policy configuration:

Figure 4: VPC Lattice service network auth policy

For monitoring at the service network level, we enable logging export to a CloudWatch log group vpc-lattice-sn-logs, as shown in Figure 5 below:

Figure 5: VPC Lattice service network monitoring configuration

The documentation details the service network creation steps. Our service network doesn’t yet have any associated VPCs or services, so let’s proceed with the next step, creating VPC Lattice services:

Step 2: Create VPC Lattice services (service owners)

After you followed the instructions in the Prerequisites section, you with have the four applications deployed in their respective accounts in your environment. We followed the same steps, so now that we have the applications deployed, we need to create each service in VPC Lattice. For this, we are leveraging the VPC Lattice native integration with Amazon EKS, EC2 and Lambda. App1, App2, App3 and App4 map to VPC Lattice Service1, Service2, Service3 and Service4 respectively.

For EKS-based services, Amazon VPC Lattice integration is achieved through the AWS Gateway API Controller, an implementation of the Kubernetes Gateway API. This allows you to discover natively and configure services deployed across EKS clusters, and associate them with VPC Lattice service networks. Check out the documentation and this blog, for more details on how you can make use of the AWS Gateway Controller with your Kubernetes workloads.

Amazon EC2 Auto Scaling also natively supports Amazon VPC Lattice. You can associate an instance-type VPC Lattice target group with an Auto Scaling group as show in Figure 6, below:

Figure 6: EC2 Auto Scaling integration with VPC Lattice.

AWS Lambda also supports Amazon VPC Lattice as a native trigger. For more details, check the documentation. You can use AWS CloudFormation templates to create the VPC Lattice services corresponding to App1, App2, App3 and App4.

After executing the templates, you will have the following account-level view of the VPC Lattice service directory:

Figure 7: VPC Lattice services in each account

For granular service-level access control, each VPC Lattice service needs to be configured with an auth policy that reflects the service communication and access patterns highlighted in Figure 2. Let’s dive deeper into the auth policy for each of the four services.

Service1 auth policy — Allows Service2 authenticated traffic to reach Service1, while it denies all other requests. The policy components detailed in Figure 8 below are:

• Principal: IAM Role assumed by Service2, which is the IAM Role associated with EKS-Cluster-2, deployed in Account-2.
• Action: vpc-lattice-svcs:Invoke, which defines the ability to invoke a VPC Lattice service.
• Resource: ARN of VPC Lattice Service1.
• (Optional) Condition: Helps ensure requests are originating from the expected VPC

Figure 8: Service1 auth policy

Service2 auth policy — Similar to Service1’s auth policy discussed above, with the difference in Principal and Resource ARNs, and no condition related to source VPC. Figure 9 below shows the IAM policy associated with Service2.

Figure 9: Service2 auth policy

Service3 auth policy – Allows Service1, Service2 and Service4 to invoke Service3, as detailed in Figure 10 below:

Figure 10: Service3 auth policy

Service4 auth policy — Given that in our setup Service4 does not need to be accessed by any other service, we have defined no specific IAM policy, and we are relying on the implicit denial of an empty policy. Figure 11 below details the setup:

Figure 11: Service4 auth policy

For monitoring and observability, we configure each service to export access logs and target group metrics to a CloudWatch log group in its account.

Now that we have VPC Lattice services and a service network, let’s bring them together in the multi-account setup, using AWS RAM.

Step 3: Share VPC Lattice service network with service owner accounts (admin)

Depending on an organization’s operational model, admins can share service networks, and service owners may choose to share VPC Lattice services with accounts owning service networks. This gives you the flexibility to decide which team has control over associating services, and VPCs to service networks – either admins, service owners, or both. To further control the actions taken by service owners or admins in their accounts, you can configure AWS Organizations Service Control Policies (SCPs). You can find more information about SCPs and examples in the documentation.

For our setup, we will walk you through one approach: sharing the service network with service owner accounts. This is required, as part of the AWS Gateway API Controller integration, to enable cross-account association of EKS-based VPC Lattice services with the VPC Lattice service network, by allowing the EKS-cluster owner to create the Kubernetes Gateway object. This allows cross-account association of VPCs with the service network.

In account 4, we create an AWS RAM resource share for the VPC Lattice service network, and we add accounts 1, 2 and 3 as Principals, as shown in Figure 12 below:

Figure 12: AWS RAM resource share for the VPC Lattice service network

When sharing a VPC Lattice service network, Principals have permissions to create service network associations for VPC Lattice services and VPCs, as shown in Figure 13 below:

Figure 13: AWS RAM permissions for VPC Lattice service network resource share

Step 4: Associate services with the service network (service owners)

Once we have shared the VPC Lattice service network lattice-sn with service owner accounts, we can associate VPC Lattice services with it, based on the service access patterns defined above:

• Service1 and Service2 need to communicate with each other bidirectionally, so both need to be associated with the service network.
• Service1, Service2 and Service4 need to access Service3, so Service3 needs to be associated with the service network.
• Service4 does not need to be accessed by other services, so it doesn’t need to be associated with the service network.

Figure 14 below shows the service network association for each service:

Figure 14: VPC Lattice services associated with the VPC Lattice service network

The green arrows in Figure 10 above show that VPC Lattice services cab be accessed through the service network. We don’t yet have any VPCs associated with the service network, so no clients to access VPC Lattice services. Let’s proceed to the next step, and associate VPCs with the service network.

Step 5: Associate VPCs with the service network (admin)

When deciding which VPCs need to be associated with the service network, we reference again our service communications matrix:

• Service1 and Service2 need to communicate with each other bidirectionally, so VPC-1 and VPC-2 need to be associated with the service network.
• Service1, Service2 and Service4 need to access Service3, therefore VPC-4 also needs to be associated with the service network.
• Service3 does not need to access other services, therefore VPC-3 doesn’t need to be associated with the service network.

Figure 15 below highlights VPCs associated with the service network, through the added blue arrows:

Figure 15: VPC associations with the VPC Lattice service network

VPC-level micro-segmentation with Amazon VPC Lattice

For the granular VPC-level network security controls, we are configuring on each VPC association with the service network, a security group that references the VPC Lattice managed prefix lists for IPv4 and IPv6. Amazon VPC Lattice manages the link-local IPv4 and IPv6 prefixes communication at an account level, in two managed prefix-lists as shown in Figure 16 below, and may be different across accounts.

Figure 16: Amazon VPC Lattice managed prefix-lists

Once VPCs are associated with the service network, we also need to update the security groups associated with resources in VPCs, to allow traffic flows to VPC Lattice services. To do this, you can reference the security group on the VPC association in the resource security group.

VPC routing with Amazon VPC Lattice

The route tables of the VPCs associated with the service network are automatically updated with the VPC Lattice-managed link-local addresses, as highlighted in Figure 17 below:

Figure 17: Route table of VPC associated with a VPC Lattice service network

Amazon VPC Lattice traffic flows

Now that we have completed the configuration of VPC Lattice services, service network, and auth policies, we can dive deeper into how traffic flows between services in the environment. Figure 18 below shows communication flow from VPC Lattice Service1 to Service2, through the VPC Lattice service network lattice-sn:

Figure 18: Service1 to Service2 request packet flow with VPC Lattice

1. Service1 resolves the DNS name of Service2, assuming it does not already have the entry cached locally. It generates a DNS query and sends it to the Amazon Route 53 VPC Resolver.
2. The Route53 VPC Resolver performs a recursive DNS resolution. Check documentation for more details on how Route 53 VPC Resolver works.
3. The Route53 VPC Resolver replies with the link-local and/or unique local IP address mapped to VPC Lattice Service2.
4. Service1 generates a SIGv4 signed request to Service2. The source IP address of the request is the IP address Service1 has in the VPC, for example 10.1.1.12. The destination IP address of the request is the link-local address received from the VPC Resolver, for example 169.254.171.23. The IP packet needs to be permitted by the security group configured on the EKS worker node, and the Network ACL configured on the worker node subnet. The packet is then routed based on the subnet route table to VPC Lattice. Remember that VPC Lattice automatically populates the route tables of VPCs with the necessary IPv4 and IPv6 routes, when they are associated with a service network
5. VPC Lattice performs all configured application layer and security functionalities, both for the service network and for the service. VPC lattice applies authentication, authorization, and traffic controls at the service network and service levels.
6. VPC Lattice sends traffic to the configured service target. The packet will have an IP source from the link-local managed prefix, for example 169.254.171.23 and an IP destination of Service2 target in VPC-2, for example 10.1.1.87.

For testing we used HTTPie with a custom auth plugin called lattice. This plugin adds the IAM credential information (via SIGv4 signing) to the requests, and VPC Lattice then verified these credentials using Lattice auth policies. Figure 19 below shows Service1 calling Service2:

Figure 19: Service1 accessing Service2

The VPC Lattice service access logs in Figure 20 below show detailed information about the request from Service1 to Service2 through the service network:

Figure 20: VPC Lattice lattice-sn service network access logs

The same information is shown in the Service2 access logs, in Figure 21 below, for the request from Service1 to Service2:

Figure 21: VPC Lattice Service3 access logs

Both service access logs and service network access logs include information regarding ARNs of the VPC Lattice service and service network, the requestMethod and requestPath. Additionally, the logs include the ID of the IAM role/user that signed the request. In case a request was denied, the authDeniedReason field shows whether the request was denied at the service network level, or the service level. This provides service owners and admins a consistent way to verify whether their configuration works as intended, and audit the setup.

Cleanup

If you used the CloudFormation templates that we provided to create AWS resources to follow along with this blog post, we recommend you delete them now to avoid future recurring charges. Check out the documentation for how to remove CloudFormation templates in your account.

Conclusion

In this blog post, we discussed how you can use Amazon VPC Lattice to provide secure and scalable connectivity between your services in a multi-account and multi-VPC environment, together with end-to-end visibility and observability. We reviewed the VPC Lattice components, design considerations, and a step-by-step guide on how to configure the service. Simplify the way you connect, secure, and monitor service-to-service communication with Amazon VPC Lattice. If you have questions about this post, start a new thread on AWS re:Post, or contact AWS Support. For more information about Amazon VPC Lattice, you can refer to the documentation and additional blogs.

About the Authors