Leveraging App Mesh with Amazon EKS in a Multi-Account environment

Today, many customers are adopting microservices to develop software faster, enable innovation, and accelerate the time-to-market for new features and products. The microservices approach is the implementation of small independent pieces of software that address specific business needs and communicate over well-defined APIs. In order to implement this development model, larger organizations are changing the way services are deployed and implementing a different strategy that uses multiple AWS accounts to run their workloads. This approach reduces the blast radius of having a single environment with all the applications from different teams and business units, but it also requires microservices across accounts to be able to talk to each other.

It is important to say that the network communication is not the only thing that needs more attention when implementing microservices. New tasks that usually consume time and can slow down the innovation process (mainly across cloud setup and governance) will occur and services such as AWS Organizations and AWS Control Tower can help facilitate the setup and governance of multi-account AWS environments.

When implementing microservices, it is very important to have the control over the communications. In a multi-account setup, there is also a need to have visibility and enforce network policies between the applications that are running in the environment. Here is where AWS App Mesh is important.

AWS App Mesh is a service mesh that provides application-level networking to make it easy for your services to communicate with each other not only across multiple types of compute infrastructure but also across different AWS accounts. App Mesh standardizes how your services communicate, giving you end-to-end visibility, and ensuring high-availability for your applications. Having a cross-account mesh increases the reliability and security of intercommunicating services, using features like mutual TLS, connection retry, and timeout handling without changing your application. This helps you to reduce the complexity of the code.

Tutorial

In this post, we are going to show how to leverage AWS App Mesh capabilities to integrate different components of an application running across different EKS clusters in different AWS accounts.

This can be applied if you are creating the entire stack from scratch (as we show bellow) or if you already have different EKS clusters running in a multi-account environment and need to implement the service mesh across them. If your case is the latter, where you already have the infrastructure provisioned, you can start it from the step 3 – Install the App Mesh components.

We will use two EKS clusters, each in its own account, and the network connectivity between these two clusters will be made through a VPC Peering connection. This approach allows us to keep the environments separated while providing connectivity between the two. Besides implementing the VPC Peering connectivity, we will use the AWS Resource Access Manager (AWS RAM) to share the mesh across both accounts so it will be visible to both EKS clusters.

In this tutorial, you will create AWS App Mesh components and deploy them using a sample application called Yelb. Yelb allows users to vote on a set of alternatives like restaurants and dynamically updates pie charts based on the votes. Additionally, Yelb keeps track of the number of page views and prints the hostname of the yelb-appserver instance serving the API request upon a vote or a page refresh. Yelb components include:

A frontend called yelb-ui is responsible for vending the JS code to the browser.
An application server named yelb-appserver, a Sinatra application that reads and writes to a cache server (redis-server), and a Postgres backend database (yelb-db).
Redis stores the number of page views and Postgres stores the votes.

The following picture shows our architecture.

There are two EKS clusters, each in its Account and VPC and a mesh spanning both Accounts. The yelb-ui component will be deployed in Cluster 1 in our Frontend Account and the yelb-appserver, redis and postgres database will be deployed in Cluster 2 in our Backend Account. The goal is to have a single mesh across the accounts.

Note that in this deployment, we are not deploying any load balancers to handle the communication between the Yelb services. Instead, we will be leveraging the client-side load balancing capabilities provided by App Mesh. With client-side load balancing, the client directly connects to a service instance, which removes an extra hop. Clients discover services by querying a service registry that maintains a list of active instances of services along with their corresponding details, such as IP addresses, hostnames, and ports. These service discovery capabilities will be provided by the integration between AWS App Mesh and AWS Cloud Map.

1. Set up the infrastructure:

To follow along, you will need to have an environment with some tooling. An AWS Cloud9 instance can be used to run this tutorial. If you want to create a Cloud9 instance in your account, follow the steps in the EKS Workshop from the chapter Create a Workspace to Update IAM Settings for your Workspace.

These are the tools we will need to have installed:

The latest version of the AWS CLI
Docker
Git
jq
Helm
eksctl
kubectl

Since we will be running this in two different accounts, you need to make sure that you have profiles with credentials for both AWS Accounts configured in ~/.aws/credentials and ~/.aws/config files. For example:

cat ~/.aws/credentials

[frontend]
aws_access_key_id = ...
aws_secret_access_key = ...
[backend]
aws_access_key_id = ...
aws_secret_access_key = ...

cat ~/.aws/config

[profile frontend]
region = us-west-2
[profile backend]
region = us-west-2

You can get more information about creating configuration files in the AWS CLI user guide.

After installing everything and configuring the profiles, let’s clone the GitHub repository that has the scripts that we will be using in this tutorial:

git clone https://github.com/aws/aws-app-mesh-examples.git
cd aws-app-mesh-examples/walkthroughs/eks-app-mesh-multi-account/

We will use CloudFormation templates to create the VPC, Security Groups, Route Tables, and the VPC Peering connection between the two accounts, if you want to take a look at them, they’re located in the infrasctructuredirectory of the repo you just cloned. You can create them by running the following script:

./infrastructure/setup.sh

Now that you have the infrastructure set up, it’s time to create our clusters.

2. Create the EKS clusters

In this step, we will use eksctl to create the EKS clusters in the Frontend and Backend accounts by running the following script:

./eks/setup.sh

You can see your new clusters by running kubectl.

kubectl config get-contexts

3. Install the App Mesh components

Before creating the mesh resources and deploying the Yelb app onto the clusters, you need to install the AWS App Mesh Controller. This controller allows you to configure App Mesh resources using kubectl. You will use Helm to install the App Mesh controller.

First, install the controller in the Frontend Account’s cluster.

kubectl config use-context <iam_user>@am-multi-account-1.<region>.eksctl.io
helm repo add eks https://aws.github.io/eks-charts
kubectl create ns appmesh-system
helm upgrade -i appmesh-controller eks/appmesh-controller \
--namespace appmesh-system

Now, install the controller in the Backend Account’s cluster.

kubectl config use-context <iam_user>@am-multi-account-2.<region>.eksctl.io
kubectl create ns appmesh-system
helm upgrade -i appmesh-controller eks/appmesh-controller \
--namespace appmesh-system

Confirm that the App Mesh controller is running:

kubectl --context=<iam_user>@am-multi-account-1.<region>.eksctl.io get pods -n appmesh-system
kubectl --context=<iam_user>@am-multi-account-2.<region>.eksctl.io get pods -n appmesh-system

You should see outputs similar to this one:

NAME                                READY STATUS   RESTARTS AGE 
appmesh-controller-66b749c78b-67n68 1/1   Running  0        6s

4 – Deploy and share the mesh

Now that you have the App Mesh controller set up in both clusters, you want to create the mesh and share it to the Backend Account.

First, let’s create the namespace where you will deploy the mesh components and the Yelb application in the Frontend Account’s cluster.

kubectl config use-context <iam_user>@am-multi-account-1.<region>.eksctl.io
kubectl create ns yelb

You need to add two labels to the yelb namespace: mesh and appmesh.k8s.aws/sidecarInjectorWebhook. These labels instruct the controller to inject and configure the Envoy proxies in the pods:

kubectl label namespace yelb mesh=am-multi-account-mesh
kubectl label namespace yelb "appmesh.k8s.aws/sidecarInjectorWebhook"=enabled

Let’s create and share the mesh and share it to the Backend Account:

./mesh/create_mesh.sh

Share the mesh using AWS Resource Access Manager:

aws --profile frontend cloudformation deploy \
--template-file shared_resources/shared_mesh.yaml \
--parameter-overrides \
"BackendAccountId=$(aws --profile backend sts get-caller-identity | jq -r .Account)" \
--stack-name am-multi-account-shared-mesh \
--capabilities CAPABILITY_IAM

Now, accept the Resource Share Invitation in the Backend Account.

Run the following commands to accept the invitation:

RESOURCE_SHARE_ARN=$(aws --profile backend ram get-resource-share-invitations \
| jq -r '.resourceShareInvitations[] | select(.resourceShareName=="mesh-share") | .resourceShareInvitationArn')

aws --profile backend ram accept-resource-share-invitation \
--resource-share-invitation-arn $RESOURCE_SHARE_ARN

The Backend Account needs a service role so App Mesh can discover yelb-appserver VirtualNode’s ip address using AWS Cloud Map, create it:

aws --profile backend iam create-service-linked-role \
--aws-service-name appmesh.amazonaws.com (http://appmesh.amazonaws.com/)

5 – Deploy the mesh resources and the application

Since the resources in the Backend Account are dependences for the resources in the Frontend Account, let’s start by creating them:

Switch to the Backend Account’s cluster context:

kubectl config use-context <iam_user>@am-multi-account-2.<region>.eksctl.io

Even though the mesh is already created and shared, the App Mesh controller in the Backend Account’s cluster needs to be aware of it, let’s run the following script:

kubectl create ns yelb
kubectl label namespace yelb mesh=am-multi-account-mesh
kubectl label namespace yelb "appmesh.k8s.aws/sidecarInjectorWebhook"=enabled
./mesh/create_mesh.sh

Now, create the Mesh components in the Backend Account, this will create VirtualNodes, VirtualServices, and VirtualRoutes for yelb-appserver, yelb-db and yelb-redis.

kubectl apply -f mesh/yelb-redis.yaml
kubectl apply -f mesh/yelb-db.yaml
kubectl apply -f mesh/yelb-appserver.yaml

Deploy yelb-appserver, yelb-db and yelb-redis.

kubectl apply -f yelb/resources_backend.yaml

Get the yelb-appserver VirtualService ARN and change mesh/yelb-ui.yaml backend configuration accordingly. This is a manual process, yelb-ui’s VirtualNode creation will fail if skipped.

kubectl -n yelb get virtualservice yelb-appserver

# Replace <yelb-appserver virtualServiceARN>
# in mesh/yelb-ui.yaml with the ARN from the 
# command above using your editor of choice

Now that you have all the Backend Account components deployed, switch to the Frontend Account’s cluster.

kubectl config use-context <iam_user>@am-multi-account-1.<region>.eksctl.io

Create the mesh components in the Frontend Account, this will create a VirtualNode, VirtualService, and a VirtualRoute for yelb-ui.

kubectl apply -f mesh/yelb-ui.yaml

Deploy yelb-ui.

kubectl apply -f yelb/resources_frontend.yaml

Get the the Load Balancer URL to test the application in your browser:

kubectl get service yelb-ui -n yelb

You should see the application as follow, go ahead and test it!

Using the AWS Console, notice that your App Mesh resource is seamlessly integrated into the architecture using AWS Resource Access Manager and its Virtual Services, Routers, and Nodes owners can be easily identified:

6. Cleanup

In order to clean up all the resources created during the execution of this tutorial, run the cleanup script with the following command:

./cleanup.sh

Note that this cleanup script won’t delete any AWS accounts created for this tutorial, but only the resources provisioned in it by the previous steps.

Summary

In this post, we created two EKS clusters (each in its own account and VPC) and a frontend service in one of the clusters and the backend resources in the other one. We used a mesh that spanned both clusters allowing us to to simplify the communication between those services while giving us the possibility of adding features such as mutual TLS, connection retry, and timeout handling without changing the application code. With this approach, customers that rely in multi-account and multi-cluster strategies can start managing the connectivity between these environments in a better way and taking advantages of all the benefits provided by a service mesh.

Also note that multi-account is just one of the features of App Mesh. You can use App Mesh to perform canary deployments, implement CI/CD pipelines, distributed tracing, encrypt the traffic between your services, and more. If you want to learn more about AWS App Mesh, you can also run the AWS App Mesh workshop.

If you are interested in trying out new App Mesh features, sign up for App Mesh Preview Channel. We would love to hear your feedback.

Containers