Traffic Encryption in AWS App Mesh across accounts using certificates from AWS Certificate Manager Private CA and AWS Resource Manager
AWS App Mesh is a service mesh that provides application-level networking to standardize how your services communicate, giving you end-to-end visibility and enabling controls to tune for high-availability of your applications. Customers building mesh architectures and enforcing the use of TLS to validate the certificate from the upstream service, is an important aspect of a zero-trust network. Using this feature, traffic is guaranteed to be encrypted before it ever leaves your compute and you’re able to validate that the upstream destination is trusted to be who they say they are. App Mesh provides an integration with AWS Certificate Manager (ACM) Private Certificate Authority (PCA), to enable TLS between connected services in a manner that does not require changes to application code.
Customers often have applications owned by different accounts in different Amazon ECS clusters. App Mesh makes it easy to connect all services into a common mesh and enable configuration of encryption via the mesh. With the support of the Cross-Account feature of AWS Certificate Manager (ACM) Private Certificate Authority (CA), customers can create a CA in one account and use AWS Resource Access Manager (RAM) to share their CAs across the enterprise. This helps manage the overall cost of CA infrastructure, when the same organization is deploying services using separate AWS accounts. You can find more details about this feature here.
- Complete managed renewal capability
Each account the Private CA is shared with that creates a private certificate in its own account, remains in full control of the certificate because the key lives in the certificate creation account and is fully protected there. The certificate creation account can associate the certificate to any resource in their account or export it for further use. Each certificate that is created also has the complete managed renewal capability of ACM. By issuing private certificates via ACM, customer certificates are automatically renewed, preventing certificate based outages.
- Cost Savings
Private CA cross-account sharing, gives you the ability to grant permissions for other accounts to use a centralized CA to generate and issue certificates by using AWS Resource Access Manager (RAM) to manage the permissions. In a configuration with a CA in each account, customers must pay a monthly fee for each CA. Using the shared CA model, customers only pay for the one CA that they are sharing, regardless of the number of to which accounts they share.
In this tutorial, you will create AWS App Mesh components and deploy them using a sample application called Yelb. Yelb allows users to vote on a set of alternatives like restaurants and dynamically updates pie charts based on the votes. Additionally, Yelb keeps track of the number of page views and prints the hostname of the
yelb-appserver instance serving the API request upon a vote or a page refresh. Yelb components include:
- A frontend called
yelb-uiis responsible for vending the JS code to the browser.
- An application server named
yelb-appserver, a Sinatra application that reads and writes to a cache server (
redis-server), and a PostgreSQL backend database (
- Redis stores the number of page views and PostgreSQL stores the votes.
The following picture depicts the architecture.
Steps to resource share ACM – Private CA across accounts:
- Create a Private CA that you will be sharing across the accounts. If you already want to use an existing CA, identify which Private CA(s) you want to share, and which accounts you want to share with.
- Create a resource share and then add your ACM Private CA to the share.
- Share the resource with a single account or with your AWS Organizations structure.
- In the shared account(s), create a certificate through the ACM console, Share your Private CA with your AWS Organizations accounts and share your Private CA with individual accounts.
Step 1: Set up the infrastructure
To follow this step by step implementation, you will need to have an environment with some tooling. An AWS Cloud9 instance can be used to run this tutorial. If you want to create a Cloud9 instance in your account, follow the steps in the EKS Workshop from the chapter Create a Workspace to Update IAM Settings for your Workspace.
These are the tools we will need to have installed:
Since we will be running this in two different accounts, you need to make sure that you have profiles with credentials for both AWS accounts configured in
~/.aws/config files. Here’s how it is going to look once you complete the profile setup successfully:
cat ~/.aws/credentials [frontend] aws_access_key_id = ... aws_secret_access_key = ... [backend] aws_access_key_id = ... aws_secret_access_key = ... cat ~/.aws/config [profile frontend] region = us-west-2 [profile backend] region = us-west-2
?You can get more information about creating configuration files in the AWS CLI user guide.
After installing everything and configuring the profiles, let’s clone the GitHub repository that has the scripts that we will be using in this tutorial:
git clone https://github.com/aws/aws-app-mesh-examples.git cd aws-app-mesh-examples/walkthroughs/eks-app-mesh-cross-account-acm/
We will be using the CloudFormation templates to create the VPC, Security Groups, Route Tables, and the VPC Peering connection between the two accounts, if you want to take a look at them, they’re located in the infrastructure directory of the repo you just cloned. You can create them by running the following script:
Step 2: Create the EKS clusters
In this step, we will use
eksctl to create the EKS clusters in the frontend and backend accounts by running the following script:
You can see your new clusters by running
kubectl config get-contexts
Step 3: Install the App Mesh components
Before creating the mesh resources and deploying the Yelb app onto the clusters, you need to install the AWS App Mesh Controller. This controller allows you to configure App Mesh resources using
kubectl. You will use Helm to install the App Mesh controller.
First, install the controller in the Frontend Account’s cluster.
kubectl config use-context <iam_user>@am-multi-account-1.<region>.eksctl.io helm repo add eks https://aws.github.io/eks-charts kubectl create ns appmesh-system helm upgrade -i appmesh-controller eks/appmesh-controller \ --namespace appmesh-system
Now, install the controller in the Backend Account’s cluster.
kubectl config use-context <iam_user>@am-multi-account-2.<region>.eksctl.io kubectl create ns appmesh-system helm upgrade -i appmesh-controller eks/appmesh-controller \ --namespace appmesh-system
Confirm that the App Mesh controller is running:
kubectl --context=<iam_user>@am-multi-account-1.<region>.eksctl.io get pods -n appmesh-system kubectl --context=<iam_user>@am-multi-account-2.<region>.eksctl.io get pods -n appmesh-system
You should see outputs similar to this one:
NAME READY STATUS RESTARTS AGE appmesh-controller-66b749c78b-67n68 1/1 Running 0 6s
Step 4: Deploy and share the mesh
Now that you have the App Mesh controller set up in both clusters, you want to create the mesh and share it to the Backend Account.
First, let’s create the namespace where you will deploy the mesh components and the Yelb application in the Frontend Account’s cluster.
kubectl config use-context <iam_user>@am-multi-account-1.<region>.eksctl.io kubectl create ns yelb
You need to add two labels to the
yelb namespace: mesh and
appmesh.k8s.aws/sidecarInjectorWebhook. These labels instruct the controller to inject and configure the Envoy proxies in the pods:
kubectl label namespace yelb mesh=am-multi-account-mesh kubectl label namespace yelb "appmesh.k8s.aws/sidecarInjectorWebhook"=enabled
Let’s create and share the mesh and share it to the Backend Account:
Share the mesh using AWS Resource Access Manager:
aws --profile frontend cloudformation deploy \ --template-file shared_resources/shared_mesh.yaml \ --parameter-overrides \"BackendAccountId=$(aws --profile backend sts get-caller-identity | jq -r .Account)" \ --stack-name am-multi-account-shared-mesh \ --capabilities CAPABILITY_IAM
Now, accept the Resource Share Invitation in the Backend Account.
Run the following commands to accept the invitation:
RESOURCE_SHARE_ARN=$(aws --profile backend ram get-resource-share-invitations \| jq -r '.resourceShareInvitations | select(.resourceShareName=="mesh-share") | .resourceShareInvitationArn') aws --profile backend ram accept-resource-share-invitation \ --resource-share-invitation-arn $RESOURCE_SHARE_ARN
The Backend Account needs a service role so App Mesh can discover
yelb-appserver Virtual Node’s ip address using AWS Cloud Map. Create it with this command:
aws --profile backend iam create-service-linked-role \ --aws-service-name appmesh.amazonaws.com
Step 5: Deploy the mesh resources and the application
Since the resources in the Backend Account are dependencies for the resources in the Frontend Account, let’s start by creating them:
Switch to the Backend Account’s cluster context:
kubectl config use-context <iam_user>@am-multi-account-2.<region>.eksctl.io
Even though the mesh is already created and shared, the App Mesh controller in the Backend Account’s cluster needs to be aware of it, let’s run the following script:
kubectl create ns yelb kubectl label namespace yelb mesh=am-multi-account-mesh kubectl label namespace yelb "appmesh.k8s.aws/sidecarInjectorWebhook"=enabled ./mesh/create_mesh.sh
Now, create the Mesh components in the Backend Account, this will create Virtual Nodes, Virtual Services, and Virtual Routes for
kubectl apply -f mesh/yelb-redis.yaml kubectl apply -f mesh/yelb-db.yaml kubectl apply -f mesh/yelb-appserver.yaml
kubectl apply -f yelb/resources_backend.yaml
yelb-appserver Virtual Service ARN and change mesh/yelb-ui.yaml backend configuration accordingly. This is a manual process, yelb-ui’s Virtual Node creation will fail if skipped.
kubectl -n yelb get virtualservice yelb-appserver # Replace <yelb-appserver virtualServiceARN># in mesh/yelb-ui.yaml with the ARN from the # command above using your editor of choice
Now that you have all the Backend Account components deployed, switch to the Frontend Account’s cluster.
kubectl config use-context <iam_user>@am-multi-account-1.<region>.eksctl.io
Create the mesh components in the Frontend Account, this will create a Virtual Node, Virtual Service, and a Virtual Route for
kubectl apply -f mesh/yelb-ui.yaml
kubectl apply -f yelb/resources_frontend.yaml
Get the the Load Balancer URL to test the application in your browser:
kubectl get service yelb-ui -n yelb
Step 6: Configuring TLS Encryption on AWS App Mesh Virtual Nodes
In App Mesh, traffic encryption works between Virtual Nodes, and thus between Envoys in the service mesh. This means that the application code is not responsible for negotiating a TLS-encrypted session, instead allowing the local proxy to negotiate and terminate TLS on application’s behalf. In the below script, CA is created and shared with the backend account.
Enforcing Virtual Node Backend Client Policy with an ACM Certificate Authority ARN
Note: The below step is not required,
apply_tls_on_yelb_nodes.sh above takes care of the required TLS encryption configurations. The screenshot is to show how it works when you are configuring via the AWS console.
The following screenshot shows a client policy set on a Virtual Node’s backend to a Virtual Service, which enforces the use of TLS on port 443 only using ACM as the trusted Certificate Authority (CA). Note the use of ACM Private Certificate Authority (PCA) within the trust ACM section.
You should see the application as follows, go ahead and test it!
Step 7: Cleanup
In order to clean up all the resources created during the execution of this tutorial, run the cleanup script with the following command:
Note that this cleanup script won’t delete any AWS accounts created for this tutorial, but only the resources provisioned in it by the previous steps.
With this step by step tutorial, customers are encouraged to test this feature out and implement secure traffic between your services thus meeting your security and compliance baselines. You will find further details about these features and more guidance in the Transport Layer Security (TLS) section of AWS App Mesh documentation. We welcome your feedback through our AWS App Mesh roadmap channel and we’re excited for you to use this feature to build secure and compliant mesh applications!