Using IAM database authentication with workloads running on Amazon EKS
Amazon Elastic Kubernetes Service (Amazon EKS) is a managed service that you can use to run Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane or nodes. When running containerized workloads on Amazon EKS, it is common to store the stateful parts of the application outside of the Kubernetes cluster in one or more SQL or NoSQL databases. However, a common challenge when using SQL databases with Kubernetes is storing the database credentials, as well as rotating the credentials on a regular basis and passing the sensitive parts securely into the Kubernetes cluster.
With Amazon Relational Database Service (Amazon RDS) and Amazon Aurora, the whole process of managing user access credentials can be simplified by using AWS Identity and Access Management (IAM) for authenticating to the database. IAM database authentication removes the burden and security risk of managing usernames and passwords for database authentication. Leveraging IAM database authentication with workloads running on Amazon EKS provides the following benefits:
- Network traffic to and from the database is encrypted using Secure Socket Layer (SSL) or Transport Layer Security (TLS).
- Authentication tokens have a lifespan of 15 minutes, so you don’t need to enforce password resets.
- Amazon EKS IAM roles for service accounts (IRSA) can be leveraged to provide IAM access keys and secret access keys to the application. Providing a secure way to retrieve tokens for IAM database authentication.
IAM database authentication works with MariaDB, MySQL, and PostgreSQL running on Amazon RDS, and Amazon Aurora MySQL and Amazon Aurora PostgreSQL.
In this blog post, we will demonstrate IAM database authentication with Amazon EKS by deploying a sample application and storing the state in an Amazon Aurora MySQL database. The demonstration application is the Product Catalog Application, which has the following architecture:
The Product Catalog Application has three microservices:
frontendservice hosting a web UI for a product catalog.
prodcatalogbackend service that performs the following actions:
- Talks to the Aurora MySQL database to:
- Add a product into the database.
- Get the product from the database.
- Calls catalog detail backend service
proddetailto get product catalog detail information.
- Talks to the Aurora MySQL database to:
proddetailbackend service that gets catalog detail, which includes version numbers and vendor names.
In this blog post, we will first be creating a local database user, workshop_user, in an Amazon Aurora MySQL database and enable this user to use IAM database authentication. We will then be creating an IAM policy called “Aurora_IAM_Policy” that will have permission to access the Aurora database as the user workshop_user. Finally, we will leverage IRSA, a Kubernetes service account, and an IAM role to securely connect to the Amazon Aurora database.
This blog assumes that you have the following setup already done:
- An existing EKS cluster with an existing node group.
- Aurora MySQL cluster using Quickstart.
- IAM DB authentication availability is enabled in this Quickstart.
- The database endpoint is not available publicly in this Quickstart, as it is not recommended to have database endpoints accessible on the internet. When following this walkthrough, a connection within the VPC to connect to the database is required. The simplest way to achieve this is through an AWS Cloud9 instance or a bastion host when running the mysql client commands.
- For simplicity, in this blog, we have assumed that the Amazon EKS cluster, the AWS Cloud9 instance/bastion host, and Aurora MySQL are all deployed in the same VPC in the same AWS account.
- Tools required on a machine with access to the AWS and Kubernetes API server. This could be your workstation or an AWS Cloud9 instance/bastion host.
- Tools required on a machine with access to the Amazon Aurora database, such as an AWS Cloud9 instance or bastion host.
- The mysql client.
Step 1. Confirm access to an Amazon EKS cluster
Ensure you have access to an EKS cluster via the kubectl client. For details on setting up the kubectl client for an Amazon EKS cluster, see the Amazon EKS User Guide.
Step 2. Confirm Aurora DB setup
Ensure the Amazon Aurora MySQL Quickstart has been deployed successfully by navigating to the database within the Amazon RDS console. If you kept the Quickstart defaults, you should see an Amazon Aurora cluster with two instances (one is Writer and one is Reader).
Next, confirm that IAM DB authentication is enabled by selecting the Configuration tab. If IAM DB authentication is not enabled, then enable it by referring to the Amazon RDS User Guide for Aurora.
Step 3. Update the database security group
To allow workloads on the Amazon EKS cluster to access the Aurora MySQL database, we need to add a few rules to the security group assigned to the Aurora MySQL database.
First, we need to retrieve the security group ID that workloads running on Kubernetes will use. By default in an Amazon EKS cluster, all Kubernetes pods share the same security group with their underlying EC2 instance. In a managed node group, this is the EKS cluster security group. This can be retrieved by browsing to the Amazon EKS console within the AWS Management Console. Select Configuration and then Networking. Copy this security group ID to your clipboard.
Next, we need to navigate to the security group associated with the Writer instance on the database cluster. Within the Amazon RDS console, navigate to the Aurora database cluster, select the Writer instance, and then select the VPC security group. This should take you to the security group console, with the relevant security group already selected.
Now we can create inbound rules on this security group for the Kubernetes pods. Add an inbound rule with Protocol TCP, Port Range 3306, and in the Source box, paste in the security group ID retrieved from the Amazon EKS cluster. If you are planning to use an AWS Cloud9 instance or bastion host to run the
mysql commands later on in this walkthrough, create a second Inbound rule with the relevant source security group ID so that traffic sourced from that instance can reach the database. Finally, select Save rules.
Step 4. Test the connectivity
Within the Amazon RDS console, select the Writer instance, and within the Connectivity & security tab, copy the database endpoint into your clipboard.
Within the AWS Cloud9 instance or bastion host, confirm that you can connect to the database with the user name msadmin and the password you set when deploying the Aurora Quickstart template. If you are unable to connect to the database, go back to Step 3 and ensure the security groups have been configured correctly for the AWS Cloud9 instance/bastion host.
In a web browser, browse to the Load Balancer URL. You should see the following UI, and you should see that there is no data found for the product catalog. At this point, the authentication for the database has not been configured, so we are unable to see the seed data in the database, hence “No Products found in the Product Catalog.”
Step 7. Create an IAM policy for DB authentication
To allow a user or a Kubernetes service account to access the Aurora MySQL database, we need to create an IAM policy. Following least privileges, the IAM policy specifies the Amazon Aurora database resource ID and the local database user workshop_user in the IAM policy document. For more details on this topic, you can also refer to the Creating and using an IAM policy for IAM database access document.
To create the policy document, we need to retrieve the Amazon Aurora Resource ID from the RDS console. Open the RDS console, select the Aurora MySQL cluster, and navigate to the Configuration tab. Copy the resource ID to your clipboard.
On a machine that has the AWS CLI installed, create an IAM policy that can access the database.
Step 8. Create an IAM role and map this to a Kubernetes service account
In this step, we will consume the IAM policy created in Step 7 and attach it to an IAM role. We will then create a trust relationship on that IAM role to map the role to a Kubernetes service account. These steps could be done by using the AWS CLI; however, the eksctl tool simplifies all of the steps into a single command.
Under the hood, the previous command carries out two things:
- It creates an IAM role and attaches the specified policy to it; in our case, arn:aws:iam::61801138XXXX:policy/Aurora_IAM_Policy
- It creates a Kubernetes service account aurora-irsa and annotates the service account with the IAM role.
View the created service account.
You can also go to the IAM console within the AWS Management Console and search for the previous role-arn. You should see that the IAM role has been created successfully.
Select the Aurora_IAM_Policy, and you should be able to confirm the previously created IAM DB authentication policy has been attached to this role.
Step 9. Redeploy the Helm Chart and specify the database credentials
Now that an IAM role has been mapped to a Kubernetes service account, the application can use the service account credentials to communicate to the Aurora MySQL database. We will pass in the nonsensitive RDS connection information to the application through environment variables in the Helm Chart.
A Helm Chart variables file is located at
workshop/helm-chart/values-aurora.yaml in the cloned repository. Once that file has been opened in a text editor, we need to update the
DB_REGION variables. These are found at lines 160 and 168.
Within the prodcatalog application code, we use the boto3 SDK to consume these environment variables when authenticating to IAM to retrieve a token. We then pass this token into our
pymysql connect command alongside the remaining database information to authenticate to the database.
So that the application can take advantage of the IAM roles for service account credentials and to pick up the new environment variables, we can redeploy the application with Helm.
To verify that IAM roles for service accounts are working correctly, use kubectl to describe the podcatalog pod.
$ kubectl describe pod prodcatalog-<pod_id> -n workshop
The following image shows that the mutating admission controller we ran in EKS (via a webhook) automatically injected the environment variables
AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE as well as the
Step 10. Confirm the application is connected to database
Finally, we will browse to the application front end again and verify that the application can now see our seed data. If you need to retrieve the front-end URL again, use kubectl to get details about the Kubernetes service.
Once you go to the Load Balancer URL in a web browser, you should be able to confirm that the Product Catalog data (Product ID: 999 and Product Name: Mountain New Bike) is coming from the Aurora MySQL DB. This means our
prodcatalog service deployed in EKS is able to talk to Aurora MySQL DB using IRSA authentication with the service account role we created for the user workshop_user.
We can also add a new product to the catalog by entering a new Product ID and Product Name into the corresponding fields.
You can confirm that the new product has been added to the Aurora MySQL database successfully.
Finally, we can verify that the new item has been added to the database by using the MySQL client on a machine with access to the database endpoint.
Deleting Aurora DB
You can delete the Aurora database using the instructions in the User Guide for Aurora.
Deleting EKS cluster
You can delete the EKS cluster using the instructions Deleting an Amazon EKS cluster in the Amazon EKS User Guide.
In this blog, we learned that you can authenticate to your Aurora MySQL DB clusters using IAM database authentication from an application running on Amazon EKS. With this authentication method, you don’t need to use a password when you connect to your database cluster. Instead, we have used a short-lived authentication token so that you don’t have to worry about the storage and lifecycle of username and password credentials. Finally, we have also seen that network traffic to and from the database is encrypted using SSL and used as part of our connection to the database.
In this blog post, we have shown an application deployed in Amazon EKS, but the IAM database authentication can be used with many other types of AWS infrastructure, such as AWS Lambda, Amazon EC2, AWS Fargate, or an Amazon ECS task. For more information, refer to: