Monitoring version compliance of Amazon Elastic Kubernetes Service by using AWS Config

Amazon Elastic Kubernetes Services (Amazon EKS) provides a managed Kubernetes service, simplifying cluster operations by offloading undifferentiated heavy lifting to AWS. With the Kubernetes release cycle of a new release every 4 months, customers have difficulty in keeping their EKS clusters up-to-date, especially across multiple AWS accounts. Additionally, keeping track of EKS version will aid your upgrade strategy and allow teams to benefit from new features, design updates, and bug fixes.

A Kubernetes cluster has a control plane and a data plane. The Kubernetes software running in the different planes need updates to remain up-to-date. The control plane updates are managed by AWS, so you only have to launch the process for new minor version. For the data plane, Amazon EKS makes the upgrade for the infrastructure layer a straightforward process by providing Amazon EKS optimized Amazon Machine Images (AMI) and the usage of Amazon EKS managed node group (MNG). However, there is a sequence of actions that need to be followed, as described in our best practices guide.

Using AWS Config to monitor Amazon EKS

In this post, we will use AWS Config, a service that allows you to continually evaluate your resources through rules. First, we will use AWS Config to monitor the version of an Amazon EKS control plane with an AWS Config managed rule. Next, we will deploy an AWS Config Custom Lambda Rule to monitor the version of Amazon Machine Images (AMI) used in your Amazon EKS managed node groups (MNG) for available update. This will provide your operations team the visibility needed to control the risk associated with upgrades and keep your infrastructure up-to-date by monitoring effective versions. To gain visibility across multiple AWS accounts, AWS Config can be configured to give you a single view using organization-wide aggregator.

AWS Config publishes the event to Amazon EventBridge. Using an event pattern, it’s possible to filter only the events of interests. You can then define one of the many available targets in EventBridge to receive those events.

Here’s the architecture diagram of the AWS Config rule that we will deploy:

The different components in this solution are :

1. An AWS Config managed rule that checks the EKS Control plane version.
2. An AWS Config Custom Lambda Rule that monitors new AMI version for our Managed Node Group.
3. All events generated by AWS Config are sent to Amazon EventBridge. Through event pattern, you can trigger notifications to stakeholders (email, http, etc.) or forward them to other targets.

Prerequisites:

You will need the following to complete the steps for the AWS managed rule section:

• AWS Config enabled in your account, if required you can use the 1-click setup directly from the console.

For the AWS Config Custom Lambda Rule section, you will need to:

• Install the AWS Command Line Interface (AWS CLI).
• Install the AWS Serverless Application Model Command Line (AWS SAM CLI).
• Install Git.
• Install Node.js and npm.

Monitor EKS control plane version through AWS Config managed rules

For Amazon EKS, we can ensure that the version of our control plane is within a supported version by using the eks-cluster-supported-version provided by AWS Config. The full list of AWS Config managed rules is available in the documentation.

To enable the eks-cluster-supported-version from AWS Config:

1. Open the AWS Config console.
2. In the navigation pane, choose Rules.
3. Choose Add rule.
4. Under AWS Managed Rules, search for EKS.
5. Select eks-cluster-supported-version, choose Next.
6. Under Parameter, in the Value column, input the latest version of EKS you want to consider compliant. At the time of this writing, I’m using 1.26.
7. Choose Next, review your rule and click Add rule.
8. Select the new rule from the list and under Actions, choose Re-evaluate.

This screenshot shows the rule details with three clusters in your account.

Extending AWS Config to monitor EKS Managed Node Groups (MNG) by leveraging AWS Lambda

The following steps create an AWS Config Custom Lambda Rule and enable it for our managed node groups using the code from our GitHub repo.

Installation procedure:

Let’s start by creating the required Lambda using sam-cli tool:

# You can download the zip from github instead of git clone
git clone https://github.com/aws-samples/aws-config-eks-mng-checkupdate.git
cd aws-config-eks-mng-checkupdate
sam build
sam deploy --guided

We suggest config-custom-rule-eks-mng as the value for the Stack Name. Use default for the other options.

Here’s the sample output of this command:

        Deploying with following values
        ===============================
        Stack name                   : config-custom-rule-eks-mng
        Region                       : ca-central-1
        Confirm changeset            : True
        Disable rollback             : False
        Deployment s3 bucket         : aws-sam-cli-managed-default-samclisourcebucket-xxxxx
        Capabilities                 : ["CAPABILITY_IAM"]
        Parameter overrides          : {}
        Signing Profiles             : {}
…
Successfully created/updated stack - config-eks-ami in ca-central-1

This will create a Lambda as well as the custom ruleEKS-MNG-CheckUpdate in AWS Config with the resource policy to invoke that Lambda. The IAM role created for Lambda has read-only permission on EKS as well as GetParameter from AWS Systems Manager (SSM) for the available version of AMIs.

Periodic trigger type

AWS Config rules can be triggered on a periodic schedule or a configuration change. AWS Config currently does not provide support for managed node group (MNG). In order to evaluate a MNG resource, we defined an AWS Config Custom Lambda Rule that runs periodically. There are multiple periodic choices, from every hour to every 24 hours. Considering this is linked to AMI update frequencies, we chose the 24 hours period through our template. Note that your AWS Config costs will increase with more rule evaluations. To learn more, see AWS Config pricing page. Choosing the right frequency for your organization’s needs will help you gain the right level of visibility without accumulating unnecessary charges. If rules need to be re-evaluated faster than the periodic check, it’s possible to trigger an on-demand evaluation from the AWS Config console. To trigger an on-demand evaluation, select the appropriate rule, and from the action menu, click the Re-evaluate option.

Here’s a view of results provided by the custom rule from the AWS Config console:

The current version of the AWS Config rule support node running the Amazon EKS-Optimized AMI of Amazon Linux 2, and Bottlerocket. See the GitHub project for more details on the supported version.

Event pattern to filter events in Amazon EventBridge

If you want to react to any compliance changes happening with our new rules inside EventBridge. Here’s an example of an event pattern that cover rules used today:

{
  "source": ["aws.config"],
  "detail-type": ["Config Rules Compliance Change"],
  "detail": {
    "configRuleName": ["EKS-MNG-CheckUpdate", "eks-cluster-supported-version"]
  }
}

For the targets, it’s possible to select from a wide-range of AWS Services and even external API, see the full list for details.

Cleanup

To prevent additional charges to your account, the following steps will delete the resources that were created:

1 – Through the AWS Config console. Select the rule eks-cluster-supported-version and from the action menu, use the Delete rule option.

2 – Delete the custom rule and associated resources via the following command:

sam delete --stack-name config-custom-rule-eks-mng

3 – In the EventBridge console, delete the rule that was added (if any).

Conclusion

In this blog post, we have demonstrated how you can track your EKS Version both at the control and data plane level. By leveraging AWS Config rules (managed and custom), you can automate the effort needed to track effective version.

To get a centralized view of your clusters and managed node group version, you can set up an organization-wide aggregator to take advantage of AWS Config aggregation capabilities. This will help you in keeping up with the Kubernetes release cycle. In planning your upgrade, you will want to consult the EKS version page, which includes information specifics for each version.