Assess compliance and configuration of Kubernetes resources with AWS Config

Many customers today rely on AWS Config for recording configuration, tracking configuration history, and evaluating compliance of their AWS resources such as Amazon Elastic Compute Cloud (EC2) instances, Amazon Simple Storage Service (S3) buckets, and even Amazon Elastic Kubernetes Service (EKS) clusters. This provides them with a comprehensive view of their AWS infrastructure configuration state and compliance posture.

However, when it comes to managing the compliance of Kubernetes resources running within Amazon EKS clusters (such as pods, deployments, and services), organizations often need to implement separate tooling. This is because Kubernetes resources exist within the cluster rather than as direct AWS resources, placing them outside the standard AWS Config recording scope.

The open-source solution described in this post bridges this gap by extending AWS Config capabilities to Kubernetes resources. This integration allows you to:

Record and track configuration changes of Kubernetes resources over time
Query Kubernetes resources using AWS Config advanced query capabilities
Apply AWS Config rules to evaluate compliance of Kubernetes resources
Maintain a single source of truth for both AWS and Kubernetes configurations
Leverage existing AWS Config dashboards and reporting for Kubernetes resources

Solution walkthrough

Deployment

The solution is deployed using AWS CloudFormation, which provisions all the necessary resources in your AWS account. For the deployment instructions, refer to the project documentation.

Configuring the solution

During deployment or afterwards, you can customize the solution with several CloudFormation parameters, including:

Resource Kinds: Specify which Kubernetes resource types to record (e.g., Pod, Deployment, Service)
Namespace Filters: Include or exclude specific Kubernetes namespaces
Label Selector: Filter resources to record based on Kubernetes labels
Resource Fields: Control which fields are included or excluded from recording

These parameters allow you to tailor the solution to your specific needs, focusing on the resources and configurations most important to your organization. For the comprehensive list of parameters, refer to the project documentation.

Viewing recorded resources

Once deployed, you can view your Kubernetes resources in the AWS Config console:

Navigate to the AWS Config console
In the left menu, choose Resources
Filter by resource type AWSCustom::EKS::KubernetesResource
Browse the list of recorded Kubernetes resources, as illustrated in the following image

Figure 1 – AWS Config console showing recorded Kubernetes resources

Choose a resource from the Resource identifier column to view its configuration details, including metadata, specifications, and AWS Config-specific information. For example, the following image shows the details of a recorded Kubernetes Deployment:

Figure 2 – Configuration details of a recorded Kubernetes Deployment in AWS Config

In the resource page, choose Resource Timeline to view how the resource has changed over time. For example, the following image shows the configuration history of a recorded Kubernetes Deployment:

Figure 3 – Resource timeline showing configuration history of a Kubernetes Deployment in AWS Config

Querying resources

One of the powerful capabilities this solution enables is the ability to retrieve your Kubernetes resources using AWS Config advanced queries. For example:

SELECT
  *
WHERE
  resourceType = 'AWSCustom::EKS::KubernetesResource'

This query returns all Kubernetes resources recorded in AWS Config, as illustrated in the image below.

Figure 4 – Results of an AWS Config advanced query for Kubernetes resources

You can also create more specific queries to find particular resources. For example:

SELECT
  *
WHERE
  resourceType = 'AWSCustom::EKS::KubernetesResource'
  AND resourceId LIKE '<your-cluster-id>/kube-system/ConfigMap%'

This query returns all Kubernetes ConfigMap resources in the kube-system namespace for a specific EKS cluster.

Evaluating compliance

With your Kubernetes resources recorded in AWS Config, you can create AWS Config rules to continuously evaluate their compliance. For example, you might want to ensure:

Deployments follow specific labeling conventions
Services use approved port ranges
Namespaces have network policies applied

You can create AWS Config custom rules using AWS CloudFormation Guard policy-as-code language or AWS Lambda functions. Here is an example Guard rule that ensures deployments have at least two replicas:

rule kubernetes_deployment_min_replicas when resourceType == "AWSCustom::EKS::KubernetesResource" {
 # Only evaluate this rule if the Kubernetes Resource is a Deployment
  when configuration.K8sResourceKind == "Deployment" {
    # Ensure the Deployment has at least 2 replicas
    configuration.K8sItem.spec.replicas >= 2
  }
}

And here is a more advanced rule that ensures all pods use images from approved registries:

let approved_registries = [/^registry\.mycorp\.com\/.*/, /^public\.ecr\.aws\/docker\/.*/]

rule kubernetes_approved_registries when resourceType == "AWSCustom::EKS::KubernetesResource" {
  # Only evaluate this rule if the Kubernetes Resource is a Pod
  when configuration.K8sResourceKind == "Pod" {
    # Check all containers use approved registry
    configuration.K8sItem.spec.containers[*] {
      # Ensure image is from approved registry
      image IN %approved_registries
    }
    
    # Also check init containers if present
    when configuration.K8sItem.spec.initContainers EXISTS {
      configuration.K8sItem.spec.initContainers[*] {
        image IN %approved_registries
      }
    }
    
    # Check ephemeral containers if present
    when configuration.K8sItem.spec.ephemeralContainers EXISTS {
      configuration.K8sItem.spec.ephemeralContainers[*] {
        image IN %approved_registries
      }
    }
  }
}

To create these rules in AWS Config:

In the left navigation pane of the AWS Config console, choose Rules
Choose Add rule, choose Create custom rule using Guard, and choose Next.
In the Configure rule page:
1. For Rule name type a meaningful name (e.g., kubernetes_approved_registries)
2. For Rule content, paste your Guard rule content
3. In the Evaluation mode section, for Scope of changes select Resources
4. For Resource type, enter and select AWSCustom::EKS::KubernetesResource
Choose Next and choose Save

After creating the rule, AWS Config continuously evaluates resources against it. The rule page shows the compliance status of all Kubernetes resources in scope, helping you quickly identify configuration issues. For example, the following image shows two compliant and two noncompliant pods.

Figure 5 – AWS Config rule compliance status for Kubernetes pods

Now you can return to the query editor and execute the following query to get all non-compliant pods in the default namespace of your cluster:

SELECT
  configuration.targetResourceId,
  configuration.targetResourceType,
  configuration.complianceType,
  configuration.configRuleList
WHERE
  configuration.complianceType = 'NON_COMPLIANT'
  AND configuration.targetResourceType = 'AWSCustom::EKS::KubernetesResource'
  AND configuration.targetResourceId LIKE '<your-cluster-id>/default/Pod/%'

Dive deep – solution architecture

Now that you’ve seen what the solution can do, the following section explains how it works behind the scenes. The solution uses AWS Step Functions to orchestrate the process of registering Kubernetes resources as AWS Config custom resources.

The initial solution deployment registers a custom resource type AWSCustom::EKS::KubernetesResource with the CloudFormation registry. This includes a schema defining the structure and properties of Kubernetes resources that enables AWS Config to properly interpret, validate, and track Kubernetes resource configurations. The deployment also provisions all necessary AWS resources for the workflow operation (Lambda functions, Step Function state machine, S3 bucket, etc.). The following diagram shows the architecture of the solution.

Figure 6 – Architecture diagram of the solution

The workflow then operates as follows:

Scheduled Execution:
1. An Amazon EventBridge rule triggers the Step Functions state machine on a configurable schedule (default: hourly).
Resource Discovery and Analysis:
1. The DiscoverResources Lambda function lists the EKS clusters in your AWS account, connects to each cluster, and retrieves the relevant resources, filtering them based on your configuration (resource kinds, namespaces, labels, etc.).
2. The DiscoverResources Lambda function queries AWS Config to get a list of Kubernetes resources currently recorded in AWS Config.
3. The DiscoverResources Lambda function calculates which resources are stale and need to be deleted from AWS Config, and which resources are new or changed and need to be registered. It saves both lists to S3.
Resources Deletion:
1. The DeleteResources Lambda function reads the resources-to-delete list from S3.
2. It deletes these stale resources from AWS Config.
Resources Registration:
1. The RegisterResources Lambda function reads the resources-to-register list from S3.
2. It creates or updates these resources in AWS Config as custom resources of type AWSCustom::EKS::KubernetesResource, making them available for auditing and compliance evaluation.

This architecture ensures that your AWS Config resources stay in sync with the actual state of your Kubernetes resources.

Use cases and benefits

Centralized configuration management

By recording Kubernetes resources in AWS Config, you gain a centralized view of your entire infrastructure, including both AWS and Kubernetes resources. This unified approach simplifies configuration management and provides a single source of truth for your cloud environment.

Compliance monitoring

The solution enables continuous compliance monitoring of your Kubernetes resources against your organization’s policies and industry standards. You can create custom rules to enforce specific requirements and receive notifications when resources drift from compliance. Additionally, you can leverage AWS Config conformance packs to group related rules and remediation actions, making it easier to apply and manage compliance at scale across your Kubernetes environments.

Security auditing

Security teams can use the recorded configuration data to audit Kubernetes resources for security best practices. For example, they can identify pods running with privileged access, containers using the root user, deployments without proper network policies, or unauthorized image repository sources.

Change tracking and troubleshooting

When issues arise, the configuration timeline in AWS Config helps you understand what changed and when. This historical data is invaluable for troubleshooting and can help you quickly identify the root cause of problems.

Conclusion

The open-source solution from this post bridges the gap between AWS and Kubernetes resource management by bringing Kubernetes resources into AWS Config’s powerful configuration and compliance framework. This integration enables unified visibility, simplified compliance, and enhanced security across your environment.

To learn more about AWS Config, visit the AWS Config documentation. For implementing this solution, check out the open-source project repository.

AWS Cloud Operations Blog