Containers

Recent changes to the CoreDNS add-on

Introduction

Amazon Elastic Kubernetes Service (Amazon EKS) add-ons were originally introduced in December 2021. At launch, they provided a mechanism for installing and managing a curated set of add-ons for Amazon EKS clusters. The add-on for CoreDNS was amongst the first add-ons we released because DNS plays such a pivotal role in Kubernetes. When advanced configuration support for Amazon EKS add-ons was added in December 2022, customers could pass their own configuration parameters for add-ons directly to the Amazon EKS Application Programming Interface ( API). This allowed customers to install and configure several operational add-ons like CoreDNS during cluster creation in a single step. This feature was among the most highly requested features on the containers roadmap, particularly customization of the CoreDNS corefile.

In the last few months the “coredns” add-on has undergone major improvements. We listened carefully to the top issues on the containers roadmap and implemented the requested functionality and closed those issues. We added new components like PodDisruptionBudget (PDB), implemented opinionated defaults (e.g., using health plugin for pod health checks) and enhanced the JSON configuration schema to allow customers to customize more parameters of the add-on.

Prerequisites

In order to follow this blog post you need:

  • an AWS account
  • an Amazon EKS cluster with the “coredns” add-on deployed
  • a recent version of the AWS Command Line Interface (CLI)
  • the kubectl command line tool compatible with your EKS cluster version

Walkthrough

Issue #2033 requests adding a topologySpreadConstraints setting on the EKS CoreDNS addon configuration schema

Version v1.9.3-eksbuild.5 added the parameter topologySpreadConstraints to add-on JSON configuration schema which maps to K8s feature Pod Topology Spread Constraints. You can use topology spread constraints to control how Pods are spread across your Amazon EKS cluster among failure-domains such as availability zones, nodes, and other user-defined topology domains. This can help to achieve high availability as well as efficient resource utilization.

The configurationValues parameter in Amazon EKS add-ons API accepts configuration as a JSON or a YAML blob. The configuration blob needs to conform to the underlying add-on configuration JSON schema, which is available through the new Amazon EKS DescribeAddonConfiguration API. The JSON schema is unique for each add-on and evolves over time as we implement new configurable parameters.

Using the describe-addon-configuration option of the AWS Command Line Interface (CLI) EKS subcommand we receive the add-on JSON configuration schema with all settings. We are using the jq utility for better readability of the output.

$ aws eks describe-addon-configuration --addon-name coredns --addon-version v1.10.1-eksbuild.2 \
  --query 'configurationSchema' --output text | jq .
{
  "$ref": "#/definitions/Coredns",
  "$schema": "http://json-schema.org/draft-06/schema#",
  "definitions": {
    "Coredns": {
      "additionalProperties": false,
      "properties": {
        "affinity": {
          "default": {
            "affinity": {
              "nodeAffinity": {
                "requiredDuringSchedulingIgnoredDuringExecution": {
                  "nodeSelectorTerms": [
                    {
                      "matchExpressions": [
                        {
                          "key": "kubernetes.io/os",
                          "operator": "In",
                          "values": [
                            "linux"
                          ]
                        },
                        {
                          "key": "kubernetes.io/arch",
                          "operator": "In",
                          "values": [
                            "amd64",
                            "arm64"
                          ]
                        }
                      ]
                    }
                  ]
                }
              },
…
    "Resources": {
      "additionalProperties": false,
      "properties": {
        "limits": {
          "$ref": "#/definitions/Limits"
        },
        "requests": {
          "$ref": "#/definitions/Limits"
        }
      },
      "title": "Resources",
      "type": "object"
    }
  }
}
Apache Configuration

The next command shows how we can evaluate if the attribute is present in the JSON configuration schema.

$ aws eks describe-addon-configuration --addon-name coredns --addon-version v1.10.1-eksbuild.2 \
  --query 'configurationSchema' --output text | jq . | grep -A3 topologySpreadConstraints
    "topologySpreadConstraints": {
      "description": "The coredns pod topology spread constraints",
      "type": "array"
    }
    
# check coredns deployment - it returns an empty output because it is not set by default 
$ kubectl get deploy -n kube-system coredns -o yaml | grep topologySpreadConstraints
Apache Configuration

Now let’s create a YAML blob to apply a topologySpreadConstraints with topologyKey: topology.kubernetes.io/zone. We use whenUnsatisfiable: ScheduleAnyway in case we have a cluster which spans only a single availability zone.

# initial add-on with default configuration
$ aws eks describe-addon --cluster-name demo-cluster-ebs-csi --addon-name coredns
{
    "addon": {
        "addonName": "coredns",
        "clusterName": "demo-cluster-ebs-csi",
        "status": "ACTIVE",
        "addonVersion": "v1.10.1-eksbuild.3",
        "health": {
            "issues": []
        },
        ...
    }
}

# add-on configuration YAML blob
$ cat topologySpreadConstraints.yaml
"topologySpreadConstraints":
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        k8s-app: kube-dns
        
# apply change to add-on
$ aws eks update-addon --cluster-name demo-cluster-ebs-csi --addon-name coredns --configuration-values 'file://topologySpreadConstraints.yaml'
{
    "update": {
        "id": "5523d967-a62b-37cc-a87c-8a4067919409",
        "status": "InProgress",
        "type": "AddonUpdate",
        "params": [
            {
                "type": "ConfigurationValues",
                "value": "\"topologySpreadConstraints\":\n  - maxSkew: 1\n    topologyKey: topology.kubernetes.io/zone\n    whenUnsatisfiable: ScheduleAnyway\n    labelSelector:\n      matchLabels:\n        k8s-app: kube-dns“
            }
        ],
        "createdAt": "2023-08-31T14:41:29.316000+02:00",
        "errors": []
    }
}

# check add-on configuration to see if it is in ACTIVE status
$ aws eks describe-addon --cluster-name demo-cluster-ebs-csi --addon-name coredns
{
    "addon": {
        "addonName": "coredns",
        "clusterName": "demo-cluster-ebs-csi",
        "status": "ACTIVE",
        "addonVersion": "v1.10.1-eksbuild.3",
        "health": {
            "issues": []
        },
…
        "tags": {},
        "configurationValues": "\"topologySpreadConstraints\":\n  - maxSkew: 1\n    topologyKey: topology.kubernetes.io/zone\n    whenUnsatisfiable: ScheduleAnyway\n    labelSelector:\n      matchLabels:\n        k8s-app: kube-dns“
    }
}

# check coredns deployment
$ kubectl get deploy -n kube-system coredns -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
…
  labels:
    eks.amazonaws.com/component: coredns
    k8s-app: kube-dns
    kubernetes.io/name: CoreDNS
  name: coredns
  namespace: kube-system
…
    spec:
…
      containers:
      - args:
        - -conf
        - /etc/coredns/Corefile
..
        name: coredns
…
      topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            k8s-app: kube-dns
        maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
…
Apache Configuration

Issue #1679 requested adding a Pod Disruption Budget (PDB) to the coredns addon and was implemented in v1.9.3-eksbuid.5 and higher.

A PDB limits the number of Pods of a replicated application, like coredns with two replica pods, that are down simultaneously during a node termination. PDB uses either minAvailable or maxUnavailable. Both can be expressed as integers or as a percentage. If you set minAvailable to x or x%, then x or x% Pods must always be available, even during a disruption. maxUnavailablemspecifies the number or percentage of pods that can be unavailable.

The PDB was first implemented in v1.9.3-eksbuild.5, which came with a default of minAvailable: 1. Starting with v1.10.1-eksbuild.2, we improved the logic: if there are two or fewer replicas of the coredns deployment, then we will set maxUnavailable to 1. Otherwise, we will set minAvailable to 2.

# "coredns" add-on v1.9.3-eksbuid.5 and v1.9.3-eksbuid.6
$ kubectl get pdb -n kube-system coredns
NAME    MIN AVAILABLE   MAX UNAVAILABLE  ALLOWED DISRUPTIONS   AGE
coredns 1               N/A              1                    13h

# "coredns" add-on v1.10.1-eksbuild.2 and v1.10.1-eksbuild.3
$ kubectl get pdb -n kube-system coredns
NAME    MIN AVAILABLE   MAX UNAVAILABLE  ALLOWED DISRUPTIONS   AGE
coredns N/A             1                1                     27h
Apache Configuration

Starting with v1.9.3-eksbuild.7 and v1.10.1-eksbuild.4, the default logic of PDB is just maxUnavailable: 1 regardless of deployment replica count. We even made the PDB user configurable by extending the JSON configuration schema as follows:

"podDisruptionBudget": {
                  "type": "object",
                  "description": "podDisruptionBudget configurations",
                  "properties": {
                    "enabled": {
                      "type": "boolean",
                      "description": "the option to enable managed PDB",
                      "default": true
                    },
                    "minAvailable": {
                      "description": "minAvailable value for managed PDB, can be either string or integer; if it's string, should end with %",
                      "anyOf": [
                        {
                            "type": "string",
                            "pattern": ".*%$"
                        },
                        {
                            "type": "integer"
                        }
                    ]
                    },
                    "maxUnavailable": {
                      "description": "maxUnavailable value for managed PDB, can be either string or integer; if it's string, should end with %",
                      "anyOf": [
                        {
                            "type": "string",
                            "pattern": ".*%$"
                        },
                        {
                            "type": "integer"
                        }
                    ]
                    }
                  }
                }
Apache Configuration

Now users can tune PDB with either minAvailable or maxUnavailable as a number or percentage according to their needs.

Issue #2026 was a request to add the lameduck option by default to the coreDNS health plugin to minimize DNS resolution failures.

lameduck delays shutdown for DURATION seconds while the health endpoint still answers 200 acceptably. Adding lameduck to the health plugin minimizes DNS resolution failures during a CoreDNS pod restart (i.e., owing to health issue, node termination, etc) or a deployment rollout. This does come at the expense of increased rollout times. For highly available environments like Amazon EKS, this trade-off is likely acceptable.

Add-on version v1.9.3-eksbuild.6 and v1.10.1-eksbuild.3 implemented this behavior.

$ kubectl get configmap -n kube-system coredns -o yaml
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
            lameduck 5s
          }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2023-08-29T11:46:03Z"
  labels:
    eks.amazonaws.com/component: coredns
    k8s-app: kube-dns
  name: coredns
  namespace: kube-system
Apache Configuration

Issue #941 requested using /ready instead of /health on readiness probe of CoreDNS

Starting with coreDNS v1.9.3-eksbuild.6 and v1.10.1-eksbuild.3 we changed the readinessProbe of the CoreDNS pods by utilizing the CoreDNS upstream ready plugin using /ready path on port 8181. As the upstream ready plugin documentation explains:

By enabling ready an HTTP endpoint on port 8181 it returns 200 OK when all plugins that are able to signal readiness have done so. If some aren’t ready, then the endpoint returns a 503 with the body containing the list of plugins that are not ready. Once a plugin has signaled it is ready, it won’t be queried again.

$ kubectl get deploy -n kube-system coredns -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
…
  name: coredns
  namespace: kube-system
…
      containers:
      - args:
        - -conf
        - /etc/coredns/Corefile
…
        name: coredns
…
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /ready
            port: 8181
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
…
The “ready” plugin is already part of the “coredns” ConfigMap:
$ kubectl get cm -n kube-system coredns -o yaml
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
            lameduck 5s
          }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2023-08-29T11:46:03Z"
  labels:
    eks.amazonaws.com/component: coredns
    k8s-app: kube-dns
  name: coredns
  namespace: kube-system
Apache Configuration

Issue #1879 requested to set labels for pods of EKS-managed addons

The coredns add-on JSON configuration schema starting with v1.9.3-eksbuild.6 and v1.10.1-eksbuild.3 allows adding and changing pod labels and annotations. Pod labels use the podLabels property and come with no defaults. Pod annotations use the podAnnotations property and come with the don-not-evict annotation mentioned in the safe-to-evict annotations section above.

# pod labels
$ aws eks describe-addon-configuration --addon-name coredns --addon-version v1.10.1-eksbuild.3 \
  --query 'configurationSchema' --output text | jq . | grep -A4 '\"podLabels\"'
        "podLabels": {
          "properties": {},
          "title": "The podLabels Schema",
          "type": "object"
        },
Apache Configuration

The following example shows adding a pod label foo: bar. Here you can see that using YAML blobs and string configuration values it isn’t necessary to use double quotes if they don’t contain special characters.

# YAML configuration blob
$ cat podLabels.yaml
podLabels:
  foo: bar
  
# apply changes
$ aws eks update-addon --cluster-name demo-cluster-ebs-csi --addon-name coredns \
  --configuration-values 'file://podLabels.yaml‘
{
    "update": {
        "id": "d49eb115-0edd-3b9f-8938-502f5a2cf54e",
        "status": "InProgress",
        "type": "AddonUpdate",
        "params": [
            {
                "type": "ConfigurationValues",
                "value": "podLabels:\n foo: bar"
            }
        ],
        "createdAt": "2023-09-01T09:08:56.905000+02:00",
        "errors": []
    }
}

# wait a while until the add-on is ACTIVE again
$ aws eks describe-addon --cluster-name demo-cluster-ebs-csi --addon-name coredns
{
    "addon": {
        "addonName": "coredns",
        "clusterName": "demo-cluster-ebs-csi",
        "status": "ACTIVE",
        "addonVersion": "v1.10.1-eksbuild.3",
        "health": {
            "issues": []
        },
...
        "tags": {},
        "configurationValues": "podLabels:\n foo: bar"
    }
}

# check deployment and pods for labels
 $ kubectl get deploy -n kube-system coredns -o yaml
apiVersion: apps/v1
kind: Deployment
...
  template:
    metadata:
      annotations:
        cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
        karpenter.sh/do-not-evict: "true"
      creationTimestamp: null
      labels:
        eks.amazonaws.com/component: coredns
        foo: bar
        k8s-app: kube-dns
        ...
          
$ kubectl get po -n kube-system -l=k8s-app=kube-dns -o custom-columns="POD-NAME":.metadata.name,"POD-LABELS":.metadata.labels
POD-NAME POD-LABELS
coredns-86bb7c8bf5-4rsxm map[eks.amazonaws.com/component:coredns foo:bar k8s-app:kube-dns pod-template-hash:86bb7c8bf5]
coredns-86bb7c8bf5-kj4s6 map[eks.amazonaws.com/component:coredns foo:bar k8s-app:kube-dns pod-template-hash:86bb7c8bf5]
Apache Configuration

Important note when changing add-on configuration values: All previous changes are overwritten. To keep the previous configuration changes, the JSON and YAML configuration blob need to contain all required changes in a merged form like:

# YAML configuration blob
$ cat coredns-conf.yaml
"podLabels":
  "foo": "bar"
"topologySpreadConstraints":
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        k8s-app: kube-dns
Apache Configuration

For adding pod annotations via podAnnotations property see the following example:

# YAML configuration blob
$ cat podAnnotations.yaml $ cat podAnnotations.yaml
"podAnnotations":
   my.log-application.com/logs: "true"

# apply changes
$ aws eks update-addon --cluster-name demo-cluster-ebs-csi --addon-name coredns   --configuration-values 'file://podAnnotations.yaml'
{
    "update": {
        "id": "a31834cb-8b62-3568-b6db-a3c580303a41",
        "status": "InProgress",
        "type": "AddonUpdate",
        "params": [
            {
                "type": "ConfigurationValues",
                "value": "\"podAnnotations\":\n   my.log-application.com/logs: \"true\""
            }
        ],
        "createdAt": "2023-09-13T16:26:28.774000+02:00",
        "errors": []
    }
}

# wait a while until the add-on is ACTIVE again
$ aws eks describe-addon --cluster-name demo-cluster-ebs-csi --addon-name coredns
{
    "addon": {
        "addonName": "coredns",
        "clusterName": "demo-cluster-ebs-csi",
        "status": "ACTIVE",
        "addonVersion": "v1.10.1-eksbuild.3",
        "health": {
            "issues": []
        },
        ...
        "tags": {},
        "configurationValues": "\"podAnnotations\":\n my.log-application.com/logs: \"true\""
      
    }
}

# check deployment and pods for annotations
$ kubectl get deploy -n kube-system coredns -o yaml
apiVersion: apps/v1
kind: Deployment
...
  template:
    metadata:
      annotations:
        ...
        my.log-application.com/logs: "true"
        ...
        
$ kubectl get po -n kube-system -l=k8s-app=kube-dns -o custom-columns="POD-NAME":.metadata.name,"POD-ANNOTATIONS":.metadata.annotations
POD-NAME                   POD-ANNOTATIONS
coredns-758ffffbfb-j7kmx   map[my.log-application.com/logs:true]
coredns-758ffffbfb-w2qhg   map[my.log-application.com/logs:true]
Apache Configuration

Call to action

With the recent changes AWS made it easier to use and customize the coredns add-on to meet customer needs. Use the Amazon Elastic Kubernetes Services (EKS) add-ons console or the AWS Command Line Interface (CLI) “aws eks create-addon” command and give it a try.

Cleaning up

Do not forget to delete any example resources if you no longer need them, to avoid incurring future costs.

Conclusion

In this post, we showed you the new features are available in the coredns add-on and how you can use them. You may provide feedback on the Amazon coredns add-on by leaving a comment or opening an issue on the AWS Containers Roadmap that is hosted on GitHub.

Stay tuned as we continue to evolve our features around CoreDNS.