Recent changes to the CoreDNS add-on
Amazon Elastic Kubernetes Service (Amazon EKS) add-ons were originally introduced in December 2021. At launch, they provided a mechanism for installing and managing a curated set of add-ons for Amazon EKS clusters. The add-on for CoreDNS was amongst the first add-ons we released because DNS plays such a pivotal role in Kubernetes. When advanced configuration support for Amazon EKS add-ons was added in December 2022, customers could pass their own configuration parameters for add-ons directly to the Amazon EKS Application Programming Interface ( API). This allowed customers to install and configure several operational add-ons like CoreDNS during cluster creation in a single step. This feature was among the most highly requested features on the containers roadmap, particularly customization of the CoreDNS corefile.
In the last few months the “coredns” add-on has undergone major improvements. We listened carefully to the top issues on the containers roadmap and implemented the requested functionality and closed those issues. We added new components like PodDisruptionBudget (PDB), implemented opinionated defaults (e.g., using health plugin for pod health checks) and enhanced the JSON configuration schema to allow customers to customize more parameters of the add-on.
In order to follow this blog post you need:
- an AWS account
- an Amazon EKS cluster with the “coredns” add-on deployed
- a recent version of the AWS Command Line Interface (CLI)
- the kubectl command line tool compatible with your EKS cluster version
Issue #2033 requests adding a topologySpreadConstraints setting on the EKS CoreDNS addon configuration schema
Version v1.9.3-eksbuild.5 added the parameter topologySpreadConstraints to add-on JSON configuration schema which maps to K8s feature Pod Topology Spread Constraints. You can use topology spread constraints to control how Pods are spread across your Amazon EKS cluster among failure-domains such as availability zones, nodes, and other user-defined topology domains. This can help to achieve high availability as well as efficient resource utilization.
The configurationValues parameter in Amazon EKS add-ons API accepts configuration as a JSON or a YAML blob. The configuration blob needs to conform to the underlying add-on configuration JSON schema, which is available through the new Amazon EKS DescribeAddonConfiguration API. The JSON schema is unique for each add-on and evolves over time as we implement new configurable parameters.
Using the describe-addon-configuration option of the AWS Command Line Interface (CLI) EKS subcommand we receive the add-on JSON configuration schema with all settings. We are using the jq utility for better readability of the output.
The next command shows how we can evaluate if the attribute is present in the JSON configuration schema.
Now let’s create a YAML blog to apply a topologySpreadConstraints with topologyKey: topology.kubernetes.io/zone. We use whenUnsatisfiable: ScheduleAnyway in case we have a cluster which spans only a single availability zone.
Issue #1679 requested adding a Pod Disruption Budget (PDB) to the coredns addon and was implemented in v1.9.3-eksbuid.5 and higher.
A PDB limits the number of Pods of a replicated application, like coredns with two replica pods, that are down simultaneously during a node termination. PDB uses either minAvailable or maxUnavailable. Both can be expressed as integers or as a percentage. If you set minAvailable to x or x%, then x or x% Pods must always be available, even during a disruption. maxUnavailablemspecifies the number or percentage of pods that can be unavailable.
The PDB was first implemented in v1.9.3-eksbuild.5, which came with a default of minAvailable: 1. Starting with v1.10.1-eksbuild.2, we improved the logic: if there are two or fewer replicas of the coredns deployment, then we will set maxUnavailable to 1. Otherwise, we will set minAvailable to 2.
Starting with v1.9.3-eksbuild.7 and v1.10.1-eksbuild.4, the default logic of PDB is just maxUnavailable: 1 regardless of deployment replica count. We even made the PDB user configurable by extending the JSON configuration schema as follows:
Now users can tune PDB with either minAvailable or maxUnavailable as a number or percentage according to their needs.
Issue #2026 was a request to add the lameduck option by default to the coreDNS health plugin to minimize DNS resolution failures.
lameduck delays shutdown for DURATION seconds while the health endpoint still answers 200 acceptably. Adding lameduck to the health plugin minimizes DNS resolution failures during a CoreDNS pod restart (i.e., owing to health issue, node termination, etc) or a deployment rollout. This does come at the expense of increased rollout times. For highly available environments like Amazon EKS, this trade-off is likely acceptable.
Add-on version v1.9.3-eksbuild.6 and v1.10.1-eksbuild.3 implemented this behavior.
Issue #941 requested using /ready instead of /health on readiness probe of CoreDNS
Starting with coreDNS v1.9.3-eksbuild.6 and v1.10.1-eksbuild.3 we changed the readinessProbe of the CoreDNS pods by utilizing the CoreDNS upstream ready plugin using /ready path on port 8181. As the upstream ready plugin documentation explains:
By enabling ready an HTTP endpoint on port 8181 it returns 200 OK when all plugins that are able to signal readiness have done so. If some aren’t ready, then the endpoint returns a 503 with the body containing the list of plugins that are not ready. Once a plugin has signaled it is ready, it won’t be queried again.
Issue #1879 requested to set labels for pods of EKS-managed addons
The coredns add-on JSON configuration schema starting with v1.9.3-eksbuild.6 and v1.10.1-eksbuild.3 allows adding and changing pod labels and annotations. Pod labels use the podLabels property and come with no defaults. Pod annotations use the podAnnotations property and come with the don-not-evict annotation mentioned in the safe-to-evict annotations section above.
The following example shows adding a pod label foo: bar. Here you can see that using YAML blobs and string configuration values it isn’t necessary to use double quotes if they don’t contain special characters.
Important note when changing add-on configuration values: All previous changes are overwritten. To keep the previous configuration changes, the JSON and YAML configuration blob need to contain all required changes in a merged form like:
For adding pod annotations via podAnnotations property see the following example:
Call to action
With the recent changes AWS made it easier to use and customize the coredns add-on to meet customer needs. Use the Amazon Elastic Kubernetes Services (EKS) add-ons console or the AWS Command Line Interface (CLI) “aws eks create-addon” command and give it a try.
Do not forget to delete any example resources if you no longer need them, to avoid incurring future costs.
In this post, we showed you the new features are available in the coredns add-on and how you can use them. You may provide feedback on the Amazon coredns add-on by leaving a comment or opening an issue on the AWS Containers Roadmap that is hosted on GitHub.
Stay tuned as we continue to evolve our features around CoreDNS.