Containers
Amazon EKS add-ons preserve customer edits
Introduction
As part of AWS re:Invent 2020, the Amazon Elastic Kubernetes Service (Amazon EKS) Team announced the release of Amazon EKS add-ons. The addition of add-ons was driven by customer feedback and a desire to simplify the management of commonly used operational software. With add-ons, you can configure, deploy, and update operational software, which provides key functionality to support your Kubernetes applications. Add-ons include critical tools for cluster networking such as the Amazon VPC CNI, as well as operational software for observability, storage, management, and security. By using add-ons, you can configure Amazon EKS to start the add-on software on the cluster and deploy new versions of the add-on with a single command.
Every add-on includes the latest security patches and bug fixes and is validated by AWS to work with Amazon EKS. This reduces the amount of work you need to do in order to start, manage, and upgrade production-ready Kubernetes clusters, which helps to keep your clusters stable and secure.
Currently, Amazon EKS add-ons support the following software:
Server-side apply and overwriting edits
Amazon EKS add-ons uses a feature, made available in Kubernetes version 1.18, called server-side apply. Server-side apply migrates the functionality of kubectl apply
to the Kubernetes application programming interface (API) server and provides fine-grained control over resolving conflicts, which can happen during configuration changes. Server-side apply introduces a new configuration merging algorithm and field ownership tracking within the Kubernetes control plane. This allows Kubernetes to detect and resolve conflicts automatically.
Once add-ons are added to an Amazon EKS cluster, you can modify the configuration of the respective managed software through the Kubernetes API. Until now, however, managed add-ons prevented configuration drift by automatically overwriting configurations every 15 minutes. This meant that any changes to managed add-ons, made through the Kubernetes API after add-on creation, were overwritten by the automated drift-prevention process. While preventing drift is a desirable goal, overwriting subsequent edits to software managed by add-ons also prevents customers from making deliberate and needed changes to Kubernetes resources under add-on management.
Solution overview
Amazon EKS add-ons behavior update: Preserving customer edits
With the latest release of Amazon EKS add-ons, we have added functionality to preserve customer edits to software managed by add-ons. Instead of overwriting software configurations every 15 minutes, we perform health checks of software managed by add-ons. This means that add-ons no longer write or overwrite anything on the cluster during steady-state operations. The only time that add-ons now write to a cluster is during create, update, and delete add-on operations that are triggered by customers.
In addition to no longer overwriting configurations every 15 minutes, a new ResolveConflicts
option, PRESERVE
, has been added to the Amazon EKS API. This new PRESERVE
option preserves any in-cluster configuration that was made through the Kubernetes API to add-on software, which is outside of the scope of add-ons. As seen below, in the Example add-on update workflow, the --resolve-conflicts PRESERVE
parameter and argument can be used with the aws eks update-addon
AWS Command Line Interface (AWS CLI) command to prevent the add-on update operation from overwriting any edits made to the add-on software, after initial creation.
Example add-on update workflow
Step 1 – Create the add-on in the cluster:
Note: To avoid lost configuration settings, you must wait to edit the add-on until after the add-on creation process has finished. The following AWS CLI command can be used to check the status of an add-on:
Step 2 – Modify desired configuration for the add-on in the cluster through the Kubernetes API:
Note: To view the server-side apply managed fields for a Kubernetes resource, you can use the --show-managed-fields
parameter, as seen below:
Step 3 – When performing an add-on update, provide the --resolve-conflicts PRESERVE
parameter and argument:
Step 4 – Confirm that the update was successful and that the edited configuration was preserved:
When performing an add-on update operation, providing the PRESERVE
option causes the operation to compare what currently exists on the cluster with what the add-on expects, to identify any field that differs or has been modified externally. Once identified, when performing the related add-on update operation, the fields that have been modified remain untouched while everything else is updated. This allows configurations made directly through the Kubernetes API to be preserved when updating an add-on. Default create and update behavior (failing on conflict during an operation) remains unchanged.
Note: Amazon EKS add-ons will allow any modified field to be preserved during an update operation. This means that when you use the PRESERVE
option, it is possible to modify a field that becomes preserved, while simultaneously incompatible in a different version of the add-on. Therefore, it is recommended to follow best practices, and test any updates on non-production clusters, with desired and specific configuration changes, to confirm that the updates work as intended before introducing like changes to production clusters.
Preventing conflicts in create and update operations
From the beginning, the Amazon EKS API has supported detecting, preventing, or overwriting conflicts during add-on create and update operations. This functionality remains unchanged. If you attempt to create or update a managed add-on after editing a field that is or will be managed by the respective add-on and you do not provide the OVERWRITE
option, existing conflicts cause the operation to fail, as seen in the following code:
To get around potential conflicts, you can use the OVERWRITE
option with create operations, or the OVERWRITE
or PRESERVE
options with update operations, as seen below, in the aws eks create-addon
command:
The previous command can be used even if the previous operation failed due to a detected conflict:
After add-on creation, you can edit the add-on resources through the Kubernetes API, and then use the PRESERVE
option with an add-on update operation to update the add-on while preserving your edits.
Another use case for the OVERWRITE
option is to reset Kubernetes resource configurations. If you modify the Kubernetes resources of software that is, or can be, managed by add-ons, you can use the OVERWRITE
option of the create or update operations to reset the configuration back to a known good configuration, prescribed by the add-on.
Preserving Kubernetes resources beyond add-on deletion
If you no longer want or need an Amazon EKS add-on in your cluster, you can delete the add-on and choose whether or not to retain the underlying Kubernetes resources. As seen below, using the --preserve
flag with the delete-addon
command will retain the underlying Kubernetes resources, while removing the add-on management from the software:
Note: When modifying production configurations, it is a best practice to ensure that configurations can be reapplied through solutions like back-up and restore, or GitOps, should the need arise.
Conclusion
In this post, we showed you how Amazon EKS add-ons reduce the heavy-lifting required to manage common operational software in your clusters. As mentioned in the previous sections of this post, every add-on includes the latest security patches and bug fixes, and is validated by AWS to work with Amazon EKS. Add-ons are an evolution of Amazon EKS, which began with us managing the control plane (managed upgrades) and data plane (managed node groups) and progressed with managing the common operational cluster software (Amazon EKS add-ons). With the latest PRESERVE
option for the Amazon EKS API add-on update operations, you can modify configurations of software under add-on management without losing your specific edits. You can use add-ons to help manage software, while still customizing the software for your individual use cases.
Calls to action
Give Amazon EKS add-ons a spin!
If you are running Amazon EKS and you haven’t tried add-ons yet, now is the time. Maybe you’ve held off due to the aforementioned drift protection and didn’t want your Kubernetes resource edits overwritten. With the new add-on update --resolve-conflicts PRESERVE
option as well as the existing add-on delete --preserve
option, you can use add-ons while preserving your edits. For the future, efforts are underway to develop a more robust configuration management system for Amazon EKS add-ons, so please stay tuned for future announcements!
Check out our containers roadmap!
If you have ideas about how we can improve Amazon EKS add-ons or other aspects of our container services, please use our containers roadmap to provide us feedback and review our existing roadmap items.