Amazon EKS now supports Kubernetes version 1.28

Introduction

The Amazon Elastic Kubernetes Service (Amazon EKS) team is pleased to announce support for Kubernetes version 1.28 for Amazon EKS and Amazon EKS Distro. Amazon EKS Anywhere (release 0.18.0) also supports Kubernetes 1.28. The theme for this version was chosen as a play on words that combines plant and Kubernetes to evoke the image of a garden. Hence, the fitting release name, Planternetes. In their official release announcement, the Kubernetes release team said this of the release, “people behind this release come from a wide range of backgrounds.”

Kubernetes 1.28 highlights

This post covers some of the notable removals, deprecations, and enhancements in the Kubernetes version 1.28 release. For starters, it’s important to note that this release brings some key changes, including the extension of the supported version skew between control plane and node components. There are also some awesome enhancements in v1.28 that we’re all excited about, such as support for advanced stateful workload management.

Below are a few enhancements that have our technical community excited about the 1.28 release. For a complete list of changes and updates in Kubernetes version 1.28, check out the Kubernetes release blog.

Changes to supported skew between control plane and node versions

Kubernetes v1.28 introduces a more lenient version compatibility policy for its core components, which expands the supported skew between the Kubernetes Application Programming Interface (API) server and the kubelet by one minor version from n-2 to n-3. This means that the node components for the oldest supported minor version can now work with the Amazon EKS control plane components for the newest supported minor version. For example, if your Amazon EKS version is 1.28, then the oldest kubelet version you can use is 1.25. This change is supported across AWS Fargate, managed node groups, and self-managed nodes. Ultimately, this change allows you a little more time to update the kubelet on the data plane. While this change does give you some additional time for kubelet minor version updates on the data plane, it’s important to note that this doesn’t negate the importance of keeping up with newer Amazon Machine Image (AMI) versions for security reasons, especially in light of potential Common Vulnerabilities and Exposures (CVEs). Remember, the n-3 skew change is only applicable to versions that are currently supported. If a version has been sunset, you’ll be limited to using only the versions that remain supported.
To learn more, see Changes to supported skew between control plane and node versions.

Stateful workload enhancements graduates to stable

Kubernetes 1.28 unveils an advanced suite of storage features that specifically enhance the handling of stateful workloads. These enhancements, described in the Kubernetes Enhancement Proposals (KEPs) (#2268, #3333), are immediately available as they have graduated to stable. These collectively offer a powerful toolset that ensures stateful workloads, such as VolumeAttachments and PersistentVolumeClaims (PVCs), can be managed more efficiently within the cluster. When it comes to handling storage without a defined StorageClass, you still have options for creating PersistentVolumes. You can manually provision and create the PV directly, such as creating a PV for an NFS mount, defining the PV with the NFS server details, without referencing a StorageClass. The PV could then be claimed via a PVC to provision storage for stateful workloads.
#2268 has graduated to stable and the NodeOutOfServiceVolumeDetach feature gate is enabled by default. This feature enables recovery from non-graceful node shutdown, ensuring that stateful workloads can successfully fail over to a different node. It addresses previous limitations in handling node shutdown on various platforms and ensures that existing VolumeAttachments are disassociated from the original shut-down node, allowing for a seamless transition of stateful applications.
#3333 has graduated to stable and the RetroactiveDefaultStorageClass feature gate is enabled by default. This feature brings the stable graduation of automatic and retroactive assignment of a default StorageClass for PVCs. If a PVC doesn’t have a StorageClassName defined, Kubernetes automatically sets one. This feature enhances the robustness of storage management for stateful workloads, ensuring consistent handling of storage resources within a Kubernetes cluster.
To learn more, refer to Kubernetes 1.28: Non-Graceful Node Shutdown Moves to GA and Kubernetes v1.28: Retroactive Default StorageClass move to GA.

Advanced topology management and fine-tuned pod placement reached beta

Kubernetes v1.28 introduces a sophisticated array of topology management features. These features, detailed in the KEP (#3545), are already enabled by default and available in beta. Together, they form a robust powerhouse that addresses the challenges of orchestrating pod placement in a way that maximizes resource efficiency, enhances performance, and fortifies fault tolerance. By using both options, you can ensure that your pods are not only placed close to each other for better performance but also adhere to other constraints like availability zones, thereby making efficient use of your cluster resources.
TopologyManagerPolicyBetaOptions empowers you with advanced settings for fine-tuning pod placement based on factors such as node topology and resource availability. A key setting is the prefer-closest-numa-nodes policy Non-Uniform Memory Access (NUMA) nodes are groups of CPUs and memory. Normally, the Topology Manager spreads pods across available NUMA nodes, but this setting tells the Topology Manager to favor nearby NUMA nodes. Enabling this option allows it to make more informed decisions about pod placement and instructs it to consider the distance between NUMA nodes, which is crucial for applications that are latency-sensitive or require high throughput.
TopologyManagerPolicyOptions provides offers an extra layer of granularity in tailoring pod placement according to unique cluster topologies. This feature allows you to define constraints based on node labels, affinity rules, and resource constraints, which enables fine-tuned control over pod placement. This allows you to define constraints using node labels, affinity rules, and resource limitations. For instance, you can specify that pods should be confined to a particular availability zone for optimized resource utilization.
Note that while these are powerful features, they require adjustments to your existing pod specifications or node configurations. We recommend testing their impact comprehensively and putting rollback plans in place for any issues that may arise. Should you encounter any issues, we recommend disabling the feature and restarting the kubelet.
To learn more, refer to Control Topology Management Policies on a node.

Deprecations and other updates

In line with the release of Kubernetes version 1.28, Amazon EKS introduces some critical changes, including deprecations that directly impact Amazon Elastic Compute Cloud (Amazon EC2) instance compatibility. It’s crucial to review these updates and adapt your Amazon EKS clusters and applications accordingly before transitioning to Kubernetes 1.28. Here are the key changes in this context.

Amazon EC2 P2 instance deprecation

Starting with Kubernetes version 28, you will no longer be able to use Amazon EC2 P2 instances with the Amazon EKS optimized accelerated Amazon Linux AMIs out of the box. These AMIs for Kubernetes versions 1.28 or later will support NVIDIA 525 series or later drivers, which are incompatible with the P2 instances. However, NVIDIA 525 series or later drivers are compatible with the P3, P4, and P5 instances, so you can use those instances with the AMIs for Kubernetes version 1.28 or later. Before upgrading your Amazon EKS clusters to version 1.28, migrate any P2 instances to P3, P4, and P5 instances. We also recommend proactively upgrading your applications to work with the NVIDIA 525 series or later.

End of life support

Amazon EKS provides support for at least four Kubernetes versions at any given time. Given the nature of the Kubernetes release cycle, it’s critical for all customers to have an ongoing upgrade plan. If you’re still running an older version of Kubernetes like 1.23 and 1.24, then please consider upgrading to one of the newer supported versions. The end-of-support for 1.23 clusters is October 11, 2023, and the end-of-support for 1.24 clusters will be in January 2024. If you have more questions concerning Amazon EKS version support, then refer to our FAQ.

Conclusion

In this post, we showed you the notable changes in Kubernetes version v1.28 and highlighted some of the most exciting features available. Be sure to check out the other improvements documented in Kubernetes v1.28 release notes. If you need assistance with upgrading your cluster to the latest Amazon EKS version, refer to our documentation here.

Containers