How do I configure Amazon EKS worker nodes to clean up the image cache at a specified percent of disk usage?

Last updated: 2020-06-10

I want to use Amazon Elastic Kubernetes Service (Amazon EKS) worker nodes to clean up the image cache at a specified percent of disk usage.

Short Description

To clean up the image cache with Amazon EKS worker nodes, use the following kubelet garbage collection arguments:

  • The --image-gc-high-threshold argument defines the percent of disk usage that triggers image garbage collection. The default is 85%.
  • The --image-gc-low-threshold argument defines the percent of disk usage that image garbage collection attempts to free. The default is 80%.

Note: In the following resolution, the kubelet garbage collection arguments are used to clean up the image cache in the worker node when the disk usage reaches 70%. The --image-gc-high-threshold is set to 70%. The --image-gc-low-threshold is set to 50%.

To add kubelet garbage collection arguments to a new Amazon EKS worker node, complete the steps in the following sections:

  1. Create a worker node with kubelet garbage collection arguments
  2. Verify that the new kubelet garbage collection arguments are in the node configz endpoint

To add kubelet garbage collection arguments to an existing Amazon EKS worker node, complete the steps in the following sections:

  1. Add the kubelet garbage collection arguments to an existing worker node
  2. Verify that the new kubelet garbage collection arguments are in the node configz endpoint

Resolution

Note: The following resolution applies to Amazon EKS-optimized Linux Amazon Machine Images (AMIs).

Create a worker node with kubelet garbage collection arguments

1.    Launch an Amazon Elastic Compute Cloud (Amazon EC2) instance. Be sure to include the kubelet arguments --image-gc-low-threshold and --image-gc-high-threshold in the UserData section of the Step 3: Configure Instance Details page.

For example:

#!/bin/bash
set -o xtrace
/etc/eks/bootstrap.sh your-cluster-name --kubelet-extra-args '--image-gc-low-threshold=50 --image-gc-high-threshold=70'

Note: Replace your-cluster-name with your Amazon EKS cluster name.

2.    Add the worker node to your Amazon EKS cluster.

3.    To verify the changes in the arguments, follow the steps in the Verify that the new kubelet garbage collection arguments are in the node configz endpoint section.

Add the kubelet garbage collection arguments to an existing worker node

Important: The following steps require that you connect to an existing worker node with SSH and have sudo access. These steps must be completed on all the existing worker nodes in your Amazon EKS cluster.

1.    Connect to an existing worker node using SSH.

2.    Open the /etc/kuberentes/kubelet/kubelet-config.json file in your worker nodes.

If the worker node was launched using EKSCTL, then open /etc/eksctl/kubelet.yaml.

For example:

sudo vi /etc/kubernetes/kubelet/kubelet-config.json

#WORKER NODES LAUNCHED USING EKSCTL

sudo vi /etc/eksctl/kubelet.yaml

3.    Add the kubelet garbage collection arguments to the kubelet-config.json file or kubelet.yaml file, depending on how your worker nodes were launched. Then, save the file.

For example:

{
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  .
  .
  .
  "imageGCHighThresholdPercent": 70,         ==> Add the argument under the same alignment as the "kind"
  "imageGCLowThresholdPercent": 50,
  "maxPods": ...
}


#WORKER NODES LAUNCHED USING EKSCTL

kind: KubeletConfiguration
kubeReserved:
  cpu: 70m
  ephemeral-storage: 1Gi
  memory: 1843Mi
serverTLSBootstrap: true
imageGCHighThresholdPercent: 70        ==> Add the arguments under the alignment "Kind" in the yaml file
imageGCLowThresholdPercent: 50

4.    To restart the kubelet service in the worker node, run the following command:

sudo service kubelet restart

5.    To verify the changes in the arguments, follow the steps in the Verify that the new kubelet garbage collection arguments are in the node configz endpoint section.

Verify that the new kubelet garbage collection arguments are in the node configz endpoint

1.    To get the name of your worker nodes, run the following command:

kubectl get nodes

2.    To open a connection to the API server, run the following command:

kubectl proxy

3.    To check the node configz, open a new terminal, and then run the following command:

curl -sSL "http://localhost:8001/api/v1/nodes/node_name/proxy/configz" | python3 -m json.tool

Note: Replace node_name with your node name from the list of nodes retrieved in step 1. If curl and python aren't available, then you can open the URL in a web browser.

The preceding command returns output from the kubeletconfig, which includes your settings from the bootstrap.sh file.

For example:

{
"kubeletconfig": {
  .
  .
  "imageGCHighThresholdPercent": 70,          <=== The new value is set to 70 as given in UserData
  "imageGCLowThresholdPercent": 50,           <=== The new value is set to 50 as given in UserData
  .
  .
}
}

Did this article help you?

Anything we could improve?


Need more help?