Why is my worker node status "Unhealthy" when I use the NGINX Ingress Controller with Amazon EKS?

Last updated: 2021-12-13

I use the NGINX Ingress Controller to expose the ingress resource, but my Amazon Elastic Kubernetes Service (Amazon EKS) worker nodes fail to use the Network Load Balancer.

Short description

The NGINX Ingress Controller sets the spec.externalTrafficPolicy option to Local to preserve the client IP. Also, requests aren't routed to unhealthy worker nodes. The following troubleshooting implies that you don't need to maintain the cluster IP address or preserve the client IP address.

Resolution

Check the health status of your worker node

Note: The following examples use the NGINX Ingress Controller v1.0.2.

1.    Create the mandatory resources for the NGINX Ingress Controller (from the Kubernetes website) in your cluster: 

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.2/deploy/static/provider/aws/deploy.yaml

By default, the NGINX Ingress Controller creates the Kubernetes Service ingress-nginx-controller with the .spec.externalTrafficPolicy option set to Local (from the GitHub website).

2.    Check if the external traffic policy (from the Kubernetes website) is set to Local:

$ kubectl -n ingress-nginx describe svc ingress-nginx-controller

Output:

Name:                     ingress-nginx-controller
Namespace:                ingress-nginx
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=ingress-nginx
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/version=1.0.2
                          helm.sh/chart=ingress-nginx-4.0.3
Annotations:              service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
                          service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: true
                          service.beta.kubernetes.io/aws-load-balancer-type: nlb
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Families:              <none>
IP:                       10.100.115.226
IPs:                      10.100.115.226
LoadBalancer Ingress:     a02245e77404f4707a725d0b977425aa-5b97f717658e49b9.elb.eu-west-1.amazonaws.com
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  31748/TCP
Endpoints:                192.168.43.203:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  30045/TCP
Endpoints:                192.168.43.203:443
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     30424
Events:                   <none>

Note: The Local setting drops packets that are sent to Kubernetes nodes that aren't running instances of the NGINX Ingress Controller. Assign NGINX pods (from the Kubernetes website) to the nodes that you want to schedule the NGINX Ingress Controller on.

3.    Check the iptables command that set up the DROP rules on the nodes that aren't running instances of the NGINX Ingress Controller:

$ sudo iptables-save | grep -i "no local endpoints"
-A KUBE-XLB-CG5I4G2RS3ZVWGLK -m comment --comment "ingress-nginx/ingress-nginx-controller:http has no local endpoints
        " -j KUBE-MARK-DROP
-A KUBE-XLB-EDNDUDH2C75GIR6O -m comment --comment "ingress-nginx/ingress-nginx-controller:https has no local endpoints " -j KUBE-MARK-DROP

Set the policy option

Update the pec.externalTrafficPolicy option to Cluster:

$ kubectl -n ingress-nginx patch service ingress-nginx-controller -p '{"spec":{"externalTrafficPolicy":"Cluster"}}'
service/ingress-nginx-controller patched

Note: By default, NodePort services perform source address translation (from the Kubernetes website). For NGINX, this means that the source IP of an HTTP request is always the IP address of the Kubernetes node that received the request. If you set a NodePort to the value of the externalTrafficPolicy field in the ingress-nginx service specification to Cluster, then you can't maintain the source IP address.

Did this article help?

Submit feedback

Do you need billing or technical support?

Contact AWS Support