Network Load Balancer Support in Kubernetes 1.9

中文版

Applications deployed on Amazon Web Services can achieve fault tolerance and ensure scalability, performance, and security by using Elastic Load Balancing (ELB). Incoming application traffic to ELB is distributed across multiple targets, such as Amazon EC2 instances, containers, and IP addresses. In addition to Classic Load Balancer and Application Load Balancer, a new Network Load Balancer was introduced last year. It is capable of handling millions of requests per second while maintaining ultra-low latencies. This guest post by Micah Hausler, who added support for Network Load Balancer in Kubernetes, explains how you can enable that support in your applications running on Kubernetes.

– Arun

In September, AWS released the new Network Load Balancer, which for many in the AWS community is an exciting advance in the load balancing space. Some of my favorite features are the preservation of the original source IP without any additional setup, and the ability to handle very long running connections. In this post, we’ll show how to create a Network Load Balancer from a Kubernetes cluster on AWS.

Classic Load Balancing in Kubernetes

I’ve been using Kubernetes on AWS for a year and a half, and have found that the easiest way route traffic to Kubernetes workloads has been with a Kubernetes Load Balancer service. An example configuration for a service might look like this:

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
  annotations: {}
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
type: LoadBalancer

This would create a Classic ELB routing TCP traffic on a frontend port 80 to port 80 on a pod. The end result is that the client’s source IP is lost and replaced with the ELB’s IP address. Workarounds have included enabling Proxy Protocol or using an X-Forwarded-For header on HTTP or HTTPS listeners with Kubernetes metadata annotations. There are a variety of additional annotations to configure ELB features like request logs, ACM Certificates, connection draining, and more.

Network Load Balancing in Kubernetes

Included in the release of Kubernetes 1.9, I added support for using the new Network Load Balancer with Kubernetes services. This is an alpha-level feature, and as of today is not ready for production clusters or workloads, so make sure you also read the documentation on NLB before trying it out. The only requirement to expose a service via NLB is to add the annotation service.beta.kubernetes.io/aws-load-balancer-type with the value of nlb.

A full example looks like this:

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
  labels:
    app: nginx
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
type: LoadBalancer

To try this for yourself, see Arun’s post on managing a Kubernetes cluster with kops and set the kubernetes-version to 1.9.1.

kops create cluster \
--name cluster.kubernetes-aws.io \
--zones us-west-2a \
--kubernetes-version 1.9.1 \
--yes

Once your cluster is created, you’ll need to grant the Kubernetes master the new permissions to create an NLB. (Once kops officially supports Kubernetes 1.9, this additional step will not be necessary.)

It can take a few minutes for the Network Load Balancer to be created and register the nodes as valid targets (even though the NLB hostname is reported back to Kubernetes). You can check the status in the AWS Console:

If you follow the above example, once the Target Group instances (the Kubernetes nodes) pass the initial setup, you’ll see one node marked as healthy and one as unhealthy.

Nodes are added to an NLB by instance ID, but, to explain a little bit of Kubernetes networking, the traffic from the NLB doesn’t go straight to the pod. Client traffic first hits the kube-proxy on a cluster-assigned nodePort and is passed on to all the matching pods in the cluster. When the spec.externalTrafficPolicy is set to the default value of Cluster, the incoming LoadBalancer traffic may be sent by the kube-proxy to pods on the node, or to pods on other nodes. With this configuration the client IP is sent to the kube-proxy, but when the packet arrives at the end pod, the client IP shows up as the local IP of the kube-proxy.

By changing the spec.externalTrafficPolicy to Local, the kube-proxy will correctly forward the source IP to the end pods, but will only send traffic to pods on the node that the kube-proxy itself is running on. Kube-proxy also opens another port for the NLB health check, so traffic is only directed to nodes that have pods matching the service selector. This could easily result in uneven distribution of traffic, so use a DaemonSet or specify pod anti-affinity to ensure that only one pod for a given service is on a node.

At this point, the Network Load Balancer is ready for use!

There are several other differences in the new Network Load Balancer from how Classic ELBs work, so read through the Kubernetes documentation on NLB and the AWS NLB documentation.

Contribute to Kubernetes!

Adding the NLB integration was my first contribution to Kubernetes, and it has been a very rewarding experience. The Kubernetes community organizes itself into Special Interest Groups (SIGs), and the SIG Cloud Provider has been very welcoming and supportive. I’m thankful to all the reviewers and collaborators from SIG Cloud Provider and from Amazon for their insight.

If you’re interested in seeing deeper integration with AWS or NLB specifically, please participate in the community! Come to a SIG Cloud Provider meeting, file feature requests, or report bugs on Github: Kubernetes is only what it is today because of the community!

Gists containing the above code snippets:

https://gist.github.com/micahhausler/4f3a2ee540f5714e6dd91b4bacace3ae

At the time of writing, Micah Hausler was a Senior Site Reliability Engineer at Skuid where he led the DevOps team and was a contributor to Kubernetes. You can (still) find him at @micahhausler on Twitter, Github, and Kubernetes Slack.

The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.