Containers

Amazon VPC CNI now supports Kubernetes Network Policies

Introduction

Today, we’re excited to announce the native support for enforcing Kubernetes network policies with Amazon VPC Container Networking Interface (CNI) Plugin. You can now use Amazon VPC CNI to implement both pod networking and network policies to secure the traffic in your Kubernetes clusters. Native support for network policies has been one of the most requested features on our containers roadmap.

By default, Kubernetes allows all pods to communicate with each other with no restrictions. Kubernetes Network Policies enable you to define and enforce rules for the flow of traffic between pods. They act as a virtual firewall, which allows you to segment and secure your cluster by specifying ingress (i.e., incoming) and egress (e.g., outgoing) network traffic rules based on various criteria such as pod labels, namespaces, IP addresses, IP blocks (CIDR ranges) and ports. Until today, customers used third-party network policy plugins to implement network policies, which often resulted in operational and management overhead. The integrated functionality in Amazon VPC CNI simplifies cluster configuration and deployment. By restricting traffic flows, you can achieve a stronger security posture without having to worry about scaling issues.

With native support for network policy in Amazon VPC CNI, when running Kubernetes on AWS, you can now create policies that isolate sensitive workloads and protect them from unauthorized access. This level of granular control enables you to implement the principle of least privilege, which ensures that only authorized pods are allowed to communicate with each other. Network policies provide defense-in-depth mechanism that extend the security features provided by Amazon VPC, such as Amazon Elastic Compute Cloud (Amazon EC2) security groups (SGs) and network access control lists (NACLs).

Amazon Elastic Kubernetes Service (Amazon EKS) fully supports the upstream Kubernetes Network Policy API, ensuring compatibility and adherence to Kubernetes standards. This means that you can use all the capabilities of the Network Policy API within your Amazon EKS cluster. You have the flexibility to define policies using different identifiers supported by the upstream API. This allows for communication definitions across pods that enhances security and isolation within your cluster.

How it works

When Kubernetes network policies were first introduced, the default and widely adopted implementation was iptables. While iptables proved effective for enforcing network policies, it presented some limitations, particularly as Kubernetes clusters scaled up in size. Managing a large number of iptables rules can be challenging, and performance issues may arise as the number of rules increase due to the sequential evaluation of rules for each packet.

To address these challenges and improve performance, Amazon EKS adopted an advanced approach by implementing network policies using extended Berkeley Packet Filter (eBPF). This technology has gained substantial traction in recent years as an alternative implementation for network policies. eBPF offers more efficient packet filtering and unlocks the potential for better overall performance when compared to iptables. It achieves this by enabling the execution of custom code directly within the kernel itself.

To facilitate the implementation of network policies, Amazon EKS introduces three key components that work seamlessly together:

  • Network Policy Controller: With the latest version of Amazon VPC CNI, Amazon EKS announces the launch of a Network Policy controller. The controller is now publicly available. When you create a new Amazon EKS cluster, the network policy controller is automatically installed on the Kubernetes control plane when the feature is enabled. It actively monitors the creation of network policies within your cluster and reconciles policy endpoints. Subsequently, the controller instructs the node agent to create or update eBPF programs on the node by publishing pod information through the policy endpoints. Network policy controller configures policies for pods in parallel to pod provisioning, until then new pods will come up with default allow policy. All ingress and egress traffic is allowed to and from the new pods until they are reconciled against the existing policies. To effectively manage network policies on self-managed Kubernetes clusters, you need to deploy a network policy controller. In this regard, you can find detailed information and key considerations for self-managed clusters in the walkthrough section of this post.
  • Node Agent: The latest version of Amazon VPC CNI also installs a node agent along with CNI binary and ipamd plugin on every node within your cluster. The node agent is bundled with VPC CNI and runs as container under aws-node Daemonset. This node agent receives policy endpoints from controllers when the network policies are applied to the cluster. The node agent plays a critical role in managing eBPF programs, which paves the way for seamless enforcement of network policies.
  • eBPF SDK (Software Development Kit): To further help with support and operations, Amazon VPC CNI includes an SDK that provides an intuitive interface to interact with eBPF programs on the node. This SDK allows for runtime introspection, tracing, and analysis of eBPF execution that aides in identifying and resolving connectivity issues.

Starting today, network policies will be supported on new Amazon EKS (IPv4 and IPv6) clusters using v1.25 and later. Amazon EKS automatically upgrades all existing clusters to the latest Amazon EKS platform version for their corresponding Kubernetes minor version. You can enforce network policy when automatic upgrades of platform versions supporting the feature are rolled out on existing clusters. For specific platform version, please refer to the Amazon EKS user guide.

To enable network policies in your cluster, ensure that your cluster is running Amazon VPC CNI version 1.14.0 or a later version. When creating a new cluster, you may specify Amazon VPC CNI version 1.14.0 or later Update Amazon VPC CNI for clusters not created with Amazon VPC CNI version 1.14.0. Once the platform version update is completed for existing clusters, make sure to update the VPC CNI to the supported version.

At launch, the network policies are supported on Amazon EKS-optimized Amazon Linux AMIs, Amazon EKS-optimized Bottlerocket AMIs, and Amazon EKS-optimized Ubuntu Linux AMIs using Kernel version 5.10 or later. Amazon EKS-optimized AMI built using Kernel version 5.10 or later mounts the eBPF file system /sys/fs/bpf automatically, which is required to enforce network policies on the nodes. On existing clusters, when the Amazon EKS platform version is updated, it is recommended that you update your nodes to the latest Amazon EKS optimized AMI. For nodes using Kernel version 5.10 or below, please refer to the Amazon EKS user guide for instructions on mounting the eBPF file system. For instructions on how to create your own custom AMI with eBPF system support, see the Amazon EKS-optimized Amazon Linux AMI build script.

Getting started

The network policy feature is supported for clusters running Kubernetes minor versions 1.25 and above but disabled by default at launch. You can enable Network policy for your cluster using Amazon VPC CNI configuration parameters when you create a cluster.

In order for the Amazon VPC CNI to manage IP addresses, it requires AWS Identity and Access Management (AWS IAM) permissions. Amazon EKS recommends creating a separate AWS IAM role with the permissions defined in AmazonEKS_CNI_Policy managed policy then associate that role with the VPC CNI using IRSA (i.e., AWS IAM roles for service account). Since IRSA requires an OpenID Connect (OIDC) endpoint, we’ll create an IAM OIDC provider as part of cluster creation. For more information, see Configuring the Amazon VPC CNI plugin for Kubernetes to use IAM roles for service accounts.

Copy the following configuration and save it to a file called cluster.yaml:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: network-policy-demo
  version: "1.27"
  region: us-west-2

iam:
  withOIDC: true

vpc:
  clusterEndpoints:
    publicAccess: true
    privateAccess: true

addons:
  - name: vpc-cni
    version: 1.14.0
    attachPolicyARNs: #optional
    - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy 
    configurationValues: |-
      enableNetworkPolicy: "true"
  - name: coredns
  - name: kube-proxy

managedNodeGroups:
  - name: x86-al2-on-demand
    amiFamily: AmazonLinux2
    instanceTypes: [ "m6i.xlarge", "m6a.xlarge" ]
    minSize: 0
    desiredCapacity: 2
    maxSize: 6
    privateNetworking: true
    disableIMDSv1: true
    volumeSize: 100
    volumeType: gp3
    volumeEncrypted: true
    tags:
      team: "eks"

eksctl create cluster -f cluster.yaml

Wait for cluster creation to be complete to confirm if Amazon VPC CNI now runs node agent.

kubectl get ds -n kube-system aws-node -o jsonpath='{.spec.template.spec.containers[*].name}{"\n"}'

Output:

aws-node aws-eks-nodeagent

Using eBPF SDK

The latest version of the Amazon VPC CNI ships an SDK that provides an interface to interact with eBPF programs on the node. The SDK is installed when the aws-node is deployed onto the nodes. You can find the SDK binary installed under /opt/cni/bin directory on the node. Consider using this SDK when you would like to identify connectivity issue. Make sure eBPF programs are being created for the pods on the node. You can check this by connecting using AWS Systems Manager (AWS SSM) through the Amazon EC2 console to a node.

At launch, the SDK provides support for fundamental functionalities, such as inspecting eBPF programs and maps. Please let us know if you have any specific usage requests by opening a GitHub issue here. Visit user guide to learn about SDK usage.

sudo /opt/cni/bin/aws-eks-na-cli ebpf progs

Deploy a sample application

Now, we’ll deploy a sample NGINX application called demo-app and a simple client application in the default namespace. Additionally, we’ll create another client application in a non-default namespace called another-ns.

Diagram of network policy restricting traffic between pods in different namespaces

git clone https://github.com/aws-samples/eks-network-policy-examples.git
cd eks-network-policy-examples
kubectl apply -f advanced/manifests/

Wait for all the pods to enter Running state.

kubectl get pods --all-namespaces --watch

Implement network policies

By default, Kubernetes allows all pods to communicate freely with all other pods. If no Kubernetes network policies are applied to a pod, then all traffic to and from the pod is permitted.

Verify access and allow all ingress and egress

You can use a different terminal to verify connectivity with a client pod that was deployed in a previous step.

kubectl exec -it client-one -- curl demo-app

You would see a response, indicating successful API call.

Deny all traffic

Let’s enable isolation by applying a deny all policy across all namespaces for the demo-app application.

kubectl apply -f advanced/policies/01-deny-all-ingress.yaml

We have selected all the pods in the network policy and any ingress traffic to demo-app will be denied.

kubectl exec -it client-one -- curl demo-app

The previous command results in a timeout.

curl: (28) Connection timed out after 3001 milliseconds<br />command terminated with exit code 28

Allow ingress from same namespace

In the following step, we’ll allow ingress traffic from client-one to demo-app from within the same namespace.

kubectl apply -f advanced/policies/03-allow-ingress-from-samens-client-one.yaml

It should return the HTML of the NGINX welcome page.

kubectl exec -it client-one -- curl --max-time 3 demo-app

Allow ingress from another-ns namespace

Next, we’ll allow traffic from another namespace.

kubectl apply -f advanced/policies/04-allow-ingress-from-xns.yaml

Now, you should be able to access demo-app from a client in another namespace. You should see a welcome message returned.

kubectl exec -it -n another-ns another-client-one -- curl --max-time 3 demo-app.default

Deny egress traffic

Apply all egress isolation from client-one pod in the default namespace.

kubectl apply -f advanced/policies/06-deny-egress-from-client-one.yaml

You can see that all egress, including DNS lookups from client one, is denied.

kubectl exec -it client-one -- nslookup demo-app

Allow egress traffic

Let’s allow egress from multiple ports and namespaces, including DNS traffic.

kubectl apply -f advanced/policies/08-allow-egress-to-demo-app.yaml

Now egress traffic to DNS and the demo-app must be allowed.

kubectl exec -it client-one -- curl --max-time 3 demo-app

Key considerations

Network Policy and Security Groups for pod

Amazon VPC CNI in IPv4 mode offers a powerful feature known as Security groups for pods. This feature enables you to use Amazon EC2 security groups to define rules governing inbound and outbound network traffic to and from the pods deployed on your nodes. By using a combination of security groups for pods and network policies, you can enhance your security posture. When network policy is enabled, security groups for pods serve as an additional layer in your defense-in-depth strategy. Network policies allow you to enforce granular controls over the flow of network traffic within your cluster whereas security groups for pods provides added protection by leveraging Amazon’s semantic controls to manage communications with resources within the Virtual Private Cloud (VPC), such as Amazon RDS databases. Amazon EKS strongly recommends employing network policies to restrict network communication between pods, thus reducing the attack surface and minimizing potential vulnerabilities. Please refer Amazon EKS Best Practices for complete guidance on using security groups alongside network policy.

Network Policy and third-party CNI plugins

Amazon VPC CNI now offers support for network policies defined using the Kubernetes NetworkPolicy API. It is important to note that clusters currently running third-party CNI plugins for pod networking must migrate to Amazon VPC CNI in order to enable the network policy feature.

As of the time of writing this post, Amazon VPC CNI doesn’t support in-place migration of third-party network policy plugins to Amazon VPC CNI. To ensure a smooth transition, Amazon EKS recommends converting your existing 3P NetworkPolicy APIs to the Kubernetes NetworkPolicy API before enabling Amazon VPC CNI network policy support. Before applying any changes to your production environment, we recommended you test your migrated policies on a separate test cluster. This allows you to identify and address any potential issues or inconsistencies in the behavior of pod communication.

To maintain consistency and avoid unexpected pod communication behavior, it is recommended to run only a single network policy plugin at any given time. It is crucial to remove any existing third-party network policy CNI plugin before upgrading to the latest version of Amazon VPC CNI. Additionally, recycling the worker nodes helps eliminate any leftover changes on a node, such as iptables or eBPF programs. For a detailed sequence of steps to migrate to Amazon VPC CNI network policy support, please refer to the documentation provided here. You can use the scripts provided in this open-source repository to convert your third-party NetworkPolicy to Kubernetes NetworkPolicy artifacts. These resources are not officially supported by Amazon EKS and are best effort only. The resources aim to help streamline and simplify the migration process, which ensures a successful transition to Amazon VPC CNI network policy support.

Self-managed Kubernetes clusters

With the latest version of Amazon VPC CNI, you can now enforce network policies on self-managed Kubernetes on AWS clusters. In addition to updating Amazon VPC CNI, you’ll install Amazon network policy controller on your self-managed Kubernetes cluster to enable network policy. Follow the instructions provided in this GitHub repository to deploy the network policy controller. It’s important to note that Amazon VPC CNI and the network policy controller currently don’t support in-place migration of third-party network policy plugins and custom NetworkPolicy objects. Therefore, please follow the procedures mentioned above to migrate to Amazon VPC CNI network policy.

Cleaning up

To avoid future costs, delete all the resources including Amazon EKS cluster created for this exercise. This action, in addition to deleting the cluster, will also delete the node group.

eksctl delete cluster -f cluster.yaml

Conclusion

In this post, we showed you that Kubernetes Network Policy support is now available in Amazon EKS, which addressed one of the most requested features on our roadmap. In this post, we have demonstrated how you can enforce fine-grained control over communication, isolate workloads, and enhance the overall security of your AWS Kubernetes clusters—all without the need to manage third-party network policy plugins.

As you dive into the realm of network policies, Amazon EKS recommends keeping in mind the specific security requirements of your applications and regularly review and update your policies to stay ahead of emerging threats. With careful planning and implementation, you can harness the full potential of Kubernetes network policies to protect your applications and data from unauthorized access. Please visit the Amazon EKS best practices guide for recommendations and additional considerations.

For installation instructions for Amazon VPC CNI, please refer to the Amazon EKS user guide. You may provide feedback on the Amazon VPC CNI plugin by leaving a comment or opening an issue on the AWS Containers Roadmap that is hosted on GitHub. Stay tuned as we continue to evolve our features and explore additional ways to leverage eBPF technology to deliver monitoring and observability capabilities within Amazon VPC CNI.