Open Source Calico for Windows Containers on Amazon EKS
In this post, we are going to walk through a tutorial on how to install and use Calico for Windows containers running on Amazon Elastic Kubernetes Service (EKS).
First, it is important for you to know that open source Calico for Windows is a networking and network security solution for Kubernetes-based Windows workloads. You can move Windows workloads like .NET applications into an EKS environment and Calico can help you manage network policy enforcement. Kubernetes network policies allow you to define rules for how groups of pods are allowed to communicate with each other and other network endpoints.
Calico has been validated in an Amazon EKS Linux environment; the process is documented here. Today, we dive deep into support for updated open source Calico on EKS Windows worker nodes. Calico on Windows nodes runs as a Windows service in contrast to DaemonSet on Linux nodes.
Calico for Windows is generally available with enterprise-grade support on Tigera’s Essentials subscription service. Please contact Tigera support for more details.
The steps below will help you enable Calico for Windows by leveraging the Amazon VPC CNI plugin, thus eliminating the need for installing any custom CNI plugins.
- Install and configure the AWS CLI, kubectl, and eksctl on the machine you will use to create and manage the EKS cluster.
- Set up the cluster
- Apply Calico resources on the EKS control plane and Linux worker node
- Install Calico on the Windows node
- Run a demo to enforce network policy.
Step 1: Set up the cluster
1.1 Create manifest file windows-manifest.yaml
--- apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: calicov2a region: us-west-2 version: '1.18' managedNodeGroups: - name: linux-ng desiredCapacity: 1 ssh: allow: true publicKeyName: oregon1 nodeGroups: - name: windows-ng instanceType: m4.4xlarge desiredCapacity: 1 minSize: 1 volumeSize: 100 ssh: allow: true publicKeyName: oregon1 amiFamily: WindowsServer2019FullContainer preBootstrapCommands: - '& mkdir c:\k' - '& Invoke-WebRequest https://amazon-eks.s3.us-west-2.amazonaws.com/1.18.8/2020-09-18/bin/darwin/amd64/kubectl -OutFile c:\k\kubectl.exe' - '& Set-ItemProperty -Path "Registry::HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment" -Name PATH -Value "$ENV:PATH ;C:\k"' - '& Invoke-WebRequest https://docs.projectcalico.org/scripts/install-calico-windows.ps1 -OutFile c:\install-calico-windows.ps1' cloudWatch: clusterLogging: enableTypes: ["*"]
Note: We have chosen to include several pre-bootstrap commands to simplify the installation process. These commands are specific to our Calico installation on Kubernetes 1.18. For general guidance on deploying a windows node group as part of your cluster deployment, please refer to the Amazon EKS documentation.
1.2 create a cluster using eksctl
eksctl create cluster -f ./windows-manifest.yaml --install-vpc-controllers
Step 2: Apply Calico resources on the EKS control plane and Linux worker node
Using Calico for Windows requires resources like the Calico DaemonSet, some custom resource definitions and appropriate ClusterRoles and RoleBindings.
2.1 Using kubectl, create the resources like below
kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.7/config/v1.7/calico.yaml
2.2 Verify that the Calico DaemonSet is running
kubectl get daemonset calico-node --namespace kube-system
Note: This also enables network policy support for Linux workloads. Try it out with the Calico support for any Linux workloads like Stars Policy Demo.
We will need to add a role to the cluster to allow the windows nodes to function as well.
2.3 Create the user-rolebinding.yaml file
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: nodes-cluster-admin roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:nodes
2.4 Apply the user role binding to the cluster
kubectl apply -f user-rolebinding.yaml
2.5 Create the calicoctl pod
Calicoctl is the CLI to manage Calico resources
kubectl apply -f https://docs.projectcalico.org/archive/v3.15/manifests/calicoctl.yaml alias calicoctl="kubectl exec -i -n kube-system calicoctl -- /calicoctl"
Step 3: Install Calico on the Windows node
3.1 Access the windows node with RDP
Follow the AWS published instructions for connecting to your Windows node.
3.2 Run install script
From Powershell, run install-calico-windows.ps1 with the following parameters.
PS C:\> ./install-calico-windows.ps1 -ServiceCidr 10.100.0.0/16 -DNSServerIPs 10.100.0.10
Once the install is complete we should check that the Calico services installed correctly and that the Kubernetes services are still running.
3.3 Verify the install
The following command checks that CalicoNode, CalicoFelix, kubelet, and kube-proxy are running.
PS C:\> Get-Service -Name Calico*, kube*
We should get the following output:
Status Name DisplayName ------ ---- ------------ Running CalicoNode Calico Windows Startup Running CalicoFelix Calico Windows Agent Running kubelet kubelet service Running kube-proxy kube-proxy service
You can also verify that the Calico services were installed properly by inspecting logs located at c:\CalicoWindows\calico-node.err.log and c:\CalicoWindows\calico-felix.err.log. Ideally, these should be empty with no errors.
Step 4: Run a demo to enforce network policy
4.1 Launch Windows and Linux deployments.
Use the pod specification below to launch Windows and Linux pods on the respective worker nodes.
--- apiVersion: apps/v1 kind: Deployment metadata: name: windows-server-iis spec: selector: matchLabels: app: windows-server-iis replicas: 1 template: metadata: labels: app: windows-server-iis spec: containers: - name: windows-server-iis image: mcr.microsoft.com/windows/servercore:1809 imagePullPolicy: IfNotPresent command: - powershell.exe - -command - "Add-WindowsFeature Web-Server; Invoke-WebRequest -UseBasicParsing -Uri 'https://dotnetbinaries.blob.core.windows.net/servicemonitor/22.214.171.124/ServiceMonitor.exe' -OutFile 'C:\\ServiceMonitor.exe'; echo '<html><body><br/><br/><marquee><H1>Hello EKS!!!<H1><marquee></body><html>' > C:\\inetpub\\wwwroot\\default.html; C:\\ServiceMonitor.exe 'w3svc'; " nodeSelector: kubernetes.io/os: windows --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx replicas: 1 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 securityContext: privileged: true nodeSelector: beta.kubernetes.io/os: linux
You can name the above spec file as sample-deployments.yaml and apply the changes to our kubernetes cluster.
kubectl apply -f sample-deployments.yaml
Verify that the pods are in ‘Running’ state.
kubectl get pods -o wide --watch
4.2 Test ping connectivity between pods
We would start by verifying that there is network connectivity among all pods. Inside each pod, ping the other two pods’ IP. Update the <nginx-pod-name> below with the pod name that you see in the cluster.
kubectl exec -it <nginx-pod-name> /bin/bash
Similarly, ‘exec’ into one of the Windows pods as well. Update the <windows-pod-name> below with the pod names that you see in the cluster.
kubectl exec -it <windows-pod-name> powershell
4.3 Enforce a network policy to restrict ping connectivity.
Apply the network policy specification to deny ping traffic to all pods.
--- apiVersion: crd.projectcalico.org/v31 kind: GlobalNetworkPolicy metadata: name: block-icmp spec: order: 200 selector: all() types: - Ingress - Egress ingress: - action: Deny protocol: ICMP - action: Deny protocol: ICMPv6 egress: - action: Deny protocol: ICMP - action: Deny protocol: ICMPv6
You can name the above spec file as deny-icmp.yaml and apply the changes to our kubernetes cluster.
calicoctl apply -f - < deny_icmp.yaml
4.4 Test ping connectivity between pods.
Follow step 4.2 and check ping connectivity once more. This time you will see that traffic is denied among the pods.
4.5 Remove the network policy enforced.
Now that you’ve tested the network policy, delete it to test whether ping connectivity is established again.
kubectl delete -f deny-icmp.yaml
That is it! You have configured your Windows worker nodes with Open source Calico for Windows.
In this post, I showed you how to install and use Open source Calico for Windows worker nodes on Amazon EKS. If you have comments or questions about this post, submit them in the comments section below.