Open Source Calico for Windows Containers on Amazon EKS

In a previous post, we provided instructions for Using Calico on Amazon EKS Windows Containers. Since that publication, Tigera has announced that Calico for Windows is Open Source.

In this post, we are going to walk through a tutorial on how to install and use Calico for Windows containers running on Amazon Elastic Kubernetes Service (EKS).

Background

First, it is important for you to know that open source Calico for Windows is a networking and network security solution for Kubernetes-based Windows workloads. You can move Windows workloads like .NET applications into an EKS environment and Calico can help you manage network policy enforcement. Kubernetes network policies allow you to define rules for how groups of pods are allowed to communicate with each other and other network endpoints.

Calico has been validated in an Amazon EKS Linux environment; the process is documented here. Today, we dive deep into support for updated open source Calico on EKS Windows worker nodes. Calico on Windows nodes runs as a Windows service in contrast to DaemonSet on Linux nodes.

Calico for Windows is generally available with enterprise-grade support on Tigera’s Essentials subscription service. Please contact Tigera support for more details.

The steps below will help you enable Calico for Windows by leveraging the Amazon VPC CNI plugin, thus eliminating the need for installing any custom CNI plugins.

Prerequisites:

Install and configure the AWS CLI, kubectl, and eksctl on the machine you will use to create and manage the EKS cluster.

Solution overview:

Set up the cluster
Apply Calico resources on the EKS control plane and Linux worker node
Install Calico on the Windows node
Run a demo to enforce network policy.

Step 1: Set up the cluster

1.1 Create manifest file windows-manifest.yaml

---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: calicov2a
  region: us-west-2
  version: '1.18'

managedNodeGroups:
  - name: linux-ng
    desiredCapacity: 1
    ssh:
      allow: true
      publicKeyName: oregon1
   
nodeGroups:
  - name: windows-ng
    instanceType: m4.4xlarge
    desiredCapacity: 1
    minSize: 1
    volumeSize: 100
    ssh:
      allow: true
      publicKeyName: oregon1
    amiFamily: WindowsServer2019FullContainer
    preBootstrapCommands:
    - '& mkdir c:\k'
    - '& Invoke-WebRequest https://amazon-eks.s3.us-west-2.amazonaws.com/1.18.8/2020-09-18/bin/darwin/amd64/kubectl -OutFile c:\k\kubectl.exe'
    - '& Set-ItemProperty -Path "Registry::HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment" -Name PATH -Value "$ENV:PATH ;C:\k"'
    - '& Invoke-WebRequest https://docs.projectcalico.org/scripts/install-calico-windows.ps1 -OutFile c:\install-calico-windows.ps1'
cloudWatch:
  clusterLogging:
    enableTypes: ["*"]

Note: We have chosen to include several pre-bootstrap commands to simplify the installation process. These commands are specific to our Calico installation on Kubernetes 1.18. For general guidance on deploying a windows node group as part of your cluster deployment, please refer to the Amazon EKS documentation.

1.2 create a cluster using eksctl

eksctl create cluster -f ./windows-manifest.yaml --install-vpc-controllers

Step 2: Apply Calico resources on the EKS control plane and Linux worker node

Using Calico for Windows requires resources like the Calico DaemonSet, some custom resource definitions and appropriate ClusterRoles and RoleBindings.

2.1 Using kubectl, create the resources like below

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.7/config/v1.7/calico.yaml

2.2 Verify that the Calico DaemonSet is running

kubectl get daemonset calico-node --namespace kube-system

Note: This also enables network policy support for Linux workloads. Try it out with the Calico support for any Linux workloads like Stars Policy Demo.

We will need to add a role to the cluster to allow the windows nodes to function as well.

2.3 Create the user-rolebinding.yaml file

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: nodes-cluster-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes

2.4 Apply the user role binding to the cluster

kubectl apply -f user-rolebinding.yaml

2.5 Create the calicoctl pod

Calicoctl is the CLI to manage Calico resources

kubectl apply -f https://docs.projectcalico.org/archive/v3.15/manifests/calicoctl.yaml alias calicoctl="kubectl exec -i -n kube-system calicoctl -- /calicoctl"

Step 3: Install Calico on the Windows node

3.1 Access the windows node with RDP

Follow the AWS published instructions for connecting to your Windows node.

3.2 Run install script

From Powershell, run install-calico-windows.ps1 with the following parameters.

PS C:\> ./install-calico-windows.ps1 -ServiceCidr 10.100.0.0/16 -DNSServerIPs 10.100.0.10

Once the install is complete we should check that the Calico services installed correctly and that the Kubernetes services are still running.

3.3 Verify the install

The following command checks that CalicoNode, CalicoFelix, kubelet, and kube-proxy are running.

PS C:\> Get-Service -Name Calico*, kube*

We should get the following output:

Status   Name            DisplayName
------   ----            ------------
Running  CalicoNode      Calico Windows Startup
Running  CalicoFelix     Calico Windows Agent
Running  kubelet         kubelet service
Running  kube-proxy      kube-proxy service

You can also verify that the Calico services were installed properly by inspecting logs located at c:\CalicoWindows\calico-node.err.log and c:\CalicoWindows\calico-felix.err.log. Ideally, these should be empty with no errors.

Step 4: Run a demo to enforce network policy

4.1 Launch Windows and Linux deployments.

Use the pod specification below to launch Windows and Linux pods on the respective worker nodes.

---
apiVersion: apps/v1
kind: Deployment
metadata:
    name: windows-server-iis
spec:
    selector:
        matchLabels:
            app: windows-server-iis
    replicas: 1
    template:
        metadata:
            labels:
                app: windows-server-iis
        spec:
            containers:
            - name: windows-server-iis
              image: mcr.microsoft.com/windows/servercore:1809
              imagePullPolicy: IfNotPresent
              command:
              - powershell.exe
              - -command
              - "Add-WindowsFeature Web-Server; Invoke-WebRequest -UseBasicParsing -Uri 'https://dotnetbinaries.blob.core.windows.net/servicemonitor/2.0.1.6/ServiceMonitor.exe' -OutFile 'C:\\ServiceMonitor.exe'; echo '<html><body><br/><br/><marquee><H1>Hello EKS!!!<H1><marquee></body><html>' > C:\\inetpub\\wwwroot\\default.html; C:\\ServiceMonitor.exe 'w3svc'; "
            nodeSelector:
              kubernetes.io/os: windows
            
---
apiVersion: apps/v1
kind: Deployment
metadata:
    name: nginx-deployment
spec:
    selector:
        matchLabels:
            app: nginx
    replicas: 1
    template:
        metadata:
            labels:
                app: nginx
        spec:
            containers:
            - name: nginx
              image: nginx:1.7.9
              securityContext:
                privileged: true
            nodeSelector:
                beta.kubernetes.io/os: linux

You can name the above spec file as sample-deployments.yaml and apply the changes to our kubernetes cluster.

kubectl apply -f sample-deployments.yaml

Verify that the pods are in ‘Running’ state.

kubectl get pods -o wide --watch

Get Calico Pods

4.2 Test ping connectivity between pods

We would start by verifying that there is network connectivity among all pods. Inside each pod, ping the other two pods’ IP. Update the <nginx-pod-name> below with the pod name that you see in the cluster.

kubectl exec -it <nginx-pod-name> /bin/bash

Calico Nginx Ping

Similarly, ‘exec’ into one of the Windows pods as well. Update the <windows-pod-name> below with the pod names that you see in the cluster.

kubectl exec -it <windows-pod-name> powershell

Calcio windows ping

4.3 Enforce a network policy to restrict ping connectivity.

Apply the network policy specification to deny ping traffic to all pods.

---
apiVersion: crd.projectcalico.org/v31
kind: GlobalNetworkPolicy
metadata:
    name: block-icmp
spec:
  order: 200
  selector: all()
  types:
  - Ingress
  - Egress
  ingress:
  - action: Deny
    protocol: ICMP
  - action: Deny
    protocol: ICMPv6
  egress:
  - action: Deny
    protocol: ICMP
  - action: Deny
    protocol: ICMPv6

You can name the above spec file as deny-icmp.yaml and apply the changes to our kubernetes cluster.

calicoctl apply -f - < deny_icmp.yaml

4.4 Test ping connectivity between pods.

Follow step 4.2 and check ping connectivity once more. This time you will see that traffic is denied among the pods.

Calico Nginx Ping Fail

Windows Ping Fail

4.5 Remove the network policy enforced.

Now that you’ve tested the network policy, delete it to test whether ping connectivity is established again.

kubectl delete -f deny-icmp.yaml

Conclusion

That is it! You have configured your Windows worker nodes with Open source Calico for Windows.
In this post, I showed you how to install and use Open source Calico for Windows worker nodes on Amazon EKS. If you have comments or questions about this post, submit them in the comments section below.

Containers