Managing Pod Security on Amazon EKS with Kyverno
This blog post was co-written by Jim Bugwadia, Co-founder and CEO, Nirmata.
This post is a follow-up to our previous blog, Implementing Pod Security Standards in Amazon EKS.
Pods are the fundamental unit of execution in Kubernetes. In this post, we’ll look at how Kyverno can be used to manage pod security for Amazon Elastic Kubernetes Service (Amazon EKS) with the new Kubernetes Pod Security Admission and Pod Security Standards features.
Pod security admission
The Kubernetes Pod Security Admission (PSA) feature went Beta in Kubernetes 1.23 and stable in 1.25. It implements a set of security controls defined in the Pod Security Standards (PSS). PSS defines 17 security controls for pod configurations organized into three levels: Privileged (unsecure), Baseline (secure), and Restricted (highly secure). PSA is a Kubernetes feature that provides an implementation of the PSS. The implementation consists of an admission controller that validates admission of pods against one of the three security levels, for each namespace, which is usually based on a static configuration file and namespace labels. The PSA uses three modes of operation:
- enforce: Policy violations will cause the pod to be rejected.
- audit: Policy violations trigger the addition of an audit annotation to the event recorded in the audit log, but are otherwise allowed.
- warn: Policy violations will trigger a user-facing warning, but are otherwise allowed.
The Kubernetes API server static configuration provides the ability to define additional PSA/PSS settings, such as exemptions based on usernames, runtime classes, and namespaces. Available (Beta) as of Amazon EKS 1.23, the PSA/PSS implementation doesn’t include the ability to customize the static configuration of PSA/PSS. This is a conscious decision to reduce Kubernetes API sever complexity and stay with default settings.
Note: In order to ensure operational stability and maintain our SLA, Amazon EKS doesn’t allow custom configuration of API server settings. Amazon EKS evaluates requests for custom configurations on a case-by-case basis, and may allow a setting to be configured if there is enough customer demand. As an example, the Amazon EKS API supports OpenID Connect (OIDC) user authentication, which is then passed as API server flags when a cluster is started. Customers can use the Containers Roadmap to open and monitor issues, and add their influence to existing issues.
The default configuration used by Amazon EKS 1.23 and beyond enables the Privileged PSS profile, utilizing the latest version of PSA modes, for all pods in a cluster, with no configured exemptions.
With this model, Kubernetes Namespaces in Amazon EKS 1.23 and beyond can opt in to more restrictive pod security settings, using the PSA/PSS labels.
While PSA is built in and enabled by default in Amazon EKS 1.23 and beyond, it’s also configured to use the Privileged PSS level by default. This means that there is no security enforced by default, unless namespace labels are used to opt in to more secure settings; pods can be executed with root privileges by default.
Another common challenge with PSA is that the configured security levels apply to all pods within a namespace. The rationale for this is that all pods in a namespace can access all service accounts and other sensitive resources in the namespace. However, using granular namespaces isn’t always practical and there is often a need to provide a policy exception to a single pod or container within a namespace, while restricting access for other pods or containers in the same namespace. Using Kyverno, users can enforce additional behaviors that complement the PSA/PSS settings, increase pod security, and provide a better user experience for Amazon EKS.
Kyverno, a Cloud Native Computing Foundation (CNCF) incubating project, is a Policy-as-Code (PaC) solution that includes a policy engine designed for Kubernetes. The Kyverno policy engine is installed into Kubernetes clusters and integrated to the Kubernetes API server as Dynamic Admission Controllers. This allows Kyverno policies to be used to mutate and validate inbound Kubernetes API server requests before the requests persist changes into the internal Kubernetes etcd data store.
Note: Kubernetes Dynamic Admission Controllers use webhook integration to call validating and mutating services. These calls have timeout settings (default 10 seconds) for how long the call will wait before a timeout occurs. The webhook configurations also include a failurePolicy setting to configure how the API server should respond when the webhook call doesn’t return within the configured timeout period. The webhook will either fail open (where the API server request is allowed to proceed) or fail closed (where the API server request is blocked). There are tradeoffs for each failure scenario. While a fail-open scenario could be seen as a potential security issue, a fail-closed scenario could cause operational issues for the cluster. For additional information about these scenarios and how they are configured in Kyverno, please see the Kyverno Security vs. Operability documentation.
Kyverno policies are declarative Kubernetes resources, written in YAML, with no new policy language to learn. Policy results are also available as Kubernetes resources and as events. Kyverno policies can be used to validate, mutate, and generate resource configurations, and validate image signatures and attestations.
To exercise the new pod security functionality provided by Kyverno, we must install the Kyverno policy engine and associated configurations into an existing Amazon EKS cluster. You can install the latest version of Kyverno using this Helm command:
Since Kyverno is built for Kubernetes, it can be used to complement and enhance the Kubernetes PSA feature. In the following sections of this post, we’ll discuss ways you can use Kyverno to apply PSS settings to your Amazon EKS clusters.
Enforcing pod security labels
As mentioned earlier, with the default PSA/PSS settings in Amazon EKS, namespaces must opt in to more restrictive pod security with PSA/PSS labels. You can use Kyverno to help manage pod security by enforcing the use of these pod security labels. The following Kyverno policy mutates inbound Kubernetes API server requests that create namespaces, and adds the pod security enforce mode label, if one is not already defined.
The result of this mutation is a namespace correctly labeled to the use the PSA enforce mode with the PSS Restricted level. All pods running in namespaces mutated by this policy will have the tightest pod security available from the PSA/PSS settings. To update existing namespaces, you can refer to this sample mutation policy to add privileged labels to existing namespaces.
After mutation, it’s still a good zero-trust practice to validate inbound Kubernetes API server requests, ensuring that the desired settings are present in the request payload, and preventing requests from making unwanted cluster changes. To that end, the following Kyverno validate policy rule can restrict the value of the PSA enforce mode to the PSS Baseline or Restricted levels.
With these policies in place, PSS is enforced on all new namespaces at either the Baseline or Restricted levels, providing an improved security posture over the default PSA settings.
Managing pod security levels and exceptions
In Kyverno release 1.8, a new validation rule type
validate.podSecurity was introduced. This declaration directly uses the in-tree Kubernetes PSS implementation while allowing more flexibility. Here are some of the major benefits:
- Cluster-wide pod security defaults do not require updating the static configuration file
- Namespace level pod security application does not require labels
- Pod security checks are automatically applied to pod controllers (e.g., deployments)
- Granular exemptions can be configured at a container image level
- Violations are reported in an in-cluster report
- Static validation and testing can be performed via the Kyverno command line interface (CLI) without requiring a cluster
The following example cluster policy applies the Restricted PSS level to all namespaces except for the kube-system namespace. Within each namespace, except kube-system, all pods have the same pod security settings, which is set to PSS Restricted level. This validation policy is used as part of a Defense-in-Depth (DiD) strategy that augments the prior namespace mutation and validation policies.
Existing namespace labels are ignored
An important point to remember about the new Kyverno
validate.podSecurity rule is that it neither requires nor considers existing PSA/PSS namespace labels. For example, given the namespace configuration below, that already includes PSA/PSS labels for the Privileged PSS level, the above Kyverno policy prevents a pod from being created, if a combination of the securityContext elements at the pod and container levels violate the Restricted PSS controls.
In this scenario, the Kyverno policy using the
validate.podSecurity rule overrides the PSA/PSS namespace labels.
Additional granularity with pod specific exemptions
As mentioned and demonstrated above, the namespace opt-in model to PSA/PSS pod security can be specified at the namespace level, with each pod in its respective namespace getting the same applied pod security. However, in the case where you need different PSA/PSS settings between pods in the same namespace, Kyverno policies can be used to provide that granularity.
Applying pod-specific exemptions to namespace PSA/PSS settings provides sub-namespace granularity. The following Kyverno policy applies the Restricted PSS level and configures an exemption for pods using the nginx:1.23.1-alpine image in the exempt namespace. All other pods in the same namespace, or in any other namespace, are subjected to the Restricted PSS level.
Rules processing – Controlling how many rules are applied
In the above policy there are two defined rules: exempt-run-as-non-root and restricted. The use of the
applyRules: One directive in the above policy instructs Kyverno to stop processing additional policy rules after the first matching rule is applied. Otherwise, the default behavior is to process all rules declared in a policy. In this example, this directive is used to create an exception that applies to a single workload using the nginx:1.23.1-alpine image in the exempt namespace, while providing a default rule that applies to all other pods, regardless of namespace.
With the 2-rule policy above, the flow (depicted below) is achieved.
- A request is submitted to the Kubernetes API server to create a pod
- If the namespace in the request is not the exempt namespace, then the Restricted PSS security profile, with the enforce PSA mode, is applied to the pod
- If the namespace in the request is the exempt namespace then check the container image in the pod
- If the container image in the pod is nginx:1.23.1-alpine, don’t apply a PSA/PSS combination
PSA operates at the pod level, but in practice pods are usually managed by pod controllers, like Deployments. Having no indication of pod security errors at the pod controller level makes issues complex to troubleshoot. The PSA enforce mode is the only PSA mode that stops pods from being created; however, PSA enforce doesn’t act on controller resources that create pods. To help this user experience, we recommend that PSA warn and audit modes are also used with enforce; PSA warn and audit modes will at least indicate that pod-creating controller resources are trying to create pods that would otherwise fail the applied PSS level. This user experience can present a challenge to adopters of the PSA/PSS features.
Using PaC solutions with Kubernetes presents another challenge: writing and maintaining policies to cover all the different Kubernetes resources used within clusters. With the Kyverno Auto-Gen Rules for Pod Controllers feature, the above pod policies auto-generate associated pod controller (Deployment, DaemonSet, etc.) policies. This Kyverno feature increases the expressive nature of policies, and reduces the effort to maintain policies for associated resources. These additional auto-generated policies also improve the above-mentioned PSA user experience where pod-creating resources are not prevented from progressing, while the underlying pods are prevented.
As seen below, the auto-generated controller policies are indicated in the status field of the original pod policy.
Kyverno includes policy reporting, using the open format defined by the Kubernetes Policy Working Group. Kyverno emits these policy reports when admission actions (CREATE, UPDATE, DELETE) are performed. These reports are also generated as a result of background scans that apply policies on existing resources. The reports can be found using the following kubectl command:
In the output above, the psa cluster policy indicates two passes. The entire report can be seen below.
Kyverno CLI shifts policy evaluation to the left
It’s not always optimal to test policy evaluation in Kubernetes clusters. It’s cheaper (time and money) to catch potential policy issues (violations, errors, etc.) upstream of Kubernetes in an automated DevOps pipeline. To shift policy testing to the left (for less cost and overhead), before we touch our clusters, we can use the Kyverno command-line interface (CLI) to apply policies to Kubernetes resource YAML files.
There are multiple ways to install the Kyverno CLI. For example, it can be installed as a kubectl plugin, using krew:
Once the Kyverno CLI is installed, the following command can be used to apply a Kyverno policy to a Kubernetes resource YAML file (no Kubernetes cluster needed):
The above command evaluated a Kubernetes Deployment resource YAML file against an example pod policy, which was mentioned earlier in this post. Even in the Kyverno CLI, the auto-generate feature created the Deployment policy, based on the pod policy, inline to the policy evaluation.
To remove Kyverno from your cluster, you can follow the uninstall instructions. The helm uninstall command is seen below:
As mentioned in the Kyverno documentation, Kyverno will try to remove all its webhook configurations. This can be done manually or as a final step with the following kubectl command:
In this post, we showed you how to augment the Kubernetes PSA/PSS configurations with Kyverno. Pod Security Standards (PSS) and the in-tree Kubernetes implementation of these standards, Pod Security Admission (PSA), provide good building blocks for managing pod security. The majority of users switching from Kubernetes Pod Security Policies (PSP) should be successful using the PSA/PSS features.
Kyverno augments the user experience created by PSA/PSS, by leveraging the in-tree Kubernetes pod security implementation, and providing several helpful enhancements for operationalization. You can use Kyverno to govern the proper use of pod security labels. In addition, you can use the new Kyverno
validate.podSecurity rule to easily manage pod security standards with additional flexibility and an enhanced user experience. And, with the Kyverno CLI, you can automate policy evaluation, upstream of your clusters.
Learn more about Kyverno at https://kyverno.io.