AWS Open Source Blog
Managing secrets deployment in Kubernetes using Sealed Secrets
Kubernetes is an open source system for automating the deployment, scaling, and management of containerized applications. It is especially suitable for building and deploying cloud-native applications on a massive scale, leveraging the elasticity of the cloud. Amazon Elastic Kubernetes Service (Amazon EKS) is a managed service for running a production-grade, highly available Kubernetes cluster on AWS without needing to stand up or to maintain the Kubernetes control plane. Amazon EKS is used today by thousands of customers to run containerized applications at scale.
Regardless of whether customers are spinning up a Kubernetes cluster using open source tools such as Kops and managing it on their own or employing a managed service such as Amazon EKS, it is a common practice for a DevOps team to use a continuous delivery pipeline to deploy various resources to their Kubernetes cluster. The YAML manifests pertaining to these cluster resources may be stored and version-controlled in a Git repository.
In this post, we will walk through the details of using tools from the Sealed Secrets open source project that will allow users to manage the deployment of sensitive information to their Kubernetes clusters, and to store them securely in a Git repository and to integrate them into their continuous delivery pipelines.
Continuous delivery using GitOps
GitOps is a term coined by WeaveWorks and is a way to do Kubernetes cluster management and continuous delivery. In this approach, a Git repository is designated as the single source of truth for deployment artifacts, such as YAML files, that provide a declarative way to describe the cluster state. As illustrated in the architecture below, a Weave Flux agent runs in the Kubernetes cluster and watches the Git repository and image registries, such as Amazon Elastic Container Registry (Amazon ECR) and Docker Hub, where the container images pertaining to application workloads reside. If changes to deployment artifacts are pushed to this config repository or a new image is pushed to the image registry by a continuous integration system such as Jenkins, the Weave Flux agent responds by pulling these changes down and updating the relevant applications workloads deployed to the cluster.
The challenge with handling Secrets
A Kubernetes cluster has different types of resources such as Deployments, DaemonSets, ConfigMaps, Secrets, etc. A Secret is a resource that helps cluster operators manage the deployment of sensitive information, such as passwords, OAuth tokens, and SSH keys. These Secrets can be mounted as data volumes or exposed as environment variables to the containers in a Kubernetes Pod, thus decoupling Pod deployment from managing sensitive data needed by the containerized applications within a Pod.
The challenge here is with integrating these Secrets into the GitOps workflow by storing the relevant YAML manifests outside the cluster, in a Git repository. The data in a Secret is obfuscated by using merely Base64 encoding. Storing such files in a Git repository is extremely insecure as it is trivial to decode the Base64-encoded data. Often developers accidentally check these files into their Git repositories, thus exposing sensitive information—such as credentials—to their production databases.
Sealed Secrets open source project addresses this challenge by providing a mechanism to encrypt a Secret object so that it is safe to store in a private or public repository. These encrypted Secrets can also be deployed to a Kubernetes cluster using normal workflows with tools such as kubectl.
How it works
Sealed Secrets comprises the following components:
- A controller deployed to cluster
- A CLI tool called kubeseal
- A customer resource definition (CRD) called SealedSecret
Upon startup, the controller looks for a cluster-wide private/public key pair, and generates a new 4096-bit RSA key pair if not found. The private key is persisted in a Secret object in the same namespace as that of the controller. The public key portion of this is made publicly available to anyone wanting to use Sealed Secrets with this cluster.
During encryption, each value in the original Secret is symmetrically encrypted using AES-256 with a randomly generated session key. The session key is then asymmetrically encrypted with the controller’s public key using SHA256 and the original Secret’s namespace/name as the input parameter. The output of the encryption process is a string that is constructed as: length (2 bytes) of encrypted session key + encrypted session key + encrypted Secret.
When a SealedSecret custom resource is deployed to the Kubernetes cluster, the controller will pick it up, unseal it using the private key, and create a Secret resource. During decryption, the SealedSecret’s namespace/name is used again as the input parameter. This ensures that the SealedSecret and Secret are strictly tied to the same namespace and name.
The companion CLI tool kubeseal is used for creating a SealedSecret custom resource definition (CRD) from a Secret resource definition using the public key. kubeseal can communicate with the controller through the Kubernetes API server and retrieve the public key needed for encrypting a Secret at runtime. The public key may also be downloaded from the controller and saved locally to be used offline.
Installing the kubeseal client
For Linux x86_64 systems, the client-tool may be installed into /usr/local/bin with the following command:
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.10.0/kubeseal-linux-amd64 -O kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal
For macOS systems, the client tool is installed as follows:
brew install kubeseal
Installing the custom controller and CRD for SealedSecret
See the Git repository for Sealed Secrets project for recent releases and detailed installation instructions. The latest release at the time of writing this post was v0.12.1 and can be installed on a Kubernetes cluster in a single step using kubectl as shown by the following command. This will install the controller and the SealedSecret custom resource definition in the kube-system namespace as well as create a Service Account and RBAC artifacts such as Role/ClusterRole and RoleBinding/ClusterRoleBinding.
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.1/controller.yaml
kubectl apply -f controller.yaml
The logs from the controller reveal the name of the Secret that is created in the kube-system namespace and that contains the private key pair used by the controller for unsealing SealedSecrets deployed to the cluster.
kubectl logs sealed-secrets-controller-6b7dcdc847-9l8pz -n kube-systemOutput from the above command looks as follows:
2020/04/09 23:06:50 Starting sealed-secrets controller version: v0.12.1
2020/04/09 23:06:50 Searching for existing private keys
2020/04/09 23:06:53 New key written to kube-system/sealed-secrets-keyhvdtf
2020/04/09 23:06:53 Certificate is
-----BEGIN CERTIFICATE-----
MIIErTCCApWgAwIBAgIQILmhsVqF7t1YIRzdZp+injANBgkqhkiG9w0BAQsFADAA
MB4XDTIwMDQwOTIzMDY1M1oXDTMwMDQwNzIzMDY1M1owADCCAiIwDQYJKoZIhvcN
AQEBBQADggIPADCCAgoCggIBAM/LlngsOuuBtDD/9nbfjIoZ2kxgepEzzBQWmpr9
gkWpw2cqOggK3ZdEv+JYw/dxJIZ7E8G/PNiY7YzcR1JDbFxclmvzuzaGDLhUdCzQ
X34ZJKOcIRAAuTYZiW60laRNVm37r68/tpok48I8+YxCD7qVtJ3y6ddRKt8iPMn2
mOZLohjy7pbZpcRp5tdS0lU8ZavdcYaUwIjxQR8XUS00nj4PIjVyKOGszjVac5bv
D4HTAZcLMNpe2iECQziZ4S/HmlBTR7+4z8hF9fk+aAy9ZeCvAZMGYBY3Um+x3qqz
3cNdlA6IF9nk1aWMzXkyx9K9CE2QIblT4WQVyB7ZYxPTtlpau9XsNGjBI4DzrcHH
VdDkUQEpNYwkxqpNndJlWSS5ZnglgoOwYLob4XT4hoqoqyrO5gCUKlarofmdmYm4
ELUeGPfYKlKP6rKOxQUQ0FHQL6Bvwx/Xx5wr6r1ylrC+++NFO6lMrQ0GhvYvtxN5
Ut0iy8fvXXuBP5HI8eBTEj0M7pAEZ+o/o+91N41KwuRoSBIOPD6q0e+ULBQ87Hth
gpuskPmQe5lT4dv0TuSTjdJg4LQuApm+w0ta4FXX9PoXoojSaHy0EoQQvw4bAr8x
8ioz6ogmGSyFPVF/qnBnQpH5RqiWAIW1jUgt2SIluNi8Zqc4IjImIaI/ZRObElqJ
qXAXAgMBAAGjIzAhMA4GA1UdDwEB/wQEAwIAATAPBgNVHRMBAf8EBTADAQH/MA0G
CSqGSIb3DQEBCwUAA4ICAQBPyUeKHuk0D0XYRjiY9WQT0hEmH5ABP7Jypz9TnElz
asuIorEtaUEhtzKj1L+zEkyNzZzeKurDE9PL2xQB3HR675y0bDPS1ndpLDDIU7qt
38rvhb7iHbgWPti7EeKteBaNVvcaGfwhG8vnYoKKI8KzTy04rk2/0OFQNzpXOHra
Yw+mlho3MHtlT5AyTAuUqSXbRXLTMKOuXQlcGwdGAxiaV5i40AkfUICDhGZnMF8F
JNujDL0kpDF04qybMrJsOlHu1ClJWKcrxQh7v/ITfFdgw3N/lnA+OBXoXOs+VL+4
s6A20IddIBAaTd04ajmIZa7FEXSj+yMA2v7+gLE8POQlQANVNH2iJ6kgePlQB2+9
gdXeQIKwWyC9cCiN/yvCWz58fb2wS5lVE0SL7Jp9A42N6C+x+Hfy4jWTSB+SAQ6F
TKGWNHh9G1ghhFbq8cDShBsOr8MrBgxeWMlXG7o0yZlSbiC64FpZn2tYE3Elc8VX
kyhv+WxwggBsuk8qvzBK8sPRT+VmQ1w/GUdsJXw/GWaL2m3KZZyfTka3OwM1R7tJ
xgvkU74FVLk+SwcTVdCewfahSkmm43vQFr8u6EHqsYN2jY2h9ka8jTOzCof1OiPS
bwMtSuxW82ty6Pu3UyskQBRLmLOAuy/nmXZKjcdtB/OLELbUO44NVgxkIhWCLrCU
ew==
-----END CERTIFICATE-----Upon starting, the controller searches for a Secret with the label sealedsecrets.bitnami.com/sealed-secrets-key in its namespace. If it does not find one, it creates a new one in its namespace and prints the public key portion of the key pair to its output logs. The contents of this Secret, which contains the public/private key pair, can be seen in YAML format with the following command:
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml
apiVersion: v1
items:
- apiVersion: v1
data:
tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVyVENDQXBXZ0F3SUJBZ0lRSUxtaHNWcUY3dDFZSVJ6ZFpwK2luakFOQmdrcWhraUc5dzBCQVFzRkFEQUEKTUI0WERUSXdNRFF3T1RJek1EWTFNMW9YRFRNd01EUXdOekl6TURZMU0xb3dBRENDQWlJd0RRWUpLb1pJaHZjTgpBUUVCQlFBRGdnSVBBRENDQWdvQ2dnSUJBTS9MbG5nc091dUJ0REQvOW5iZmpJb1oya3hnZXBFenpCUVdtcHI5CmdrV3B3MmNxT2dnSzNaZEV2K0pZdy9keEpJWjdFOEcvUE5pWTdZemNSMUpEYkZ4Y2xtdnp1emFHRExoVWRDelEKWDM0WkpLT2NJUkFBdVRZWmlXNjBsYVJOVm0zN3I2OC90cG9rNDhJOCtZeENEN3FWdEozeTZkZFJLdDhpUE1uMgptT1pMb2hqeTdwYlpwY1JwNXRkUzBsVThaYXZkY1lhVXdJanhRUjhYVVMwMG5qNFBJalZ5S09Hc3pqVmFjNWJ2CkQ0SFRBWmNMTU5wZTJpRUNRemlaNFMvSG1sQlRSNys0ejhoRjlmaythQXk5WmVDdkFaTUdZQlkzVW0reDNxcXoKM2NOZGxBNklGOW5rMWFXTXpYa3l4OUs5Q0UyUUlibFQ0V1FWeUI3Wll4UFR0bHBhdTlYc05HakJJNER6cmNISApWZERrVVFFcE5Zd2t4cXBObmRKbFdTUzVabmdsZ29Pd1lMb2I0WFQ0aG9xb3F5ck81Z0NVS2xhcm9mbWRtWW00CkVMVWVHUGZZS2xLUDZyS094UVVRMEZIUUw2QnZ3eC9YeDV3cjZyMXlsckMrKytORk82bE1yUTBHaHZZdnR4TjUKVXQwaXk4ZnZYWHVCUDVISThlQlRFajBNN3BBRVorby9vKzkxTjQxS3d1Um9TQklPUEQ2cTBlK1VMQlE4N0h0aApncHVza1BtUWU1bFQ0ZHYwVHVTVGpkSmc0TFF1QXBtK3cwdGE0RlhYOVBvWG9valNhSHkwRW9RUXZ3NGJBcjh4Cjhpb3o2b2dtR1N5RlBWRi9xbkJuUXBINVJxaVdBSVcxalVndDJTSWx1Tmk4WnFjNElqSW1JYUkvWlJPYkVscUoKcVhBWEFnTUJBQUdqSXpBaE1BNEdBMVVkRHdFQi93UUVBd0lBQVRBUEJnTlZIUk1CQWY4RUJUQURBUUgvTUEwRwpDU3FHU0liM0RRRUJDd1VBQTRJQ0FRQlB5VWVLSHVrMEQwWFlSamlZOVdRVDBoRW1INUFCUDdKeXB6OVRuRWx6CmFzdUlvckV0YVVFaHR6S2oxTCt6RWt5TnpaemVLdXJERTlQTDJ4UUIzSFI2NzV5MGJEUFMxbmRwTERESVU3cXQKMzhydmhiN2lIYmdXUHRpN0VlS3RlQmFOVnZjYUdmd2hHOHZuWW9LS0k4S3pUeTA0cmsyLzBPRlFOenBYT0hyYQpZdyttbGhvM01IdGxUNUF5VEF1VXFTWGJSWExUTUtPdVhRbGNHd2RHQXhpYVY1aTQwQWtmVUlDRGhHWm5NRjhGCkpOdWpETDBrcERGMDRxeWJNckpzT2xIdTFDbEpXS2NyeFFoN3YvSVRmRmRndzNOL2xuQStPQlhvWE9zK1ZMKzQKczZBMjBJZGRJQkFhVGQwNGFqbUlaYTdGRVhTait5TUEydjcrZ0xFOFBPUWxRQU5WTkgyaUo2a2dlUGxRQjIrOQpnZFhlUUlLd1d5QzljQ2lOL3l2Q1d6NThmYjJ3UzVsVkUwU0w3SnA5QTQyTjZDK3grSGZ5NGpXVFNCK1NBUTZGClRLR1dOSGg5RzFnaGhGYnE4Y0RTaEJzT3I4TXJCZ3hlV01sWEc3bzB5WmxTYmlDNjRGcFpuMnRZRTNFbGM4VlgKa3loditXeHdnZ0JzdWs4cXZ6Qks4c1BSVCtWbVExdy9HVWRzSlh3L0dXYUwybTNLWlp5ZlRrYTNPd00xUjd0Sgp4Z3ZrVTc0RlZMaytTd2NUVmRDZXdmYWhTa21tNDN2UUZyOHU2RUhxc1lOMmpZMmg5a2E4alRPekNvZjFPaVBTCmJ3TXRTdXhXODJ0eTZQdTNVeXNrUUJSTG1MT0F1eS9ubVhaS2pjZHRCL09MRUxiVU80NE5WZ3hrSWhXQ0xyQ1UKZXc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlKS1FJQkFBS0NBZ0VBejh1V2VDdzY2NEcwTVAvMmR0K01paG5hVEdCNmtUUE1GQmFhbXYyQ1JhbkRaeW82CkNBcmRsMFMvNGxqRDkzRWtobnNUd2I4ODJKanRqTnhIVWtOc1hGeVdhL083Tm9ZTXVGUjBMTkJmZmhra281d2gKRUFDNU5obUpiclNWcEUxV2JmdXZyeisybWlUandqejVqRUlQdXBXMG5mTHAxMUVxM3lJOHlmYVk1a3VpR1BMdQpsdG1seEdubTExTFNWVHhscTkxeGhwVEFpUEZCSHhkUkxUU2VQZzhpTlhJbzRhek9OVnB6bHU4UGdkTUJsd3N3CjJsN2FJUUpET0puaEw4ZWFVRk5IdjdqUHlFWDErVDVvREwxbDRLOEJrd1pnRmpkU2I3SGVxclBkdzEyVURvZ1gKMmVUVnBZek5lVExIMHIwSVRaQWh1VlBoWkJYSUh0bGpFOU8yV2xxNzFldzBhTUVqZ1BPdHdjZFYwT1JSQVNrMQpqQ1RHcWsyZDBtVlpKTGxtZUNXQ2c3Qmd1aHZoZFBpR2lxaXJLczdtQUpRcVZxdWgrWjJaaWJnUXRSNFk5OWdxClVvL3FzbzdGQlJEUVVkQXZvRy9ESDlmSG5DdnF2WEtXc0w3NzQwVTdxVXl0RFFhRzlpKzNFM2xTM1NMTHgrOWQKZTRFL2tjang0Rk1TUFF6dWtBUm42aitqNzNVM2pVckM1R2hJRWc0OFBxclI3NVFzRkR6c2UyR0NtNnlRK1pCNwptVlBoMi9STzVKT04wbURndEM0Q21iN0RTMXJnVmRmMCtoZWlpTkpvZkxRU2hCQy9EaHNDdnpIeUtqUHFpQ1laCkxJVTlVWCtxY0dkQ2tmbEdxSllBaGJXTlNDM1pJaVc0Mkx4bXB6Z2lNaVlob2o5bEU1c1NXb21wY0JjQ0F3RUEKQVFLQ0FnQlVraWFHZUhIdWdkYUZqdGVQb0FKQi9xMmpJaTBnUmJXTWczcWZGQWhlTSs2c1lUcEhKYXowTU8zcgp3SGJaa1hudEpkQnZyVmFsVFBCNXdQbGlHTURVZ25aU0wxdUZvRjh5OG1ScUROQ2dzTGtCd2J5UEY4eEpvWEVXCjFuYUU3Vmo4NEUrcmdzSGQwSi9GNFMwcmtZTjNUQkM3ckM3U0RGM25mTGJDK0JOWXYzV1VzK0s1RUpIdjg2NFkKK3NOU0g0ZTl3QjNCU1c5bkROR1ZSdGNxRDkxTG9yc29oM0x1RG5mS3JTcVlSbW5JUzhtODRMZ2NXRGhzOE0vTgpESXZpOTFqdDBrZEVWNEp4bjAreUJsMHd1akRwbGpDTTF5NXFQRS9YMTh1cExCVll1eEJVTGIyUFdCeEFDYU5pCjdYRDRheWtpOEVOWmV6TWptZDNkK0ZuanF1bnU5bHdpUHhKZTl2Zng5QUJqaUhPRjZmWFpKT2dPUzhESWFaazkKc0Fpb2xveCt1VklFQUl1L0x4bytFaHJyTnA0QWFGVUdJWlNyemk1UFRSZWE1ZHFkeVVuemt2TGk5T3JSejNOUQprazJjYlRYUmZtZEtCTVNkZ1czM0R4dk0vTndId0d4VUNEeDFJRUtHYTB5YnFtM3h1QnNqeXJrYVZRNmYveUhyCkI3OUJzS0YwanU1MFhtRUhRbkFsZklkc0RFRi9uaWdBeFFLM1ZEN3ZDRVdmeU9mZUJnNFo4bkVzanhxRGlOQkgKQ1NjT1hoSHVVWllZMUJ2UXJQSzF0dVJpOU1yR0N5dmxrN1JnaHBBVDVtck1OSFlVc2haRE9ISlhYRDh3WkcrdgptbllkZ2JjTU15cnJ1Yzk2TkFCN3A4Y1ViaWxySVlHNXN4NE1QYlpPOUlRaWVBQjVrUUtDQVFFQTd3elQvL09yCmJHMVR2Nkg1UDFlR0hhMXpUeG5GTzI5UDl0SVRmSWdtZjB2TkFJeGNEdExJamxrcTRQVVg4eGZMQWdWUDFFOWQKdEE0L0d4dzFqQWNRVDBNalZIS3JhWEtkRGh5WU8wSytlekcrc1MyZ0xmV1NxOGh5MGYxclNuNktMcjJhQ1U4Ywp2allVbmtCclI3Mk1tMXYvV1g5YjZyV1NjdGdLMkpZclVNWXVEUlcrbTl0L2dDdU95Y3JXYnRqQXoyMURzOHR5CnBWOWxIMlczRTM2bVR5UTJVZzZzRnJ2NGtLMXVTeGxNTWRFQU1odWhVc1lseFhQYmVBWWxxYk5tUmNlbjJPMlAKam5jZHY1elJRZ3ZRLzJXVjBBWGdnU3pkK3ZjbXRRUFJ5ZG5EeFNXY3hiNWl1MWNnbFVoN3k2elB6akNDa0NFQQo4RjdqME5YNFdOdlJEd0tDQVFFQTNvZHVKNFhCcFlqdkJTcmY4NEduRUI4T2U2VlVYOGErRUxkMmg4ZTB3NWM1Cm9rZ1BCR0oxZVpnRXQyWXJQVkJ1RExvcEphUWVLMVlpcHBabDdHUG5lWUVrSTE3OWR1WnVmWE41YTZQelRTTncKeHZTTTZ0SGFKUnJLZno5NzFGS2M4dkVUQjVadjJUZFEzWFUvR1NrcWdKMHcreXd3MHZLUVpnc2wxK2YyY0s0TQpYcktwdnU1Y2hIRkhsWDFoYmZ2N2xzanN4UkNHSXl2NkpmbkpETy9JNDgwa21SQ3hUdHJubStuZVlxZjBEcXovCkVsUC9IQVlwUUpYMmJkbEJiZFRGaDF3VGFJQUJxSXFVdmZYR0s5Z3J6bXRPNDh5dHcxVDdnS0ZTRUdiR0JiVlcKbGN5WFZIVWc4R0FYVDJYUUt4SnFmK2V4VXh2WWh0UmpKaWwzY3NKZ2VRS0NBUUVBeGxqVWh1azk4WWQ5RitKagpsMFVlQ1Aza1VWdkdwUndsTTF6M3dqcU9CczEwV2VJY2VFZzVGTE96dWxoaStOZGpJRmdiOXNPcnNqeW42K3lxCkdYZTY5cWwwWlJ1SVVzUkF3SGJGY1ZaZUNvWXAvWVVvQlRwZjZwMDFlRHRYak1ZV0RkWlFPeTBqWWtncEwyMncKRmlTV3lFbTdSQjFDdlNyUFN1OHJnSzZKWGtveDU3V0ZKSGtwLzhVa2d4Y0VlWkRyMnJDRW5taE94aHl6SVN3YQpqZGhtVWdCditnSW1rKzUvdmp2STZoTWhmNncxQjE2WnFyNnlsSFVmUXlXR2xwbytYK1BieDBqRjlxV3JUMVBrCjVYSThoYzFhVXZLdFowRTlKb0Z2NG40NjBjc1lmenBJTEdOZU5LZUVaNWx1N242REpraGw3UVVWYkZ5dmxwWVQKckZjbnpRS0NBUUJiOEc1MWk5RFBHTDFRVUQrSTl3ZFVKTkN5QzBQSjhtM3lzQ29idlVvVkNYVDVkSFluNUpvTwpxOTArL01wZW9jMW1Hc0FIV2tCUXZWekJvUi9wUS9tTi9PbzJadmVuMlZyTElCdUplb3A2VTJzeitEUUVqTUZwClZTRlc0NTdBd1lVdzVxTnJIaCtHQ2xHeHZkQmREK0lNazJWNlVPNjNLUnE1M2w4N1RnNUd6ZEkwaWZLUi9SOWkKWlA4aloxTUt3dkpXZ1JzNTdETFBjMHI5eDY3bVZtZVVudHhCRldGOFovc0xNdHY3dk5LY0FhTzlLZEViL2Z1cgpRSW81Sm1yZSt1ckZteWcxbzdXTHNmMzBZZ2dIYzEvZC8vM3ZKbENnaElzSXdSNEx3cnFML3prUDJTQ285MSt0CmtMWHd5dXJ2OE1McHA0dGZBQUU4NjZFdlVqQ1V6SFJaQW9JQkFRRE81UGgzb0d1TVB4MWRkdlFDZ0N4bWplaWIKaEVYREdGR1FzaGx2eFZtUDdQdkdQRk5pWEFnUnlvdkZhd3BZZEkyWlhXbW8xaitTS2dsM0MrQUFpL0Uyd2d5QwpKN0QrME9ralRJemVaYzNHL3VBWWNyTVc1OHVnZEZrLzROdkJyQkQvdGd3SlhNaTRKOXFsa0NjMjRSUzJVQ0ZSClZnZldnN2xUYjJmM1gzSmZrUVZlOVJ5b1R6MUpIbXQ3UXQzc0NOOFEvRzRKbWsrY3k0VXdzQ2lmRDVWWTB1d2QKaDkxcGRNdDBWbUJQS1kzRUVyemhLR29ZVHdsYnJ6YlRnWDEvNWxQeSt3Q3BjTnBxR3RxK29jZ2xTbWI5VHhrTQo2NzMxeTc3NmFKTTNEb2VZM1lDU0VLUjVzU1hxaXRFNm93TWU1Z20vR2NqMEhVK1FqcE5jRUMzaXdGeEoKLS0tLS1FTkQgUlNBIFBSSVZBVEUgS0VZLS0tLS0K
kind: Secret
metadata:
creationTimestamp: "2020-04-09T23:06:53Z"
generateName: sealed-secrets-key
labels:
sealedsecrets.bitnami.com/sealed-secrets-key: active
name: sealed-secrets-keyhvdtf
namespace: kube-system
resourceVersion: "302785"
selfLink: /api/v1/namespaces/kube-system/secrets/sealed-secrets-keyhvdtf
uid: cd53c49a-7ab6-11ea-a485-0adbb667fc0b
type: kubernetes.io/tls
kind: List
metadata:
resourceVersion: ""
selfLink: ""Sealing the Secrets
Since 1.14, kubectl supports the management of Kubernetes objects using Kustomize, which provides Resource Generators to create Kubernetes resources, such as Secrets and ConfigMaps. The Kustomize generators should be specified in a kustomization.yaml file. The YAML manifest for a Secret is generated from literal key-value pairs using kubectl and kustomize as shown in the example below:
cat << EOF > kustomization.yaml
namespace: octank
secretGenerator:
- name: database-credentials
literals:
- username=admin
- password=Tru5tN0!
generatorOptions:
disableNameSuffixHash: true
EOF
kubectl kustomize . > secret.yamlUsing this Secret, the YAML manifest for the SealedSecret CRD is created using kubeseal as follows:
kubeseal --format=yaml < secret.yaml > sealed-secret.yamlkubeseal encrypts the Secret using the public key that it fetches at runtime from the controller running in the Kubernetes cluster. If a user does not have direct access to the cluster, then a cluster administrator may retrieve the public key from the controller logs and make it accessible to the user.
A SealedSecret CRD is then created using kubeseal as follows using the public key file:
kubeseal --format=yaml --cert=public-key-cert.pem < secret.yaml > sealed-secret.yamlThe generated Secret with Base64 encoded value for username and password keys is as follows:
apiVersion: v1
kind: Secret
metadata:
name: database-credentials
namespace: octank
data:
password: VHJ1NXROMCE=
username: YWRtaW4=
type: OpaqueThe generated SealedSecret with encrypted values is shown below.
Note that the keys in the original Secret—namely, username and password—are not encrypted in the SealedSecret, only their values are. You may change the names of these keys, if necessary, in the SealedSecret YAML file and still be able to deploy it successfully to the cluster. However, you cannot change the name and namespace of the SealedSecret. Doing so will invalidate the SealedSecret because the name and namespace of the original Secret are used as input parameters in the encryption process. This provides the additional safeguard that even if a user were to gain access to the YAML manifest of a SealedSecret created for a certain namespace in a Kubernetes cluster that they do not have access to, they will not be able to merely edit that manifest and deploy the SealedSecret to a different namespace even within the same cluster.
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: database-credentials
namespace: octank
spec:
encryptedData:
password: AgBIpx1gp2VsE0gcHzaFgXDdhA1hu0p+D4U1ys1FsQ1h0g4IRVVJqDUh56YRFBkj9bSt9dVXevRiO27mTPwyKXP6vnS2Oe/fM7x0LMQ1oXd6gTe2NB2o8r6ek3AenTghObRn1lbKigAHFtBFYN+Pq6wQptoewsaRhv+fTxRFk1PMjK5PHu/ETwsm74KeZ3VjOzEUkrAaRN2BPML0UvLstEy/yDMHD3+hLaGP/slwd5oeyhbVWutsofpyNNwLiZ5EvKhEr6vuCQ4CF0gvtFhXVF7e84m3SmZcZ1AvYicqW026PxjbL1b42T8Hk9PUboYfydSsBUjFAmXhzHLu+FrBuHIHdbikpd4kPOOqHO9z9wt39lwhiHf6f3YgesMFmVsWpco+ghn79fumfzTCrVHdwZGQ/7oYNDrcBNQB+kKQSOy+p3W2YR6BhhoGKHxELcqMZU9/1gKHIfbHPFoJtsK1SW2KwfpGeh0kvn/NtWj5sBeLrMYD9fev9j+jaDSOkzmB7ftpNx85DhOUVYJ2O/e4qUkzf0xASLq4XzwhknPOAbaRZ8oxBHQBKZKCW6x6RUpxOUXe7uImKEdodbDbCyrcIMH7a1SILohg5jpEDUUbxQBGtt7bFt4EJoF0LKLMDvzO/R02kHhuFiIRUwH2NJM8IpupwfLVdOBEBJ8yh4agmRvAm8bioH+bQELMQvc4DZHbiIOq0L2DlkzTyw==
username: AgA5lKkkdSFgQbO8NXgEb/ohsWlUH5ngoxa4SAdQt/kN84eowrUev71KLnZsahJ1sDHr0GQHWW32hJlemG+GnD3ASSEjvJABWwMIdpeJzi5vEPtp5fNiHZq5pmlJGnJZxHRvVlLSOfhf473r9Sbo/dS3OUvzsDabnb6hQzSVMJjS6RYTQp6JnsmvjDrOGk0P7Gik6bdgNivqmxrEiddqkanSqjS+MXMsY2HYnECr5EG7QRawq2yWl6UVQRfVLjoB9n8MAGHELkIXo5aiH1MwYdH0jnPukWZrwIsz1BbeBW4jX9wjtjXpZKKxrywd6AOhoodL6YIRGhWvRXTswQfktAdk5Y7Gu7vlnQMIkeaKjd7sXwO+TgoW4T8WTwTY7LPe4vLNI87AX1ZWCmLUexX1YlmwNJUGPIP8+WfFlkQO2YkcOt8Imt5lJI13+CGAPl13RZ5jVVAsO8HnE4GaufWRDHglb1GOAtiwnKRYp5JS0ipzQZspH+tpACFIFKLqGykMPBoRoxEqqtdouYs1b8k2cAj7uhhWaibn3f0T1gYMMuAwu5FD/A1KX0YRtcBXab2VPDOxD0cLwwjimkAjWgXrzVjCHq9yn9CyCsd9Hie56lesphmk+/kiZ0fwr5T9UZnJQrL/REplDgUVnFdLZfdY9PFCeylAeTv6KUjHsmc6O6cooPwXKFhntGDHm4GGYz9T8cfQWI7NFA==
template:
metadata:
creationTimestamp: null
name: database-credentials
namespace: octank
type: Opaque
status: {}The YAML manifest that pertains to the Secret is no longer needed and may be deleted. The SealedSecret is the only resource that will be deployed to the cluster as follows:
kubectl create namespace octank
kubectl apply -f sealed-secret.yaml
Once the SealedSecret CRD is created in the cluster, the controller becomes aware of it and unseals the underlying Secret using the private key and deploys it to the same namespace. This is seen by looking at the logs from the controller:
2020/04/09 23:06:53 HTTP server serving on :8080
2020/04/09 23:19:16 Updating octank/database-credentials
2020/04/09 23:19:16 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"octank", Name:"database-credentials", UID:"882f5e6c-7ab8-11ea-9ebe-1243fd9383c9", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"304045", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfullyThe YAML file, sealed-secret.yaml, that pertains to the SealedSecret is safe to be stored in a Git repository along with YAML manifests pertaining to other resources—such as DaemonSets, Deployments, and ConfigMaps—deployed in the cluster. We can now use a GitOps workflow and securely manage the deployment of Secret resources to your cluster.
Securing the sealing key
Without the private key that is managed by the controller, there is no way to decrypt the encrypted data within a SealedSecret. In the event that you are trying to restore the original state of a cluster—say, after a disaster or you want to leverage GitOps workflow to deploy the same set of Kubernetes resources, including SealedSecrets, from a Git repository and stand up a separate instance of a Kubernetes cluster—the controller deployed in the new cluster must use the same private key to be able to unseal the SealedSecrets. Absent this private key, all SealedSecrets will have to be regenerated using a new private/public key pair, which could become an onerous task for a deployment containing a large number of Secret resources.
The private key can be retrieved from the controller using the following command. In a production environment, one would typically make use of Kubernetes RBAC to grant the permissions required to perform this operation to a restricted set of clients.
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > master.yaml
To test how this works, let’s first delete the installation of the controller, the Secret that it created in the kube-system namespace that contains the private key, the SealedSecret resource named database-credentials, as well as the Secret that was unsealed from it:
kubectl delete secret database-credentials -n octank
kubectl delete sealedsecret database-credentials -n octank
kubectl delete secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key
kubectl delete -f controller.yaml
Now, let’s reinstate the Secret containing the private key back into the cluster using the master.yaml file:
kubectl apply -f master.yaml
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key
Output from the above command shows the new Secret created in the kube-system namespace using the private key from the master.yaml file:
NAME TYPE DATA AGE
sealed-secrets-keyhvdtf kubernetes.io/tls 2 5s
Next let’s redeploy the SealedSecret CRD, controller, and RBAC artifacts to the cluster. Note that we are using the same YAML manifest for the SealedSecret that was generated earlier:
kubectl apply -f controller.yaml
kubectl logs sealed-secrets-controller-6b7dcdc847-jkrz8 -n kube-system
As shown in the new controller’s logs, it is able to find the existing Secret sealed-secrets-keyhvdtf in the kube-system namespace and thus does not create a new private/public key pair:
2020/04/09 23:26:07 Starting sealed-secrets controller version: v0.12.1+dirty
controller version: v0.12.1
2020/04/09 23:26:07 Searching for existing private keys
2020/04/09 23:26:07 ----- sealed-secrets-keyhvdtf
2020/04/09 23:26:07 HTTP server serving on :8080Now let’s redeploy the SealedSecret and verify that the controller is able to successfully unseal it:
kubectl apply -f sealed-secret.yaml
kubectl logs sealed-secrets-controller-6b7dcdc847-jkrz8 -n kube-system
2020/04/09 23:26:07 Starting sealed-secrets controller version: v0.12.1
controller version: v0.12.1
2020/04/09 23:26:07 Searching for existing private keys
2020/04/09 23:26:07 ----- sealed-secrets-keyhvdtf
2020/04/09 23:26:07 HTTP server serving on :8080
2020/04/09 23:29:33 Updating octank/database-credentials
2020/04/09 23:29:33 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"octank", Name:"database-credentials", UID:"f7d22e88-7ab9-11ea-a485-0adbb667fc0b", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"305137", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfullyIf the master.yaml file that contains the public/private key pair generated by the controller is compromised, then all the SealedSecret manifests can be unsealed and the encrypted sensitive information they store revealed. Hence, this file must be guarded by granting least privilege access. For additional guidance on sealing key renewal, manual sealing key management, and so on, consult the documentation.
One option to secure the private key is to store the master.yaml file contents as a SecureString parameter in AWS Systems Manager Parameter Store. The parameter could be secured using a AWS Key Management Service (KMS) Customer managed key (KMS Key) and you can use the Key policy to restrict the set of AWS Identity and Access Management (IAM) principals who can use this key in order to retrieve the parameter. Additionally, you also may enable automatic rotation of this Key in KMS. Note that Standard tier parameters support a maximum parameter value of 4096 characters; given the size of the master.yaml file, you will have to store it as a parameter in the Advanced tier.
When a Secret resource is created, the Kubernetes API server persists the data in an etcd database that is part of the Kubernetes Control Plane. The default behavior is to persist the Secret data in Base64-encoded format.
When you launch an Amazon EKS cluster using Kubernetes version 1.13 or later, you may choose whether to enable or disable envelope encryption of Kubernetes secrets using AWS KMS. If you enable envelope encryption, the Kubernetes Secrets are encrypted using the customer managed key (KMS Key) that you select and then stored in the etcd database, thus adhering to the security best practice of encrypting data at rest. This feature is discussed in detail in “Using EKS encryption provider support for defense-in-depth”. Using this strategy in conjunction with the one discussed in this post gives users a robust mechanism for managing sensitive data needed for deploying their Kubernetes workloads.
Feature image via Pixabay.