AWS Open Source Blog
Using Kubernetes Service Catalog and AWS Service Broker on EKS
In a previous post we discussed using the AWS Service Broker on Kubernetes. A lot has changed since then: EKS is now available, and the Service Broker has evolved significantly, so we thought it would be good to revisit the topic with a focus on EKS.
If you are a Kubernetes user, you may have found, as we did, that managing the lifecycle and credentials for disparate services such as databases can be challenging. Aside from having to work out how to provide your application access to needed credentials, it often involves linking multiple disparate tool chains, which adds undifferentiated heavy lifting. This can result in best practices such as least-privilege models being hard to enforce, often being left up to individual development teams to implement. In this post, we will discuss some patterns that help address these challenges. We’ll take a look at Kubernetes Service Catalog, an extension API that enables applications running in Kubernetes clusters to easily use external-managed software offerings, by consuming service brokers that implement the Open Service Broker API specification. The AWS Service Broker is an open source project that (in combination with Kubernetes Service Catalog) provides a catalog of AWS services that can be managed and connected to your Kubernetes applications using familiar Kubernetes APIs and tooling. We’ll also walk through what using the AWS Service Broker looks like in practice, by provisioning an S3 bucket and connecting it to your application.
Core Service Broker Concepts
First, we want to go over some of the key concepts that we’ll be using in this post. For more details, have a look at the Open Service Broker API Specification and Kubernetes Service Catalog design documentation.
ClusterServiceClass – A Kubernetes resource that Service Catalog generates. Service Catalog fetches the catalog of each installed broker and merges the new catalog entries into ClusterServiceClasses in Kubernetes. In the case of the AWS broker, these classes represent AWS services like S3, SQS, etc.
ClusterServicePlan – Each ClusterServiceClass contains one or more plans. A plan represents a configuration of the service; in the AWS broker, many services have opinionated plans for production and development use cases, as well as custom plans that allow the user to configure all of the available options for the service.
ServiceInstance – An instance of a ClusterServicePlan. With the AWS broker, this will represent an instance of the AWS service created when the provision API is called.
Provision – The provision API is what gets called when creating a new ServiceInstance
using kubectl
, the api or svcat
(the Kubernetes Service Catalog CLI tool). In the case of the AWS Service Broker, the provision call accepts various parameters depending on the plan, which are then used to create the requested AWS services.
Bind – Binding is the API call that requests credentials and metadata from the broker for a given ServiceInstance. The Service Catalog then creates a Kubernetes secret and stores the credentials in it. Applications can then map the secret to gain access to the credentials. For example, with an Amazon RDS database, the bind call returns the endpoint url, a username and a password. For services that require IAM at run time, a least-privilege policy is created and attached to an IAM user/role.
Prerequisites
- An EKS cluster with at least one node configured; three or more nodes are recommended for HA/production use. For details, see the EKS Getting Started Guide. The steps in this guide should work for any Kubernetes cluster (v1.10 and up), but your mileage may vary.
kubectl
andaws-iam-authenticator
installed/configured to connect to the above cluster. For details see the reference.awscli
installed and configured with AWS credentials. For details, see the AWS CLI documentation.jq
is used to inspect the output of the sample application; it can be obtained from your operating system’s package manager
In this post we’ve chosen to use native Kubernetes tools wherever possible, to highlight what the Service Catalog types look like and how they can be used natively. If you are regularly interacting with the catalog via the cli, you may want to look at the
svcat
cli tool, which simplifies describing and managing service catalog types. See the Service Catalog documentation on GitHub for more details
Installing Service Catalog
We’ll use Helm to handle the installation of the needed components.
curl https://raw.githubusercontent.com/helm/helm/master/scripts/get > get_helm.sh
chmod 700 get_helm.sh
./get_helm.sh
kubectl create serviceaccount tiller --namespace kube-system
kubectl create clusterrolebinding tiller \
--clusterrole=cluster-admin \
--serviceaccount=kube-system:tiller
# Install helm and tiller into the cluster
helm init --service-account tiller
# Wait until tiller is ready before moving on
until kubectl get pods -n kube-system -l name=tiller | grep 1/1; do sleep 1; done
Once completed, you should see a running tiller pod in kube-system namespace:
NOTE: This type of Tiller installation is not recommended for public-facing or production clusters. To learn more about installing Tiller in production clusters, see Securing your Helm Installation.
Now that we have Helm and Tiller set up, we can move on to using Helm to install Kubernetes Service Catalog.
helm repo add svc-cat https://svc-catalog-charts.storage.googleapis.com
helm install svc-cat/catalog \
--name catalog --namespace catalog --wait
Once install is complete we should see the api-server and controller-manager pods running in the catalog
namespace:
Installing the AWS Service Broker
The first step is to set up the prerequisites. This can be done easily using a CloudFormation template that creates the required IAM User and DynamoDB table. The following code block uses the AWS CLI to launch the template and gather the needed outputs. Be sure to set the REGION variable to the AWS region where you would like to have the broker provision resources.
REGION=us-west-2
# Download the template
wget https://raw.githubusercontent.com/awslabs/aws-servicebroker/master/setup/prerequisites.yaml
# Create stack
STACK_ID=$(aws cloudformation create-stack \
--capabilities CAPABILITY_IAM \
--template-body file://prerequisites.yaml \
--stack-name aws-service-broker-prerequisites \
--output text --query "StackId" \
--region ${REGION})
# Wait for stack to complete
until \
ST=$(aws cloudformation describe-stacks \
--region ${REGION} \
--stack-name ${STACK_ID} \
--query "Stacks[0].StackStatus" \
--output text); \
echo $ST; echo $ST | grep "CREATE_COMPLETE"
do sleep 5
done
# Get the username from the stack outputs
USERNAME=$(aws cloudformation describe-stacks \
--region ${REGION} \
--stack-name ${STACK_ID} \
--query "Stacks[0].Outputs[0].OutputValue" \
--output text)
# Create IAM access key. Note down the output, we'll need it when setting up the broker
aws iam create-access-key \
--user-name ${USERNAME} \
--output json \
--query 'AccessKey.{KEY_ID:AccessKeyId,SECRET_ACCESS_KEY:SecretAccessKey}'
Now we’re ready to install the broker, first adding the repository to Helm:
helm repo add aws-sb https://awsservicebroker.s3.amazonaws.com/charts
The broker has several configurable properties, to list them run:
helm inspect aws-sb/aws-servicebroker --version 1.0.0-beta.3
In this post we will cover a minimal installation. For details about advanced install options, see the AWS Service Broker documentation. Replace <ACCESS_KEY_ID> and <SECRET_ACCESS_KEY> below with the output saved from the output of the aws create-access-key
command:
helm install aws-sb/aws-servicebroker \
--wait \
--name aws-servicebroker \
--namespace aws-sb \
--version 1.0.0-beta.3 \
--set aws.region=${REGION} \
--set aws.accesskeyid=<ACCESS_KEY_ID> \
--set aws.secretkey=<SECRET_ACCESS_KEY>
NOTE: If setting aws.targetaccountid on the helm cli, do not use —set, use —set-string, see the related Helm issue for more info
Now verify that the broker pod is running:
helm ls --namespace aws-sb
kubectl get ClusterServiceBrokers
Now you can list the available services:
kubectl get ClusterServiceClasses \ -o=custom-columns=NAME:.spec.externalName,DESCRIPTION:.spec.description
If you ran into any issues, you can troubleshoot by having a look at the Service Broker’s logs:
kubectl logs $(kubectl get pods -n aws-sb -o name) -n aws-sb
Provisioning and Binding to a Sample Application
Let’s take the broker for a spin by creating a sample application, provisioning an S3 bucket, and binding the bucket to our application.
First create the sample application; we’ve provided a simple application that validates that the connection to S3 is functional:
cat <<EOF > sample.yaml
---
apiVersion: v1
kind: Namespace
metadata:
name: s3-demo
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: s3-demo
namespace: s3-demo
spec:
selector:
matchLabels:
app: s3-demo
replicas: 1
template:
metadata:
labels:
app: s3-demo
spec:
containers:
- name: s3-demo
image: awsservicebroker/s3-demo:latest
ports:
- containerPort: 8080
EOF
kubectl apply -f sample.yaml
# Wait for deployment to complete
until kubectl get pods -n s3-demo | grep 1/1; do sleep 1; done
Now we can check what the application returns when we curl it. Note that we haven’t set up a Kubernetes Service
or Ingress
for brevity, so we’ll run the curl using kubectl exec
:
kubectl exec \
$(kubectl get pods -o name --namespace s3-demo | awk -F '/' '{print $2}') \
--namespace s3-demo -- \
curl -s http://127.0.0.1:8080 | jq .
The above output shows that we’ve got some missing environment variables. Let’s resolve that by creating our S3 service and binding it to the application
cat <<EOF > s3-instance.yaml
---
apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceInstance
metadata:
name: s3-bucket
namespace: s3-demo
spec:
clusterServiceClassExternalName: s3
clusterServicePlanExternalName: production
EOF
kubectl apply -f s3-instance.yaml
That will start the provisioning process. You should see a CloudFormation stack being created in your account:
aws cloudformation list-stacks \
--region ${REGION} \
--query 'StackSummaries[?starts_with(StackName,`aws-service-broker-s3-`)]'
Shortly after the stack has completed, the ServiceClass will be ready to bind. You can check the status by describing the ServiceInstance
and checking .status.conditions[0].message
kubectl get ServiceInstance/s3-bucket -n s3-demo -o yaml
Now that our ServiceInstance has been created, we can bind to it so that our sample application can access those credentials from the kubernetes environment itself. Here’s what we need to do to create a binding resource:
cat <<EOF > s3-binding.yaml
---
apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceBinding
metadata:
name: s3-binding
namespace: s3-demo
spec:
instanceRef:
name: s3-bucket
EOF
kubectl apply -f s3-binding.yaml
Now that we have created a ServiceBinding object, we can verify if the status is “Ready”:
kubectl describe ServiceBinding s3-binding -n s3-demo
The binding action creates a Kubernetes secret containing the credentials needed to access the S3 bucket in “s3-demo” namespace. To verify that the secret wss created, run this command:
kubectl describe secrets/s3-binding -n s3-demo
All that’s left to do now is to attach the secrets to our pod:
cat <<EOF >> sample.yaml
env:
- name: S3_AWS_ACCESS_KEY_ID
valueFrom: { secretKeyRef: { name: s3-binding, key: S3_AWS_ACCESS_KEY_ID } }
- name: S3_AWS_SECRET_ACCESS_KEY
valueFrom: { secretKeyRef: { name: s3-binding, key: S3_AWS_SECRET_ACCESS_KEY } }
- name: S3_REGION
valueFrom: { secretKeyRef: { name: s3-binding, key: S3_REGION } }
- name: BUCKET_ARN
valueFrom: { secretKeyRef: { name: s3-binding, key: BUCKET_ARN } }
- name: BUCKET_NAME
valueFrom: { secretKeyRef: { name: s3-binding, key: BUCKET_NAME } }
- name: LOGGING_BUCKET_NAME
valueFrom: { secretKeyRef: { name: s3-binding, key: LOGGING_BUCKET_NAME } }
EOF
kubectl apply -f sample.yaml
# wait for deployment to complete
until kubectl get pods -n s3-demo | grep -c s3-demo | grep 1; do sleep 1; done
Once the deployment has completed, we should see that the application is now able to access the bucket:
kubectl exec \
$(kubectl get pods -o name --namespace s3-demo | awk -F '/' '{print $2}') \
--namespace s3-demo -- \
curl -s http://127.0.0.1:8080 | jq .
Cleanup
Let’s delete the sample application, binding, and instance. (Notice that the delete call results in the CloudFormation stacks being deleted.)
kubectl delete -f s3-binding.yaml
kubectl delete -f s3-instance.yaml
kubectl delete -f sample.yaml
You’re welcome to play around some more, but if you would prefer to also remove the broker and catalog from your cluster, you can run Helm to do so:
helm delete --purge aws-servicebroker
helm delete --purge catalog
# Remove cloudformation stack containing prerequisites
aws cloudformation delete-stack \
--stack-name $(echo $STACK_ID | awk -F '/' '{print $2'}) --region=$REGION
Summary
In this post, we’ve highlighted the power of using Kubernetes Service Catalog and the AWS Service Broker to manage your AWS services, providing separation of code and configuration, as well as providing baked-in best practices like least-privilege IAM. This is just the tip of the iceberg in terms of the functionality provided by the broker! For more advanced scenarios such multi-account provisioning, building your own custom catalog with CloudFormation templates, and using bind to attach to pod-level IAM roles, see the AWS Service Broker documentation. If you’re interested in exploring other ways to integrate AWS services with Kubernetes, we recommend that you also take a look at Chris Hein’s post on the AWS Service Operator.