Sailing with Spinnaker on AWS

We have recently begun contributing to an exciting open source project, Spinnaker from Netflix. Spinnaker is a cloud-based continuous delivery platform for releasing software changes rapidly and reliably. Spinnaker makes it easier for developers to focus on writing code without having to worry about the underlying cloud infrastructure. It integrates seamlessly with tools such as Jenkins and TravisCI. Spinnaker provides the flexibility to deploy applications on virtual machines running in the cloud or in your container platform of choice, such as Amazon ECS or Kubernetes. AWS customers like Capital One are using Spinnaker to provide their developers with a single CI/CD pipeline to deploy their applications.

Today, we are excited to share that Spinnaker can now be deployed on Amazon Elastic Container Service for Kubernetes (Amazon EKS). To ensure proper installation of Spinnaker on top of Amazon EKS, we recently updated the official documentation.

Integrating Spinnaker and Amazon EKS

Amazon EKS runs the Kubernetes management infrastructure across multiple AWS Availability Zones. EKS automatically detects and replaces unhealthy control plane nodes, and provides on-demand upgrades and patching. You simply provision worker nodes and connect them to the provided EKS endpoint.

When running Spinnaker on Amazon EKS, the application deployment tool inherits all the scaling and reliability benefits that Kubernetes offers.

In this post, we will walk you through how to install Spinnaker on Amazon EKS and enable other AWS-powered cloud providers:

AWS EC2 option: allows you to manage Amazon EC2 Instances.
AWS ECS option: allows you to manage containers on Amazon ECS.
Kubernetes option: allows you to manage containers on Amazon EKS.

Setup

Follow these instructions to configure a Kubernetes V2 (manifest-based) Clouddriver to run Spinnaker on EKS.

Set Up the Managing Account

In the managing account, create a two-subnet VPC, IAM roles, instance profiles, and a Security Group for EKS control-plane communications and an EKS cluster. This step will take around 15-20 minutes to complete.

curl -O https://d3079gxvs8ayeg.cloudfront.net/templates/managing.yaml
aws cloudformation deploy --stack-name spinnaker-managing-infrastructure-setup --template-file managing.yaml \
--parameter-overrides UseAccessKeyForAuthentication=false EksClusterName=spinnaker-cluster --capabilities CAPABILITY_NAMED_IAM

After the stack creation succeeds, run the following:

VPC_ID=$(aws cloudformation describe-stacks --stack-name spinnaker-managing-infrastructure-setup --query 'Stacks[0].Outputs[?OutputKey==`VpcId`].OutputValue' --output text)
CONTROL_PLANE_SG=$(aws cloudformation describe-stacks --stack-name spinnaker-managing-infrastructure-setup --query 'Stacks[0].Outputs[?OutputKey==`SecurityGroups`].OutputValue' --output text)
AUTH_ARN=$(aws cloudformation describe-stacks --stack-name spinnaker-managing-infrastructure-setup --query 'Stacks[0].Outputs[?OutputKey==`AuthArn`].OutputValue' --output text)
SUBNETS=$(aws cloudformation describe-stacks --stack-name spinnaker-managing-infrastructure-setup --query 'Stacks[0].Outputs[?OutputKey==`SubnetIds`].OutputValue' --output text)
MANAGING_ACCOUNT_ID=$(aws cloudformation describe-stacks --stack-name spinnaker-managing-infrastructure-setup --query 'Stacks[0].Outputs[?OutputKey==`ManagingAccountId`].OutputValue' --output text)
EKS_CLUSTER_ENDPOINT=$(aws cloudformation describe-stacks --stack-name spinnaker-managing-infrastructure-setup --query 'Stacks[0].Outputs[?OutputKey==`EksClusterEndpoint`].OutputValue' --output text)
EKS_CLUSTER_NAME=$(aws cloudformation describe-stacks --stack-name spinnaker-managing-infrastructure-setup --query 'Stacks[0].Outputs[?OutputKey==`EksClusterName`].OutputValue' --output text)
EKS_CLUSTER_CA_DATA=$(aws cloudformation describe-stacks --stack-name spinnaker-managing-infrastructure-setup --query 'Stacks[0].Outputs[?OutputKey==`EksClusterCA`].OutputValue' --output text)
SPINNAKER_INSTANCE_PROFILE_ARN=$(aws cloudformation describe-stacks --stack-name spinnaker-managing-infrastructure-setup --query 'Stacks[0].Outputs[?OutputKey==`SpinnakerInstanceProfileArn`].OutputValue' --output text)

Set Up the Managed Account

In each of managed accounts, create an IAM role that can be assumed by Spinnaker. This needs to be executed in the managing account as well:

curl -O https://d3079gxvs8ayeg.cloudfront.net/templates/managed.yaml  

aws cloudformation deploy --stack-name spinnaker-managed-infrastructure-setup --template-file managed.yaml \
--parameter-overrides AuthArn=$AUTH_ARN ManagingAccountId=$MANAGING_ACCOUNT_ID --capabilities CAPABILITY_NAMED_IAM

kubectl and heptio Authenticator Configuration

Install and configure kubectl and heptio authenticator for aws on the workstation/instance from which you are running Halyard. The Halyard version must be >=1.5.0.

When an Amazon EKS cluster is created, the IAM entity (user or role) that creates the cluster is added to the Kubernetes RBAC authorization table as the administrator. Initially, only that IAM user can make calls to the Kubernetes API server using kubectl.

If you use the console to create the cluster, you must ensure that the same IAM user credentials are in the AWS SDK credential chain when you are running kubectl commands on your cluster.

In the setup as shown above, we used the AWS CLI. Ensure that the server/workstation from which you are running the kubectl commands in step 2 below have the same AWS credentials.

Create default kubectl configuration file

Paste the following into your kubeconfig file, replacing <endpoint-url>, <base64-encoded-ca-cert> and <cluster-name> with the values for $EKS_CLUSTER_ENDPOINT, $EKS_CLUSTER_CA_DATA, and $EKS_CLUSTER_NAME as noted above:

apiVersion: v1
clusters:
- cluster:
    server: <endpoint-url>
    certificate-authority-data: <base64-encoded-ca-cert>
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: aws
  name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: heptio-authenticator-aws
      args:
        - "token"
        - "-i"
        - "<cluster-name>"
        # - "-r"
        # - "<role-arn>"
      # env:
        # - name: AWS_PROFILE
        #   value: "<aws-profile>"

(Optional) To have the Heptio authenticator assume a role to perform cluster operations (instead of the default AWS credential provider chain), uncomment the -r and <role-arn> lines and substitute an IAM role ARN to use with your user.

(Optional) To have the Heptio authenticator always use a specific named AWS credential profile (instead of the default AWS credential provider chain), uncomment the env lines and substitute <aws-profile> with the profile name to use.

Create the necessary service accounts and cluster role bindings

CONTEXT=aws
kubectl create namespace spinnaker
kubectl apply -f https://d3079gxvs8ayeg.cloudfront.net/templates/spinnaker-service-account.yaml 
kubectl apply -f https://d3079gxvs8ayeg.cloudfront.net/templates/spinnaker-cluster-role-binding.yaml 
TOKEN=$(kubectl get secret --context $CONTEXT \
   $(kubectl get serviceaccount spinnaker-service-account \
       --context $CONTEXT \
       -n spinnaker \
       -o jsonpath='{.secrets[0].name}') \
   -n spinnaker \
   -o jsonpath='{.data.token}' | base64 --decode)
kubectl config set-credentials ${CONTEXT}-token-user --token $TOKEN
kubectl config set-context $CONTEXT --user ${CONTEXT}-token-user

Enable Kubernetes Cloud Provider Using Halyard

This option will allow you to manage containers on EKS . You can replace “kubernetes-master” in the commands below with an account name of your choice.

hal config provider kubernetes enable
hal config provider kubernetes account add kubernetes-master --provider-version v2 --context $(kubectl config current-context)
hal config features edit --artifacts true

Enable AWS EC2 Cloud Provider Using Halyard

This option will allow you to manage EC2 instances. Replace ${NAME_OF_YOUR_AWS_ACCOUNT} with a friendly name for your AWS account and ${YOUR_AWS_ACCOUNT_ID} with your AWS account ID.

hal config provider aws account add ${NAME_OF_YOUR_AWS_ACCOUNT} \
    --account-id ${YOUR_AWS_ACCOUNT_ID} \
    --assume-role role/spinnakerManaged

hal config provider aws enable

Enable Amazon ECS Provider Using Halyard

This option will allow you to manage containers on ECS. Replace ${NAME_OF_YOUR_AWS_ACCOUNT} with the friendly name for your AWS account that you used in configuring the EC2 cloud provider above, and the ${ECS_ACCOUNT_NAME} with a friendly name that you want to use for ECS.

hal config provider ecs account add ${ECS_ACCOUNT_NAME} --aws-account ${NAME_OF_YOUR_AWS_ACCOUNT}
hal config provider ecs enable

Choose Halyard Distributed Deployment

This step will configure Halyard to deploy Spinnaker microservices on EKS.

hal config deploy edit --type distributed --account-name kubernetes-master

Choose Persistent Storage to S3 Using Halyard

We will use Amazon Simple Storage Service (S3) as persistent storage.

hal config storage s3 edit \
    --access-key-id ${ACCESS_KEY_HAVING_S3_ACCESS} \
    --secret-access-key \
    --region us-west-2

hal config storage edit --type s3

Launch and Configure Amazon EKS Worker Nodes

Worker nodes launched using the commands below are standard Amazon EC2 instances and use EKS optimized AMIs.

curl -O https://d3079gxvs8ayeg.cloudfront.net/templates/amazon-eks-nodegroup.yaml
aws cloudformation deploy --stack-name spinnaker-eks-nodes --template-file amazon-eks-nodegroup.yaml \
--parameter-overrides NodeInstanceProfile=$SPINNAKER_INSTANCE_PROFILE_ARN \
NodeInstanceType=t2.large ClusterName=$EKS_CLUSTER_NAME NodeGroupName=spinnaker-cluster-nodes ClusterControlPlaneSecurityGroup=$CONTROL_PLANE_SG \
Subnets=$SUBNETS VpcId=$VPC_ID --capabilities CAPABILITY_NAMED_IAM

Join the Nodes with the Spinnaker EKS Cluster

Replace $SPINNAKER_ROLE_ARN with $AUTH_ARN value and save it as aws-auth-cm.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: $SPINNAKER_ROLE_ARN
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes

Join the nodes with the cluster:

kubectl apply -f aws-auth-cm.yaml

Watch the status of your nodes and wait for them to reach the Ready status:

kubectl get nodes --watch

Deploy Spinnaker Using Halyard

List the available versions:

hal version list

Set the version you want to use:

hal config version edit --version ${version}
hal deploy apply

Connect


hal deploy connect

What’s Next

AWS will be presenting at the Spinnaker Summit this October 8th and 9th in Seattle. If you would like to learn more about the Spinnaker contributions we’re making, please join us on Slack #aws-general or email us at spinnaker-aws@amazon.com.

Select your cookie preferences

AWS Open Source Blog