Managing Spinnaker using Spinnaker Operator in Amazon EKS

• Halyard – Halyard is a command-line administration tool that manages the lifecycle of your Spinnaker deployment, including writing and validating its configuration, deploying each of Spinnaker’s microservices, and updating the deployment.
• Kleat – Intended to replace Halyard, Kleat is a tool for managing Spinnaker configuration files. It is currently under active development, but it is ready for early adopters to try in development clusters.
• Spinnaker Operator – is a Kubernetes operator to deploy and manage Spinnaker using familiar tools. You can install a basic version of Spinnaker or use kustomize files for advanced configuration.

• A Custom Resource Definition (CRD), which extends Kubernetes APIs
• A service account, role, and role binding to grant the operator access to the Kubernetes objects it needs to watch or modify.
• A Deployment object for the operator itself.

• Amazon Elastic Kubernetes Service (Amazon EKS) Cluster.
  • Here I have used Amazon EKS 1.20.
  • The cluster name is “eksworkshop-eksctl"
  • You will need minimum “m5.large” instance type as Spinnaker install has many pods to be deployed as part of Spinnaker service.
• Following tools are installed:
  • AWS Command Line Interface (AWS CLI)
  • The eksctl utility used for creating and managing Kubernetes clusters on Amazon EKS
  • The kubectl utility used for communicating with the Kubernetes cluster API server
  • The jq JSON processor
  • The Helm CLI used for installing Helm Charts
• GitHub account

Later in this blog, when we deploy this same application using Spinnaker pipeline, we will see new version of CatalogDetail which shows two vendors “ABC.com” and “XYZ.com”.

• Create an S3 bucket first: You can create an S3 bucket either using Admin Console or using AWS CLI. Use one of the options from below.
  • Using Admin Console:
    Go to AWS Console and go to Amazon S3 service and create the bucket as below.
    
    
  • Using AWS CLI:

    export S3_BUCKET=spinnaker-workshop-$(cat /dev/urandom | LC_ALL=C tr -dc "[:alpha:]" | tr '[:upper:]' '[:lower:]' | head -c 10)
    aws s3 mb s3://$S3_BUCKET
    aws s3api put-public-access-block \
    --bucket $S3_BUCKET \
    --public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"
    echo $S3_BUCKET

• Set up an IAM service account to access Amazon S3
  • Create the OIDC provider for your cluster.

    eksctl utils associate-iam-oidc-provider --cluster eksworkshop-eksctl --approve

    The output will be:

    [ℹ]  using region {AWS_REGION}
    [ℹ]  will create IAM Open ID Connect provider for cluster "eksworkshop-eksctl" in "{AWS_REGION}"
    [✔]  created IAM Open ID Connect provider for cluster "eksworkshop-eksctl" in "{AWS_REGION}"

  • Now, create the service account to access the S3 bucket that uses the managed policy.

    eksctl create iamserviceaccount \
        --name s3-access-sa \
        --namespace spinnaker \
        --cluster eksworkshop-eksctl \
        --attach-policy-arn arn\:aws\:iam::aws\:policy/AmazonS3FullAccess \
        --approve \
        --override-existing-serviceaccounts
    
    export S3_SERVICE_ACCOUNT=s3-access-sa
    

  • Confirm the details on the service account.

    kubectl describe sa s3-access-sa -n spinnaker
    
    echo $S3_SERVICE_ACCOUNT

    The output will be:

    Name:                s3-access-saNamespace:           spinnaker
    Labels:              app.kubernetes.io/managed-by=eksctl
    Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/eksctl-eksworkshop-eksctl-addon-iamserviceac-Role1-1WHK24KTB8MC3
    Image pull secrets:  <none>
    Mountable secrets:   s3-access-sa-token-pvf4k
    Tokens:              s3-access-sa-token-pvf4k
    Events:              <none>
    
    s3-access-sa

• Create an Amazon ECR repository
  We will be testing the trigger in Spinnaker pipeline for Amazon ECR image of one of our microservice proddetail. We need to push a test container image to the newly created ECR repository because an empty ECR repository does not show up in the Spinnaker UI when we set up the trigger in Pipeline. Below we are building a test container image for proddetail and pushing it to the newly created ECR repo.

  cd ~/environment/eks-app-mesh-polyglot-demo
  export ECR_REPOSITORY=eks-workshop-demo/test-detail
  export APP_VERSION=1.0
  aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
  aws ecr describe-repositories --repository-name $ECR_REPOSITORY >/dev/null 2>&1 || \
    aws ecr create-repository --repository-name $ECR_REPOSITORY >/dev/null
  TARGET=$ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPOSITORY:$APP_VERSION
  docker pull nginx
  docker tag nginx $TARGET
  docker push $TARGET

  Go to the Amazon ECR repository in AWS console and check if this repository has been pushed by Docker.

  

• Create a ConfigMap Create the ConfigMap for ECR token.

  cd ~/environment
  
  cat << EOF > config.yaml
  interval: 30m # defines refresh interval
  registries: # list of registries to refresh
    - registryId: "$ACCOUNT_ID"
      region: "$AWS_REGION"
      passwordFile: "/etc/passwords/my-ecr-registry.pass"
  EOF
  
  kubectl -n spinnaker create configmap token-refresh-config --from-file config.yaml

  Output should be as below:

  namespace/spinnaker createdconfigmap/token-refresh-config created

  Confirm if the ConfigMap is created correctly.

  kubectl describe configmap token-refresh-config -n spinnaker

  Output looks like below:

  Name:         token-refresh-config
  Namespace:    spinnaker
  Labels:       <none>
  Annotations:  <none>
  
  Data
  ====
  config.yaml:
  ----
  interval: 30m # defines refresh interval
  registries: # list of registries to refresh
    - registryId: "405710XXXXX"
      region: "us-west-2"
      passwordFile: "/etc/passwords/my-ecr-registry.pass"

• Spinnaker is configured with one or more “Cloud Provider” Kubernetes accounts, which you can think of as deployment targets.
• For each Kubernetes account, Spinnaker is provided a kubeconfig to connect to that Kubernetes cluster.
• The kubeconfig should have the following components:
  • A Kubernetes kubeconfig cluster
  • A Kubernetes kubeconfig user
  • A Kubernetes kubeconfig context
  • Metadata, such as which context to use by default
• Each Kubernetes account is configured in the SpinnakerService manifest under spec.spinnakerConfig.config.providers.kubernetes.accounts key. Each entity has these fields, among others:
  • name: A Spinnaker-internal name
  • kubeconfigFile: A file path referencing the contents of the kubeconfig file for connecting to the target cluster.
  • onlySpinnakerManaged: When true, Spinnaker only caches and displays applications that have been created by Spinnaker.
  • namespaces: An array of namespaces that Spinnaker will be allowed to deploy to. If this is left blank, Spinnaker will be allowed to deploy to all namespaces
  • omitNamespaces: If namespaces is left blank, you can blacklist specific namespaces to indicate to Spinnaker that it should not deploy to those namespaces
• If the kubeconfig is properly referenced and available, Operator will take care of the following:
  • Creating a Kubernetes secret containing your kubeconfig in the namespace where Spinnaker lives
  • Dynamically generating a clouddriver.yml file that properly references the kubeconfig from where it is mounted within the Clouddriver container
  • Creating/Updating the Kubernetes Deployment (spin-clouddriver) which runs Clouddriver so that it is aware of the secret and properly mounts it in the Clouddriver pod

• Download the latest spinnaker-tools release
  • This tool helps to create the ServiceAccount, ClusterRoleBinding, kubeconfig for the service account for the Amazon EKS/Kubernetes account

    cd ~/environment
    git clone https://github.com/armory/spinnaker-tools.git
    cd spinnaker-tools
    go mod download all
    go build

    Output should be:

    Cloning into 'spinnaker-tools'...remote: Enumerating objects: 278, done.
    remote: Counting objects: 100% (6/6), done.
    remote: Compressing objects: 100% (6/6), done.
    remote: Total 278 (delta 0), reused 4 (delta 0), pack-reused 272
    Receiving objects: 100% (278/278), 84.72 KiB | 4.71 MiB/s, done.
    Resolving deltas: 100% (124/124), done.

• Set up environment variables
  • Run these commands to set and check the environment variables:

    export CONTEXT=$(kubectl config current-context)
    export SOURCE_KUBECONFIG=${HOME}/.kube/config
    export SPINNAKER_NAMESPACE="spinnaker"
    export SPINNAKER_SERVICE_ACCOUNT_NAME="spinnaker-ws-sa"
    export DEST_KUBECONFIG=${HOME}/Kubeconfig-ws-sa
    echo $CONTEXT
    echo $SOURCE_KUBECONFIG
    echo $SPINNAKER_NAMESPACE
    echo $SPINNAKER_SERVICE_ACCOUNT_NAME
    echo $DEST_KUBECONFIG

    If you do not see output from the above command for all the above environment variables, do not proceed to next step.

• Create the service account
  • Create the kubernetes service account with namespace-specific permissions

    ./spinnaker-tools create-service-account   --kubeconfig ${SOURCE_KUBECONFIG}   --context ${CONTEXT}   --output ${DEST_KUBECONFIG}   --namespace ${SPINNAKER_NAMESPACE}   --service-account-name ${SPINNAKER_SERVICE_ACCOUNT_NAME}

    Output should be:

    Cloning into 'spinnaker-tools'...remote: Enumerating objects: 278, done.
    remote: Counting objects: 100% (6/6), done.
    remote: Compressing objects: 100% (6/6), done.
    remote: Total 278 (delta 0), reused 4 (delta 0), pack-reused 272
    Receiving objects: 100% (278/278), 84.72 KiB | 4.71 MiB/s, done.
    Resolving deltas: 100% (124/124), done.
    
    Getting namespaces ...
    Creating service account spinnaker-ws-sa ...
    Created ServiceAccount spinnaker-ws-sa in namespace spinnaker
    Adding cluster-admin binding to service account spinnaker-ws-sa ...
    Created ClusterRoleBinding spinnaker-spinnaker-ws-sa-admin in namespace spinnaker
    Getting token for service account ... 
    Cloning kubeconfig ... 
    Renaming context in kubeconfig ... 
    Switching context in kubeconfig ... 
    Creating token user in kubeconfig ... 
    Updating context to use token user in kubeconfig ... 
    Updating context with namespace in kubeconfig ... 
    Minifying kubeconfig ... 
    Deleting temp kubeconfig ... 
    Created kubeconfig file at /home/ec2-user/Kubeconfig-ws-sa

• Configure EKS account
  • Open the SpinnakerService manifest located under deploy/spinnaker/basic/spinnakerservice.yml.

    providers:
              dockerRegistry:
                enabled: true
              kubernetes:
                enabled: true
                accounts:
                - name: spinnaker-workshop
                  requiredGroupMembership: []
                  providerVersion: V2
                  permissions:
                  dockerRegistries:
                    - accountName: my-ecr-registry
                  configureImagePullSecrets: true
                  cacheThreads: 1
                  namespaces: [spinnaker,workshop]
                  omitNamespaces: []
                  kinds: []
                  omitKinds: []
                  customResources: []
                  cachingPolicies: []
                  oAuthScopes: []
                  onlySpinnakerManaged: false
                  kubeconfigFile: kubeconfig-sp  # File name must match "files" key
                primaryAccount: spinnaker-workshop  # Change to a desired account from the accounts array
          files:
              kubeconfig-sp: |
                 <REPLACE_ME_WITH_FILE_CONTENT> # Content from kubeconfig created by Spinnaker Tool

  • Replace the <REPLACE_ME_WITH_FILE_CONTENT> in the above section of deploy/spinnaker/basic/spinnakerservice.yml with the kubeconfig content from ${HOME}/Kubeconfig-ws-sa.
    • From the terminal, Go to ${HOME}/Kubeconfig-ws-sa (in my case it was /home/ec2-user/Kubeconfig-ws-sa) and copy the kubeconfig text starting from “apiVersion…” to the end of file.
    • Align the tab of the added file content to look as below
• Verify the Spinnaker manifest
  • By now we have completed our configuration for Spinnaker, and the SpinnakerService manifest located at deploy/spinnaker/basic/spinnakerservice.yml should look similar to below.
  • Note: “$$$” in the YAML below is just a placeholder. Do not copy the content under “kubeconfig-sp:” in this file. Copy the content from ${HOME}/Kubeconfig-ws-sa to this section.

    apiVersion: spinnaker.io/v1alpha2
    kind: SpinnakerService
    metadata:
      name: spinnaker
    spec:
      spinnakerConfig:
        config:
          version: $SPINNAKER_VERSION # the version of Spinnaker to be deployed
          persistentStorage:
            persistentStoreType: s3
            s3:
              bucket: $S3_BUCKET
              rootFolder: front50
              region: $AWS_REGION
          deploymentEnvironment:
            sidecars:
              spin-clouddriver:
              - name: token-refresh
                dockerImage: quay.io/skuid/ecr-token-refresh:latest
                mountPath: /etc/passwords
                configMapVolumeMounts:
                - configMapName: token-refresh-config
                  mountPath: /opt/config/ecr-token-refresh
          features:
            artifacts: true
          artifacts:
            github:
              enabled: true
              accounts:
              - name: $GITHUB_USER
                token: $GITHUB_TOKEN # GitHub's personal access token. This fields supports `encrypted` references to secrets.
          providers:
    	        dockerRegistry:
                  enabled: true
                kubernetes:
                  enabled: true
                  accounts:
                  - name: spinnaker-workshop
                    requiredGroupMembership: []
                    providerVersion: V2
                    permissions:
                    dockerRegistries:
                      - accountName: my-ecr-registry
                    configureImagePullSecrets: true
                    cacheThreads: 1
                    namespaces: [spinnaker,workshop]
                    omitNamespaces: []
                    kinds: []
                    omitKinds: []
                    customResources: []
                    cachingPolicies: []
                    oAuthScopes: []
                    onlySpinnakerManaged: false
                    kubeconfigFile: kubeconfig-sp  # File name must match "files" key
                  primaryAccount: spinnaker-workshop  # Change to a desired account from the accounts array
        files:
            kubeconfig-sp: |
                apiVersion: v1
                clusters:
                - cluster:
                    certificate-authority-data: $$$$
                    server: $$$$$
                  name: arn:aws:eks:$AWS_REGION-2:$ACCOUNT_ID:cluster/eksworkshop-eksctl
                contexts:
                - context:
                    cluster: arn:aws:eks:$AWS_REGION:$ACCOUNT_ID:cluster/eksworkshop-eksctl
                    namespace: spinnaker
                    user: spinnaker-token-user
                  name: spinnaker
                current-context: spinnaker
                kind: Config
                preferences: {}
                users:
                - name: spinnaker-token-user
                  user:
                    token: $$$$$
        profiles:
            clouddriver:
              dockerRegistry:
                enabled: true
                primaryAccount: my-ecr-registry
                accounts:
                - name: my-ecr-registry
                  address: https://$ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
                  username: AWS
                  passwordFile: /etc/passwords/my-ecr-registry.pass
                  trackDigests: true
                  repositories:
                  - $ECR_REPOSITORY
            igor:
              docker-registry:
                enabled: true
        service-settings:
          front50:
            kubernetes:
              serviceAccountName: $S3_SERVICE_ACCOUNT
              securityContext:
                fsGroup: 100
      # spec.expose - This section defines how Spinnaker should be publicly exposed
      expose:
        type: service  # Kubernetes LoadBalancer type (service/ingress), note: only "service" is supported for now
        service:
          type: LoadBalancer

• You are done with the Spinnaker configuration for all the Spinnaker services. Now, let’s install Spinnaker.

• Access Spinnaker UI: Grab the load balancer URL from the previous chapter, or use the below command to get the load balancer URL.

  kubectl -n spinnaker get spinsvc spinnaker -w

  Open the URL in the browser. You should see the below Spinnaker UI.

  

• Create application: Click on Create Application and enter name as product-detail and email as your email. Leave the rest of the fields as default. Then, click on “Create.”
  
  

• Create pipeline: Click on Pipelines under product-detail and click on link Configure a new pipeline and add the name helm-pipeline.
  

• Set up trigger: You should now be in the Configuration page.
  • • Now click on Add Trigger under Automated Triggers
    • Select Type as Docker Registry.
    • In the Registry Name dropdown you should see the value my-ecr-registry, select that.
    • In the Organization dropdown you should see the value eks-workshop-demo, select that.
    • In the Image dropdown you should see the value eks-workshop-demo/test-detail, select that.
    • Click on Save Changes.

  This is the ECR registry we set up in Spinnaker manifest in Step 7 – Configure ECR Artifact.

  

• Evaluate variable configuration

• • Click on Add Stage and select type as Evaluate Variables from the dropdown.
  • Add the variable name as image_name and value as ${trigger['tag']}.
  • Add another variable name as repo_name and value as $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/eks-workshop-demo/test-detail. Replace $ACCOUNT_ID and $AWS_REGION based on your setup.
  • Click on Save Changes. We will be using these variables in the next Bake Stage.
    

• Set up bake stage

• • Click on Add Stage and select Type as Bake Manifest from the dropdown.
  • Select Template Renderer as Helm3 and enter name as workshop-detail and enter workshop as namespace.
  • Select Expected Artifact as Define a new artifact
    • Select your Git account that shows in dropdown. This is the Git account we had setup in Spinnaker manifest in Step 8.
    • And then enter the below git location in the Content URL and add master as the Commit/Branch. This is to provide the the Helm template for the deployment.
      • https://api.github.com/repos/aws-containers/eks-app-mesh-polyglot-demo/contents/workshop/productcatalog_workshop-1.0.0.tgz
    • Keep the branch as “master.”
  • Under the Overrides section, select Add new artifact.
    • Select Expected Artifact as Define a new artifact
    • Select your Git account that shows in dropdown. This is the Git account we had setup in Spinnaker manifest in Step 8.
    • Enter the below git location in the Content URL. This is to provide the overrides for the Helm template using values.yaml.
      • https://api.github.com/repos/aws-containers/eks-app-mesh-polyglot-demo/contents/workshop/helm-chart/values.yaml
    • Keep the branch as “master”
  • Under the Overrides key/value section, click on “Add override.”
    • Enter first key as detail.image.repository for repository and value as ${repo_name}.
    • Enter second key as detail.image.tag for tag and value as ${image_name}.
    • The keys are based on the Values.yaml from the Helm chart and the values are the variables that we set in previous step “Evaluate Variables.”
  • Edit the Produces Artifacts and change the name to helm-produced-artifact and click Save Artifact. Then, click Save Changes.
    
    

• Set up deploy stage

• • Click on Add Stage and select “Type” as Deploy (Manifest) from the dropdown, and give a name as Deploy proddetail
  • Select Account as spinnaker-workshop from the dropdown. This is the EKS account we had setup in Spinnaker manifest in Step 9 – Add EKS account.
  • Select Artifact and then select helm-produced-artifact from the dropdown for Manifest Artifact and click Save Changes.
    

• Push the new container image to ECR for testing trigger. To ensure that the Amazon ECR trigger will work in Spinnaker UI:
  • First, change the code to generate a new docker image digest. Note: The Amazon ECR trigger in Spinnaker does not work for same docker image digest.
  • Go to ~/environment/eks-app-mesh-polyglot-demo/workshop/apps/catalog_detail/app.js and replace the line "vendors":[ "ABC.com"] with "vendors":[ "ABC.com","XYZ.com"]
  • Ensure that the image tag (APP_VERSION) you are adding below does not exist in the Amazon ECR repository eks-workshop-demo/test-detail otherwise the trigger will not work. Spinnaker pipeline only triggers when a new version of image is added to ECR.
  • Then, run the below command.

    cd ~/environment/eks-app-mesh-polyglot-demo/workshop
    aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
    export APP_VERSION=5.0 ## pick a version that is not there in the ECR
    export ECR_REPOSITORY=eks-workshop-demo/test-detail
    TARGET=$ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPOSITORY:$APP_VERSION
    docker build -t $TARGET apps/catalog_detail
    docker push $TARGET

• Watch the pipeline getting triggered
  • You should see the image version 5.0 get triggered.
    

• • You will see that Docker push triggers a deployment in the pipeline.
    

• • Below are the Execution Details of pipeline:
    
    
    

• Get deployment details

• • Click on Clusters and you can see the deployment of frontend, prodcatalog, and proddetail service below.
    

• • Click on the LoadBalancer icon link for frontend and you should see below information. Click on that Load Balancer link or Paste the link on browser.
    
  • You should see the service up and running as below.
    

• • Now add a product. Below, we’ve used “1“ as “id“ and “Table“ as “name.“ And you see that an additional vendor, XYZ.com, is added  from the new container image for proddetail service we pushed into ECR.
    
  • You can also go to the terminal and confirm the deployment details.
    kubectl get all -n workshop
    You can see the below output
    
NAME READY STATUS RESTARTS AGE
pod/frontend-7b78bc4cbb-fr2mz 1/1 Running 0 16m
pod/prodcatalog-f6d7bffb5-rjbz2 1/1 Running 0 16m
pod/proddetail-75cd46fb7b-k82lg 1/1 Running 0 16mNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    service/frontend LoadBalancer 10.100.221.87 aa76467f53d53419aa273bf96b8cdd47-XXXXX.us-east-2.elb.amazonaws.com 80:32022/TCP 10h
    service/prodcatalog ClusterIP 10.100.213.2 <none> 5000/TCP 10h
    service/proddetail ClusterIP 10.100.144.79 <none> 3000/TCP 10h

    NAME READY UP-TO-DATE AVAILABLE AGE
    deployment.apps/frontend 1/1 1 1 10h
    deployment.apps/prodcatalog 1/1 1 1 10h
    deployment.apps/proddetail 1/1 1 1 10h

• https://spinnaker.io/docs/concepts/
• https://spinnaker.io/docs/reference/architecture/microservices-overview/
• https://spinnaker.io/docs/guides/user/managed-delivery/
• https://github.com/armory/spinnaker-operator
• https://docs.armory.io/armory-enterprise/installation/armory-operator/
• https://docs.armory.io/armory-enterprise/overview/architecture/
• http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html
• https://www.armory.io/blog/spinnaker-operator/