AWS Cloud Operations & Migrations Blog

Auto discovery and dynamic route registration of services running in Amazon Elastic Kubernetes Service (EKS) with AWS Migration Hub Refactor Spaces

AWS Migration Hub Refactor Spaces automates infrastructure management, and traffic routing between monolithic and microservices endpoints. It uses two key constructs, service and route, to distribute incoming user traffic between monolithic and microservices endpoints. You can create the service and route configuration in Refactor Spaces from either the AWS Management Console, CLI, or the service APIs . To learn how to configure Services and Routes with Refactor Spaces, refer to the blog “AWS Migration Hub Refactor Spaces Helps to Incrementally Refactor Your Applications.”

In this blog, we focus on automating the discovery of endpoint URLs of microservices running inside Amazon Elastic Kubernetes Service (EKS), and registering those as Routes in Refactor Spaces.

Why we need service discovery and dynamic route registration

In an incremental modernization project, development teams working on microservices 1) create application container images, 2) Kubernetes manifest files, 3) deploy these on EKS, 4) and then manually register those new service endpoint URLs with Refactor Spaces. In a large modernization projects, where a high number of microservices are used, this manual process creates additional complexity.  In addition, the volume of manual changes grows as you promote Refactor Spaces service and route configurations through each environment in your software development lifecycle.

In this blog we will address how to reduce the manual effort and provide the following benefits from the solution:

  • Microservices deployed in EKS are auto discovered and dynamically registered with Refactor Spaces.  This also keeps both EKS and Refactor Spaces in synch. When a new microservice is deployed in EKS, it also gets registered with Refactor Spaces. Similarly when a microservice is deleted in EKS, the corresponding routing configuration in Refactor Spaces also gets deleted.
  • If routing configuration of Refactor Spaces is maintained as files in a source control system, it becomes version controlled and auditable.
  • Endpoint URLs of microservices exposed as Kubernetes service objects (NodePort, Ingress, LoadBalancer) are auto discovered and registered with Refactor Spaces.  There is no static configuration required. This makes the application promotion across environments (dev, test, stage, prod) less complex.

About the solution

The solution uses a custom-built Kubernetes operator, which looks for Kubernetes services of type LoadBalancer, Ingress and NodePort in the EKS cluster where the operator is installed.  It then uses API operations to manage the service and route configuration in Refactor Spaces.

You can check the full implementation of the solution here. [https://github.com/aws-samples/refactorspaces-operator]

Prerequisites:

You must have the following installed:

  • AWS CLI
  • helm
  • eksctl
  • kubectl installed

Walkthrough:

First, you must install the AWS Migration Hub Refactor Spaces EKS operator in your EKS cluster. And then configure roles to be used by the operator to interact with Refactor Spaces.

  1. Enable IAM OIDC provider in your EKS cluster, if not done already.
eksctl utils associate-iam-oidc-provider --cluster <cluster-name> --approve
  1. Create a role with name “refactorspaces-role-cluster” with the following managed policies –
  • AWSMigrationHubRefactorSpacesFullAccess
  • AWSResourceAccessManagerFullAccess

Edit the Trust Relationship of the role as follows.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::{ACCOUNTID}:oidc-provider/oidc.eks.{REGION}.amazonaws.com/id/{OIDC_ID}"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.{REGION}.amazonaws.com/id/{OIDC_ID}:sub": "system:serviceaccount:refactorspaces-system:refactorspaces-account",
                    "oidc.eks.{REGION}.amazonaws.com/id/{OIDC_ID}:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

Replace {ACCOUNTID} and {REGION} with your own account ID and Region code. You can retrieve the OIDC ID with the following command and replace {OIDC_ID} placeholder:

aws eks describe-cluster –name {EKS-CLUSTER-NAME} –query “cluster.identity.oidc.issuer” –output text

Use the output value that appears after /id/.

https://oidc.eks.us-east-1.amazonaws.com/id/XXXXXXXXXXXXXXXXX

  1. Clone the github repository
git clone https://github.com/aws-samples/refactorspaces-operator
cd refactorspaces-operator/helm
  1. Edit values.yaml and replace the “<accountID>” with your account ID.
Figure 1. Sample values.yaml file

Figure 1. Sample values.yaml file

  1. Log in to the EKS cluster and install the helm chart
aws configure
aws eks --region <region name> update-kubeconfig --name <cluster-name>
helm install rs-operator .
  1. Verify that rs-operator helm chart is installed property.
helm list

Output looks like:

Figure 2. Output of helm list command

Figure 2. Output of helm list command

Refactor Spaces EKS Operator (rs-operator) uses two Custom Resource Definition (CRDs) – a) RefactorSpacesConfig and b) RefactorSpacesService. Issue the following command to verify if both the CRDs are created:

kubectl get crd

Output looks like:

Figure 3. Output of command - kubectl get crd

Figure 3. Output of command – kubectl get crd

Check the status of deployment with the following command:

kubectl get deploy -n refactorspaces-system

You should find the deployment refactorspaces-operator in ready state.

Figure 4. Output of command - kubectl get deploy

Figure 4. Output of command – kubectl get deploy

  1. Now, create a namespace and a RefactorSpacesConfig object in that namespace.

Create banking-app-config.yaml file with the following content:

apiVersion: v1
kind: Namespace
metadata:
  name: refactorspaces-config-ns

---
apiVersion: eks.amazonaws.com/v1
kind: RefactorSpacesConfig
metadata:
  name: banking-app-config
  namespace: refactorspaces-config-ns
spec:
  environmentIdentifier: <ENVID>
  applicationIdentifier:  <APPID>
  vpcId: <VCPID>
  reconciliation: Enabled

Replace the placeholders ENVID, APPID, and VPCID in the manifest with the values from Refactor Spaces:

Figure 5. AWS Migration Hub Refactor Spaces environment detail page

Figure 5. AWS Migration Hub Refactor Spaces environment detail page

Figure 6. AWS Migration Hub Refactor Spaces application detail page

Figure 6. AWS Migration Hub Refactor Spaces application detail page

Execute the following command to create the object.
kubectl create -f banking-app-config.yaml

On success, you will see the following output:

Figure 7. Output of kubectl create command

Figure 7. Output of kubectl create command

  1. Now, deploy a microservice. For the demonstration purpose, lets create an NGINX deployment and expose the deployment as LoadBalancer service.

Create microservices.yaml with the following content:

---
apiVersion: v1
kind: Namespace
metadata:
  name: micro

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx-app
  name: nginx-app
  namespace: micro
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-app
  template:
    metadata:
      labels:
        app: nginx-app
    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: nginx
        ports:
        - containerPort: 80
          protocol: TCP
        resources: {}

---

apiVersion: v1
kind: Service
metadata:
  name: nginx-loadbalancer
  namespace: micro
  labels:
     app: nginx-loadbalancer
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx-app
  sessionAffinity: None
  type: LoadBalancer

Create the deployment and service objects with the following command:

kubectl create -f microservices.yaml

Check if the deployment and service has been created:

kubectl get svc,deploy -n micro -o wide

You should see the following output.

Figure 8. Output of kubectl command

Figure 8. Output of kubectl command

  1. Now, the final magic step. We will create a RefactorSpacesService manifest file with the following content, and name it nginx-rss.yaml.
apiVersion: eks.amazonaws.com/v1
kind: RefactorSpacesService
metadata:
  name: nginx-another-rss
  namespace: micro
spec:
  route: /someAnotherPath.jsp
  prefix: 
  healthCheckPrefix: 
  protocol: HTTP
  method: GET
  serviceSelector:
     matchLabels:
        app: nginx-loadbalancer
  configurationName: refactorspaces-config-ns.banking-app-config

In this spec, we have mentioned to look for NodePort or LoadBalancer service or Ingress in the namespace ‘micro’ with label “app: nginx-loadbalancer”. Once a match is found, get the endpoint URL of the service and create entries in the Refactor Spaces.  Notice that we are referring to the RefactorSpaceConfig object with the statement “configurationName: refactorspaces-config-ns.banking-app-config”, that we created in the earlier step.

Create the RefactorSpacesService object with following command:

kubectl create -f nginx-another-rss.yaml

Once the object is created, check the status:

kubectl get rss/nginx-rss -n micro -o yaml

You will see the following output:

apiVersion: eks.amazonaws.com/v1
kind: RefactorSpaceService
metadata:
  annotations:
    kopf.zalando.org/last-handled-configuration: |
      {"spec":{"configurationName":"refactorspace-config-ns.banking-app-config","method":"GET","protocol":"HTTP","route":"/somepath.jsp","serviceSelector":{"matchLabels":{"app":"nginx-loadbalancer"}}}}
  creationTimestamp: "2022-02-08T16:10:50Z"
  finalizers:
  - refactorspace-operator.eks.amazonaws.com/finalizer
  generation: 2
  name: nginx-rss
  namespace: micro
  resourceVersion: "7776160"
  uid: 3c98569b-84ed-4d05-9659-04b8798ee475
spec:
  configurationName: refactorspace-config-ns.banking-app-config
  method: GET
  protocol: HTTP
  route: /somepath.jsp
  serviceSelector:
    matchLabels:
      app: nginx-loadbalancer
status:
  create_fn:
    appId: app-BB2YAvAfP4
    endpoint: http://XXXXXXXXXXXXXXXXXXX-1790482514.us-east-1.elb.amazonaws.com
    envId: env-k4pEXTyttB
    k8sService: nginx-loadbalancer
    route: /somepath.jsp
    serviceId: svc-Iz67tWa2wk
    state: SUCCESS
    vpcId: vpc-0990928a0015c2e9b

Notice the status section of the object.

Now, on the AWS Management Console navigate to Refactor Spaces. You will be able to see that the services and routes are configured:

Figure 9. Migration Hub Refactor Spaces page for routes and services

Figure 9. Migration Hub Refactor Spaces page for routes and services

Note:  The first service and route you create for your Refactor Spaces Application will show the route with a “Source Path” as “/” instead of “/somepath.jsp.” This is because an application must have a default route before any other routes are created.

  1. Now, lets create another RefactorSpacesService object. Copy the following content, and save in a file named nginx-another-rss.yaml.
apiVersion: eks.amazonaws.com/v1
kind: RefactorSpacesService
metadata:
  name: nginx-another-rss
  namespace: micro
spec:
  route: /someAnotherPath.jsp
  prefix: 
  healthCheckPrefix: 
  protocol: HTTP
  method: GET
  serviceSelector:
     matchLabels:
        app: nginx-loadbalancer
  configurationName: refactorspaces-config-ns.banking-app-config

Create the RefactorSpacesService object with following command:

kubectl create -f nginx-another-rss.yaml

On the AWS Management Console navigate to AWS Migration Hub, and then select AWS Refactor Spaces. You will be able to see that another service and route are configured:

Figure 10. Migration Hub Refactor Spaces page for routes and services

Figure 10. Migration Hub Refactor Spaces page for routes and services

  1. Now, delete the RefactorSpacesService objects.
kubectl delete rss/nginx-rss -n micro
kubectl delete rss/nginx-another-rss -n micro
  1. Once the objects are deleted in EKS, the corresponding services and routes are also deleted in Refactor Spaces. On the Refactor Spaces AWS Management Console, you will see that the configurations no longer exist.

Cleanup:

If you are not using the solution, remember to uninstall the operator using following helm uninstall command.

kubectl delete service/nginx-loadbalancer -n micro
kubectl delete deployment.apps/nginx-app -n micro
kubectl delete rsc/banking-app-config -n refactorspaces-config-ns
kubectl delete ns refactorspaces-config-ns
kubectl delete ns micro
helm uninstall rs-operator

Conclusion

In this blog, we demonstrated how to auto discover the endpoint url of microservices deployed in Amazon Kubernetes Services (EKS) and dynamically register those Routes with Refactor Spaces.  This helps you to continue refactoring your application while it is in use and take further advantage of the microservices you’ve created.

Author:

Niladri Sekhar Adhikari

Niladri Sekhar Adhikari is a Solutions Architect with the Global System Integrator (GSI) team at Amazon Web Services. He is passionate about solving customer challenges with innovative solutions. Niladri lives in Bangalore, India with his wife, son and daughter.