Kubernetes RBAC and IAM Integration in Amazon EKS using a Java-based Kubernetes Operator

Introduction

A Kubernetes native application is one that is deployed on a Kubernetes cluster and managed both using Kubernetes APIs as well as client-side tools such as kubectl. A Kubernetes Operator is an abstraction for deploying non-trivial Kubernetes applications such as an etcd database cluster or a Prometheus monitoring/alerting system. It provides a mechanism to extend Kubernetes functionality using custom resources and controllers with domain-specific knowledge needed by such applications.

Developing custom controllers requires low-level APIs to interact with the Kubernetes controller-runtime. The GoLang developer community has long benefited from a rich set of frameworks and stable SDKs. Though Java is one of the most popular programming languages in the world, such library resources were not available to the Java developer community. Recently, the official Kubernetes Java SDK project announced that many of the salient features of the GoLang SDK have been ported over to Java in order to create a controller-builder Java SDK that is helpful for developing custom controllers.

In this blog post, we will walk through the details of implementing a Kubernetes Operator that leverages new features in Kubernetes Java SDK to help manage the integration of Kubernetes role-based access control (RBAC) and AWS Identity and Access Management (IAM) in a Kubernetes cluster provisioned using Amazon Elastic Kubernetes Service (Amazon EKS).

Authentication and authorization in Amazon EKS

Amazon EKS uses IAM to provide authentication to your Kubernetes cluster. When you use kubectl to interact with Amazon EKS under the hood it uses the aws eks get-token command, available in version 1.18.49 or later of the AWS CLI, or the AWS IAM Authenticator for Kubernetes to fetch an authentication token, which is passed along in the Authorization header of an HTTP request sent to the Kubernetes API server. By default, it will generate this token using the IAM credentials that are returned with the aws sts get-caller-identity command. Amazon EKS uses a token authentication webhook to authenticate the request but it still relies on native Kubernetes RBAC for authorization. The set of permissions granted by IAM policies associated with an authenticated IAM principal has no bearing whatsoever on what the client can or cannot do in an Amazon EKS cluster. The crux of this integration between IAM and RBAC is the aws-auth ConfigMap applied to the Amazon EKS cluster that provides the mappings between IAM principals (roles/users) and Kubernetes Subjects (Users/Groups). The latter in turn is associated with Kubernetes Roles/ClusterRoles and RoleBindings/ClusterRoleBindings that control the access granted to the client.

When an AWS account administrator modifies the attributes of an IAM user by adding/removing the user to/from one or more IAM groups, it will be desirable if those actions could automatically trigger corresponding updates to the aws-auth ConfigMap in an Amazon EKS cluster, thereby controlling the level of access granted to that user in the Amazon EKS cluster. The Kubernetes Operator is an ideal pattern for designing such automation.

Authentication and Authorization of a client in an Amazon EKS cluster

Architecture

The architecture used to implement this automation of integration between Kubernetes RBAC and IAM comprises the following key elements.

An Operator implemented using Kubernetes Java SDK. This operator packages a custom resource named IamUserGroup defined by a CustomResourceDefinition, a custom controller implemented as a Deployment, which responds to events in the Kubernetes cluster pertaining to add/update/delete actions on the IamUserGroup custom resource, Role/RoleBinding resources that allow the custom controller to make changes to the aws-auth ConfigMap.
A Kubernetes Java client implemented as an AWS Lambda function whose execution is triggered whenever an IAM user is added or removed from an IAM group. This is made possible using Amazon EventBridge, which is a serverless event bus service that makes it easy to deliver a stream of real-time data from the IAM service and route that data to targets such as AWS Lambda.
Role/RoleBinding resources that control the access granted to Kubernetes Subjects that an IAM group is mapped to.

Kubernetes RBAC and IAM integration with a custom controller

Custom controller implementation

Creating a custom controller using the Java SDK entails providing an implementation of io.kubernetes.client.informer.SharedInformer interface. A SharedInformer is responsible for informing clients such as a controller about actions taken by the Kubernetes API server – actions such as creating/updating/deleting cluster resources. A separate instance of a SharedInformer is required for each cluster resource such as a Pod, ConfigMap etc. whose lifecycle events we want our custom controller to watch. Each custom controller is represented by an implementation of io.kubernetes.client.extended.controller.Controller interface. The SDK provides a default implementation class, namely, DefaultController, which would suffice for most use cases and it contains the necessary plumbing to respond to cluster events. Next, an implementation of the io.kubernetes.client.extended.controller.reconciler.Reconciler interface is associated with each custom controller. A reconciler works by comparing the state specified in an object by a user against the actual cluster state, and then perform operations to make the actual cluster state reflect the state specified by the user. Finally, an implementation of io.kubernetes.client.extended.controller.builder.ControllerWatch interface is required for each custom controller. It is responsible for filtering event notifications pertaining to a Kubernetes resource sent from the Informer framework and defining what a custom controller responds to. These classes make up the core of the custom controller-builder framework in the Java SDK.

If you develop your Kubernetes application using Spring Framework, then you can leverage the Spring-integration components in the SDK to wire up the aforementioned classes using declarative semantics with Spring annotations. The code below shows a SharedInformerFactory class, which is declared as a Spring bean using @KubernetesInformer annotation. A Spring BeanDefinitionRegistryPostProcessor in the SDK will process such beans and instantiate a SharedInformer for each @KubernetesInformer annotation and register that as a Spring bean.

@KubernetesInformers({
    @KubernetesInformer(
            apiTypeClass = IamUserGroupCustomObject.class, 
            apiListTypeClass = IamUserGroupCustomObjectList.class, 
            groupVersionResource = @GroupVersionResource(
                    apiGroup = "octank.com", 
                    apiVersion = "v1", 
                    resourcePlural = "iamusergroups"), 
            resyncPeriodMillis = 60    * 1000L)
    })
public class SpringSharedInformerFactory extends SharedInformerFactory {
    public SpringSharedInformerFactory (ApiClient apiClient) {
        super (apiClient);
    }
}

Next, we provide an implementation of a Reconciler and declare it as a Spring bean using the @KubernetesReconciler annotation. A Spring BeanFactoryPostProcessor in the SDK will process these beans and create a Controller for each Reconciler. Each Reconciler should provide a set of methods annotated with @AddWatchEventFilter, @UpdateWatchEventFilter and @DeleteWatchEventFilter which return true/false depending on whether the Controller has to reconcile the respective lifecycle event of the custom resource. For each @KubernetesReconcilerWatch annotation, a ControllerWatch instance is created and is assigned the aforementioned methods as handlers that help filter add/update/delete event notifications pertaining to a Kubernetes resource. The code below shows the skeleton of the Reconciler bean (download the complete source code from the Git repository for details).

@KubernetesReconciler(
        value = "iamUserGroupController", 
        workerCount = 2,
        watches = @KubernetesReconcilerWatches({
            @KubernetesReconcilerWatch(
                    apiTypeClass = IamUserGroupCustomObject.class, 
                    resyncPeriodMillis = 60*1000L)
            }))
public class IamUserGroupReconciler implements Reconciler {
    private GenericKubernetesApi<V1ConfigMap, V1ConfigMapList> apiConfigMap;
    private SharedInformer<IamUserGroupCustomObject> iamUserGroupInformer;
    public IamUserGroupReconciler(ApiClient apiClient, SharedInformer<IamUserGroupCustomObject> iamGroupInformer) {
        this.iamUserGroupInformer = iamGroupInformer;
        
        this.apiConfigMap = new GenericKubernetesApi<V1ConfigMap, V1ConfigMapList>(
                V1ConfigMap.class, 
                V1ConfigMapList.class,
                "", 
                "v1", 
                "configmaps", 
                apiClient);
    }

    @AddWatchEventFilter(apiTypeClass = IamUserGroupCustomObject.class)
    public boolean onAddFilter(IamUserGroupCustomObject iamUserGroup) {
    }

    @UpdateWatchEventFilter(apiTypeClass = IamUserGroupCustomObject.class)
    public boolean onUpdateFilter(IamUserGroupCustomObject oldIamUserGroup, IamUserGroupCustomObject newIamUserGroup) {
    }

    @DeleteWatchEventFilter(apiTypeClass = IamUserGroupCustomObject.class)
    public boolean onDeleteFilter(IamUserGroupCustomObject iamUserGroup, boolean deletedFinalStateUnknown) {
    }

    @Override
    public Result reconcile(Request request) {
    }
}

Next, we need an implementation of the custom resource. We start by defining a CustomResourceDefinition using the YAML manifest shown below. The CRD defines the schema for custom resource named IamUserGroup.

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: iamusergroups.octank.com
spec:
  group: octank.com
  version: v1
  versions:
    - name: v1
      served: true
      storage: true
  scope: Namespaced    
  names:
    kind: IamUserGroup
    plural: iamusergroups
    singular: iamusergroup
    shortNames:
    - ig
  preserveUnknownFields: false
  validation:
    openAPIV3Schema:
      type: object
      properties:
        spec:
          type: object
          properties:
            iamUser:
              type: string
            iamGroups:
              type: array
              items:
                type: string
            username:
              type: string

To deploy an instance of IamUserGroup custom resource, a YAML manifest such as the one shown below is used. The manifest provides the association between an IAM principal’s username, Amazon Resource Name (ARN) and as list of IAM groups.

---
apiVersion: octank.com/v1
kind: IamUserGroup
metadata:
  namespace: kube-system
  name: viji-developers
spec:
  iamUser: 'arn:aws:iam::937351234567:user/viji'
  iamGroups: 
    - developers
  username: viji

The custom controller manages the IamUserGroup custom resource using a set of Java classes, namely, IamUserGroupCustomObject, IamUserGroupCustomObjectList, and IamUserGroupCustomObjectSpec. IamUserGroupCustomObject is an object representation of the entire manifest; IamUserGroupCustomObjectSpec corresponds to the spec section of the manifest and IamUserGroupCustomObjectList represents a collection of IamUserGroup resources. The reconcile method of the Reconciler uses these classes to retrieve the iamUser, iamGroups, and username attributes of an IamUserGroup resource and make relevant changes to the aws-auth ConfigMap depending on whether the said IamUserGroup resource was added, updated, or deleted.

The final set of artifacts in the operator package is a Kubernetes ServiceAccount, ClusterRole, and ClusterRoleBinding definition shown below. The custom controller runs under the identity of the service account named iamusergroup, which is granted permissions to execute all actions against the IamUserGroup and ConfigMap resources.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: iamusergroup
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: iamusergroup-role
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/name: iamusergroup-operator
rules:
- apiGroups:
  - octank.com
  resources:
  - iamusergroups
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - '*'

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: iamusergroup-rolebinding
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/name: iamusergroup-operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: iamusergroup-role
subjects:
- kind: ServiceAccount
  name: iamusergroup
  namespace: kube-system

Another set of Role/RoleBinding definitions is required to control the access granted to an authenticated user. These RBAC resources are not part of the operator package per se and these definitions depend on how one wants to manage Kubernetes subjects and their access in the target environment. For demonstration purposes, in this implementation, the IAM groups listed in an IamUserGroup custom resource are considered to have a one-to-one association with a Kubernetes group. Thus, in the YAML file show below, the IAM group developers is associated with the Kubernetes group developers and the latter is bound to the Role named developers-role, which grants full access to all Kubernetes resources in the dev namespace.

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: developers-role
  namespace: dev
rules:
  - apiGroups:
      - ""
      - apps
      - batch
      - extensions
      - rbac.authorization.k8s.io
    resources: ["*"]
    verbs:
      - get
      - list
      - watch
      - create
      - update
      - patch
      - delete
---  
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: developers-role-binding
  namespace: dev
roleRef:
  kind: Role
  name: developers-role
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: Group
    name: developers
    apiGroup: rbac.authorization.k8s.io

Kubernetes client using AWS Lambda

The Lambda function is designed to handle event notifications sent from the IAM service when an IAM user is added to or deleted from an IAM group. These notifications are sent in the form of JSON data shown below.

{
   "version":"0",
   "id":"5628e467-803a-8ffb-7194-e2b6f9831549",
   "detail-type":"AWS API Call via CloudTrail",
   "source":"aws.iam",
   "account":"937351234567",
   "time":"2020-05-15T15:47:12Z",
   "region":"us-east-1",
   "resources":[
   ],
   "detail":{
      "eventVersion":"1.05",
      "userIdentity":{
      },
      "eventTime":"2020-05-15T15:47:12Z",
      "eventSource":"iam.amazonaws.com",
      "eventName":"AddUserToGroup",
      "awsRegion":"us-east-1",
      "sourceIPAddress":"72.21.196.64",
      "userAgent":"console.amazonaws.com",
      "requestParameters":{
         "groupName":"developers",
         "userName":"viji"
      },
      "responseElements":null,
      "requestID":"02814799-fb64-4f01-96f4-a674ebdc9119",
      "eventID":"22648995-6e51-4edc-acc7-2d719df264d8",
      "eventType":"AwsApiCall"
   }
}

The eventSource field in the data is be set to iam.amazonaws.com. The eventName field indicates which action was executed. AddUserToGroup and RemoveUserFromGroup are the two events that we are interested in. The requestParameters field contains the names of the IAM user and group associated with the action. Given this data, the Lambda function creates an instance of IamGroupCustomObject class and invokes the Kubernetes API to either create or delete an instance of IamUserGroup custom resource.

The Lambda function runtime environment will not be able to execute either the aws eks get-token or aws-iam-authenticator token commands to get the authentication token required for communicating with the Kubernetes API server. Hence, this token will have to be constructed programmatically. The documentation for AWS IAM Authenticator for Kubernetes provides details about how this token is constructed under the section titled API Authorization from Outside a Cluster. The token is generated with the AWS Signature Version 4 algorithm using the helper classes provided under Signature Calculation Examples Using Java. The authentication token is a base64 encoded version of an HTTP URL constructed as shown below:

https://sts.us-east-1.amazonaws.com/? 
Action=GetCallerIdentity& 
Version=2011-06-15& 
X-Amz-Algorithm=AWS4-HMAC-SHA256& 
X-Amz-Credential=ASIA12345NGDGUPABCDEF/20200517/us-east-1/sts/aws4_request& 
X-Amz-Date=20200517T131023Z& 
X-Amz-Expires=60& 
X-Amz-Security-Token=IQoJb3JpZ2luX2VjEHYaCXVzLWVhc3QtMSJHMEUCIQCSdHOdiAh4c1+FiOM/diA8NNkFSMnwORgyfO68rdxdgAIgAXEVzjsu9H5WDvbqkihQ94Ugw5MzczNTE5MzA5NzUiDGon8DFp20Zw0pgiayqzASBP4b/bAa0gZtoP8U4bM+gFiH5xpVtj18HyVEy5uGm1xgCpb3P/z0/lFmWm3bQbqwOeA809NyDWfaYD79FgqydT4lD3H0AQg4IsvLx/qQSSK0pXwoMYe82xhbG8yQQzW7x5flYqiP80xe1R1HPrlmAVfhCnCyKifghmQujucJoRfW4wcC0wGwetSXDpu1fCsCfZkMO2CD2dgDdnliz07kFig0A9EVMawCzoOnQP9jYRhfxWML7vhPYFOuMBlpzWN219LlZo982Q/4o/lSE5USnCCbAsKd59JkzjrH2G5HEp1EGs4rlRrZNYHwFMVeW3FbED8+crQcOz2F95AN9D8gmaFHn4my68Vz/b8gW7F2C11q7wNk18at0/EGrKl4ty18vMjmB/qxUk2xvO5YDLxcnwThXS3AL5HL1eT15xg0wLIz1OzZ0n4Cd94iLR/U29IgY5VlV3G03RjdP8+uvv+OrHheL96CsCDY4MQ+55J7wbNWIPLWjqFtGFZgNUIslifZv4bAX5F8dj2cQAidMxyK2EXi+aAgmP/B4WtSM38OQ=& 
X-Amz-SignedHeaders=host;x-k8s-aws-id& 
X-Amz-Signature=fd5b5f2f8a3d0c7b128ca785d9fb71f3b583741c8648f036e20d80d2c0450503

In order for the Lambda function to be able create or delete IamUserGroup custom resources, the following configuration settings are required:

The Amazon Resource Name (ARN) of an IAM role, say, K8s-Lambda-Client-Role, which has been mapped to a Kubernetes group, say, lambda-clients, in the mapRoles section of the aws-auth ConfigMap in the Amazon EKS cluster. The temporary credentials of this IAM role will be used to generate the authentication token.
The credentials (access key, secret access key) of an IAM user which, adhering to the least privilege security guidelines, is granted only the permission to assume the said IAM role.
A Role and RoleBinding definition in the Amazon EKS cluster that grants the lambda-clients Kubernetes group permission to add/update/delete IamUserGroup custom resources.
API server endpoint URL and certificate authority data for the Amazon EKS cluster, which are both retrieved using the aws eks update-kubeconfig command.

The code below shows the skeleton of the Java handler for the Lambda function (download the complete source code from the Git repository for details).

public class IAMEventHandler implements RequestStreamHandler {
    private GenericKubernetesApi<IamUserGroupCustomObject, IamUserGroupCustomObjectList> apiIamGroupClient = // Initialize Kubernetes API client
    public void handleRequest(InputStream inputStream, OutputStream outputStream, Context context) throws IOException {
        String inputString = // Retrieve from input stream
        JsonObject inputObject = new JsonObject (inputString);

        String account = inputObject.getString("account");
        String eventName = inputObject.getJsonObject("detail").getString("eventName");
        String eventSource = inputObject.getJsonObject("detail").getString("eventSource");
        String groupName = inputObject.getJsonObject("detail").getJsonObject("requestParameters").getString("groupName");
        String userName = inputObject.getJsonObject("detail").getJsonObject("requestParameters").getString("userName");
        String userArn = String.format("arn:aws:iam::%s:user/%s", account, userName);
        
        String objName = userName.concat("-").concat(groupName).toLowerCase();
        String objNamespace = "kube-system";
        
        IamUserGroupCustomObject iamUserGroup =
                new IamUserGroupCustomObject()
                .apiVersion("octank.com/v1")
                .kind("IamUserGroup")
                .metadata(new V1ObjectMeta()
                        .name(objName)
                        .namespace(objNamespace))
                .spec(new IamUserGroupCustomObjectSpec()
                        .iamUser(userArn)
                        .username(userName)
                        .group(groupName));
                        
        if (eventName.equals(ADD_USER_TO_GROUP)) {
            KubernetesApiResponse<IamUserGroupCustomObject> createResponse = apiIamGroupClient.create(iamUserGroup);
        }
        else if (eventName.equals(REMOVE_USER_FROM_GROUP)) {
            KubernetesApiResponse<IamUserGroupCustomObject> createResponse = apiIamGroupClient.delete(objNamespace, objName);
        }
    }
}

Testing the operator in an Amazon EKS cluster

Now, let’s try this Kubernetes operator out in an Amazon EKS cluster. The following YAML manifest named operator.yaml defines all the artifacts needed to deploy this operator to a Kubernetes cluster.

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: iamusergroups.octank.com
spec:
  group: octank.com
  version: v1
  versions:
    - name: v1
      served: true
      storage: true
  scope: Namespaced    
  names:
    kind: IamUserGroup
    plural: iamusergroups
    singular: iamusergroup
    shortNames:
    - ig
  preserveUnknownFields: false
  validation:
    openAPIV3Schema:
      type: object
      properties:
        spec:
          type: object
          properties:
            iamUser:
              type: string
            iamGroups:
              type: array
              items:
                type: string
            username:
              type: string

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: iamusergroup
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: iamusergroup-role
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/name: iamusergroup-operator
rules:
- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - '*'  
- apiGroups:
  - octank.com
  resources:
  - iamusergroups
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - '*'

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: iamusergroup-rolebinding
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/name: iamusergroup-operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: iamusergroup-role
subjects:
- kind: ServiceAccount
  name: iamusergroup
  namespace: kube-system
  
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: iamusergroup
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: iamusergroup
      role: operator
  template:
    metadata:
      labels:
        app: iamusergroup
        role: operator
      annotations:
        prometheus.io/scrape: 'false'     
    spec: 
      serviceAccountName: iamusergroup
      containers:          
        - name: java  
          image: eksworkshop/k8s-iam-operator:latest
          imagePullPolicy: Always   
          ports:
            - containerPort: 8080
              name: http 
              protocol: TCP
          resources:
            requests:
              cpu: "100m"
              memory: "256Mi"
            limits:
              cpu: "500m" 
              memory: "1000Mi"
          livenessProbe:
            httpGet: 
              path: /live
              port: 8080
            initialDelaySeconds: 15
            timeoutSeconds: 1
            periodSeconds: 10
            failureThreshold: 3            
          readinessProbe:
            httpGet: 
              path: /ready
              port: 8080
            initialDelaySeconds: 15
            timeoutSeconds: 1
            periodSeconds: 10
            failureThreshold: 3        
---
apiVersion: v1
kind: Service
metadata:
  name: iamusergroup-svc
  namespace: kube-system
spec:
  sessionAffinity: None
  type: ClusterIP  
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: iamusergroup
    role: operator

Deploy this to an Amazon EKS cluster using kubectl apply -f operator.yaml. The following is the output from this command.

customresourcedefinition.apiextensions.k8s.io/iamusergroups.octank.com created 
serviceaccount/iamusergroup created 
clusterrole.rbac.authorization.k8s.io/iamusergroup-role created 
clusterrolebinding.rbac.authorization.k8s.io/iamusergroup-rolebinding created 
deployment.apps/iamusergroup created service/iamusergroup-svc created

The initial state of aws-auth ConfigMap in the cluster contains the mapping that allows worker nodes to join the Amazon EKS cluster. Modify this ConfigMap by applying the changes in the YAML manifest shown below. This will create a mapping that associates clients with the lambda-clients Kubernetes group if they were authenticated using temporary credentials that belong to the role K8s-Lambda-Client-Role.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::937351234567:role/eks-worker-stack-WorkerNodeInstanceRole-1FET9MJ4MU56
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
    - rolearn: arn:aws:iam::937351234567:role/K8s-Lambda-Client-Role 
      username: lambda-client 
      groups: 
        - lambda-clients

The Kubernetes group lambda-clients is granted full access only to IamUserGroup custom resource within the kube-system namespace by creating the Role/RoleBinding definitions shown in the YAML manifest below. Thus, by using the role K8s-Lambda-Client-Role for authenticating the Lambda client to the Amazon EKS cluster, we limit the scope of its actions within the cluster.

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: lambda-clients-role
  namespace: kube-system
rules:
  - apiGroups:
    - octank.com
    resources:
    - iamusergroups
    verbs:
    - '*'
    
---  
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: lambda-clients-rolebinding
  namespace: kube-system
roleRef:
  kind: Role
  name: lambda-clients-role
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: Group
    name: lambda-clients
    apiGroup: rbac.authorization.k8s.io

The Lambda client for Amazon EKS is deployed using the command aws lambda create-function with the following JSON file as input:

{
    "FunctionName": "K8sClientForIAMEvents",
    "Runtime": "java8",
    "Role": "arn:aws:iam::937351234567:role/Lambda-Execution-Role",
    "Handler": "com.octank.IAMEventHandler::handleRequest",
    "Description": "K8s Client to Process IAM Notifications",
    "Timeout": 30,
    "MemorySize": 512,
    "Code": {
        "S3Bucket": "sarathy-lambda-handlers",
        "S3Key": "eksLambda.jar"
    },    
    "Environment": {
        "Variables": {
            "REGION": "us-east-1",
            "STS_ENDPOINT": "sts.us-east-1.amazonaws.com",
            "ACCESS_KEY_ID": "AKIATRU5TN0127H5VTXM",
            "SECRET_ACCESS_KEY": "sIbm2UbhXhusmk8sDjy-+Gens85n6ym///rQySTMb7Aw",
            "ASSUMED_ROLE": "arn:aws:iam::937351234567:role/K8s-Lambda-Client-Role",
            "CLUSTER_NAME": "k8s-sarathy-cluster",
            "API_SERVER": "https://9E700EEF26B4378A9109685E2C99D393.gr7.us-east-1.eks.amazonaws.com",
            "CERT_AUTHORITY": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01EVXhPREF4TURNd05sb1hEVE13TURVeE5qQXhNRE13Tmxvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBT2IwCkQrT3ZESDRRYy9wcXRVR0FjRlVnMWUrNFhta0lBY0VUNzZXaEVWYVFRWDNLMzkwNUZ1Y1g4U2ppY3hQd3hUSEYKczBhR0puWEl0WDh3V2hrQU53Q0VkUnM5bGdaNTQxcEdKQ1JTdFVYbTR5UWNoVUg1Uk0xR2Fpelk0OVAyQ0RlVApuMzh2TXhDb2JSSWdacW5IaFFCeWNQY21hT3p6dnNjQXFlRCtveGYzLzFSVWhRdEdvZE5iS284KzdnbHhiVVhsCmFTS2VWamhBcWZMMzJTK3plUFlncFUzN0pjSlo5cEY0VTNoYUlDN0ZPWUhvNlVWU1VUN1o2SjlvcFVnT1AyemgKRzBhVUVCaUpaYkdIRlluOVJ5aUJxc1lHRk9VZ1J2OEYwdWRsUnNXUWFkVVJrOVRmQitBMW15YnZVUzRTK09xbAo4cjlBcHhQZSthVUpTZSszL0RrQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFBVVhLeDFENkdHcnc4SGhrZ3VzNmFTTmlzcDgKQSt1TkhSS2J3aTRxTjR1SVBHR05rOTdXNjJwZkJvc09lMElYdjN1dlBXZDl4M2kwVG9qWVphZ2xTOVRUOWNycwpod0xhNlpNWmRpMkZFeThYRC8vdVpoRUszditMNlY0cXY1b2E1SWw3a3IrVDZpYUVOMzhZMllSaGJ0MDNVUUJVClV2ZXlUVDVKdENuZkFMaUJWRzBJQk0xUUkxRkpiRVJqOWpTVXFiWHpWUnNBUHJYTktSSkJxdktjRG5DK09OWkoKVkpCWTN2RDJVVGI2N1VtVXhaS2tKRHpiK1N5SEFkV3lLTzYvWCtyNEZWRHl2NjRyME1Kd0RQLzc3OGd1OGlrbgo0MzhDUVFSQmIyMUN6ZFhkOEw0MDBpbzNYVUNJQTNvajZ2aWViRXR0VmVYZUFSTTV5aG1yQTBOejl3dz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="
        }
    }
}

Next, an EventBridge rule is created and the Lambda function is assigned as its target with the following set of commands:

EVENT_RULE_ARN=$(aws events put-rule --name IAMUserGroupRule --event-pattern "{\"source\":[\"aws.iam\"]}" --query RuleArn --output text)

aws lambda add-permission \
--function-name K8sClientForIAMEvents \
--statement-id 'd6f44629-efc0-4f38-96db-d75ba7d06579' \
--action 'lambda:InvokeFunction' \
--principal events.amazonaws.com \
--source-arn $EVENT_RULE_ARN

aws events put-targets --rule IAMUserGroupRule --targets file://lambdaTarget.json

Now, let’s go to the IAM dashboard and edit the user named viji and add that user to the groups named developers and testers. Note that this will trigger two separate event notifications to the Lambda function as seen from the CloudWatch Logs because IAM AddUserToGroup API supports adding only one group to a user at a time though the UI allows you to select more than one.

Managing user-group associations in the AWS IAM dashboard

CloudWatch logs for AWS Lambda handling event notifications from AWS IAM

Running the kubectl get configmap aws-auth -n kube-system -o yaml command gives the following output, confirming the changes made to the aws-auth ConfigMap that are consistent with the actions executed in the IAM dashboard.

apiVersion: v1
data:
  mapRoles: |
    - rolearn: arn:aws:iam::937351234567:role/eks-worker-stack-WorkerNodeInstanceRole-134DR0KSSN2K
      username: system:node:{{EC2PrivateDNSName}}
      groups: ['system:bootstrappers', 'system:nodes']
    - rolearn: arn:aws:iam::937351234567:role/K8s-Lambda-Client-Role
      username: lambda-client
      groups: [lambda-clients]
  mapUsers: |
    - userarn: arn:aws:iam::937351234567:user/viji
      groups: [testers, developers]
      username: viji
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system

Now, let’s remove the user viji from the developers group using the command aws iam remove-user-from-group –group-name developers –user-name viji. The output from the kubectl get configmap aws-auth -n kube-system -o yaml command now looks as follows:

apiVersion: v1
data:
  mapRoles: |
    - rolearn: arn:aws:iam::937351234567:role/eks-worker-stack-WorkerNodeInstanceRole-134DR0KSSN2K
      username: system:node:{{EC2PrivateDNSName}}
      groups: ['system:bootstrappers', 'system:nodes']
    - rolearn: arn:aws:iam::937351234567:role/K8s-Lambda-Client-Role
      username: lambda-client
      groups: [lambda-clients]
  mapUsers: |
    - userarn: arn:aws:iam::937351234567:user/viji
      groups: [testers]
      username: viji
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system

Note that the adding/removing an IAM user to/from an IAM group may be done using kubectl command as well by creating appropriate YAML manifests that define an IamUserGroup custom resource. These events are handled in exactly the same manner by the custom controller.

Source code

The complete source code for the custom controller and the Lambda function can be downloaded from the following links:

https://github.com/aws-samples/k8s-rbac-iam-java-operator/tree/master/java-operator

https://github.com/aws-samples/k8s-rbac-iam-java-operator/tree/master/lambda-client

Concluding remarks

An operator is a great way of building non-trivial applications on top of Kubernetes that encapsulate domain-specific logic. The Kubernetes controller-builder Java SDK enables developers to write custom controllers and easily wire up relevant components to the Kubernetes controller runtime. Organizations that have adopted Kubernetes for orchestrating their container workloads and have a lot of in-house expertise in Java can now leverage the Kubernetes Java SDK to build custom tooling in Java over the Kubernetes API.

Containers