Fine-grained IAM roles for Red Hat OpenShift Service on AWS (ROSA) workloads with STS

Red Hat OpenShift Service on AWS (ROSA) is a fully managed OpenShift service, jointly supported by both Red Hat and Amazon Web Services (AWS) and managed by the Red Hat SRE team. This relieves customers of cluster lifecycle management, allowing them to focus on building applications rather than maintaining the OpenShift clusters.

ROSA has recently been integrated with the AWS Security Token Service (AWS STS). AWS STS is an AWS service that allows AWS users, authenticated via AWS Identity and Access Management (IAM) or Federation, to request temporary security credentials for your AWS resources. ROSA users can allocate administrative permissions on demand. The temporary security credentials work exactly like regular long-term security access key credentials allocated to IAM users, except the lifecycle of the access credentials is shorter. Additionally, the temporary credentials are not stored with the user and instead are generated dynamically and provided to the user on demand. When the temporary credentials expire, the user can simply request new ones.

IAM supports federated identities using an OpenID Connect (OIDC) identity provider. This feature allows customers to authenticate AWS API calls with supported identity providers and receive a valid OIDC JSON web token (JWT). PODs can request and pass this token to the AWS STS AssumeRoleWithWebIdentityAPI operation and receive temporary IAM role credentials.

Currently, you can create a ROSA cluster using the AWS STS service, using the rosa create cluster --sts option. This will be discussed in greater detail later. During cluster creation, the operator IAM roles and the OpenID Connect (OIDC) identity provider are created. The operator IAM roles and endpoint are mapped to OpenShift resources within the ROSA cluster and use the OIDC to authenticate.

In this blog post, we will discuss how to use the OIDC identity provider created during cluster installation and use it with IAM roles for service accounts (IRSA). IRSA allows us to associate an IAM role with a Kubernetes service account, which can then be used by pods for authentication and fine-grained permissions.

Using IRSA has the benefit of using the least privileged recommendation and credential isolations, meaning that the container within the pod can only retrieve credentials for the IAM role associated with the service account to which the pod belongs. We can also get better auditing, having access and event logging available through AWS CloudTrail.

Walkthrough

In the following section, we demonstrate how ROSA with STS enabled can use IAM roles for service accounts to provide access for a Kubernetes pod to an Amazon Simple Storage Service (Amazon S3) bucket. The guide will not walk through the creation of a ROSA cluster with STS enabled. For more detail on how to provision a ROSA cluster using STS, see the documentation.

Prerequisites

AWS account
The AWS, rosa, and oc CLI
ROSA enabled in the AWS account
A ROSA cluster with STS enabled

Step 1. Set working variables

We are going to reuse the existing OIDC endpoint created during the ROSA STS cluster deployment. To do this, we are going to create a set of variables for items that will be reused later, then connect to the OpenShift cluster using the oc CLI.

$ export ROSA_CLUSTER_NAME=<cluster-name>
$ export SERVICE="s3"
$ export AWS_REGION=<aws-region-id>
$ export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
$ export APP_NAMESPACE=iam-app
$ export APP_SERVICE_ACCOUNT_NAME=iam-app-$SERVICE-sa

Step 2. Get the OIDC endpoint URL of the cluster

We will now use the oc CLI to connect to the ROSA cluster and get the existing OIDC endpoint details. In the documentation we will find how to connect to the cluster using oc CLI in more detail. Once connected, we can collect the endpoint details.

$ export OIDC_PROVIDER=$(oc get authentication.config.openshift.io cluster -ojson | jq -r .spec.serviceAccountIssuer | sed 's/https:\/\///')

Step 3. Create the application namespace and service account

$ oc create new-project $APP_NAMESPACE
$ oc create serviceaccount $APP_SERVICE_ACCOUNT_NAME -n $APP_NAMESPACE

Step 4. Create an IAM role for the application service account

$ cat <<EOF > ./trust.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:${APP_NAMESPACE}:${APP_SERVICE_ACCOUNT_NAME}"
        }
      }
    }
  ]
}
EOF

$ export APP_IAM_ROLE="iam-app-${SERVICE}-role"
$ export APP_IAM_ROLE_ROLE_DESCRIPTION='IRSA role for APP $SERVICE deployment on ROSA cluster'
$ aws iam create-role --role-name "${APP_IAM_ROLE}" --assume-role-policy-document file://trust.json --description "${APP_IAM_ROLE_ROLE_DESCRIPTION}"
$ APP_IAM_ROLE_ARN=$(aws iam get-role --role-name=$APP_IAM_ROLE --query Role.Arn --output text)

Step 5. Attach IAM policy to the IAM role

Attach the required Amazon S3 policy and apply them to the newly created IRSA role.

Note: You can choose to attach to the user custom or specific policies required by your needs.

$ POLICY_ARN=$(aws iam list-policies --query 'Policies[?PolicyName==`AmazonS3FullAccess`].Arn' --output text)
$ aws iam attach-role-policy --role-name "${APP_IAM_ROLE}" --policy-arn "${POLICY_ARN}" > /dev/null

Step 6. Associate the IAM role created to the service account

For the moment, the Amazon Resource Name (ARN) of the IAM role created is not yet an annotation for the service account. Use the following command to associate an IAM role to the service account.

$ export IRSA_ROLE_ARN=eks.amazonaws.com/role-arn=$APP_IAM_ROLE_ARN
$ oc annotate serviceaccount -n $APP_NAMESPACE $APP_SERVICE_ACCOUNT_NAME $IRSA_ROLE_ARN

Let’s describe the service account

$ oc describe serviceaccount $APP_SERVICE_ACCOUNT_NAME -n $APP_NAMESPACE
 Name:                iam-app-s3-sa
 Namespace:           iam-app
 Labels:              <none>
 Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<APP_IAM_ROLE>
 Image pull secrets:  iam-app-s3-sa-dockercfg-wk7tz
 Mountable secrets:   iam-app-s3-sa-token-pqpkd
                      iam-app-s3-sa-dockercfg-wk7tz
 Tokens:              iam-app-s3-sa-token-pqpkd
                      iam-app-s3-sa-token-vngcg
 Events:              <none>

Step 7. Application deployment

Next, we will deploy a sample application onto the ROSA cluster. For simplicity, we will deploy a single container in a deployment called awscli. This container will have the AWS CLI binary preinstalled and will allow us to demonstrate accessing cloud services from ROSA using fine-grained IAM roles.

$ cat <<EOF | oc create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: ${APP_NAMESPACE}
  name: awscli
  labels:
    app: awscli
spec:
  replicas: 1
  selector:
    matchLabels:
      app: awscli
  template:
    metadata:
      labels:
        app: awscli
    spec:
      containers:
        - image: amazon/aws-cli:latest
          name: awscli
          command:
            - /bin/sh
            - "-c"
            - while true; do sleep 10; done
          env:
            - name: HOME
              value: /tmp
      serviceAccount: ${APP_SERVICE_ACCOUNT_NAME}
EOF

When AWS clients or SDKs connect to AWS APIs, they look for credentials in a variety of different places, such as an ~/.aws/config file or an AWS_ACCESS_KEY_ID environment variable. When using a Web Identity, the AWS clients and SDKs look for environment variables called AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN. For the Python Boto3 SDK, this is documented here.

The EKS Pod Identity Webhook running in the OpenShift cluster can place these variables within a running pod for us as long as a service account with an IAM role defined has been attached to your pod. For the demonstration workload running in our ROSA cluster, we can see these variables have been successfully inserted into the pod by running:

$ oc describe pod $(oc get pod -l app=awscli -o jsonpath='{.items[0].metadata.name}') | grep "^\s*AWS_"
 AWS_ROLE_ARN: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<APP_IAM_ROLE>
 AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token

Note also TokenExpirationSeconds in the Volumes section of the pod:

Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400

Step 8. Create an S3 bucket

Outside of the ROSA cluster, we can use the AWS CLI on our workstation to create an S3 bucket.

$ export S3_BUCKET_NAME=$ROSA_CLUSTER_NAME-iam-app
$ aws s3api create-bucket --bucket $S3_BUCKET_NAME --region $AWS_REGION --create-bucket-configuration LocationConstraint=$AWS_REGION > /dev/null

Step 9. Verify the solution

Verify the solution by starting a remote shell into the running pod and then use the AWS CLI to write to the S3 bucket.

$ oc rsh $(oc get pod -l app=awscli -o jsonpath='{.items[0].metadata.name}')

List the bucket using awscli

sh-4.2$ aws s3 ls

Create a file, upload it into the S3 bucket, and check the content of the S3 bucket.

sh-4.2$ echo "Hello from ROSA" > /tmp/rosa.txt
sh-4.2$ aws s3 cp /tmp/rosa.txt s3://<S3_BUCKET_NAME>/rosa.txt
upload: ../tmp/rosa.txt to s3://<S3_BUCKET_NAME>/rosa.txt
sh-4.2$ aws s3 ls s3://<S3_BUCKET_NAME> --recursive --human-readable --summarize
2022-01-17 13:50:58   39 Bytes rosa.txt

Total Objects: 1
   Total Size: 39 Bytes

Summary

In this blog post, we have explored how to use RBAC for implementing IAM roles for service accounts in a ROSA cluster to provide fine-grained permissions to pods and access AWS API securely. For more information and getting started with Red Hat OpenShift Service on AWS (ROSA) as well as other Red Hat solutions on AWS, see the AWS Red Hat partner page.

Containers