IBM & Red Hat on AWS

Configuring ROSA for fine-grained ECR repository access

Customers running Red Hat OpenShift Service on AWS (ROSA) frequently use Amazon Elastic Container Registry (ECR) for storing, sharing, and deploying their container images. As the number of containerized workloads continually grows, along with the associated container images, customers need enhanced methods to segment their container repositories with improved security features, limiting access to specific namespaces or projects that require them.

In this blog, we will demonstrate how to use ROSA with ECR and AWS Identity and Access Management (IAM) to protect access to your container images on a pod, namespace, node, or cluster level. Implementing controlled ECR repository access in ROSA helps enhance your security controls and enables improved governance for your ROSA workloads. This includes encryption of data at rest and in transit, vulnerability scanning of container images, detailed access logging through AWS CloudTrail, and automated compliance reporting.

Overview of solution

This tutorial will demonstrate the installation and configuration of the ECR Secret Operator in a ROSA cluster. Once configured, we will use the ECR Secret Operator to deploy project level secrets for workload-based service accounts. Additionally, we will configure AWS IAM roles that will negotiate with ECR permissions for access to container images in secure repositories.

Reference Architecture

Solution architecture containing ROSA cluster pulling images from ECR

Figure 1. Reference Architecture

Prerequisites

For this walkthrough, you should have the following prerequisites:

  • An AWS account with administrative access
  • A ROSA Cluster with cluster-admin privileges
  • The OpenShift command-line-interface (oc) installed and configured
  • Docker Desktop or alternative image builder
  • Enabled AWS CloudTrail logging

Prepare the ECR environment

  1. Log in to your AWS account and search for Elastic Container Registry
Amazon Elastic Container Registry

Figure 2. Amazon Elastic Container Registry

  1. Select Create to begin configuring a private repository

Use awsrosa/testrepo for the namespace/repo-name values, leave the other options as-is, and click Create

Creating a private ECR repository

Figure 3. Creating a private ECR repository

  1. Once the repository is created, take note of the URI value
URI value of the new private repository

Figure 4. URI value of the new private repository

Create the IAM Role to be used for ECR access

  1. Set up environment variables for the OIDC provider, AWS account ID, and ECR repository name
export OIDC_PROVIDER=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer| sed -e "s/^https:\/\///")
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export REPOSITORY_NAME=awsrosa/testrepo
Bash
  1. Create a new policy (ECRLoginPolicy1) with permissions to get the ECR authorization token
cat <<EOF > /tmp/iam_policy.json
{    
  "Version": "2012-10-17",    
  "Statement": [        
    {            
      "Effect": "Allow",            
      "Action": [                
        "ecr:GetAuthorizationToken"            
      ],
      "Resource": "*"        
    }    
  ]
}
EOF

aws iam create-policy --policy-name ECRLoginPolicy1 --policy-document file:///tmp/iam_policy.json
Bash
  1. Create a new IAM role (ECRLogin1) and attach a trust policy with ECR and the IAM policy defined in step 2.
cat <<EOF > /tmp/trust_policy.json
{    
  "Version": "2012-10-17",    
  "Statement": [        
    {            
      "Effect": "Allow",            
      "Principal": {                
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"           
      },           
      "Action": "sts:AssumeRoleWithWebIdentity",            
      "Condition": {                
        "StringEquals": {                    
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:ecr-secret-operator:ecr-secret-operator-controller-manager"                
        }            
      }        
    }    
  ]
}
EOF

aws iam create-role --role-name ECRLogin1 --assume-role-policy-document file:///tmp/trust_policy.jsonaws iam attach-role-policy --role-name ECRLogin1 --policy-arn arn:aws:iam::${AWS_ACCOUNT_ID}:policy/ECRLoginPolicy1
Bash
  1. Create an ECR repository policy providing permissions to the ECRLogin1 role to manage images
cat <<EOF > /tmp/repo_policy.json
{    
  "Version": "2012-10-17",    
  "Statement": [        
    {            
      "Sid": "AllowPushPull",            
      "Effect": "Allow",            
      "Principal": {                
        "AWS": [                    
          "arn:aws:iam::${AWS_ACCOUNT_ID}:role/ECRLogin1"                
        ]            
      },            
      "Action": [                
        "ecr:BatchGetImage",                
        "ecr:BatchCheckLayerAvailability",                
        "ecr:CompleteLayerUpload",                
        "ecr:GetDownloadUrlForLayer",                
        "ecr:InitiateLayerUpload",                
        "ecr:PutImage",                
        "ecr:UploadLayerPart"            
      ]        
    }    
  ]
}
EOF 

aws ecr set-repository-policy --repository-name ${REPOSITORY_NAME} --policy-text file:///tmp/repo_policy.json
Bash
  1. Create an STS kubernetes secret and apply it to a new project (ecr-secret-operator1)
cat <<EOF > /tmp/credentials
[default]
role_arn = arn:aws:iam::${AWS_ACCOUNT_ID}:role/ECRLogin1
web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
EOF 

oc new-project ecr-secret-operator1oc create secret generic aws-ecr-cloud-credentials1 --from-file=credentials=/tmp/credentials -n ecr-secret-operator1
Bash

Install and configure the ECR Secrets Operator

The ECR Private Registry Authentication service uses an authorization token with a 12-hour expiry. The ECR Secret Operator automatically refreshes the authorization token prior to expiration, eliminating the need to manage it as part of the authentication flow.

  1. Search for the ECR Secret Operator in the OperatorHub of your ROSA cluster
OperatorHub catalog in ROSA

Figure 5. OperatorHub catalog in ROSA

  1. Install the operator using the All namespaces on the cluster (default) installation mode and choose the ecr-secret-operator namespace created during the ECR setup.
  1. Create a new project (app-ecr-operator) in the ROSA cluster
oc new-project app-ecr-operator
Bash
  1. Create a new secret in the app-ecr-operator namespace
cat << EOF | oc apply -f -
apiVersion: ecr.mobb.redhat.com/v1alpha1
kind: Secret
metadata:  
  name: ecr-secret  
  namespace: app-ecr-operator
spec:  
  generated_secret_name: ecr-docker-secret  
  ecr_registry: [AWS-Account-ID].dkr.ecr.us-east-1.amazonaws.com  
  frequency: 10h  
  region: us-east-1
EOF
Bash
  1. Create a deployment from your private ECR repository to the new app-ecr-operator1 namespace
cat << EOF | oc apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ecr-deployment
  namespace: app-ecr-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-ecr
  template:
    metadata:
      labels:
        app: app-ecr
    spec:
      containers:
      - name: app-ecr-container
        image: [ECR Repository URI]/awsrosa/testrepo:latest
        ports:
        - containerPort: 8080
      imagePullSecrets:
      - name: ecr-docker-secret
EOF
Bash

Control access to the repository using the permissions policy in the ECR console

Once the pod has deployed successfully, we can validate the permissions and controls in the ECR console.

  1. Log in the AWS console and navigate to the awsrosa/testrepo repository created previously
  1. Select the Permissions link for the repository

Notice the IAM Entities associated with the repository permissions. These are the only roles permitted to access the repository. Additionally, only the specified actions listed in the permissions policy can be executed by the IAM entities

Permissions section of the ECR repository

Figure 6. Permissions section of the ECR repository

  1. Select Edit policy JSON on the Permissions screen and remove the ECRLogin1 role from the list of Principals and Save the changes
Permissions JSON with ECRLogin1 removed

Figure 7. Permissions JSON with ECRLogin1 removed

  1. Return to the ROSA console and delete the pod you previously deployed to require a new pull of the image from the ECR repository
Delete the pod in the ROSA console

Figure 8. Delete the pod in the ROSA console

The pod will fail to deploy since we removed the ECRLogin1 role from the ECR repository permissions

Pod deployment failure

Figure 9. Pod deployment failure

  1. The pod deployment failure can be validated in the CloudTrail event history for the ecr.amazonaws.com event source. The error message will include the user (ECRLogin1) and authorization error
Cloud Trail error message for the failed ECR image pull

Figure 10. Cloud Trail error message for the failed ECR image pull

  1. Add the ECRLogin1 IAM entity back in to the ECR repository permissions and save the change
ECR permissions with the ECRLogin1 role

Figure 11. ECR permissions with the ECRLogin1 role

  1. The pod will now be able to deploy properly with the required ECR access in place
Successful deployment

Figure 12. Successful deployment

Cleaning up

  1. Remove the ecr-deployment from the ROSA console
  2. Uninstall the ECR Secret Operator from the ROSA Installed Operators screen
  3. Remove the app-ecr-operator and ecr-secret-operator projects from ROSA
  4. Delete the awsrosa/testrepo repository from ECR

Conclusion

Integrating Amazon Elastic Container Registry (ECR) with Red Hat OpenShift Service on AWS (ROSA) provides enhanced operational capabilities with improved security features. When properly configured with the security controls outlined above, this integration helps organizations maintain compliance requirements while enabling efficient container management. In this blog we highlighted how to configure fine-grained access control to your repositories using ECR Permissions and AWS IAM policies and roles. This additional level of security and governance provides customers with a trusted model for ensuring the right users are accessing the right images.

Next Steps

Now that you understand how to secure access to your ECR repositories, explore methods for repository cross-account and cross-Region replication to ensure your images are readily available to all your clusters

Steve Mirman

Steve Mirman

Steve Mirman is an ex-Red Hatter, ex-IBMer, and current Partner Solutions Architect at AWS. He has over 20 years of experience helping customers architect, develop, deploy, and migrate enterprise applications.