Using Windows Authentication with gMSA on Linux Containers on Amazon ECS

UPDATE: On July 17^th 2023, AWS launched support for Windows authentication with gMSA on non-domain-joined (domainless) Amazon ECS Linux container instances. This blog post has been updated to cover both modes, making domainless mode the default.

Introduction

Today, we are announcing the availability of Credentials Fetcher integration with Amazon Elastic Container Service (Amazon ECS). This integration makes it easier for developers to implement Windows Authentication in Linux containers running on Amazon ECS using Microsoft Active Directory (AD) group Managed Service Account (gMSA). The Credentials Fetcher daemon allows containers running on Linux hosts to authenticate using gMSA credentials.

A gMSA is a managed domain account that provides automatic password management. While a typical AD account requires an IT administrator to manually set, rotate, and synchronize the password, gMSA passwords are automatically managed by Active Directory, including seamless synchronization across multiple clients. This type of account is ideal for containerized applications in Amazon ECS, because all instances of a task definition should have the same permissions, and the number of running instances can scale dynamically.

Prior to this release, workloads running on Amazon ECS using gMSA credentials had to run using Windows containers. Linux containers provide cost savings, increased uptime, and improved scalability – but workloads running in Linux containers had to use a solution, such as a “sidecar container”, that authenticated to the AD using credentials stored in AWS Secrets Manager. One disadvantage of this approach is that the sidecar container doesn’t automatically rotate or synchronize the AD account password. When the password changes in the AD, the container’s authentication fails because the stored credentials are no longer valid.

With today’s release, customers modernizing their applications by taking advantage of Linux containers can now use Windows authentication via the Kerberos protocol with automatic password management. This gives customers the ability to deploy workloads that are secure, easy-to-manage, cost-efficient, and scalable.

Solution overview

To explore how the gMSA support for Linux containers on Amazon ECS works, you’ll use the following sample solution:

Sample solution architecture diagram

Figure 1: Sample solution architecture diagram

You‘ll deploy:

AWS Directory Service for Microsoft Active Directory directory deployed in two Availability Zones (AZs).
Amazon Relation Database (Amazon RDS) for SQL Server instance with a sample database.
Amazon Elastic Compute Cloud (Amazon EC2) instance to aid in managing the AD.
Amazon Elastic Container Registry (Amazon ECR) repository to store the container image.
Amazon ECS cluster backed by an Amazon EC2 Auto Scaling Group using Amazon Linux 2023.
Active Directory gMSA, security group and user.
Amazon CloudWatch log group with the logs from the Amazon ECS tasks.
Supporting resources including AWS Systems Manager parameters, associations, documents and AWS Secrets Manager secrets.

After deploying these components, you’ll generate a Credential Specification (CredSpec) file for the created gMSA account, and upload it to Parameter Store, a capability of AWS Systems Manager, where Credentials Fetcher retrieves it.

Finally, you’ll build and deploy a simple .NET Web application in a Linux container on Amazon ECS. The Web application is configured to use the Windows Integrated Security to securely connect to the database. Credentials Fetcher is responsible for obtaining the Kerberos ticket from the gMSA and provides it to the Linux container.

The sample solution uses the AWS Cloud Development Kit (AWS CDK) to provision cloud resources using TypeScript. The AWS CDK allows developers to build AWS infrastructure in a variety of familiar programming languages, including JavaScript, C#, Python, Java, and Go.

Prior to any production deployment, you should always consult with your local security team to review security controls and requirements based on your environment and security posture.

Prerequisites

For the tutorial, you should have the following prerequisites:

An AWS account
Complete the AWS CDK getting started guide, including installing the AWS CDK and learning the key concepts
Create an Amazon EC2 key pair and record its name
Install the AWS Command Line Interface (AWS CLI) and set up your AWS credentials for command-line use if you are using a Bash-compatible shell
- If you prefer to use PowerShell, then install the AWS Tools for PowerShell and set up your AWS credentials for PowerShell
Install a Microsoft Remote Desktop (RDP) client
Install the latest version of the Docker runtime
Install the latest .NET 8 SDK

Domainless and domain-joined modes

There are two modes in which you can support Windows authentication using gMSA for your applications, non-domain-joined (domainless) mode and domain-joined mode.

In domainless mode, the Amazon ECS container instances doesn’t need to be joined to the target AD domain. This is the recommended mode for most workloads, especially when scaling is needed.

In domain-joined mode, you to have the Amazon ECS container instances joined to the target AD domain prior to deploying tasks on them. Use this mode if you don’t want to manage individual AD user accounts.

This blog post will guide you using the domainless mode, but will point the differences when using domain-joined mode.

Deploy the infrastructure

To start, create a directory for the sample solution in your local computer. Clone this GitHub repository into the directory.

Open a terminal in the cdk-typescript directory of the cloned repository, replace {KEY_PAIR_NAME} with your Amazon EC2 key pair name, and run the following commands if you are using Bash:

export AWS_DEFAULT_REGION={YOUR REGION}
export EC2_INSTANCE_KEYPAIR_NAME="{KEY_PAIR_NAME}"
export MY_SG_INGRESS_IP=$(curl checkip.amazonaws.com)
export DOMAIN_JOIN_ECS=0

npm install
cdk deploy "*" --require-approval "never"

If you are using PowerShell, then run the following commands:

$Env:AWS_DEFAULT_REGION = "{YOUR REGION}"
$Env:EC2_INSTANCE_KEYPAIR_NAME = "{KEY_PAIR_NAME}"
$Env:MY_SG_INGRESS_IP = $(Invoke-WebRequest -URI https://checkip.amazonaws.com).ToString().Trim()
$Env:DOMAIN_JOIN_ECS = 0   

npm install
cdk deploy "*" --require-approval "never"

Note: To use domain-joined mode, set the DOMAIN_JOIN_ECS variable to 1 before deploying the AWS CDK solution.

This starts the deployment of three AWS CloudFormation stacks that contain the sample solution. The deployment takes around one hour to complete.

When the deployment completes, navigate to the AWS CloudFormation console. Something similar to the following image will be displayed:

CloudFormation resources in AWS console

Figure 2: CloudFormation resources in AWS console

During the deployment, a security group, user, and gMSA are created in the AD. The user is a set as member of the security group, which is authorized to retrieve password from the gMSA. The AD user’s password is randomly generated, and stored in the secret with name:

aws/directory-services/[directory-id]/seamless-domain-join.

Walkthrough

In the following sections, you’ll explore how to configure Amazon ECS and Credentials Fetcher to use these AD security principals in the web application.

Install Credentials Fetcher in Amazon ECS

First, it’s necessary to install Credentials Fetcher in Amazon ECS. To do this automatically for all Amazon ECS container instances, you need to update the User Data inside the Auto Scaling Group’s Launch Template/Configuration adding the following commands:

echo "ECS_GMSA_SUPPORTED=true" >> /etc/ecs/ecs.config
dnf install dotnet realmd oddjob oddjob-mkhomedir sssd adcli krb5-workstation samba-common-tools credentials-fetcher -y
systemctl enable credentials-fetcher
systemctl start credentials-fetcher

The sample solution already applies this configuration. You can continue to the next section.

Configure an Active Directory identity that retrieves gMSA password

To retrieve the password from a gMSA, the Credentials Fetcher requires a security principal to authenticate with AD and retrieve the gMSA password. For domainless mode, you use an AD user, while for domain-joined mode, you use the host computer security principal. In both cases, the security principal should be authorized to retrieve the gMSA password, either directly or through an AD security group. The sample solution creates and uses an AD Security Group named SampleWebAppGmsaPrincipals to access the gMSA passwords.

In domainless mode, credentials fetcher retrieves the gMSA password using credentials stored in a Secrets Manager secret that you provide in the HostAccountConfig object of the CredSpec file as you will do in the next section..

In domain-joined mode, you need to join the Amazon ECS instances to the domain, and add the computer security principals to an authorized AD security group. You can automatically join the Amazon ECS container instances to the AD domain using Systems Manager associations and seamless domain join for Linux. Currently, there’s no available command on Linux to add computer security principals to an AD security group, so you need to do it via the Add-ADGroupMember PowerShell cdmlet running on Windows. This must happen every time a new Amazon ECS container instance is created.

As part of the sample solution, you’ll find a PowerShell script inside the AD management Amazon EC2 instance that‘ll add all Amazon ECS container instances in a cluster to the appropriate AD security group. You’ll find the script in the path C:\SampleConfig\Add-ECSContainerInstancesToADGroup.ps1. You need to pass as argument the name of the Amazon ECS’s ASG, which you’ll find in the ECSAutoScalingGroupName output of the amazon-ecs-gmsa-linux-infrastructure AWS CloudFormation stack.

Generate the CredSpec file

The CredSpec file is a JSON document, which contains metadata about the gMSA account that’s used within each container. Credentials Fetcher uses the CredSpec file to request a Kerberos ticket and then make it available to the container. The CredSpec file can only be generated in a Windows computer when logged in as an admin user of the AD.

To start the process of generating it, navigate to the AWS Secrets Manager Console and copy the value of the amazon-ecs-gmsa-linux/active-directory-administrator-password secret.

Active Directory Password secret in the AWS Console

Figure 3: Active Directory Password secret in the AWS Console

You can also get the secret value by running this command in the Bash terminal:

aws secretsmanager get-secret-value –secret-id amazon-ecs-gmsa-linux/active-directory-administrator-password

If you are using PowerShell, then you can get the secret value by running this command:

Get-SECSecretValue -SecretId amazon-ecs-gmsa-linux/active-directory-administrator-password

Next, navigate to the Amazon EC2 console, then select the instance named amazon-ecs-gmsa-linux-bastion/active-directory-management-instance.

Active Directory Management instance in the AWS Console

Figure 4: Active Directory Management instance in the AWS Console

Follow these instructions to connect to the instance using Remote Desktop. Use the username directory.amazon-ecs-gmsa-linux.com\admin, and the password you retrieved from Secrets Manager. (If you can’t log in, then the instance is still setting up the database and AD. Wait 10–15 minutes and try again.)

In the Remote Desktop session, open a PowerShell window and run the following command:

C:\SampleConfig\Generate-CredSpec.ps1

Note: In domain-joined mode, append the following parameter while running the script: -DomainlessArn "".

This command creates the CredSpec file and stores its minimized contents as both a Systems Manager parameter, and an Amazon Simple Storage Service (Amazon S3) object, where the credential fetcher retrieves can retrieve it. In domainless mode the CredSpec file will have the following additional object:

{
  ...
  "ActiveDirectoryConfig": { 
    ...
    "HostAccountConfig": {
      "PortableCcgVersion": "1",
      "PluginGUID": "{859E1386-BDB4-49E8-85C7-3070B13920E1}",
      "PluginInput": {
 	   "CredentialArn": "[SECRET ARN]"
      }
    } 
  }
}

This provides a reference to a Secrets Manager secret containing the username and password for an AD user used to retrieve the gMSA password and name of the target AD domain. The sample solution automatically creates this secret under the name credentials-fetcher-identity. There’s nothing more to do in this Amazon EC2 instance, so you can sign out of the Remote Desktop session.

You can now continue to the next section where you will build the .NET sample application.

Build the .NET application container

In the sample repository, there’s an ASP.NET Core application that connects to a table in the sample SQL Server database. This application helps you validate that the connection to SQL Server is indeed using Integrated Security. The Visual Studio solution file is located at /web-app/web-app.sln. Make sure that it builds successfully using Visual Studio, Visual Studio Code or the following command in the web-app folder:

dotnet build web-app.sln

dotnet build output

Figure 5: dotnet build output

Once the application has built successfully, you need to build the Docker container and push it to Amazon ECR. To do this, navigate to the Amazon ECR console. Select the amazon-ecs-gmsa-linux/web-site repository, then select View push commands.

Amazon ECR console

Figure 6: Amazon ECR console

Follow the directions to tag and push your image to the Amazon ECR repository.

If you are building the application in a Mac computer with an Apple M1/M2 processor, container are built using a arm64 processor architecture by default. Credentials Fetcher only supports x86-64 containers, so to build an x86-64 container, you need to run the following command instead of the one shown in the dialog window: docker build --platform=linux/amd64 -t amazon-ecs-gmsa-linux/web-site ..

Amazon ECR push commands in the AWS console

Figure 7: Amazon ECR push commands in the AWS console

Now that you have pushed the application container image into Amazon ECR, your next step is to configure the Amazon ECS task definition to support Windows Authentication with gMSA.

Configure the Amazon ECS task definition

To enable gMSA support in your Amazon ECS task definition, you need to set the credentialSpecs property with a link to your minimized CredSpec. The CredSpec can be stored in an Amazon Simple Storage Service (Amazon S3) bucket, a Parameter Store parameter, or on a local file accessible by container. If you chose Amazon S3 or Parameter Store, then you must set the Amazon ECS task execution role property to an AWS Identity and Access Management (AWS IAM) role with permission to read the parameter or bucket. For more information, refer to the Amazon ECS documentation for Using gMSAs for Linux Containers.

The Amazon ECS task definition looks similar to:

"containerDefinitions": [
        {
            ...
            " credentialSpecs": [
                " credentialspecdomainless:arn:aws:ssm:us-west-2:111111111111:parameter/testgmsa"
            ],
           ...
        }
]

Note: In domain-joined mode, replace the “credentialspecdomainless:” text with ” credentialspec:”.

Deploy the application to Amazon ECS

To deploy the Amazon ECS service with your application, go back to the terminal you used to deploy the infrastructure. Run the following commands if you are using Bash:

export DEPLOY_APP=1
cdk deploy "*" --require-approval "never"

Run the following commands if you are using PowerShell:

$Env:DEPLOY_APP = 1
cdk deploy "*" --require-approval "never"

Once the deployment is complete, go to the CloudFormation console and click on the value of the output named like websiteec2serviceServiceURLXXXXXXXX to navigate to the web app. The web application will run and authenticate to the AD using the gMSA.

Working sample web application

Figure 8: Working sample web application

The ASP.NET Core application configuration is identical to any other environment, including on-premises. The only particularity you need to keep in mind is that the SQL Server address in the connection string MUST be defined using the AD domain name. If you use an IP address or a different DNS name, then the authentication method defaults to Microsoft NTLM, which will fail.

Troubleshoot

There are two main types of errors you may encounter when deploying the sample solution. The first, and most simple, type of error is when there is an error connecting to SQL Server. To diagnose the problem, navigate to the Amazon ECS console, select your cluster name, then the service name, and finally select Logs to display the errors logged by the application.

Amazon ECS task logs

Figure 9: Amazon ECS task logs

The second type of error is when the Amazon ECS task cannot be launched successfully. These root cause of these error are usually found in the Credentials Fetcher. To explore Credential’s Fetcher logs, you need to connect to one Amazon ECS container Linux instance, and run the following command: sudo journalctl -u credentials-fetcher -e. The logs tell you more accurately what problem is preventing the launch of the task.

Clean up

To avoid incurring future charges, delete the resources. You can use the cdk destroy command to delete the stack. Run this command in the terminal or PowerShell window:

cdk destroy "*" --require-approval "never"

Finally, manually delete the Amazon CloudWatch log group created by AWS CloudFormation.

Conclusion

In this post, we showed you how to integrate Credentials Fetcher with Amazon ECS. This integration enables AWS customers to modernize applications to Linux containers while using Windows authentication via gMSA. gMSA authentication, previously only available on Windows containers, allows applications and services to authenticate against an AD without manually managing passwords. This reduces complexity and improves security by reducing the risk of a password leak.

The sample application in the post shows how to use gMSA to support Windows authentication in a .NET application running in a Linux container, while connecting to a SQL Server database. This pattern enables enterprises to run modern containerized workloads that unlock the cost savings, reliability, and scalability of Linux – without sacrificing the secure authentication provided by AD.

Containers