How to grant least privilege access to third-parties on your private EC2 instances with AWS Systems Manager

AWS Systems Manager Session Manager provides a more secure way to manage your Amazon Elastic Compute Cloud (EC2) instances without the need to open inbound ports, maintain bastion hosts, or manage SSH keys. Furthermore, you can use it with a combination of AWS services to give access to external third-parties.

Due to business requirements, you might need to grant access to your EC2 instances to a third-party entity. This could be a customer to which you are proposing a managed environment on AWS or a partner who might be performing some testing for you. Considering that inadvertent disclosure of data can originated from third-parties, a more secure method of granting them access is imperative.

In this post, we will show you how to provide least privilege access to third-parties on your private Linux and macOS EC2 instances by using Session Manager run as support, AWS Security Token Service (AWS STS) and Attribute-Based Access Control (ABAC) to grant access to specific EC2 instances based on the resource tags defined.

For demonstration purposes, the environment in this post consists of EC2 instances hosted in private subnets. AWS PrivateLink endpoints are used to connect the EC2 instances to AWS services without requiring internet connectivity. Additionally, Session Manager preferences are configured to stream session log data to Amazon CloudWatch Logs. Finally, the third-party in this post is defined as an external party to your organization.

Throughout this blog post, third-parties’ resources will be referred as “their” resources. Opposite to the resources in your organization, refer to as “your” resources.

Solution overview

The Figure below describes the workflow of how a third-party can connect on your private EC2 instances from their account:

1. Alice will assume an IAM role in the local account
2. Then, assume another IAM role in your account using External ID
3. With the temporary credentials, open a session on Session Manager on one of the authorized EC2 instance
4. The session opened on Session Manager will use a local OS (Operating System) user os_user_a_alice.

Figure 1: Solution architecture diagram

Two main steps are necessary: (1) your third-parties need to set up their accounts, and (2) you will need to configure your account too.

First part: Configuration in their account

Third-parties can start a session on your private EC2 instances based on the configuration you have set. They must:

• Know the External ID that you have set for them. It must be unique per third-party.
• Know the ARN of the IAM role that they have to assume in your AWS account to start a session on your private EC2 instances.
• Create an IAM role in their AWS account used to assume the IAM role created for them in your AWS account.
• Use the AWS Command Line Interface (CLI) and the Session Manager plugin on their local computers to start a session with Session Manager.

Second part: Configuration in your account hosting the private EC2 instances

Your AWS account hosts the private EC2 instances to which you would like to grant access to third-parties while taking into account security best practices such as:

1. the need-to-know principle – you would like to:
   • Utilize attribute-based access control (ABAC) policies to grant access only to specific tagged instances per third-party.
   • Control permissions and avoid the confused deputy problem by using IAM policies and External ID.
   • Create an IAM role per third-party that can be assumed to open a session on your EC2 instances.
2. the least privilege principle – you want to enforce it at the operating system (OS) level, by creating unprivileged local users on the EC2 instance using Systems Manager Run Command. Note:  an unprivileged user no_sudo_user is specified as the default operating system user name in the Session Manager preferences in the event an IAM user or role is not tagged with SSMSessionRunAs.
3. the logging of session data for archival purpose – you want to track the commands executed by third-parties during their sessions on your EC2 instance by streaming their activity to CloudWatch Logs.

Prerequisites

This solution assumes that the following are in place:

• One AWS account that will host your EC2 instances. Refer to as Your account.
• One AWS account that you can use as mock external third-parties’ AWS accounts to test the solution. Refer to as Their account.
• AWS CLI and Session Manager plugin for the AWS CLI installed on local computer.

Implement the solution

Two CloudFormation templates are used to deploy the solution in this post. They are available in this GitHub repository.

Use the region us-east-1 to deploy two stacks. When deploying the CloudFormation stacks, you must include the CAPABILITY_NAMED_IAM value to acknowledge the template’s capabilities due to the hardcoded names for IAM roles.

Part 1: Configure the third-party AWS account

In their AWS account, the third-parties will need to assume the IAM role created later in section Part 2 > Step 5. In order to make that possible, a few resources are needed.

Step 1 : Take note of the AWS account id

In the upcoming section, you will need to use the AWS account of this account. It will be identified by THEIR_ACCOUNTID.

Step 2 : Create resources in Their AWS Account

In the Third party aka Their AWS account, launch a CloudFormation stack using this template that will:

1. create a user group GroupRemoteAccess;
2. create and add an IAM user alice to that group – alice’s credentials will be stored in AWS Secrets Manager;
3. create one IAM role 3P_IAM_Assume_Role to be assumed by alice, and permitting to assume the IAM role SSMStartSession_IAM_Role_ThirdPartyA in your AWS account.

Figure 2: Resources created by the CloudFormation template in Their account

Important: Note that for the testing purpose of this post, an IAM user alice is used for the sake of simplicity. However, in practice, your third-party user should be a federated identity.

Part 2: Configure Your AWS account

In your account, deploy the CloudFormation template to create resources. Then perform additional actions such as tag your instances, create OS users and configure Session Manager preferences.

Take note of your AWS account ID, it will be later referred as YOUR_ACCOUNTID.

Step 1 : Create resources in Your AWS Account

In Your account, launch this CloudFormation template to create the resources below:

• One VPC with one private subnet
• Three VPC endpoints for AWS Systems Manager and one VPC endpoint for CloudWatch Logs in the VPC
• Two tagged EC2 instances in the private subnet having an IAM instance profile for Systems Manager with a managed policy AmazonSSMManagedInstanceCore. SSM agent is preinstalled by default on some Amazon Machine Images (AMIs) such as Amazon Linux 2, which is used for this post
• Security Group (SG) for the VPC endpoints that allows HTTPS traffic inbound from network interfaces that are assigned to the same Security Group. An additional HTTPS inbound rule, from EC2 instances SG
• Security Group (SG) for the EC2 instances that only allows HTTPS outbound traffic to the VPC Endpoints SG
• One CloudWatch Log group.

Figure 3: Resources created by the CloudFormation template in Your account

Important: Note that you can enable Systems Manager Default Host Management Configuration across your accounts, so that your EC2 instances will be automatically managed by Systems Manager without the requirement to attach an IAM instance profile to them.

Step 2 : Tag your EC2 instances

On your private EC2 instances, enforce the need-to-know principle by granting access to your third-parties through tag-based permissions. The EC2 instances created by the template will have tags with the key ExternalAccessGrantedTo and the value can be Third_Party_A or Third_Party_B.

The tag key-value pair is used in an IAM policy to restrict access to only the private EC2 instances provisioned for the respective third-party.

Figure 4:Tags on your EC2 instances

Step 3 : Create OS users

To create the local Operating System (OS) user on the EC2 instances, use Systems Manager Run Command. Based on your EC2 instances tags, Run Command will run the commands of your choice with the AWS-RunShellScript document.

1. On AWS System Manager page, click on Run Command
2. Click on the button Run a Command
3. On the search bar for Command document search for AWS-RunShellScript
4. On the document page, click on Run command
5. In the Command parameters section, create two OS users by entering:
   • sudo useradd os_user_a_alice -g users -c 'Third_Party_A – alice' -m;
   • sudo useradd no_sudo_user -g users -c 'Unprivileged user’ -m
6. In the Targets section, select Specify instance tags: Tag key ExternalAccessGrantedTo and Tag value Third_Party_A.
7. Leave the other options as the default.
8. Click on Run button.

As per best practices, you should consider adding the creation of each required OS user in the AMI used to create the instance or as part of the instance launch configuration. For the purpose of this post, we use Run Command to create the OS user on each EC2 instance.

Step 4 : Configure Session Manager Preferences

We will configure Session Manager to use an OS user name no_sudo_user instead of the system-generated ssm-user. Any session opened on your EC2 instance with Session Manager without the SSMSessionRunAs tag present, will use no_sudo_user.

Step 4a: To Configure Run As

1. From AWS System Manager, select Session Manager.
2. Click on Configure Preferences
3. On the Preferences tab, in the General Preferences section, check Enable Run As support for Linux instances.
4. Under Operating system user name, type no_sudo_user.
5. Click on the Save button.

Figure 6: Enable Run As on Session Manager

Step 4b: To Configure CloudWatch logging

To see in near real-time the OS commands entered by your external third-parties on your EC2 instances during their session, we will configure the streaming of the logs to CloudWatch Logs. A CloudWatch log group named SSMSessionManager_LogGroup is created by the CloudFormation template with a retention period defined as 7 days.

From AWS System Manager, select Session Manager.

1. Click on Configure Preferences
2. On the Preferences tab, in the CloudWatch logging section, select the Enable checkbox.
3. Select Stream session logs (Recommended) to have the session data streams sent continuously to CloudWatch logs and specify the log group SSMSessionManager_LogGroup.
4. Uncheck Allow only CloudWatch log groups that are encrypted by customers master keys (CMK). By default, log group data is encrypted in CloudWatch Logs using server-side encryption.
5. Select Choose a log group name from the list and select SSMSessionManager_LogGroup.
6. Click the Save button.

Step 5 : Create an IAM role for your third-party

The CloudFormation template 2-YourAWSAccount_Configuration.yaml creates the IAM role SSMStartSession_IAM_Role_ThirdPartyA, with tags and associated permissions for the third party.

Step 5a: To add mandatory tags

To enforce the ABAC, this IAM role required as least the following tags:

• ExternalAccessGrantedTo: should match the principal tag assuming the role
• SSMSessionRunAs: with the value of the OS user to be used to open the session.

Figure 8: tags for the IAM Role to be assume to open sessions

Step 5b: To Configure IAM policy allowing third-parties to open a session on your EC2 instances

This IAM policy is created by the CloudFormation template and defines the permissions granted to third-parties when they assume the role to open a session on your EC2 instances. Below a presentation of the main sections of that policy.

1. Strict restriction to Start Session

Third-parties can list your EC2 instances.

{     
    "Sid": "DescribeAllEC2",
    "Effect": "Allow",
    "Action": [
        "ec2:DescribeInstances",
        "ssm:GetConnectionStatus",
        "ssm:DescribeInstanceProperties"
        ],
    "Resource": "*"
},

Using ABAC, it’s only possible to start a session on the EC2 instances that have the tag value associated to the right third-party.

{
    "Sid": "StartSessionOnInstances",
    "Effect": "Allow",
    "Action": "ssm:StartSession",
    "Resource": "arn:aws:ec2:REGION:YOUR_ACCOUNTID:instance/*",
    "Condition": {
        "StringEquals": {
           "ssm:resourceTag/ExternalAccessGrantedTo": "Third_Party_A"
           }
       }
},

2. Resume/Terminate Session Third-parties’ users should only be able to terminate/resume their session. The restriction is applied to their username.

{
     "Effect": "Allow",
     "Action": ["ssm:TerminateSession", "ssm:ResumeSession"],
     "Resource": "*",
     "Condition": {
        "StringEquals": { "ssm:resourceTag/aws:ssmmessages:session-id": ["${aws:userid}"] } 
       } 
},

3. Forbidden actions

Third-parties’ users shouldn’t be able to assume any other IAM role in your account. Nor should they be able to see other SSM sessions.

{
    "Effect": "Deny",
    "Action": [
        "sts:AssumeRole",
         "ssm:DescribeSessions"
        ],
     "Resource": "*" 
}

4. Amazon CloudWatch

This policy allows log Session Manager session data to the CloudWatch Log Group created in section Part 1 >Step 4 > Step 4b based on its ARN.

{
    "Effect": "Allow",
    "Action": [
       "logs:CreateLogStream",
       "logs:PutLogEvents",
       "logs:DescribeLogGroups",
       "logs:DescribeLogStreams"
       ],
     "Resource":"arn:aws:logs:REGION:YOUR_ACCOUNTID:log-group:SSMSessionManager_LogGroup:*"
}

Step 6 : Create a Trust Policy with External ID

This policy allows a third-party AWS account THEIR_ACCOUNTID to assume the IAM role 3PA_IAM_Assume_Role created earlier in Part 1 > Step 2. But ONLY if the correct External ID is provided by the third-party.

{
    "Version": "2012-10-17",
    "Statement": [{
       "Effect": "Allow", 
       "Principal": {
          "AWS": "arn:aws:iam::THEIR_ACCOUNTID:role/3PA_IAM_Assume_Role"
          },
       "Action": "sts:AssumeRole",
       "Condition": 
           { "StringEquals": {"sts:ExternalId": "ExternalID3rdPartyA"} }
     }]
}

The usage of External ID addresses and prevents the confused deputy problem. External ID are generally GUIDs. These identifiers are unique per third-party and are sensible values to use. Learn more about how to use trust policies with IAM roles in this post. See the global condition keys.

Test the solution

Test for yourself the solution after you followed the implementation steps.

Step 1 : Use AWS CLI to open the session

Your third-parties can use AWS CLI to open a cross-account session from Their AWS account to your private EC2 instances.

Step 2 : Role chaining

Your third-parties’ users will do role chaining to gain access to your account. First, this will be done by assuming the local IAM role and using the temporary credentials provided. Execute the commands below in AWS CLI. Replace <THEIR_ACCOUNTID> by the value of the account you selected as your third-party account and <YOUR_ACCOUNTID> with the AWS account id you selected as your account.

aws sts assume-role \
    --role-arn arn:aws:iam::<THEIR_ACCOUNTID>:role/3P_IAM_Assume_Role \
    --role-session-name switchrole1

Then, by assuming the cross-account role by providing the External ID secret.

aws sts assume-role \
    --role-arn arn:aws:iam::<YOUR_ACCOUNTID>:role/SSMStartSession_IAM_Role_ThirdPartyA \
    --external-id ExternalID3rdPartyA \
    --role-session-name switchrole2
	

Using the new temporary security credentials, access is granted to your AWS account hosting the EC2 instances. To assume role using AWS CLI, read this.

Figure 9: Third-party use role chaining to gain access to Your AWS account

Step 3 : Get a list of the EC2 instances on which the 3rd parties can open sessions

Use AWS CLI and the command ec2 describe-instances to list the authorized EC2 instances for a specific third-party tag-value. This method is scalable and doesn’t require you to regularly send a list of authorized EC2 instances to your third-parties.

aws ec2 describe-instances \
     --filters "Name=tag-value,Values= <tag_value>" \ 
    --query 'Reservations[*].Instances[*].{Instance:InstanceId, Status:State.Name, PublicIP:PublicIpAddress, ExternalAccess:Tags[?Key==`<tag_key>`]|[0].Value, Name:Tags[?Key==`Name`]|[0].Value}' \
    --output table
	

Replace <tag_value> by Third_Party_A and <tag_key> is ExternalAccessGrantedTo as per the tags of your EC2 instance shown on Figure 4:Tags on your EC2 instances.

Figure 10: Your third-party lists your EC2 instances they can have access to based on their tags

Step 4 : Open session on any of authorized EC2 instances

The following is an example of a session opened on an authorized EC2 instance using the assume role temporary security credentials provided after the role chaining in section Role chaining.

Figure 11: Session opened from cross account on Third_Party_A account to one of your private EC2 instance

Step 5 : Open session on an unauthorized EC2 instance

If the third-party attempts to open a session on EC2 instances on which they don’t have access based on ABAC, the session is denied.

An error occurred (AccessDeniedException) when calling the StartSession operation: 
User: … is not authorized to perform: ssm:StartSession 
on resource: …instance/i-05567a4c26c0d2f13 
because no identity-based policy allows the ssm:StartSession action

Step 6 : Session activity logging

As your third-party user enters commands during the session, those commands and results are streamed to the specified CloudWatch Logs log group.

Amazon CloudWatch Logs can be used to receive a continual stream of session data logs.

Step 7 : Audit all activities performed by the third-party in your account

AWS CloudTrail monitors and records account activity across your AWS infrastructure. AWS CloudTrail can be used to audit activities performed by third-parties in your account. A filter can be used to find the events generated by the IAM role that you created for them.

Clean up

To delete the resources created by the CloudFormation templates, go to the AWS CloudFormation console in the each of the two AWS accounts used. Choose the stack you created, and then choose Delete.

Conclusion

In this post, you’ve seen how you can use AWS Systems Manager Session Manager to grant access to an external party on your private EC2 instances while applying security best practices such as least privilege principle, need to know principle, and logging for archival purpose.

You can keep your EC2 instances private without internet connectivity and still successfully deploy this solution. VPC interface Endpoints are used for connectivity to make sure that your EC2 instances don’t become exposed to internet traffic.

Tags are used for ABAC and to enforce security principles. External ID is used to avoid the confused deputy problem. The Run As feature of Session Manager is used to restrict privileges of your third-party on your EC2 instances’ OS.

About the authors: