How can I determine which SageMaker notebook instance made a particular API call if all the instances use the same IAM role?

4 minute read
0

I have multiple Amazon SageMaker notebook instances. They all use the same AWS Identity and Access Management (IAM) role. The AWS CloudTrail event for each API action shows the same principalId (session name), no matter which notebook instance performed the action. How can I tell which notebook instance performed which API actions?

Short description

When you have multiple SageMaker instances with the same IAM role, you can't determine which notebook instance performed a particular API action with the CloudTrail event.

Example:

{
    "eventVersion": "1.05",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AAAAAAAAAAAAAAAAAA:SageMaker",
       
    "arn": "arn:aws:sts::111122223333:assumed-role/AmazonSageMaker-ExecutionRole/SageMaker",

Resolution

1.    Create an IAM execution role for the SageMaker notebook instance. Or, use an existing execution role. In the following steps, the Amazon Resource Name (ARN) for the execution role is arn:aws:iam::111122223333:role/service-role/AmazonSageMaker-ExecutionRole.

2.    Attach an IAM policy that includes sts:AssumeRole to the execution role. The sts:AssumeRole action allows the execution role to assume itself using a different session name.

Example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111122223333:role/service-role/AmazonSageMaker-ExecutionRole"
        }
    ]
}

3.    Create a Start notebook lifecycle configuration script similar to the following. This example script retrieves the notebook instance name and then uses the name as the session name.

#Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

#Permission is hereby granted, free of charge, to any person obtaining a copy of this
#software and associated documentation files (the "Software"), to deal in the Software
#without restriction, including without limitation the rights to use, copy, modify,
#merge, publish, distribute, sublicense, and/or sell copies of the Software, and to
#permit persons to whom the Software is furnished to do so.

#THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
#INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
#PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
#HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
#OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
#SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

#!/bin/bash

set -ex

# Obtain the name of the notebook instance
nbname=$(jq -r '.ResourceName' /opt/ml/metadata/resource-metadata.json)
echo "Notebook Name = $nbname"

# Use the AWS Command Line Interface (AWS CLI) to obtain the Amazon Resource Name (ARN) of the IAM execution role
nbinfo=$(aws sagemaker describe-notebook-instance --notebook-instance-name $nbname)
nbrole=$(jq -r '.RoleArn' <<< "$nbinfo")
echo "Notebook Role = $nbrole"

# Obtain the Region of the notebook instance
nbregion=$(aws configure get region)
echo "Notebook Region = $nbregion"

# Write Assume Role Provider Settings to a new config file
echo "Writing new config file"
cat > /home/ec2-user/.aws/config.new <<EOF1
[default]
region=$nbregion
role_arn = $nbrole
credential_source = Ec2InstanceMetadata
role_session_name = $nbname
sts_regional_endpoints = regional
EOF1

echo "Moving new config to config file"
sudo mv /home/ec2-user/.aws/config.new /home/ec2-user/.aws/config

# Secure the "config" file so that it can't be deleted/updated without root user permissions
sudo chattr +i /home/ec2-user/.aws/config

4.    Create a SageMaker notebook instance (for example, test-2), and then attach the lifecycle configuration script that you created in the previous step.

5.    Create a SageMaker notebook instance with root access turned off. This restricts the user ec2-user from deleting or updating the config files.

6.    To identify the notebook instance that performed an API action, check the CloudTrail event. Under the userIdentity object, the principalId and arn show the name of the notebook instance. For example, the following event detail shows that the SageMaker notebook instance test-2 made the API call.

{
    "eventVersion": "1.05",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AAAAAAAAAAAAAAAAAAAA:test-2",
        "arn": "arn:aws:sts::111122223333:assumed-role/AmazonSageMaker-ExecutionRole/test-2",
        "accountId": "111122223333",
        "accessKeyId": "AAAAAAAAAAAAAAAAAAAA",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "AAAAAAAAAAAAAAAAAAAA",
                "arn": "arn:aws:iam::111122223333:role/service-role/AmazonSageMaker-ExecutionRole",
                "accountId": "111122223333",
                "userName": "AmazonSageMaker-ExecutionRole"
            },
            "webIdFederationData": {},
            "attributes": {
                "mfaAuthenticated": "false",
                "creationDate": "2020-09-12T00:45:04Z"
            }
        },
        "invokedBy": "im.amazonaws.com"
    },
    "eventTime": "2020-09-12T00:49:04Z",
    "eventSource": "sagemaker.amazonaws.com",
    "eventName": "CreateEndpoint",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "im.amazonaws.com",
    "userAgent": "im.amazonaws.com",
    "requestParameters": {
        "endpointName": "sagemaker-mxnet-ep",
        "endpointConfigName": "sagemaker-mxnet-epc",
        "tags": []
    },
    "responseElements": {
        "endpointArn": "arn:aws:sagemaker:us-east-1:111122223333:endpoint/sagemaker-mxnet-ep"
    },
    "requestID": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "eventID": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "eventType": "AwsApiCall",
    "recipientAccountId": "111122223333"
}

Related information

SageMaker roles

AWS OFFICIAL
AWS OFFICIALUpdated 2 years ago