How do I resolve Amazon S3 AccessDenied errors in Amazon SageMaker training jobs?

Last updated: 2020-09-03

My Amazon SageMaker training job failed with an AccessDenied error, even though the AmazonSageMakerFullAccess policy is attached to the execution role.

Short description

AccessDenied errors indicate that your AWS Identity and Access Management (IAM) policy doesn't allow one or more the following Amazon Simple Storage Service (Amazon S3) actions:

  • s3:ListBucket
  • s3:GetObject
  • s3:PutObject

The permissions that you need depend on the SageMaker API that you're calling. For example, the only Amazon S3 action that the CreateModel API requires is s3:GetObject. However, the CreateTrainingJob API requires s3:GetObject, s3:PutObject, and s3:ListObject. For more information about the permissions that are required for each API, see SageMaker roles.

Resolution

AccessDenied errors commonly happen in the following scenarios.

Encrypted input bucket

If the data in the S3 bucket is encrypted with AWS Key Management Service (AWS KMS):

  • Be sure that the IAM policy that's attached to the execution role allows the kms:encrypt and kms:decrypt actions. For more information, see SageMaker roles.
  • Be sure that the AWS KMS key policy grants access to the IAM role. For more information, see Using key policies in AWS KMS.
  • If you specify a KMS key for the machine learning (ML) storage volume in the resource configuration of your job, the IAM policy must allow the kms:CreateGrant action. For more information, see Using grants. For more information about encrypting the ML storage volume, see Protect data at rest using encryption.

Permissions boundaries

If you define permissions boundaries for the execution role, SageMaker can execute only the actions that are allowed by both the IAM policy and the permissions boundaries. Be sure that the IAM policy and the permissions boundaries allow the required Amazon S3 actions.

Bucket policies

If the input bucket uses a bucket policy, be sure that the bucket policy allows the execution role to perform the required Amazon S3 actions. For more information about bucket policies, see Policies and permissions in Amazon S3.

Here's an example of a bucket policy that denies access to the SageMaker execution role and causes an AccessDenied error:

{
    "Version": "2012-10-17",
    "Id": "ExamplePolicy01",
    "Statement": [
        {
            "Sid": "ExampleStatement01",
            "Effect": "Deny",
            "Principal": {
                "AWS": "arn:aws:iam::Account-ID:role/SageMakerExecutionRole"
            },
            "Action": [
                "s3:GetObject",
                "s3:GetBucketLocation",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::awsdoc-example-bucket/*",
                "arn:aws:s3:::awsdoc-example-bucket"
            ]
        }
    ]
}

Cross-account Amazon S3 access

If a different AWS account owns the Amazon S3 data:

  • Be sure that both accounts have access to the AWS KMS customer master key (CMK). If you don't specify a CMK for the training job, SageMaker defaults to an Amazon S3 server-side encryption key. A default Amazon S3 server-side encryption key can't be shared with or used by another AWS account.
  • Be sure that the IAM policy for the SageMaker execution role and the S3 bucket policy have cross-account permissions.

For more information, see How can I deploy an Amazon SageMaker model to a different AWS account?


Did this article help?


Do you need billing or technical support?