I set up replication between my buckets, but new objects aren't replicating. How can I troubleshoot this?

Last updated: 2022-03-30

I set up cross-Region replication (CRR) or same-Region replication (SRR) between my Amazon Simple Storage Service (Amazon S3) buckets. However, objects aren't replicating to the destination bucket. How can I fix this?

Resolution

To troubleshoot objects that aren't replicating to the destination bucket, check the different types of permissions for your bucket. Also, check the public access settings, and bucket ownership settings.

Tip: Make sure to test the replication after each configuration change by uploading an object to the source bucket. It's a best practice to make one configuration change at a time to identify any replication setup issues.

After you resolve the issues causing replication to fail, there might be objects in the source bucket that weren't replicated. By default, S3 replication doesn't replicate existing objects or objects with a replication status of FAILED or REPLICA. Replicate these objects using S3 batch replication.

Minimum Amazon S3 permissions

Confirm that the AWS Identity Access Management (IAM) role has the correct permissions. If the source and destination buckets are in different accounts, confirm that the destination account's bucket policy also grants sufficient permissions to the replication role.

The following example shows an IAM policy with the minimum permissions required for replication.    Based on the replication rule options, you might need to grant additional permissions.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::SourceBucket"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObjectVersionForReplication",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersionTagging"
            ],
            "Resource": [
                "arn:aws:s3:::SourceBucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateTags"
            ],
            "Resource": "arn:aws:s3:::DestinationBucket/*"
        }
    ]
}

Note: Replace SourceBucket and DestinationBucket with the names of your S3 buckets.

The IAM role must have a trust policy, which allows Amazon S3 to assume the role to replicate objects. For example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

If the destination bucket is in another account, then the destination bucket policy must grant the following permissions:

{
    "Version": "2012-10-17",
    "Id": "Policy1644945280205",
    "Statement": [
        {
            "Sid": "Stmt1644945277847",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789101:role/s3-replication-role"
            },
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateTags"
            ],
            "Resource": "arn:aws:s3:::DestinationBucket/*"
        }
    ]
}

Note: Replace arn:aws:iam::123456789101:role/s3-replication-role with the ARN of your replication role.    

Additional Amazon S3 permissions

If the replication rule is set to Change object ownership to the destination bucket owner, then the IAM role must have s3:ObjectOwnerOverrideToBucketOwner permissions. This permission is placed on the S3 object resource. For example:

{
    "Effect":"Allow",
         "Action":[
       "s3:ObjectOwnerOverrideToBucketOwner"
    ],
    "Resource":"arn:aws:s3:::DestinationBucket/*"
}

The destination account must also grant the s3:ObjectOwnerOverrideToBucketOwner permission through the bucket policy:

{
  "Version": "2012-10-17",
  "Id": "Policy1644945280205",
  "Statement": [
    {
      "Sid": "Stmt1644945277847",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789101:role/s3-replication-role"
      },
      "Action": [
        "s3:ReplicateObject",
        "s3:ReplicateTags",
        "s3:ObjectOwnerOverrideToBucketOwner"
      ],
      "Resource": "arn:aws:s3:::DestinationBucket/*"
    }
  ]
}

Note: If the destination bucket's object ownership settings include Bucket owner enforced, then you don't need Change object ownership to the destination bucket owner in the replication rule. This change will occur by default.

If the replication rule has delete marker replication activated, then the IAM role must have s3:ReplicateDelete permissions. If the destination bucket is in another account, then the destination bucket owner must also grant this permission through the bucket policy. For example:

{
    "Effect":"Allow",
         "Action":[
       "s3:ReplicateDelete"
    ],
    "Resource":"arn:aws:s3:::DestinationBucket/*"

}

Note: Replace DestinationBucket with the name of your S3 bucket.

The same permission must also be granted through the bucket policy on the destination bucket:

{
  "Version": "2012-10-17",
  "Id": "Policy1644945280205",
  "Statement": [
    {
      "Sid": "Stmt1644945277847",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789101:role/s3-replication-role"
      },
      "Action": [
        "s3:ReplicateObject",
        "s3:ReplicateTags",
        "s3:ObjectOwnerOverrideToBucketOwner",
        "s3:ReplicateDelete"
      ],
      "Resource": "arn:aws:s3:::DestinationBucket/*"
    }
  ]
}

AWS KMS permissions

If a bucket's source objects are encrypted with an AWS Key Management Service (AWS KMS) key, then the replication rule must be configured to include KMS-encrypted objects.

To include objects encrypted with AWS KMS:

1.    Open the Amazon S3 Console.

2.    Choose the S3 bucket that contains the source objects.

3.    On the Management tab, select a replication rule.

5.    Choose Edit.

6.    Under Encryption, select Replicate objects encrypted with AWS KMS.

7.    Under AWS KMS key for encrypting destination objects, select an AWS KMS key. The default option is to use the AWS KMS key (aws/S3).

Important: If the destination bucket is in a different AWS account, then specify a KMS customer managed key which is owned by the destination account. Don't use the default aws/S3 key. This encrypts the objects with the AWS managed key that is owned by the source account, and can't be shared with another account. As a result, the destination account can't access the objects in the destination bucket.

To use an AWS KMS key that belongs to the destination account to encrypt the destination objects, the destination account must grant the replication role in the KMS key policy:

{
    "Sid": "AllowS3ReplicationSourceRoleToUseTheKey",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::123456789101:role/s3-replication-role"
    },
    "Action": ["kms:GenerateDataKey", "kms:Encrypt"],
    "Resource": "*"
}

Note: If you use an asterisk (*) for Resource in the AWS KMS key policy, then the policy grants permission for the KMS key only to the replication role. The policy doesn't allow the replication role to elevate its permissions.

Additionally, the source account must add the following minimum permissions to the replication role's IAM policy:

{
    "Effect": "Allow",
    "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey"
    ],
    "Resource": [
        "SourceKmsKeyArn"
    ]
},
{
    "Effect": "Allow",
    "Action": [
        "kms:GenerateDataKey",
        "kms:Encrypt"
    ],
    "Resource": [
        "DestinationKmsKeyArn"
    ]
}

By default, the KMS key policy grants the root user full permissions to the key. These permissions can be delegated to other users in the same account. Unless there are deny statements in the source KMS key policy, then using an IAM policy to grant the replication role permissions to the source KMS key is sufficient.

Explicit Deny and Conditional Allow Statements

If your objects still aren't replicating after you've validated the permissions, check for any explicit deny statements:

  • Deny statements in the destination bucket policy, or KMS key policies that restrict access to specific CIDR ranges, VPC endpoints, or S3 access points can cause replication to fail.
  • Deny statements or permissions boundaries attached to the IAM role can cause replication to fail.
  • Deny statements in AWS Organizations service control policies attached to either the source or destination accounts can cause replication to fail.

Tip: Before removing any explicit deny statements, confirm the reason for using the deny and determine whether the statement has an impact on data security.

If either the source or destination KMS keys grant permissions based on the encryption context, then confirm that S3 Bucket Keys are turned on for the buckets. If the buckets have Bucket Keys turned on, then the encryption context must be for the bucket level resource:

"kms:EncryptionContext:aws:s3:arn": [
     "arn:aws:s3:::SOURCE_BUCKET_NAME"
     ]
"kms:EncryptionContext:aws:s3:arn": [
     "arn:aws:s3:::DESTINATION_BUCKET_NAME"
     ]

Note: Replace SOURCE_BUCKET_NAME and DESTINATION_BUCKET_NAME with the names of your source and destination buckets.    

If bucket keys aren't turned on for the source or destination buckets, then the encryption context must be the object level resource:

"kms:EncryptionContext:aws:s3:arn": [
     "arn:aws:s3:::SOURCE_BUCKET_NAME/*"
     ]
"kms:EncryptionContext:aws:s3:arn": [
     "arn:aws:s3:::DESTINATION_BUCKET_NAME/*"
     ]

Note: Replace SOURCE_BUCKET_NAME and DESTINATION_BUCKET_NAME with the names of your source and destination buckets.

Object ACLs and Block Public Access

Check whether the source and destination buckets are using ACLs. If the object has an ACL attached that allows public access but the destination bucket is using block public access, then replication fails.

Source Object Ownership

If the objects in the source bucket were uploaded by another AWS account, then the source account might not have permission to those objects. Check the source bucket to see if ACLs are deactivated. If the source bucket has ACLs deactivated, then the source account is the owner of all objects in the bucket. If the source bucket doesn't have ACLs deactivated, then check to see if Object Ownership is set to Object owner preferred or Bucket owner preferred. If the bucket is set to Bucket owner preferred, then source bucket objects need a bucket-owner-full-control ACL for the bucket owner to become the object owner.

The source account can take ownership of all objects in their bucket by deactivating ACLs. Most use cases don't require using ACLs to manage access. It's a best practice to use IAM and bucket policies to manage access to S3 resources. To deactivate ACLs on your S3 bucket, see Controlling ownership of objects and disabling ACLs for your bucket. Be sure to evaluate the current usage of ACLs on your bucket and objects. Your current bucket and IAM policies must grant sufficient permissions so that you can deactivate ACLs without impacting Amazon S3 access.


Did this article help?


Do you need billing or technical support?