How do I restore data from an Amazon OpenSearch Service domain in another AWS account?

Short description

To restore data from an OpenSearch Service domain in another AWS account, you must set up cross-account access. Cross-account access must be established between your OpenSearch Service domain and the domain that you're trying to restore data from. You must also allow your domain to access the Amazon Simple Storage Service (Amazon S3) bucket that stores your data.

To create this cross-account access, perform the following steps:

1.    In Account A, set up the following:
Source: OpenSearch Service domain with fine-grained access control
Source: Amazon S3 bucket

2.    In Account B, set up your destination (OpenSearch Service domain) with fine-grained access control.

Note: You don't need to create an S3 bucket in the destination (Account B). A single S3 bucket is used to restore the data across the AWS accounts. This setup also works for the OpenSearch Service domains without fine-grained access control.

Resolution

Note: The examples in this article use Python and Postman code.

Set up cross-account access for Account A

1.    Create an S3 bucket in Account A in the same Region as the OpenSearch Service domain.

2.    Create an AWS Identity Access Management (IAM) policy to provide S3 bucket access permissions:

{
     "Version": "2012-10-17",
     "Statement": [{
          "Action": [
               "s3:ListBucket"
          ],
          "Effect": "Allow",
          "Resource": [
               "arn:aws:s3:::snapshot"
          ]
     },
     {
          "Action": [
               "s3:GetObject",
               "s3:PutObject",
               "s3:DeleteObject",
               "iam:PassRole"
          ],
          "Effect": "Allow",
          "Resource": [
               "arn:aws:s3:::snapshot/*"
          ]
     }]
}

Note: Replace "arn:aws:s3:::snapshot" with your bucket ARN from Step 1.

3.    Create an IAM role and select Amazon Elastic Compute Cloud (Amazon EC2) as your service.

4.    Add the IAM policy (created in Step 2) to your newly created IAM role.

5.    Open your IAM role and choose Trust relationships.

6.    Update the trust relationship of the following policy:

{
     "Version": "2012-10-17",
     "Statement": [{
          "Effect": "Allow",
          "Principal": {
               "Service": "es.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
     }]
}

Note: Replace "Service": "ec2.amazonaws.com" with "Service": "es.amazonaws.com". Also, record the role ARN, which you'll need for later steps.

7.    Choose one of these options:
Update the policy (from Step 2) to include the "iam:PassRole" permissions, attaching the policy to your IAM role. This permission allows OpenSearch service to have write access to an S3 bucket.

-or-

Create a new IAM policy, attaching the policy to your IAM role.

Note: You can have all your permissions set under one IAM role by updating the policy. Or, if you want to create a new IAM policy and split up the permissions, you can reuse the IAM policy for another use case.

Here's an example policy with the required IAM permissions:

{
     "Version": "2012-10-17",
     "Statement": [{
          "Effect": "Allow",
          "Action": "iam:PassRole",
          "Resource": "arn:aws:iam::Account A:role/cross"
     },
     {
          "Effect": "Allow",
          "Action": "es:ESHttpPut",
          "Resource": "arn:aws:es:us-east-1:Account A:domain/srestore/*"
     }]
}

This policy must be attached to the IAM user or role that is used to sign the HTTP request.

Note: Replace "arn:aws:iam::Account A:role/cross" with the role that you created in Step 3. Also, update "arn:aws:es:us-east-1:Account A:domain/srestore/*" with the OpenSearch Service domain listed as the Source in Account A. The Source in Account A is used for cluster snapshots.

8.    Create an IAM user and attach the policy that you created in Step 2 (which includes the required permissions to access Amazon S3). This IAM user must have Admin access to the OpenSearch Service domain in Account A to provide access to the read/write API using FGAC. For more information about using fine-grained access control, see Map the snapshot role in OpenSearch Dashboards (if using fine-grained access control).

9.    (Optional) If you're using Python code to register the S3 bucket to OpenSearch Service, launch an Amazon EC2 machine in Account A. Then, attach the IAM role created in Step 3.

Note: Make sure that your security group can access the OpenSearch Service domain.

Register the S3 bucket to the Source in Account A

To register the S3 bucket to the Source domain in Account A, perform the following steps:

1.    Update the PUT field with a URL that includes the OpenSearch Service domain endpoint and S3 bucket name. For example:

https://endpointofdomain.amazonaws.com/_snapshot/snapshot

2.    Choose the Authorization tab.

3.    Update the AccessKey and SecretKey of the IAM user.

4.    Update the AWS Region and Service Name.

5.    Choose Save.

6.    Choose the Headers tab.

7.    Select Content-Type for your key type.

8.    Select Application/JSON for your key value.

9.    Choose Save.

10.    Choose the Body tab.

11.    Use the following code:

{
     "type": "s3",
     "settings": {
          "bucket": "snapshot",
          "region": "us-east-1",
          "role_arn": "arn:aws:iam::Account A:role/cross"
     }
}

12.    Choose Send to submit the query through the OpenSearch Service console. After the registration completes, you'll receive a Status Code: 200 OK message.

13.    Log in to OpenSearch Dashboards in Account A. Then, check the available data on the S3 bucket.

14.    Use the following command to take a new snapshot:

PUT /_snapshot/<registered_snapshot_repository>/<snapshot_name>

Here's an example output:

GET _cat/snapshots/casnapshot
today SUCCESS 1585190280 02:38:00 1585190284 02:38:04 3.9s 4 4 0 4

This output verifies the completion of the S3 bucket registration to the OpenSearch Service domain in Account A.

Set up cross-account access for Account B

1.    Create a policy and IAM role in Account B, specifying the same S3 bucket ARN as Account A:

{
     "Version": "2012-10-17",
     "Statement": [{
          "Action": [
               "s3:ListBucket"
          ],
          "Effect": "Allow",
          "Resource": [
               "arn:aws:s3:::snapshot" --> S3 bucket ARN from Account A
          ]
     },
     {
          "Action": [
               "s3:GetObject",
               "s3:PutObject",
               "s3:DeleteObject",
               "iam:PassRole"
          ],
          "Effect": "Allow",
          "Resource": [
               "arn:aws:s3:::snapshot/*"
          ]
     },
     {
          "Effect": "Allow",
          "Action": "iam:PassRole",
          "Resource": "arn:aws:iam::Account B:role/cross" --> Role created in Account B 
     },
     {
          "Effect": "Allow",
          "Action": "es:*",
          "Resource": "arn:aws:es:us-east-1:Account B:domain/restore/*" --> Destination ES domain in Account B
     }]
}

Here's an example trust policy for your role:

{
     "Version": "2012-10-17",
     "Statement": [{
          "Effect": "Allow",
          "Principal": {
               "Service": "es.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
     }]
}

2.    Attach the IAM role that you previously created to the IAM user in Account B. The same IAM user must have administrator access to the Destination (domain with FGAC) in Account B. For more information about updating IAM user access, see Registering a manual snapshot repository.

3.     Update the S3 bucket policy for your bucket in Account A, providing Account B access to your bucket:

{
     "Version": "2012-10-17",
     "Id": "Policy1568001010746",
     "Statement": [{
          "Sid": "Stmt1568000712531",
          "Effect": "Allow",
          "Principal": {
               "AWS": "arn:aws:iam::Account B:role/cross" --> Role which is created in Account B 
          },
          "Action": "s3:*",
          "Resource": "arn:aws:s3:::snapshot"
     },
     {
          "Sid": "Stmt1568001007239",
          "Effect": "Allow",
          "Principal": {
               "AWS": "arn:aws:iam::Account B:role/cross" --> Role which is created in Account B 
          },
          "Action": "s3:*",
          "Resource": "arn:aws:s3:::snapshot/*"
     }]
}

4.    Register the S3 bucket to your domain (in Account B).

Note: You must use the authentication credentials of the IAM user in Account B. Make sure to choose OpenSearch Service as your destination.

{
     "type": "s3",
     "settings": {
          "bucket": "snapshot",
          "region": "us-east-1",
          "role_arn": "arn:aws:iam::Account B:role/cross" --> Role which is created in Account B.
     }
}

5.    Log in to OpenSearch Dashboards in Account B.

6.    Check the snapshots from Account A that are available in the S3 bucket:

GET _cat/snapshots/casnapshot

Here's an example of the output:

today SUCCESS 1585190280 02:38:00 1585190284 02:38:04 3.9s 4 4 0 4

This output confirms that cross-account access is successfully set up in Account B.

---

Related information

How can I migrate data from one Amazon OpenSearch Service domain to another?

Attach a bucket policy to grant cross-account permissions to Account B

Migrating Amazon OpenSearch Service indices using remote reindex