How do I restore data from an Amazon Elasticsearch Service domain in another AWS account?

Last updated: 2021-02-10

I want to restore data from an Amazon Elasticsearch Service (Amazon ES) domain in another AWS account. How can I do this?

Short description

To restore data from an Amazon ES domain in another AWS account, you'll need to set up cross-account access between your domain and the Amazon ES domain that you're trying to restore data from. You must also allow your domain to access the Amazon Simple Storage Service (Amazon S3) bucket that stores your data.

To create this cross-account access, perform the following steps:

1.    In Account A, set up the following:

Source: Amazon ES domain with fine-grained access control
Source: Amazon S3 bucket

2.    In Account B, set up your destination (Amazon ES domain) with fine-grained access control.

Note: You don't need to create an Amazon S3 bucket in the destination (Account B). A single Amazon S3 bucket is used to restore the data across the AWS accounts. This setup also works for the Amazon ES domains without fine-grained access control.

Resolution

Note: The examples in this article are written in Python and Postman code.

Set up cross-account access for Account A

1.    Create an Amazon S3 bucket in Account A in the same Region as the Amazon ES domain.

2.    Create an AWS Identity Access Management (IAM) policy to provide Amazon S3 bucket access permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::snapshot" --> bucket ARN created in step 1.
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "iam:PassRole"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::snapshot/*"
            ]
        }
   ]
}

Replace "arn:aws:s3:::snapshot" with your bucket ARN.

3.    Create a role and select Amazon Elastic Compute Cloud (Amazon EC2) as your service.

4.    Add the IAM policy that you just created.

5.    Choose Trust relationships.

6.    Update the trust relationship of the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "es.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Replace "Service": "ec2.amazonaws.com" with "Service": "es.amazonaws.com".

Note: You use the role ARN for later steps.

7.    Edit the policy that you created:

{
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::Account A:role/cross"
        },
        {
            "Effect": "Allow",
            "Action": "es:ESHttpPut",
            "Resource": "arn:aws:es:us-east-1:Account A:domain/srestore/*"
        }

Replace "arn:aws:iam::Account A:role/cross" with the role that you created in Step 3. Also, update "arn:aws:es:us-east-1:Account A:domain/srestore/*" with the Amazon ES domain listed as the Source in Account A. The Source in Account A is used for cluster snapshots.

8.    Create an IAM user and attach the policy that you created in Step 3. This same IAM user must have admin access to the Amazon ES domain that's in Account A to provide access to read/write API using FGAC. For more information about using fine-grained access control, see Mapping roles to users.

9.    (Optional) If you're using Python code to register the Amazon S3 bucket to Amazon ES, launch an Amazon EC2 machine in Account A. Then, attach the IAM role created in Step 3.

Note: Make sure that your security group can access the Amazon ES domain. For more information, see Mapping roles to users.

Register the Amazon S3 bucket to the Source in Account A

To register the Amazon S3 bucket to the Source domain in Account A, perform the following steps:

1.    Update the PUT field with a URL that includes the Amazon ES domain endpoint and S3 bucket name. For example:

https://endpointofES domain.amazonaws.com/_snapshot/snapshot

2.    Choose the Authorization tab.

3.    Update the AccessKey and SecretKey of the IAM user.

4.    Update the AWS Region and Service Name.

5.    Choose Save.

6.    Choose the Headers tab.

7.    Select Content-Type for your key type.

8.    Select Application/JSON for your key value.

9.    Choose Save.

10.    Choose the Body tab.

11.    Use the following code:

{
    "type": "s3",
    "settings" : {
        "bucket" : "snapshot",
        "region" : "us-east-1",
        "role_arn" : "arn:aws:iam::Account A:role/cross"
}

12.    Choose Send to submit the query through the Amazon ES console. After the registration completes, you receive a Status Code: 200 OK message.

13.    Log in to Kibana in Account A. Then, check the available data on the Amazon S3 bucket.

14.    Use the following command to take a new snapshot:

PUT /_snapshot/<registered_snapshot_repository>/<snapshot_name>

Here's an example output:

GET _cat/snapshot/casnapshot
today SUCCESS 1585190280 02:38:00 1585190284 02:38:04 3.9s 4 4 0 4

This output verifies the completion of the Amazon S3 bucket registration to the Amazon ES domain in Account A.

Set up cross-account access for Account B

1.    Create a policy and IAM role in Account B, specifying the same Amazon S3 bucket ARN as Account A:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::snapshot"  --> S3 bucket ARN from Account A
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "iam:PassRole"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::snapshot/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::Account B:role/cross"  --> role created in Account B 
        },
        {
            "Effect": "Allow",
            "Action": "es:*",
            "Resource": "arn:aws:es:us-east-1:Account B:domain/restore/*"  --> Destination ES domain in Account B
        }
    ]
}

Here's an example trust policy for your role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "es.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

2.    Attach the IAM role that you previously created to the IAM user in Account B. The same IAM user must have administrator access to the Destination (Amazon ES domain with FGAC) in Account B. For more information about updating IAM user access, see Mapping roles to users.

3.     Update the Amazon S3 bucket policy for your bucket in Account A, providing Account B access to your bucket:

{
    "Version": "2012-10-17",
    "Id": "Policy1568001010746",
    "Statement": [
        {
            "Sid": "Stmt1568000712531",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::Account B:role/cross"  --> Role which is created in Account B 
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::snapshot"
        },
        {
            "Sid": "Stmt1568001007239",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::Account B:role/cross" -->  Role which is created in Account B 
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::snapshot/*"
        }
    ]
}

4.    Register the Amazon S3 bucket to your Amazon ES domain (in Account B).

Note: You must use the authentication credentials of the IAM user in Account B. Make sure to choose Amazon ES as your destination.

{
    "type": "s3",
    "settings" : {
        "bucket" : "snapshot",
        "region" : "us-east-1",
        "role_arn" : "arn:aws:iam::Account B:role/cross" ->  role which is created in Account B.
}

5.    Log in to Kibana in Account B.

6.    Check the snapshots from Account A that are available in the Amazon S3 bucket:

GET _cat/snapshots/casnapshot

Here's an example of the output:

today SUCCESS 1585190280 02:38:00 1585190284 02:38:04 3.9s 4 4 0 4

This output confirms that cross-account access is successfully set up in Account B.