How can I use DataSync to transfer data between Amazon EFS file systems in different Regions over a private network?

Last updated: 2020-08-28

I want to use AWS DataSync to transfer data between Amazon Elastic File System (Amazon EFS) file systems in different AWS Regions. I want to transfer data over a private network. How can I do that?

Short description

Follow these steps to enable a DataSync cross-Region transfer between Amazon EFS file systems over an Amazon Virtual Private Cloud (Amazon VPC) peering connection:

  1. Create a cross-Region VPC peering connection.
  2. Configure the security group rules in the source and destination Amazon EFS file systems.
  3. Create a VPC endpoint for DataSync in the Region of the destination Amazon EFS file system.
  4. Create the DataSync agent in the source Region, and then activate the agent in the destination Region.
  5. Create the locations for the source and destination Amazon EFS file systems.
  6. Create the DataSync task, and then run the task.

Resolution

The following configuration steps are based on this example environment:

  • The source AWS Region is US East (N. Virginia) (us-east-1).
  • The source VPC CIDR is 10.10.0.0/16 (one public subnet).
  • The DataSync agent virtual machine's (VM) IP address is 10.10.3.124. The DataSync VM is deployed in the source Region.
  • The destination Region is US East (Ohio) (us-east-2).
  • The destination VPC CIDR is 10.20.0.0/16.

Important: You must configure your security group rules based on your environment's source and destination VPC CIDRs.

Create a cross-Region VPC peering connection

Create a VPC peering connection between the VPCs of the source and destination Amazon EFS file systems.

Before you proceed to the next step, use the Amazon VPC console to verify the following:

  1. View the peering connection. Confirm that the status is Active.
  2. View the source VPC. Review the VPC's route table to confirm that there is an active route to a target that begins with pcx. This route is for the peering connection.
  3. View the destination VPC. Review the VPC's route table to confirm that there is an active route to a target that begins with pcx.

Configure the security group rules for the source and destination Amazon EFS file systems

Important: The following example security group rules are based on the example VPC CIDRs. You must configure your security group rules based on your environment's VPC CIDRs.

Modify the security group rules of both the source and destination Amazon EFS file systems to meet the network requirements for the DataSync agent.

For the security group of the source Amazon EFS file system, configure an inbound rule that allows the DataSync agent VM to mount locally, similar to the following:

Type Protocol Port Range Source Description
NFS TCP 2049 10.10.3.124/32 NFS

For the default security group of the VPC that the source Amazon EFS file system is in, configure an inbound rule similar to the following:

Type Protocol Port Range Source Description
All traffic All All (The security group's ID)  

When you create tasks using a DataSync agent in a VPC, the tasks are private and each task creates four elastic network interfaces in the same subnet that the VPC endpoint for DataSync is in. These network interfaces are used for transferring data. This is important to note for the destination Amazon EFS file system's security group because the DataSync agent must be able to route to all the network interfaces. For more information, see Using AWS DataSync in a virtual private cloud.

For the security group of the destination Amazon EFS file system, configure inbound rules that allow traffic from the DataSync agent VM, similar to the following:

Type Protocol Port Range Source Description
All traffic All All (The security group's own ID)  
HTTPS TCP 443 10.10.3.124/32 HTTPS
NFS TCP 2049 10.10.3.124/32 NFS

When you activate the DataSync agent in the destination VPC, DataSync creates elastic network interfaces (ENIs) in the destination VPC subnets. For the security group of the destination Amazon EFS file system, configure an outbound rule that allows the ENIs within the destination VPC to communicate with each other, similar to the following:

Type Protocol Port Range Destination Description
NFS TCP 2049 10.20.0.0/16  

Create a VPC endpoint for DataSync in the Region of the destination Amazon EFS file system

  1. Open the Amazon VPC console in the Region of the destination Amazon EFS file system. Then, create an interface endpoint for DataSync.
  2. Configure the VPC endpoint's security group inbound rules to meet the network requirements for a VPC endpoint for DataSync, similar to the following:

Important: These example security group rules are based on the example VPC CIDRs. You must configure your security group rules based on your environment's VPC CIDRs.

Type Protocol Port Range Source Description
Custom TCP Rule TCP 1024 - 1064 10.10.0.0/16 DataSyncEndpoint
HTTPS TCP 443 10.10.0.0/16 DataSyncEndpoint

Create the DataSync agent in the source Region, and then activate the agent in the destination Region

Note: The following steps are for creating an agent using the DataSync console. You can also create a DataSync agent using the AWS Command Line Interface (AWS CLI).

  1. Open the DataSync console in the Region of the source Amazon EFS file system.
  2. From the navigation pane, choose Agents.
  3. Choose Create agent.
  4. For Service endpoint, select VPC endpoints using AWS PrivateLink.
  5. For VPC Endpoint, select the VPC endpoint that you created in the destination Region.
  6. For Subnet, select the subnet that your VPC endpoint is in.
  7. For Security Group, select the security group of the VPC endpoint.
  8. Choose Get key.
  9. Activate the agent in the same Region as the destination Amazon EFS file system. You can activate the DataSync agent using either its public IP address or private IP address. If you have only its private IP address, then you must activate the agent from a machine that's in the same subnet as the agent.

Create the locations for the source and destination Amazon EFS file systems

Create the source location:

  1. Open the DataSync console.
  2. From the navigation pane, choose Locations.
  3. Choose Create location.
  4. For Location type, select Network File System (NFS).
  5. For Agents, select the DataSync agent that you deployed.
  6. For NFS Server, enter the source Amazon EFS file system's subnet IP address.
  7. Choose Create location.

Create the destination location:

  1. Open the DataSync console.
  2. From the navigation pane, choose Locations.
  3. Choose Create location.
  4. For Location type, select Amazon EFS file system.
  5. For EFS File system, select the destination Amazon EFS file system.
  6. For Mount path, enter the mount path of the destination Amazon EFS file system.
  7. For Subnet, select the subnet where the destination Amazon EFS file system resides.
  8. For Security Group, select the destination Amazon EFS file system's security group.
  9. Choose Create location.

Create the DataSync task, and then run the task

Configure the task settings. After the task status shows as Available, then you can run the task. The task then runs through multiple steps. For more information on each phase of the task, see Understanding task execution statuses.


Did this article help?


Do you need billing or technical support?