How can I troubleshoot slow performance when I copy local files to Storage Gateway?
Last updated: 2020-10-19
I want to copy local files to my Network File System (NFS) or Server Message Block (SMB) file share on AWS Storage Gateway, but the transfer is slow. How can I improve the upload performance?
Consider the following ways to improve the performance when you copy local files to a file share on Storage Gateway:
Note: A file gateway is an object-store cache, not a file server. This means that a file gateway's performance characteristics differ from those of file servers.
1. For the best performance, scale your workload by adding threads or clients. When you transfer a directory of files, a file gateway scales best when the workload is multi-threaded or involves multiple clients. Review your file-management tool and confirm whether the tool runs single-threaded uploads by default.
It's a best practice to use multiple threads or clients when you transfer small or large files. You get the highest MiB per second throughput when you transfer large files (tens or hundreds of MiB each) using multiple threads. Because of the overhead of creating new files, transferring many small files results in a lower MiB per second throughput when compared to the same workload with large files.
Note: For transfers of smaller files, measure the transfer rate in files per second, instead of MiB per second. The rate of file creation can take up a lot of the workload associated with transferring smaller files.
2. Tune your cache disk size to the size of the active working set. A cache that uses multiple local disks can parallelize access to data and lead to higher IOPS. For more information, see Performance guidance for file gateways.