Posted On: Sep 3, 2019
Amazon SageMaker customers can now use smaller Amazon FSx for Lustre file systems as the data source for training machine learning models. Until today, the smallest FSx for Lustre file system that could be created was 3.6 TBs. For training sets that are smaller than this size, customers can now create and use file systems as small as 1.2 TB.
Amazon FSx for Lustre is a high performance file system that works with Amazon S3 data and is optimized for workloads such as machine learning, analytics, and high performance computing.
With FSx for Lustre, customers can accelerate their Amazon SageMaker training jobs. Until support for Amazon FSx for Lustre was announced on August 27, 2019, Amazon SageMaker transparently downloaded a full training set from Amazon S3 to local file storage at the start of a training job, when using the File input mode. Now, with Amazon FSx for Lustre, customers can accelerate their File mode training jobs by avoiding the initial Amazon S3 download time. When linked to Amazon S3, Amazon FSx for Lustre file systems automatically copy objects from Amazon S3 to the file system when the objects are accessed for the first time. The same Amazon FSx for Lustre file system can also be used for iterative Amazon SageMaker training jobs running on the same datasets, preventing repeated downloading of common objects. With support for smaller file systems, Amazon FSx for Lustre can now be used for even more machine learning workloads.
Smaller Amazon FSx for Lustre file systems are available now in the following AWS Regions: US East (N. Virginia), US East (Ohio), US West (Oregon), US West (N. California), EU (Ireland), EU (Frankfurt), and Asia Pacific (Tokyo). For details on region availability please check the AWS region table.
Visit the documentation for more information and read the blog post for how to use the feature.