Faculty Uses Amazon EFS to Scale Innovative Machine-Learning Platform

Faculty began using Amazon Elastic File System (Amazon EFS), a cloud-native shared-file system and scalable file storage, to overcome its challenges. “Amazon EFS was the best solution for our shared storage needs. Using this service, we no longer have to provision storage or worry about managing the network file system ourselves,” says Stevenson.

All Faculty data is stored on a single Amazon EFS file system, with one subdirectory per data science project. Amazon EFS gives Faculty users a shared workspace for code and data, which can be mounted on each customer’s data science environment. Any changes to a machine-learning model are reflected immediately on collaborators’ and data scientists’ machines.

Faculty also uses AWS CloudFormation scripts to provision code. “Using AWS CloudFormation, we have completely automated the deployment of Faculty,” Stevenson says.

Using Amazon EFS, Faculty no longer needs to manage file storage for data science projects. With all data stored in a central file system, the company has a reliable, secure, and highly available solution for storing data. “The sign of a great technology is that you forget it’s there,” says Stevenson. “Amazon EFS just works. It requires zero maintenance on our end. Instead of trying to build and manage our own storage system, which would be technically challenging, we can rely on Amazon EFS to manage it for us.”

By migrating the Faculty file-storage system to Amazon EFS, engineers can spend more time innovating. For example, Faculty built a new feature that enables data scientists to spin servers up and down on demand and to train machine-learning models in parallel. “This is something data scientists have struggled to do on their own, which we were able to accomplish due to the time saved by moving to Amazon EFS,” says Stevenson.

Faculty is taking advantage of the elastic scalability of Amazon EFS to better support its customers’ growing data storage needs. Recently, Faculty worked with the U.K. government’s Home Office on a machine-learning project to automatically identify terrorist propaganda online. During this project, the Home Office stored 10 terabytes of video data on the Faculty platform—representing thousands of videos. The data scientists used Faculty to create and train a machine-learning model that detected 94 percent of propaganda and automatically rejected extremist content. “The data scientists working on the project didn’t have to worry about how to store 10 terabytes of data; Amazon EFS easily accommodated that scale,” says Brookes. Additionally, Faculty did not have to do any provisioning or resizing of the file system to handle the increased volume of data during batch workloads.

By automating the provisioning of Amazon EFS file systems through AWS CloudFormation, Faculty can more quickly deploy the Faculty platform for new customers. “We deploy Faculty to new customers weekly, and because we’ve built the provisioning into code through AWS CloudFormation, everything is automated,” says Stevenson. “We are saving deployment engineers days of effort every time we deploy the solution.”

Faculty has also improved collaboration between data scientists. “There is more consistency in the data science development process because everyone is working with the most updated code base thanks to Amazon EFS,” says Stevenson. “This ultimately helps us deliver a better product to our customers.”