Persistent File Storage for Modern Applications
Why is persistent file storage important for modern application development?
Development teams are modernizing their applications by adopting containers, serverless, and microservices-based architectures. Because containers are transient in nature, long-running applications can benefit from keeping state in durable storage. Distributed applications like machine learning training, and web serving, benefit from a shared storage layer. Amazon Elastic File System (Amazon EFS) is a simple, serverless, set-and-forget, cloud native file system, enabling you to build modern applications, persist and share data from your AWS containers and serverless applications, with zero management required.
Data persistence for your serverless applications
Serverless computing enables you to be more agile while spending less time dealing with the security, scalability and availability of your applications. At the same time, modern data-intensive applications require fast access to large volumes of shared data. AWS Lambda lets you run large-scale, and mission-critical serverless applications. Amazon EFS provides highly available and durable serverless storage for those applications, and simplifies the sharing of data that needs to persist beyond and between executions of Lambda functions and AWS Fargate tasks. This powerful combination is ideal for building machine learning applications, loading large models, libraries, and other reference data, processing and backing up large amounts of data, hosting web content, and developing internal build systems.
Benefits
Simple
Amazon EFS attachments are configured in the application metadata,
such as the Amazon Elastic Container Service (Amazon ECS) task definition or the Kubernetes persistent
volume, including connectivity, so developers can focus on their applications,
not infrastructure.
Elastic
Amazon ECS, Amazon Elastic Kubernetes Service (Amazon EKS), AWS Fargate, and Amazon EFS are fully elastic, scale up and down rapidly based on demand.
Available and Durable
Amazon ECS, Amazon EKS, AWS Fargate, AWS Lambda, and Amazon EFS are regional services. You can build applications that span multiple availability zones, with automatic failover.
Secure
AWS provides a secure cloud computing environment. Access to Amazon EFS can be controlled based on the IAM role of the Amazon ECS task.
Cost-optimized
Pay only for the storage and compute you use. Amazon EFS scales on demand from zero to petabytes with no disruptions, growing and shrinking automatically as you add and remove files, and Amazon ECS, Amazon EKS, and AWS Fargate Cluster Auto Scaling enables capacity to grow and shrink to meet demand.
Use Cases
Web serving and content management
Web serving and content management systems require shared data access across multiple containerized application instances, data persistence, and data durability. Examples include applications like WordPress and Drupal, that benefit from scaling out to multiple instances for performance and redundancy, and need to share uploads, plugins, and templates.
Stateful microservices
A stateful microservice is part of a loosely coupled application that needs to remember some things about its state each time it runs, as in the way a weather application remembers your home city. For these modern applications, Amazon EFS is a data foundation, operating in concert with containers and serverless technologies to reliably and consistently deploy to AWS, allowing data to persist application state.
ML and AI
Containers are a way to deploy machine learning (ML) training jobs, inference endpoints, and tools that run consistently in multiple environments. Using containers can solve problems such as consistency, portability, and dependency management that complicate matters for data scientists and developers whose focus should really be on their code, datasets, and training models, not on infrastructure. ML tools like Amazon SageMaker Notebooks, as well as open source tools like Jupyter use Amazon EFS to manage data scientist home directories. Amazon FSx for Lustre can be used for your most compute-intensive ML training jobs where the highest throughput is required.
Case Studies
Blog posts
Running WordPress on Amazon EKS with Amazon EFS Intelligent-tiering
Mount Amazon EFS file systems cross-account from Amazon EKS
Analyze terabyte-scale geospatial datasets with Dask and Jupyter on AWS
Choosing between storage mechanisms for ML inferencing with AWS Lambda
Johnson & Johnson reduces analysis time by 35% with their data science platform using Amazon EFS
Deploy multiple machine learning models for inference on AWS Lambda and Amazon EFS
Migrate Resources Between AWS Accounts
Virus scan S3 buckets with a serverless ClamAV based CDK construct
Manage your Digital Microscopy Data using OMERO on AWS
Analyze controlled unclassified data with SAS Viya EKS on AWS GovCloud (US)
Persistent Storage using EFS for EKS on Bottlerocket
Deploy serverless Drupal applications using AWS Fargate and Amazon EFS
Event-driven data transfer to container-shared storage on AWS
Scaling RStudio/Shiny using Serverless Architecture and AWS Fargate
Introducing Amazon EFS CSI dynamic provisioning
Running stateful workloads with Amazon EKS on AWS Fargate using Amazon EFS
AWS re:Invent recap: Modernize your applications with Amazon EFS
Persistent storage for container logging using Fluent Bit and Amazon EFS
New - AWS Fargate for Amazon EKS now supports Amazon EFS
Deploying Jenkins on Amazon EKS with Amazon EFS
New – A Shared File System for Your Lambda Functions
Using Amazon EFS for AWS Lambda in your serverless applications
Developers guide to using Amazon EFS with Amazon ECS and AWS Fargate – Part 1
Developers guide to using Amazon EFS with Amazon ECS and AWS Fargate – Part 2
Developers guide to using Amazon EFS with Amazon ECS and AWS Fargate – Part 3
Digital transformation at Discover using AWS Storage solutions
Amazon Elastic Container Service & AWS Fargate, now support Amazon Elastic File System
Best Practices for Using Amazon EFS for Container Storage
Deploy Stateful Docker Containers with Amazon ECS and Amazon EFS
Read Docker's blog »
Read about the latest launches and best practices in the Storage blog »
Read about the latest launches and best practices in the Containers blog »
Resources
Learn more about Amazon EFS, scalable, elastic, cloud-native file system
Persistent storage on containers using Amazon EFS (Level 200)

Machine Learning Inference with AWS Lambda and Amazon EFS (Level 200)

AWS Online Tech Talk: Modernize Content Management Systems with Containers

Modernize Your Applications with Containers, Serverless, and Amazon Elastic File System, an on-demand webinar

Persistent Storage for Amazon ECS and AWS Fargate Using Amazon EFS, an on-demand webinar

Read the Tech Target white paper
