Cost-optimized file storage with Amazon FSx for NetApp ONTAP and Komprise

As enterprises pursue digital transformation and smart operations, they’re challenged by the limitations of traditional file systems. Machine-generated data from connected systems and automation has pushed legacy storage solutions beyond their capabilities. In manufacturing, healthcare, logistics, financial services, and other industries, organizations need reliable access to data across globally distributed locations. These organizations face rising storage costs, poor data and usage visibility, and file access delays. Without modern, scalable storage solutions that offer a data lifecycle management strategy, organizations will struggle to achieve their smart operations goals.

Amazon FSx for NetApp ONTAP combined with Komprise offers a cost-optimized hybrid-cloud enterprise file storage solution. It provides organizations with high-performance edge and cloud-hosted access to active data while automatically moving inactive data to more cost-effective storage tiers over its lifecycle. Organizations can use the global infrastructure of Amazon Web Services (AWS), the high-performance hybrid-cloud file system of NetApp ONTAP, and the data movement specialization of Komprise to reduce storage costs by over 50%¹, eliminate manual data management tasks, prevent production delays due to slow file access, and gain insights from their data through integration with analytics and AI systems.

In this post, we explore how one such organization, a global manufacturing enterprise, modernized their file storage infrastructure by replacing their legacy file system with FSx for ONTAP and Komprise. They migrated nearly 3 PB of storage with no downtime, reduced storage costs by over 50%, consolidated over 2 PB of previously unmanaged shop-floor data, and integrated their file data with their enterprise data repository. Using their implementation as a model, we demonstrate how to implement a high-performance hybrid-hosted cloud native file storage solution that supports smart operations and optimizes costs with intelligent data lifecycle management. Whether managing operational data, enterprise file shares, or both, your organization can apply this solution for similar results.

File storage challenges to smart operations

A manufacturing company with over 100 sites worldwide was beginning an Industry 4.0 transformation to create smart factories by connecting their production systems, automating operations, and implementing data analytics for real-time decision making. They encountered high operating costs and lower production output due to their legacy storage infrastructure. Even though this file storage system used cloud-hosted storage, it lacked the design and functionality of modern file systems to allow them to capitalize on the benefits of cloud computing.Storage expenses had become unsustainable, with their legacy file system storage and access costs reaching $1 per GB annually.

The enterprise file data footprint was increasing by over 10% annually and new data streams from Industry 4.0 transformations at their manufacturing sites were writing over 1 PB of new production artifacts to file storage per year. Without data lifecycle management, all of the data, whether accessed daily or annually, consumed expensive primary storage space. Performance and visibility issues also negatively impacted their operations. At busy manufacturing sites, engineers faced frequent delays accessing files. Without file details and metadata for advanced querying, quality teams investigating production issues spent days sorting through historical data.To avoid high storage costs for data, which would have doubled or even tripled within 5 years, and performance concerns, the business’ manufacturing sites deployed on-premises NAS devices and storage arrays outside of the enterprise data ecosystem. These “shadow IT” storage arrays did not guarantee the necessary performance or reliability and did not make data available for smart factory operations. The use of local storage solutions compounded the problems. The business had no way to track these unmonitored and unmanaged solutions’ total storage costs, understand data usage patterns, or enforce enterprise-wide data retention and protection policies. Manufacturing teams wanted to use enterprise-wide data analytics and machine learning (ML), but their fragmented storage infrastructure made this impossible. Sites had to manually share data for cross-facility analysis, and production insights were trapped in local storage silos. Without a unified, performant storage solution, the business’ smart factory initiatives were stalled while competitors implemented data-driven optimizations.

These challenges demonstrated that a successful Industry 4.0 transformation needs more than just moving storage to the cloud. It needs a modern data management architecture that balances performance, cost, and accessibility across a global manufacturing operation.

A new “cloud-native” storage approach

To address these challenges, we designed a multi-tiered, cloud-based, and hybrid-deployed file storage solution using FSx for ONTAP, Komprise, and Amazon S3. This solution provides a globally scalable, high-performance file system with intelligent data lifecycle management capabilities. It enables smart operations with visibility into data and usage patterns, and integration capabilities for other systems.

Architecture considerations

When designing the solution, we focused on the following key business requirements:

1. Global accessibility for users across 100+ worldwide locations.
2. High-performance storage for active data, particularly large files such as production line images.
3. Automated tiering to move cold data to cost-effective storage.
4. Identification and archiving of infrequently accessed data.
5. Insights into data usage patterns and storage consumption.
6. Integration with existing systems and future smart factory initiatives.

Solution Architecture

The main functions and components of the solution are indicated in Figure 1:

Figure 1: Solution Architecture using FSx for ONTAP and Komprise for File Storage

The solution includes the following components:

• Cloud-hosted managed file system: FSx for ONTAP is a scalable, high-performance, AWS-hosted file system. It provides automatic storage tiering, multiprotocol (NFS, SMB) file access, data de-duplication, compression, and compaction, and data protection with snapshots and replication.
• Edge cache volumes: The FSx for ONTAP FlexCache cache volumes are optionally deployed at locations that need low-latency access to their frequently used files. They provide writable, persistent copies of requested data from the main file system.
• Data lifecycle: Komprise Intelligent Data Management, a third-party data solution, manages the files’ lifecycles and provides visibility into data and its usage. Using metadata, access patterns, and configured thresholds, it archives data to cost-effective Amazon S3 storage. Archived files are accessible from both the FSx for ONTAP or Amazon S3 file systems and can be restored through self-service or process driven data retrieval.
• Archival storage: Archived data is stored in cost-effective S3 Glacier object storage with its metadata preserved by Komprise. Different S3 Glacier tiers are available for an optimal storage cost and retrieval experience. S3 Intelligent-Tiering is optionally used for automated archival tiering into deeper, more cost-effective storage as data ages.
• Dedicated connectivity: AWS Direct Connect provides dedicated network connections from user locations to AWS, providing consistent, low-latency access to the FSx for ONTAP file system. This is necessary for FlexCache performance and improves the performance of file activities against the cloud-hosted file system.

Solution implementation

The business’ issues were immediate, but remaining prepaid licensing, migration complexity, and requirements for business continuity meant that they needed to be solved carefully. We used a phased rollout strategy to address the most impacted locations first.

We configured the FSx for ONTAP file system and an Amazon Elastic Compute Cloud (Amazon EC2) instance hosting Komprise in a centrally located Region for the business. Although multi-Region deployments are possible, a single Region was acceptable here.

Reliable, dedicated connectivity from user hubs into AWS was a requirement for this solution. The business already had the necessary Direct Connects in place and sized adequately. If they had not, then one of our first steps would have been to start implementing these to avoid lead times becoming a blocker.FlexCache cache volumes were deployed to regional hubs and major sites where the user needed low-latency file access. Some locations already had NetApp storage arrays that made the deployment process more streamlined, but most hosted their caches on virtual machines (VMs) running on KVM hypervisors. As data was requested from the FlexCaches, it would be replicated from the file system. More FlexCaches could be deployed later when requested by users.

The data migration was our biggest hurdle. Moving 3 PB of file data is not a challenge for the AWS cloud, but user bandwidth and cost are considerations. An AWS Snowball can move large amounts of data from user networks into AWS. AWS DataSync also provides online file transfer without any hardware to manage. However, most of the data was already hosted by a cloud provider, so we used Komprise Elastic Data Migration for the migration and change replication. Komprise delivered built-in resiliency, optimized migration speed (up to 25x) for large and small files, and chain-of custody reporting for the business’ regulated business segments. During the migration, Komprise’s Transparent Move Technology allowed users to find their data at the original location or in the AWS cloud.

When the data was moved into FSx for ONTAP by Komprise with its metadata intact, we ran the first archival operation based on the last-accessed date, which resulted in nearly 70% of old data being moved into S3 Glacier storage. The remaining 30% stayed in FSx for ONTAP and 60% of that was moved to the Capacity pool tier within 30 days, resulting in over 90% of the initial data being optimized.The business’ manufacturing sites agreed to migrate the remaining 2 PB of shop-floor data after the initial rollout demonstrated its performance, stability, and cost-effectiveness. Users could use the NFS and SMB options for connecting to data, thus redirecting their applications to the new file store’s DNS record, and decommissioning their old storage.

Business impacts of a storage transformation

The implementation of a modern file system using FSx for ONTAP and Komprise transformed the business’ file storage infrastructure and unblocked their Industry 4.0 evolution. It delivered over 50% in storage cost reductions, consolidated 2 PB of shop-floor data, and reduced file retrieval time for production issues from days to minutes. It also added production data insights for enhanced anomaly detection, historical trend analysis, and quality control.The business used the improved data and usage visibility offered by Komprise and FSx for ONTAP to track company-wide data growth for proactive capacity planning.

As the sites continued transforming into smart factories, files for new shop floor workloads were stored in FSx for ONTAP and garnered the same efficiencies without any other configuration. The deployment of NAS and storage arrays by manufacturing sites has since ceased in favor of the new modern, scalable file system.The new storage infrastructure is a foundational capability of the organization’s Industry 4.0 strategy for smart factories and offices. Teams can now consolidate IT and Operational Technology data into one file store with insight into the content and thus integrate production line data with enterprise knowledge to use it in analytics and AI-driven workloads. This capability, impossible with their previous storage solution, has prevented production downtime by proactively identifying issues, reducing product defect resolution time from hours to minutes by offering troubleshooting guidance, and saving the business hundreds of hours of undifferentiated storage activities annually.

Conclusion

In this post, we explored how a global manufacturing enterprise modernized their file storage infrastructure by replacing their legacy file system with FSx for ONTAP and Komprise. They migrated nearly 3 PB of storage with no downtime, reduced storage costs by over 50%, consolidated over 2 PB of previously un-managed shop-floor data, and unified their file data in an enterprise data repository.The business implemented a modern hybrid-cloud storage architecture using Amazon FSx for NetApp ONTAP and Komprise to transform their global file storage infrastructure from an operational burden into a lean manufacturing asset. The solution enabled their smart factory initiatives through improved data accessibility and management at an optimized price point with better performance and scalability.

The key to our success was designing this system by working backward from how the business uses their data and what its goals were. Rather than just migrating the existing storage to a new service, we worked with the business to design an offering that could be deployed to their internal customers with a scalable and customizable pattern. Using this approach, we provided a zero-downtime, cost-optimized migration that unlocked new capabilities necessary for an Industry 4.0 transformation while maintaining operational continuity for users and applications.If your organization is facing similar challenges with growing storage costs and complex data management requirements, then you can get started with this solution today:

1. Evaluate your file storage needs and challenges using the AWS Migration Evaluator Storage Assessment.
2. Learn more about Amazon FSx for ONTAP and its enterprise storage capabilities.
3. Explore how Komprise can help optimize your data lifecycle management.
4. Contact your AWS account team or an AWS Storage Competency Partner to begin planning your implementation.

For more information about enterprise file storage solutions on AWS, visit the AWS Storage Blog.

¹FSx for ONTAP and Komprise: FSx for ONTAP – 768TB data, 768TB backups, 30% SSD, 8GBps throughput, 320k SSD IOPs, 6m capacity tier read/write reqs, S3 Glacier IR – 4.2TB data, 10TB/mo retrievals, NetApp and Komprise third party licensing costs = 50% savings on $1/GB/yr.