File Sync and Share with Qumulo Continuous Replication on AWS

By Kumar Nachiketa, Partner Solutions Architect at AWS

With the increased demand from various applications and massive growth of unstructured data from workloads such as high performance computing (HPC), video rendering, or editorial workflows, traditional Network Attached Storage (NAS) solutions are a poor fit for these workloads due to their limited scalability and performance constraints.

Interesting use cases emerge when scale-out file storage goes beyond the boundaries of customers’ datacenters and utilizes Amazon Web Services (AWS) for ultimate scalability, security, and cost benefits.

The AWS Cloud is an ideal location for these types of file-based, frequently shared workloads. AWS Partner Network (APN) Partner solutions use AWS services as building blocks to provide unique offerings to customers with file sync and share needs.

In this post, we’ll look at Qumulo, an APN Standard Technology Partner whose continuous replication feature gives you the ability to easily share data while enabling use cases like bursting workloads across locations.

Qumulo File Fabric (QF2) for AWS

Headquartered in Seattle, Qumulo can run on-premises or on the AWS Cloud and is an enterprise file storage system for organizations with data-intensive workloads that need storage at scale.

Qumulo File Fabric (QF2) is an excellent fit for use cases that require storage performance coupled with real-time statistics on how storage is being used at scale. Qumulo also fits where there’s an existing on-premises deployment and elasticity is required to extend the capacity of the workload to AWS.

Figure 1 – Qumulo File Fabric for AWS.

QF2 has a distributed architecture where many individual computing nodes work together to form a cluster with scalable performance and capacity with a single, unified file system.

Clusters start at four instances and can scale to 1,000 instances. Customers can access the file system through NFS or SMB protocols so that both Windows and Linux hosts can access the same file share.

What distinguishes Qumulo from other file sync and share solutions is its ability to provide real-time views of data usage and up-to-the-minute file system analytics for billions of files and directories. The use of aggregated metadata enables this view into the storage, even as it scales out.

Trends for capacity and performance for the cluster are visible to file-level granularity. The analytics also give administrators immediate access to the information needed to diagnose problems, as well as enable proactive planning for capacity and performance tiers regardless of the scale.

This instant view into cluster use can work in conjunction with the real-time quota feature.

For more details on the solution, go to https://qumulo.com/product/cloud/aws/.

Components

The AWS components required to build QF2 are shown in Figure 2. It’s comprised of Amazon Elastic Compute Cloud (Amazon EC2) instances, Amazon Elastic Block Store (Amazon EBS) SSD volume, and Amazon EBS throughput optimized volume (st1).

Figure 2 – QF2 building blocks on AWS.

Deployment

Qumulo is available through AWS Marketplace and offers a range of prepackaged editions (20TB Node, 5TB Node, and AWS free edition). The 5TB or 20TB node deployments allow you to create a cluster of different sizes, and the AWS free edition allows you to get started with QF2 on AWS.

The following are AWS CloudFormation template topologies for the AWS free trial edition, as well as a 20TB cluster built from four 5TB nodes.

Figure 3 – Free trial node deployment topology.

Figure 4 – 20TB node deployment topology.

QF2 AWS Free Trial Deployment

QF2 can be deployed using an Amazon Machine Image (AMI) or the CloudFormation template. In the example below, we use an CloudFormation template. To proceed with the deployment, visit AWS Marketplace, choose “QF2 for AWS – Free Edition” and then subscribe.

Figure 5 – QF2 is available on AWS Marketplace.

This will bring you to the series of steps to follow, details of which can be found here: Steps to deploy QF2 in AWS: Free Trial Instance with AWS CloudFormation.

Configuration

After the CloudFormation setup is complete, use the URL shown in outputs. You’ll see a login screen similar to the one below, and enter the user “admin” and password set during the CloudFormation setup earlier.

The QF2 dashboard provides an overview of the cluster status and activity. For example, you can look at the various setting status (e.g., cluster protection status, cloud-based monitoring, DHCP, NTP). You can also see capacity info performance related info (IOPS and Throughput).

Next, we have created an SMB share and mapped it as a network drive to a Windows host.

Create SMB Share

On the QF2 dashboard, click on Sharing > SMB Shares.

On the screen below, enter the necessary details to create an SMB share (sample details are filled below), and click Save.

The share is now created, and it appears as below on QF2 dashboard.

The SMB share mapped as a network drive looks like this:

When you’re deploying Qumulo in a production environment, consider these factors:

• What’s the best way to deploy Qumulo cluster so it can handle failures?
• How do I scale my cluster after the fact, both up and down?
• What performance and sizing guidance are available?

For details and various deployment models on AWS, refer to Qumulo Knowledgebase Documents or QF2 on Cloud: Various Deployment Models. For complex requirements, contact Qumulo or AWS.

Qumulo Use Case with FuseFX

FuseFX, which provides visual effects for all major Hollywood studios and concurrently works on more than 60 television shows, has many large visual effects rendering jobs that must meet the tight deadlines which characterize episodic television.

FuseFX knew AWS could provide the elastic compute resources they needed to support their workflows and fast-paced deadlines. The studio also identified they needed a modern, highly-scalable file storage system that could meet the high performance compute requirements of their workloads within AWS.

Already a Qumulo customer, FuseFX learned we were debuting QF2 for AWS and jumped at the chance to try it out. FuseFX set up a four-node QF2 cluster on AWS to support 1,000 32 Core Amazon EC2 instances. QF2’s peak performance was 40,000 IOPS and the peak throughput was 3.87GB/sec.

The flexibility to use the AWS Cloud both for large-scale rendering and storage helped FuseFX keep their commitments, accelerate production, and reduce the risk of downtime that can occur with on-premises render farms.

Read more about how FuseFX leveraged the Qumulo solution.

Summary

By utilizing the continuous replication feature in QF2, you can stitch together your namespace from both on-premises and Amazon EC2 instances.

With QF2 on AWS, workloads can be supported with the on-demand scale and elasticity of Amazon EC2 instances running with Qumulo. This avoids over-provisioning for individual projects, expensive rental fees, and wasted time to deploy permanent or temporary infrastructure on site.

Several other use cases, such as genomic sequencing, research imaging, or editorial workflows can benefit from the Qumulo solution. To get started, visit AWS Marketplace.

.

Qumulo – APN Partner Spotlight

Qumulo is an APN Standard Technology Partner. Their QF2 platform is a modern, highly-scalable file storage system that runs in the data center and the AWS Cloud. Mix large and small files, store as many files as you need.

Contact Qumulo | Solution Overview | Customer Success | Buy on Marketplace

*Already worked with Qumulo? Rate this Partner

*To review an APN Partner, you must be an AWS customer that has worked with them directly on a project.