How AWS Partners Are Utilizing Amazon S3 to Help Customers Solve for Scale

By Henry Axelrod, Principal Solutions Architect at AWS

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. It was released back in 2006, on Pi Day, March 14.

When Amazon S3 first launched, its primary purpose was to provide highly scalable storage for web developers. Since then, the service has evolved and adoption has increased significantly, with millions of active customers and tens of thousands of data lakes now built on Amazon S3.

Designed to store and retrieve any amount of data from anywhere, Amazon S3 includes seven storage classes, lifecycle management, replication, and many other features.

This year, as we celebrate Amazon S3’s 15th birthday, we are excited to share stories of how AWS Partners are using S3 to provide a wide variety of solutions to address customer needs. It’s rare to find an ISV solution these days that does not support Amazon S3 in some way. AWS Partners have helped customers all across the world ingest exabytes of data into S3, and some AWS Partners have built entire businesses on it.

“MSP360 would never exist without the launch of Amazon S3,” says Brian Helwig, CEO at MSP360. “Our customers demand the reliability and durability AWS Storage provides, and they count on Amazon’s continued innovation and flexible pricing models to meet their business needs. Now, we are an integral part of the S3 ecosystem and look forward to another 15 years.”

In this post, let’s take a look at some cases in which AWS Partners are helping customers solve for scale on Amazon S3.

Backup and Restore

Backup is one of the most common use cases for customers starting out with Amazon Web Services (AWS). Customers that are used to traditional on-premises backup infrastructure, such as disk and tape, find many benefits by moving their backups to AWS and Amazon S3.

One of those benefits is the scalability of both performance and capacity. When using Amazon S3 for backups, backup administrators no longer have to worry about late-night pages because backups are failing due to no more disk capacity.

With AWS, backup administrators don’t have to wait for their next order of tapes to arrive to ensure they have enough to keep up with the growth of their business. They also don’t need to worry about having the capacity to be able to make next week’s offsite copy.

With data generated in the world doubling every two years, keeping up with growth is more important than ever.

Veeam added support for Amazon S3 a few years ago as part of their Scale-Out Backup Repository, and recently added Amazon S3 Glacier support to enable customers to scale even more cost-effectively. “Amazon S3 has provided a simple, scalable, and secure storage offering for Veeam customers to ensure digital resilience,” says Danny Allen, Chief Technology Officer at Veeam Software.

Backup vendors have been using the familiar and simple S3 APIs for years. Many AWS Partners started with gateway appliances, but now most of them support writing to S3 directly. Almost every major backup vendor supports Amazon S3 in some way, including companies like Clumio, Cohesity, and Druva that have built full software-as-a-service (SaaS) backup solutions on AWS.

“Druva designed its data protection as a service around Amazon S3 storage, and its durability, simplicity, and scale enabled us to build an industry pioneer in data protection,” says Stephen Manley, Chief Technology Officer at Druva. “Almost a decade later, with the power of S3 behind us, Druva runs more than 6 million daily backups on an award-winning platform that keeps customer data safe, secure, and protected.

“Choosing AWS’s industry-leading cloud storage has been foundational to our success,” adds Manley, “and we are incredibly excited about the future to come.”

It’s easier than ever to move your backups to AWS. As you can see in the diagram below, starting to use Amazon S3 can be as simple as configuring your existing backup solution to point to your S3 bucket in your AWS account.

Figure 1 – Basic backup architecture with AWS.

Customers can also run most Partner solutions on AWS to restore data from on premises to AWS, or even back up workloads running on AWS. Customers can also use an AWS Partner SaaS backup solution to avoid having to run any backup infrastructure for protecting both on premises and AWS workloads.

Parsons, an engineering firm with multiple global sites and hundreds of petabytes of data, has been able to eliminate their tape infrastructure by using Commvault to protect PBs worth of data on Amazon S3. Read the full case study about Why Parsons Considers its Data Protection Strategy a Business Advantage.

Long-Term Retention

There are many different ways customers use backups with AWS, including for offsite backup, but one of the ever-increasing use cases is long-term retention.

Customers generally have two main types of backups: operational and long-term retention backups. Operational backups are most commonly used for restore purposes. These can be the latest 7, 15, or 30 days of backups depending on an organization’s policies. Rarely are backups used for normal operational restores after 30-60 days.

However, either due to organizational policies or compliance requirements, many customers need to maintain backups for a much longer length of time. Backups that go beyond the operational window are typically referred to as long-term retention. Amazon S3 has become the best destination for storing long-term retention data.

Back in 2012, AWS first introduced Amazon S3 Glacier which was initially a separate storage service designed for archiving. Since then, S3 Glacier has become an even more integral part of Amazon S3, as it provides a lower-cost option for customers. AWS is continually introducing new features and services for S3 Glacier based on customer demand.

In 2019, AWS introduced a game changing Amazon S3 storage class called S3 Glacier Deep Archive. Since customers were working with AWS Partners supporting S3 when S3 Glacier Deep Archive was released, storage partners supported it quickly to meet long-term archival needs.

Support happened rapidly for S3 Glacier Deep Archive because AWS Partners knew there was significant customer demand for long-term scalable storage at a lower price point. Cloudberry Lab (now MSP360), Commvault, and Veritas were even launch partners for S3 Glacier Deep Archive, so customers using these backup solutions were able to take advantage of the new storage class on day one.

For more information on AWS Partner support for S3 Glacier Deep Archive, see my blog post on Partner Backup Solutions and Amazon S3 Glacier Deep Archive.

Since the launch of S3 Glacier Deep Archive, many additional AWS Partners have added support. Customers can now store data for as low as $1/TB per month, which is often a lower cost than offline tape. Some AWS Partners even support using deduplication with the S3 Glacier storage classes to further reduce costs.

“Veritas began our support of Amazon S3 10 years ago, and now we are several iterations in on bringing new innovation to our customers,” says Lisa Erickson, Director, Product Management at Veritas. “With S3 integration, we’ll continue to add increased performance, efficiency, scalable, and secure data protection for our customers. Veritas looks forward to our continued innovation and collaboration with AWS.”

Profuturo Group, a Mexican retirement fund administrator, was able to move their long-term retention backups to Amazon S3 Veritas NetBackup and save 50 percent on storage costs. They were able to use NetBackups deduplication technology to save on the amount of data they needed to send on the wire and the cost of storing the backup.

Read the full case study about How Profuturo Breathes New Life into Data Protection.

Archiving

Archiving is a popular use of these storage classes. While it has many similarities to long-term retention and may use the same software, there are some differences.

The primary difference is that an archive generally represents the authoritative version of data, and in some cases is the only copy, whereas backups are typically a copy of data and sometimes even the second and third copy.

Thanks to Amazon S3 being designed for 99.999999999% (11 9’s) of durability, it’s a great location for storing any archives. S3 storage classes also provide protection across three or more AWS Availability Zones (AZs), including the S3 Glacier and S3 Glacier Deep Archive storage classes.

Amazon S3 also offers S3 One Zone-IA that stores data in a single AZ and costs 20 percent less than S3 Standard-IA. S3 One Zone-IA is ideal for customers who want a lower-cost option for infrequently accessed data but do not require the availability and resilience of S3 Standard or S3 Standard-IA.

What separates S3 from other cloud providers, is that an Availability Zone is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region. AWS Partner solutions make it easier to move customer data to S3 for archiving use cases. This is because AWS Partners want to ensure customers can easily and cost-effectively store as much data as they need using the Partner’s solutions.

“Over the years, Amazon S3 has enabled Commvault to deliver and evolve secure, cost-optimized backup and archival storage, cloud migration, and disaster recovery services to our joint customers,” says John Tavares, Vice President of Channel and Alliances at Commvault. “As our customers embark on their cloud journey, Amazon S3 plays a critical role in providing storage for all stages of the application lifecycle.

“Customers leveraging Commvault with Amazon S3 can reduce their S3 storage and egress fees and enable simplified, self-service data management for their business,” adds Tavares.

Primary Storage

In addition to backup and archiving, Amazon S3 is increasingly becoming a destination for primary storage partner solutions.

AWS Partners are utilizing S3 in various ways in conjunction with their solutions to help customers. Some have created solutions that provide front-end gateways to serve traditional network attached storage (NAS) use cases on premises while being backed by the virtually unlimited capacity of S3. This helps customers avoid traditional capacity management challenges.

AWS Partners have also enabled customers to open up new modern use cases like global collaboration and enterprise file sync and share (EFSS). For an example of EFSS, see my blog post about Sharing Files Across Your Organization with CTERA’s File Sync and Share on AWS. Amazon S3 is a great location for this type of data as you can store data in a location that can be easily accessed from anywhere.

For an example of an AWS Partner solution that enables global collaboration, see my blog post on How to Get the Global Scalability of AWS Storage at Local Speed with Nasuni.

“When Amazon S3 debuted 15 years ago, people would have thought you were crazy to build a file system on top of object storage,” says Russ Kennedy, Chief Product Officer at Nasuni. “S3 enables us to offer a cloud-native global file storage platform that scales to petabytes, unlimited numbers of files, versions, shares, and snapshots. S3’s semantics enable us to store every file change as an immutable, infinite version history, ensuring that customer data can never be corrupted. None of this would be possible without S3.”

Movado is a large manufacturing and design company and was able to use Nasuni and S3 to enable cross-office collaboration across seven locations in the U.S., Europe, and Asia. Since this was based on S3 compared to a traditional NAS, they were able to gain virtually unlimited file storage and reduce cost by 50 percent.

Read the full case study about How Movado Unifies Global Design and Manufacturing Teams with Nasuni.

Many AWS Partners have added the ability to tier data from their primary storage solutions to Amazon S3 as a cost-savings method. Some solutions include tiering local snapshots with solutions like NetApp FabricPool, or Pure Storage CloudSnap. NetApp FabricPool can also tier colder data to S3.

Many primary storage Partners also provide solutions that run on AWS and provide the ability to replicate data using their native replication features. Being able to replicate data to AWS and run Partner virtual storage arrays on AWS has allowed customers to replace secondary data centers.

“NetApp’s portfolio of solutions combined with Amazon S3 helps thousands of customers worldwide increase public cloud consumption and accelerate digital transformation,” says Anthony Lye, VP and GM at NetApp Public Cloud Services. “Our strategic relationship allows us to innovate faster with increased flexibility both in the cloud and on premises.”

“We’re building amazing solutions like Cloud Volumes ONTAP that provide customer value with NetApp Cloud Tiering of cold data to S3 to reduce customer’s cloud waste and best optimize their cost and storage resources,” adds Lye.

In addition to traditional storage use cases, AWS Partners that generate and process data in the logging, data and analytics, machine learning, media and entertainment, health care and life sciences, and other areas have made significant uses of Amazon S3 as their main storage location.

Amazon S3 can provide massive parallel performance that allows AWS Partners to process huge datasets at a cost-effective rate. For example, Weka.io was a central aspect to help Untold Studios Unlock Creative Potential by Running Entirely on AWS.

AWS Storage Competency Partners

The AWS Storage Competency recognizes validated AWS Partners who have built storage solutions for AWS and have demonstrated proven success with customers.

AWS Storage Competency Partners provide industry-leading consulting and technology services for a variety of use cases, including backup and restore operations to, from, and within the AWS environment; primary storage using IP File or Block protocols and object storage; active and passive data archiving capabilities; and business continuity/disaster recovery (BCDR) solutions.

For a complete list of validated Partners, see the AWS Storage Competency page.

Summary

Amazon S3 has a long history of serving customer and AWS Partner needs. Today, it has seven different storage classes to meet the needs of virtually any workload. AWS Partners have been able to build solutions on top of S3 across a variety of use cases, including but not limited to backup and restore, long-term retention and archive, and primary storage.

AWS Partners that build their solutions on top of Amazon S3 are able to help customers gain access to the benefits of S3, such as virtually unlimited scalability, 11 9’s of durability, and global accessibility.

If you’re looking to improve the scale, durability, or cost of your on-premises solution, consider a proof of concept (PoC) of Amazon S3 with an AWS Partner solution. These can solve for many use cases and enable an easy transition to S3.

To get started, check with your cloud team or your AWS account team if you already have an AWS account, or sign up for a new account which includes AWS free tier.

With Amazon S3 you get virtually unlimited scalability but you only pay for the storage you consume, which makes it very easy and cost-effective to start testing out your use case. Once you setup an account you can be sending data to Amazon S3 within minutes.