Meeting record compliance requirements using SnapLock with Amazon FSx for NetApp ONTAP

Financial services customers that are regulated by the U.S. Securities and Exchange Commission (SEC) are required by regulations (17 CFR § 240.17a-4 and 17 CFR § 240.18a-6, or, briefly, “17a-4” and “18a-6”) to store their electronic books and records under certain protective measures. Both 17a-4 and 18a-6 require the ability to retrieve the original, unmodified version of a record.

One option to do this is to “preserve the records exclusively in a non-rewriteable, non-erasable format.” To address this record retention requirement, organizations often use write-once-read-many (WORM) technology that provides data immutability by preventing premature modification and deletion. You can use SnapLock for Amazon FSx for NetApp ONTAP to help meet these requirements and help comply with SEC regulations, as well as those of FINRA Rule 4511 and CFTC Regulation 1.31.

Read this article to understand how SnapLock works and how it can be used to address regulatory record preservation requirements. We outline some common configurations and deployment options for SEC-regulated customers to consider when designing compliant record storage systems. Customers subject to other regulations, such as the Health Insurance Portability and Accountability Act (HIPAA) or Payment Card Industry Data Security Standard (PCI DSS), may also benefit from these architectures.

How SnapLock can be used to help meet compliance obligations

17a-4 and 18a-6 outline both technical and non-technical processes required for compliance. This article describes technical configuration and deployment options for SnapLock that can help meet compliance requirements; however, it remains your responsibility to make sure your deployment and business processes align with your organization’s obligations under the regulations. For more information, see the Compliance Assessment Report for Amazon FSx for NetApp ONTAP.

SnapLock can help you to comply with key technical requirements laid out in SEC Rules 17a-4 and 18a-6, including:

1. Immutable storage of records
2. Automated verification of record integrity and accuracy
3. Audit logging of all administrative actions performed on records
4. A redundant system for record storage with the same protections as the original
5. Immediate access to records when requested by regulators

How SnapLock protects your files

From the perspective of an application or an end user, a SnapLock volume looks like any other shared storage using Network File System (NFS) or Server Message Block (SMB) mount points. Placing a file under WORM protection on a SnapLock volume can be done explicitly by a user or application, or automatically by the SnapLock volume itself (using the Autocommit feature).

SnapLock volumes can be deployed in two modes: Compliance and Enterprise. Compliance mode can help meet the regulatory requirement for immutable storage because it does not provide any mechanism to delete files prior to the expiration of their retention periods. Enterprise mode, however, can be configured to allow early deletion of files. Consider using Enterprise mode only for testing purposes or to help meet other organizational retention requirements.

For a review of the basic features of SnapLock Compliance volumes, let’s assume you’ve already created an Amazon FSx for NetApp ONTAP filesystem containing a SnapLock Compliance volume (as outlined in the FSx for ONTAP documentation) inside of an Amazon Virtual Private Cloud (Amazon VPC). Make sure to set an appropriate maximum retention on your SnapLock Compliance volume. You also need a way of mounting a SnapLock volume and issuing filesystem commands. For the purposes of this blog post, we’re using a Linux-based Amazon Elastic Compute Cloud (Amazon EC2) instance.

When you configure a SnapLock volume, the minimum, maximum, and default retention settings are used to enforce limits on what users or applications can specify for retention and to provide a reasonable default if they do not specify a retention period at all. This limits the risk of accidentally under- or over-retaining files. Similarly, specifying an Autocommit period setting enables WORM-protection automatically, even if the user or application doesn’t act to enable WORM-protection. In this example, the SnapLock Compliance volume is configured with the following settings:

default retention: 5 minutes
minimum retention: 0 minutes
maximum retention: 30 minutes
mode: compliance
Autocommit: enabled
Volume append mode: disabled

IMPORTANT: With SnapLock compliance volumes, be careful not to commit a file to WORM for longer than intended for the testing purposes in this post. The file(s), volume, and filesystem cannot be deleted until the end of the retention period you specified, during which time you are billed for these resources. To avoid this situation, make sure to set a short maximum retention period on your volume before executing any of the following steps. Maximum retention limits can be configured through the FSx for ONTAP console on the configuration page for the volume with the instructions in the FSx for ONTAP documentation.

Using the terminal of your EC2 instance, mount the volume with NFS:

$ sudo mkdir /snaplock_volume
$ sudo mount -t nfs svm-nfs-dns-name:/snaplock-volume-name /snaplock_volume

For demonstration purposes, let any user modify files when SnapLock is not active:

$ sudo chmod 777 /snaplock_volume/

Let’s write a file and then set a five-minute retention period by modifying the atime parameter of the file and then committing the file to WORM by setting the file to be read-only:

$ echo "Hello, world!" > /snaplock_volume/test.txt
$ touch -a -t `date -d'5 minutes' +%Y%m%d%H%M` /snaplock_volume/test.txt
$ chmod -w /snaplock_volume/test.txt

Try to delete the file before five minutes goes by:

$ sudo rm /snaplock_volume/test.txtrm: cannot remove '/snaplock_volume/test.txt': Read-only file system

You’re prevented from doing so, even using elevated permissions. You can also try to modify the file or its permissions before five minutes is up:

$ sudo echo "Some more data" >> /snaplock_volume/test.txt-bash: /snaplock_volume/test.txt: Permission denied

You’re prevented from doing this, too.

What about deleting the volume before five minutes goes by?

$ aws fsx delete-volume --volume-id=fsvol-0bd3bebxxxxxxxxx{
    "VolumeId": "fsvol-0bd3bebxxxxxxxxx",
    "Lifecycle": "DELETING"
}

After a few moments, check on the volume you tried to delete:

$aws fsx describe-volumes --volume-id=fsvol-0bd3bebxxxxxxxxx
{
    "Volumes": [
        {
            ...
            "FileSystemId": "fs-0bd3bebxxxxxxxxx",
            "Lifecycle": "CREATED",
            "Name": "vol1",
            ...
            "VolumeType": "ONTAP",
            "LifecycleTransitionReason": {
                "Message": "Cannot delete the volume because it contains unexpired WORM files."
            }
        }
    ]
}

We’re prevented from deleting the SnapLock Compliance volume “vol1” because the retention period has not expired on our test file. To verify this, wait another five minutes and try the deletion again. The volume deletion succeeds.

Now that it’s clear how SnapLock protects files by preventing deletion or modification once the file has been transitioned to WORM, we can architect a record storage system that addresses the technical requirements of the regulations.

Architecture options for a financial services record storage system

This article presents two configurations that address the preceding requirements: a stand-alone architecture, which can be used without a document management application, or a backing storage architecture for a document management application serving as an archive.

For either of these two architectures, the requirement for immediate access to records implies a need for a highly available and durable system spanning multiple Availability Zones (AZs). Note that both the stand-alone and the backing storage architectures, shown in the next section, specify a Multi-AZ deployment of FSx for ONTAP. Although a Single-AZ deployment does provision a pair of file servers in an active-standby configuration in separate fault domains in a single AZ, a Multi-AZ architecture provides continuous access to your records, as required by the regulations, even if an AZ becomes unavailable. If your organization has a requirement for a minimum distance between copies of its records, consider using a Multi-Region architecture.

The stand-alone architecture is simpler to deploy and maintain and allows you to use only native FSx for ONTAP features to address the SEC requirements. The trade-off is that it requires manual application of folder and file naming conventions and lacks the advanced management features of an enterprise content management application (like versioning, fine-grained access control, etc.).

The backing storage architecture allows SnapLock to be used as the storage for an application, such as an enterprise content management (ECM) application (e.g., IBM FileNet, OpenText Documentum) or an operational business application. This is useful when you already have an application that stores your records and you want to continue to use that application with a filesystem that supports WORM. Unlike in the stand-alone architecture, in the backing storage architecture, the technical features necessary to address the requirements are split between SnapLock and the application. This implies that the application might need to be designed to a higher availability standard than it might otherwise have.

For example, an application that presents a record access interface to an end-user typically shows files named in a way that a business user – and a regulator – can understand (e.g., “Q1 Balance Sheet” or “John Doe’s July account statement”). In many enterprise applications, these filenames are metadata that’s preserved in the application itself, typically in a relational database, while the files written to the underlying storage system may be named using a numerical or otherwise non-human-readable naming convention. In this case, if your business application is unavailable for any reason, you won’t be able to simply look at your redundant SnapLock volume and pull records upon request because you won’t have the metadata to search them. To solve for this, use a SnapLock volume in a Multi-AZ File System as the backing storage for a business application that is deployed in a redundant fashion across the same multiple AZs.

Stand-alone architecture

In the stand-alone architecture, shown in Figure 1, record creators and retrievers simply use a SnapLock volume over NFS or SMB in the same way they’d connect to any non-SnapLock shared filesystem. Users create folders and save records into those folders using whatever naming convention they choose.

Figure 1: stand-alone architecture built on a multi-AZ filesystem for replication

Backing storage architecture

In the backing storage architecture, users interact with a document management system or other business application that, in turn, interacts with the SnapLock volume through NFS or SMB, as shown in Figure 2.

Figure 2: backing storage architecture built on a multi-AZ filesystem for replication of the storage layer

How these architectures address the requirements

• Immutable storage of records: Whether you’re using the stand-alone or the backing storage architecture, configure the SnapLock volume with a minimum, maximum, and default retention period, and an appropriate auto-commit time. These cause all files written to the volume to be committed to WORM state, even if the user or application doesn’t explicitly do so. In the backing storage architecture, if the business application has the capability to interact with SnapLock directly through its API or filesystem metadata, then the application can apply a retention period other than the default (provided it’s between the minimum and maximum retention periods defined in the SnapLock volume’s settings).
• Automated verification of record integrity and accuracy: FSx for ONTAP stores a checksum with every block of data it writes. This checksum is automatically checked during retrieval. Additionally, individual hash fingerprints can be calculated on-demand for any file stored on an ONTAP volume using the ONTAP command-line interface or APIs.
• Audit logging of all administrative actions to records: ONTAP deploys a SnapLock audit volume that records the creation and deletion of SnapLock administrator accounts, modifications to the log volume, etc. Additionally, FSx for ONTAP has built-in capability to audit file access, and AWS CloudTrail can provide audit information on any AWS API actions applied to your SnapLock volumes. If you’re using the backing storage architecture with a business application, that application may have its own auditing capabilities that you must configure to capture all relevant recordkeeping events, including human-readable record names. Make sure you understand the auditing capabilities the application provides and whether they align with your requirements.
• A redundant system for record storage with the same protections as the original: In both architectures, standby storage infrastructure is deployed in a different AZ from the active infrastructure in the same AWS Region. Any changes written to your SnapLock volume are synchronously replicated across AZs to the standby with the same WORM protection. In the backing storage architecture, because the record retrieval process could require the use of the business application to find and retrieve records, it’s up to you to configure the application for redundancy.
• Immediate access to records when requested by regulators: The stand-alone architecture provides a highly available SnapLock volume to business users to retrieve WORM records without the need for intervention by technology administrators, even during failover. In the backing storage architecture, while the SnapLock volume automatically remains highly available, if your record retrieval process relies on the application and the data stored in it, then you should make sure that it also supports synchronous replication and a failover access path to retain immediate access to records without intervention by technology administrators.

Which architecture should I choose?

When selecting an architecture to address the regulatory requirements, there is an inherent trade-off between the two architectures outlined in this article. Although the stand-alone architecture is less complex to deploy and maintain, it lacks some of the advanced features that a large organization might require. Although the backing storage architecture is more complex and divides responsibility for record management between SnapLock and a business application, it can support more advanced features that large enterprises might require.

To guide your decision, consider using the standalone architecture if you:

• Store your regulated records in a file/folder structure on a shared storage system
• Do not rely on a document management application to store and retrieve your records

Consider using the backing-storage architecture if you:

• Seek to continue to use an existing content management or business application that contains regulated records
• Require more advanced document management features such as granular record access control or document versioning capabilities

Other things to consider

Although this post has addressed many of the basic components that can be part of a compliant record storage system using SnapLock, there are other areas to explore in future posts in this series. Some of these components are:

Legal hold

Legal hold is the ability to apply separate retention protection for records that are relevant to a particular legal action. This protection can be enabled or removed as the legal proceedings require. This functionality is supported by SnapLock Compliance volumes.

Event-based retention

What if you don’t know at the time you write a file how long it needs to be protected? Consider a requirement to store account statements immutably for three years after the closure of an account. How can you make sure that the file is under WORM protection for long enough to satisfy the SEC requirements and no longer? Event-based retention is the solution and SnapLock has the capability to help meet this requirement.

Migrations and hybrid architectures

NetApp ONTAP is widely used in the financial services industry and many readers of this article might already have on-premises SnapLock volumes that they’ve been using to help meet their compliance requirements. Using the SnapMirror feature built into ONTAP, these users can migrate their existing SnapLock deployments to AWS without any additional software. SnapMirror also facilitates hybrid deployments in which data is synchronized between on-premises and cloud-based record management systems while enforcing WORM policies.

Cleaning up

If you followed along and would like to remove resources used in this solution to avoid incurring any unwanted future charges, wait for the retention period to expire for all the files you stored. Once the retention periods have expired, follow the “clean up resources” section of the FSx for ONTAP user guide.

Conclusion

SnapLock on FSx for ONTAP gives SEC-regulated customers, and customers subject to regulations related to payments or healthcare, access to fully managed file systems in the cloud with compliance-grade WORM functionality. This article has proposed two architectures that regulated customers might use to help achieve a compliant record storage system and has demonstrated the basic features of SnapLock and its WORM functionality. SnapLock on FSx for ONTAP has been assessed independently by Cohasset Associates and is available today to help meet the record storage needs of regulated customers.