AWS Storage Blog
Centralize data access using AWS Transfer Family and AWS Storage Gateway
More stringent and ever-changing compliance regulations have challenged financial institutions in a variety of ways, and regulatory compliance is frequently translated into specific operational requirements. One such operational requirement is that financial institutions must provide information accurately and in a timely fashion to external organizations. Apart from reporting, financial institutions must adhere to data retention guidelines after information has been reported. Retaining data for long periods of time in a cost-effective way, while still maintaining centralized access to data, is a common need for organizations faced with compliance requirements. Institutions that prepare for compliance requirements and are able to adapt quickly enjoy a distinct competitive advantage in the financial services market.
Let’s look at a few scenarios where financial institutions are required to generate reports and then must securely share those reports with recipients outside their organization.
- Credit bureau reporting: Credit bureaus receive information from lenders and creditors like banks, credit unions, and consumer credit card companies in Metro2 format. These reports contain information on debtors, such as whether they are paying their debts on time, whether there are any missing payments, if a bankruptcy has been filed, if there is a co-owner on the account — each of which can affect an individual’s credit score. Debtors who find inaccurate information on their credit reports can file a dispute with the credit bureaus or with the creditors who provided the incorrect data. As mandated, institutions have to retain this data for as long as 7 years.
- Office of Foreign Assets Control (OFAC) reporting: To maintain compliance with the Office of Foreign Assets Control sanctions programs, businesses and organizations are required to report certain transactions to the OFAC. These businesses and organizations not only have to report such transactions, but they also have to retain these transactions and should be able to make them available for examination, if requested, for at least 5 years after the date of such transactions.
- Regulatory reporting: In order to provide investor protections, promote effective capital markets, and address issues that could pose a “systemic risk” to the financial system and the economy, there are a number of established regulations that must be followed and reported by market participants to regulators. Organizations must also retain trade and transaction data for at least 5 years or longer, as mandated.
In this blog post, we show you how you can use AWS Storage Gateway for files (File Gateway) to store financial reports generated by applications in your on-premises environment in Amazon S3. We also show you how you can securely make these reports available to external entities using the AWS Transfer Family. Using File Gateway and the AWS Transfer Family together, you can reduce your on-premises infrastructure while maintaining your existing file transfer workflows. Furthermore, by moving your file transfer servers to AWS Transfer Family, you eliminate the use of an on-premises DMZ for third-party access.
The solution described in this blog post focuses on a specific use case for generating and sharing financial reports. The same solution can be applied to any workflow that requires sharing data between applications or users that use transfer protocols such as SFTP, FTP, or FTPS and storage protocols such as NFS or SMB.
Services Overview
Let’s review the two key services that are used to build the solution: AWS Transfer Family and File Gateway.
AWS Transfer Family
The AWS Transfer Family provides fully managed support for file transfers directly into and out of Amazon S3. The service supports the SFTP, FTPS, and FTP protocols and each AWS Transfer Family server scales to support thousands of concurrent connections while providing redundancy across multiple Availability Zones within a Region.
The AWS Transfer Family enables seamless migration by allowing you to import host keys, use static IP addresses, and use existing hostnames for your servers. With these features, user scripts and applications that use your existing file transfer systems continue working without changes. The AWS Transfer Family also provides several options for user authentication and enables you to integrate with your existing authentication systems, further easing migration. Because data is stored in S3, you can take advantage of the additional benefits that the AWS Cloud offers around machine learning, analytics, and more.
File Gateway
Our solution uses File Gateway to provide on-premises applications low-latency access to data stored in Amazon S3. Because File Gateway uses industry standard storage protocols such as NFS and SMB, customers can reduce, or even eliminate, on-premises storage for their reports without changing the applications that generate those reports.
File Gateway stores all data durably and reliably in Amazon S3 and uses a local cache to provide low-latency access to frequently accessed data. File Gateway can be deployed on-premises as a hardware or virtual appliance (VMware, Hyper-V, KVM). It can also be deployed in AWS as an EC2 instance. All data transferred between the gateway and AWS is encrypted in flight and Amazon S3 encrypts data at rest.
You can use Amazon S3 Object Lock to store objects using a write-once-read-many (WORM) model, letting you prevent an object from being deleted or overwritten. S3 Object Lock helps you meet regulatory requirements that require WORM storage, and adds another layer of protection against object changes and deletion. Additionally, to reduce costs, historical reports can be archived by using S3 Lifecycle to transition data to lower-cost storage classes, such as Amazon S3 Glacier or S3 Glacier Deep Archive.
Solution overview
The preceding diagram shows how File Gateway and AWS Transfer Family can be used together for your reporting workflow.
- Your application servers generate reports and save them to File Gateway through an SMB or NFS share.
- The reports are written immediately to the local cache on the File Gateway.
- The reports are uploaded in the background from the File Gateway cache to your Amazon S3 bucket. There the reports are stored durably, securely, and reliably, preserving metadata such as file ownership, permissions, and timestamps in the user metadata section of each S3 object. File Gateway encrypts all data in transit using HTTPS.
- With the reports in the S3 bucket, they can then be made available to third parties and other external organizations via common file transfer protocols such as SFTP, FTPS, or FTP using AWS Transfer Family.
- To create a more automated workflow, applications that generate the reports and save them to File Gateway can integrate with the NotifyWhenUploaded API to trigger an Amazon CloudWatch event when a file is written to Amazon S3. This event can be used in conjunction with other AWS services, such as Amazon SNS, to notify your external parties that the data is available to be retrieved.
- Cloud-native applications can listen for these events to perform additional automated processing, deriving further business value.
Getting started
In this section, we walk you through the high-level steps to get started in building this solution. First, you deploy a File Gateway and configure access to your S3 bucket. You then deploy an AWS Transfer Family server and configure user access to the same S3 bucket.
Deploy File Gateway
As mentioned earlier, a File Gateway can be deployed on-premises or in AWS. To take advantage of the low-latency cache the gateway provides, you want to deploy it as close to your application servers as possible.
Once the gateway is deployed and activated, you then create an NFS or SMB file share, depending upon whether your on-premises application is running in a Linux or Windows environment. The file share will be backed by an S3 bucket located in the same Region in which the gateway was activated.
Once the file share has been created, you can then mount it from your application servers using standard methods. When your application writes a file to the file share, it is written to the local cache and then uploaded in the background to the S3 bucket. When you read a file, the file data is read from cache, if available, or it is read from S3.
Deploy AWS Transfer Family
With the File Gateway successfully deployed and connected to your application servers, the next step is to deploy an AWS Transfer Family server in the same Region where your S3 bucket is located. When creating your server, you must choose which protocols to make available, the identity provider type, and the endpoint type. For maximum flexibility, we recommend using a VPC hosted endpoint type. This allows for both public and internal traffic, and enables you to keep your endpoint IP address static and restrict access to your server. Once you create your server, it takes a few minutes to deploy.
Access to the S3 bucket from AWS Transfer Family is configured on a per-user basis. For each user session, the AWS Transfer Family server assumes an IAM role that you provide that dictates how the user can access the bucket. In addition to the IAM role, you can provide a scope-down policy to further control access and use logical directory mappings to safely lock down your users to designated folders in your S3 bucket. If you are using the service-managed identity provider, then you create your users through the AWS Transfer Family service. If you are using the custom identity provider, then you must implement a Lambda function to connect to your authentication provider.
With a server deployed and user authentication configured, you can access your AWS Transfer Family server using standard file transfer programs that support your selected protocols. When reports are generated by your application servers and written to File Gateway, they are accessible to external parties through your AWS Transfer Family server.
Hands-On Workshop
To help you get started quickly, we’ve created a simple workshop on GitHub that you can use to run through the solution outlined in this post on your own. In the workshop, you use AWS CloudFormation to deploy an AWS Transfer Family server, a File Gateway, and an S3 bucket. You use a Linux server to access these resources, and then you walk through how to configure them and transfer files between them. The workshop takes about 30 minutes to complete.
Conclusion
In this blog post, we talked about some of the data transfer and storage challenges faced by customers in the Financial Services industry, particularly when using applications that must remain on-premises, while needing to make generated reports accessible to external entities. We’ve shown you how to use File Gateway and AWS Transfer Family to centralize access to your financial reports in Amazon S3 and make them available to third parties and external entities. Using these services, you can reduce your on-premises storage and server footprint, minimize operational overhead, and archive your reports for long-term retention, while still maintaining your existing workflows. We also provided you with a workshop to run through the solution on your own.
To learn more about the services covered in this solution, check out the following links:
- Getting started with AWS Storage Gateway
- Getting started with AWS Transfer Family
- Getting started with Amazon S3
To try out the solution mentioned in this blog post, use the following workshop link:
Thank you for reading and feel free to leave any comments in the comments section.