Enabling SMB access for serverless workloads with Amazon FSx

Many customers adopting a cloud-first strategy prioritize serverless technologies and cloud file storage that offer automatic scaling, built-in high availability, and a pay-for-use billing model to increase agility and optimize costs. As customers adopt serverless architectures, they may need shared data access on a persistent storage layer. While AWS Lambda includes a 512-MB temporary file system for your code, this is an ephemeral scratch resource not intended for durable storage. SMB integration for Lambda allows you to share data across function invocations, read large reference data files, and write function output to a persistent and shared store.

In this post, I focus on integrating AWS Lambda with Amazon FSx for Windows File Server. You can also use the solution I demonstrate with any SMB-compatible file system including Amazon FSx for NetApp ONTAP. This solution allows you to run file and directory operations such as listing, storing, retrieving, and deleting files and folders on file shares hosted on Amazon FSx for Windows File Server, Amazon FSx for NetApp ONTAP, SMB-compatible data stores running on an on-premises Windows file server, or on Amazon EC2 Windows file servers.

Use cases

Distributed applications like content management and web serving, home directories, and business-critical workloads can benefit from shared file storage. The following are some common serverless use case examples:

• Windows home directories and department shares, and using Lambda functions to write report files to a user’s home directory or department shares. You can use Amazon FSx for Windows File Server or Amazon FSx for NetApp ONTAP for a wide variety of use cases, such as hosting home directories, department shares, content management, and highly available Microsoft SQL Server deployments.
• Hosting content in Amazon FSx and performing automated serverless actions in response to file change events. You can use file access auditing support on FSx for Windows File Server to generate event logs in Amazon CloudWatch whenever a file change event occurs. You can also trigger a Lambda function upon such a CloudWatch Log entry being generated, and perform an operation on the changed file (for example, you can copy the changed file to another data source where the latest content needs to be reflected).
• Sharing files with your organization through a public facing SFTP server. You can leverage AWS Transfer for SFTP with data stored in Amazon S3 and download the external files to your SMB file shares. Through this serverless integration, Lambda can automatically download files from S3 and save to a shared file system.
• Using Amazon Athena to query data stored in Amazon S3. The query results are stored in a comma separated file (.csv), and you want to copy the results to an SMB-compatible file share. Scheduling Athena queries using Lambda is a common use case, extending the Lambda function to copy the results to a Windows file share automates the process.

Solution overview

AWS Lambda supports multiple languages through the use of runtimes. There are several implementations of SMB in common programming languages, such as Python, .NET, Java, and Node.js. I demonstrate two Lambda functions with different runtimes. The first function uses a Python runtime and imports smbprotocol – an open-source Python library that implements the SMBv2 and SMBv3 protocol. The second function (an alternative for customers more comfortable with .NET) imports SMBLibrary – an open-source C# SMB 1.0/CIFS, SMB 2.0, SMB 2.1, and SMB 3.0 server and client implementation.

In the Lambda code provided, I implement two operations (store file and list shares) on a preexisting FSx for Windows File Server share. You can extend the operations to your needs by referring to the documentation for each respective 3PP library. I also detail security aspects and credential management using AWS Secrets Manager.

Walkthrough

Let’s walk through AWS Lambda integration with SMB files shares running in Amazon FSx Windows File Server, Windows File Server on Amazon EC2, or your on-premises environment.

Figure 1: Lambda function integrating with SMB file shares running in Amazon FSx, launched on EC2 or on-premises environment

1. Create a test event with dummy payload (“test”) to trigger a Lambda invocation. Lambda integrates with other AWS services to invoke functions.
   • Note: Amazon S3 is shown as example service integration that can trigger Lambda, Lambda can read and write from Amazon S3 using AWS SDKs.
2. Lambda retrieves the stored credentials of the file server (for example, user name, password, host, and share name) from AWS Secrets Manager.
3. Lambda function creates a test file and stores on the file share.

Demo using AWS Serverless Application Model (SAM)

View source code

To deploy the example application, you need a few prerequisites:

Prerequisites

• AWS credentials that provide the necessary permissions to create the resources. This example uses admin credentials.
• An available Amazon FSx file share or an SMB-compatible file share with valid credentials and a route to the VPC. See the getting started exercise to set up Amazon FSx for Windows File Server.
• .NET SDK installed if deploying template with Lambda .NET core runtime.
• The AWS SAM CLI As of this writing, version 1.29.0.
• Clone the GitHub repository.

Deployment steps

1. Navigate to the cloned repo directory, or run sam init then choose Custom Template Location and paste the repo URL.
2. Build the AWS SAM application, replace <templateName> with Python or the .NET template:

sam build -t <templateName>

3. Deploy the AWS SAM application:

sam deploy –guided

Figure 2: Example input values for guided AWS SAM deployment

AWS SAM performs the following steps:

• Creates a secret with dummy values stored in Secrets Manager.

• Creates a VPC endpoint for Secrets Manager in your selected VPC. This is required for Lambda functions connected to a VPC.

• Creates a prefix list for easier management and reference of CIDR (Classless Inter-Domain Routing) blocks.

Depending on the selected template, AWS SAM performs either action:

• Builds and deploys a Lambda function with Python runtime, and builds a Lambda layer with library dependencies and associates it with the Lambda function.

OR

• Builds a Lambda function with .NET Core and its dependencies.

Testing

To test the integration, I show the steps for the Python runtime function:

1. Replace the dummy values in the created secret in Secrets Manager with the IP address or a resolvable hostname, server name, user name, and password of your file share.

Note: From the AWS Management Console, as the root user, go to Secrets Manager and copy the resource policy to a notepad. You must remove the resource policy first to edit. After replacing with your secret key pair values, reinstate the resource policy again.

Note: For Amazon FSx for Windows File Server, configure the host value in the secret with the ‘Preferred File Server IP Address’ found in the console.

2. In the Lambda console, select your function and create test event with payload from the json file in the repo.

Figure 3: Successful Lambda function test event

3. Verify that the file is created on the file share.

Figure 4: Test file is created on the file share

Monitoring and troubleshooting

To verify file share access and file access, you can enable file access auditing and view the logs in CloudWatch if using Amazon FSx for Windows File Server. If your Windows file server is running on EC2 or on-premises, check the Windows Event Logs.

Security and connectivity

Per the Security pillar of the AWS Well-Architected Framework, apply security at all layers. Apply a defense in depth approach with multiple security controls to all layers (edge of network, VPC, load balancing, every instance and compute service, operating system, application, and code). Note: This solution is a proof-of-concept on how to achieve SMB access for serverless workloads. Customers should keep their security requirements in mind when developing their solutions.

The solution requires a few security configurations:

VPC and subnets

I configure the Lambda function (in this demo solution) with a VPC and two private subnets for resiliency. This setup enables reachability from the Lambda function to your private AWS resources. In real environments, configure proper routing from Lambda’s subnet(s) based on your needs.

In the case of on-premises file servers, you need Amazon VPN or AWS Direct Connect with routing rules that allow SMB traffic to pass through to the Amazon VPC.

Security groups

The Lambda security group is configured with two outbound rules and no inbound rule:

• For secret retrieval, an outbound rule that allows HTTPS (port 443) access to Secrets Manager VPC endpoint’s security group (destination).

• For SMB traffic, an outbound rule that allows SMB (port 445) access to CIDR block of your file server (destination).

Note: Configure the created managed prefix list with your on-premises network CIDR blocks.

Figure 5: Lambda security group outbound rules

The security group of the VPC endpoint of Secrets Manager is configured with one inbound rule:

• For secret retrieval, an inbound rule that allows HTTPS (port 443) communication from Lambda’s security group (source)

Figure 6: Secrets Manager VPC endpoint security group rules allowing inbound HTTPS from Lambda’s security group

Finally, make sure the security group (or on-premises firewall appliance) of your file share servers allow inbound access (port 445) from Lambda security group.

Credential management

The AWS Lambda function must authenticate with the SMB file share. A good practice is to avoid hardcoding credentials or storing in files or environment variables. For this reason, I integrate the Lambda function with AWS Secrets Manager to store and retrieve the file server credentials and parameters. Secrets Manager stores a secret in form of key-value pairs as shown in the following screenshot. We recommend customizing the name of the secret, and using a dedicated set of scoped-down credentials instead of Admin credentials.

Figure 7: A secret with configured credentials and parameters in AWS Secrets Manager

Secrets Manager provides code examples that illustrate how to retrieve the secret in your application. To view them, go to the AWS Management  Console>AWS Secrets Manager>Secrets>MySecretName.

Encryption at rest and in transit

Amazon FSx for Windows File Server supports two forms of encryption for file systems, encryption of data at rest and in transit. Encryption of data at rest is automatically enabled when creating an Amazon FSx file system. Encryption of data in transit is supported on file shares that are mapped on a compute instance that supports SMB protocol 3.0 or newer. Amazon FSx automatically encrypts data in transit using SMB encryption as you access your file system without the need for you to modify your applications.

In the following screenshot, I am sniffing the AWS Lambda function traffic on a Windows file server running on Amazon EC2. Similar behavior is expected for Amazon FSx for Windows File Server. You can see protocol negotiated to SMB 3.1.1 dialect, between the file server and Lambda.

Figure 8: AWS Lambda function sending a “Negotiate Protocol Request” to a Windows 2019 file server on Amazon EC2

After the protocol negotiation is complete, you can see the encrypted traffic between AWS Lambda and the file server in the following screenshot:

Figure 9: Packet trace showing encryption in transit between AWS Lambda and Windows 2019 file server on Amazon EC2

You can restrict unencrypted access in Amazon FSx for Windows File Server for clients that do not support SMB 3.x, by enforcing encryption in transit as detailed in the documentation on best practices for administering Amazon FSx file systems.

Note: This configuration will fail any connected client (other than AWS Lambda function) to your file systems that does not support SMB 3.x.

Figure 10: Amazon FSx for Windows File Server rejecting unencrypted access from SMB2 clients

IAM policies

In order for the Lambda function and the VPC endpoint of Secrets Manager to access my secret, I configure their inline policy as follows:

• GetSecretValue action locked down to the file server secret as defined in the resource field

        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal: '*'
            Action:
              - 'secretsmanager:GetSecretValue'
            Resource: !Sub ${MySecret}

Resource policies

Additionally, I configure a resource policy for the secret stored in Secrets Manager. I tell Secrets Manager to deny any request explicitly unless it originates from a defined source SPC endpoint.

Note: AWS SAM template creates the following policy:

Version: "2012-10-17"
        Statement:
        - Sid: EnableSecretsManagerpermissions
          Effect: "Allow"
          Principal:
            AWS: !Sub "${AWS::AccountId}"
          Action: "secretsmanager:*"
          Resource: "*"
        - Sid: RestrictGetSecretValueoperation
          Effect: "Deny"
          Principal: '*'
          Action: "secretsmanager:GetSecretValue"
          Resource: "*"
          Condition:
            StringNotEquals:
              'aws:sourceVpce': !Sub '${SecretsManagerVPCEndpoint}'

Cleaning up

To clean up the demo:

1. Delete the CloudFormation stack or run sam delete from your terminal.

Conclusion

In this post, I showed you how to integrate your Lambda functions with Amazon FSx for Windows File Server or Amazon FSx for NetApp ONTAP or SMB-compatible file shares using open-source libraries for Python and .NET. The demo solution is serverless and easily deployed with AWS SAM. I also walk through the security of all layers of the solution. This integration enables AWS Lambda to perform file operations on SMB-compatible persistent data stores.

Integrating AWS Lambda with Amazon FSx file shares allows you to share data across function invocations unlocking new use cases for Lambda such as automatically saving files to Windows home directories, taking programmatic actions upon file change events, or use cases simply requiring a larger and persistent data store for your function invocations.

You can experiment with other Lambda runtimes and open-source SMB libraries such as Java or Node.js based on your preference. All the demo artifacts are available on the GitHub repo and you can extend it to your needs.

Thanks for reading this blog post. If you have any comments or questions, don’t hesitate to leave them in the comments section.