Using Amazon Cognito as an identity provider with AWS Transfer Family and Amazon S3

In highly regulated industries, securely exchanging files business-to-business is a crucial business practice. When building out a Managed File Transfer (MFT) environment, it is common to consider using a third-party identity solution for authenticating users. This approach offers simplicity for businesses that already use an identity service, allowing them to maintain identities for a variety of purposes in a single location. Although on the surface this reduces operational overhead, sometimes identity solutions lack mechanisms that allow for the storage and administration of user entitlement details. When working with these types of identity providers, this gap creates a challenge with maintaining appropriate authorization for each user. Authentication is a critical feature of any MFT, but appropriate authorization is just as important.

Well-designed authorization mechanisms help prevent unauthorized users from accessing, modifying, or deleting sensitive data. To address this, AWS Transfer Family offers a flexible architecture for integrating user’s identity providers. With Transfer Family custom identity providers, users can design customizable controls that integrate alongside the identity provider. These controls make sure users only access permitted data by dynamically generating and enforcing session policies and logical directory mappings. This makes sure that users can only access, and/or alter, permitted data as defined by an administrator.

In this post, we walk through setting up and testing an SFTP server using AWS Transfer Family, wherein we configure the SFTP server with Amazon Cognito as its custom identity provider. We store entitlements in Amazon DynamoDB and use AWS Transfer Family features, such as logical directories and session policies, to provide customized access. We also deploy an AWS Lambda function that facilitates both the authentication of the user through Amazon Cognito and the retrieval and enforcement of the authorization entitlements found in DynamoDB. This solution allows regulated organizations to exchange files securely without compromising their ability to control user entitlements and prevent unauthorized data access.

Solution overview

For this solution, you are configuring the following AWS services to build the file transfer solution.

• AWS Transfer Family for managing secure FTP transfers.
• Amazon Simple Storage Service (Amazon S3) for scalable object storage.
• Amazon Cognito for user identity and access management.
• Lambda to enable custom authentication workflows.
• DynamoDB to store user entitlements and access controls.

With AWS Transfer Family you have flexibility in how you manage user identities, whether through service managed users, Active Directory integration, or a custom identity provider (IDP). In this post, we use Amazon Cognito, paired with a Lambda authorizer function, as a custom identity provider. The Lambda function validates user credentials against Amazon Cognito and checks their access entitlements in DynamoDB before authorizing access to specific S3 buckets and folders. This allows fine-grained control over user permissions for transfers.

The following diagram summarizes the overall workflow of this solution as just described:

Prerequisites

To follow along with this walkthrough, you must have the following:

• An AWS account with a role that has sufficient access to provision the necessary resources.
• Your chosen AWS Region – we use us-west-2 (make sure to use the same Region for all activities).
• You need at least one additional AWS account with administrator level privilege (AdministratorAccess policy attached to your role).

Walkthrough

To set up the solution, you must perform the following steps:

1. Deploy resources using AWS CloudFormation.
2. SFTP endpoints and entitlements.
3. Create a Transfer Family SFTP server.
4. Set up custom IDP.
5. Testing the SFTP server.

1. Deploy resources using CloudFormation

Run the CloudFormation template to setup lab resources in the AWS Region us-west-2. This is a mandatory step, as instructions provided in the subsequent sections are based on these resources.

1.1. Download the CloudFormation template.

1.2. Navigate to CloudFormation in the AWS Management Console, select Create stack, upload the template that was downloaded earlier, and select Next.

1.3. Give the stack a name, such as “file-transfer-solution”.

1.4. Accept the default values in Configure stack options and select Next.

1.5. Check I acknowledge that AWS CloudFormation might create IAM resources and select Submit.

1.6. Once the deployment is complete, review the resources created by navigating to CloudFormation Resources and selecting the CloudFormation stack you created.

2. SFTP endpoints and entitlements

In this section, create an SFTP server with a custom identity provider using AWS Transfer Family, Amazon Cognito, and Lambda.

Use the resources in the following table, all provisioned as part of the environment setup, to configure the SFTP server, users in the IDP, and log in to the SFTP server.

us-west-2_<<xxxx>>

file-transfer-solution-databucket-<<xxxx>>

file-transfer-solution-AuthLambda-<<xxxx>>

You need names for the preceding resources from your AWS environment. To copy the resource names, navigate to the CloudFormation Outputs and select the CloudFormation stack that you created. Copy the resource names of AuthLambda, AuthLambdaRole and DataBucket (highlighted below) for use in the following sections

3. Create an AWS Transfer Family SFTP Server

Next, create and configure an SFTP server.

3.1. Navigate to the Transfer Family console, and choose Create server.

3.2. In the Choose protocols page:

a) Check the SFTP (SSH File Transfer Protocol) – file transfer over Secure Shell option from the list.

b) Leave all other options unchecked and choose Next.

3.3. In the Choose an identity provider page:

a) Choose Custom Identity Provider.

b) Choose Use AWS Lambda to connect to your identity provider.

c) Select file-transfer-solution-AuthLambda-<<xxxx>>, in which xxxx is a unique alphanumeric identifier from the AWS Lambda function dropdown list.

d) Choose Next.

3.4. In the Choose an endpoint page:

a) Choose Publicly accessible.

b) Leave other settings in their default setting.

c) Choose Next.

3.5. In the Choose a domain page:

a) Choose Amazon S3.

b) Choose Next.

3.6. Leave the default options in Configure additional details, then choose Next.

3.7. On the Review and create page, choose Create.

a) Now, the SFTP server is active and available.

Note that it takes a couple of minutes for the server to be created and reach an Online state. Wait before moving to the next step.

3.8. Select server-id and note the endpoint address, as you need it in later sections to connect to it.

This concludes the steps for creating the SFTP server. Next, set up the custom IDP using Amazon Cognito and Lambda.

4. Set up custom IDP

In this section, configure the custom IDP using Amazon Cognito to access pre-provisioned user entitlements stored in DynamoDB. We’ll cover:

4.1. Configure Amazon Cognito users
4.2. Configure authentication Lambda

With Amazon Cognito, you can add user sign-up and sign-in features, and control access to your web, mobile, and thick client applications. It provides an identity store that scales to millions of users, supports social and enterprise identity federation, and offers advanced security features to protect your consumers and business. Built on open identity standards, it supports various compliance regulations and integrates with frontend and backend development resources.

4.1. Configure Amazon Cognito users

4.1.1. Navigate to the Amazon Cognito console.

4.1.2. Select the pre-provisioned user pool with user pool name TransferUserPool-xxxx. A user pool is a user directory for web and mobile app authentication and authorization. User pools are an OpenID Connect (OIDC) identity provider (IDP) that add layers of additional features for security, identity federation, app integration, and customization of the user experience.

4.1.3. In the Users tab, select Create user to create two users: John Doe (john) and Jane Doe (jane).

4.1.4. Create user for John Doe. Enter username as “john” and a strong password of your choice. Make sure you remember the password, as it’s needed for log in later. To ensure comprehensive security, it’s crucial to implement MFA across all user accounts. For more details on how to enable it within Amazon Cognito user pool, please refer to Adding MFA

4.1.5. Create user for Jane Doe. Enter username as “Jane” and a password of your choice. Make sure you remember the password, as it’s needed for log in later.

4.2. Configure authentication Lambda

The pre-provisioned Lambda file-transfer-solution-AuthLambda-xxxx was built with the needed code for AWS Transfer Family to authenticate against an Amazon Cognito user pool and obtain the associated user entitlements stored in DynamoDB.

However, Lambda functions need to have permission enabled to allow other resources to invoke the Lambda function. Add a permissions statement to allow the newly created SFTP server to invoke the Authentication Lambda for user authentication and entitlement information.

4.2.1. First, you need to obtain the ARN for the SFTP server. Currently, this is not visible through the console, so you get it through a CLI command using AWS CloudShell.

a) Run the following command:

aws transfer list-servers

b) Copy and save the ARN value in the result.

c) Note that you may see multiple servers’ details if there are servers already configured in this account and need to copy the ARN value of the correct server.

4.2.2. Navigate to the Lambda console.

4.2.3. Select the file-transfer-solution-AuthLambda-<<xxxx>> function.

4.2.4. In the Configuration tab, choose the Permissions item in the left menu.

4.2.5. In the Resource-based policy statements section, choose Add permissions.

a) Choose AWS Service.

b) Choose Other from the Service

c) Enter a unique Statement ID, e.g., “file-transfer-access.”

d) Enter “amazonaws.com” as the Principal.

e) In the Source ARN field, enter the ARN that was obtained through the CLI command in Step 1.

f) From the Action dropdown, choose lambda:InvokeFunction.

g) Save the permissions statement.

4.2.6. To verify that the SFTP server can invoke the Lambda function, navigate to the AWS Transfer Family console and select the SFTP server.

a) From the Actions dropdown, choose Test.

b) Enter the username and password of either “john” or “jane”, and “0.0.0.0” as the Source IP since the SFTP server is publicly accessible. Choose Test.

This concludes the steps for setting up the custom IDP. Next log in to the SFTP server and to configure entitlements for controlling folder visibility and access.

5. Testing the SFTP server

With the following steps, we test how the identity provider determines and enforces which buckets in Amazon S3 each user can access:

5.1. Review folder structure and entitlements
5.2. Log in using CloudShell

The identity provider featured in this post uses the logical directories feature of AWS Transfer Family to create a virtual directory structure based on user entitlements. With logical directories, you can construct a virtual directory structure that uses user-friendly names that your users navigate when they connect to your S3 bucket or Amazon Elastic File System (Amazon EFS) file system. When you use logical directories, you can minimize disclosing absolute directory paths, S3 bucket names, and Amazon EFS file system names to your end users.

The identity provider featured in this post uses a session policy to control the user’s access based on entitlements. A session policy is an AWS Identity and Access Management (IAM) policy that restricts users to certain portions of an S3 bucket. It does so by evaluating access in real time. You can use a session policy when you need to give the same access to a group of users to a particular portion of your S3 bucket. For example, a group of users might only need access to the home directory. That group of users shares the same IAM role. The session policy in this solution is generated dynamically during runtime based on the user entitlements.

5.1. Review folder structure and entitlements

Let’s review the folder structure of the data in S3, and the entitlements for each user the DynamoDB table:

5.1.1. Navigate to your S3 buckets.

5.1.2. Open the file-transfer-solution-data-bucket-xxxx bucket that is storage for the SFTP server. Note that all of the folders are contained in the bucket, but the SFTP user entitlements restrict the folders that are visible to users and whether they can upload into those folders.

5.1.3. Navigate to DynamoDB.

5.1.4. Choose one file-transfer-solution-UserProducts-<<xxxx>> table. Select the Explore table items button from right top side of the page to view the user entitlements.

5.1.5. You can see John has the equities product subscribed and the indices product published. Jane has the indices product subscribed and the credit, equities products published. Users have read-only access to subscribed products and read/write access to published products.

The custom IdP code (lambda function) retrieves the user’s entitlements from the DynamoDB table based on their username. For instance, when John Doe logs in, the Lambda function queries the DynamoDB table and finds that John has the “equities” product subscribed, granting him read-only access, and the “indices” product published, allowing him read and write access. The function then interprets these entitlements and constructs a logical directory mapping representing the virtual file structure that John should see. This mapping is sent back to the Transfer Family service, which uses it to enforce the appropriate access controls. Specifically, John would see directories for “equities” (read-only) and “indices” (read/write), while any other directories in the underlying S3 bucket would be hidden from him based on his entitlements stored in DynamoDB. This logical directory mapping abstracts away the actual S3 bucket structure, providing a user-friendly virtual view tailored to each user’s permissions.

5.2. Log in using CloudShell

Now we log in to the SFTP server using CloudShell. Although it is possible to use a variety of SFTP clients for the purposes of this post, here we cover using the command line SFTP client available in CloudShell to connect to the SFTP server. Using CloudShell, a browser-based pre-authenticated shell that you can launch directly from the AWS Management Console, you can run AWS Command Line Interface (AWS CLI) commands using your preferred shell, such as Bash, PowerShell, or Z shell, without downloading or installing AWS CLI tools.

5.2.1. Open CloudShell and create a test file for John.

a) Run the following command:

echo “This is just some text.” > index1-2023.txt

b) Log in to the SFTP server as John using the following command: sftp john@REPLACE_WITH_SFTP_ENDPOINT_URL. Enter yes for a RSA key fingerprint. Press the enter key. Enter the password you created in Amazon Cognito for the user. Press enter.

5.2.2. View the products to which John has access and perform file operations. View the products that John has published and subscribe access by executing the following commands. The published folder is where John can publish or upload files, whereas the subscribed folder is where John can subscribe/download files.

ls published
ls subscribed

5.2.3. Upload the index1-2023.txt file to the published indices product folder. The upload is successful, as users can upload files to their published products.

cd published/indices
put index1-2023.txt
ls

5.2.4. Upload the index1-2023.txt file to the subscribed equities product folder. The upload is unsuccessful as users can only upload files to their published products. Close the SFTP connection to complete testing as John.

cd ../../subscribed/equities ls 
put index1-2023.txt
get equity2-2019.txt 
exit

5.2.5. Create a test file for Jane.

a) Run the following command:

echo “This is just some text.” > equity1-2023.txt

b) Log in to the SFTP server as Jane using the following command: sftp jane@REPLACE_WITH_SFTP_ENDPOINT_URL. Browse the Published and Subscribed folders. Note how the visible folders are different for Jane than for John.

5.2.6. View the products to which Jane has access to and perform file operations. View the products that Jane has published and subscribe access by executing the following commands:

ls published
ls subscribed

5.2.7. Upload the equity1-2023.txt file to the published equities folder. The upload is successful as users can upload files to their published products. Close the SFTP Connection to complete testing as Jane.

cd published/equities
put equity1-2023.txt
ls
exit

Cleaning up

You have created several components that generate cost. To avoid incurring future charges, remove the resources with the following steps:

Amazon S3 clean up

1. Select Amazon S3 to navigate to the dashboard.
2. Select the file-transfer-solution-databucket-<<xxxx>> and select Empty. If you see a warning No objects found in bucket xxxxx-xxxxx , then select Cancel.
3. Otherwise enter “permanently delete” into the confirmation box, and then select Empty.
4. When you see the green banner at the top stating that the bucket is empty, select Exit.

Delete SFTP server

1. Open the Transfer Family console
2. In the left navigation pane, choose Servers.
3. Select the check box of the server that you created during the workshop.
4. For Actions, choose Delete.
5. In the confirmation dialog box that appears, enter the word “delete”, and then choose Delete to confirm that you want to delete the server.
6. The server is deleted from the Servers page.

Remove the CloudFormation stack

• Select the stack on the CloudFormation console that was created as part of the post, then select Delete and confirm.

Conclusion

In this post, we dove into the process of setting up a secure SFTP server with AWS Transfer Family, leveraging Amazon Cognito for user authentication and Amazon DynamoDB to restrict user data access. By implementing this customized access control, users in regulated industries can confidently exchange files without worrying about data leakage or unauthorized modifications.

Transfer Family, a fully managed service, takes care of the scalability and failover, allowing you to focus on your applications and business needs. With its robust security features, Transfer Family makes sure that your data is always protected, even in the event of a disaster or outage. Use Transfer Family’s scalability and failover to achieve agility, elasticity, and cost savings. Visit our documentation and product page to learn more about how Transfer Family can help you streamline your file transfer processes and meet your security requirements. AWS Transfer Family now provides an interactive workshop for building file transfer solutions using Secure File Transfer Protocol (SFTP). To get started with the workshop, visit Transfer Family – SFTP Workshop.

We hope you have enjoyed our post, happy building!

If you have any comments or questions, feel free to leave a comment in comments section.