How to securely transfer files with presigned URLs

Securely sharing large files and providing controlled access to private data are strategic imperatives for modern organizations. In an era of distributed workforces and expanding digital landscapes, enabling efficient collaboration and information exchange is crucial for driving innovation, accelerating decision-making, and delivering exceptional customer experiences. At the same time, the protection of sensitive data remains a top priority since unauthorized exposure can have severe consequences for an organization.

Presigned URLs address this challenge while maintaining governance of internal resources. They provide time-limited access to objects in Amazon Simple Storage Service (Amazon S3) buckets, which can be configured with granular permissions and expiration rules. Presigned URLs provide secure, temporary access to private Amazon S3 objects without exposing long-term credentials or requiring public access. This enables convenient collaboration and file transfers.

Presigned URLs are also used to exchange data among trusted business applications. This architectural pattern significantly reduces the payload size over the network by not using file transfers. However, it’s critical that you implement safeguards to help prevent inadvertent data exposure when using presigned URLs.

In this blog post, we provide prescriptive guidance for using presigned URLs in AWS securely. We show you best practices for generating and distributing presigned URLs, security considerations, and recommendations for monitoring usage and access patterns.

Best practices for presigned URL generation

Ensuring secure presigned URL generation is paramount. Aligning layered protections with business objectives facilitates secure, temporary data access. Strengthening generation policies is crucial for responsible use of presigned URLs. The following are some key technical considerations to securely generate presigned URLs:

Tightly scope AWS Identity and Access Management (IAM) permissions to the minimum required Amazon S3 actions and resources reduces unintended exposure.
Use temporary credentials such as roles instead of access keys whenever possible. If you’re using access keys, regularly rotating them as a safeguard against prolonged unauthorized access if credentials are compromised.
Use VPC endpoints for S3, which allow your Amazon Virtual Private Cloud (Amazon VPC) to connect directly to S3 buckets without going over internet address space. This improves isolation and security.
Require multi-factor authentication (MFA) for generation actions to add identity assurance.
Just-in-time creation improves security by making sure that presigned URLs have minimal lifetimes.
Adhere to least privilege access and encrypting in transit to mitigate downstream risks of unintended data access and exposure when using presigned URLs.
Use unique nonces (a random or sequential value) in URLs to help prevent unauthorized access. Verify nonces to prevent replay attacks. This makes guessing URLs difficult when combined with time-limited access.

Solution overview

Presigned URLs simplify data exchange among trusted business applications, reducing the need for individual access management. Using unique, one-time nonces enhances security by minimizing unauthorized use and replay attacks. Access restrictions can further improve security by limiting presigned URL usage from a single application and revoking access after the limit is reached.

This solution implements two APIs:

Presigned URL generation API
Access object API

Presigned URL generation API

This API generates a presigned URL and a corresponding nonce, which are stored in Amazon DynamoDB. It returns the URL for accessing the object API to customers.

The following architecture illustrates a serverless AWS solution that generates presigned URLs with a unique nonce for secure, controlled, one-time access to Amazon S3 objects. The Amazon API Gateway receives user requests, an AWS Lambda function generates the nonce and a presigned URL, which is stored in DynamoDB for validation, and returns the presigned URL to the user.

Figure 1: Generating a presigned URL with a unique nonce

The workflow includes the following steps:

The producer application sends a request to generate a presigned URL for accessing an Amazon S3 object.
The request is received by Amazon API Gateway, which acts as the entry point for the API and routes the request to the appropriate Lambda function.
The Lambda function is invoked and performs the following tasks:
1. Generates a unique nonce for the presigned URL.
2. Creates a presigned URL for the requested S3 object, with a specific expiration time and other access conditions.
3. Stores the nonce and presigned URL in a DynamoDB table for future validation.
The producer application shares the nonce with other trusted applications it shares data with.

Access object API

The consumer applications receive the nonce in the payload from the producer application. The consumer application uses the Access object API to access the Amazon S3 object. Upon the first access, the nonce is validated and then removed from DynamoDB. Thus, limiting use of the presigned URL to one. Subsequent requests to the URL are prohibited by the Lambda authorizer for added security.

The following architecture illustrates a serverless AWS solution that facilitates a secure one-time access to Amazon S3 objects through presigned URLs. It uses API Gateway as the entry point, Lambda authorizer for nonce validation, another Lambda function for access redirection by interacting with DynamoDB, and subsequent nonce removal to help prevent further access through the same URL.

Figure 2: Solution for secure one-time Amazon S3 access through a presigned URL

The workflow consists of the following steps:

The consumer application requests the Amazon S3 object using the nonce it receives from the producer application.
API Gateway receives the request and validates it using the Lambda authorizer to determine whether the request is for a valid nonce or not.
The Lambda authorizer ValidateNonce function validates that the nonce exists in the DynamoDB table and sends the allow policy to API Gateway.
If Lambda authorizer finds the nonce doesn’t exist, it means the nonce has already been used and it sends a deny policy to API Gateway, thus not allowing the request to continue.
When API Gateway receives the allow policy, it routes the request to the AccessObject Lambda function.
The AccessObject Lambda function:
1. Retrieves the presigned URL (unique value) associated with the nonce from the DynamoDB table.
2. Deletes the nonce from the Dynamo DB table, thus invalidating the presigned URL for future use.
3. Redirects the request to the S3 object.
Subsequent attempts to access the S3 object with the same presigned URL will be denied by the Lambda authorizer function, because the nonce has been removed from the DynamoDB table.

To help you understand the solution, we have developed the code in Python and AWS CDK, which can be downloaded from Presigned URL Nonce Code. This code illustrates how to generate and use presigned URLs between two business applications.

Prerequisites

To follow along with the post, you must have the following items:

An AWS account associated with your space along with an associated IAM role
The AWS Command Line Interface (AWS CLI)
Python 3.x Installed Python 3.x installed with pip
The AWS Cloud Development Kit (AWS CDK) installed

To implement the solution

Generate and save a strong random nonce string using the GenerateURL Lambda function whenever you create a presigned URL programmatically:

def create_nonce():
    # Generate a nonce with 16 bytes (128 bits)
    nonce = secrets.token_bytes(16)    
    return nonce

def store_nonce(nonce, url):
    res = ddb_client.put_item(TableName=ddb_table, Item={'nonce_id': {'S': nonce.hex()}, 'url': {'S': url}})
    return res

Include the nonce as a URL parameter for easier extraction during validation. For example: The consumer application can request the Amazon S3 object using the following URL: https://<your-domain>/stage/access-object?nonce=<nonce>

When the object is accessed, use Lambda to extract the nonce in Lambda authorizer and validate if the nonce exists.

Look up the extracted nonce in the DynamoDB table to validate it matches a generated value:

def validate_nonce(nonce):
    try:
        response = nonce_table.get_item(Key={'nonce_id': nonce}) 
        print('The ddb key response is {}'.format(response))

    except ClientError as e:
        logger.error(e)
        return False

    if 'Item' in response:
        # Nonce found
            return True
    else:
        # Nonce not found    
        return False

If the nonce is valid and found in DynamoDB, allow access to the S3 object. The nonce is deleted from DynamoDB after one use to help prevent replay.

	 if validate_nonce(nonce):
        logger.info('Valid nonce:'+ nonce)
        return generate_policy('*', 'Allow', event['methodArn'])
    else:
        logger.info('Invalid nonce: '+ nonce)
        return generate_policy('*', 'Deny', event['methodArn'])

Note: You can improve nonce security by using Python’s secrets modules. Secrets.token_bytes(16) generates a binary token and secrets.token_hex(16) produces a hexadecimal string. You can further improve your defense against brute-force attacks by opting for a 32-byte nonce.

Clean up

To avoid incurring future charges, use the following command from the AWS CDK toolkit to clean up all the resources you created for this solution:

cdk destroy –force

For more information about the AWS CDK toolkit, see Toolkit reference.

Best practices for presigned URL sharing and monitoring

Ensuring proper governance when sharing presigned URLs broadly is crucial. These measures not only unlock the benefits of URL sharing safely, but also limit vulnerabilities. Continuously monitoring usage patterns and implementing automated revocation procedures further enhances protection. Balancing business needs with layered security is essential for fostering collaboration effectively.

Use HTTPS encryption enforced by TLS certificates to protect URLs in transit using and S3 policy.
Define granular CORS permissions on S3 buckets to restrict which sites can request presigned URL access.
Configure AWS WAF rules to check that a nonce exists in headers, rate limit requests, and if origins are known, allow only approved IPs. Use AWS WAF to monitor and filter suspicious access patterns, while also sending API Gateway and S3 access logs to Amazon CloudWatch for monitoring:
1. Enable WAF Logging: Use WAF logging to send the AWS WAF web ACL logs to Amazon CloudWatch Logs, providing detailed access data to analyze usage patterns and detect suspicious activities.
2. Validate nonces: Create a Lambda authorizer to require a properly formatted nonce in the headers or URL parameters. Block requests without an expected nonce. This prevents replay attacks that use invalid URLs.
3. Implement rate limiting: If a nonce isn’t used, implement rate limiting by configuring AWS WAF rate-based rules to allow normal usage levels and set thresholds to throttle excessive requests originating from a single IP address. The WAF will automatically start blocking requests from an IP when the defined rate limit is exceeded, which helps mitigate denial-of-service attempts that try to overwhelm the system with high request volumes.
4. Configure IP allow lists: Configure IP allow lists by defining AWS WAF rules to only allow requests from pre-approved IP address ranges, such as your organization IPs or other trusted sources. The WAF will only permit access from the specified, trusted IP addresses, which helps block untrusted IPs from being able to abuse the shared presigned URLs.
Analyze logs and metrics by reviewing the access logs in CloudWatch Logs. This allows you to detect anomalies in request volumes, patterns, and originating IP addresses. Additionally, graph the relevant metrics over time and set CloudWatch alarms to be notified of any spikes in activity. Closely monitoring the log data and metrics helps identify potential issues or suspicious behavior that might require further investigation or action.

By establishing consistent visibility into presigned URL usage, implementing swift revocation capabilities, and adopting adaptive security policies, your organization can maintain effective oversight and control at scale despite the inherent risks of sharing these temporary access mechanisms. Analytics around presigned URL usage, such as access logs, denied requests, and integrity checks, should also be closely monitored.

Conclusion

In this blog post, we showed you the potential of presigned URLs as a mechanism for secure data sharing. By rigorously adhering to best practices, implementing stringent security controls, and maintaining vigilant monitoring, you can strike a balance between collaborative efficiency and robust data protection. This proactive approach not only bolsters defenses against potential threats, but also establishes presigned URLs as a reliable and practical solution for fostering sustainable, secure data collaboration.

Next steps

Conduct a comprehensive review of your current data sharing practices to identify areas where presigned URLs could enhance security and efficiency.
Implement the recommended best practices outlined in this blog post to securely generate, share, and monitor presigned URLs within your AWS environment.
Continuously monitor usage patterns and access logs to detect anomalies and potential security breaches and implement automated responses to mitigate risks swiftly.
Follow the AWS Security Blog to read more posts like this and stay informed about advancements in AWS security.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Want more AWS Security news? Follow us on Twitter.