Secure and Cost-Effective Video Streaming using CloudFront signed URLs

In this blog, you will learn how to solve for a common challenge you may face when streaming video on demand (VOD) – limiting access to media streams for select and/or paying users.

In this solution, you will use HTTP Live Streaming (HLS) and Amazon CloudFront. HLS is a protocol that segments media files optimizing them for streaming. HLS enables media players to play segments with the highest quality resolution that is supported by their network connection during playback. Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data to end-users globally with low latency and high transfer speeds.

Solution overview

CloudFront offers advanced security capabilities, integrating with AWS Shield and AWS Web Application Firewall to protect against multiple types of attacks including network and application layer DDoS attacks. CloudFront also works seamlessly with any AWS origin, such as Amazon Simple Storage Service (Amazon S3), Amazon API Gateway, Elastic Load Balancing, or with any custom HTTP origin.

To securely serve your private content, you can configure CloudFront to require that your users access your files using CloudFront signed URLs. A signed URL includes additional information, such as an expiration date and time, that gives you more control over access to your content. This additional information appears in a policy statement which is based on a custom policy.

In order to restrict access to files and media streams, you will need:

• CloudFront, to securely deliver your content and CloudFront signed URLs to control access to it.
• API Gateway HTTP API, as the CloudFront origin and the entry point for your APIs.
• AWS Lambda, to sign URLs to your HLS files.
• AWS Secrets Manager, to securely store private key that will be used by the Lambda function to sign URL.
• Amazon S3, to store the main playlist and HLS files.

This solution assumes that the user visiting your web page is authenticated with your application, and this user wants to load a player. Figure 1 below illustrates the flow that a user’s request goes through to playback your media files.

Figure 1: User request flow to playback media files

Let’s look at how user request flow as illustrated in Figure 1 above works.

1. Client requests signed URL to *.m3u8 file by making API call to a protected resource.
2. CloudFront behavior is matched based on path pattern. Request is forwarded to origin.
3. Lambda gets CloudFront private key and caches for subsequent requests.
4. Using the private key, the URL to *.m3u8 file is signed and returned to the client.
   

   Note: Before signing the URL, you want to verify if the user is authorized to watch the requested content. Only if this is the case, you sign the URL and return it back to the client application.

5. Client requests *.m3u8 file, passing custom query params Key-Pair-Id-PREFIX, Policy-PREFIX and Signature-PREFIX.
6. CloudFront behavior is matched and request is forwarded to the origin.
7. CloudFront makes origin request and passes custom headers.
8. API Gateway forwards the request to Lambda that has proxy integration configured with a greedy path variable ({proxy+}). This will allow Lambda function to programmatically identify how to get the manifest from S3 bucket in the next step.
9. Lambda function gets the original *.m3u8 manifest file from S3.
10. Lambda function modifies the .m3u8 manifest by appending signed URL params to each *.ts file name.
11. Client requests *.ts file, passing query params as part of request URL.
12. CloudFront behavior is matched and request is forwarded to the origin.
13. CloudFront makes origin request to S3 and returns *.ts file.

Solution walkthrough

In this section, you will learn additional details about how each of the key components work and how they are configured.

Use CloudFront origin and behavior to forward client request

An origin is the location where you store your content, connecting it to CloudFront so that it can serve this content to your viewers.

In this solution, you will need three origins:

• API Gateway endpoint when signing a URL
• API Gateway endpoint when requesting the main manifest and child manifest, if applicable.
• S3 bucket to request the manifest and HLS files

Following Figure 2 illustrates CloudFront origin configuration for each of the origins CloudFront will be forwarding the request.

Figure 2: CloudFront origins

• sign-url.api.myapp.com, is the origin for API Gateway endpoint when signing a URL
• media-files.api.myapp.com, is the origin for API Gateway endpoint when requesting the main manifest
• my-content.s3.amazonaws.com, is the origin for S3 bucket to request the manifest and HLS files

CloudFront behavior allows you to route the request for an object based on path patterns and based on the precedence. If the URL matches the path patterns of two or more behaviors, behavior with higher precedence is selected.

Figure 3: CloudFront behaviors

Let’s look at the CloudFront behaviors setup as illustrated in Figure 3 above.

• When the player makes a request to fetch the main manifest, CloudFront behaviors route the request to the origin sign-url.api.myapp.com.
  

  Note: By default, API Gateway is created with a default base URL that follows this pattern: https://api-id.execute-api.region.amazonaws.com/stage. You can create a custom domain name and associate it with your API Gateway default URL. To learn how to create a custom domain name, see Setting up custom domain names for HTTP APIs.

• When the client application receives the signed URL, to fetch the main manifest, the request path will be matched with the pattern *.m3u8. The CloudFront behavior will then route the request to the origin media-files.api.myapp.com. Next, a Lambda function is invoked, getting the main manifest from your S3 bucket, modifying it, and returning the manifest to your client application.
• The player makes a request to fetch the associated media segments. The request path will be matched with the pattern *.ts. The CloudFront behavior routes the request to the origin my-content.s3.amazonaws.com.

Using CloudFront Trusted Key Group to protect media content

To create signed URLs, you need a signer. A signer is either a trusted key group (recommended) or a CloudFront key pair that you create in CloudFront, or an AWS account that contains a CloudFront key pair. Each signer that you use to create CloudFront signed URLs must have a public–private key pair. The signer, a Lambda function in this case, uses a private key to sign the URL, and CloudFront uses the public key to verify the signature. To learn more about trusted keys, how to create a new key pair, and how to upload the public key to CloudFront, see this document about Creating key pairs for your signers.

Note: To help secure your applications, I recommend that you rotate key pairs periodically. For more information, see Rotating key pairs.

The last step in protecting your media files with CloudFront trust key group is to associate the media-files.api.myapp.com and my-content.s3.amazonaws.com behavior. To learn how to add a signer to your CloudFront distribution, see Adding a signer to a distribution using the CloudFront console.

Figure 4 illustrates CloudFront behaviors when certain behavior are protected with trusted key group.

Figure 4: CloudFront behavior protected with trusted key group

Signing URLs with a Lambda function

When the client application makes a request to get the signed URL for the main manifest, CloudFront behaviors will forward this request to the API Gateway as show in Figure 1 step 2. API Gateway will forward the request to the Lambda function to sign the URL and return the signed URL to the client application.

Assume that your media files are stored in an S3 bucket that has a folder structure as shown in Figure 5 below.

Figure 5: S3 bucket folder structure to store media files

For illustration purposes, also assume that the request URI is /content/media/signurl?movie=1. The Lambda function will get the query parameter movie=1 from the event object then query your data store to identify the s3 bucket name and the path to the index.m3u8 file. Once again, assume query result return movies/movie_1/index.m3u8 as the path to the movie=1 media content. Knowing the path to the index.m3u8 file, Lambda function has all the required information to sign a URL.

The code below shows the Lambda function code that gets a private key from Secrets Manager and uses it to sign the URL.

import datetime
import json
import rsa
import os
import base64
import boto3

from botocore.signers import CloudFrontSigner

CF_KEY_ID = os.environ.get('CF_KEY_ID')
CF_URL = os.environ.get('CF_URL')
SM_SECRET_NAME = os.environ.get('SM_SECRET_NAME')

required_vars = [CF_KEY_ID, CF_URL, SM_SECRET_NAME]

if not all(required_vars):
    raise KeyError(f'Missing required environment variable/s. Required vars {str(required_vars)}.')

client = boto3.client('secretsmanager')
cf_private_key_base64 = client.get_secret_value(SecretId=SM_SECRET_NAME)['SecretString']


def rsa_signer(message):
    private_key = base64.b64decode(cf_private_key_base64)
    return rsa.sign(message, rsa.PrivateKey.load_pkcs1(private_key), 'SHA-1')


cf_signer = CloudFrontSigner(CF_KEY_ID, rsa_signer)


def lambda_handler(event, context):
    movie_id = event['queryStringParameters'].get('movie')

    if movie_id is None:
        # or handle otherwise when required query param is missing
        return {'statusCode': 400, 'body': ''}

    movie_path = '<Use movie_id to look up the path to media files in your s3 bucket.>'
    movie_path = 'pexels-pixabay-371589.jpg'
    # if  movie_path starts with '/', remove it.
    movie_path = movie_path[1:] if movie_path[0] == '/' else movie_path
    movie_folder_path = '/'.join(movie_path.split('/')[:-1])

    try:
        # if CF_URL ends with '/', remove it.
        url = CF_URL[:-1] if CF_URL[-1:] == '/' else CF_URL
        full_url = '/'.join([url, movie_path])
        # if movie_folder_path = '', don't join with '/'
        uri = str('' if movie_folder_path == '' else '/').join([movie_folder_path, '*'])
        resource = '/'.join([url, uri])

        # Hardcoded for simplicity. It can be dynamic value that's retrieved from another source.
        expire_token_in_hours = 6

        current_time = datetime.datetime.utcnow()
        expire_date = current_time + datetime.timedelta(hours=expire_token_in_hours)

        policy = {
            "Statement": [
                {
                    "Resource": resource,
                    "Condition": {
                        "DateLessThan": {
                            "AWS:EpochTime": int(expire_date.timestamp())
                        }
                    }
                }
            ]
        }

        policy_json_str = json.dumps(policy)
        signed_url = cf_signer.generate_presigned_url(full_url, policy=policy_json_str)

        return {
            'statusCode': 200,
            'body': signed_url
        }
    except Exception as e:
        print(e)

    return {'statusCode': 500, 'body': ''}

Let’s go over the key areas in this python code:

1. 1. CF_KEY_ID environment variable is the ID of the public key in CloudFront.
      

      Note: To find the public key ID, navigate to CloudFront console and select Key management.

   2. CF_URL environment variable is either the default CloudFront domain name or an alternate domain name (CNAME) assigned to a CloudFront distribution. The protocol must be included in the CF_URL variable, for example, if the CNAME is myapp.com, then the environment variable value is https://myapp.com.
   3. SM_SECRET_NAME environment variable is the name of your secret in Secrets Manager. The secret value is expected to be encoded in base64. You will need to base64 encode the private key and then store it in Secrets Manager so that Lambda function can successfully get the value. To learn how to create secrets in Secrets Manager, see Creating a secret.
      

      Note: The python code assumes the secret is type: Other type of secrets and displayed in the Plaintext option. You can change this as needed, but remember to update the code to properly get the secret.

   4. movie_id variable which is movie=1 from example above.
   5. movie_path variable, is the path to the index.m3u8 file. From example above it would be movies/movie_1/index.m3u8.
   6. movie_folder_path variable the path to the folder in s3 bucket, not including the file itself. From example above it would be /content/media/signurl.
   7. full_url variable is the full path URL to the index.m3u8 that the client application makes a request to get the main manifest. If the CNAME is myapp.com, that the user uses to access the web application, then the full path URL is https://myapp.com/movies/movie_1/index.m3u8.
   8. resource variable is the path to the movie folder: movie_folder_path, plus the wildcard that will allow the client application to access index.m3u8 and the media segments of that movie. From the Figure 5, the media segments are sequence_01.ts, sequence_02.ts, etc.
   9. expire_token_in_hours variable, defines how long the signed URL will be valid from current time of code execution. You can have more control over the expiration time by passing second, minutes, hours, days or other supported time unites to the datetime.timedelta function.
   10. policy variable, the object that represents the CloudFront signed URL custom policy. This policy is used by CloudFront to verify if user has access to the requested resource. To learn more about the custom policy, see Creating a signed URL using a custom policy.
   11. signed_url variable, is the signed URL using the private key. This url is returned to the client and is used by the client to get the main manifest.

The signed URL looks similar to the one below:

https://myapp.com/movies/movie_1/index.m3u8\
?Policy=eyJTdGF0ZW1lbnQiOiBbeyJSZXNvdXJjZSI6ICJodHRwczovL215YXBwLmNvbS9tb3ZpZXMvbW92aWVfMS8qIiwgIkNvbmRpdGlvbiI6IHsiRGF0ZUxlc3NUaGFuIjogeyJBV1M6RXBvY2hUaW1lIjogIjxleHBpcmF0aW9uLXRpbWVzdGFtcD4ifX19XX0K__\
&Signature=<SIGNATURE>\
&Key-Pair-Id=ABC0123456789

A CloudFront signed URL has three query parameters that CloudFront requires to be present on each request:

• Policy, base64 encoded version of policy statement
• Signature, hashed, and signed version of the policy statement
• Key-Pair-Id, public key ID for the CloudFront public key whose corresponding private key you’re using to generate the signature

Since the policy is encoded in base64, you can run the following command in your terminal to see the JSON object.

# Note: Works on Linux and MacOS only.
# You can also use any online available tool to decode a base64 encoded string,
# such as base64decode.
$ base64 -D <<< eyJTdGF0ZW1lbnQiOiBbeyJSZXNvdXJjZSI6ICJodHRwczovL215YXBwLmNvbS9tb3ZpZXMvbW92aWVfMS8qIiwgIkNvbmRpdGlvbiI6IHsiRGF0ZUxlc3NUaGFuIjogeyJBV1M6RXBvY2hUaW1lIjogIjxleHBpcmF0aW9uLXRpbWVzdGFtcD4ifX19XX0K__

The JSON object of the sample base64 encoded policy is:

{
   "Statement" : [
      {
         "Resource" : "https://myapp.com/movies/movie_1/*",
         "Condition" : {
            "DateLessThan" : {
               "AWS:EpochTime" : "<expiration-timestamp>"
            }
         }
      }
   ]
}

The video player in your client application receives the signed URL as request response, and makes the request to the https://myapp.com/movies/movie_1/index.m3u8 URL to get the main manifest. To authorize the request, CloudFront looks at all attributes in the JSON policy object. The Resource attribute in the policy verifies that the resource pattern in the Resources attributes matches with the request URL. The wildcard in https://myapp.com/movies/movie_1/* means that the policy allows access to any file in the movie_1 directory, as indicated by the * wildcard character. In this example, requests to https://myapp.com/movies/movie_1/index.m3u8 and https://myapp.com/movies/movie_1/sequence_01.ts will be authorized by CloudFront. To learn more about the policy statement, see Values that you specify in the policy statement for a signed URL that uses a custom policy.

Forwarding signed URL query parameters to the origin

When your media player makes a request to get the main manifest, three CloudFront query parameters need to be passed to the origin so that those can be used to modify the main manifest and append the CloudFront query parameters to each segment inside the manifest file. As Key-Pair-Id, Policy and Signature are intended to be consumed by CloudFront, CloudFront removes them from the URL before forwarding the request to the origin. In order to pass those query params values, you will need to append additional query params with different key name, however use the same values as in Key-Pair-Id, Policy and Signature.

Following request illustrates a URL that makes it possible to pass signed URL query param values to the origin:

https://myapp.com/movies/movie_1/index.m3u8\
?Policy=<base64_encoded_policy>\
&Signature=<SIGNATURE>\
&Key-Pair-Id=ABC0123456789\
&Policy-PREFIX=<base64_encoded_policy>\
&Signature-PREFIX=<SIGNATURE>\
&Key-Pair-Id-PREFIX=ABC0123456789

CloudFront removes Key-Pair-Id, Policy and Signature, however Key-Pair-Id-PREFIX, Policy-PREFIX and Signature-PREFIX query parameters are forwarded to the origin. The appended string -PREFIX represents a patterns that is used by the second Lambda function to identify which query param are meant to be used to modify the main manifest. This value can be anything you like, Lambda function only needs to know what it is for this to work.

There are two main options for constructing the URL:

• client side once client application gets the signed URL
• server side in the Lambda function to append additional query params before returning the signed URL to the client

Note: Ensure that your CloudFront distribution is configured to forward the query params to the origin. To learn how to configure CloudFront to forward query params to your origin, see Origin request settings.

Modifying the main manifest

When the second Lambda function receives the request, it gets the main manifest from your S3 bucket, modifies it by appending CloudFront query params to each segment in the manifest file, and returns the manifest to the client.

The following code below illustrates how it can be achieved in the Lambda function.

import boto3
import os


KEY_PREFIX = os.environ.get('KEY_PREFIX')
S3_BUCKET = os.environ.get('S3_BUCKET')
SEGMENT_FILE_EXT = os.environ.get('SEGMENT_FILE_EXT', '.ts')

required_vars = [KEY_PREFIX, S3_BUCKET]
if not all(required_vars):
    raise KeyError(f'Missing required environment variable/s. Required vars {required_vars}.')


s3 = boto3.client('s3')


def lambda_handler(event, context):
    try:
        s3_key = event['pathParameters']['proxy']
        obj = s3.get_object(Bucket=S3_BUCKET, Key=s3_key)

        body = obj['Body'].read().decode('utf-8')
        qp = event['queryStringParameters']

        params = ['?']
        # reconstruct query param uri
        [(params.append(p.replace(KEY_PREFIX, '') + '=' + qp[p] + "&")) for p in qp if KEY_PREFIX in p]
        sign_params = ''.join(params).rstrip("&")

        # append query params to each segment
        resp_body = body.replace(SEGMENT_FILE_EXT, ''.join([SEGMENT_FILE_EXT, sign_params]))

        return {
            'statusCode': 200,
            'body': resp_body
        }
    except Exception as e:
        print(e)

    return {'statusCode': 500, 'body': ''}

Let’s go over the key areas in the python code:

1. KEY_PREFIX environment variable is the prefix pattern that Lambda uses to identify the CloudFront query params. In my example it is -PREFIX.
2. S3_BUCKET environment variable the S3 bucket name to which Lambda will make the request to get the main manifest.
   

   Note: For illustration purposes I used an environment variable that is set, for instance, when the Lambda function is created. You can change this part to have a lookup logic, which is especially helpful if you have different S3 buckets that are used to store media content.

3. SEGMENT_FILE_EXT environment variable file extension of your media file. It default’s to .ts, you can override the value by setting this environment variable in Lambda function configuration.
4. s3_key variable is the path to the file in your S3 bucket which is represented as a URL that is used by the client to make the request. In my example the URL is https://myapp.com/movies/movie_1/index.m3u8 and the s3_key is movies/movie_1/index.m3u8, which matches exactly with the s3 bucket folder structure as shown earlier in Figure 5. By following this convention, when creating a S3 folder structure for the URL path, s3_key will dynamically get the correct path to the file in your S3 bucket. To learn more about Lambda proxy integration, see Set up a proxy integration with a proxy resource.
5. sign_params variable is the reconstructed CloudFront signed URL query parameters.
6. resp_body variable, is the final modified main manifest that is returned to the client. The replace function appends the CloudFront signed URL query parameters to each segment in the manifest file. The final result is assigned to the resp_body variable.

This Lambda function is getting the manifest file in its original form from your S3 bucket and modifying it so that when the playback player makes a request to get the next segment, the request already includes the CloudFront signed URL query parameters. This allows you to restrict access to the video content based on user permissions for each video. Figure 6 illustrates manifest before and after the modification.

Figure 6: Original and modified manifest file side-by-side

Making a request to an endpoint to get the ts segments

The video player makes requests to get each segment from your S3 bucket. This part is completely transparent to the client application and doesn’t require any configuration in the video player. CloudFront matched the path pattern to *.ts and forwards the request to my-content.s3.amazonaws.com origin.

To securely stream content from your S3 bucket to CloudFront, use a special CloudFront user called an origin access identity (OAI), associate it with your distribution, configure your S3 bucket permissions so that CloudFront can use the OAI to access the files in your bucket and serve them to your users. To learn more about how to create OAI and configure S3 bucket permissions, see Restricting Access to Amazon S3 Content by Using an Origin Access Identity.

Summarizing the request flow

At this point, you have walked through each step of the request flow, from requesting CloudFront signed URL, through getting the main manifest file, to getting media segments. Figure 7 below summarizes this request flow.

Figure 7: End-to-end request flow

Cost

Aside the costs of using CloudFront service to stream VOD, this solution incurs costs from two primary AWS services – API Gateway HTTP API and Lambda function.

If you have one main manifest file per video, the API Gateway and the Lambda function are invoked only once per video. If you have a main manifest and several child manifests, the API Gateway and the Lambda function are invoked one time for the main manifest and one consecutive time for each child manifest. For instance, if you have one main manifest and three child manifest for your video, API Gateway and the Lambda function are invoked in total of four times for that video. To learn about AWS service pricing, see AWS pricing.

Replacing API Gateway HTTP API with Application Load Balancer

Depending on the scale of your application, it can be more cost effecting to replace API Gateway HTTP API with Application Load Balancer (ALB). The primarily factors are the amount of media content streamed and how many manifests per media content. Generally, as your application reaches higher scale the more cost effective ALB will become. You can use the AWS pricing calculator to estimate costs for each of these approaches and better understand when you might want to replace API Gateway with ALB.

Conclusion

You have now learned how to secure your VOD streams with Amazon CloudFront. You have also learned how to pass CloudFront query parameters to Lambda function, sign URLs using CloudFront private key, and modify the manifest by appending CloudFront signed URL query parameters to each segment in the file.

If you have questions about this post, start a new thread on the Amazon CloudFront forum or contact AWS Support.