How to set up a simple video-on-demand service on AWS with TrackIt

The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.

The production of new video content driven by ubiquitous and relatively high-quality acquisition devices (mobile phones, action cameras, etc.), along with the accelerating re-use of old libraries, has seen an explosion of user interest in streaming distribution of their assets.

At TrackIt, we have been engaged by multiple content creators to create easy-to-use and simple video-on-demand (VOD) solutions they can implement to start sharing their content with users.

In September of 2019, when we architected our first deployment, there was not yet an offering available in the AWS marketplace that served as a full-package plug-and-play VOD solution. Recognizing the need, TrackIt developed and shared a clean and simple VOD workflow that could be quickly deployed on AWS with minimal technical expertise.

Our goal was to build a simple service that content owners could use to stream their library. We outline the process in the following step by step instructions to facilitate easy implementations, and touch on ancillary requirements such as user access management that are typically required for production use in a complete implementation.

Challenges for content creators

Limited control over video content

When using AWS, content creators typically choose to store their video content in Amazon Simple Storage Service (Amazon S3) buckets. S3 is an object storage built to store and retrieve any amount of data from anywhere. With TrackIt’s solution, additional video metadata such as location, name of the video creator, etc. can also be included.

Potential security risks

In order to allow viewers to access the videos from a website, content creators with minimal technical experience often implement one of two scenarios:

I. Make the S3 bucket public: In this case, everyone can access all of the content – including private videos.

II. Add specific policies to the bucket: Using these approaches configures and ensures secure access to their stored content assets.

Issues with video quality or resolution

In the absence of a purpose-built VOD solution, content creators generally only provide their viewers with videos of a single quality – the original quality of the uploaded video. Thus, viewers with low-bandwidth connections might have trouble buffering the content, and those with high bandwidth don’t have the option of experiencing higher-quality video.

The VOD workflow

TrackIt’s workflow has been designed as an easy-to-implement and elegant VOD solution. Our workflow implements only the minimum services required.

Amazon API Gateway is a fully managed service that acts as a central point for API integrations while enforcing clean programming principles is used to make requests. S3 buckets, the most common form of AWS storage, are used to store video content and AWS Elemental MediaConvert transcodes content and manages format resolutions.

AWS Elemental MediaConvert is a file-based video transcoding service that allows users to easily create video-on-demand (VOD) content for broadcast and multiscreen delivery at scale. Using MediaConvert allows content creators to provide users with multiple resolutions for each of video. The source video is converted into two different kinds of output: MP4 and HLS. HLS is an adaptive bitrate video format that optimizes the quality of video playback based on the available bandwidth and target device. HLS gives viewers access to multiple resolutions their players can switch to during playback. MediaConvert also generates thumbnails by extracting multiple evenly distributed frames in the video to provide a preview capability.

Amazon CloudFront is a content delivery network (CDN) used to geographically distribute and accelerate the web front-end we have developed in ReactJS along with the Video.js video player. CDNs provide a globally distributed network of proxy servers which cache content more locally to consumers, thus improving access speed for downloading content. CloudFront provides content creators with a simple and straightforward solution for video delivery with in-built caching mechanisms. A content creator simply needs to create a new distribution and select the right S3 buckets as targets, and CloudFront does the rest of the work. In the use case detailed in this blog, we use signed cookies to control access using S3 and CloudFront.

Amazon DynamoDB, a fully managed key-value and document database, is used to store information as RTMP and playback endpoints, configuration information, and streaming status.

Amazon Lambda, a serverless programmatic tool that provides scalable computing without provisioning or managing servers, functions are used to initiate the streaming process and handle all API calls.

How the workflow works

The TrackIt Workflow creates S3 buckets that are used to store video content
A clean API-based modern software design approach is used to manage VOD content
An input bucket is created to store raw video content uploaded by the content creator
Once a file is uploaded into this bucket, a Lambda function triggers the creation of a MediaConvert job to transcode the video
The transcoded videos are stored in an output bucket
The API Gateway provides a method for users make requests to retrieve the list of transcoded videos available in the output bucket

Features of the workflow

API that allows the content creator to perform CRUD operations on the content
Content tagging in the S3 buckets
Amazon Cognito integration for user management
Content metadata stored in DynamoDB
White label branding capability
Easy installation with just a single command required to deploy the full stack
M3U8 support through Apple HLS

Instructions

Pre-requirements

AWS CLI

All the steps require you to have AWS CLI installed.

Your credentials need to be set inside ~/.aws/credentials

S3 bucket

A bucket is needed for the Serverless framework. This bucket will be used to store data during deployment.

The other buckets are created during deployment.

CloudFront key pair

A CloudFront key pair is needed for the pre-signed URLs generation. For instructions, see Creating CloudFront Key Pairs for Your Trusted Signers.

You will need to save the key pair ID and the private key; they will be used in the next section for the configuration.

Node.js / NPM

Node.js and NPM must be installed. For instructions, see Node.js.

Serverless

Serverless must be installed. For instructions, see Get started with Serverless Framework Open Source & AWS.

Go to your AWS Management Console and create an access key for Serverless by following the Creating AWS Access Keys Tutorial. There are details on how to use your access key with Serverless following the tutorial.

Configuration

A configuration file named config.json is available to customize this workflow. Two mandatory modifications are detailed in the following subsections: deployment and distribution. Once you have made those modifications, you can rename the other values in this configuration file based on your needs.

Deployment modification

Set “deployment bucket” as the name of the bucket created in the preceding step. This is necessary for Serverless to push content to this bucket for deployment. You can also change the Region in which you want to deploy the workflow.

Distribution modification

First, you need to add information about the key pair generated above into the configuration file:

“keyPairId” must be the generated key pair ID
“privateKey” is the content of the private key that you have downloaded (Note: Each new line should be replaced by \n, so the private key can be stored as one single line)

Next, perform the first deployment so Serverless can set up everything required. Once the first deployment is finished, go to the AWS Management Console and search for the CloudFront distribution created with this workflow. A base URL is displayed. This value needs to be stored as “baseURL” in the configuration file.

Deployment

First, install the required libraries:

$> npm install

You can now deploy the workflow with the following command:

$> serverless deploy

$> serverless s3deploy

This command will add events notification support to trigger Lambda for transcoding

Using the API

First, you need to upload a video by generating an upload URL:

$> curl -X POST -H 'Content-type: application/json' -H 'Authorization: $TOKEN' -d '{"name": "vod_workflow_demo.avi"}'

http://vod-workflow-api/videos
{
"upload_url": "$UPLOAD_URL"
}

Then, once this URL is generated, you can upload your filet to S3:

$> curl --upload-file vod_workflow_demo.avi $UPLOAD_URL

You can then track the transcoding status by requesting details about your video to the API:

$> curl -H 'Authorization: $TOKEN' http://vod-workflow-api/videos
[
      {
            "status": "Transcoding",
            "job_id": "1234567890123-vod123",
            "id": "f8b49d03354b410bcc20518787bfc64b",
            "name": "vod_workflow_demo.avi",
            "last_update": 1234567890
      }
]
$> curl -H 'Authorization: $TOKEN' http://vod-workflow-api/videos/f8b49d03354b410bcc20518787bfc64b
{
      "status": "Transcoding",
      "job_id": "1234567890123-vod123",
      "id": "f8b49d03354b410bcc20518787bfc64b",
      "name": "vod_workflow_demo.avi",
      "last_update": 1234567890
}

Depending on your video, transcoding might take some time.

Once the transcoding is done, you can get the transcoded output and thumbnails:

$> curl -H 'Authorization: $TOKEN' http://vod-workflow-api/videos/f8b49d03354b410bcc20518787bfc64b
{
      "status": "Ready",
      "job_id": "1234567890123-vod123",
      "id": "f8b49d03354b410bcc20518787bfc64b",
      "name": "vod_workflow_demo.avi",
      "last_update": 1234567890,
      "content": {
            "mp4": "$CLOUDFRONT_PRESIGNED_URL",
            "hls": {
                  "master_low.m3u8": "$CLOUDFRONT_PRESIGNED_URL",
                  "master_high.m3u8": "$CLOUDFRONT_PRESIGNED_URL",
                  "master_med.m3u8": "$CLOUDFRONT_PRESIGNED_URL"
            },
            "thumbnails": [
                  "$CLOUDFRONT_PRESIGNED_URL",
                  "$CLOUDFRONT_PRESIGNED_URL"
            ]
      }

}

Three kinds of content are available:

MP4: MP4 version of your video
HLS: M3U8 playlist files with multiple resolutions (High, Medium & Low)
Thumbnails: Extracted frames from your video

If you want to store more details about the video, you can add data into DynamoDB using the API:

$> curl -X POST -H 'Content-type: application/json' -H 'Authorization: $TOKEN' -d '{"author": "myself"}' http://vod-workflow-api/videos/f8b49d03354b410bcc20518787bfc64b
{
      "status": "Ready",
      "job_id": "1234567890123-vod123",
      "id": "f8b49d03354b410bcc20518787bfc64b",
      "name": "vod_workflow_demo.avi",
      "last_update": 1234567890,
      "author": "myself",
      "content": {
            "mp4": "$CLOUDFRONT_PRESIGNED_URL",
            "hls": {
                  "master_low.m3u8": "$CLOUDFRONT_PRESIGNED_URL",
                  "master_high.m3u8": "$CLOUDFRONT_PRESIGNED_URL",
                  "master_med.m3u8": "$CLOUDFRONT_PRESIGNED_URL"
            },
            "thumbnails": [
                  "$CLOUDFRONT_PRESIGNED_URL",
                  "$CLOUDFRONT_PRESIGNED_URL"
            ]
      }

}

More detailed documentation about the API is available on GitHub.

Cognito for user management

Users are managed using Amazon Cognito authentication, authorization, and user management for web and mobile applications. While we do not go into the detail of the Cognito implementation here, we have created reference implementations and boilerplates that are available upon request.

The two primary user pools in this solution are:

vod-workflow-user-pool-users : For the Users
vod-workflow-user-pool-admins : For the Admins

A simple VOD solution designed for rapid deployment

This post detailed how content creators with minimal AWS-related technical experience can rapidly assemble and deploy their own VOD solution to stream their library. The architecture of the TrackIt workflow and the instructions provided in this post allows you to quickly and easily assemble your own VOD solution and have it running right away. We have designed this VOD workflow as a turnkey entry-level solution that a content creator doesn’t need to worry about once it’s set up properly.

For more information, contact AWS Advanced Consulting Partner TrackIt.

About the Author:

Ludovic François founded TrackIt, a full-service cloud engineering and consulting company in 2014 and serves as its CEO & CTO. Ludovic François has over 17 years of experience with start-up, cloud, storage and M&E businesses. Prior to founding TrackIt, Ludovic has held numerous roles at Thales Airborne Systems, DataDirect Networks and Nexsan. Ludovic is passionate about system administration, automation, education and sailing.

About TrackIt:

TrackIt Logo

TrackIt is a 6-year old cloud management, consulting and software development solutions company based in Venice, CA. We specialize in Modern Software Development; DevOps, Infrastructure-As-Code, serverless, CI/CD and containerization with special expertise in Media & Entertainment workflows, High-Performance Computing environments and data storage.

AWS for M&E Blog