How to reduce latency with Amazon Kinesis Video Streams – Part 2

In this, part 2 on how to reduce latency in media managed by Amazon Kinesis Video Streams (KVS) I describe the techniques in which to configure KVS, the media producer and the media player for optimal latency settings. Then, I introduce the Amazon Kinesis Video Stream Web Viewer and perform a number of experiments on KVS to validate the latency figures you can expect to achieve under a variety of conditions.

In part 1, I covered the fundamentals of streaming media, KVS Operation, common design patterns and the leading contributing factors to latency in streaming media managed by KVS.

Reducing Latency of KVS Live Media

The most effective means of reducing latency of streaming media in KVS are the following:

Optimize KVS provided HLS Playlist settings for live media,
Reduce the fragment length set by the KVS media producer,
Where supported by the media player:
- Programmatically reduce fragment buffering by the media player,
- Set the playback speed to slightly above x1 (i.e x1.05 / x1.1) to keep up with the live edge of the media at all times.

Be aware that adjusting buffer and fragment settings for low-latency typically comes as a trade off with video quality and re-buffering, especially during network instability. It’s recommended to thoroughly test applied settings against the range of expected network conditions.

For robust live video quality with low-latency across a variety of network conditions:
- Provide 5 – 10 fragments in the HLS Media Playlist and
- Use fragment length of 1 second.
To aggressively tune for latency (where network quality and / or media player supports):
- Provide 3 fragments in the HLS Media Playlist,
- Use fragments length of 0.5 seconds,
- Programmatically limit the media player buffer to 2 – 3 fragments depending on network quality and
- Set the media playback speed to slightly above x1 (x1.05 / x1.1) so as to keep up with the live edge of the media even during dropped fragments (this can cause more frequent re-buffering).

Ref: Latency too high between producer and player

Optimizing KVS HLS Playlist Settings for Live Media

The KVS Archived Media API supports the GetHLSStreamingSessionURL call to request the HLS Media Playlist and access media fragments as shown in the following (simplified) sequence:

Figure 1 – KVS Request (HLS) Playlist URL process flow.

When an application requests the HLS Playlist URL from KVS, a number of settings are available to optimize for low latency streaming media. The below example is a simple python script that makes a request to the KVS GetHLSStreamingSessionURL API to return the URL of the HLS Media Playlist for the requested stream and settings:

import boto3

KVS_STREAM_NAME = '[ENTER_KVS_STREAM_NAME]'
REGION = '[ENTER_AWS_REGION]'

#=====================================================
# Create the AWS Kinesis Video client
kvs = boto3.client("kinesisvideo", region_name=REGION)

#=====================================================
# Get the KVS endpoint for this stream
endpoint = kvs.get_data_endpoint(
APIName="GET_HLS_STREAMING_SESSION_URL",
StreamName=KVS_STREAM_NAME)['DataEndpoint']

print(f'\nKVS ENDPOINT: {endpoint}')

#=====================================================
# Get the KVS Archive media client
kvam = boto3.client("kinesis-video-archived-media", endpoint_url=endpoint, region_name=REGION)

# Get LIVE HLS URL
hls_playback_url = kvam.get_hls_streaming_session_url(
StreamName=KVS_STREAM_NAME,
PlaybackMode="LIVE",
MaxMediaPlaylistFragmentResults=3,
Expires=43200)['HLSStreamingSessionURL']

print(f'\nLIVE HLS_STREAMING_SESSION_URL: {hls_playback_url}\n')

The output of this script is the URL for the KVS Stream HLS Master Playlist (.m3u8) file. This URL can be passed directly as a link in supporting browsers such as Safari or to desktop media players like VLC to play the KVS media stream with the selected parameters. The HLS URL is returned with a SessionToken that provides time limited access to the KVS media so be careful with how you manage this URL.

In the above code, there are two parameters in particular to consider:

PlaybackMode: When set to LIVE, the HLS media playlist is continually updated with the latest fragments as they become available. This will typically display the live notification in the media player and in some players will select preconfigured fragment buffer settings more suitable to low-latency media.
MaxMediaPlaylistFragmentResults: The maximum number of fragments that are returned in the HLS media playlists. Limiting this can also limit fragment buffering by the media player and reduce latency. When the PlaybackMode is LIVE, the most recent fragments are returned up to this value. When the PlaybackMode is ON_DEMAND , the oldest fragments available are returned, up to this maximum number.

Therefore; to reduce latency when requesting the KVS HLS Playlist URL, ensure the PlaybackMode is set to LIVE and it’s recommended that MaxMediaPlaylistFragmentResults is set to at least 3 for low latency with robust media quality.

Reduce the Fragment Length Set by the KVS Media Producer

Fragment length of a media stream impacts latency as it determines the initial buffering and fragmentation time on the producer and the buffering time on the media player. Fragment lengths are set at the KVS producer and are preserved through the KVS service. And so, the fragment length created by the producer is what will be received and buffered by the media player or consumer. Below is an example of how to set the fragment length using a simple GStreamer media pipeline to KVS.

Using the GStreamer KVS Plugin:

The below GStreamer pipeline preforms a live screen capture on a MacOS device (avfvideosrc plugin), encodes the raw video to H.246 frames (vtenc_h264_hw plugin) and passes the encoded stream to the KvsSink plugin which ingests it to KVS. In the example, the requested frame rate from the camera and the key-frame interval on the encoder are both 20 FPS resulting in 1 second fragments.

gst-launch-1.0 avfvideosrc capture-screen=true \
! videoconvert \
! videoscale \
! video/x-raw,width=1280,height=720,framerate=20/1 \
! vtenc_h264_hw allow-frame-reordering=FALSE realtime=TRUE max-keyframe-interval=20 \
! h264parse \
! video/x-h264,stream-format=avc,alignment=au,profile=baseline \
! kvssink stream-name=$STREAM_NAME aws-region=$REGION access-key=$ACCESS_KEY secret-key=$SECRET_KEY retention-period=$RETENTION_HRS

To reduce the fragments to 0.5 seconds, adjust the key-frame interval to 10 by adjusting the max-keyframe-interval value. With a key-frame interval of 10 and a frame-rate of 20 fps this will trigger 0.5 second fragments by the kvssink element. It’s also valid in this example to increase the frame-rate to 40 fps by adjusting the framerate value if it’s supported by the camera but this will have a much greater impact on the network bandwidth and KVS consumption costs for the media stream.

Using the KVS Producer Libraries:

If using the KVS Producer Libraries, you as the developer are responsible for the number of frames per fragment and the fragment length using the programmatic means available to adjust key-frame interval and frame rate ingested to KVS.

Programmatically Reducing Media Player Fragment Buffering

If developing a custom web or mobile application for consuming KVS media, most popular media player libraries support programmatic tuning of settings such as the number of fragments to buffer. An example of a KVS integrated web application is the Amazon Kinesis Video Streams Web Viewer, an AWS provided code sample you can host in your own AWS account to test and consume KVS media.

KVS Web Viewer Demo Application:

Figure 2 – KVS Web Viewer Application Screen Shot.

The KVS Web Viewer uses the react-player which calls on HLS.js library to play HLS based media such as presented from KVS. The KVS Web Viewer exposes all of the programmatic settings available in the hlsConfigs.js file. In particular, the hlsConfig.js parameters liveSyncDurationCount and liveMaxLatencyDurationCount have been reduced from the defaults to optimize for lower latency.

Using this example, you can deploy a KVS consumer web application with HLS and buffer settings exposed via a simple configuration file for test and development in your environment.

Setting the Media Playback Speed to Slightly Above x1

Media players will often slip behind the live edge of the streaming media adding to video latency. This can be due to slight imperfections in timing between media producer and consumers or due to lost seconds during media re-buffering or network instability. Because this effect is accumulative, the impact is proportional to the length of the playback session and so the longer the media is expected to run in live playback mode the higher the impact. Therefore, this is especially relevant where a media source is displayed permanently on live monitor such as for a security camera or similar.

One technique to reduce latency being introduced due to media player falling behind the live edge is to set the media playback speed to slightly above x1 (i.e: x1.05 / 1.1 depending on media player support). This puts the media player into ‘Catch-Up’ mode that is constantly trying to out run the live edge. This keeps the media player as close to the live edge as possible but will in some cases cause the media to overrun the current fragment which will result in the media player being choppy. Because if this, it’s only recommended to use this technique for long running or permanently monitored media sources.

The KVS Web Viewer discussed supports both x1.05 and x1.1 playback speeds, use this as a means to test the increased playback speed technique in your environment.

Validating KVS Latency Values

In this section, I will test the described KVS HLS settings, fragment lengths, the custom KVS Web Viewer compared to direct viewing of a KVS stream in the Safari web browser and across global networks to validate latency values achievable under these specific conditions.

Test Set-Up:

This test uses the above described GStreamer pipeline to ingest a live screen capture from a MacOS device to KVS. The Mac screen displays a millisecond accuracy stopwatch and the same KVS stream is consumed on a second monitor for comparison. The test is being performed from Melbourne, Australia using the AWS Sydney Region across a physical distance of ~900Kms and therefore an ~1800 Km round trip.

Figure 3 – KVS latency test set-up and distance.

Using KVS Recommended Settings for Robust Video Quality:

HLS PlaybackMode: Live
HLS MaxMediaPlaylistFragmentResults: 7
Fragment Length: 1 Second

Figure 4 – KVS latency test for robust video quality results.

Using the KVS Web Viewer with latency optimized buffer configuration, the recommended KVS HLS settings for robust video quality and 1-Second fragment length, latency was measured at 1.95 Seconds.

This exact test was run consuming the KVS HLS Playlist URL directly in the Safari web browser and measured 4.072 seconds. This difference is due to the fact that we can’t control the number of fragments the Safari browser buffers and browsers are typically tuned for Video on Demand with media quality being a priority over low latency.

And finally, this test was run again with the KVS Web Viewer but in the N.Virginia region, a round trip from Melbourne Australia of over 30,000Kms (20,000 mi) which measured 3.32 seconds end to end latency. This gives an approximate scale to the impact of distance on media latency. This could be reduced further by using services like the AWS Global Accelerator.

Using KVS Latency Optimized Settings:

HLS PlaybackMode: Live
HLS MaxMediaPlaylistFragmentResults: 5
Fragment Length: 0.5 Seconds

Figure 5 – KVS latency test for latency optimised settings results.

By reducing the Fragment length to 0.5 seconds we were able to achieve an end to end latency of 1.412 seconds. Here, because we are using the KVS Web Viewer that allows us to programmatically configure the number of buffered fragments the MaxMediaPlaylistFragmentResults value that can otherwise influence media player buffer settings is not a leading contributor.

Running this same test but consuming the KVS media directly from the Safari web browser resulted in latency of 3.55 seconds.

Conclusion

In this series on how to reduce end to end latency of media managed by Amazon Kinesis Video Streams, in part 1; you learned the fundamentals of streaming media, KVS operation and common design patterns and finally the leading contributing factors to latency of streaming media in KVS.

In this, part 2; you learned the means in which to reduce latency of streaming media by reducing fragment lengths, media player buffer size and how to select optimal HLS playlist settings when requesting a HLS URL from the KVS API for low latency media.

Following this, you tested recommended settings for robust video quality and latency optimized media streams using the Amazon KVS Web Viewer. You also compared Amazon KVS Web Viewer to viewing the same media stream directly in the Safari web browser. These results were compared to geographically near and distant AWS Regions to also validate the impact of distance on media latency.

Through this testing I was able to demonstrate how to achieve latency on KVS managed media of 1.95 seconds using settings that deliver robust video quality over a variety of network conditions and latency of 1.412 seconds when latency optimized settings were applied.

About the author

Dean Colcott is a Snr IoT and Robotics Specialist Solution Architect at Amazon Web Services (AWS) and a SME for Amazon Kinesis Video Streams. He has areas of depth in distributed applications and full stack development, video and media, video analytics and computer vision, Industrial IoT and Enterprise data platforms. Dean’s focus is to bring relevance, value and insight from IoT, Industrial data, video and data generating physical assets within the context of the Enterprise data strategy.

The Internet of Things on AWS – Official Blog