Part 3: How to compete with broadcast latency using current adaptive bitrate technologies

Part 1: Defining and Measuring Latency
Part 2: Recommended Optimizations for Encoding, Packaging, and CDN Delivery
Part 3: Recommended Optimizations for Video Players (this post)
Part 4: Reference Architectures and Tests Results

Part 3: Recommended Optimizations for Video Players

In previous installments of this blogs series, we looked at the possible optimizations that can be applied to encoding, packaging and CDN delivery steps in the video workflow. We now look at the most crucial area of latency improvement – the parameters of the video player. Even if you optimized your workflow parameters upstream, these efforts might be invalidated by a video player that doesn’t integrate low latency-oriented mechanisms. In this section, we’ll go through optimization suggestions for open source video players, and for the commercial players we’ll present the approaches of AWS Elemental technology partners castLabs and Accedo, who were kind enough to work with us on this topic.

Video Playback Optimizations

Video players are often optimized to provide uninterrupted playback to end-users, which means prioritizing the buffer length over lowering the stream latency. It does not mean that they totally lack options to enable low latency, but rather that these options are not enabled by default in each player’s initialization settings.

Here is a non-exhaustive list of problems that can prevent a player from providing the lowest possible latency for your streams:

• Initial buffer size: most players are designed to buffer a specific number of segments, of seconds, or of megabytes (MB), before triggering stream playback. With 1 and 2 second segments, this is usually fine and allows for under 10 second latency if the player doesn’t buffer more than three segments. But some players might be designed to buffer a specific amount of time if a long DVR window is presented in the live playlist/manifest. In this case, you would end up with 30 to 40 seconds buffered even if your segments are 1 second long, which prevents a low-latency experience. This is why you should check the player default buffering policy and look for ways to limit the buffer length at startup if the player is too conservative. Typically, limiting the buffer to 3 or 4 seconds is a reasonable trade-off between latency and playback stability. Going under 3 seconds can obviously improve the latency but also jeopardizes the user experience, with regular rebuffering phases happening alongside the playback session.
• Initial positioning of the player in the live timeline, compared to the live edge time: for most players, it’s redundant with the previous point, but not always. Setting a low buffer duration at startup in your player settings might be inefficient if it leverages another setting which forces the playhead to start with x segments or seconds of delay compared to the live edge time of your stream. That’s a complementary setting that you would then need to customize.
• Live Edge-Time stickiness: even if your player starts the playback with the expected delay compared to the live edge time, it’s likely that in the event of a rebuffering, it will resume the playback at the last known timecode before the rebuffering. Meaning that if your player takes as little as 100 ms to rebuffer, you will automatically be late by the same amount of time compared to the live edge time, after the rebuffering phase. This is usually what happens by default in all players, but some of them provide options to reload the playlist/manifest after a dry buffer effect (when the buffer for an audio or video track goes to zero seconds and gets stuck to it), or seek forward in time and stay tuned on the live edge time in such situations. If your priority is maintaining the lowest possible latency across the playback session, and not having the end-user go through every single second of content in the live session, this is the kind of option to leverage or to add if your player is open source. In any case, it’s a crucial feature to have in your player if you don’t want your latency to drift over time.
• Resilience to segments unavailability: it may be that a given media segment is not available at all or available with some delay compared to the player’s expectations. In this case, the player will retry to load the segment a couples of times and possibly stall the playback session if the segment is not available at all after all the retry attempts. For those use cases, you might want to look for player options to increase the retry attempts, to switch to lower bitrates, or to skip a missing slice in the timeline.

Open Source Players

hls.js

This open source HLS player for MSE (Media Source Extensions) environments does expose a lot of different parameters in its config.js initialization file. You should experiment and reduce some of its parameters:

• The maxBufferLength(default: 30s) this is the maximum number of seconds that the player will try to buffer
• The maxBufferSize(default: 60MB) this is the maximum content size in MB that the player will try to buffer
• The maxStarvationDelay(default: 4s) this is the maximum allowed number of seconds of rebuffering. Reducing it could prevent large rebuffering phases by forcing the player to switch to inferior bitrates.
• The liveSyncDurationCount(default: 3) this is the number of segments behind the last referenced segment at startup. Lowering it will make the player start closer to the live edge time.

Prior to hls.js 0.9.1, if you need to use a playlist reload interval inferior to one second, then you will have to reduce the hardcoded 1000 value in the following level-controller.js line:
reloadInterval = Math.max(1000, Math.round(reloadInterval));

dash.js

This open source DASH player for MSE environments provides several methods to set the initial delay compared to the live edge time. The player.setLiveDelayFragmentCount(default: 4) allows specification of the number of segments behind the live edge time, and player.setLiveDelayspecifies it in seconds. “The final achieved delay is a function of the segment duration, when the stream is requested with respect to the segment boundaries, as well as the amount of data the source buffers need to begin decoding”, says one of the live-delay player samples. The other methods parameters that you might want to customize are:

• player.setFragmentLoaderRetryInterval(default: 1000ms) to bring the failed fragment load attempt interval to a third of your segment’s length
• player.setFragmentLoaderRetryAttempts(default: 3) that you can increase to compensate for the previous customization
• player.setBandwidthSafetyFactor(default: 0.9) that you can lower to 0.5 to enforce a more conservative throughput calculation
• player.setStableBufferTime(default: 12s) that you can lower significantly to restrict the post startup/seeks buffer target and ease quicker bitrate switches
• player.setBufferToKeep(default: 20s) that you can lower significantly to restrict the source buffer length and allow a more reactive rotation of the buffered bitrate variants
• player.setBufferTimeAtTopQuality(default: 30s) that you can lower significantly to restrict the internal buffer target when playing the highest quality
• player.setLowLatencyEnabled(default: false) : starting from v.2.6.8, you can set this parameter to ‘true’ in order to leverage the browser fetch API instead of the traditional XHR loading mechanism. It has a very efficient impact on long DVR windows latency.

Exoplayer

This open source player for Android is compatible with several streaming formats, including HLS and DASH. In HLS, Exoplayer can have some difficulties with playlists referencing too few segments. In DASH, the suggestedPresentationDelay advertised in the manifest is honored by default. There are some parameters that you can modify in order to optimize the experience for low latency:

• You can increase the DEFAULT_MIN_LOADABLE_RETRY_COUNTvalue (default: 3) in HlsMediaSource and DashMediaSource classes, in order to allow more retries on segments in 404
• You can decrease DEFAULT_TRACK_BLACKLIST_MS(default: 60000) in ChunkedTrackBlacklistUtil class, in order to reduce the probability of a bitrate variant being blacklisted because of successive 404s
• Finally you can decrease the default buffer values in the DefaultLoadControl class: DEFAULT_MAX_BUFFER_MS(default: 3000), DEFAULT_BUFFER_FOR_PLAYBACK_MS(default: 2500) and DEFAULT_BUFFER_FOR_PLAYBACK_AFTER_REBUFFER_MS(default: 5000) in order to control more tightly how much content is loaded at the playback session startup, after a seek or a rebuffering

Shaka player

This open source HLS and DASH player for MSE environments provides several parameter options that can be modified in order to achieve lower latency, as the default values are conservative:

• streaming.bufferingGoal(default: 10s) is the amount of content seconds that the player is trying to buffer, it will impact the number of loaded segments
• streaming.rebufferingGoal (default: 2s) is the amount of content seconds that the player needs to have buffered before starting the playback. It’s valid for startup and rebuffering phases. If a DASH manifest minBufferTime is greater than this value, it will override it.
• retryParameters.maxAttempts (default: 2) is the maximum number of requests for a given segment before the player fails
• retryParameters.baseDelay (default: 1000ms) is the base delay between retries
• retryParameters.timeout (default: 0, equals never) is the delay in ms defining a request timeout
• retryParameters.backoffFactor (default: 2) is the multiplicator applied to baseDelay to amplify the delay between retries

All retryParameters can also be applied to manifests.

Commercial players

DISCLAIMER: The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.
These results are from testing using a reference architecture that match scenario 4 from part 4 of this blog series.

Common-configuration and Common-ABR for all castLabs Players
Our PRESTOplay suite of player SDKs feature a ‘Common-configuration’ framework. Using the same JSON objects across players, Common-configuration saves on development time when building playback apps across multiple devices and platforms and simplifies A/B testing across players.

Unique capabilities of the PRESTOplay SDKs allow:

• Same configuration to be used across all of castLabs’ PRESTOplay SDKs
• Easier development processes and unified end-user experience
• “Common-ABR”: Improved heuristics and buffering, including unique optimizations for low-latency live streaming
• Manual override of standard stream type “safety requirements” (such as overcoming the 3 segments latency for HLS) to achieve best-of-class latency

Common-configuration is complemented by our ‘Common-ABR’ feature. Common-ABR is the unique capability of castLabs’ PRESTOplay products to employ the same adaptive bitrate heuristic/algorithm and buffering logic across all player platforms. This explicitly includes iOS which other player vendors can only support via the native player with standard ABR and buffering logic. castLabs has developed Common-ABR with unique capabilities to achieve tightly controlled ultra low-latency far below industry averages and beyond typical guarantees/standards of current streaming protocols.

We’re able to achieve less than 5 seconds latency and can keep players as closely aligned as +/-0.5 second accuracy (as long as the encoder produces segments not longer than 2 seconds). In the best case castLabs’ players can achieve 2.5 second latency with 1 second segments for MPEG-DASH.

Our Common-configuration includes player initialization settings for live low latency scenarios. For example, the options below are relevant to low-latency playback and are the same across our Browser, Android, and iOS player technology. This will also be the same for all future player platforms we support. Below are the Common-configuration options which work equally across all PRESTOplay player platforms to enable the unique Common-ABR low-latency mode.

{
…
“lowLatencyMode”: true,
“suggestedPresentationDelayMs”: 0,
“lowLatencyCatchupMode”: “speedup|seek|none”,
“lowLatencyCatchupThresholdMs”: 3000,
“lowLatencyCatchupRate”: 1.1
...
}

When “lowLatencyMode” is enabled, the player will try to stick to the live and play as close to it as possible. If not specified explicitly, the distance from the live edge will be set to 0 in low latency mode. This can be further tweaked with the “suggestedPresentationDelayMs” parameter.

Dynamic live-edge tracking
The “lowLatencyCatchupMode” can be used to configure the behaviour of the player when it falls behind the current live edge. This can happen for instance when a buffer under-run occurs and the player is waiting for data. If the parameter is set to “none”, the player will not catch up and just continue to play. If set to “speedup”, the player will increase the playback rate once the current playback position is more than “lowLatencyCatchupThresholdMs” milliseconds behind the current live edge. The speedup rate can be controlled through the “lowLatencyCatchupRate” parameter and will be set back to 1.0 once the player is close enough to the live edge again. If the parameter is set to “seek”, the player will jump to the live edge if it falls behind more than the configured threshold.

Here are the results measured with AWS Elemental tests streams and castLabs players (the tests were performed by the AWS Elemental team):

At Accedo, our core focus is in enabling premium video user experience, and while we are not providing our own proprietary player, we have looked into best practices when it comes to video & live feed streaming. For us, it is both about stream start time & hand-waving latency.

The first parameter will have a direct impact on the UX as adding a perception of “delay” the longer this preparation phase takes while the second might affect the user if the buffer is not large enough to deliver a reliable feed. For developers we recommend to consider the following to optimise the stream start time:

• Limit (try to eliminate) synchronous calls (e.g. concurrency check) made before initiating the stream, performing them asynchronously and after the stream has begun
• Pre-fetch the license when a playback is anticipated (e.g. a user is on the asset detail page which will probably lead to a playback request)
• Pre-fetch entitlements and determine if the user is authorised to stream the asset
• Use QoE tools to analyze Time To First Frame and failed starts to aid in tuning settings as network characteristics vary from region to region

When it comes to the hand-waving latency, there is a balance to have between buffer size and latency. If we don’t consider the Apple HLS guidelines for segment size, it is indeed interesting to consider smaller chunks and see how parameters such as the duration of media to buffer when starting, while streaming, the duration of media to buffer after a rebuffering event, and also the maximum size of the buffer can play a role in decreasing latency.

iOS tests
Tests performed on iOS 11.2.5 with default-buffer-control=true and AVPlayer automaticallyWaitsToMinimizeStalling = true.

Android tests
Tests performed on Android 7.0 with Exoplayer 2.5.4 with parameters default-buffer-sizes-for playback-start=2.5s, default-buffer-sizes-after-rebuffer=5s, stream-download-buffe-minimum=15s, stream-download-buffe-maximum=30s

While these tests highlight how smaller segments allow for lower latency, we need to not forget how the stream reliability is key. At Accedo, we recommend setting buffer-related parameters in our AWS-powered Accedo One Cloud platform for our customer to control them & find out their optimal values. The delivery chain is unique per service deployment (Encoding, Packaging, CDN, regional network) and while any service launch should be initialized with a safe configuration, our customers should look into tailoring those parameters to their unique technology chain to decrease the latency as much as possible. Thanks to our A/B testing service called Optimise, our customers will be able to test different configurations and evaluate the impact on UX (re-buffering, playback retention time, etc.), leading them to define their own optimized setup.

In our next blog post, we’ll look at several solutions architectures leveraging AWS Elemental products and services, as well as the associated tests results in terms of end-to-end latency measured with reference players.

————

Part 1: How to Compete with Broadcast Latency Using Current Adaptive Bitrate Technologies – Defining and Measuring Latency

Part 2: How to Compete with Broadcast Latency Using Current Adaptive Bitrate Technologies – Recommended Optimizations for Encoding, Packaging, and CDN Delivery

Part 3: How to Compete with Broadcast Latency Using Current Adaptive Bitrate Technologies – Recommended Optimizations for Video Players (this post)

Part 4: How to Compete with Broadcast Latency Using Current Adaptive Bitrate Technologies – Reference Architectures and Tests Results