MISB ST 1910 and KLV: Changing the game in intelligence, entertainment, and beyond

Introduction

From media and entertainment (M&E) to defense, science, and agriculture, video is pervasive. Whether shot on a high-end camera in a studio or for a tactical mission from an unmanned aerial system (UAS), video and the environment in which it’s captured can provide invaluable information, such as sensor data, geolocation, instrument readings, or other forms of telemetry. Historically, military and intelligence organizations have leveraged Key-Length-Value (KLV), defined by the Society of Motion Picture and Television Engineers (SMPTE), to carry important information alongside high-definition video collected by UAS for intelligence, surveillance, and reconnaissance operations. While highly effective, previously this data was not being carried using the latest adaptive bitrate streaming (ABS) protocols, hindering organizations from capitalizing on the benefits of cloud-powered mission video systems.

Identifying these short-comings, a team at Amazon Web Services (AWS) developed a solution that would facilitate the carriage of KLV metadata in-band of open standard ABS protocols. Further, in collaboration with the National Geospatial-Intelligence Agency’s Motion Imagery Standards Board ¬(MISB) – the organization responsible for managing interoperable standards for the US military and intelligence agencies – AWS participated in the standardization of this practice to create MISB Standard 1910. These technology advancements are made possible through AWS Elemental MediaLive video processing and AWS Elemental MediaPackage video packaging via cloud, an evolution that also provides new benefits across a range of industries including media and entertainment, sports, and science.

Connecting the dots between video in UAS and M&E

Before diving into the new approach and its implications, it’s important to take a step back and examine the characteristics of video in the UAS industry compared to the M&E industry. There are two terms used within UAS communities that characterize mission video: Full Motion Video (FMV) and Motion Imagery (MI). FMV implies a very narrow subset of MI, one that assumes geospatial metadata, commercial image format, and a playback rate. A few commonly encountered FMV/MI formats include 1080p60, 720p60, 1080p30, or 720p30, in addition to codecs such as H.265/HEVC or H.264/AVC. For M&E professionals, these are familiar video formats and codecs, as is KLV, the associated metadata encoding technique used in FMV/MI systems. Specified by SMPTE ST 336M, KLV is actually a well-known data encoding standard. In fact, many video and metadata technologies used in UAS, KLV included, are derived from the professional M&E industry.

Breaking down KLV

The UAS community has long standardized on the use of KLV to carry dozens of fields of data from aerial sensing systems. Just as video wouldn’t be carried without captions in broadcast, video in mission systems wouldn’t be carried without KLV data. At its core, KLV comprises a trio of data elements: key, length, and value. The key simply identifies the data, such as “Sensor Latitude,” the length expresses the length of the data in bytes, and the value is a numeric or character set representing the actual data itself. KLV extends a highly interoperable and efficient data encoding technique, making it ideal for carrying nearly any kind of information imaginable with low protocol overhead, while also offering optimal video-to-metadata utilization over bandwidth-constrained environments. Interestingly, the standard permits the grouping of KLV data elements into related sets to reduce overhead and improve bit efficiency. For example, the MISB defines a Local Data Set (LDS) prescribed to its UAS data types. Similarly, organizations may opt to define an LDS applicable to their data types, eliminating Universal Label ambiguity. KLV data elements are packed into larger KLV PDUs (Protocol Data Units) that organize the data into a stream. This data stream can then be multiplexed within an MPEG-2 transport stream (MPEG-TS) and synchronized with other media elements such as video and audio for transmission over terrestrial and IP networks. Receivers on the other end tune to the stream where the MPEG-TS is demultiplexed and individual media elements are decoded. The resulting video, audio, and metadata can then be presented back to a viewer in a synchronous playback fashion.

The influence of video distribution protocols on KLV

Historically, FMV/MI streams would flow end-to-end over vast IP networks using MPEG-TS and either UDP/IP or RTP/IP encapsulation. While MPEG-TS continues to be a staple in the video contribution space, consumers of video at the other end are turning to newer, more efficient, adaptive streaming protocols such as CMAF, MPEG-DASH, and HLS to satisfy scalable video distribution demands. Adaptive streaming protocols extend a variety of performance enhancements for distribution, including the use of the HTTP(S) protocol, which greatly improves stream reliability between sender and receiver over varying network conditions. The introduction of adaptive bitrate (ABR) technologies has also enabled video professionals to dynamically tune client-side picture quality based on bandwidth availability. Furthermore, inherent scalability, or one-to-many distribution techniques used in segmented-file delivery systems, combined with cloud-based distribution have made it simpler to disseminate live and file-based content over managed or unmanaged networks. Thanks to these and many other advancements in video distribution protocols, several improvements to KLV handling were able to be realized.

Envisioning a new approach

Leveraging ISO Base Media File Format (ISO BMFF), the teams at AWS and MISB envisioned a new way to carry KLV data using Event Message (emsg) Boxes, allowing web-enabled players a standards-based approach to the orchestration and playback of media and associated event messages. Event Messages were introduced as part of the MPEG-DASH specification, augmenting content with supplemental information important to the timing and relationship of media during playback. The handling of KLV in emsg boxes is now made possible using AWS Elemental MediaLive and AWS Elemental MediaPackage. These managed cloud services create a bridge between legacy MPEG-TS containing KLV to newer adaptive streaming protocols with KLV. Moreover, the practice is consistent with broadly adopted CMAF and MPEG-DASH specifications, while compliant with the newly formed MISB ST 1910, supporting the carriage of KLV metadata in ABR content encoding. This means that UAS video and metadata can seamlessly interface with newer cloud-based architectures, allowing the processed stream to be made accessible to a myriad of web-enabled video players. This is similar to a Netflix-, Hulu-, or YouTube-style implementation for mission video, leveraging over-the-top (OTT) streaming technologies.

Exploring MISB ST 1910 applications in intelligence, M&E, and science

Using the method described in the preceding paragraph, FMV/MI can now use web and mobile platforms for the playback of video, audio, and associated KLV metadata. Similar to the OTT industry, these modernizations are game-changing for UAS, because previously many dashboard experiences required heavy, thick-client applications to manage video and associated KLV metadata. Today, military, intelligence, and even first responders can quickly access mission-critical content on nearly any device in near-real-time through the power of modern web-based video dissemination methods.

M&E professionals who leverage streaming content can also look to KLV metadata as an enabler to carry available data sources alongside video to build new immersive, metadata-rich fan experiences. For instance, a motorsports broadcaster could fuse in-car telemetry with racing video, or a multimedia provider could help viewers feel closer to the live stream of a shuttle launch by displaying tracking and propulsion readings in a simulated heads-up display (HUD). The real-time aspect of carrying metadata in-band with video could play a key role in scientific explorations. Imagine a group of scientists conducting deep-sea research using remotely operated underwater vehicles (ROVs). These ROVs could be equipped with special sensors capable of sending important metadata relative to the video, such as operating depth and temperature, while the scientists observe the behavior of species as it happens.

Conclusion

Whether your data represents longitude and latitude coordinates for UAS, or engine manifold pressure in motorsports, using KLV provides a universal, self-describing data encoding method to carry rich contextual information in-band with video from contribution to distribution. Not only will this allow UAS video processing to leverage the AWS Cloud for intelligence, surveillance, and reconnaissance workflows, but it will also open up new opportunities to carry data from a range of sensor systems across industries creating spatial and situational awareness in video-centric applications. A number of APN Partners have already begun developing solutions using the new standard and associated services to implement innovative, web-native geospatial capabilities for streaming video. The possibilities are nearly infinite, and we can’t wait to see how MISB ST 1910 and KLV are used to develop other applications in the future.

To learn more, please visit: https://aws.amazon.com/media/resources/government/