AWS for M&E Blog

Unified multimodal search across live events and VOD assets on AWS

In Part 1 of this series, we described how two ingestion pipelines—one for live broadcast events using a Time-Addressable Media Store (TAMS), and one for video on demand (VOD) assets using a media asset management (MAM) system—can enrich media with AI-generated embeddings, structured scene analysis, and speech transcription. Each piece of content is distilled into searchable documents containing four multidimensional vectors (visual, audio, speech, and combined channels), natural language descriptions, and full transcripts.

Now comes the harder question: How do you search it?

A single query such as “goalkeeper diving to save a penalty” needs to match against visual content (what the camera captured), spoken content (what the commentator said), and descriptive content (what the AI scene analysis produced). No single search technique handles all three. A keyword search finds the word “penalty” in transcripts but misses visual-only scenes. A vector search against video embeddings captures the visual action but might miss a moment when the commentator describes a penalty without showing it.

This post describes how a multi-leg hybrid search architecture solves this problem by combining vector similarity and keyword matching across multiple embedding channels in a single query, with results spanning both live events (searchable within 90 seconds of broadcast) and VOD archives simultaneously.

A single query, five search strategies

When a user submits a natural language query, the search system doesn’t rely on a single matching strategy. Instead, it generates multiple mathematical representations of the query and executes five parallel searches against different document fields in a single round trip to Amazon OpenSearch Service:

User query: "goalkeeper diving to save a penalty"

│──> Cross-modal embedding (Marengo) ─────> kNN search: visual channel (leg 1)

│──> Cross-modal embedding (Marengo) ─────> kNN search: fused channel (leg 2)

│──> Cross-modal embedding (Marengo) ─────> kNN search: transcription channel (leg 3)

│──> Text embedding (Titan) ────────────> kNN search: text descriptions (leg 4)

│──> Keyword matching (BM25) ──────────> Full text: transcripts, descriptions, titles, tags (leg 5)

│

Score normalization + Fusion

│

Ranked results (live events + VOD assets combined)

Figure 1: A single natural language query generates embeddings from two models (Marengo cross-modal + Amazon Bedrock Titan text) and executes five parallel searches: three k-Nearest Neighbor (kNN) legs on video, audio, and speech channels, one kNN leg on text descriptions, and one BM25 keyword leg, fused into a single ranked result set

Two embedding models in parallel

TwelveLabs Marengo 3.0 (using Amazon Bedrock) converts the text query into the same 512-dimensional space used for video and audio content at ingest. This is cross-modal: Text can be mathematically compared to video. The Marengo vector searches three document channels: visual (matching what is seen), fused (overall content), and transcription (what is said, searching semantically rather than by keyword). These correspond to legs one to three in Figure 1.

The fourth leg is enabled by Amazon Titan Text Embeddings V2, which produces a 512-dimensional text-only vector used to search against descriptions of our assets generated by Amazon Bedrock Data Automation (Amazon BDA). This same-modal signal is strongest when the query closely matches how the AI described a scene.

Both models complete in approximately 250 milliseconds running in parallel.

BM25 keyword search

The fifth leg is traditional BM25 full-text search across titles, transcripts, scene descriptions, tags, and key moment labels. This handles proper nouns (for example, “Bukayo Saka”), exact phrases (for example, “VAR review”), and terminology that embedding models might not capture. For example, “Ronaldo” and “Mbappé” are semantically nearly identical to a model, but a researcher searching one doesn’t want the other.

Score normalization and fusion

Raw scores from different legs are incomparable. A cosine similarity of 0.25 might be an excellent visual match, while a BM25 score of 3.0 might be barely relevant.

The system uses per-leg normalization with empirically calibrated bands. Each leg has a noise floor and a ceiling:

normalized_score = clamp(0, 1, (raw_score - floor) / (ceiling - floor))

For example, the Amazon Titan Text Embeddings text leg uses floor=0.04, ceiling=0.55. The BM25 leg uses floor=2.0, ceiling=10.0. These values are corpus-dependent and measured by scoring known-unrelated pairs to find the noise floor and known-matching pairs to find the ceiling.

Strongest signal wins

After normalization, each document’s final score is the maximum of its normalized leg scores. A document that perfectly matches the transcription leg (the commentator said what the user searched for) isn’t diluted by a poor visual match (camera showing a replay). The best leg advocates for the document.

The tradeoff: Max fusion trusts a single leg. If any channel produces a spuriously high score, that document is promoted on its strength alone, making guards against degenerate embeddings essential. A configurable minimum threshold filters documents that don’t reach meaningful confidence on any leg, preventing weak matches from filling the result set.

Alternative: Reciprocal Rank Fusion

For recall-prioritized scenarios, Reciprocal Rank Fusion (RRF) ranks documents by position within each leg rather than absolute scores.

This recovers documents that rank highly on one leg even with low absolute cosine, which is common for visual searches where scores cluster in a narrow range.

Reverse media search

Beyond text, the system supports searching with media. Users can upload an image, video clip, or audio sample to find similar content across the catalog. The following figure shows how you can architect this capability within AWS:

AWS architecture diagram for multimodal asset search in us-east-1. Users upload assets to an S3 bucket and access a frontend via CloudFront and API Gateway, authenticated by Cognito. A Lambda function generates embeddings with Amazon Bedrock, runs kNN search in Amazon OpenSearch, and fetches metadata from DynamoDB before returning matched results.

Figure 2: Reverse media search generates per-channel embeddings from the uploaded file and performs channel-matched kNN search with visual queries searching visual embeddings and audio queries searching audio embeddings, preventing cross-modal confusion

The uploaded media is submitted to Marengo, generating per-channel embeddings identical to those stored at ingest. Each query channel then searches only its corresponding document channel (visual searches visual, audio searches audio). This channel-matched design is critical for precision. When a user uploads a clip of a crowd cheering, the audio channel must match documents where similar noise was heard, not documents that happen to show a crowd visually but with different sound.

For cross-modal search (for example, finding video content from a still image), the fused channel bridges modalities. Both share the same embedding space, meaning an image of a stadium can surface video recorded at that location.

In our testing, median backend search latency, including embedding generation, Amazon OpenSearch Service retrieval, and result preparation was approximately 1.2 seconds for image queries using synchronous InvokeModel API calls and approximately 6.5 seconds for 4-second video and audio clips using StartAsyncInvoke with polling, measured on warmed-up AWS Lambda functions and excluding file uploads. (These measurements cover short reference clips and don’t establish how latency scales with content duration.)

Silent audio guard

Marengo generates a near-constant placeholder vector for silent content (title cards, still images). Without mitigation, any quiet query would score nearly perfectly against silent documents.

The system stores a silence fingerprint and drops the audio channel from queries where the embedding falls within the threshold. This is a pattern that generalizes to any modality producing degenerate embeddings.

Searching live events in real time

One of the solution’s most compelling capabilities is finding moments within a broadcast while it’s still happening.

Because the TAMS ingestion pipeline indexes segments within 60–90 seconds of capture, the search index grows continuously during a live event. A user searching “penalty kick” during a live football match can find the moment within seconds of indexing, complete with a playback URL that seeks directly to that point in the livestream.

Key moments detected by AWS Elemental Inference (within 20–30 seconds of on-air occurrence) surface as top-level results with rich labels, outranking generic segments because their concentrated metadata (moment type, player name, AI-generated description) produces higher scores across both keyword and semantic legs. This means the system answers the question “what just happened?” before a content manager could scrub to the right timecode manually.

Unified results across inventory systems

The search spans both indices—live events (TAMS) and VOD assets (MAM)—in a single request. The default returns unified results ranked by content relevance, not source system.

A VOD clip with a perfect description match outranks a live segment with only a moderate visual match.

Result grouping

Raw hits are flat documents in the form of individual segments and moments. Results resolving to the same underlying asset are collapsed into one entry carrying its strongest score, then organized for consumption:

  • Live events – Matching segments grouped under the parent event with HTTP Live Streaming (HLS) playback URLs
  • VOD assets – Matching clips grouped under the parent asset with thumbnails and streaming URLs
  • Key moments – Standalone top-level results with labels and precise timestamps

A search for “celebration” might return a key moment (“Goal celebration: 73rd minute”), a live event with three matching segments, and a VOD highlight reel with two clips. Backend search latency includes query embedding generation, Amazon OpenSearch Service retrieval, and result grouping and enrichment. For text queries, the Marengo and Amazon Titan Text Embeddings are generated in parallel.

The following table summarizes median backend search latency measured in our prototype tests on warmed-up AWS Lambda functions, excluding file uploads and browser rendering. Video and audio queries used 4-second reference clips; these measurements do not establish how latency scales with content duration.

Query type Median search latency
Text search Approximately 1.1 seconds
Reverse image search Approximately 1.2 seconds
Reverse video and audio search (4-second reference clip) Approximately 6.5 seconds

Tuning and optional reranking

Search behavior—including normalization bands, fusion mode, score thresholds, active legs—is controlled using environment variables, facilitating relevance tuning in minutes without redeployment. We recommend bands be recalibrated when the content mix changes significantly.

For maximum precision (editorial curation, compliance), an optional cross-encoder reranking step (using Amazon Bedrock) rescores top candidates holistically, capturing negation, qualification, and multipart queries that embedding retrieval might miss. This adds 500–1,000 milliseconds and is turned on or off per environment.

The whole picture

Taken together with the ingestion pipelines in Part 1, this architecture delivers on the vision of a content intelligence layer that spans media inventory systems:

  • Content is enriched automatically at ingest with no manual tagging required
  • Live events are searchable within 90 seconds of broadcast capture
  • A single search spans TAMS live events and MAM VOD archives
  • Natural language queries match visual, audio, spoken, and descriptive content
  • Reverse media search finds similar content from an image, video clip, or audio sample
  • Key moments are detected and surfaced as first-class results
  • The AI layer is additive; existing TAMS tools and workflows continue unchanged

The producer looking for “the moment the goalkeeper dives to save a penalty” no longer needs to know which system holds the footage, or manually scrub hours of video to find it. They can use one query across everything.

For media organizations operating across multiple content systems, unified content discovery is achievable using AWS services. The approach is extensible. Adding a new inventory source requires implementing one ingest connector while the search infrastructure, models, and user experience remain the same.

To discover how your organization can use unified content discovery powered by AWS, contact your AWS representative.

Further reading

Mohan Kumar Guruswamy

Mohan Kumar Guruswamy

Mohan Kumar Guruswamy is an applied AI solutions engineer on the Prototyping and AI Customer Engineering (PACE) team at Amazon Web Services (AWS). Mohan partners with customers to build working prototypes on AWS using their own use cases and data, demonstrating what's possible and accelerating informed decisions on their cloud journey.

Ankit Patel

Ankit Patel

Ankit Patel is an applied AI solutions engineer on the Prototyping and AI Customer Engineering (PACE) team at AWS. Ankit helps customers bring their innovative ideas to life by rapidly prototyping, using AWS to build, orchestrate, and manage custom applications.