Part 3: contextualized viewer engagement and monetization for live OTT events

This is the third and final post in the series of blog posts on contextualized viewer engagement and monetization for live over-the-top (OTT) events. This post details how Amazon Personalize, which allows developers to quickly build and deploy curated recommendations and intelligent user segmentation at scale using machine learning (ML), can be adopted to determine the right set of viewer engagement tools at the right point in time during an event.

This post is a deep dive into the solution architecture, emphasizing the design of its key solution blocks. It is recommended to read the first and the second posts of this series to get a better understanding of the problem that is being addressed through this solution.

Solution overview

To better illustrate the dynamics of real-time, contextualized viewer engagement and monetization, we will adopt a fictitious example of a live soccer match streamed OTT. The following sections of this post will reference this example for better articulation and clarity.

The solution entails two key elements: a dynamic component-based client application and a personalization server that can stream user experience customization instructions to client applications at every window of opportunity. The following section dives deeper into each of these solution elements.

Dynamic component-based client application: This is developed based on the microfrontend framework and implemented on ReactJS. Microfrontends open up the option to deliver the best-in-class engagement tool for the consumer by extending the system beyond its own enterprise boundaries. In this example, though live media streaming is the primary service, there can be collaboration with food delivery, ecommerce, sports analytics, betting, and other partners to deliver a compelling and wholesome consumer experience while they watch a live soccer match.

Figure 1. Wireframe for a typical OTT application

The client application in this example has one parent container and multiple child components embedded within it. These components are independently hosted and intelligently orchestrated to deliver a unified experience to the consumer. Such components can be selectively swapped based on the consumer persona, behavior, or on the content context. For example, with the scores leveled between the two teams and 5 minutes remaining in the match, would you like to place a bet, chat with a friend, order a pizza, or see the statistical probability of your favorite team winning? This is where the personalization server comes in.

Personalization server: This uses Amazon Personalize as its foundational component. This post assumes that the reader has a basic familiarity of the Amazon Personalize service. Please check out this link should a refresher be necessary. Amazon Personalize requires three sets of data, such as the user profile, item metadata, and the user-item interaction information. Classical implementations of Amazon Personalize assign media titles or merchandise goods as items. For this solution, we improvised and assigned a collection of web components as the item dataset. For a set of context inputs, Amazon Personalize would respond with the best set of items or web components that should be used by a client application to decorate the user experience.

Solution architecture

The sequence diagram below summarizes the overall system dynamics.

Figure 2. Sequence diagram for the overall system behavior

The client module creates a subscription request to the WebSocket server for contextual recommendations, proxied through the application server. The event context service listens for context state changes during the length of the event. Such context changes could be initiated by changing social media sentiments, player injuries, tiebreakers, weather conditions, or anything else that a consumer might be interested in. The event context service dispatches context change events to the WebSocket server, which then invokes the Amazon Personalize campaign to get contextual recommendations. This is denoted in the sequence diagram above as the “getComponents()” function call. The response from the Amazon Personalize campaign is further decorated by the WebSocket server by querying necessary event metadata or business-specific constraints. This is represented in the sequence diagram through the function call “getBizConstraints().” The enriched response is streamed back to the client, and it contains recommendations for the different web components that should be swapped out to augment the liveliness and interactivity of the user interface.

Client application design

Let us review how the client application handles the dynamicity to plug and swap web components. The parent application handles the connection with the WebSocket server and dynamically determines the child components that should be loaded based on a given context. Each child component is an independent application that is dynamically plugged into the parent container application. This architectural design not only delivers modularity, scalability, and separation of architectural concerns, but it also facilitates the potential of a syndicated business model as indicated in the earlier sections.

Please refer to the sequence diagram below for further details.

Figure 3. Sequence diagram for the client application behavior

Technical architecture

The diagram below outlines the overall technical architecture for this solution, followed by a synopsis of each of the key modules labeled in the diagram.

Figure 4. Technical architecture for contextual recommendation

1. Training data management: This module facilitates extraction of the training datasets for Amazon Personalize, such as the consumer profile and the clickstream. Amazon Glue, a serverless data integration service, is used for data catalog management of unstructured clickstream data and to initiate extract, train, load (ETL) jobs to deliver datasets in the required format to the target buckets in Amazon Simple Storage Service (Amazon S3)—an object storage service offering industry-leading scalability, data availability, security, and performance.
2. Amazon Personalize: Please refer to the section below for a detailed configuration overview of Amazon Personalize in terms of dataset ingestion, selection of recipe, training of solution, campaign setup, and facilitating an event tracker.
3. Real-time live stream analyzer: To build up context based on the streamed content, the live stream can be analyzed for its visuals and audio using managed services such as Amazon Rekognition, which offers pretrained and customizable computer vision, and Amazon Transcribe, which automatically converts speech to text. Please refer to the AWS blogs on how visual and audio information can be extracted from a media live stream.
4. Event of interest generator: This is another channel to capture live event data by subscribing to sources such as social media firehose (example: Twitter) or third-party sports analytics API or by harvesting from other social media sources. Such subscription streams are ingested into Amazon Kinesis Data Streams, a serverless streaming data service, using Kinesis Producer Library The stream is further analyzed through Amazon Kinesis Data Analytics, which is an easy way to transform and analyze streaming data in real time, to detect anomalies or events of interest (example: three goals scored in the last 10 minutes of a soccer match). Such anomalies are posted to Amazon Simple Notification Service (SNS), a fully managed Pub/Sub service for A2A and A2P messaging. Anomalies are subsequently dispatched to the context builder module.
5. Context builder: This module listens to the context information from deep analysis of the streamed content and from the signals detected from social media feeds or sports analytics services. This information is aggregated, refined, and decorated further with supplementary metadata information stored in Amazon DynamoDB, a fast, flexible NoSQL database service for single-digit millisecond performance. This component is built on Python running on a machine on Amazon Elastic Compute Cloud (Amazon EC2), secure and resizable compute capacity for virtually any workload.
6. Personalize WebSocket server: This module hosts the WebSocket server, which manages the client connections and subscription requests for user experience personalization and contextualization. This module communicates with the context builder and listens for the context event dispatch. Each such event is processed and enriched before being streamed back to individual client applications. This component is built on Python running on Amazon EC2 machines.

Amazon Personalize configuration

This section depicts how Amazon Personalize is configured for this solution. There are five key steps to be implemented.

1. Set up a dataset group and select “custom” as the domain name. The choice of item here is neither a media title nor a retail catalog. It consists of the components of a modular client application.

Figure 5. Configure dataset group in Amazon Personalize

2.  Upload three specific datasets, such as “user,” “item,” and “user-item.” It might be noted that only the “user-item” dataset is mandatory in Amazon Personalize. But inclusion of “user” and “item” with detailed relevant feature sets does help Amazon Personalize solutions to learn and generalize better, similar to any other ML-training job. The figures below provide sample illustrations of each of these datasets.

Figure 6. Sample data for users

Figure 7. Sample dataset for items

Figure 8. Sample dataset for user-item interaction

These datasets are first uploaded to an Amazon S3 bucket and subsequently imported into Amazon Personalize.

Figure 9. Confirmation of dataset import in Amazon Personalize

3. The Amazon Personalize recipe of “aws-user-personalization” is chosen to build the Amazon Personalize solution. The recipe of “aws-user-personalization” recommends a list of items for a given user. Feel free to experiment with different hyperparameter settings, depending on the available dataset and model performance requirements.

Figure 10. Creation of solution in Amazon Personalize

4. A real-time Amazon Personalize campaign is created, along with business-required customizations, such as threshold throughput, exploration-to-exploitation ratio, and more. This is a logical endpoint with a specific Amazon Resource Name (ARN) that can be used for user-item recommendation requests.

Figure 11. Campaign creation in Amazon Personalize

5. This is the key step for real-time contextualization, and it has two aspects. First, how the consumer is interacting with the user interface. Amazon Personalize provides an event tracker to capture the consumer clickstream and contextualize the recommendation output from campaign invocations.

Figure 12. Event tracker creation in Amazon Personalize

The second aspect is the context generation from the streaming content itself. This can be generated by live analysis of the video stream as explained in the second part of this blog series. In the middle of a live event, a consumer is less likely to interact with an application interface that might zero out the clickstream data and its associated context. The content context is the key driver in such scenarios.

Amazon Personalize does have the concept of moderating recommendations based on context. Depending on the use case, it is required to design suitable context variables for the user-item dataset. In this example of live OTT streaming for a soccer match, such variables are choice of match venue, match event (goal, yellow card, or player injury), team composition (players for a given match), type of match (league or finals), and score difference (level, 1, 1). These same context variables are passed as inputs to Amazon Personalize campaign requests. This facilitates refining the recommendation results based on the state of the soccer match.

Conclusion

In this three-part blog series, we outlined a complete perspective of how to maximize monetization and user engagement for live streaming content. The second part of the blog illustrated how Amazon Rekognition with an event driven serverless workflow can facilitate capturing content metadata for live streaming. This post outlined how Amazon Personalize can be used to deliver a real-time and contextualized user experience. It is left to the imagination and creativity of different businesses of how this solution framework can be adopted for different types of content, business objectives, and target consumers. Happy innovating!