Generate video summaries with voice narration, powered by computer vision technology
This Guidance demonstrates how Amazon Sagemaker and Large Language Models (LLMs) can be used to create short-form video summaries compiled from a longer, original video file. The summary is used to identify the most relevant video segments, which are compiled into a final video output with voice narration. It helps media organizations automate the process of generating video summaries to enhance the viability, scalability, and efficiency of content production throughout the supply chain process. It also helps media organizations improve their audiences' experiences through personalized content.
Please note: [Disclaimer]
Architecture Diagram
[text]
Step 1
Amazon Simple Storage Service (Amazon S3) hosts a static website for the video summarization workload, served by an Amazon CloudFront distribution. Amazon Cognito provides customer identity and sign-in functionality to the web application.
Step 2
Amazon S3 stores the source videos, which are uploaded through pre-signed URLs.
Step 3
To perform a video summarization, make an API call to Amazon API Gateway that invokes an AWS Lambda function to put the request into an Amazon Simple Queue Service (Amazon SQS) queue. New messages added to the queue are processed by a Lambda function that processes a new AWS Step Functions workflow.
Step 4
Amazon Transcribe converts the speech in the source video into text, generating an output subtitle file containing the transcript and timestamps.
Step 5
Amazon SageMaker foundation model endpoint summarizes the text, retaining the story from the original video but in shorter form.
Step 6
Amazon Polly generates a voice narration. The SageMaker text embedding model endpoint pairs each sentence in the summarized text with its corresponding sentences in the original subtitle file. The output is the most relevant video segments and their timestamps.
Step 7
AWS Elemental MediaConvert transcodes the final video output using the original video input clipping timestamps. It inserts the voice narration, generated from Amazon Polly, with optional background music of your preference.
Step 8
Amazon S3 stores the output video that offers durable, highly available, and scalable data storage at low cost.
Step 9
Amazon DynamoDB tables store profiling and task metadata. This helps you keep track of the tasks’ status and other relevant information.
Step 10
Amazon CloudWatch and Amazon EventBridge monitor in near real-time every component, and can be used to integrate this workflow into other systems.
Step 1
Amazon Simple Storage Service (Amazon S3) hosts a static website for the video summarization workload, served by an Amazon CloudFront distribution. Amazon Cognito provides customer identity and sign-in functionality to the web application.
Step 2
Amazon S3 stores the source videos, which are uploaded through pre-signed URLs.
Step 3
To perform a video summarization, make an API call to Amazon API Gateway that invokes an AWS Lambda function to put the request into an Amazon Simple Queue Service (Amazon SQS) queue. New messages added to the queue are processed by a Lambda function that processes a new AWS Step Functions workflow.
Step 4
Amazon Transcribe converts the speech in the source video into text, generating an output subtitle file containing the transcript and timestamps.
Step 5
Amazon SageMaker foundation model endpoint summarizes the text, retaining the story from the original video but in shorter form.
Step 6
Amazon Polly generates a voice narration. The SageMaker text embedding model endpoint pairs each sentence in the summarized text with its corresponding sentences in the original subtitle file. The output is the most relevant video segments and their timestamps.
Step 7
AWS Elemental MediaConvert transcodes the final video output using the original video input clipping timestamps. It inserts the voice narration, generated from Amazon Polly, with optional background music of your preference.
Step 8
Amazon S3 stores the output video that offers durable, highly available, and scalable data storage at low cost.
Step 9
Amazon DynamoDB tables store profiling and task metadata. This helps you keep track of the tasks’ status and other relevant information.
Step 10
Amazon CloudWatch and Amazon EventBridge monitor in near real-time every component, and can be used to integrate this workflow into other systems.
Well-Architected Pillars
The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.
The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.
-
Operational ExcellenceRead the Operational Excellence whitepaper
CloudWatch, EventBridge, Lambda, API Gateway, and Step Functions are AWS services purpose-built to enhance your operational excellence when operating this Guidance. CloudWatch monitors and collects metrics, logs, and events from Lambda, API Gateway and Step Functions. In addition, using CloudWatch helps you gain insights into the workload performance and identify issues quickly through the proactive monitoring, automated anomaly detection, and near real-time visualization of data. With EventBridge, routing and processing events from Lambda, API Gateway, Step Functions and other sources are supported to make responsive actions during the workflow. EventBridge also reduces tight coupling and enables asynchronous interactions between workflow components, providing an event-driven architecture that can respond effectively to changes while being easily extended.
-
SecurityRead the Security whitepaper
Amazon Cognito, Amazon CloudFront and AWS Shield Standard are deployed in this Guidance to enhance your security. Amazon Cognito enables you to manage user identities and authentication for the application. CloudFront improves website security with secure data transmission and access control, limiting unauthorized access to AWS resources by using origin access control (OAC). Shield Standard is automatically enabled when you use AWS services, such as CloudFront, and defends against the most common, frequently occurring network and transport layer distributed denial of service (DDoS) attacks.
-
ReliabilityRead the Reliability whitepaper
Amazon S3, CloudWatch, EventBridge, and Lambda are used throughout this Guidance to enhance the reliability of your workloads. Amazon S3 does this by storing objects, including videos and other media file formats, in a highly scalable, highly available manner. Amazon S3 also supports versioning, which enables you to retain and manage historical data versions, contributing to data integrity and availability. EventBridge and CloudWatch monitor and respond to events, such as changes in states in CloudWatch alarms or in Step Functions, to process Lambda functions with remediation logic for the workload.
-
Performance EfficiencyRead the Performance Efficiency whitepaper
Lambda, DynamoDB, Amazon S3, and CloudFront help enhance your performance efficiency. With Lambda, you can run code with zero administration, as it will manage everything required to run and scale the code. It also offers event-driven scaling, improving application performance and resource efficiency. Furthermore, DynamoDB provides a fully managed NoSQL database with single-digit millisecond latency at any scale, where all the profiling and video summarization task metadata is stored. Moreover, using Amazon S3 to host static web applications and store media files reduces latency and increases throughput in data access. Additionally, CloudFront includes a caching capability and this service, coupled with Amazon S3, brings content closer to the users with a global network of 550+ Points of Presence, further improving performance and reducing latency.
-
Cost OptimizationRead the Cost Optimization whitepaper
Amazon S3, DynamoDB, and Lambda enhance your cost optimization framework in a number of ways. For one, Amazon S3 Lifecycle Management manages your videos and media objects so that they are stored cost effectively throughout their lifecycle. Second, DynamoDB Time to Live (TTL) allows you to define a timestamp by item to determine when an item in the table is no longer needed. And third, with Lambda, you're only charged for the actual compute time used during the processing of your code. There are no charges for idle resources, as Lambda automatically scales resources up or down based on the demand.
-
SustainabilityRead the Sustainability whitepaper
Lambda, Amazon S3, DynamoDB, and CloudFront are all used in this Guidance to enhance the sustainability of your workloads. The Lambda serverless architecture dynamically allocates resources based on demand, ensuring efficient utilization and reducing energy consumption. It also provides content filtering options for Amazon SQS, so that the Lambda function is only invoked by Amazon SQS under the filtering criteria you specify, which reduces Lambda function processing. Amazon S3 Intelligent-Tiering storage class and Lifecycle features help automatically move data to appropriate storage tiers based on access patterns, reducing the need for higher-cost storage and deletes unnecessary data. DynamoDB automatically scales read and write capacity based on usage, eliminating over-provisioning and minimizing energy consumption associated with excessive resources. Finally, CloudFront, with global edge locations, reduces the need to repeatedly access origin servers, conserving resources and reducing energy consumption by serving cached content closer to end users.
Implementation Resources
A detailed guide is provided to experiment and use within your AWS account. Each stage of building the Guidance, including deployment, usage, and cleanup, is examined to prepare it for deployment.
The sample code is a starting point. It is industry validated, prescriptive but not definitive, and a peek under the hood to help you begin.
Related Content
Video summarization with AWS artificial intelligence (AI) and machine learning (ML) services
Disclaimer
The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.