[SEO Subhead]
This Guidance demonstrates how to conduct a semantic video search powered by generative artificial intelligence (AI). It streamlines the process of finding and extracting relevant video segments from vast amounts of video data. With natural language queries, it can quickly search for specific scenes, actions, concepts, people, objects, and more without the need to watch the full length of original videos. Designed to drive cost efficiency and scalability in media supply chain management, this Guidance is built for those who want to efficiently generate content such as trailers, sports highlights, captions, and descriptions from their existing video assets through automation.
Please note: [Disclaimer]
Architecture Diagram
[Architecture diagram description]
Step 1
Amazon Simple Storage Service (Amazon S3) hosts a static website for the semantic video search, served by an Amazon CloudFront distribution. Amazon Cognito provides customer identity and access management for the web application.
Step 2
Upload videos to Amazon S3 with Amazon S3 pre-signed URLs.
Step 3
After a video is uploaded successfully, an API call to Amazon API Gateway invokes AWS Lambda to queue new indexing-video requests in Amazon Simple Queue Service (Amazon SQS).
Step 4
Lambda processes new messages in the Amazon SQS queue, initiating an AWS Step Functions workflow.
Step 5
Amazon Rekognition detects multiple video shots from the original video, containing the start, end, and duration of each shot. Shot metadata is used to generate a sequence of frames, which are grouped by individual video shots and stored in Amazon S3.
Step 6
Step Functions uses the Map state to run a set of workflows for each frame stored in Amazon S3 in parallel.
Step 7
Amazon Rekognition detects celebrities and text in the frames.
Step 8
A foundation model (FM) in Amazon Bedrock generates image descriptions from the frames data and detected text or celebrities in the frames. The FM creates a description of the video shot based on a sequence of descriptions of its associated frames.
Step 9
An embedding model in Amazon Bedrock generates the embeddings of a video’s descriptions. Amazon OpenSearch Service stores the embeddings and other related information in a vector database.
Step 10
An embedding model in Amazon Bedrock generates the embedding of the users’ query, which is then used to perform semantic search for the videos from the OpenSearch Service vector database.
Step 11
Amazon DynamoDB tables store profiling and video indexing job metadata to keep track of the jobs’ status and other relevant information.
Step 1
Amazon Simple Storage Service (Amazon S3) hosts a static website for the semantic video search, served by an Amazon CloudFront distribution. Amazon Cognito provides customer identity and access management for the web application.
Step 2
Upload videos to Amazon S3 with Amazon S3 pre-signed URLs.
Step 3
After a video is uploaded successfully, an API call to Amazon API Gateway invokes AWS Lambda to queue new indexing-video requests in Amazon Simple Queue Service (Amazon SQS).
Step 4
Lambda processes new messages in the Amazon SQS queue, initiating an AWS Step Functions workflow.
Step 5
Amazon Rekognition detects multiple video shots from the original video, containing the start, end, and duration of each shot. Shot metadata is used to generate a sequence of frames, which are grouped by individual video shots and stored in Amazon S3.
Step 6
Step Functions uses the Map state to run a set of workflows for each frame stored in Amazon S3 in parallel.
Step 7
Amazon Rekognition detects celebrities and text in the frames.
Step 8
A foundation model (FM) in Amazon Bedrock generates image descriptions from the frames data and detected text or celebrities in the frames. The FM creates a description of the video shot based on a sequence of descriptions of its associated frames.
Step 9
An embedding model in Amazon Bedrock generates the embeddings of a video’s descriptions. Amazon OpenSearch Service stores the embeddings and other related information in a vector database.
Step 10
An embedding model in Amazon Bedrock generates the embedding of the users’ query, which is then used to perform semantic search for the videos from the OpenSearch Service vector database.
Step 11
Amazon DynamoDB tables store profiling and video indexing job metadata to keep track of the jobs’ status and other relevant information.
Well-Architected Pillars
The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.
The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.
-
Operational ExcellenceRead the Operational Excellence whitepaper
Use Amazon CloudWatch to monitor your workload and get insights into performance so that your application runs and responds effectively. Complement this with Step Functions to orchestrate your serverless workflow, handle errors automatically, and maintain consistency in processing. These services give you the observability and control you need to proactively identify and resolve issues.
-
SecurityRead the Security whitepaper
Utilize Cognito to authenticate users and authorize access to your resources. Pair this with CloudFront to securely deliver your content globally, with built-in distributed denial-of-service (DDoS) protection from AWS Shield Standard. These services ensure only valid users can access your application while safeguarding your data.
-
ReliabilityRead the Reliability whitepaper
Amazon S3 stores your videos, images, and other media objects with high availability and durability, minimizing the risk of data loss or downtime. It also supports features like versioning, cross-Region replication, and lifecycle policies, contributing to the integrity and availability of your data. Use this with CloudWatch to monitor your application and automatically respond to incidents when thresholds are breached. Finally, use Step Functions to orchestrate your serverless workflows, with built-in error handling and retry mechanisms that support consistent processing and fault tolerance.
-
Performance EfficiencyRead the Performance Efficiency whitepaper
Optimize your application's performance by taking advantage of serverless computing with Lambda. You can use this in tandem with Amazon Bedrock for high-performing machine learning inference, and benefit from the global reach of CloudFront to deliver content with low latency. These services work together to scale your application efficiently and deliver a responsive user experience.
-
Cost OptimizationRead the Cost Optimization whitepaper
Optimize costs by using Amazon S3 Intelligent-Tiering and Amazon S3 Lifecycle configuration to automatically store your media in the most cost-effective manner. Amazon S3 Lifecycle transitions files to cheaper storage tiers or deletes them based on rules, keeping your costs low. Lambda and Amazon Bedrock also help, as you only pay for the resources you use without provisioning servers. Lambda scales on demand, charging only for compute time used, and Amazon Bedrock is invoked as needed, avoiding idle infrastructure costs.
-
SustainabilityRead the Sustainability whitepaper
Lambda scales dynamically based on demand, improving efficiency and eliminating constant server provisioning. It also supports event filtering for Amazon SQS to reduce traffic to your Lambda functions. Complement this with Amazon S3 Intelligent-Tiering and Amazon S3 Lifecycle, which automatically shift data to the most suitable storage tiers based on usage. This minimizes expensive storage and unnecessary data. Finally, CloudFront caches content closer to users, lowering network traffic and energy consumption for data transfers.
Implementation Resources
A detailed guide is provided to experiment and use within your AWS account. Each stage of building the Guidance, including deployment, usage, and cleanup, is examined to prepare it for deployment.
The sample code is a starting point. It is industry validated, prescriptive but not definitive, and a peek under the hood to help you begin.
Related Content
[Title]
Disclaimer
The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.
References to third-party services or organizations in this Guidance do not imply an endorsement, sponsorship, or affiliation between Amazon or AWS and the third party. Guidance from AWS is a technical starting point, and you can customize your integration with third-party services when you deploy the architecture.