Translate multiple languages into American Sign Language (ASL) with an avatar-based translator
This Guidance demonstrates how to build an application that provides multimodal, high-fidelity translations from over 100 spoken languages into ASL through a realistic avatar. It uses generative artificial intelligence (AI) to transcribe and simplify input phrases from any supported language. Each simplified English phrase is then interpreted, rearranged, and translated directly into ASL. The application also generates a background image from Stable Diffusion to visually portray the input phrase.
Please note: [Disclaimer]
Architecture Diagram
[Architecture diagram description]
Step 1
User authenticates to Amazon Cognito using an Amazon CloudFront-hosted website or web API (through Amazon Cognito-based JWT access token).
Step 2
User types or speaks an input phrase in a chosen language, which Amazon Transcribe transcribes. Transcription is stored in an Amazon Simple Storage Service (Amazon S3) bucket.
Step 3
User requests an action (like ASL translate, change avatar, or change background image) through the website or web API (Amazon API Gateway endpoint).
Step 4
Based on the user-requested action, API Gateway routes its request to a corresponding AWS Lambda function for processing that action.
Step 5
For ASL Translation requests, a matching Lambda function invokes Amazon Bedrock to form an ASL phrase for the provided input phrase and to obtain a contextual 2D image (to be stored in an S3 bucket).
Step 6
Amazon Comprehend and Amazon Bedrock perform multi-lingual toxicity checks on the input phrase. Amazon Rekognition performs visual toxicity checks on 2D-generated images. Toxicity check results are returned to respective Lambda functions.
Step 7
All Lambda functions generate a JSON-based payload to capture a user-requested action for Epic Games Unreal Engine. Each payload is sent to a corresponding Amazon Simple Notification Service (Amazon SNS) topic: Translation and Non-Translation.
Step 8
Each Amazon SNS-based payload is transmitted to its corresponding Amazon Simple Queue Service (Amazon SQS) queue for later consumption by Unreal Engine.
Step 9
Using the AWS SDK, the Unreal Engine application polls and dequeues Amazon SQS action-based payloads from its queues. Background images are fetched from an S3 bucket for translation requests.
Step 10
Based on each payload received, the Unreal Engine application performs a user-requested action and displays resulting video output on that user’s system. This output provides an ASL-equivalent interpretation of an input phrase by displaying a MetaHuman 3D avatar animation with ASL-transformed text displayed.
Step 1
User authenticates to Amazon Cognito using an Amazon CloudFront-hosted website or web API (through Amazon Cognito-based JWT access token).
Step 2
User types or speaks an input phrase in a chosen language, which Amazon Transcribe transcribes. Transcription is stored in an Amazon Simple Storage Service (Amazon S3) bucket.
Step 3
User requests an action (like ASL translate, change avatar, or change background image) through the website or web API (Amazon API Gateway endpoint).
Step 4
Based on the user-requested action, API Gateway routes its request to a corresponding AWS Lambda function for processing that action.
Step 5
For ASL Translation requests, a matching Lambda function invokes Amazon Bedrock to form an ASL phrase for the provided input phrase and to obtain a contextual 2D image (to be stored in an S3 bucket).
Step 6
Amazon Comprehend and Amazon Bedrock perform multi-lingual toxicity checks on the input phrase. Amazon Rekognition performs visual toxicity checks on 2D-generated images. Toxicity check results are returned to respective Lambda functions.
Step 7
All Lambda functions generate a JSON-based payload to capture a user-requested action for Epic Games Unreal Engine. Each payload is sent to a corresponding Amazon Simple Notification Service (Amazon SNS) topic: Translation and Non-Translation.
Step 8
Each Amazon SNS-based payload is transmitted to its corresponding Amazon Simple Queue Service (Amazon SQS) queue for later consumption by Unreal Engine.
Step 9
Using the AWS SDK, the Unreal Engine application polls and dequeues Amazon SQS action-based payloads from its queues. Background images are fetched from an S3 bucket for translation requests.
Step 10
Based on each payload received, the Unreal Engine application performs a user-requested action and displays resulting video output on that user’s system. This output provides an ASL-equivalent interpretation of an input phrase by displaying a MetaHuman 3D avatar animation with ASL-transformed text displayed.
Get Started
Well-Architected Pillars
The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.
The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.
-
Operational ExcellenceRead the Operational Excellence whitepaper
The AWS Cloud Development Kit (AWS CDK) and AWS CloudFormation stack output provide insights into the current deployment state. Amazon CloudWatch logs enable operational analysis and visibility into the workload. GitLab facilitates source code management, and the AWS CDK deploys changes to the AWS Cloud. Infrastructure can be re-provisioned through AWS CDK in case of failure and can be undeployed or redeployed as a unit using CloudFormation.
-
SecurityRead the Security whitepaper
AWS Identity and Access Management (IAM) roles and policies, Amazon Cognito-based authentication, and an API Gateway endpoint protect and manage resources. Amazon Cognito restricts access to a specific user pool, with access tokens expiring after one hour. The API Gateway endpoint provides encrypted web API access to specific paths and services, enforcing policies and API authorization. All AWS services use IAM for authentication and authorization, leveraging IAM roles for short-term credentials.
-
ReliabilityRead the Reliability whitepaper
The resilience of serverless services underpins the architecture. Lambda decouples low-level input processing from the Unreal Engine sample application's presentation layer. API Gateway throttles requests, queuing them in Amazon SQS. The Unreal Engine sample application retries failed AWS SDK operations, such as dequeuing from Amazon SQS. Standard CloudWatch logs and metrics store informational messages, error conditions, and enable notifications of errors.
-
Performance EfficiencyRead the Performance Efficiency whitepaper
Amazon CloudFront caches web page data at edge locations, scaling based on traffic. API Gateway throttling mechanisms (such as steady-state rate and burst request limits) manage API requests. For Amazon S3, edge locations (such as Transfer Acceleration) decrease latency. Adjusting provisioned throughput from Amazon Bedrock optimizes performance.
-
Cost OptimizationRead the Cost Optimization whitepaper
Serverless AWS services allow you to run code without provisioning or managing servers, paying only for specific on-demand usage. Costs scale according to the number of ASL translation requests, the amount of words translated into ASL in each request, and the quantity of sentence simplification operations. The serverless architecture avoids persistent AWS resources beyond S3 buckets when the overall solution is inactive. CloudFront's caching reduces requests to the web interface origin server hosted on Amazon S3. As ASL translation requests scale, Amazon SQS queues these requests as needed for later consumption.
-
SustainabilityRead the Sustainability whitepaper
Resource utilization scales on demand according to the number of ASL translation requests, the quantity of words translated into ASL in each request, and the amount of sentence simplification operations. The serverless architecture reduces the overall utilization and energy consumption footprint, eliminating the need to provision and manage servers or physical machines.
Related Content
[Title]
Disclaimer
The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.
References to third-party services or organizations in this Guidance do not imply an endorsement, sponsorship, or affiliation between Amazon or AWS and the third party. Guidance from AWS is a technical starting point, and you can customize your integration with third-party services when you deploy the architecture.