Every day millions of users around the world communicate with each other through email, social networks, and other online communities including chat platforms and message boards. Many times, users find a community interesting and want to participate in the chat conversation, but the primary language of the online community might be different than the language that user understands.
One of those communities is Twitch, the world’s leading social video service for gamers. Each day, millions of community members gather to watch, talk, and chat about shared interests. Chat is built into every stream, so instead of passively watching the stream, you can be part of the show. You will find many streamers on Twitch who don’t necessarily speak your language, but you might still enjoy their streams and would want to participate in the chat.
To make this communication possible, AWS provides Amazon Translate, a neural machine translation service that delivers fast, high-quality, and affordable language translation. Amazon Translate can provide on-demand translation to enable cross-lingual communication between users. By adding real-time translation to chat, email, helpdesk, and ticketing applications, an English-speaking agent or employee can communicate with customers across multiple languages.
This blog post covers how you can use Amazon Translate to enable cross-lingual chat for a Twitch channel. You will also learn how to use Amazon Polly, a Text-to-Speech service, to help voice incoming real-time messages. This additional functionality provides further access and offer users the option to listen to the messages while concentrating on other tasks at hand.
Solution overview
In this post, you will create an HTML page with CSS and JavaScript. You will use a JavaScript library to connect to a Twitch channel and start receiving messages. As those real-time messages come in, you will use the AWS SDK to call Amazon Translate and get those messages translated and displayed in the UI. If you choose to use the user-enabled voice option, you’ll use AWS SDK to call Amazon Polly and get synthesized speech for those messages and play them back to the users.
Create and Configure a Twitch account
- Go to twitch.tv to sign up for a Twitch account.
- After successfully signing up for your Twitch account, go to https://twitchapps.com/tmi to create a Twitch OAuth token.
- Click on the button “Connect with Twitch” and note the OAuth token.
Create an IAM user
AWS Identity and Access Management (IAM) enables you to manage access to AWS services and resources securely. Using IAM, you can create and manage AWS users and groups, and use permissions to allow and deny their access to AWS resources. To get started, create an IAM user with the minimum required permissions to run this example.
- Go to https://console.aws.amazon.com/iam/home?region=us-east-1#/users.
- Choose Add user.
- Enter the user name, check Programmatic access, and choose Next Permissions.
- Choose Attach existing policies directly, and search for translate.
- Select the check box next to TranslateReadOnly and choose Next Review.
- Choose Create user and note the access key ID and secret access key.
Create and run the application
- Create an HTML page and copy the code from the following code block.
<!doctype html>
<html lang="en">
<head>
<title>Amazon Translate</title>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
<style>
.topHeader
{
background-color: #6441a4;
padding: 10px;
border-bottom: solid 1px #cacaca;
color: white
}
.panelHeading
{
background-color: #6441a4 !important;
}
.panelBody
{
min-height: 450px; max-height: 450px;overflow-y: scroll;
}
body{
margin-left: 0px;
margin-right: 0px;
height: 100%;
}
</style>
</head>
<body>
<div class="container-fluid">
<div class="row topHeader">
<div class="col-md-12">
<h4>Amazon Translate - Artificial Intelligence on AWS - Powerful machine learning for all Developers and Data Scientists</h4>
</div>
</div>
<div class="row">
<div class="col-md-12">
<p class="bg-info">
<div id="connecting-div"></div>
</p>
</div>
</div>
<div class="row" style="padding: 10px;">
<div class="col-md-6">
<div class="form-inline">
<div class="form-group">
<input type="text" id="channel" class="form-control" value="" placeholder="Channel"/>
</div>
<div class="form-group">
<select id="sourceLanguage" class="form-control">
<option value="en">en</option>
<option value="ar">ar</option>
<option value="de" selected="selected">de</option>
<option value="es">es</option>
<option value="fr">fr</option>
<option value="pt">pt</option>
<option value="zh">zh</option>
</select>
</div>
<div class="form-group">
<select id="targetLanguage" class="form-control">
<option value="en" selected="selected">en</option>
<option value="ar">ar</option>
<option value="de">de</option>
<option value="es">es</option>
<option value="fr">fr</option>
<option value="pt">pt</option>
<option value="zh">zh</option>
</select>
</div>
<div class="form-group">
<button type="button" class="form-control" id="btn-go" onclick="connect()">Go</button>
<button type="button" class="form-control" id="btn-stop" onclick="location.href='index.html';">Stop</button>
<span id="status"></span>
</div>
</div>
</div>
<div class="col-md-6">
<div class="form-inline">
<div class="form-group">
<input type="checkbox" id="cbSpeak" value="Speak"> Speak Live Translation
<input type="text" id="follow" class="form-control" value="" placeholder="follow"/>
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-6">
<div class="panel panel-primary">
<div class="panel-heading panelHeading">Live Chat</div>
<div id="livechatc" class="panel-body panelBody">
<div class="subscribe" id="livechat"></div>
</div>
</div>
</div>
<div class="col-md-6">
<div class="panel panel-primary">
<div class="panel-heading panelHeading">Live Translation</div>
<div id="livetranslationc" class="panel-body panelBody">
<div class="imageDetected" id="livetranslation"></div>
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-11">
<input type="text" id="message" class="form-control"/>
</div>
<div class=" col-md-1">
<button type="button" class="form-control btn btn-default" id="btn-send" onclick="sendMessage()">Send</button>
</div>
</div>
</div>
<script src="https://code.jquery.com/jquery-3.2.1.slim.min.js" integrity="sha384-KJ3o2DKtIkvYIK3UENzmM7KCkRr/rE9/Qpg6aAZGJwFDMVNA/GpGFF93hXpG5KkN" crossorigin="anonymous"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
<script src="aws-js-sdk/dist/aws-sdk-all.js"></script>
<script src="http://cdn.tmijs.org/js/1.2.1/tmi.min.js" integrity="sha384-eE0n7sm1W7DOUI2Xh5I4qSpZTe6hupAO0ovLfqEy0yVJtGRBNfssdmjbJhEYm6Bw" crossorigin="anonymous"></script>
<script>
cred = {
twitchUsername: "Twitch user name",
twitchOAuthToken: "Twitch OAuth token",
awsAccessKeyId: "access key",
awsSecretAccessKey: "secret key"
};
AWS.config.region = 'region';
ep = new AWS.Endpoint('endpoint');
AWS.config.credentials = new AWS.Credentials(cred.awsAccessKeyId, cred.awsSecretAccessKey);
window.translator = new AWS.Translate({endpoint: ep, region: AWS.config.region});
function connect(){
init();
var options = {
options: {
debug: false
},
connection: {
cluster: "aws",
reconnect: true
},
identity: {
username: cred.twitchUsername,
password: cred.twitchOAuthToken
},
channels: [con.channel]
};
window.client = tmi.client(options);
window.client.connect();
window.client.on("chat", onChat);
window.client.on("connecting", onConnecting);
window.client.on("connected", onConnected);
document.getElementById("sourceLanguage").disabled = true;
document.getElementById("targetLanguage").disabled = true;
document.getElementById("channel").disabled = true;
document.getElementById("btn-go").disabled = true;
}
function init(){
var lc = document.getElementById("livechat");
var lt = document.getElementById("livetranslation")
var lcc = document.getElementById("livechatc");
var ltc = document.getElementById("livetranslationc")
var cbspeak = document.getElementById("cbSpeak")
var follow = document.getElementById("follow");
var sendMessage = document.getElementById("message");
con = {
channel: document.getElementById("channel").value,
sourceLanguage: document.getElementById("sourceLanguage").value,
targetLanguage: document.getElementById("targetLanguage").value,
liveChatUI: lc,
liveTranslationUI: lt,
liveChatUIContainer: lcc,
liveTranslationUIContainer: ltc,
cbSpeak: cbspeak,
follow: follow,
sendMessage: sendMessage
}
lc.innerHTML = '';
lt.innerHTML = '';
var voiceId = "Joanna";
if(con.targetLanguage == "en")
voiceId = "Joanna";
else if(con.targetLanguage == "de")
voiceId = "Marlene";
else if(con.targetLanguage == "es")
voiceId = "Conchita";
else if(con.targetLanguage == "fr")
voiceId = "Celine";
else if(con.targetLanguage == "pt")
voiceId = "Ines";
else
voiceId = "Joanna";
window.audioPlayer = AudioPlayer(voiceId);
}
function onChat (channel, userstate, message, self) {
if (self) return;
if (message) {
var username = userstate['username'];
var params = {
Text: message,
SourceLanguageCode: con.sourceLanguage,
TargetLanguageCode: con.targetLanguage
};
window.translator.translateText(params, function onIncomingMessageTranslate(err, data) {
if (err) {
console.log("Error calling Translate. " + err.message + err.stack);
}
if (data) {
console.log("M: " + message);
console.log("T: " + data.TranslatedText);
con.liveChatUI.innerHTML += '<strong>' + username + '</strong>: ' + message + '<br>';
con.liveTranslationUI.innerHTML += '<strong>' + username + '</strong>: ' + data.TranslatedText + '<br>';
if(con.cbSpeak.checked){
if(con.follow.value == "" || username == con.follow.value)
audioPlayer.Speak(username + " says " + data.TranslatedText);
}
con.liveChatUIContainer.scrollTop = con.liveChatUIContainer.scrollHeight;
con.liveTranslationUIContainer.scrollTop = con.liveTranslationUIContainer.scrollHeight;
}
});
}
}
function onConnecting (address, port) {
document.getElementById("status").innerHTML = " [ Connecting...]"
}
function onConnected (address, port) {
document.getElementById("status").innerHTML = " [ Connected ]"
window.audioPlayer.Speak("Connected to channel " + con.channel + ". You should now be getting live chat messages.");
}
function sendMessage(){
if(con.sendMessage.value){
message = con.sendMessage.value;
var params = {
Text: con.sendMessage.value,
SourceLanguageCode: con.targetLanguage,
TargetLanguageCode: con.sourceLanguage
};
window.translator.translateText(params, function onSendMessageTranslate(err, data) {
if (err) {
console.log("Error calling Translate. " + err.message + err.stack);
}
if (data) {
console.log("M: " + message);
console.log("T: " + data.TranslatedText);
window.client.action(con.channel, data.TranslatedText);
con.sendMessage.value = "";
con.liveTranslationUI.innerHTML += '<strong> ME: </strong>: ' + message + '<br>';
con.liveChatUI.innerHTML += '<strong> ME: </strong>: ' + data.TranslatedText + '<br>';
con.liveChatUIContainer.scrollTop = con.liveChatUIContainer.scrollHeight;
con.liveTranslationUIContainer.scrollTop = con.liveTranslationUIContainer.scrollHeight;
}
});
}
}
function AudioPlayer(voiceId) {
var audioPlayer = document.createElement('audio');
audioPlayer.setAttribute("id", "audioPlayer");
document.body.appendChild(audioPlayer);
var isSpeaking = false;
var speaker = {
self: this,
playlist:[],
Speak: function (text) {
if (isSpeaking) {
this.playlist.push(text);
} else {
speakTextMessage(text).then(speakNextTextMessage)
}
}
}
function speakTextMessage(text) {
return new Promise(function (resolve, reject) {
isSpeaking = true;
getAudioStream(text).then(playAudioStream).then(resolve);
});
}
function speakNextTextMessage() {
var pl = speaker.playlist;
if (pl.length > 0) {
var txt = pl[0];
pl.splice(0, 1);
speakTextMessage(txt).then(speakNextTextMessage);
}
}
function getAudioStream(textMessage) {
return new Promise(function (resolve, reject) {
var polly = new AWS.Polly();
var params = {
OutputFormat: 'mp3',
Text: textMessage,
VoiceId: voiceId
}
polly.synthesizeSpeech(params, function (err, data) {
if (err)
reject(err);
else
resolve(data.AudioStream);
});
});
}
function playAudioStream(audioStream) {
return new Promise(function (resolve, reject) {
var uInt8Array = new Uint8Array(audioStream);
var arrayBuffer = uInt8Array.buffer;
var blob = new Blob([arrayBuffer]);
var url = URL.createObjectURL(blob);
audioPlayer.src = url;
audioPlayer.addEventListener("ended", function () {
isSpeaking = false;
resolve();
});
audioPlayer.play();
});
}
return speaker;
}
</script>
</body>
</html>
- Update the following code snippet with path to the AWS JavaScript SDK.
- Update the following code snippet with the AWS region and endpoint.
- Update the following code snippet with your Twitch username, OAuth token, AWS access key ID and AWS secret access key.
- Open the web page in a browser.
- Enter the name of a Twitch channel.
- Select source and destination language and click Go.
You should now see real-time messages along with their translation on the screen. If you compare Twitch chat with this example application, you will see messages appear in the chat simultaneously as they appear in the Twitch chat showing the performance and very low latency from Amazon Translate.
You can also view the video demonstration showing the real-time translation of Twitch chat at re:Invent 2017.
Receive and Translate Messages
The following code block shows how you receive real-time incoming messages and then use AWS SDK to call Amazon Translate to show the translated message in the UI. As you see most of the code that follows is about input and UI handling, while the code to translate the messages is only few lines.
Send translated messages
The following code block shows how you translate an outgoing message and then send it the Twitch channel.
Summary
In this post you learned how you can use Amazon Translate for real-time translation of chat messages. This blog used a Twitch channel as an example, but you can use it as a starting point for other real-time streaming text like other chat platforms, customer service interactions, message boards and more.
About the Authors
Kashif Imran is a Solutions Architect at Amazon Web Services. He works with some of the largest strategic AWS customers to provide technical guidance and design advice. His expertise spans application architecture, serverless, containers, NoSQL and machine learning.