AWS Machine Learning Blog

Whooshkaa and Amazon Polly: Combining Eyes and Ears to Widen Publishing Horizons

This is a guest blog post by Robert Loewenthal, CEO & Founder of Whooshkaa. 

Based in Australia, Whooshkaa is a creative audio-on-demand podcast platform that helps publishers and advertisers reach their audiences. We’re always trying new products and techniques, and combining them to pioneer new solutions for our customers.

The Amazon Polly Text-To-Speech (TTS) feature is a great example of this. Already, we have top-tier publishers, sporting bodies, and Australia’s biggest telecommunications company using Amazon Polly to augment their established delivery methods.

Those traditional information-providers are finding that today’s customers don’t want to just read information, they want to listen to it. With Amazon Polly TTS, Whooshkaa gives information providers the ability to speak to their audiences – in any of 52 voices and 25 languages.

Earlier this year, Amazon Polly gave a voice to The Australian, our country’s premier national newspaper. Amazon Polly will read aloud newspaper stories, recipes, or sports scores to subscribers while they drive, exercise, or otherwise keep their hands and eyes busy.

Powered by Amazon Polly, Whooshkaa makes it easy for selected partners to pick any news story and convert it into a podcast episode in seconds. We also provide the tools to merge multiple stories and to customize the sound by changing its accent, pitch, rate, and volume.

Whooshkaa has an extensive distribution network, which means that listeners can choose from a wide range of possibilities to consume content. The most obvious choice is through their favorite podcasts app. However, because Whooshkaa has a unique partnership with Facebook, our podcast episodes can be played through their native audio player. Our customizable web player is also supported on Twitter, but it can be embedded on any website.

We believe that when the technology is ready, publishers will be able to make their news stories available in any language, in every part of the world. News stories could be customized to their listeners’ preferences and needs.

We are also working with Australia’s largest telecommunications company, Telstra, and the National Rugby League, to deliver live sport results of a user’s favorite team through any connected smart speaker. Our users can simply ask their device for the current score, and they’ll get it read back to them instantly.

Our developer Christian Carlsson believes the immediacy of Amazon Polly TTS and the range of languages bring limitless opportunities to any type of publisher.

“By integrating artificial intelligence with Whooshkaa’s already-powerful platform it’s now possible to create a fully automated podcast episode from text in less than 30 seconds – and this is just the beginning,” Carlsson says.

Technical implementation of the AFL integration

The Australian Football League (AFL) wanted their fans to be able to follow their favorite team through voice commands to a smart speaker. To do so Whooshkaa needed to create an RSS feed that got updated every 2 minutes with the latest results. The following diagram shows  a simplified overview of our implementation.

To trigger a crawl of AFL’s API that contained the data we needed, we set up a simple AWS Lambda function that would call our API. The Whooshkaa API would fetch the data, parse it, convert the text to speech, and publish the newly created RSS feed to Amazon S3.

First, we got the serverless.yml file that is responsible for initializing the requests every 2 minutes. Nothing fancy here.

Serverless.yml:
createAFLFeeds:
 handler: api.createAFLFeeds
 events:
   - schedule:
       rate: rate(2 minutes)
       enabled: ${self:custom.${opt:stage}.ScheduledEvents}

This triggers the following code:

WhooshkaaAPI.js
createAFLFeeds() {
    return new Promise((resolve, reject) => {
      this.fetchAFLTeams().then(result => {
        for (const team of result) {
          this.createAFLFeedByTeamID(team['id']);
        }
      }, error => {
        console.log(error);
        reject(error);
      });
      resolve({message: "success"});
    });
}

Next, the createAFLFeedByTeamID method sends a POST request to our endpoint which does the following:

  1. Fetches the data from the AFL API.The data normalization is abstracted to a separate AFL package to make this method as readable as possible. A few different conditions determine what data to parse. A team’s match data is fetched if the team is playing or has played in the last 24 hours, otherwise we default to the latest news about the team.
  2. Makes sure that the returned data is new by storing its hash in Amazon S3.$this->publisher is once again an abstracted class that contains three different storage adapters: local, Whooshkaa S3 bucket, and AFL S3 bucket. When working with the data we use the local adapter, we store the hash in the Whooshkaa S3 bucket, and we publish the generated RSS feed to the AFL S3 bucket.
  3. Takes the text and converts it to an audio stream through Amazon Polly.You can see in the makeAudio method how we manipulate some of the words so they sound the way we expect them to. For example, MCG, which is a sports stadium, was interpreted as ‘McGee,’ so we tell Amazon Polly to spell it out instead.
  4. Creates the RSS feed and publishes it to AFL’s S3 bucket.
AFLController.php:
public function team(string $id)
{
    if (!$team = Team::findById($id)) {
        $this->response->errorNotFound('Invalid team ID.');
    }

    if ($team->isPlayingOrHasRecentlyPlayed()) {
        $story = $team->match;
    } else {
        $story = $team->news;
    }

    $this->publisher->setTeamId($id);
    $this->publisher->setStory($story->getStory());

    $hash = Hash::make($story, $this->publisher->getRemoteStorageAdapter());
    if ($hasBeenUpdated = $hash->hasBeenUpdated()) {
        $fileName = $this->publisher->getFileName();

        $audio = $this->makeAudio($story);
        $this->publisher->store($fileName, $audio->getContent());

        $feed = $this->makeFeed($team, $story);
        $this->publisher->store('feed.xml', $feed->getContent());

        $this->publisher->moveToCloud([$fileName, 'feed.xml']);
        $this->publisher->cleanUp();

        $hash->store();
    }

    return response([
        'rss' => $this->publisher->getRemoteUrl('feed.xml'),
        'updated' => $hasBeenUpdated,
    ]);
}

private function makeAudio($story)
{
   $polly = new Polly;
   $polly->setPhonemes(['live' => 'laɪve']);
   $polly->setProsody('AFL', ['rate' => 'fast']);
   $polly->setSayAs(['MCG' => 'spell-out']);

   $text = $story->getStory();
   // Trim the text to a maximum of 1500 characters.
   if (strlen($text) > 1499) {
       $text = $this->text->setText($text)->trimToWordBoundary(1499);
   }

   try {
       $audioStream = $polly->fetchAudioStream($text);
   }
   catch (\Exception $e) {
       $this->response->error($e->getMessage(), $e->getStatusCode());
   }

   return response()->make($audioStream)->header('Content-Type', 'audio/mpeg');
}

private function makeFeed(Team $team, $story)
{
   $feed = new Feed($this->publisher->getRemoteURL('feed.xml'));
   $feed->setTitle($team->getName() . "'s Official Live Feed");
   $feed->setDescription('An official live feed from the Australian Football League.');
   $feed->setLink('http://www.afl.com.au');
   $feed->setOwner('The Australian Football League', 'podcast@afl.com.au');
   $feed->setImage($team->getImage());
   $feed->appendElements([
       'itunes:subtitle' => "Follow {$team->getName()}'s Live Matches and Latest News",
       'itunes:explicit' => 'no',
       'language' => 'en-us',
       'lastBuildDate' => Carbon::now('UTC')->toRssString(),
       'ttl' => 2,
       'copyright' => 'The Australian Football League',
   ]);

   $feed->setCategories([
       'Sports & Recreation' => [
           'Professional',
       ]
   ]);

   $fileName = $this->publisher->getFileName();
   $metaData = $this->getMetaData($fileName);

   $item = $feed->addItem([
       'title' => $story->getTitle(),
       'link' => $story->getArticleURL(),
       'pubDate' => Carbon::now('UTC')->toRssString(),
       'itunes:duration' => $metaData['playtime_string'],
   ]);
   $item->appendDescription($story->getStory());
   $item->appendEnclosure($this->publisher->getRemoteUrl($fileName, true), $metaData['filesize'], $metaData['mime_type']);
   $item->append('itunes:image', null, ['href' => $team->getImage()]);
   $item->append('guid', $this->publisher->getGuid(), ['isPermaLink' => 'false']);

   return response()->make($feed->output())->header('Content-Type', 'text/xml');
}

Technical implementation of the Australian’s ‘Daily News’

The Australian is a newspaper publisher under the News Corp umbrella. They wanted their top 10 news headlines of the day available to their listeners in audio. Since their requirements were that the headlines should be updated five times a day as a podcast episode, our integration with Amazon Polly made it fairly easy to implement. The following diagram provides a simplified overview of our implementation.

 

This implementation has striking similarities with the AFL integration, but with one exception. Instead of generating an RSS feed, we instead publish the episode to a specified show on the Australian’s account on Whooshkaa. This makes the episode almost immediately available on iTunes, Pocket Casts, or any other podcast player.

To build this implementation, we set up a AWS Lambda function, as we did for the AFL implementation because we need to trigger our ‘Daily News’ endpoint at specific times throughout the day.

Serverless.yml
createDailyNewsStory:
 handler: api.createDailyNewsStory
 events:
   - schedule:
       rate: cron(0 2,6,10,22 * * ? *)
       enabled: ${self:custom.${opt:stage}.ScheduledEvents}
   - schedule:
       rate: cron(30 14 * * ? *)
       enabled: ${self:custom.${opt:stage}.ScheduledEvents}
WhooshkaaAPI.js
createDailyNewsStory() {
 const options = {
   hostname: this.commonOptions.hostname,
   port: this.commonOptions.port,
   path: '/news-corp/daily-news',
   method: 'POST',
 };
 return new Promise((resolve, reject) => {
   this.sendRequest(options).then(result => {
     return resolve(result);
   }, error => {
     console.log(error);
     return reject('Could not create "Daily News" story.');
   });
 });
}

Next, the createDailyNewsStory handler calls the createDailyNewsStory function which triggers the dailyNews endpoint on our API, as follows.

NewsCorpController.php
public function dailyNews()
{
   $show = Show::find(DailyNewsStory::SHOW_ID);
   $storyBuilder = new StoryBuilder($show);

   $dateTime = Carbon::now('Australia/Sydney')->format('F j (g:00 a)');
   $title = $show->title . ' - ' . $dateTime;

   $story = new DailyNewsStory;
   $story->setLimit(10);
   $story->setTitle($title);
   $story->setDescription($title);

   $episode = $storyBuilder->fetch($story)->publish();

   return $this->response->item($episode, new EpisodesTransformer);
}

The DailyNewsStory extends a StoryBase class that in turn has a dependency injection of a NewsCorpApi class. The values from DailyNewsStory are passed through to the NewsCorpApi class where they fetch and normalize the data.

Next, we’ll generate the audio for all of the stories we have fetched and publish it as a single episode. This is done in the StoryBuilder class, as follows..

StoryBuilder.php
public function publish()
{
   $title = $this->story->getTitle();
   $description = $this->story->getDescription();

   if (!$episode = $this->episodes->findByTitleAndDescription($title, $description)) {
       $audio = $this->makeAudio();
       $fileName = $this->storage->putContent($audio->content(), Polly::OUTPUT_FORMAT);

       $data = [
           'podcast_id' => $this->show->id,
           'title' => $title,
           'description' => $description,
           'media_file' => $fileName,
           'length' => $this->storage->getSize($fileName),
       ];

       $episode = $this->episodes->create($data);
   }

   return $episode;
}

public function makeAudio()
{
   $polly = new Polly;

   $audioStream = null;
   foreach ($this->story->getBody() as $body) {
       $audioStream .= $polly->makeAudioStream($body);
   }

   return $polly->makeAudioResponse($audioStream);
}

We loop through $this->story->getBody() because it’s an array containing all ten stories previously mentioned. This creates a continuous audio stream from Amazon Polly. The audio stream is then uploaded as an mp3 file to our S3 bucket, and the filename, with the rest of the information, is saved to the database and returned in the request.

Many of our customers generate significant amounts of rich content. We give them a platform, powered by Amazon Polly, to convert their content to audio and then distribute, analyze, and commercialize. One news publisher plans to make its recipe library available through Whooshkaa and Amazon Polly text-to-voice.

Whooshkaa  is always looking for ways to innovate with audio. We seek new markets and technology to give our creators the widest distribution network possible. We’ve found that traditional publishers and Amazon Polly are a winning combination.

About the Author

Robert Loewenthal is the CEO & Founder of Whooshkaa. Based in Sydney Australia, Whooshkaa is a full service, audio on-demand company that help creators and brands produce, host, share, track and monetize content.