Enhancing live selling experiences using Amazon Interactive Video Service

This blog was co-authored by Hector Zelaya, WebRTC.Ventures WebRTC Developer Advocate.

Let your customers experience the future of shopping with live selling: the ultimate blend of online convenience and live video excitement. No crowds, no endless searching—just dynamic, engaging shopping online. In the same vein, allow sellers to add images and QR codes in real time to their live stream and turn viewers into happy buyers more easily.

In this blog post, we describe how you can enhance your own live selling application—powered by Amazon Interactive Video Service (Amazon IVS)—by adding the ability to embed images and QR codes right into your video streams in real time by leveraging the Insertable Streams for MediaStreamTrack API.

Understanding the solution

The solution presented here is a web application written in Typescript using the Next.js framework. It has two main pages: one for sellers and another for viewers. Both pages feature the video stream and the chat. The seller’s view has additional options for adding images and QR codes to the stream.

The live selling application in action. A seller uses an image to support a sales pitch.

The solution uses the Amazon IVS Broadcast SDK to send seller’s media streams to Amazon IVS, and the Amazon IVS Player SDK to consume them. Additionally, a websocket connection is made to an Amazon IVS chat room to enable chat functionality as shown in the following graphic.

Depiction of how the application works.

1. Sellers (publishers) access the seller page, which uses the Amazon IVS Broadcast SDK to broadcast media streams to a previously created channel.
2. Buyers (subscribers) access the buyer page, which uses the Amazon IVS Player SDK to consume the stream from the channel.
3. Both pages include a chat component that maintains a websocket connection to a chat room using Amazon IVS Chat.

For simplicity, all the required credentials feed into the application as public environment variables. This is not recommended for production applications. Instead, it’s recommended to set up a backend service with proper authentication and authorization mechanisms to retrieve those values in runtime.

We highlight the most important parts in the following example. If you’re interested in learning how to use Amazon IVS to build the live streaming application, visit the WebRTC.ventures blog post where we cover integrating live streaming experiences using Amazon IVS, and which includes a link to the GitHub repository.

Adding QR codes and images using insertable streams

By leveraging Amazon IVS live streaming capabilities, you can support millions of potential customers in your live selling events with low latency. These experiences can be taken to the next level using the power of the Insertable Streams API.

By providing the ability to process video streams in real time, the live streaming experience becomes more engaging and sellers have the tools to convert viewers into buyers with only a few clicks, or as in this example: with just one QR scan!

First, we need to convert the video stream into something we can read from and write to. To do this, we leverage MediaStreamTrackProcessor and MediaStreamTrackGenerator interfaces. The first one gives us access to the video stream track. The latter allows us to create a processed video track.

// get a readable object from the track
const trackProcessor = new MediaStreamTrackProcessor({ track })

// create a writable object for the new track
//   set the same 'kind' as the original track
const trackGenerator = new MediaStreamTrackGenerator({ kind: track.kind })

// create the TransformStream instance passing the transform function
const transformer = new TransformStream({ transform })

Depiction of a video processing pipeline.

We do this by getting trackProcessor’s readable attribute piped through the transformer, and then pipe the result to trackGenerator’s writable attribute.

// bundle everything together
trackProcessor.readable
  .pipeThrough(transformer)
  .pipeTo(trackGenerator.writable)

With this in place, it’s now time to create a transform function. As we mentioned in a previous post from WebRTC.ventures, a nice approach is to use a transform function factory that generates custom functions to transform the streams.

Let’s use an example of a transform function factory for generating QR codes. The qrcode library is used for QR code generation and OffscreenCanvas to render everything.

The transform function receives one video frame at a time and a controller object for enqueueing each frame of the final track. Transformation consists of setting up the canvas, drawing the original frame on it, and adding the QR code on top of it. Next, we enqueue this new frame into the controller and we properly dispose of the original frame.

// transform function factory for functions that render QR codes
const showQr = (
  Params: QRParamsType
) => {
  // setup OffscreenCanvas
  const canvas = new OffscreenCanvas(1, 1)
  const ctx = canvas.getContext('2d')

  // setup QR code
  const qrCanvas = document.createElement('canvas')
  QRCode.toCanvas(qrCanvas, Params.text)

  // the actual transform function
  //   It receives one frame at a time and a controller
  return function transform(frame, controller) {
    // adjust the canvas to the frame
    const width = frame.displayWidth
    const height = frame.displayHeight
    canvas.width = width
    canvas.height = height

    // let's make Typescript happy
    //   making sure we have a context
    if (ctx) {
      // clear any previous configuration
      ctx.clearRect(0, 0, width, height)
      // render the original frame
      ctx.drawImage(frame, 0, 0, width, height)
      // render the QR code on top of it
      ctx.drawImage(qrCanvas, params.x, params.y, params.width, params.height)
    }

    // get the timestamp of the current frame before closing it
    const { timestamp } = frame
    frame.close()

    // create a new frame and add it to the queue
    const newFrame = new VideoFrame(canvas, { timestamp })
    controller.enqueue(newFrame)
  }
}

const cleanStream = () => {
  return function transform(frame, controller) {
    controller.enqueue(frame)
  }
}

// get the video stream using getUserMedia
const videoStream = await navigator.mediaDevices.getUserMedia({
  video: {
    // get video from a specific device
    deviceId: { exact: params.device.deviceId }
    // the rest of constraints
    ...
  }
})

// get a processed track from the pipeline created before
const pTrack = createProcessedTrack({
  track: videoStream.getVideoTracks()[0],
  transform
})

// create a new stream using the processed track
const newVideoStream = new MediaStream([pTrack])

// add the stream to IVS by specifying the stream, a layer name and 
//   the rest of params
client.addVideoInputDevice(
  newVideoStream,
  params.name,
  params
)

// start with a clean track
const selectedTransform = useRef(cleanStream())

// build the transform function based on the reference
const transformFn = (frame, controller) => 
  selectedTrasnform.current(frame, controller)

// pass transformFn to your media initialization function
yourAddVideoFn(...otherParams, transformFn)

Switching to a different transformation is a matter of updating the reference. For example, here is how to show a QR code:

seletectedTransform.current = showQr()

A demo application from this example is shown in the following image.

Conversion almost complete! The seller shows a QR code that viewers can scan to complete the purchase.

Conclusion

Amazon IVS along with the Insertable Streams for MediaStreamTrack API allows us to shape and enhance live selling experiences by adding the ability to manipulate video streams and elements such as QR codes and images to aid sellers in converting customers.

About WebRTC.ventures

If you’re looking into building your own live selling application with Amazon IVS that takes advantage of the power of Insertable Streams, contact  WebRTC.ventures. They are expert integrators of real-time capabilities and an AWS Partner. You can find WebRTC.ventures on the Amazon Marketplace or at www.webrtc.ventures.