JibJab launches machine learning-powered product line with help from Mission and AWS

JibJab, a provider of personalized eCards and other digital entertainment products, needed a managed cloud services partner with deep Amazon Web Services (AWS) and machine learning (ML) expertise to help build out a novel head cutting technique for its new product, Starring You Books. JibJab required an ML algorithm that could quickly and automatically crop a user’s face and hair from an uploaded image, and then produce print-quality images that customers can place within stories they create. Mission successfully built the ML solution that JibJab required to launch the product – which prepares customized images from customer uploads in an average of five seconds and with 90% accuracy.

To support its new Starring You Books product line, JibJab sought a technology solution for improving both the customer experience and the quality of the images going into these printed book products. Traditionally, JibJab enabled users to place faces of themselves or others into eCards and other media by uploading photos, and then using a simple interface featuring a “peanut shape” cropping tool. The result was a cropped oval face placed onto characters within selected products (making yourself into a dancing elf, for example). 

With Starring You Books, users would be able to create and fully customize multi-page books featuring face images, which JibJab then prints and delivers as a physical product. However, shifting to a printed book offering called for higher-quality images and a new design for how faces are displayed – moving beyond the “peanut” to also include a person’s full face and hair (while ignoring everything else in the uploaded image). JibJab also wanted to offer customers a much faster and less manual experience around image preparation. 

JibJab recognized the potential to leverage a ML computer vision algorithm able to detect faces within uploaded photos, automatically crop both face and hair from the image, and perform image post-processing to arrive at a print-quality result. However, JibJab lacked the in-house expertise to complete this ML project. Exploring the marketplace, JibJab also realized that it didn’t want to pay the high licensing fees associated with utilizing an existing ML algorithm. This left the company seeking a cloud services provider with deep expertise in designing, building, and training ML algorithms and the project-delivery and support experience to help JibJab achieve its project goals.

JibJab had worked with Mission Cloud Services on a previous AWS infrastructure project and trusted Mission as a highly-capable and hands-on partner. “We brought in Mission before, and that partnership went really well,” said Cielecki. “Our VP of Technical Operations, who had led that engagement from our side, recommended Mission when the need came up again to find an AWS Partner.” 

As it now scoped out this new initiative, JibJab’s legal department advised that the company would need its own ML algorithm created completely from scratch to avoid any potential licensing issues, increasing the pressure to select a reliable partner able to build ML solutions from the ground up. 

In talking with Mission about its project needs and the unique capabilities of Mission’s data, analytics, and machine learning (DAML) practice, Mission impressed JibJab by detailing not just how to achieve the requisite ML solution, but the specific reasons behind what was being done. “It became evident from day one that Mission wasn’t just going to throw something over the fence for us to use; the team was going to ensure that we understood the rationale behind the processes and technologies put into action,” said Cielecki. Mission gave JibJab full confidence that the project would be successful, and that the ultimate handoff to the JibJab team would go smoothly.

Mission began training the ML algorithm to automatically recognize faces and hair in user-uploaded photos. First, Mission performed image data labeling by leveraging LabelMe and Amazon SageMaker Ground Truth (achieving 70% time-savings with its AI-assisted automatic segmentation tool) to create a training data set and begin data preparation. With that labeled data set of 1000 images, Mission then performed data augmentation – adding blur, sharpening, rotation, noise and other alterations – to expand the data set to 17,000 images. Using that robust data set, Mission leveraged Detectron2 running in Amazon SageMaker to detect objects within images and perform instance segmentation to identify faces and hair. 

From a workflow perspective, the solution’s process begins when a customer uploads a new image, which lands in an Amazon S3 bucket. Amazon Rekognition a powered face detection then looks at the image. If it detects a single face, it moves to the next step; if there are multiple faces, the solution asks the user to crop the image down to a single face. The solution then performs an image quality pre-check, assessing whether the image shows the front of the face, if it is blurry, if anything blocks the face, and if the image size supports a quality result. This step is crucial to the product success, since enforcing high image quality reduces book returns and increases customer satisfaction. The image then goes to a Detectron2 model running as an endpoint in SageMaker, which performs instance segmentation to find the face and hair to use in the final image. The next step is image post-processing to remove the image background and blur the edges to improve the final appearance. Finally, the solution extracts facial landmarks that JibJab uses to place the head into the product. The customer can then position the final image within the book they are creating using a GUI and approve it for printing.

With the ML-fueled product, JibJab customers can now upload photos and have the algorithm complete its work and provide a final book-ready image in just five seconds. While Mission’s early discussions with JibJab set a goal for the algorithm to deliver high-quality images with 85% accuracy, in practice the algorithm is achieving 90% accuracy. 

If customers were asked to prepare images themselves instead of the algorithm, that image preparation work would take at least a minute, and without the same guarantee of a high-quality result. Now, customers have a seamless self-service method for creating their entire Starring You Books in just minutes. The ML algorithm enables a much better experience for the customer and creates a much better product. Mission is continuing to improve on the ML algorithm powering JibJab’s solution, with the targeted goal of increasing its accuracy to 95%.