Amazon Reduces Infrastructure Costs on Visual Bin Inspection by a Projected 40% Using Amazon SageMaker

The VBI team builds autonomous counting systems that use computer vision, ML, and image-processing algorithms to analyze images across its network of FCs worldwide. Initially, the VBI team ran its own solution on Amazon Elastic Compute Cloud (Amazon EC2) C4 Instances. This solution did not support piloting new models—that is, it did not enable the team’s new models to handle requests alongside the old ML model nor did it allow the team to test it using real data and without risking service disruptions. Consequently, the VBI team had to develop ML models offline and validate and test them manually, which often took 3–6 months. The system also did not provide the capability to divert volume to new model launches. And inference—the process of making predictions using a trained model—was limited to CPU-based single input predictions, so the VBI team couldn’t reduce costs with graphics processing units (GPUs) or batch execution.

The VBI team was drawn to Amazon SageMaker because it solved the inefficiencies of the team's legacy software solution while reducing the amount of software and infrastructure that needed to be maintained. The fully managed service enabled the software development team to focus on core competencies instead of system management.

The VBI team developed the new solution on Amazon SageMaker over 12 weeks; during that time, the team containerized the inference code and built infrastructure for automating deployment. By comparison, the VBI team needed 8–12 months to develop its legacy solution. Then the VBI team conducted performance testing for another 2 months and completed its migration in April 2020 with no disruption to normal operations.

Due to the migration, Lalat Nayak, senior engineer for the VBI team at AFT, projects: "We will cut infrastructure costs by 40 percent with batch inference—just by using the features that come with Amazon SageMaker.” The team also gained a simplified system, trading in 1,000 CPU instances for one endpoint that automatically scales a fleet of fewer than 100 GPU instances. That transition to GPUs has cut the time-to-predict latency by 50 percent. The single Amazon SageMaker endpoint, which handles traffic from sensor towers across North America, Europe, and Japan, processes millions of bin images per day.

When AFT launches FCs in new regions, it now expects to deploy the Amazon SageMaker solution in 1–2 weeks, versus 1–2 months with its legacy solution. “Relying on Amazon SageMaker to host the model gives us the ability to decide whether we use the same model for all the warehouses or just some,” explains Nayak. “It helps us run experiments freely because we can release a feature for just a handful of warehouses and then expand to the rest after successful testing.”

Amazon SageMaker saves the software development engineer team about 1 month per year in maintaining infrastructure and software. It also enabled the VBI team to remove Amazon Simple Queue Service (Amazon SQS), a fully managed message-queuing service, and Amazon Simple Storage Service (Amazon S3), an object storage service, from its ML pipeline, which will save AFT 40 percent per month on AWS costs. Amazon SageMaker has scaled to handle demand spikes automatically, whereas the software development engineer team previously would have had to order more hardware, delaying implementation until its delivery.

Amazon SageMaker also reduced the load on the research wing of the VBI team, which can now launch models in pilot mode and roll out a new model in about 2 weeks, with no overhead needed to handle volume increases. And because Amazon SageMaker is ML-framework agnostic, the VBI team can layer whichever framework it wants on top of it. Converting from Caffe to Apache MXNet a few years ago took the VBI team 6–8 months, but Nayak anticipates a future conversion will be faster and simpler on Amazon SageMaker: “We want to be able to seamlessly transition to whatever technology is working best without investing a lot of time in integrating that technology into our framework. Amazon SageMaker does that for us.”

Using Amazon SageMaker, AFT’s VBI team reduced cost and the overhead on software developers while maximizing the efficiency of its solution. The VBI team expects that by the end of 2020, the Amazon SageMaker model will be used in all of its FCs and processing millions images in each center. With the ability to efficiently monitor its FCs to quickly find misplaced inventory, AFT can better fulfill Amazon’s promise that customers will receive their packages on time.