Proactively detect and prevent manufacturing defects with SAP on AWS

This post was written by Ganesh Suryanarayanan, Krishnakumar Ramadoss, Joseph Rosing and Manoj Muthukrishnan.

Introduction
Machine learning increasingly enables a lower total cost of quality by detecting defects faster (in some cases predicting them) and augmenting traditional six sigma business process improvements with scalable, low-cost solutions. However, like many industrial machine learning use cases, the machine learning insights are only valuable to the extent that someone acts on them. As a result, it is key to integrate insights into existing quality management workflows or business processes. This blog details how users can automate this integration by applying computer vision-based defect detection with Amazon Lookout for Vision and sending the defect notifications to SAP Quality Management (QM) in order to improve first pass yield(FPY), Rolled throughput yield(RTY), and increased customer satisfaction. The usage of computer vision for quality provides the ability for an increased sample size for inspection and this has a direct correlation with yield and defect reduction.

Amazon Lookout for Vision
Customers such as Dafgard and General Electric have used Amazon Lookout for Vision to automate and scale computer vision-based inspections. The managed service uses computer vision to identify misplaced or missing components in electronic products, surface defects or damage to metal structures, irregularities in production lines, and even minuscule defects in silicon wafers. As a result, customers can eliminate more costly, fixed visioning systems or inconsistent manual inspection points with lower- cost cameras and machine learning to catch defects at the source, improving quality control. More details around benefits and pricing can be found here.

SAP Quality Management (QM)
Within SAP, the QM application supports tasks associated with quality planning, quality inspection, and quality control. In addition, it controls the creation of quality certificates and manages problems with the help of corrective action plans.

SAP QM is an integral part of several key business processes including procurement, production, and sales. For example, it can be used to validate quality compliance of raw materials when they’re first delivered by vendors and as well as logging quality records for materials during production and postproduction. Finally, QM is also used to ensure compliance with customers’ quality specifications before finished goods are shipped. The quality plan involves setting up the master data, inspection plans, and the codes/class/characteristics/info records for inspection and calibration. To record quality inspections during the production process, users create an inspection lot, perform the physical inspection, and record the inspection results with assigned corrective actions, if needed. Samples business flows for Sales, Production and Procurement are as follows:

Sales

Production

Procurement

Integrating Amazon Lookout for Vision and SAP QM
The solution is made of two parts:
Setting up the anomaly detection model using AWS Look out for vision . To get started with Lookout for Vision, we create a project, create a dataset, train a model, and run inference on test images. We finally host the model for use as described in the later section of the solution.

Setting up the workflow and Integration with SAP where we use the Amazon lookout for vision anomaly detection API to identify a defect utilizing the Lookout for Vision model as part of the inspection process, and take appropriate actions in SAP, which we will dive deep in the following sections.

The solution is composed of the following building blocks
1. Equipment(Camera): The images from a manufacturing facility camera can be ingested either directly by the camera, which supports compute, or via a client application that collects images from the cameras and uploads them to S3. For simplicity we are going to manually upload the image to the S3 bucket.
2. S3 bucket: This bucket will be used as a landing zone for images captured by the equipment.
3. Lambda for Orchestration and API integration: This sample package contains Lambda layer to connect with SAP and consume OData services through the HTTP layer. When an image gets landed in the bucket, the S3 object notification invokes the Lambda function, which will orchestrate the process of detecting whether an image is anomalous by making a call to look out for vision, and then the inference result is passed back to SAP for creating a defect.
4. Amazon Lookout for Vision Anomaly model: Amazon look out for vision model which was trained using sample data set, tested and hosted for use as described in the previous section.
5. Metadata data store for SAP Integration: We also use a DynamoDB table for storing the metadata
6. Secret Manager: SAP credentials for accessing the OData service via Lambda
7. SAP Gateway: SAP gateway would be required for exposing the OData API’s
8. SAP S/4 HANA or SAP ECC.

Solution Deployment
There are three key steps in this deployment process. The first part is to setup the model

Setting up the Anomaly Detection Model using Amazon Lookout for Vision
You can build the anomaly detection model using lookout for vision using the console or the SDK.

To keep it simple, we are going to use the cloud formation stack for creating a Sagemaker notebook instance. The instance also comes with the required sample circuit board images and a Jupyter notebook for creating the look out for vision project, train the model, and host the model. To access the cloud formation template and instructions for deploying the vision model follow the instructions in samples repository here.

SAP configuration
Prerequisites
1. SAP S/4 HANA system with embedded gateway where VPC traffic is allowed to access the OData services
2. In case your SAP gateway is deployed as a hub deployment, then the SAP gateway system should be accessible from the VPC.
3. You should have master and configuration data set up for Notification type and other relevant QM configurations as required.

In the SAP backend system activate the OData services, the following OData services are delivered as part of SAP S/4 HANA appliance and for this sample integration, we are relying on those services for creating defects and attaching images to the defect.

1. Activate service API_DEFECT_SRV via transaction /IWFND/MAINT_SE

2.  Activate service API_CV_ATTACHMENT_SRV

3. Create a material in material master for which inspection needs to be performed. We are using the Material CB-FL-001 that was created in the material master.

4. Create a service user in SAP with required authorizations. The authorization should include the start authorizations for the OData service in the back-end system and the business authorizations for creating defects.

AWS deployment and configuration of integration solution

Prerequisites

• AWS Cloud Development Kit (CDK) lets you define your cloud infrastructure as code in one of five supported programming languages. For ease of deployment this sample integration solution building blocks has been packaged as CDK for creating the necessary resources. For AWS CDK installation steps, refer to the AWS CDK documentation.
• Create secret manager resource for storing the SAP credentials, these credentials will be used by lambda for authenticating with SAP for accessing OData services.
• AWS Cloud 9 environment as development environment. See AWS Cloud9 setup instructions here to create an EC2 environment, with t2.micro as the instance type and Amazon Linux2 as the underlying OS.

Deployment

The entire integration has been packaged and can be cloned from the samples repository here. The stack deployment can be verified in AWS Cloudformation. A sample successful deployment would display like below:

1.Maintain the DynamoDB table created through this stack with required metadata as shown below for creating the defect in SAP.

Here is the sample JSON which can be used for creating the DynamoDB Item , update the material and plant as required.

{
 "notiftype": "06",
 "equipment": "CAM-01",
 "DefectClass": "92",
 "DefectCode": "1",
 "DefectCodeCatalog": "9",
 "plant": "1710",
 "DefectCategory": "06",
 "DefectCodeGroup": "QM-E",
 "material": "CB-FL-001"
}

Testing the Solution

Now that we’ve successfully deployment the solution, let’s run a test!

1. From the console navigate to s3 bucket that got created by stack

2. Navigate to the following folder, where the image of product captured by an equipment will be ingested as part of inspection. To simulate an anomaly, we will be manually uploading a defective image to this folder from the additional circuit board data set that is made available part of the repository we used for model training .

3. To simulate an anomaly, from the cloned git repo navigate to the following folder circuitboard>extra_images, and upload the file extra_images-anomaly_5.jpg as shown below to the s3 bucket

The defective image(extra_images-anomaly_5.jpg) is missing a led bulb component compared to the normal image of a circuit board

3. The S3 object notification will trigger the lambda, the Lambda function will orchestrate the process of detecting whether an image is anomalous by making a call to look out for vision, and then the inference result is then passed back to SAP for creating defect. You can check the lambda invocations and logs in the Lambda console as shown below

The logs written by Lambda to cloud watch shows that the image was sent to the look out for vision model as part of anomaly detection and the inference result was passed back to the SAP OData service for creating a defect in SAP and the defect number was logged.

4. Validate the defect created in SAP. Launch the Fiori launchpad and search for the app “manage defects” which is delivered by SAP as part of QM and search for the defect that got created through the Lambda function.

Clean up
Don’t forget to clean up your AWS account to avoid ongoing charges for resources you created in this blog. In order to delete all resources created by this CDK app, run the following command

cdk destroy

In order to delete the Lookout for vision model and data sets, choose CloudFormation – Select your stack and hit delete stack.

Conclusion
In summary, the benefits of Amazon Lookout for Vision include the ability to scale computer vision-based quality inspection at a lower total cost in order to reduce inspection variation and improve quality yields. However, the output of machine learning in industrial operations only provides valuable results if action is taken on the machine learning insights. To reduce the burden of change management and ensure action is taken on the Lookout for Vision insights, this blog has demonstrated how to automatically record the Lookout for Vision defects in SAP Quality management. As a result, users can integrate Lookout for Vision into their existing SAP QM driven quality management process to accelerate the value of computer vision-based machine learning. Learn more about Amazon Lookout for Vision by going to the Amazon Lookout for Vision Resources page.