Industrial Automation Software Management on AWS—Best Practices for Operational Excellence

Factory or intralogistics automation, which can include assembly stations, sortation stations, robots, and conveyors, are programmed and operated by their respective native controllers or programmable logic controllers (PLCs). These controllers execute programs to control physical equipment to accomplish a task—for example, painting a car with the specified color or sorting an item by its barcode according to its final destination. Depending on the equipment and the process, PLC programs may be developed by equipment vendors, systems integrators, or factory automation engineers. In a factory, multiple systems running various PLC codes are integrated and commissioned end-to-end for seamless operations of the production workflow. The code, its recipes, and its versions also need to be continuously maintained to run the automation at its optimum level. Given the complexity of dealing with multiple vendor systems and code collaborators, it can be quite challenging to implement change control and manage the lifecycle of these critical code assets. This blog post is the first of a three-part-series that details how Amazon Web Services (AWS) brings modern DevOps, Digital Twin and Virtual Commissioning technologies to develop, test, commission and maintain the automation code. This article covers automation programming best practices from operational perspective.

Traditional automation code development and maintenance

Imagine an automation task of “depalletizing” the incoming goods in a warehouse – where robots pick up, or destack, boxes off of a pallet that comes to the station on a conveyor. Such an automation is a mix of PLC and robot programming and controls logic of the conveyor, proximity sensors that recognize pallet position, computer vision to identify the boxes, robot programming for reaching the boxes, vacuum triggers to activate or deactivate the gripping action, and so on. This automation solution could be done by a single supplier or by multiple vendors programming the various pieces of equipment involved, and it is ultimately integrated by the warehouse owner themselves or a third-party integrator. The commissioning of the automation includes development of PLC code, robot code, integration of sensors via input-output (IO) modules, wiring and connecting sensors with their latest firmware updates, code testing, integration, and so on. The automation is designed and executed for the functional logic of the pallet in position for the robot, then the robot’s motion, followed by the gripper activation, etc. This is coordinated via PLC or robot, with one of them being the master controller. This logical PLC programming can be visualized by three graphical and one textual programming language standards:

• ladder diagram (LD): graphical programming language standard
• function block diagram (FBD): graphical programming language standard
• structured text (ST): textual programming language standard
• sequential function chart (SFC): graphical programming language standard

PLC code can be tested virtually using a virtual PLC with virtualized IOs, or by real-life commissioning on the actual equipment. PLC code is generally edited using a proprietary vendor integrated development environment (IDE) and is stored in proprietary binary file formats. Similarly, robot programming is typically written in structured text and uses the proprietary programming standards of the robot vendors. The robot programs and their behavior can be virtualized in their native simulation environments or by using third-party tools such as Siemens’s Process Simulate or Dassault’s Delmia. After the automation is deployed and commissioned, operations and maintenance teams continuously monitor and keep track of PLC and robot programs. Their code requires changes or tuning whenever:

• performance adjustments need to be made—for example, to improve cycle time if the operations team wants to increase robot acceleration while depalletizing boxes;
• new product variants are to be manufactured on the same line, which produces different recipes of the code;
• bugs and errors need to be corrected; or
• the underlying hardware or firmware needs to be updated and the corresponding PLC/robot code reverified and redeployed.

These operational and maintenance tasks can become complex, and change control becomes challenging as the number of PLCs and robotics or other automation systems increases. Problems arise when the right version and right configuration of the code is not found. While code and configuration management is a standard DevOps practice for software development, these practices are not as common in the world of industrial automation, primarily due to lack of good tooling. These challenges can now be solved through systematic, secure, and easily accessible solutions in the AWS cloud.

AWS-backed automation code maintenance

AWS offers partner-led solutions that can automatically back up PLC and robot code to the cloud, maintain versions and its variants (recipes), offer tools to compare code, and assist in code reviews before bringing back the right code from the cloud to PLCs. These solutions follow strict security guidelines, supervisory checks, and sign-offs to ensure integrity of the code. One such solution is Copia Automation’s Git-based source control (Git is an open-source DevOps tool for source code management). Copia Automation brings the power of a modern source control system (specifically, Git) to industrial automation, solving the operational problems outlined in the section above. Strong version control practices help organizations recognize changes made to the system, who made it, and when and why it was made. This allows a deeper understanding of changes and the ability to switch between any previous version, increasing the efficiency and reducing unplanned downtime.

Traditional tools such as Github or Gitlab primarily support text-based programming languages (Python, JavaScript, Java, C#, etc.) and do not fully support the IEC 61131-3 standard (LD, FB, ST, SFC) used in PLC programming. This leaves the automation professional with limited options, and they often turn to the file folders on local servers, or many times their own workstations, for version control.

Copia Automation’s tool set not only back-up PLC code versions to the AWS cloud, but they also empowers engineers to visualize their PLC code from a variety of vendors and a variety of PLC programming languages. This unlocks traditional Git workflows such as logic comparison, branching, merging, code reviews, and code commenting for automation professionals. Copia Automation’s source control solution works natively with vendors’ binary file types and provides a powerful visualization layer that helps them utilize the power of Git, which is not possible with standard Git providers.

Figure 1: Copia’s desktop application for source control

Built on top of its source control solution, Copia Automation’s DeviceLink helps organizations systematically back up industrial devices and store them in the cloud. DeviceLink shows up-to-date code from the floor in one centralized location, regardless of how much equipment is managed or how many sites they’re spread across. Copia Automation’s DeviceLink works with a multitude of vendors (Allen-Bradley, CODESYS, Siemens, Schneider, and FANUC, to name a few) and can schedule code backups at any time or frequency. The organization can also detect or be notified about any changes in the backed-up version versus the last known version.

Overall, Copia Automation’s cloud-based source control and automated backup system provides increased reliability, allows for seamless disaster recovery, and minimizes downtime in case of unforeseen events. It provides for seamless code collaboration, pull-request-based code update workflows, code analysis, and collaborative code reviews. In short, it brings the modern DevOps practices to industrial automation, increasing the overall efficiency and maintainability of automation code.

Figure 2: Copia Automation’s secure connectivity from plant to cloud for automation code backups

Advantages

Reliability and disaster recovery

One of the primary advantages of backing up PLC code to the AWS cloud is the increased reliability it offers. Traditional backup methods, such as on-premises servers or local storage, are susceptible to failures, data loss, and disasters. By using cloud storage, organizations can help ensure that their PLC code is protected against hardware malfunctions, software crashes, and physical damage. Cloud-based backups provide an extra layer of redundancy, allowing for seamless disaster recovery and minimizing downtime in case of unforeseen events.

Accessibility to remote experts and reduced downtime

Most organizations today still look for who has the latest PLC code, in which thumb drive, and to call that person to fix the code. AWS-based PLC code backups provide remote accessibility, allowing authorized personnel to access the PLC code and its previous versions from anywhere that has an internet connection. This averts the need to hunt for the right version of code, the right thumb drive, and the right expert who needs to be physically on-site to fix the code. This level of accessibility empowers organizations to collaborate seamlessly across teams and locations—even with external stakeholders—and to quickly find root causes to problems in order to minimize unscheduled downtime due to PLC or robot code fixes. Additionally, cloud backups can minimize downtime in case of hardware failures, software crashes, or natural disasters, allowing for faster recovery and reduced impact on operations. By quickly restoring PLC code from cloud backups, organizations can resume production and minimize revenue loss associated with unplanned downtime.

Systematic code reviews to avoid errors in deployment and code maintenance

Because multiple experts can more easily review the code from different geolocations, conducting PLC code reviews using cloud technologies can be an effective way to catch code errors and bugs. This can be accomplished using the following steps:

1. Version control and collaboration: Use version controls as with other cloud-based software platforms, such as GitHub or GitLab, to the hosted PLC/robot code with its variants. AWS code repository provides centralized location for code review.
2. Pull request workflow: Adopt a pull request workflow, commonly used in software development, to facilitate code reviews. When a developer completes a new feature or fixes a bug, they create a pull request to lock the code until it is reviewed and approved for deployment. Other team members can review the PLC/robot code changes, provide feedback, and suggest improvements.
3. Automated code analysis: Use cloud-based code analysis tools to automatically check the PLC code for errors, style violations, and potential bugs. Code analysis tools—from structured text analytics to rendered code (e.g., ladder logic) comparisons—can be used to find differences in code versions.
4. Collaborative code review: Utilize cloud-based collaborative code review tools to facilitate efficient and effective review processes. Platforms such as GitHub and GitLab provide built-in code review features, allowing reviewers to add comments, suggest changes, and discuss code directly within the pull request. This promotes collaboration, encourages knowledge sharing, and helps catch errors and improve code quality.
5. Code simulations and testing: Emulators and simulators specific to your PLC system can be deployed in the cloud. These environments allow running the code in a controlled setting and provide functional testing and validation before deployment.

Scalability and flexibility

Cloud storage offers unparalleled scalability and flexibility compared to traditional backup methods. As industrial operations expand or new projects emerge, the storage needs for PLC code may grow exponentially. AWS offers scalable solutions, allowing organizations to dynamically adjust their storage capacity based on demand. This scalability ensures that businesses can accommodate future growth without investing in additional hardware or storage infrastructure. Furthermore, cloud-based backups can accommodate various PLC programming languages, making them compatible with a wide range of PLC systems.

Improved security and data protection

Data security is a paramount concern for organizations handling PLC code, as it contains proprietary information and operational intelligence. At AWS, security is “job zero,”—meaning that it’s more important than any number one priority—and it is implemented in a shared-responsibility model. AWS cloud offers robust security measures, including encryption, access controls, and threat detection systems to safeguard data against unauthorized access and cyberthreats. Organizations can implement security in the cloud by using these measures as well as by implementing user-access, levels, and organizational authority over the PLC or robot code. AWS also provides granular traceability on the changes to the code regarding who accessed the code and when it was changed. This information is often not recorded, or can get lost, in an on-premises system. Code backups in the AWS cloud often comply with industry-specific data protection regulations. For example, the Life Sciences’ Good Manufacturing Practice (GMP) aids organizations in meeting compliance requirements and potentially avoiding penalties. AWS cloud storage also offers georedundancy, which distributes data across multiple regions and availability centers, thereby minimizing the risk of data loss due to physical or localized incidents. Regular backups to the cloud ensure that even if a local system is compromised, the code remains secure and recoverable.

Cost savings

Functional and performance cost savings include:

• operational efficiencies in locating the right version of code and finding root causes for code-related failures;
• reduced unscheduled downtime;
• reduced errors in code deployments due to systematic code reviews with experts, who can be remote; and
• secure and governable access to code.

In addition, this approach saves costs in other ways:

• Cloud storage eliminates the need for significant upfront investments in on-premises infrastructure, hardware, and maintenance costs.
• Pay-as-you-go pricing models offered by AWS allow organizations to align expenses with actual storage usage – optimizing cost efficiency.
• Cloud backups can free up local storage resources, potentially reducing the need for additional storage infrastructure which saves on equipment and maintenance costs.

Amazon Warehouse automation ops case study

Problem statement

Amazon fulfillment has a wide range of material handling equipment and work cells using multiple PLC manufacturers. Buildings and work cells that shared the same original design often would have code changed over time by local controls technicians, which caused variability throughout the network and lacked appropriate feedback mechanisms to design and development teams. Collaboration and file backups were done manually that allowed co-development, and maintaining the latest backups was enforced only through an honor system. Amazon reviewed various internal and external solutions and selected Copia Automation as its pilot software, based on its flexibility with various PLC manufacturers and its ability to allow customers to self-host on their own AWS servers.

Implementation

In 2023, Amazon engineers collaborating with Copia Automation completed a proof-of-concept (PoC) of automatic backups of automation software at three warehouses. The backups included PLCs and configuration files for equipment, such as safety scanners. During the PoC, Amazon’s business and design needs were verified, security and performance of the solution were tested, and Copia Automation’s functionality was customized to be integrated with Amazon’s workflow. After successful completion of the PoC, Amazon extended Copia Automation’s solution to 10 more warehouses, with over 500 PLCs to back-up code daily and provide smart reporting of changes. Copia Automation supported the Amazon team in being able to host agent and conversion servers on internal instances of Amazon Elastic Compute Cloud (Amazon EC2)—which provides secure and resizable compute capacity for virtually any workload—where Amazon had full control of approved traffic flow to PLCs. Alerts are generated over email and Slack, and network-level dashboards were created by connecting databases to AWS QuickSight—which provides unified business intelligence at hyperscale—for flexible reporting.

Solution Architecture and Best Practices

The Copia solution is deployed in Amazon’s own AWS account. In this type of deployment model, Amazon is responsible for managing and configuring its own infrastructure needed to run Copia’s software. The typical architecture is shown in Figure 3. The primary software application (App Server) is run on Amazon Elastic Kubernetes Services (Amazon EKS). It uses Amazon Elastic Block Storage (Amazon EBS) as the storage layer for storing Git repositories. The metadata needed to run the application is stored in a Amazon Relational Database Service (Amazon RDS) database while Amazon Simple Storage Service (Amazon S3) is used for storing backups. On the Factory side, the Copia Agent software can be installed on Agent Servers running Vendor-specific software for Integrated Development Environment (IDE) and connectivity. The Agent software works in conjunction with the vendor software to connect with PLCs and other devices. The agent software connects to the App Server over HTTPS. Amazon Route 53 is used to resolve the DNS, and Application Load Balancer (Amazon Elastic Load Balancer) serves as the end point for receiving and distributing the traffic to the appropriate App Server pods. Shop floor users or vendors can access the application through a web interface. A Conversion Server Instance on Amazon EC2 is used by the Copia application to translate PLC programs from vendor proprietary language to a common intermediate format for display on the web.

The Amazon EKS cluster is configured to be horizontally scalable. Connectivity between the Factory and the AWS cloud over AWS Direct Connect (or AWS Site-to-Site VPN) keep all the traffic private and encrypted. This architecture allows Amazon to privately store their code repositories and enable a centralized source management practice for their Automation code. It enables Amazon to standardize code management and code collaboration practice both internally and externally (with vendors).

Figure 3: Solution Architecture for Copia Deployment

Benefits

Amazon expects to reduce unexpected downtime from unapproved controls changes by 80 percent and significantly improve scalability of known code improvements that should be propagated across the entire network. High-severity issue resolution time is expected to improve by 25 percent or more by having proper change history and file backups and by reducing the time of downloading and sending files with internal or external original equipment manufacturer (OEM) customer support. External OEM vendors can be provided read-only access to support troubleshooting, and they can be given full access if they are supporting implementation of new equipment.

Conclusions

Manufacturing enterprises need a durable and remotely accessible solution for managing their PLC and Robot program versions. AWS, together with our partner Copio.io, offers toolsets to store such automation code, render the programs in a web portal to visually understand the differences between versions, and provides systematic workflows for code reviews, code merging and deployment. This solution helps manufacturers manage disaster recovery, code rollbacks, and remote collaboration with vendors or experts. Because the code repositories are to be accessed via the cloud, a strong security and code integrity protocls can be implemented –access to the code can be implemented as per company roles and policies; supervisory controls monitor, trace, approve or reject, or alert appropriately whenever code is accessed; and the code is encrypted at rest and in transit. Amazon deployed automation at 13 warehouses, connected about 500 PLCs to back-up to the AWS cloud, and expect to reduce 80% of unexpected downtime due to PLC code errors.