What actions customers can take to protect, detect, and respond to Log4j vulnerabilities in Operational Technology (OT) and Industrial Internet of Things (IIoT) environments

In this post we will provide guidance to help industrial customers respond to the recently disclosed Log4j vulnerability. This post covers how to identify if you are susceptible to the issue, and then how to address the vulnerability in OT and IIoT environments.

The Log4j vulnerability (CVE-2021-44228, CVE-2021-45046) is a critical vulnerability (CVSS 3.1 base score of 10.0) in the ubiquitous logging platform Apache Log4j. This vulnerability allows an attacker to perform a remote code execution on the vulnerable platform. Version 2 of Log4j, between versions 2.0-beta-9 and 2.15.0, is affected. The vulnerability uses the Java Naming and Directory Interface (JNDI) which is used by a Java program to find data, typically through a directory, commonly an LDAP directory in the case of this vulnerability.

Log4j is an open source Java logging library used extensively by developers. The use of third-party libraries in applications isn’t just an IT problem but also an OT/IIoT problem as industrial digital transformation is driving changes to the OT landscape. As these environments continue to evolve, OT environments are leveraging more IT solutions to improve productivity and efficiency of production operations. It’s no surprise that Log4j is embedded in Operational Technology (OT) and Industrial Internet of Things (IIoT) systems and OT/IIoT vendors have released advisories on how their products are impacted. A key challenge with Log4j in Industrial Control System and Operational Technology (ICS/OT) environments is that at this stage it’s hard to identify exactly what’s affected. In the weeks and months ahead, we will begin to understand the pervasiveness and extent of this particular vulnerability in OT infrastructure; but it is most certainly used to perform critical OT/IIoT logging functions making that system vulnerable to remote code execution. Additionally, adversaries can leverage this vulnerability in proprietary Supervisory Control and Data Acquisition (SCADA), engineering workstations, Human Machine Interfaces (HMI), Energy Management Systems (EMS), and IIoT systems, which use Java in their codebase.

ICS/OT vendors, and IIoT platform providers have started sharing exactly which of their systems are affected, releasing patches that will fix the vulnerability, and providing detailed mitigation plans. Customers should immediately act to identify assets affected by Log4j, upgrade Log4j assets to the latest version as soon as patches are available, and remain alert to vendor software updates. Customers also need to ensure they monitor and protect network access to these systems and implement cybersecurity best practices across their industrial operations to protect against exploitation of this vulnerability. Here are some mitigation steps customers can take to protect, detect, and respond to Log4j vulnerabilities in Operational Technology (OT) and Industrial Internet of Things (IIoT) environments.

Protect – Patch your devices and to understand what you need to patch make sure you know what you have and where. Limiting the scope of network connectivity where possible reduces the risk/exposure from the Log4j vulnerability.

Identify the location of potentially affected digital assets. You can use your asset/software inventory to identify known applications that were published as vulnerable to Log4j. You can follow https://github.com/cisagov/log4j-affected-db#software-list to view a maintained vulnerable software list and it’s important to track individual vendor sites for the most up to date information.

Scan ICS/OT assets. When asset/software inventory is not available or to augment the inventory, you can run a targeted scan of ICS/OT assets with tools like CERTCC’s (published by CISA), both for Linux and Windows systems. Many OT Intrusion Detection System (IDS) vendors also provide vulnerability scanning tools for OT and IIoT assets, so check with your vendor and scan where feasible after taking the necessary precautions not to impact production operations.

Understand software components in your systems – By understanding the software components in your systems, you can check if you have security vulnerabilities in your dependencies. If a flaw is discovered in one of the libraries your code depends on, you can view the dependency tree of the software you build/procure to determine if you are affected. Note that OT systems are typically proprietary in nature. A full software inventory of these systems is often not available. Work with your vendors to collect, maintain, and update software inventory to keep track of what components vendors may use in their systems.

Patching – If you identify these vulnerabilities within an application or endpoint in your OT/IIoT environment, remediate as soon as possible through patching if available. Asset owners can rely upon vendors to provide patches to impacted software products. It’s important to do a risk assessment and use an up to date network architecture to determine how these vulnerable systems can be accessed from external networks and to identify and patch the most critical assets first. When patching vendor systems, follow the steps provided by the vendor and conduct extensive testing of patches before applying them to production systems. AWS provides AWS IoT jobs to define a set of remote operations that you send to and execute on one or more IIoT devices connected to AWS IoT and AWS Systems Manager Patch Manager to patch on premise computers and edge gateways.

Quarantine unsupported systems – With the longer hardware and software refresh cycles in ICS/OT environments, it is common to find ICS/OT products that are no longer under active support or whose software vendors no longer exist. OT environments typically have lots of equipment that can’t be patched, End-of-Life (EOL), cyber fragile, or “insecure by design.” In these cases, patching may not be possible. Quarantine any vulnerable asset that can’t be patched such that it cannot be directly accessible (e.g., where there is a high likelihood of the affected software being probed by adversaries on the internet) or used within a larger networked system of systems. You can significantly reduce the risk of impact on industrial systems by using micro-network segmentation of the IT/OT networks and this is a general best practice regardless of the Log4j vulnerability.

Detect – Monitor to detect whether this vulnerability exists in your environment, respond to alerts from OT/IIoT systems and pay particular attention to the IT environments that they connect to.

Security Monitoring – If Intrusion Detection System (IDS) and network monitoring systems are deployed in the OT network, monitor for odd traffic patterns (e.g., JNDI LDAP/RMI outbound traffic, DMZ systems initiating outbound connections) and configure with Log4Shell indicators of compromise (IOCs) to detect and escalate the alerts for faster response to potential exploitation. Note the advisories from OT IDS vendors for information related to updates on their network-based detections for ICS based active Log4j exploitation. OT/IIoT systems closest to enterprise networks and the internet have the greatest risk exposure and could likely be used by a threat actor as a pivot point into OT. Similarly, threat actors can pivot into the enterprise or cloud environment through compromised OT/IIoT systems. Security monitoring should be comprehensive and cover the entire attack surface, which includes OT, industrial edge, IT, and Cloud. Customers can use AWS IoT Device Defender to audit and monitor their fleet of IoT devices and detect anomalies in device behavior, Amazon Inspector to look for Log4j vulnerability for all supported AWS Systems Manager managed instances including on premise computers and edge gateways and Amazon GuardDuty to detect indicators of compromise associated with exploiting the Log4j vulnerability in the cloud.

Respond – Build automation to respond and quarantine assets where you see suspicious activity. Prepare an incident response plan/runbooks and test these regularly.

Investigation and Incident Response – Customers can investigate potential compromise and hunt for signs of malicious activity by using threat detection methods and tools like logs, SIEM, etc. and respond and remediate where applicable.

Customers can use AWS Security Hub with AWS IoT Device Defender, Amazon Inspector and Amazon GuardDuty to aggregate alerts and enable automatic remediation and response. In the short term, we recommend that you use Security Hub to set up alerting through AWS Chatbot, Amazon Simple Notification Service, or a ticketing system for visibility when Inspector finds this vulnerability in your environment. In the long term, we recommend you use Security Hub to enable automatic remediation and response for security alerts when appropriate. Here are ideas on how to setup automatic remediation and response with Security Hub.

Review and monitor Apache Log4j Security Vulnerabilities webpage for AWS updates and mitigation guidance and work closely with your vendors to monitor any updates on affected systems. In addition, follow the Apache Log4j Vulnerability Guidance provided by CISA, AWS Security Bulletins and learn more about AWS security services to protect against, detect, and respond to the Log4j vulnerability.

Finally, Plan to replace legacy unsupported systems as soon as possible. OT systems are likely to include components that are 20-30 years old, or even older. Some systems may be so old that they predate any and all concerns about cybersecurity, and other systems may simply have inadequate security measures or reached their end of life. If you find that aging OT infrastructure is presenting a significant risk to your operations, then the mitigation strategy might include a plan to upgrade, replace or decommission outdated assets. When considering a holistic risk management program, modernizing the OT environment can be a major enabler to reducing the risks facing an organization.

Conclusion

In this blog post, we outlined key actions that enable customers to adopt a layered approach to help them protect, detect, and respond to the Log4j vulnerability in OT and IIoT environments. Given the ease of exploitation for this vulnerability, the scope of potentially impacted applications, and the time required to apply comprehensive fixes across large organizations, AWS recommends taking a multi-layered and defense in depth approach to secure the ICS/OT, IIoT, and cloud environments by following the AWS security golden rules for IIoT solutions, AWS Security Best Practices for OT and AWS Well Architected Framework, IoT Lens to design, deploy, and architect IIoT workloads aligned with architectural best practices.

About the author

Ryan Dsouza is a Principal Solutions Architect for IIoT at AWS. Based in New York City, Ryan helps customers design, develop, and operate more secure, scalable, and innovative solutions using the breadth and depth of AWS capabilities to deliver measurable business outcomes. Ryan has over 25 years of experience in digital platforms, smart manufacturing, energy management, building and industrial automation, and OT/IIoT security across a diverse range of industries. Before AWS, Ryan worked for Accenture, SIEMENS, General Electric, IBM, and AECOM, serving customers for their digital transformation initiatives.