How to securely extend utility OT data to the cloud

Utilities are considering the strategic advantages of modernizing their operational technology (OT) networks to drive business value. OT systems are rich with data that could be used to support simulations, incident response, and business decisions, but the networks they are connected to are often limited by the storage and compute power available onsite. An OT network is typically designed to run in a physically isolated and secured location, disconnected from the outside world to protect it from intrusion. What if you could use the cloud to extend your OT network without sacrificing isolation and security?

By extending your OT network to the cloud and using cloud services from Amazon Web Services (AWS), you gain elasticity, scalability, and resilience that facilitate the efficient collection and analysis of valuable OT system data to drive business decisions and improve the reliability of your operations. The extension does not require changes to your existing OT network but instead offers a way to move data—including North American Electric Reliability Corporation Critical Infrastructure Protection (NERC CIP) Bulk Electric System Cyber System Information (BCSI)—to the cloud for storage and analysis using AWS services. Some utility use cases include:

• Contingency Analysis and Planning: Run dynamic contingency simulations in OT environments using real system data to help identify, prevent, and prepare for potential reliability impacts.
• Incident Response: Store OT system backups for longer periods of time to use in the event of an operational issue or cyber event. Collect event logs from OT systems for correlation and analysis to identify and investigate operational and cyber events.
• Advanced Analytics: Perform demand forecasting, predictive maintenance, and outage management using Amazon Machine Learning tools to drive business decisions.
• OT Security Monitoring: Apply modern security controls offered by AWS for central network visibility, and for OT security monitoring that leverages automation to correlate events.

Utilities extending their OT networks to the cloud using proven secure methods like AWS virtual private clouds (VPCs) through Amazon Virtual Private Cloud (VPC) have complete control over their virtual networking environments with the security, scalability, and resilience they need.

Solution Approach – AWS Networking Services

AWS offers over 200 services designed to meet your computing needs. A key feature of these services is their inherent focus on security. An Amazon VPC is a user-defined network in the cloud that by default offers no route to permit inbound or outbound traffic unless explicitly configured to. This default isolation is the first step in creating a secure extension of an OT environment into the cloud. The obvious question then is this: If an Amazon VPC offers no inbound or outbound routes for network traffic, how can an OT network extend into it?

There are several ways assets inside a VPC that can communicate with those outside, including:

1. An internet gateway for access to the open internet.
2. A VPN gateway or AWS Direct Connect, for establishing a site to site or point to site private connection to your AWS environment.
3. Network Address Translation (NAT) Gateway so that instances in a private subnet can connect to services outside your VPC but external services cannot initiate a connection with those instances.
4. A peering connection between two VPCs.
5. A Transit Gateway to connect multiple VPCs and VPN connections.
6. If you use VPC endpoints, AWS Server Manager Session Manager for interactive console access to Amazon Elastic Compute Cloud (EC2) instances without them needing to have a public IP address.
7. Private endpoints to various AWS services using AWS PrivateLink.

By default, none of the above capabilities are configured. Only a person with the appropriate permissions for AWS Identity and Access Management (IAM)—which lets you manage access to AWS services and resources securely—can configure these services. Each service requires a deliberate setup and additional configuration of route tables and security group rules (firewall rules) to be fully functional. In addition, by using AWS Config—a service that lets you assess, audit, and evaluate the configurations of AWS resources—you can set AWS Config rules to evaluate and monitor the configuration status of any of these services and notify you in near real time of any changes. Furthermore, you can use AWS Organizations—which helps you centrally manage and govern your environment as you grow and scale your AWS resources—to create a service control policy that will prevent anyone from configuring these services irrespective of their permission level.

It should be noted that a VPC is created in an AWS Region. A Region is a physical location around the world made up of multiple data centers. We call each group of logical data centers an Availability Zone (AZ). Each AWS Region consists of multiple, isolated, and physically separate AZs within a geographic area. You choose the AWS Region(s) in which your content is stored. AWS will not move or replicate your content outside of your chosen AWS Region(s) without your agreement, except as necessary to comply with the law or a binding order of a governmental body.

A utility running a distribution, transmission, or generation system can extend its OT environment to AWS by setting up a VPN connection from the OT network to an Amazon VPC. The encrypted nature of a VPN tunnel will protect data in transit from an OT network to the cloud-based Amazon VPC. In the VPC you create private subnets with only a local route and a route to the VPN Gateway. This way servers in these subnets can only access the VPN tunnels and cannot be accessed from anywhere but via the VPN tunnels. Use AWS Key Management Service (AWS KMS) to encrypt all data at rest.

Figure 1 – Dual VPN to Amazon VPC for resiliency

Let’s take this approach and add more resilience to it. A single VPN connection may not be the best approach as any failure will stop data flow. Adding an additional VPN from additional telecommunications or internet service providers significantly increases the resilience of your connectivity to the Amazon VPC. Each AWS VPN connection offers the ability to create two tunnels; so with two internet service providers, you can have four VPN tunnels to the Amazon VPC. As a result, the resilience of your environment increases, and your electronic access points are well defined and completely in your control.

The configuration so far is secure and resilient but does not guarantee bandwidth and network performance between your OT network and the Amazon VPC. To get that additional assurance you can use AWS Direct Connect to establish a dedicated network connection between your OT network and AWS which uses industry-standard 802.1q VLANs to connect to Amazon VPCs using private IP addresses. Communication over Direct Connect supports IEEE 802.1AE (MACsec) which provides native, near line-rate, and point-to-point encryption for 10 Gbps and 100 Gbps links, thereby securing your data in transit at the fastest rates available. As in the VPN scenario with Direct Connect, you can configure redundant connections for a highly secure environment extending your OT network into the cloud over highly resilient network connections that are encrypted and offer dedicated bandwidth to help you meet your reliability requirements. See figure 2 below.

Figure 2 – Dual Direct Connect to AWS VPC for resiliency

Solution Approach – Additional AWS Services

We have now covered the networking basics, but what about other services such as compute, storage, and analytics?

AWS offers a wide range of services to store, process, and analyze information. How do those services benefit from the isolation offered by an Amazon VPC? Most AWS services can be deployed into a VPC. If you want to launch a containerized application for processing and analyzing your data you can use Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (Amazon EKS) both of which can be configured to be deployed into your VPC of choice. If you want serverless compute you can deploy AWS Lambda functions into your VPC. For high performance computing, you can deploy Amazon EMR into your VPC.

For your database needs, you can deploy an instance of SQL Server, MySQL, PostgreSQL, Oracle, or MariaDB using Amazon Relational Database Service (RDS) or MySQL and PostgreSQL using Amazon Aurora into a VPC of your choice.

For your storage of all data types, Amazon Simple Storage Service (Amazon S3) can be used including automated lifecycle management capabilities to move less frequently accessed data from hot, to warm, to cold storage. Amazon S3 resides outside of your own VPC within your region, but you can configure a VPC endpoint so any communication between the storage service and your VPC is internal to the AWS network over TLS. Furthermore, you can create bucket policies restricting access to your bucket based on least privileges. It should be noted that all the services mentioned above and many more natively support encryption using AWS KMS.

Examples of these additional services are depicted in the Reference Architecture at the end of this article.

For controlling access to servers in the Amazon VPC, you have a choice to extend your OT domain to the cloud or create a domain in the cloud. AWS Directory Service lets you set up a managed Active Directory Service on AWS which is also hosted in your VPC.

NERC CIP Compliance Considerations

Utilities with systems in-scope under the NERC CIP Standards may be wondering what controls can be applied to help support their compliance if they choose to use this solution to store and analyze BCSI from their NERC in-scope systems. The table below includes the AWS controls and services identified above, and the CIP Requirement that each one helps to support. Be aware that these controls and services include shared responsibilities between the utility and AWS.

Reference Architecture: BCSI on AWS

Clicking on the diagram below will take you to the AWS Reference Architecture showing how you can securely extend your OT network without changing any existing OT technology by creating VPN or AWS Direct Connect connections to an Amazon VPC thus opening to the door to various AWS services in a secure manner.

Customers can also use multiple AWS accounts to set a strong security boundary and separate by environment, workload, or security domain. For further guidance, see Organizing Your AWS Environment Using Multiple Accounts.

Visit AWS Power and Utilities for more information.