Common architecture patterns to securely connect IoT devices to AWS using private networks

Introduction

Increasingly, business leaders are adopting Internet of Things (IoT) solutions to drive revenue growth, streamline operations, and reduce costs. Managing security and safety considerations while connecting your assets to the cloud, whether they’re industrial machines or autonomous vehicles, can be challenging. In the Ten security golden rules for Industrial IoT (IIoT) Solutions, AWS recommends establishing secure connections from industrial environments to the cloud and secure remote access to resources on-premises. Similarly, connected mobility solutions commonly use private cellular networks to connect vehicles to cloud services.

This blog covers common architecture patterns and best practices to safely and securely connect IoT devices to AWS using private networks. Using the Virtual Private Cloud (VPC) endpoint feature for AWS IoT Core credential provider, it is now possible to operate an AWS IoT Greengrass-powered device in a VPC, without public internet access. In addition, these devices can access other AWS services, such as Amazon Elastic Container Registry (Amazon ECR), AWS Secrets Manager, and Amazon CloudWatch logs, using AWS PrivateLink. This approach provides you more flexibility in securing your connected solutions by isolating network traffic from the internet by establishing private connections, and it also helps you comply with your organization’s security best practices.

Solution overview

The solution described enables you to connect your IoT devices to AWS IoT Core and AWS IoT Greengrass using a private endpoint in Amazon VPC. Private endpoints use private IP addresses from a virtual network address space to connect your devices privately to AWS IoT Core data endpoints and AWS IoT Greengrass within your VPC.  Interface VPC endpoints are used to connect to services powered by AWS PrivateLink, an AWS service that you can use to establish connectivity between VPCs and AWS services without exposing data to the internet. Network traffic between connected devices and AWS IoT Core and AWS IoT Greengrass use AWS site-to-site VPN or AWS Direct Connect, eliminating exposure on the public internet. Let’s go over the solution architecture and solution components.

Scenario 1: IoT devices connecting to AWS IoT Core using private network

Figure 1: IoT devices in organizations connecting to AWS IoT Core through private networks

Solution description

The flow contains the following steps:

1. An asset located in the factory needs to resolve an ‘AWS IoT data endpoint’ domain name. The AWS IoT device data endpoints support a publish/subscribe protocol that is designed for the communication needs of IoT devices. It sends the query to its pre-configured Domain Name System (DNS) Resolver.
2. The DNS Resolver in the corporate data center has a conditional forwarder rule that points all DNS queries for ‘AWS IoT data endpoint’ DNS domains to the Amazon Route 53 Resolver Inbound Endpoint.
3. The forwarded query arrives at the Amazon Route 53 Resolver Inbound Endpoint through either AWS Direct Connect or an AWS Site-to-Site VPN.
4. All inbound DNS queries flow through this VPC on the way to the Resolver. To improve reliability, Resolver requires that you specify two IP addresses for DNS queries. We recommend that you specify IP addresses in two different Availability Zones for high availability.
5. The inbound endpoint sends the query to Route 53. The Route 53 Resolver resolves the DNS queries for AWS IoT Core Data domain names.
6. The Private Hosted Zone associated with the VPC holds the DNS records for AWS IoT Core Data endpoint so that the Route 53 Resolver can resolve the query.
7. Traffic destined for the AWS IoT Core Data endpoint is resolved to the private IP addresses of the endpoint network interfaces using DNS, and then sent to the AWS service using the connection between the VPC endpoint and AWS IoT Core privately.

For security considerations,

• Set VPC Interface endpoint with security groups and network ACL on endpoint Elastic Network Interface
• Use VPC condition context keys to control access to AWS IoT Core Data over VPC endpoints.

The following table shows the required details for AWS IoT data VPC endpoint. For more details please visit the documentation.

Figure 2: VPC endpoints with corresponding DNS aliases for IoT devices

Figure 3:  Setting up VPC endpoints in AWS console

Note: Find more details on creating an interface VPC endpoint along with creating AWS IoT Core with interface VPC endpoint. For more information, on creating a private hosted zone in Amazon Route 53 refer to the documentation.

Scenario 2: AWS IoT Greengrass-powered device connecting to AWS IoT Core using AWS IoT credentials VPC endpoint

Figure 4: AWS IoT Greengrass powered devices connecting to AWS IoT Core over private networks

Solution description

The flow contains the following steps:

1. The sensors, which are IoT Greengrass client devices, connect and communicate with an IoT Greengrass core device over MQTT. The IoT Greengrass core software at the edge needs to resolve an ‘AWS IoT data endpoint,’ ‘AWS IoT credentials,’ and ‘Amazon Simple Storage Service (Amazon S3)’ domain name. It sends the query to its pre-configured DNS Resolver. Based on your use case, additional endpoints may be needed.
2. The DNS Resolver in the corporate data center has a conditional forwarder rule that points all DNS queries for ‘AWS IoT data endpoint,’ ‘AWS IoT credentials,’ and ‘Amazon S3’ DNS domains to the Amazon Route 53 Resolver Inbound Endpoint.
3. The forwarded query arrives at the Amazon Route 53 Resolver Inbound Endpoint through either AWS Direct Connect or an AWS Site-to-Site VPN.
4. All inbound DNS queries will flow through this VPC on the way to Resolver. To improve reliability, Resolver requires that you specify two IP addresses for DNS queries. We recommend that you specify IP addresses in two different Availability Zones for high availability.
5. The inbound endpoint sends the query to Route 53. The Amazon Route 53 Resolver resolves the DNS queries for ‘AWS IoT data endpoint’, ‘AWS IoT credentials’ and ‘Amazon S3.’
6. The Private Hosted Zone associated with the VPC holds the DNS records for ‘AWS IoT data,’ ‘AWS IoT credentials,’ and ‘Amazon S3’ endpoint so that the Amazon Route 53 Resolver can resolve the query.
7. Traffic destined for the ‘AWS IoT data,’ ‘AWS IoT credentials,’ and ‘Amazon S3’ endpoint is resolved to the private IP addresses of the endpoint network interfaces using DNS, and then sent to the AWS service using the connection between the VPC endpoint and AWS IoT Core privately.

Note:

1. When the AWS IoT Greengrass core software deploys a component, it downloads the component’s artifacts from AWS. By configuring a VPC endpoint for Amazon S3, you enable the Greengrass core devices to access these artifacts securely and more efficiently.
2. In AWS IoT Greengrass nucleus configuration, greengrassDataPlaneEndpoint must be set to iotdata. For more information, see Greengrass nucleus configuration. This setting specifies the endpoint that the Greengrass nucleus uses to communicate with AWS IoT Greengrass service. By setting it to iotdata, Greengrass core uses the AWS IoT Data Plane endpoint to communicate with AWS IoT Greengrass. This configuration is crucial for enabling the core device to communicate effectively with AWS IoT Core, ensuring that it can send and receive necessary data for its operations and deployments.

The following table gives information about the corresponding custom private DNS aliases. For more information, visit the documentation.

Figure 5: VPC endpoints with corresponding DNS aliases for AWS IoT Greengrass powered devices

AWS IoT endpoint (com.amazonaws.region.iot.data) is used to manage components, deployments, and core devices from the AWS IoT Greengrass service.

Authentication and authorization with this endpoint is done using X.509 certificates as described in ‘Device authentication and authorization for AWS IoT Greengrass’.

Depending on your IoT use cases and the features you use, you might need additional endpoints. For example, for AWS-provided AWS IoT Greengrass components, please refer to the documentation to understand what services are required for the component to function. A few common examples:

Figure 6: Examples of AWS service VPC endpoints

AWS IoT Core credentials provider endpoints (com.amazonaws.[region].iot.credentials) are used to communicate with other AWS cloud services that do not support X.509 authentication and authorization, like Amazon Simple Storage Service (Amazon S3) and Amazon Elastic Container Registry (Amazon ECR). In these cases, AWS IoT Core or an AWS IoT Greengrass component will call AWS IoT Core credential provider endpoint using the X.509 certificate to authenticate and get authorized. The endpoint will issue a temporary security token for the client to use in the call to the services not supporting X.509. Calls to Amazon S3 and Amazon ECR services are required during the IoT Greengrass component deployments. The IoT Greengrass component will also require a security token if they use AWS SDKs to communicate with other cloud services that do not support X.509 certificates authentication and authorization mechanism. If you are using your own component, you may need to review the dependencies and perform additional testing to determine if any additional endpoints are required.

Controlling access to AWS IoT Core over VPC endpoints

You can restrict device access to AWS IoT Core to be allowed only though VPC endpoints by using VPC condition context keys. You can use SourceVpc key to check whether the request comes from the VPC that you specify in the policy. Use the SourceVpce key to compare the VPC endpoint identifier of the request with the endpoint ID that you specify in the policy to restrict access to a specific VPC endpoint. With the VPCSourceIp, you can compare the IP address from which a request was made with the IP address that you specify in the policy.

Note: This policy would deny connection attempts to your public IoT data endpoint.

Creating a VPC endpoint policy for AWS IoT Greengrass

When you create an interface VPC endpoint for AWS IoT Greengrass control plane operations, such as CreateDeployment and ListEffectiveDeployments, you can use a VPC endpoint policy to  controls access to AWS IoT Greengrass control plane operations which helps to improve your security posture. The policy specifies the following information:

• The principal that can perform actions.
• The actions that the principal can perform.
• The resources that the principal can perform actions on.

The following is an example of an endpoint policy for AWS IoT Greengrass. When attached to an endpoint, this policy grants access to the listed AWS IoT Greengrass actions for all principals on all resources.

{
    "Statement": [
        {
            "Principal": "*",
            "Effect": "Allow",
            "Action": [
                "greengrass:CreateDeployment",
                "greengrass:ListEffectiveDeployments"
            ],
            "Resource": "*"
        }
    ]
}

Limitations of AWS IoT data VPC endpoints and AWS IoT Core credential provider endpoints

At the time of writing this blog, IoT data VPC endpoints and credentials provider endpoints have some limitations. For example,

• IoT data VPC endpoints’ MQTT-based keep alive periods are limited to 230 seconds and each VPC endpoint supports up to 100,000 concurrent devices.
• Only IPv4 traffic is allowed by both endpoints.
• Both endpoints will serve Amazon Trust Service (ATS) certificates only and VPC endpoint policies are not supported.

However, despite these restrictions, AWS IoT Core data endpoints and AWS IoT Core’s credentials provider feature do provide a secure way to connect large numbers of devices to AWS using private networks. Check the AWS documentation for the most up-to-date information on capabilities and constraints.

Conclusion

With devices deployed in a variety of different environments, locations, and scenarios, you need flexibility and security when implementing IoT solutions. In this blog, we discussed the architecture and best practices to securely connect IoT and IoT Greengrass-powered devices to AWS IoT Core and other AWS services using private networks. This solution provides you the ability to isolate your connected devices and network from the internet and use a private network to send data to AWS. This approach helps establish secure communications over a private network, helps protect AWS resources from security events in public networks, and allows you to align your operations in line with your organization’s security best practices and requirements. To learn more, visit Security in AWS IoT.

Resources:

• Using AWS IoT Core with interface VPC endpoints – https://docs.aws.amazon.com/iot/latest/developerguide/IoTCore-VPC.html
• AWS Direct Connect – https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html
• Controlling Access to AWS IoT Core over VPC endpoints – https://docs.aws.amazon.com/iot/latest/developerguide/IoTCore-VPC.html#Control-VPC-access
• Resolving DNS queries between VPCs and your network – https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver.html
• Back to Basics: Understanding IOT Core VPC Endpoint Patterns – https://www.youtube.com/watch?v=r0NzJjMjhyw
• AWS IoT Greengrass and interface VPC endpoints (AWS PrivateLink) – https://docs.aws.amazon.com/greengrass/v2/developerguide/vpc-interface-endpoints.html