How to centrally manage AWS IoT Greengrass devices using AWS Systems Manager

Remotely managing vast fleets of disparate systems and applications can be a challenging task for edge device administrators. AWS IoT Greengrass can help these system administrators manage their edge device application stack. However, system software on these devices must be updated and maintained separately via operational policies consistent with those of their larger IT organizations. Moreover, customers have had to build or integrate custom tools to ensure that edge devices can be managed alongside  Amazon Elastic Compute Cloud (Amazon EC2)  and on-premises instances through a consistent set of policies.

We released integration between AWS Systems Manager and AWS IOT Greengrass at re:Invent 2021. This release simplified the management and maintenance of system software for edge devices. When coupled with the AWS IoT Greengrass Client Software, AWS Systems Manager lets edge device administrators remotely access and securely manage many devices they own – from OS patching to deployment applications. Administrators can utilize almost all of the Systems Manager service’s operational and node management features.

This post will go through the setup to integrate an AWS IoT Greengrass device with Systems Manager. From there, we’ll demonstrate how to use Session Manager to open remote login to an edge device, patch them using Patch Manager, and run automated scripts through Run Command.

Install AWS IoT Greengrass Core software

First, we must install Greengrass Core software on edge devices. There are multiple options available to install the AWS IoT software, including:

• Automatic provisioning
• Manual provisioning
• Fleet provisioning
• Custom provisioning

Follow the AWS IoT Greengrass documentation to understand the differences among these provisioning methods.

We use the automatic provisioning method to install the Greengrass Core software on the edge device for this post. In this method, the installer provisions required peripherals, such as registering the AWS IoT thing and creating the AWS IoT thing group, the AWS Identity and Access Management (IAM) role, and the AWS IoT role alias. This enabled a quicker set up. Automatic provisioning works on both Linux and Windows devices. We’re using an Ubuntu Linux device for the following steps. Device requirements for running AWS IoT Greengrass Core software can be found in the AWS IoT Greengrass documentation.

Once the Greengrass Core software installs successfully, you should see the Core devices in the AWS IoT Greengrass console with the Healthy status. Refer to the troubleshooting guide if the installation isn’t successful or the device doesn’t show up in the Healthy status

Figure 1: Edge device in Healthy status is visible in the AWS IoT Greengrass Console, under Greengrass, Core devices.

Deploy Systems Manager Agent

Now that the edge device is communicating with AWS IoT Greengrass, we can deploy software components to the device. You can also deploy components to a group of devices with AWS IoT Greengrass. Deployments define the component and its configuration to apply to the device’s existing software, and they’re continuous. AWS IoT Greengrass rolls out the deployment to the online target devices when you create a deployment. If a target device isn’t online, it receives the deployment the next time it connects to AWS IoT Greengrass.

The Systems Manager Agent (SSM Agent) will be deployed as a component through Greengrass to enable the Systems Manager service to update, manage, and configure Greengrass core devices. The agent processes and runs requests from the Systems Manager service in the AWS Cloud. Then, the agent sends status and runtime information back to the Systems Manager service.

Follow the steps in Install the Systems Manager Agent documentation to complete Systems Manager setup steps and deploy the Systems Manager Agent component. The process consists of four stages that are summarized as follows:

1. Complete general Systems Manager setup steps.
2. Create an IAM service role for Systems Manager.
3. Add permissions to the token exchange role.
4. Deploy the Systems Manager Agent component.

The deployment can take several minutes to complete.

After deploying the Systems Manager component, navigate to AWS IoT Greengrass console, choose Core devices, and select your device. On the core device’s details page, find the Systems Manager instance property, which links to the SSM console.

Figure 2: Edge device is now a managed node, as visible from the Systems Manager Console under Fleet Manager.

Edge devices registered to Systems Manager via Greengrass will have a Node ID starting with ‘mi-’. To manage the device using Systems Manager and its capabilities, the SSM Agent ping status for the given node must be Online. You can verify the node’s ‘SSM Agent ping status’ by selecting the Node ID.

Figure 3: Node overview page showing details of the managed node.

Systems Manager refers to an edge device as a managed node. You can now access the managed node, perform patching, and gather monitoring data among other tasks.

1. Open Terminal on the Edge device

Select the ‘Node actions’ dropdown on the Node overview page and select Start terminal session. This action will launch a terminal session on the edge device through Systems Manager Session Manager.

Note: Session Manager requires the advanced instance tier for use to interactively access the edge device since they’re registered with Systems Manager as a hybrid instance. Furthermore, it has additional pricing, which can be reviewed in Systems Manager Pricing. Follow the Systems Manager documentation steps to enable the Advanced instance tier from the console.

Figure 4: Interactive session window opened in browser for the managed node.

Sessions use a secure bidirectional communication channel between the client (you) and the remote managed node that streams inputs and outputs for commands. Traffic between a client and a managed node is encrypted using TLS 1.2, and requests to create the connection are signed using Sigv4. This two-way communication allows interactive bash and PowerShell access to managed nodes.

Furthermore, use an AWS Key Management Service (AWS KMS) to encrypt data beyond the default TLS encryption further. We also have options for logging and auditing session histories in the AWS account through integration with numerous AWS services, such as Amazon Simple Storage Service (Amazon S3), AWS CloudTrail, and Amazon CloudWatch Logs. One option is to configure preferences within sessions, which are customizable and allows for the definition of shell preferences, environment variables, working directories, running multiple commands, etc., when a session is started.

2. Patch the Edge device

Patch Manager, a capability of Systems Manager, provides predefined patch baselines for each supported operating system that typically approves Security-related patches. Utilize these baselines as they are currently configured (we can’t customize them) or create custom patch baselines. Custom patch baselines allow for greater control over which patches are approved or rejected for the environment. For this post, we will use a predefined patch baseline. For more information, refer to the predefined and custom patch baselines documentation.

For patching the edge device:

1. Open the Node overview page of the managed node.
2. From the Node actions dropdown, select Patch node. We’ll see the AWS Systems Manager Patch Manager page to patch the edge device. Read through the options on the page which shows the basic configuration for patching. By default, the patching operation is Scan, which only scans for missing patches and doesn’t install them.
3. Keep all of the defaults, scroll to the bottom of the page, and select Patch now.

Figure 5: Patch now window with default options selected.

4. On the next page, we’ll see the patch execution summary. Select the Execution ID hyperlink, followed by the Output on the page after that, to view the details and results of the scan operation.

The output page contains three steps. Select the step that matches the OS running on the edge device in the step name. For example, if it’s running Linux, expand the output and error of the step PatchLinux. Note that the edge device must be able to reach its respective repositories to pull the patches.

Figure 6: Association Execution Summary window showing details of Patch operation in Scan mode.

Figure 7: The output of the scan operation shows a summary of the managed node’s available patches.

For more information, visit the How Patch Manager operations work documentation.

Since these are edge devices with intermittent connections and limited technical specs, the installation time to complete may vary. Patch Manager uses Run Command (another capability of Systems Manager that we’ll see in the following section) to perform the patching operation on the managed node.

3. Send a command remotely to the Edge device

Use Run Command, a capability of Systems Manager, to configure managed nodes at scale without logging in to them. We gather processes running on the edge device through the Run Command for this post. Go back to the Node overview page, and from the Node actions dropdown, select Execute run command. This method will bring users to the Systems Manager Run Command page. Notice a list of pre-configured command documents that are available. Search for AWS-RunShellScript, and select that document. Our edge device runs on Ubuntu, so we’ll send the command ps aux to get the processes running. Under Command parameters, type ps aux in the Commands text box.

Figure 8: Run a Command window with AWS-RunShellScript document selected and supplied shell command.

Scroll down to target. We should see our node already selected. Under Output options, uncheck ‘Enable an S3 bucket’. For production usage, it’s recommended to write all of the command outputs to an Amazon S3 bucket for audit purposes. We only test to see if the Run command is successful for this post. We also recommend ‘Enable Cloudwatch logs’ for production use for additional auditing purposes.

Select Run. It will bring you to a page that shows the Run command results. Select the radio button next to the Instance ID and select View output to see the command’s output on our target IoT device. This action will bring us to a different page. The select output will show us the result of the ‘ps aux’ command sent to our edge device.

Figure 9: Output of shell command ran through Run Command.

An alternative way to gather running processes on a managed node is to use Systems Manager Fleet Manager. At re:Invent 2021, a new feature was released that offers an easy console-based experience for customers to view and manage processes on their instances. Customers can see granular details, such as process name, process details, and utilization metrics for every active process. They can query processes by name and quickly sort the columns based on any of the process parameters. In addition, you can terminate unwanted processes from the console or start new processes.

Fleet Manager utilizes the Session Manager capability to manage Processes on managed nodes. Furthermore, it requires AWS KMS encryption to be enabled in the Session Manager preferences. Visit the documentation for steps to update Session preferences. Note the AWS KMS key that we selected in the previous step, and open this key in the AWS KMS console.

Previously in this post, you created and deployed the SSM Agent component which would have created an IAM service role. Suppose the AWS IoT Greengrass documentation was followed to deploy the SSM agent. In that case, the name of the IAM service role should be SSMServiceRole. You’ll need to update the encryption keys by following the AWS KMS documentation steps to add SSMServiceRole as one of the Key users.

Open our managed node in the Systems Manager console as described in Section 2 above to see how this works. In the left column, under Node overview, select Processes.

Figure 10: Node details page showing Process and Resource usage information of the managed node.

Since the IoT Greengrass Core software and Systems Manager Agent could control thousands of remote devices, following the AWS Well-Architected security best practice of least privilege is essential. Least privilege is the principle of granting only the permissions required to complete a task. AWS-managed policies that are setup with automatic provisioning can be a good starting place to see the actions typically associated with a particular service or job function. However, they may not fit a company’s compliance standards or specific use cases.

For assistance in writing least privilege IAM policies, refer to this blog.

Cleanup

If you no longer wish to manage your Edge device with Systems Manager, deregister the device and uninstall the Systems Manager Agent from it. Follow the steps from the documentation Uninstall the Systems Manager Agent. Similarly, you can uninstall the Greengrass Core Software to remove it from the Edge device. Find the instructions to stop, disable, and remove the Greengrass service in Uninstall the AWS IoT Greengrass Core software documentation.

Conclusion

This post demonstrated how to install the Systems Manager agent on the edge device as an AWS IoT Greengrass component and utilize various Systems Manager features to manage edge devices alongside Amazon EC2 and on-premises nodes centrally. We began by showing how to install the AWS IoT Greengrass core software on the edge device, as well as how to deploy the SSM Agent component, and then we walked through three device management use cases.

These use-cases included:

• Gaining interactive shell access to edge devices while improving security and audit posture, centralizing access control, and restricting inbound access. Start the session using AWS Console, AWS Command Line Interface (AWS CLI), or one of the supported AWS SDKs. Refer to Console, CLI, and SDK access to Session Manager capabilities for more information.
• Patching the Operating System of the edge device with fine-grained control, over which patches can be installed. Patching can be automated across fleets of devices using a well-defined Maintenance Window, during which it is acceptable for disruptive actions to occur. Visit patching schedules using maintenance windows for more information.
• Execute remote scripts on the edge devices in a controlled manner that allows for safe and secure administration at scale. Install or bootstrap applications using Run Command, track inventory, apply software updates, or modify system settings. Furthermore, thousands of edge devices can be targeted in a single Run Command using tags on managed nodes. More information on Run Command can be found in the AWS Systems Manager Run Command documentation.

Authors: