Artificial Intelligence
Introducing managed cluster extensions for observability and user management in Amazon SageMaker HyperPod Slurm
Keeping a shared Slurm cluster for foundation model development in sync entails keeping several separately maintained pieces of configuration matched to what’s running on the nodes. The cluster’s shape lives on multiple configuration files, its software in custom scripts, its network topology in files that must be edited as nodes change, and its users in a roster that an administrator must maintain and rerun. Over time these pieces drift away from what’s actually running, and nothing tells you when they do.
We believe cluster configurations should be managed the way you manage the rest of your AWS infrastructure. You declare what you want through an API or the console, AWS validates it before it’s applied, the cluster is kept in that state continuously, and you can see whether it’s working. During 2026, Amazon SageMaker HyperPod brought that approach to Slurm clusters, with API-driven Slurm configuration, AMI-based configuration, continuous provisioning, automatic topology management, and deep health checks. Today, we launched cluster extensions, which bring the same approach to the software and settings on running nodes, with managed extensions for observability and user management. However, through all of it, you still keep full access to every node in your cluster, including the controller node. You can SSH in, configure nodes directly, and run your own lifecycle scripts. The APIs give you less to maintain, direct access gives you more control, and you choose the balance.
In this post, we walk through the features we introduced in 2026 and explain why the software on running nodes needed a different mechanism. We then show how cluster extensions work, through both the API and the console, and follow a platform team through a quarter on a cluster that uses them.
A declarative foundation for Slurm cluster operations

We can think of a Slurm cluster’s configuration grouped into six layers. Five of them describe the cluster itself: its definition, node setup, capacity, network topology, and hardware health. The sixth covers the software running on its nodes, which we discuss in the next section. The following figure shows the six layers.
Earlier in 2026, we brought declarative management to the first five layers. With API-driven Slurm configuration, administrators define node roles, partitions, and Amazon FSx for Lustre and Amazon FSx for OpenZFS mounts in CreateCluster and UpdateCluster requests, and connect an external accounting database. HyperPod updates partition settings while preserving other controller-node Slurm customizations. AMI-based configuration replaces routine setup scripts with images preconfigured with Slurm, Docker, Enroot, Pyxis, and home directories. A custom AMI or OnInitComplete script handles organization-specific software. Continuous provisioning for Slurm retries capacity requests in the background; AMI-based configuration works with it, minimum capacity requirements let teams specify MinInstanceCount, and cluster operations track scaling requests.
Automatic topology management maintains the network layout as nodes scale or are replaced. On Slurm 25.11 or later, partition-level topology supports tree or block topology per partition. For hardware health, on-demand deep health checks stress GPUs and test NVIDIA Collective Communications Library (NCCL) connectivity, while deep health checks with continuous provisioning run as nodes are provisioned. Nodes under check stay in Slurm maintenance reservations until checks finish, keeping jobs off them.
Introducing cluster extensions
The five layers described so far are configurations that HyperPod can derive or apply once, when it provisions a node. The sixth layer, the software that runs on the nodes, behaves differently because it keeps changing for the life of the cluster. As researchers join and leave, a team asks for deeper GPU metrics for a week, and an exporter ships a fix. For this layer, lifecycle scripts have been the main mechanism so far, and they have drawbacks when the software keeps changing. A script runs only when a node is provisioned, so each change means editing it and then replacing nodes or rerunning it across the fleet. A mistake shows only when the script runs on a node, and nothing reports what’s installed, so nodes drift apart over time.
We wanted the software to be declared in one place, checked when you submit it, kept consistent across nodes, and visible afterward. In November 2026, we launched cluster extensions, which are software packages managed by AWS that you install, configure, update, and remove through APIs or the Amazon SageMaker AI console, starting with observability and user management, including directory integration, as the first software extensions. Compared with scripts, extensions have these advantages:
- Validated up front: HyperPod checks an extension’s settings against its schema when you submit them, through the API or the console, so a misspelled setting is rejected before anything reaches a node.
- Consistent on every node: HyperPod applies the extension to every node, including nodes added by scale-up or replacement, and keeps the nodes consistent as the cluster scales and heals.
- Changed through the API or the console: You change a setting or upgrade a version with an API call, or from the SageMaker AI console, instead of editing scripts and replacing nodes. Both paths use the same validated configuration, and this post shows console screenshots alongside the API examples.
- Visible: Each extension reports its status and per-node health.
Use cluster extensions when HyperPod offers a managed software capability. Use custom AMIs and lifecycle scripts for organization-specific software or initialization that the available extensions don’t cover. HyperPod supports two cluster managed extensions today:
- The observability extension (sagemaker-observability) collects node, GPU, Elastic Fabric Adapter (EFA), Slurm job, and Slurm cluster metrics and sends them to Amazon CloudWatch or Amazon Managed Service for Prometheus.
- The user management extension (sagemaker-user-management) creates users and groups with the same IDs on every node, controls who can use SSH and sudo on each node type, and registers users with Slurm accounting. It can also connect the cluster to Active Directory, for example AWS Directory Service for Microsoft Active Directory.
The following diagram shows how an extension reaches your nodes.
Figure 2: HyperPod can install, edit, or remove an extension. HyperPod records the desired configuration, a node agent applies the extension package on every node, including new and replaced ones, and health probes report status and health issues back
Using cluster extensions
You can use cluster extensions from the SageMaker AI console or through an API. On the console, the Extensions tab on a cluster’s details page lists each extension with its status, version, and ARN, and it’s where you install a new one. Each extension adds controls of its own, such as the metric store and an open dashboard in Grafana button for observability, and the number of users and groups and a Manage users button for user management. You can also manage every extension through APIs, from the AWS CLI or an SDK, to script installs, keep settings in version control, repeat a setup across clusters, or check status and health from your own tooling. The following table lists the six cluster extension APIs, from installing an extension to listing the versions you can upgrade to, along with console view for extensions.
Figure 3: Left, the six cluster extension APIs and what each one does. Right, the console view to add and remove extensions
The following diagram shows what happens after you call one of these APIs.
Figure 4: You declare an extension’s version and settings through the API. HyperPod validates and records them, applies them to every node for its role, and reports status and health
When you create or update an extension, HyperPod validates your settings against the version’s schema during the API call, so a misspelled setting is rejected immediately. It then records the version and settings as the desired state and applies them, with each node installing only what its role needs, such as GPU exporters on GPU nodes only. New and replacement nodes get the same state a few minutes after they start, and every node reports whether the extension applied and whether it’s healthy.
Updates send only the settings that you want to change, and HyperPod merges them into the stored settings. Problems that show up only on a node, such as a port or user ID that something else already uses, appear in the Health field of DescribeClusterExtension, grouped by cause with a code, a message, and a count of affected nodes. HyperPod retries internal failures on its own, and for anything else you fix the cause and call UpdateClusterExtension with no changes to retry the affected nodes.
An extension’s status is Creating or Updating while a change rolls out, Active when every node runs the requested version and settings and none reports a problem, and Degraded when some nodes failed or report a problem. Each extension has its own ARN, so a platform team can own sagemaker-user-management while a monitoring team owns sagemaker-observability. Each health issue names the component, describes the effect on your cluster, gives the next step, and counts the affected nodes, so you can see where to look without signing in to each node, as the following figure shows.
When to use cluster extensions and when to use lifecycle scripts
A useful way to decide is to ask who owns the software and how often it changes. Some of the software on a cluster is specific to your organization, such as security agents, license servers, and the storage layout your data pipeline expects. Lifecycle scripts and custom AMIs are the right tools for it, because only your team knows what it needs. The rest is software that almost every cluster needs, like GPU and job metrics and user accounts with consistent IDs and access rules. That software must match on every node and keeps changing, so cluster extensions are the better fit, because you declare it once, HyperPod validates it, and each extension reports whether it’s healthy.
The two approaches can coexist, but each component should have one owner. Don’t manage the same exporter, user, or service through both a lifecycle script and an extension, because competing configurations can cause installation or runtime conflicts. Cluster extensions work with every lifecycle configuration option.
If you already run these capabilities from scripts, you can move to extensions one capability at a time, and scripts and extensions can run side by side while you do. For each capability, the migration takes four steps:
- Call UpdateClusterSoftware to move the cluster to the November 2026 HyperPod AMI or later.
- Turn the capability off in your lifecycle scripts and upload them to your S3 bucket, so that new nodes skip it.
- During a planned maintenance window, stop and remove the script-installed agents on every node. This is the only manual, fleet-wide step.
- Call CreateClusterExtension to install the extension on the clean cluster.
If anything is left behind, the install names the conflict, such as a port that’s already in use, and DescribeClusterExtension shows it under Health. With user management, you keep the same usernames and user IDs. The following figure shows four steps.
What you need to get started with cluster extensions
You need a HyperPod Slurm cluster on or after the November 2, 2026 HyperPod AMI or later with continuous provisioning enabled. An extension runs software as root on every node, so treat CreateClusterExtension, UpdateClusterExtension, and DeleteClusterExtension as administrator permissions. We suggest that you do not grant them to the cluster’s execution role, because every user who can sign in to the nodes can use its credentials. That role does need access to the destinations that the extensions use, which means Amazon CloudWatch or aps:RemoteWrite for observability and secretsmanager:GetSecretValue for the Active Directory secrets that user management reads. If those secrets are encrypted with a customer managed AWS Key Management Service (AWS KMS) key, the role also needs kms:Decrypt on that key. User management also needs a shared file system, such as Amazon FSx for Lustre, for home directories.
For procedures, IAM policies, and configuration references, see Cluster extensions for Amazon SageMaker HyperPod Slurm clusters in the Amazon SageMaker AI Developer Guide.
A platform team’s journey managing a shared Slurm cluster using cluster extensions
To see how cluster extensions are used in practice, consider a platform team that manages a shared Slurm cluster for several research groups. The team needs a reviewable cluster definition, GPU and job metrics, corporate directory access, local accounts for a partner lab, and separate ownership of observability and user management. The following examples show how the team configures these capabilities and responds to routine changes and failures.
Create the cluster from one definition
The administrator declares the whole cluster, including node roles, partitions, FSx mounts, minimum capacity, deep health checks, and the external accounting database, in a single CreateCluster request. For a complete example request, see Getting started with SageMaker HyperPod using the AWS CLI.
The request has no lifecycle configuration, topology file, or Slurm configuration file. Each SlurmConfig gives its instance group a role, and the GPU group joins the train partition. AMI-based configuration sets up the nodes, and the security team’s compliance agent is already in the custom AMI. The GPU group asks for 8 nodes and requires 6, so jobs can start as soon as 6 are ready while HyperPod provisions the rest and health checks each GPU node as it arrives. HyperPod selects tree topology for the P5 nodes on its own, and accounting writes to an Amazon Relational Database Service (Amazon RDS) database that the team owns. The administrator can still SSH into the controller node to inspect or tune Slurm directly.
Install observability
Leadership wants GPU and job metrics, so the administrator installs the observability extension. In the SageMaker AI console, on the cluster’s Extensions tab, the administrator chooses the observability extension, picks the metric categories and their levels and the collection interval, and selects one of two metric stores: Amazon CloudWatch or Amazon Managed Service for Prometheus. The following screenshot shows the install form with one store selected. The other option is available in the same field.
Figure 7: Installing the observability extension from the console, with configurable metric levels and a choice of metric store
The same installation through the API takes one create-cluster-extension call. The following example uses Amazon CloudWatch as the metric store. To send metrics to Amazon Managed Service for Prometheus instead, you set the Prometheus destination in the same configuration.
The administrator installs the extension with one create-cluster-extension call:
The call returns the extension’s ARN, which the administrator uses in later calls:
Whichever way it’s installed, each node runs only the components relevant to its role, such as DCGM and EFA exporters on GPU nodes and Slurm cluster metrics on the controller. Schema errors are rejected before rollout, while node-specific conflicts appear in extension health.
After the install, the extension appears on the cluster’s Extensions tab with its version, status, and ARN. The screenshot shows the tab for a cluster that has both the observability and user management extensions installed, and both are Active.
Figure 8: The Extensions tab for a cluster with both the observability and user management extensions installed
Whichever metric store you choose, you can add an Amazon Managed Grafana workspace to the installation, so the team doesn’t need to build dashboards from scratch. After signing in to Grafana, they find two pre-created dashboards, Clusters and Jobs, as shown in the following screenshot.
Figure 9: The Clusters and Jobs dashboards that come with the observability extension in Amazon Managed Grafana
The Clusters dashboard gives a single view of your HyperPod Slurm clusters, built on the metrics that the extension collects and sends to your chosen metric store. At the top, it summarizes cluster health: instance count and health, GPU and GPU memory totals, vCPUs, memory, and Amazon FSx for Lustre capacity. It also shows the state of Slurm itself, including node states (idle, allocated, mixed, down, and draining), job queues, failed or preempted jobs, and scheduler activity. From there, you can filter by cluster, node, or GPU to drill into EFA networking, GPU performance and errors, and host-level CPU, memory, disk, and network metrics, as shown in the following screenshots. That lets you go from “is my cluster healthy?” to the root cause of an issue without leaving the dashboard.
Figure 10: The Clusters dashboard, from cluster-wide health and Slurm state down to per-node GPU, EFA, and host metrics
The Jobs dashboard moves from cluster-level health down to individual Slurm jobs. It ties each job’s resource to use back to the job itself, using metrics from the Slurm job exporter in either metric store. You can filter by cluster, job ID, job name, partition, user, or account, and see CPU, memory, GPU utilization, GPU memory, and disk I/O for each job, plus accelerator and NCCL metrics. That makes it easier to spot underused GPUs, stuck jobs, or communication bottlenecks in distributed training, as in the following screenshot.
Turn on deeper GPU and NCCL metrics
Later, a research team that is tuning multi-node training asks for more detail, so the administrator sends only the settings to change:
HyperPod merges these settings into the stored configuration and applies them to every node, restarting only the components whose settings changed. When the investigation is over, setting both to null returns them to their defaults, and at no point are nodes replaced or scripts rerun.
Fix a metrics delivery problem
One morning the extension shows as Degraded. A new instance group has an execution role that lacks aps:RemoteWrite, and describe-cluster-extension shows the effect:
Once the administrator adds the permission, the nodes report healthy and the extension returns to Active, with nothing to reinstall.
Give people access
The user management extension manages the Linux users and groups on every node, and who can sign in and use sudo on each node type. You declare who should exist and what they can do, and HyperPod applies it to every node, including nodes added later or replaced, so no one connects to nodes or runs scripts to add or remove a user. Users come from local accounts that you list in the configuration, from Active Directory over LDAPS, or from both, and each group names the node types (controller, login, or compute) where its members can sign in over SSH and where they get passwordless sudo.
For the platform team, security wants corporate identities, with sudo reserved for administrators, and a partner lab that isn’t in the directory needs accounts of its own. The administrator declares both sources in one configuration, with group rules that say which node types each group can sign in to and use sudo on:
Access is explicit, so after the extension is installed only the groups that grant SSH access, along with the default ubuntu user, can sign in. Directory users keep their identities and group membership in Active Directory, while the partner lab’s users get the same IDs, home directories, and the default Slurm root account.
The same configuration can be managed from the console. Choosing Manage users on the user management extension opens the cluster’s Users tab, where the administrator can add, edit, and remove groups and users, and set which node types each group can sign in to and use sudo on, as shown in the following screenshot.
Figure 12: Managing groups and users for the user management extension on the console
Upgrade an extension to get new features without AMI updates
When AWS publishes a new version of an extension, the administrator lists the versions that can be installed and upgrades in place without AMI updates:
HyperPod checks the stored settings against the new version’s schema and rejects the request, changing nothing, if they aren’t valid for it. Otherwise, the new version rolls out to every node, and nodes that join later get it too. Upgrades only go forward, so the administrator picks a maintenance window. Any new settings or capabilities that come with the version can then be turned on with an update like the earlier one.
Scale the cluster and replace failed nodes
Before the end-of-quarter run, the administrator starts an on-demand deep health check on the GPU group and then adds nodes with batch-add-cluster-nodes. The request returns as soon as HyperPod accepts it, so the administrator follows it as a cluster operation:
As capacity arrives, each node is health checked, joins Slurm, takes its place in the topology, and gets both extensions with the same version and settings as every other node. If a GPU fails during the run, the replacement is configured and picks up the extensions the same way.
Review who has access
At the end of the quarter, security asks who has sudo on the controller, and the administrator answers from the API without signing in to a node:
The answer is cluster-admins and ClusterAdmin@example.com, and the directory group’s membership is audited in the directory. Because each extension has its own ARN, the platform team’s role can update only sagemaker-user-management and the monitoring team’s role only sagemaker-observability, while security has read-only access to both. None of these permissions belongs on the cluster’s execution role. When an extension is Active, ActiveCount equals TotalCount, and Health is empty, every node runs exactly that configuration.
Availability and pricing
Cluster extensions, including the observability and user management extensions, have been available since November 2026 in all the regions where Amazon SageMaker HyperPod is supported, and the other improvements in this post were announced earlier in 2026, with each announcement listing its Regions. There’s no additional charge for cluster extensions. You pay for the AWS resources that you use with them, such as Amazon CloudWatch, Amazon Managed Service for Prometheus, and AWS Secrets Manager.
Conclusion
A cluster is easier to run when more of it is declared, validated, converged, and observable. This year’s improvements brought that to a HyperPod Slurm cluster’s definition, node setup, capacity, network topology, and hardware health, and cluster extensions, available since November 2026, bring it to the software on the nodes, the layer that changes most often. Lifecycle scripts and custom AMIs stay focused on what’s specific to your organization, and the controller node stays yours.
To create a cluster as described in this post, see Getting started with SageMaker HyperPod using the AWS CLI. To get started with cluster extensions, see Cluster Extensions in the Amazon SageMaker AI Developer Guide. Share your feedback in the comments or through your AWS account team.
Acknowledgment
This feature has contributions across several teams. Some of the people involved in developing this include Hengyu Lou, Aman Pratap Singh, Karthick Ravi, Shubh Nisar, Nitish Sai Kommasai, Uday Singamala, Min Badar, Devanshi Desai, Yahya Hassan, Connie Chen, Kashish Tayal, Oleg Talalov, Takuma Yoshitami, Kiran Penta, Dhawal Parkar, Pradeep Cruz.



