Amazon SageMaker HyperPod introduces enhanced lifecycle scripts debugging
Amazon SageMaker HyperPod now provides enhanced troubleshooting capabilities for lifecycle scripts, making it easier to identify and resolve issues during cluster node provisioning. SageMaker HyperPod helps you provision resilient clusters for running AI/ML workloads and developing state-of-the-art models such as large language models (LLMs), diffusion models, and foundation models (FMs).
When lifecycle scripts encounter issues during cluster creation or node operations, you now receive detailed error messages that include the specific CloudWatch log group and log stream names where you can find execution logs for lifecycle scripts. You can view these error messages by running the DescribeCluster API or by viewing the cluster details page in the SageMaker console. The console also provides a "View lifecycle script logs" button that navigates directly to the relevant CloudWatch log stream, making it easier to locate logs. Additionally, CloudWatch logs for lifecycle scripts now include specific markers to help you track lifecycle script execution progress, including indicators for when the lifecycle script log begins, when scripts are being downloaded, when downloads complete, and when scripts succeed or fail. These markers help you quickly identify where issues occurred during the provisioning process. These enhancements reduce the time required to diagnose and fix lifecycle script failures, helping you get your HyperPod clusters up and running faster.
This feature is available in all AWS Regions where Amazon SageMaker HyperPod is supported. To learn more, see SageMaker HyperPod cluster management in the Amazon SageMaker Developer Guide.