All Dask processes, including the scheduler and workers, publish logs to Amazon CloudWatch. These logs will show if a failure has occurred in the system. For a more detailed analysis, users have access to a real-time dashboard application showcasing various metrics such as current worker connections, in-flight processes, and CPU usage. By manually adjusting the desired count of scheduler tasks and worker clusters, you can resolve issues such as a task refresh to unfreeze processes. 

Read the Operational Excellence whitepaper

The principle of least privilege is applied throughout the Guidance, limiting each resource’s access to only what is required for the resource to complete its function. External users would only need SageMaker notebook access, given this is the primary tool to interact with Dask. Additionally, all resources are deployed into private subnets. OpenSearch Service has configured node-to-node encryption and enforces HTTPS connections only. The SageMaker notebook and FSx for Lustre file system (where the data to query sits) both leverage AWS Key Management Service (AWS KMS) to encrypt the underlying storage of each service.

Read the Security whitepaper

By deploying private subnets across different Availability Zones, we decrease the chance of disruption to services. If one Availability Zone fails, the autoscaling group attached to the Amazon ECS cluster would spin up a replacement instance in another Availability Zone. This Guidance contains loosely coupled dependencies throughout each of the Regions. Dask workers connect independently into the client Region’s scheduler and mount to an FSx for Lustre file system on launch. Telemetry from the workers notify the scheduler of failed workers to terminate. Each of the Dask workers have a running “nanny process” which monitors the health of the workers and restarts them if necessary. In this process, the scheduler detects and requests a new worker to be spun up in-place should the worker fail to process its request.

Read the Reliability whitepaper

In this Guidance, we selected Amazon ECS for its ability to execute a long-running Dask container and mount to an FSx for Lustre file system. FSx for Lustre provides a high IOPS service for data access patterns, and regional clusters scale compute based on CPU usage. We use Lambda for short-term processes, such as a daily sync of the FSx for Lustre file system to Amazon S3 or AWS Fargate for the scheduler that doesn’t require mounting to a file system. You can optimize your data by performing on a more relevant metric to scale than CPU usage (such as a published metric from the worker framework detailing when it should scale out) and building it into the Dask framework.

Read the Performance Efficiency whitepaper

Instances follow an on-demand pricing model which charge on an hourly basis. There is a scaling metric on CPU usage attached to the cluster of Dask workers. This metric streams into CloudWatch, a service that provides insight as to whether the existing supply of workers is sufficient or whether additional workers should be provisioned. This visibility helps you use only the minimum resources required and reduce overall costs.

Read the Cost Optimization whitepaper

Dask workers are kept to a minimal amount, configured to 10 workers per Amazon ECS task. As the scheduler receives jobs for that pool of workers, CPU usage of the workers increases. If those workers remain at a high enough CPU threshold, an alert will trigger to scale to more workers. The number of workers only increases as demand increases, helping you keep the amount of resources you use to a minimum.

Read the Sustainability whitepaper