Using SaltStack to Run Commands in Parallel on Amazon EMR
Miguel Tormo is a Big Data Support Engineer in AWS Premium Support
Amazon EMR provides a managed Hadoop framework that makes it easy, fast, and cost-effective to process vast amounts of data across dynamically scalable Amazon EC2 instances. Amazon EMR defines three types of nodes: master node, core nodes, and task nodes.
It’s common to run commands on each node by using SSH agent forwarding and running a loop on the master node to connect through SSH to every core or task node. However, there are cases in which you might want to run commands on select nodes only (for example, to generate a report on a particular instance type). For this reason, it helps to have an alternative approach for automating command execution on Amazon EMR clusters.
SaltStack is an open source project for automation and configuration management. It started as a remote execution engine designed to scale to many machines while delivering high-speed execution. Saltstack uses its own protocol, which is based on the ZeroMQ library.
SaltStack bootstrap action
You can use the new bootstrap action that installs SaltStack on Amazon EMR. It provides a basic configuration that enables selective targeting of the nodes based on instance roles, instance groups, and other parameters. Even if an instance group gets resized, each new node will execute the bootstrap action that installs SaltStack and registers the node with the master.
After your Amazon EMR cluster is up and running, and SaltStack is successfully deployed, you can now use the SaltStack CLI to configure and run commands on your cluster nodes.
Here are some examples of salt commands:
To check connectivity to all registered nodes
sudo salt '*' test.ping
To restart the YARN NodeManager on every task instance
sudo salt -N task service.restart hadoop-yarn-nodemanager
To run a report of a YARN queue
If SaltStack is installed in external or syndicated mode, you can print a report of the status of the default YARN queue (for example, for every registered EMR cluster running EMR version 4.7.2):
sudo salt -C 'G@emr:version:4.7.2 and G@emr:instance_role:master' cmd.run 'yarn queue -status default'
To execute a script located on the master
This command will execute the script located in the path /srv/salt/myscript of the SaltStack master on all nodes in the ig-FFFFFFFFFFFF instance group:
sudo salt -G 'emr:instance_group_id:ig-FFFFFFFFFFFF' cmd.script salt://myscript
You can find the bootstrap action with instructions and more examples here:
Happy salting! If you have a question or suggestion, please leave a comment below.