Updating chef-client on AWS OpsWorks for Chef Automate nodes

Throughout the life of your AWS OpsWorks for Chef Automate environments there will be numerous updates and changes to one of the most crucial components – the chef-client. How can you ensure that you have the most up-to-date version of chef-client running on your OpsWorks for Chef Automate infrastructure? Can you make these updates in the “Chef way”? In this article I’ll show you how to make sure that your chef-client is updated on your Chef nodes, and I’ll explain why this is important.

Why should you update your chef-client?

There are multiple reasons why should you make sure that you are running an updated version of chef-client on your nodes. Here are just a few:

·      Security patches (newer version of Ruby, RubyGems, and other dependencies)
·      Avoiding the end of support for previous releases of Chef (support for Chef 12 should end in April 2018)
·      Support for more resources (e.g., windows_path and zypper_repository)
·      Additional properties of existing resources (e.g., remote_user, remote_password, and remote_domain on the remote_file resource)
·      More support for Windows (support for additional features, properties, and resources for Windows nodes)

Things to consider before updating

In the latest version of chef-client new features are being introduced, but at the same time some features are deprecated and/or removed from the current version of chef-client. For example, in chef-client version 13.6.4 the deploy resource is deprecated. It is removed in Chef 14, which was released on April 2nd of this year. You must ensure that the cookbooks running in your environments are not affected by the client version changes because these changes might break your environments’ functionality.

The best place to look for changes in the versions is in the release notes on the documentation page of Chef. For an even more detailed look at the changes, check out the full changelog on GitHub.

Prerequisites – assumptions

For this blog post I assume that you are managing your OpsWorks for Chef Automate server from a workstation using knife from the root directory of your OpsWorks for Chef Automate starter kit. Also, we will be using berkshelf to manage all the cookbooks we need.

This guide was developed on and is intended to work with Chef nodes running Amazon Linux, but the same steps should apply to different Linux distributions. The approach for updating Windows Nodes is sligthly different (eg. No knife ssh commands).

How to update the version of chef client

A brief overview of the procedure
To update the version of our chef-client we will use the chef_client_updater cookbook available from the chef supermarket, and follow these steps.

1.     Provision the cookbook use Berkshelf.
2.     Create a role that has this cookbook attached to its run list.
3.     Assign this role to our nodes.
4.     (Optionally) Trigger a chef-client run on our nodes.

Obtaining the cookbooks
In your Berksfile, located at the root of your OpsWorks for Chef Automate starter kit, append the name of the chef_client_updater so that berkshelf can obtain the cookbook from the Chef Supermarket. Here is how my Berksfile looks:

source "https://supermarket.chef.io"
cookbook "chef-client"
cookbook "chef_client_updater"

Now, to get the cookbook to your workstation and upload it to the OpsWorks for Chef Automate server, run the following command:

berks install
berks upload

This way berkshelf downloads the cookbook with all its dependencies and uploads it to the Chef Automate server.

Creating the role
To make this process as clean as possible, we will create a role for updating the chef-client, add the chef_client_updater cookbook and attach it to the nodes we want to update. This way, we can remove the role from these nodes if we want to prevent further updates.

So, let’s create a role from a JSON file. Here is the JSON file that contains the definition of our role:

{
	"name": "update_chef_client",
	"description": "Role used for updating Chef client",
	"json_class": "Chef::Role",
	"default_attributes": {
		"chef_client_updater": {
			"version": "latest",
			"prevent_downgrade": true
		}
	},
	"chef_type": "role",
	"run_list": ["recipe[chef_client_updater::default]"
	]
}

Note that we have set the value of version to latest, on the chef_client_updater attribute. This lets us have the latest version of chef-client on our nodes. But, if you want to remain at a specific version you can set the version to a version number (e.g., 13.6.0). Additionally, we set the prevent_downgrade to true. This will prevent the downgrade of the client version if there is a newer version already installed on the node.

Save this JSON file in the roles directory as update_chef_client.json.

Then from the root of your starter kit, create the role by using knife:

knife role from file roles/update_chef_client.json

Now we have a role that, when attached to nodes, will allow them to run the chef_client_updater cookbook and update chef-client.

Applying the role to the nodes

There are a few ways to approach assigning a role to our instances. You can do it manually. or you can use knife exec to get multiple nodes based on search criteria. I’ll demonstrate both of these approaches.

Apply a role to a single node
To apply a role (add it to a nodes run_list), use the following knife command:

knife node run_list add i-0c8xxxxxxxx308d45 'role[update_chef_client]'

You can do this for each of your nodes, but if your environment is large, I advise you to follow the next example.

Apply a role to multiple nodes at once
In the following example I use knife exec and Ruby to search for nodes that contain a specific recipe, and then add the role to their run_list. I add the role to all nodes containing the chef-client cookbook recipe in their run_list. In my case, this would reflect all nodes attached to my OpsWorks for Chef Automate server.

knife exec -E 'nodes.find("recipe:chef-client") {|n| puts n.name; n.run_list.add("role[update_chef_client]"); n.save}'

This will loop through all the nodes, print out their node names, and add the role update_chef_client to their run_list.

Now that you have added the role to our nodes, you are good to go. You can either wait for the nodes to check in with the OpsWorks for Chef Automate server (by default the chef-client cookbook sets this to 30 minutes), or you can run chef-client by using knife ssh on all of your nodes.

Run chef-client on all nodes
You can use knife ssh to trigger a Chef run on all nodes that have the update_chef_client role. To do this, use the following command:

knife ssh 'role:update_chef_client' 'sudo chef-client' --ssh-user ec2-user --identity-file ~/.ssh/my_key.pem --attribute cloud.public_hostname

This will trigger parallel SSH connections to all of the nodes that have the role set on them, and run sudo chef-client. This performs the execution of the chef_client_updater cookbook and updates your chef-client.

Note: The chef_client_updater cookbook needs to kill the current running chef-client, during the update procedure. This Chef run will fail, even though the update of the chef-client will be successful. Because of the kill command, it will show in the Chef run log as failed due to SystemExit. This is perfectly normal, and during the next Chef run everything should be okay.

Learn Chef Modules

In addition to this walkthrough, you can check out the Upgrade chef-client Learn Chef Module from Chef. It guides you through upgrading the chef-client on OpsWorks, testing it in a safe environment, and verifying the installed version through InSpec.

Conclusion

In this blog post we demonstrate how you can use the chef_client_updater cookbook to keep your chef-client updated to the latest version. This cookbook allows us to perform updates in a clean and easy manner.

Note that you should always test the chef-client version against your cookbooks, so you don’t run into issues that can break the functionality of your cookbook. Another best practice is to use a specific version (closest to the latest version) that you are certain works with your cookbooks, instead of using latest as the version in your chef_client_updater attribute.

About the Author

Darko Meszaros is a Solutions Architect based in Berlin, Germany. He is a subject matter expert for OpsWorks and CodeDeploy. Outside of work, he loves collecting video games and old computers.

AWS Cloud Operations Blog