AWS Cloud Operations Blog

How to execute Chef recipes using AWS Systems Manager

It’s exciting to see how many AWS customers are taking advantage of AWS Systems Manager to manage and deploy infrastructure configuration at scale. I have previously blogged about the benefits of using AWS Systems Manager with configuration management tools, including Ansible and Salt. Recent improvements to the configuration management functionality, has made the service even more powerful and flexible for customers. Today, I’m excited to introduce a new AWS Systems Manager document that helps customers configure their instances at scale, using Chef recipes.

Introduction

Chef is a popular configuration management tool used to configure Amazon Elastic Compute Cloud (Amazon EC2) and on-premises instances. While customers have been able to use Chef recipes in AWS OpsWorks for years, we got feedback that customers wanted similar capabilities within AWS Systems Manager. To address this demand, we recently launched a new Run Command document, named AWS-ApplyChefRecipes, that enables customers to apply Chef recipes to instances. At the same time, taking advantage of all the AWS Systems Manager benefits, such as velocity controls and compliance reporting.

In this blog post, I explain the features of the AWS-ApplyChefRecipes document and walk you through an example of how to use it to configure an Amazon EC2 instance, using open-sourced Chef recipes.

We are not going to dive deep into how Chef works in this article. We assume you have some basic knowledge of Chef and experience with related tools, such as the ChefDK. However, as quick overview, Chef is a tool that simplifies the task of managing configuration across a large number of servers or instances, as we call them in AWS. It does this by allowing customers to define desired state using a construct called a recipe. Chef recipes are a collection of resources and instructions, that help achieve a desired configuration state of an instance. This can involve installing software packages, copying files, etc. Recipes are grouped into a set, called a cookbook. For more details, visit Chef’s web documentation.

AWS Systems Manager is an ideal platform to manage the distribution and execution of Chef recipes across your instances. It augments Chef’s native functionality, with powerful features such as scheduled execution, rate control, compliance reports, and enhanced logging capabilities.

Walkthrough

Let’s walk through an example of how to use the new AWS-ApplyChefRecipes document using Run Command.

Let’s say that I want to automate installation of the ntp package across a large set of Linux instances. Instead of writing a cookbook from scratch to achieve this, I can check Chef’s Supermarket platform to see if there is a pre-existing cookbook I can use. The Chef supermarket is a central repository of community-maintained Chef cookbooks, and an excellent resource to find reusable cookbooks for use in production or as a starting point. You can search Chef supermarket using their console, or using the following command with knife.

knife supermarket search ntp

That command returns a list of cookbooks that match the string ntp. One of the results is the one I’m looking for:

ntp:
cookbook: https://supermarket.chef.io/api/v1/cookbooks/ntp
cookbook_description: Installs and configures ntp as a client or server
cookbook_maintainer: chef
cookbook_name: ntp

I can now download that cookbook to my computer with the following command.

knife supermarket download ntp

The command downloads the cookbook as a compressed tar.gz file. I can now extract the file, examine its content, and even make changes, if necessary. For the purposes of this example, I deploy the cookbook as is.

The AWS-ApplyChefRecipes document makes it easy to execute cookbooks stored on GitHub or Amazon Simple Storage Service (Amazon S3). First, I’m going to upload the compressed cookbook to Amazon S3. Now that my cookbook is ready in an accessible S3 bucket, let’s walk through the process of executing it using AWS Systems Manager’s Run Command feature.

  1. Log in to the AWS Management Console
  2. Go to the AWS Systems Manager Console
  3. Click on the Run Command tab in the left navigation bar
  4. Click on the orange “Run Command” button on the right side of the screen
  5. Find the AWS-ApplyChefRecipes document in the list and click to select it
  6. Click the “Run Command” button

The next screen enables you to customize parameters made available by the AWS-ApplyChefRecipes document. Let’s explain what these parameters are and how to use them.

  • Source Type: This parameter specifies where your cookbook files are stored. AWS-ApplyChefRecipes uses the aws:downloadcontent plugin for AWS Systems Manager that supports both GitHub and S3.
  • Source info: In order to download your cookbooks, the aws:downloadcontent plugin requires structured JSON information about where to retrieve them. You can find more details on how the aws:downloadcontent plugin works and source info examples here.
  • Run list: This is a list of the Chef recipes that will be applied for example: recipe[cookbook_name1::recipe_name]. You can find some good information about run lists here.
  • JSON Attributes Content: JSON Attributes to pass to the Chef client, which applies your recipes.
  • Chef Client version: This is the Chef version that will be installed and executed in the target instance. You can direct the AWS Systems Manager document to automatically install the version of Chef client specified on the target instance. If you specify None, the document will not install Chef before trying to execute the recipe.
  • Chef Client Arguments: Extra arguments to be passed to the Chef client when recipes are applied. For more information about available arguments, run chef-client -h from an instance where it is installed.
  • Why Run: When set to true, Chef will show what would happen if the recipes were executed, but will not actually perform any changes.
  • Compliance Severity: Defines the severity level to apply in AWS Systems Manager compliance reporting, rendered when instance configuration does not match recipes specified in the run list, during a run or why-run. To turn off compliance reporting, select “None”. For more information on AWS Systems Manager Compliance reporting, please visit this link.
  • Compliance Type: The AWS Systems Manager compliance type to use when reporting compliance results. For more information on AWS Systems Manager Compliance reporting, please visit this link.
  • Compliance Report Bucket: The name of an existing Amazon S3 bucket in which to store details about every Chef run executed by this document. This includes resource configuration and compliance results. Make sure the AWS Identity and Access Management (IAM) Role used to execute the Run Command, has write privileges to this bucket.

For the purposes of our test, lets setup the parameters as follows:

  • Chef Client version : 14
  • Run list: recipe[ntp::default]
  • Source info (replace with your bucket name) : {“path”: “https://your-s3-bucket.s3.amazonaws.com/ntp-3.7.0.tar.gz”}
  • Why Run: False
  • Compliance Type: Custom:Chef
  • Source Type: S3
  • Compliance Severity: None
  • JSON Attributes Content: Blank
  • Compliance Report Bucket: Blank
  • Chef Client Arguments: Blank

Now, under targets, select an instance to test. Make sure that the instance meets AWS System Manager prerequisites . Also ensure the you have an IAM instance profile for AWS Systems Manager.

Now, click the orange “Run” button to execute the Chef recipe on the target instance. This takes you to the command status page seen here.

 

After a few minutes, the execution completes and the status will change to Successful to indicate is completed.

Conclusion

As you can see, it is now simple to apply Chef cookbooks and recipes using Run Command. You can also leverage AWS Systems Manager State Manager to enforce periodic execution or evaluation of compliance against Chef recipes.

AWS Systems Manager State Manager is a secure and scalable configuration management service, that automates the process of keeping your Amazon EC2 and hybrid infrastructure in a state that you define. You can use Chef recipes and this new document to define that state. Click on the links to learn more about Systems Manager Run Command or State Manager.

About the Author

Andres Silva is a Principal Technical Account Manager for AWS Enterprise Support. He has been working with AWS technology for more than 9 years. Andres works with Enterprise customers to design, implement and support complex cloud infrastructures. When he is not building cloud automation he enjoys skateboarding with his 2 kids.