AWS Cloud Operations Blog

Applying customizations automatically in AWS OpsWorks for Chef Automate

During mandatory maintenance, AWS OpsWorks for Chef Automate performs Chef Server and Chef Automate version, security, and operating system patching. Version updates for Chef go through extensive testing to verify they are production ready and don’t disrupt existing environments.

When an instance is replaced, customers often ask how they can restore their custom settings in a programmatic manner after the maintenance is completed. This blog post explores a solution that uses AWS Lambda and AWS Systems Manager to allow you to automatically re-apply the configurations after the maintenance period.

Overview

This solution creates all the necessary resources to run a Systems Manager document after the server has completed the maintenance. It is launched using AWS CloudFormation and creates an Amazon S3 trigger for a Lambda function, which will be the core of the solution, a Systems Manager Document to run the commands in the OpsWorks for Chef Automate instance, an Amazon CloudWatch Event to trigger the Lambda function on a schedule, an Amazon SNS topic for notifications and all the necessary AWS Identity and Access Management (IAM) permissions.

Workflow

When the OpsWorks for Chef Automate maintenance starts, the Lambda function will be invoked via the Amazon S3 trigger. This function will first configure the CloudWatch event to invoke the function every minute after a 20 minute window has passed.

After the server maintenance is complete, and it is in a healthy state, the Lambda function will execute a Systems Manager Run Command with the OpsWorks for Chef Automate instance as the target. This command will download a zip file that includes a Chef recipe to set the custom configurations.

After this is done, the Lambda function disables the CloudWatch event to prevent any unnecessary invocations. Additionally, if there is an issue with maintenance, an SNS topic is used to send you a notification.

Under the hood

Here’s the sequence of events that occur when maintenance is triggered, whether manually or using OpsWorks for Chef Automate Maintenance Windows.

  • 1. An object is placed in the Amazon S3 bucket.
  • 2. The event triggers the Lambda function.
  • 3. Lambda modifies the CloudWatch Event to run every minute after a 20 minute window and enables it.
  • 4. AWS Lambda verifies the state of the OpsWorks for Chef Automate server.
  • 5. After 20 minutes, the CloudWatch Event triggers the Lambda function.
  • 6. If the server is in the “Healthy” state, AWS Lambda executes a Systems Manager Run Command against the instance.
  • 6a. If the server is in any other state, AWS Lambda publishes a message to the SNS topic and disables the CloudWatch Event.
  • 6b. SNS sends an email to notify you of the state of the server.
  • 7. Systems Manager downloads the zip archive, extracts it into the /tmp/chow/ directory and executes the recipe using “chef-apply.”

Set up

Before you can actually launch the CloudFormation template, you will need to write and upload a recipe to an S3 bucket. In addition, you will also need to gather the name of the OpsWorks for Chef Automate server and the name of the S3 bucket created by OpsWorks for Chef Automate to be used in the CloudFormation stack.

To simulate a change to the Chef Automate configuration, include a comment in the /etc/delivery/delivery.rb file. Make a copy of the file from the Chef Automate instance and save it on your workstation, adding a line to the top: “#Hello AWS!”

For the recipe, use the remote_file resource to copy the delivery.rb file that was modified into the correct location, followed by the execute resource to reconfigure Chef Automate:

execute 'reconfigure' do
  command 'delivery-ctl reconfigure'
  action :nothing
end

remote_file 'delivery.rb' do
  source 'file:///tmp/chow/delivery.rb'
  path '/etc/delivery/delivery.rb'
  owner 'root'
  group 'root'
  mode '640'
  notifies :run, 'execute[reconfigure]', :immediately
end

This recipe will copy the delivery.rb file located in the temporary directory /tmp/chow/ to the location of the Chef Automate configuration /etc/delivery/delivery.rb. After this action is concluded it notifies the reconfigure resource to execute the delivery-ctl reconfigure command and update the configuration.

With both “delivery.rb” and “recipe.rb” created, go ahead and create a new zip archive with the two files at top level, and upload it to an S3 bucket in the same AWS Region as your Chef Automate server.

Launching the stack

Now that all supporting files are configured, launch the CloudFormation stack from the template that you can download here.

This template has 5 parameters, 4 of which are required before you can launch it:

ServerName: The name of your OpsWorks for Chef automate server.

OWCABucketName: This is the bucket that is created by OpsWorks for Chef Automate. It contains the server backups and Systems Manager logs. You can find this bucket in your Amazon S3 console. It will have the naming scheme “aws-opsworks-cm-SERVER-NAME-abcd123456” where “abcd123456” is a random string of numbers, and “SERVER-NAME” is the name of your Chef Automate server. This bucket will be in the same Region as your Chef Automate server.

RecipeBucketName: The name of the bucket where you uploaded the zip file.

RecipeKeyName: The key of the zip file. If you upload it to the root of the bucket it will just be the name of the object. However, if you upload it to a folder within the bucket, the key will follow this naming scheme: “folder/sub-folder/object.zip”.

NotificationEmail: An email address to receive notifications if there is an issue with maintenance. This parameter is optional, so if you do not want to receive notifications, you can leave this empty.

After you have all the parameters filled out, go ahead and launch the stack. It will take a few minutes to create all the resources.

After the stack resources have finished being provisioned and the stack is in the “CREATE_COMPLETE” state, you can test it out. To do this just manually trigger maintenance in the Chef Automate server using the AWS CLI to run the command:

aws opsworks-cm start-maintenance --server-name $Server-Name --region $Region

This process will take roughly 20 minutes to complete while a backup is created and the Chef Automate instance replaced. You can verify that the Lambda function “CHOW” is being invoked by looking through the CloudWatch Logs for the function.
When the maintenance is complete and the server back in the “Healthy” state, you can use SSH to connect to it and open the /etc/delivery/delivery.rb file to verify that all changes were applied!

Conclusion

This blog post showed how you can execute a chef recipe after every maintenance period. The solution is portable, because it is backed by CloudFormation. You can use this to apply the customizations you want to your server, and ensure they are reapplied automatically after the maintenance is completed.

The change performed was a simple one —replace a file— but you can use any Chef resource to perform additional modifications, such as installing monitoring software, adding additional files, or anything else you need!

About the Author

Vasco Cardoso is a Cloud Support Engineer who supports customers using DevOps technologies, specializing in configuration management and continuous integration and deployment. Outside of work he enjoys riding his motorcycle and playing video and board games.