Introduction to quickstart-linux-utilities

To properly signal AWS CloudFormation during bootstrapping, CloudFormation requires some functions (qs_update-os, qs_bootstrap_pip, and qs_aws-cfn-bootstrap) to be installed on your Linux Amazon EC2 instances. The bootstrap helper scripts, cfn-signal and cfn-init, assist with common operations like package installation, service management, CloudFormation signaling, and metadata retrieval. Such tools allow you to standardize your installation and reduce maintenance overhead. The code I provide in this post allows you to easily invoke the latest version of this utility at stack launch. Note that AWS conveniently provides a Linux distribution called Amazon Linux, which has this utility built-in.

Quick Start Linux utilities provide a simple and easy way to automate the installation of AWS CloudFormation tools across common Linux distributions. Examples include the Red Hat OpenShift, Corda Blockchain, and HashiCorp Consul Quick Starts. Depending on which distribution is running your Amazon EC2 instance, use package managers, such as yum/dnf, apt/apt-get, or zypper, to install the aws-cfn-bootstrap tools. This can be accomplished by running the OS-specific install commands via UserData. Note that installing these tools requires dependency handling to be part of your code, which can bloat your UserData. After installing this utility, you can call wrapper commands, such as qs_aws-cfn-bootstrap, that install the prerequisite Python pip (if needed) before installing aws-cfn-bootstrap-latest.

If you plan to support different Linux distributions in your CloudFormation template, additional logic is needed, but the extra logic will also bulk up your UserData. This blog post details an approach of using quickstart-linux-utilities to offload tedious tasks and keep your UserData streamlined.

This utility supports the following Linux distributions.

Amazon Linux
CentOS 7
Fedora
Red Hat Enterprise Linux 7
Ubuntu (16.04)
Ubuntu (18.04)—coming soon

How to make quickstart-linux-utils available in UserData

Git-based method (recommended)

Depending on your OS, you may need to replace yum/dnf with apt-get/apt or zypper. Alternatively, you can use curl or wget to pull the files directly from GitHub using the raw URL. If you forked the repo update, the following commands will invoke your forked URLs.

Red Hat
```
yum install -y git
```
Debian
```
apt-get install git
```

Clone the source

git clone https://github.com/aws-quickstart/quickstart-linux-utilities.git

Loading the utility in UserData

To make the qs utilities available during bootstrapping, load the main script, source quickstart-linux-utilities/quickstart-cfn-tools.source. When the tool is loaded, it simply runs qs_aws-cfn-bootstrap to install the AWS CloudFormation helper.

The qs_aws-cfn-bootstrap function determines the OS type and then issues the following commands.

Install Python and its dependencies.
Install pip and its dependencies.
Install cfn-init.

The following section includes code snippets that can be cut and pasted into the UserData section on any supported Linux distribution.

YAML

UserData: !Base64
  Fn::Sub:
    - |
      #!/bin/bash -x
      until git clone https://github.com/aws-quickstart/quickstart-linux-utilities.git; do echo "Retrying"; done
      cd /quickstart-linux-utilities
      source quickstart-cfn-tools.source
      qs_update-os || qs_err
      qs_bootstrap_pip || qs_err
      qs_aws-cfn-bootstrap || qs_err

JSON

{
    "UserData": {
        "Fn::Base64": {
            "Fn::Sub": [
                "#!/bin/bash -x\n",
                "until git clone https://github.com/aws-quickstart/quickstart-linux-utilities.git; do echo \"Retrying\"; done\n",
                "cd /quickstart-linux-utilities \n",
                "source quickstart-cfn-tools.source\n",
                "qs_update-os || qs_err\n",
                "qs_bootstrap_pip || qs_err\n",
                "qs_aws-cfn-bootstrap || qs_err\n"
            ]
        }
    }
}

The latest snippets of code can be found on GitHub (YAML, JSON). The previous snippets use two other Quick Start utility functions to satisfy dependencies for the CloudFormation helper scripts.

qs_update-os updates the OS with the latest patches.
qs_bootstrap_pip installs Python and pip.

The code qs_bootstrap_pip uses qs_get-ostype, qs_get-osversion, and qs_get-python-path, which are commands that can be called during other bootstrapping tasks.

When it’s complete, cfn-init cfn-signal, cfn-get-metadata, and cfn-hup become available for use in UserData. For more information on cfn helper scripts, see CloudFormation helper scripts.

Here are some examples of quickstart-linux-utilities that have been implemented.

Contribution and Support

If you want to report bugs or contribute support for other operating systems, you can use the repo issues register.

Conclusion

Using quickstart-linux-utilities reduces development time when implementing UserData scripts via CloudFormation templates. Additionally, it provides consistency and maintainability by providing a single location where these helper utilities can be patched and enhanced as needed.

For additional reference, see the examples in Red Hat OpenShift, R3 Corda Blockchain, and HashiCorp Consul Quick Starts. If you have any feedback about this post, please let us know in the comments.

Integration & Automation