Automate Ansible playbook deployment with Amazon EC2 and GitHub

Ansible is an open-source automation tool that uses playbooks to enable you to make deployments faster and scale to various environments. Think of playbooks as recipes that lay out the steps needed to deploy policies, applications, configurations, and IT infrastructure. You can use playbooks repeatedly across multiple environments. Customers who use Ansible playbooks typically deploy periodic changes manually. As complex workloads increase, you might be looking for ways to automate them. In this post, we show you how to automate an Ansible playbook deployment using Amazon Elastic Compute Cloud (Amazon EC2) and GitHub.

In the section “Walkthrough for automating Ansible playbook deployment,” we explain how to configure the pipeline, step by step. In the section “Alternative procedure: Use an AWS CloudFormation template,” we present a quick and repeatable solution. You can modify either method to suit your requirements.

Process overview

The following diagram shows the pipeline flow.

A push event triggers a webhook request, which is sent to an Amazon EC2 instance. We use NGINX to route the request to an Express server running on the EC2 instance. The Express server then runs an Ansible command to pull and run the newly pushed playbook.

First, set up Ansible on an Amazon EC2 instance running an Amazon Linux 2 Amazon Machine Image (AMI) connected to a GitHub repository that stores your playbooks. Second, configure a webhook, which is a way for an app to send other applications real-time information during a push event. This allows you to automatically configure multiple environments, which saves you the time and energy that would have otherwise been spent on manual processes.

Prerequisites

This blog post assumes that you’re familiar with AWS CloudFormation templates, Amazon EC2, and GitHub.

For this walkthrough, you need the following:

• An AWS account
• An Amazon EC2 key pair
• An Amazon EC2 instance running an Amazon Linux 2 AMI
• A security group that allows SSH (Secure Shell) and HTTPS access
• A GitHub repository to store playbooks

Walkthrough

Step 1: Set up webhook processing

To use Ansible with GitHub webhooks, set up webhook processing on the EC2 instance. This procedure uses NGINX as a reverse proxy to route the request to an Express server. Git is not required to process the webhook, but it is necessary for Ansible to pull the playbook from the repository.

1. Access the EC2 instance using SSH. See Connecting to your Linux instance using SSH.
2. Enable the Extra Packages for Enterprise Linux (EPEL) repository by running the following command.

   amazon-linux-extras install epel

   The output looks like this:

   

3. Apply the updates to the packages.

   yum update -y

4. Install Ansible, NGINX, and Git.

   yum install ansible -y
   yum install nginx -y
   yum install git -y

Your webhook processing is now set up.

Step 2. Install Node.js and set up the Express server

1. Install Node.js.

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash
. ~/.nvm/nvm.sh
nvm install node

2. Choose a location for the Express server. In this example, we create a directory called server to store the relevant files.

mkdir server && cd server
npm install express

3. When the installation completes, create a JavaScript file that contains the code to handle the webhook request. We create a sample named app.js that runs the ansible-pull command to pull and run the playbook.yml file from a GitHub repository. The server is configured to listen on port 8080. You must know that the server is configured to listen on port 8080 because the NGINX configuration needs to know where to route the traffic that it receives. The port number is arbitrary, but the specification in the NGINX configuration file must match the port that is defined in the Express server code. In this example, replace <GitHubUser>, <repo-name>, and <playbook> with your information.

var express = require('express');
var app = express();
const util = require('util');
const exec = util.promisify(require('child_process').exec);

app.post('/', function(req, res){
    try {
        console.log('executing deployment...');
        exec("ansible-pull -U git@github.com:<GitHubUser>/<repo-name>.git <playbook>.yml", (error, stdout, stderr) => {
                if (error) {
                console.log(`error: ${error.message}`);
                return;
        }
        if (stderr) {
                console.log(`stderr: ${stderr}`);
                return;
        }

        console.log(`stdout: ${stdout}`);

        });
    } catch (e) {
        console.log(e);
    }

    res.json({ received: true });
});

app.listen(8080, '127.0.0.1');

4. Specify the GitHub user and repository where the playbooks are stored. This is required by the ansible-pull command. In this example, replace <GitHubUser>, <repo-name>, and <playbook> with your information.

exec("ansible-pull -U git@github.com:<GitHubUser>/<repo-name>.git <playbook>.yml")

5. Run the Express server.

node app.js

Step 3. Set up a deploy key for your repository

1. Create an SSH key on your instance. In this example, replace <your_email@example.com> with your email address.

ssh-keygen -t rsa -b 4096 -C <your_email@example.com>

Keep the default settings, and press Enter to skip through the prompts.

2. When the key is created, run the following code.

eval "$(ssh-agent -s)"

The output looks similar to this:

Agent pid 1111

You will use this deploy key later in the procedure.

Step 4. Configure NGINX to route traffic

1. Use the following basic configuration to listen on port 80 and route traffic to the port that the Express server listens to.

   server {
       listen       80 default_server;
       listen       [::]:80 default_server;
       server_name _;
   
       location / {
           proxy_pass http://127.0.0.1:8080;
           proxy_http_version 1.1;
           proxy_set_header Upgrade $http_upgrade;
           proxy_set_header Connection 'upgrade';
           proxy_set_header Host $host;
           proxy_cache_bypass $http_upgrade;
       }
   
       error_page 404 /404.html;
           location = /40x.html {
       }
   
       error_page 500 502 503 504 /50x.html;
           location = /50x.html {
       }
   }

2. Start NGINX.

systemctl start nginx 
systemctl enable nginx

Step 5. Set up GitHub to configure the webhook

1. Log in to your GitHub account, and navigate to your repository settings. The page looks like this:

2. Add the deploy key from Step 3.
3. On your instance, navigate to the .ssh directory, where the public key is stored.
4. Open the id_rsa.pub file and copy its contents. In this example, replace <your_email@example.com> with your email address. It should look like the following.

ssh-rsa <your_email@example.com>

5. Choose Add deploy key, and paste the contents into the Key section. The page looks like this:

6. On the Webhooks tab, choose Add webhook.

7. Copy and paste your EC2 instance’s public IP address into the Payload URL section. This adds the webhook that is triggered when a push event occurs. When the webhook is created and a request is sent to the EC2 instance, the Recent Deliveries section looks like this:

This response indicates that the Express server received the request.

Alternative procedure: Use an AWS CloudFormation template

Use the following AWS CloudFormation template to provision the Ansible stack. (See Creating a stack on the AWS CloudFormation console). This stack does not create the web server code or the NGINX configuration file. For a sample configuration and sample code, see the previous section, “Walkthrough for automating Ansible playbook deployment.” This AWS CloudFormation template runs only in the US East (N. Virginia) Region, and you must use a public subnet with internet access. To use this template in another Region, configure the Mappings section to match your Region with the latest AMI ID.

AWSTemplateFormatVersion: 2010-09-09
Parameters:
  SubnetID:
    Type: AWS::EC2::Subnet::Id
    Description: Subnet to deploy EC2 instance into
  SecurityGroupIDs:
    Type: List<AWS::EC2::SecurityGroup::Id>
    Description: List of Security Groups to add to EC2 instance
  KeyName:
    Type: AWS::EC2::KeyPair::KeyName
    Description: >-
      Name of an existing EC2 KeyPair to enable SSH access to the instance
  InstanceType:
    Description: EC2 instance type
    Type: String
    Default: t2.micro
Mappings:
  AWSRegionToAMI:
    us-east-1:
      AMIID: ami-0a887e401f7654935
Resources:
  EC2Instance:
    Type: AWS::EC2::Instance 
    Properties:
      ImageId:
        !FindInMap 
          - AWSRegionToAMI 
          - !Ref AWS::Region
          - AMIID
      InstanceType: !Ref InstanceType
      KeyName: !Ref KeyName
      SecurityGroupIds: !Ref SecurityGroupIDs
      SubnetId: !Ref SubnetID
      UserData:
        Fn::Base64: 
          !Sub |
              #!/bin/bash -ex
              amazon-linux-extras install epel
              yum update -y
              yum install ansible -y
              yum install nginx -y
              yum install git -y
              systemctl start nginx
              systemctl enable nginx
              curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash
              export NVM_DIR="$HOME/.nvm"
              [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
              nvm install node
              cat <<EOF >> /home/ec2-user/.bashrc
              export NVM_DIR="/.nvm"
              [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
              EOF
              cd /home/ec2-user
              mkdir server && cd server
              npm install express
      Tags: 
        -
          Key: Name
          Value: Ansible - CloudFormation    
        -
          Key: Environment
          Value: Development

Cleanup

This procedure provisions an EC2 instance in your AWS account. If you choose an instance type within the free tier, it incurs no costs as long as it follows free-tier limits.

To remove your stack, see Deleting a stack on the AWS CloudFormation console. To remove your instance after provisioning the environment through the console, see Terminate your instance.

Conclusion

In this post, we walked through setting up a pipeline that enables you to deploy your Ansible playbooks through a push event using a combination of webhooks and Amazon EC2. We also provided an AWS CloudFormation template that spins up the resources automatically. Whichever method you choose, the configuration of multiple environments is automated to save time and energy. When you no longer must push playbooks manually, your system can respond to events in seconds using automated playbooks. This allows you to engineer a process that maintains consistency in your environment and ensures quality performance.

Learn about another way to use Ansible on AWS: Running Ansible playbooks using EC2 Systems Manager Run Command and State Manager.

Please let us know your thoughts in the comments.