AWS Partner Network (APN) Blog
Getting Started with Ansible and Dynamic Amazon EC2 Inventory Management
Editor’s note: For the latest information on Amazon EC2, visit the Amazon EC2 website or the AWS Blog category for Amazon EC2.
By Brandon Chavis, Partner Solutions Architect at AWS
Today, the options for configuration and orchestration management seem nearly endless, making it daunting to find a tool that works well for you and your organization. Here at AWS, we think Ansible, an APN Technology Partner, provides a good option for configuration management due to its simplicity, agentless architecture, and ability to interact easily with your ever-changing, scaling, and dynamic AWS architecture.
Instead of having to push an agent to every new instance you launch via userdata, roll an agent into an AMI, or engage in similarly management-intensive deployments of your config management software, the Ansible framework allows administrators to run commands against Amazon Elastic Compute Cloud (Amazon EC2) instances as soon as they are available, all over SSH. This document intends to examine ways that your Amazon EC2 inventory can be managed with minimal effort, despite your constantly changing fleet of instances.
This post assumes you already have Ansible installed on either your workstation or an Amazon EC2 instance – Ansible has great documentation for installation (http://docs.ansible.com/intro_installation.html) and getting started (http://docs.ansible.com/intro_getting_started.html).
I’ve chosen to use a RHEL7 Amazon EC2 instance for my Ansible “master”. The reason I’ve done this is first, for convenience, and also, because the dynamic Amazon EC2 inventory script Ansible provides runs on top of Boto. This is significant because Boto can automatically source my AWS API credentials provided by an Amazon EC2 Identity and Access Management (IAM) role (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html). I’ve given my Amazon EC2 role Power User privileges, for simplicity – you may choose to lock this down further.
Should you decide to run Ansible on your workstation, you’ll need to set environment variables for your Secret and Access key:
$ export AWS_ACCESS_KEY_ID='YOUR_AWS_API_KEY'
$ export AWS_SECRET_ACCESS_KEY='YOUR_AWS_API_SECRET_KEY'
To get started with dynamic inventory management, you’ll need to grab the EC2.py script and the EC2.ini config file. The EC2.py script is written using the Boto EC2 library and will query AWS for your running Amazon EC2 instances. The EC2.ini file is the config file for EC2.py, and can be used to limit the scope of Ansible’s reach. You can specify the regions, instance tags, or roles that the EC2.py script will find. Personally, I’ve scoped Ansible to just look at the US-West-2 region.
https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.py
https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.ini
* Linked from this Ansible documentation: http://docs.ansible.com/ansible/intro_dynamic_inventory.html#example-aws-ec2-external-inventory-script
Use Wget/Curl/Git to pull those files down into the directory /etc/ansible/. I had to create this directory myself because I installed Ansible through pip.
Now you’ll need to set a few more environment variables for the inventory management script –
$ export ANSIBLE_HOSTS=/etc/ansible/ec2.py This tells Ansible to use the dynamic EC2 script instead of a static /etc/ansible/hosts file.
Open up ec2.py in a text editor and make sure that the path to the ec2.ini config file is defined correctly at the top of the script:
$ export EC2_INI_PATH=/etc/ansible/ec2.ini This tells ec2.py where the ec2.ini config file is located.
As a quick aside, I’ll address SSH connectivity. The Ansible documentation covers this fairly comprehensively (http://docs.ansible.com/intro_getting_started.html), but we’ll cover this briefly to get up and running. There are several options for how you’d like to handle authentication. Passwords, while not recommended, are allowed, and you can also use an SSH agent for credential forwarding.
Using an SSH agent is the best way to authenticate with your end nodes, as this alleviates the need to copy your .pem files around. To add an agent, do
$ ssh-agent bash
$ ssh-add ~/.ssh/keypair.pem
More on SSH-agents: https://developer.github.com/guides/using-ssh-agent-forwarding/
At this stage, you should be ready to communicate with your instances. Here’s your mid-blog post checklist:
-Ansible is installed and has access to your Secret and Access key (via EC2 role or environment variable)
-Ec2.py and Ec2.ini inventory scripts are downloaded and configured
-ANSIBLE_HOSTS environment variable set
-Ansible.cfg exists
-SSH agent is running (You can check with “ssh-add -L”)
Now we’re ready to see Ansible shine. You can run a command or playbook against any collection instances based on common Amazon EC2 instance variables. These variables are listed here: http://docs.pythonboto.org/en/latest/ref/ec2.html#module-boto.ec2.instance
If you call the Amazon EC2 inventory script directly, you’ll see your Amazon EC2 inventory broken down and grouped by a variety of factors. To try this, run $ /etc/ansible/ec2.py --list
For a useful example of leveraging the Amazon EC2 instance variables, I currently have two instances with the tag “Ansible Slave” applied to them. Let’s ping them real quickly.
$ ansible -m ping tag_Ansible_Slave
10.1.2.137 | success >> {
"changed": false,
"ping": "pong"
}
10.1.2.136 | success >> {
"changed": false,
"ping": "pong"
}
Now, I’ll manually launch another instance from the Amazon EC2 console using the “Launch more like this” option, for simplicity. While this can quickly be done through Ansible, that’s another blog post!
Five minutes later, I can run the command again –
$ ansible -m ping tag_Ansible_Slave
The authenticity of host '10.1.2.193 (10.1.2.193)' can't be established.
ECDSA key fingerprint is a7:6c:44:ef:dc:04:68:64:38:be:60:d8:d0:f7:2c:e0.
Are you sure you want to continue connecting (yes/no)? yes
10.1.2.193 | success >> {
"changed": false,
"ping": "pong"
}
10.1.2.136 | success >> {
"changed": false,
"ping": "pong"
}
10.1.2.137 | success >> {
"changed": false,
"ping": "pong"
In this example, I was prompted to accept the new SSH key for my new host. Clearly this won’t scale well, so in /etc/ansible/ansible.cfg, you can simply uncomment the line “host_key_checking = False” to prevent needing any manual input to connect to new hosts.
I can easily query by another variable- let’s ping all instances that use my “ansible_slaves” .pem file:
$ ansible -m ping key_ansible_slaves
10.1.2.193 | success >> {
"changed": false,
"ping": "pong"
}
10.1.2.136 | success >> {
"changed": false,
"ping": "pong"
}
10.1.2.137 | success >> {
"changed": false,
"ping": "pong"
}
Now, we need to update all of my instances tagged Ansible Slave, because there is a critical security patch needed – we can simply use the Ansible yum module (http://docs.ansible.com/yum_module.html) to upgrade all packages: $ ansible -s -m yum -a "name=* state=latest" tag_Ansible_Slave
I’ve snipped the output from this for brevity – Ansible can be very verbose when running “yum update” across several instances.
Similarly, I can install Nginx on my Ansible_Slave tagged instances. Here, I’m providing a URL to the Nginx .rpm, since it isn’t in the default RHEL7 Repos.
$ ansible -s -m yum -a
"name=http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-
centos-7-0.el7.ngx.noarch.rpm state=present" tag_Ansible_Slave
10.1.2.193 | success >> {
"changed": true,
"msg": "",
"rc": 0,
"results": [
"Loaded plugins: amazon-id, rhui-lb\nExamining /var/tmp/yum-
root-2WRpGo/nginx-release-centos-7-0.el7.ngx.noarch.rpm: nginx-release-
centos-7-0.el7.ngx.noarch\nMarking /var/tmp/yum-root-2WRpGo/nginx-
release-centos-7-0.el7.ngx.noarch.rpm to be installed\nResolving
Dependencies\n--> Running transaction check\n---> Package nginx-
release-centos.noarch 0:7-0.el7.ngx will be installed\n--> Finished
Dependency Resolution\n\nDependencies
Resolved\n\n==============================================================
==================\n
Package Arch Version Repository
Size\n==============================================================
==================\nInstalling:\n nginx-release-
centos\n noarch 7-0.el7.ngx /nginx-release-centos-7-
0.el7.ngx.noarch 1.5 k\n\nTransaction
Summary\n==============================================================
==================\nInstall 1 Package\n\nTotal size: 1.5 k\nInstalled
size: 1.5 k\nDownloading packages:\nRunning transaction check\nRunning
transaction test\nTransaction test succeeded\nRunning
transaction\n Installing : nginx-release-centos-7-
0.el7.ngx.noarch 1/1 \n Verifying : nginx-
release-centos-7-0.el7.ngx.noarch 1/1
\n\nInstalled:\n nginx-release-centos.noarch 0:7-
0.el7.ngx \n\nComplete!\n"
]
}
10.1.2.137 | success >> {
"changed": true,
"msg": "",
"rc": 0,
"results": [
"Loaded plugins: amazon-id, rhui-lb\nExamining /var/tmp/yum-
root-nU1W_b/nginx-release-centos-7-0.el7.ngx.noarch.rpm: nginx-release-
centos-7-0.el7.ngx.noarch\nMarking /var/tmp/yum-root-nU1W_b/nginx-
release-centos-7-0.el7.ngx.noarch.rpm to be installed\nResolving
Dependencies\n--> Running transaction check\n---> Package nginx-
release-centos.noarch 0:7-0.el7.ngx will be installed\n--> Finished
Dependency Resolution\n\nDependencies
Resolved\n\n==============================================================
==================\n
Package Arch Version Repository
Size\n==============================================================
==================\nInstalling:\n nginx-release-
centos\n noarch 7-0.el7.ngx /nginx-release-centos-7-
0.el7.ngx.noarch 1.5 k\n\nTransaction
Summary\n==============================================================
==================\nInstall 1 Package\n\nTotal size: 1.5 k\nInstalled
size: 1.5 k\nDownloading packages:\nRunning transaction check\nRunning
transaction test\nTransaction test succeeded\nRunning
transaction\n Installing : nginx-release-centos-7-
0.el7.ngx.noarch 1/1 \n Verifying : nginx-
release-centos-7-0.el7.ngx.noarch 1/1
\n\nInstalled:\n nginx-release-centos.noarch 0:7-
0.el7.ngx \n\nComplete!\n"
]
}
10.1.2.136 | success >> {
"changed": true,
"msg": "",
"rc": 0,
"results": [
"Loaded plugins: amazon-id, rhui-lb\nExamining /var/tmp/yum-
root-PszTju/nginx-release-centos-7-0.el7.ngx.noarch.rpm: nginx-release-
centos-7-0.el7.ngx.noarch\nMarking /var/tmp/yum-root-PszTju/nginx-
release-centos-7-0.el7.ngx.noarch.rpm to be installed\nResolving
Dependencies\n--> Running transaction check\n---> Package nginx-
release-centos.noarch 0:7-0.el7.ngx will be installed\n--> Finished
Dependency Resolution\n\nDependencies
Resolved\n\n==============================================================
==================\n
Package Arch Version Repository
Size\n==============================================================
==================\nInstalling:\n nginx-release-
centos\n noarch 7-0.el7.ngx /nginx-release-centos-7-
0.el7.ngx.noarch 1.5 k\n\nTransaction
Summary\n==============================================================
==================\nInstall 1 Package\n\nTotal size: 1.5 k\nInstalled
size: 1.5 k\nDownloading packages:\nRunning transaction check\nRunning
transaction test\nTransaction test succeeded\nRunning
transaction\n Installing : nginx-release-centos-7-
0.el7.ngx.noarch 1/1 \n Verifying : nginx-
release-centos-7-0.el7.ngx.noarch 1/1
\n\nInstalled:\n nginx-release-centos.noarch 0:7-
0.el7.ngx \n\nComplete!\n"
]
}
And we’re done!
I’d be remiss to leave out the playbook running abilities of Ansible in this blog post. Playbooks allow you to create repeatable routines of tasks in a single .yml file. Above, we update all of the packages on my instances and installed Nginx with ad-hoc commands; here we’ll condense this into a single command by running a playbook to my instances. Let’s take a look at a simple playbook to do both of those things:
---
- hosts: tag_Ansible_Slave
user: ec2-user
sudo: yes
tasks:
- name: Update all packages to latest
yum: name=* state=latest
- name: Install specific nginx package for centos 7
yum: name='http://nginx.org/packages/centos/7/noarch/RPMS/nginx-\
release-centos-7-0.el7.ngx.noarch.rpm' state=present
We simply add the code above into a “playbook.yml” file on the Ansible master and run it with the “ansible-playbook playbook.yml” command. See here for more info on playbooks: http://docs.ansible.com/playbooks_intro.html
$ ansible-playbook playbook.yml
PLAY [tag_Ansible_Slave] ******************************************************
GATHERING FACTS ***************************************************************
ok: [10.1.2.193]
ok: [10.1.2.137]
ok: [10.1.2.136]
TASK: [Update all packages to latest] *****************************************
changed: [10.1.2.136]
changed: [10.1.2.193]
changed: [10.1.2.137]
TASK: [Install specific nginx package for centos 7] ***************************
ok: [10.1.2.193]
ok: [10.1.2.136]
ok: [10.1.2.137]
PLAY RECAP ********************************************************************
10.1.2.136 : ok=3 changed=1 unreachable=0 failed=0
10.1.2.137 : ok=3 changed=1 unreachable=0 failed=0
10.1.2.193 : ok=3 changed=1 unreachable=0 failed=0
As you can see, there are a lot of possibilities with Ansible and Amazon EC2. Group instances by common attributes, tags, security groups, and so on, and you’ll be able to administer all of your machines with just a few commands.
To get started yourself, take a look at the Ansible installation documentation (http://docs.ansible.com/intro_installation.html) and grab the Amazon EC2 inventory python script and ini config file. The steps in this post can get you to a functional baseline, at which point you’ll be able to move toward becoming an Amazon EC2-orchestrating and playbook-writing Ansible wizard.
To learn more about Ansible on AWS, click here.