Index ¦ Archives ¦ Atom

Setting Up Ansible for AWS with Dynamic Inventory (EC2)

Personally, I find AWS's documentation on setting up dynamic inventory/EC2 for use with Ansible to be a bit cryptic, so thought I'd write what is (at least to me) a simpler way to do it.

We're gonna start from the absolute basics, and assume that right now, you don't even have Ansible installed and are just beginning. (So it's basically you, a dream and an AWS account, hehe)

First things first, decide where you will be running Ansible from. It can be your laptop, another ec2 instance or something similar. This part doesn't really matter. For the purposes of this tutorial, I am running Ansible from a rhel ec2 box in the same Availability Zone as the nodes I'll be managing. This isn't a hard requirement, but it does help a bit with performance to have your Ansible box running in the same AZ as your managed nodes.

1) Spin up an EC2 instance to be the box you will run Ansible from.

In this tutorial, I am using the standard AWS RHEL 7.5 AMI. For now, just pick a t2.micro, default Storage settings (my box is using a 10GB volume) and default "Instance Details" (ie defaults for whatever your vpc and subnet is). There are two important things though: don't use the default security group setting AWS gives you-you'll end up with an instance that is open to the world. Instead, when you come to the SG part, just open Port 22 for "My IP".

LaunchControlBox

2) Enable the Epel repo.

Once you've got your instance spun up, go ahead and log into it using your chosen method (putty on Windows, terminal on Mac, etc) We're going to do a few things to get prepped for Ansible...I like to install it using Pip, so we'll need to get the box ready for that. First enable the EPEL repo:

cd /tmp
wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
ls *.rpm

Confirm that when you do ls the rpm shows up that you just downloaded. If it does, go ahead and install it:

yum install epel-release-latest-7.noarch.rpm

Then confirm the repo is enabled and ready to go by doing the below command and confirming the EPEL repo is listed:

yum repolist

3) Install Pip

Now that we have the EPEL repo enabled on the box, go ahead and install Pip:

yum install python-pip

Then do a pip --version to confirm it's there:

PipConfirm

4) Install Ansible (for this tutorial, I am using Ansible 2.4.3)

yum install ansible

BONUS NOTE: Ansible tends to upgrade their versions pretty frequently, and if for some reason you need to install a specific version for consistency, instead of doing a straight 'yum install ansible' and taking whatever they give you, here's how to install a specific version:

pip install ansible==2.4.3

You can get a list of the versions here: https://releases.ansible.com/ansible/

The above command will also work if you specify two digits out, like so:

pip install ansible==2.5

Or four:

pip install ansible==2.0.1.0

5) Set up Your Ansible Config File

Now that you have Ansible installed, let's set up your Ansible config file. When installing through pip, it doesn't create the config directory, so you'll need to go to /etc and create a dir called 'ansible'.

cd /etc
mkdir ansible
cd ansible

Then make a empty config file named ansible.cfg:

touch ansible.cfg

6) Your Basic Config File

Here is where we could do a deep-dive into Ansible's configs, as in their documentation's sample config file they list a LOT: Sample Config File

We could spend a lot of time discussing this. For now, I'll just present you with what I feel is a good starting point for a file...

[defaults]
forks = 999
gathering = explicit
timeout = 60
var_compression_level = 9

display_skipped_hosts = False
host_key_checking = False
retry_files_enabled = False
roles_path = roles:platform/roles:application/roles

jinja2_extensions = jinja2.ext.do

callback_whitelist = profile_tasks

stdout_callback = debug
stderr_callback = debug

[ssh_connection]
control_path = %(directory)s/%%h-%%r

Explanation of each setting:

forks = 999
I work with a LOT of machines, so setting a high number of forks will help you with performance. If you won't be managing hundreds of servers, feel free to set this to 5. That's the basic starter number.

gathering = explicit
By default, Ansible will gather facts about the nodes it manages, like their OS and IP. This can take time and slow down the execution of a playbook, especially if you're running tasks against a lot of instances. I turn this to explicit, which means it only runs if I tell it to in my playbook. Otherwise, it doesn't spend time grabbing this info.

timeout = 60
Pretty self-explanatory...this is how long Ansible will wait before giving up when trying to SSH to an instance.

var_compression_level = 9
This controls the var compression sent to worker processes. Default is zero, highest is 9.

display_skipped_hosts = False
I don't want to see messages during my playbook run output listing skipped hosts, so I just set this to false. (It will still show you the task name).

host_key_checking = False
Disables SSH key host checking. Set to True if you want this.

retry_files_enabled = False
When a playbook fails to execute, Ansible will save a .retry file so you can try again using this file. I never do-if a playbook fails, I just edit the playbook to fix what went wrong and rerun the bash command I used to run it the first time, so I leave this set to False, as the .retry files can really clutter up your directory when you are developing & testing playbooks.

roles_path = roles:platform/roles:application/roles
Specifies where Ansible will look for roles when you use the role section in your playbook.

jinja2_extensions = jinja2.ext.do
List any jinja2 extensions you may use here. (Ansible uses Jinja as its template engine)

callback_whitelist = profile_tasks
stdout_callback = debug
stderr_callback = debug
These all control callback plugin output you get when running your playbook. I have them set to debug so that I can see plenty of info when something goes wrong.

[ssh_connection]
control_path = %(directory)s/%%h-%%r
This controls the directory for the ControlPath sockets. When you're using EC2, you'll often have long hostnames and can run into an error with "too long for Unix domain socket", so this setting fixes that.

For more details, please see the Ansible documentation

7) Set Up Dynamic Inventory

Now that you have your config file set up in /etc/ansible/ansible.cfg, it's time to get to the heart of things. In order to set up dynamic inventory, there are three parts:

1-Set up an IAM user for Ansible to use when executing playbooks and managing your AWS resources.
2-Set up Boto
3-Copy over the standard Boto EC2 scripts (ec2.py and ec2.ini)

8) Set up An Ansible IAM User

So step one is, we are going to set up an IAM user named 'ansible_user' and give it full access to the resources we'll be having it manage. In this case that is EC2, and S3 (S3 is in here because often you'll deploy files from S3 onto your EC2 instances, such as deploying application artifacts, so it'll be handy to have S3 access set up from the beginning.)

Set up your user:

SetUpUser

Click on "Attach existing policies directly" and choose the ec2 and s3 FullAccess policies. (Yes, you could select the Admin policy that gives you full access to EVERYTHING in your account, but that is pretty risky and I don't recommend it.)

EC2Policy

S3Policy

9) Copy IAM Credentials

Once you get to the final screen, copy the Access key id and Secret Access key details (either copy paste the details from the screen or download the CSV AWS gives you.)

10) Install Boto

Now that you have your IAM user set up, let's get Boto going. Back on your Ansible control box, install boto (specifically Boto3):

pip install boto3

11) Set Up the Boto Credentials File

There's lots of ways to set up creds for Boto...I think this is one of the simplest ways, and as a bonus, it will also allow you to access multiple AWS accounts with the same Ansible install.

cd
mkdir .aws
cd .aws
touch credentials

12) Define Credentials

In this credentials file, input your access key id and secret access key info from Step 9 (Please note: you don't need any quotes around anything...just input the key info directly). Also put in your region. If for example, you are using instances in the Northern California Availability Zone, put in us-west-1. One final thing....Boto requires at least one 'default' profile in this file in order for it to work, so make sure you name at least one of your AWS accounts 'default'. (If you only have one AWS account, then this is easy and you know what to do ;)

[default]
aws_access_key_id = YOUR_KEY
aws_secret_access_key = YOUR_SECRET
region = YOUR_REGION

13) Test Your Boto Install

Now that you have your credentials set up, let's do a test. Create an S3 bucket (if you don't have one already) and copy over this test python script to your Ansible control box:

#!/usr/bin/python

import boto3

s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
    print(bucket.name)

Then chmod it so it's executable:

chmod +x test.py

Then run your test script. What it does is use Boto to talk to your AWS account and display a list of all your S3 buckets. If you run this script from your Ansible control box and it outputs your S3 bucket, you are doing great and your Boto is successfully configured.

./test.py

14) Copy Over the Boto EC2 Scripts

Almost there! You now have Ansible installed and configured, Boto installed and configured, and your IAM user set up. Now it's time to do the last part, and that is to copy over the standard Boto EC2 scripts from Ansible. From your control box, run the following commands to download the files to whatever directory you'll be running Ansible from.

wget https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.py
wget https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.ini

Make sure you make the ec2.py script executable, otherwise you'll get cryptic errors when you try to run the Ansible playbook.

chmod +x ec2.py

15) Update Security Groups for Ansible

Now that we have everything set up and ready to go, let's run a test playbook. In this case we are going to be installing Apache on an EC2 instance via Ansible. But before we can do that, we need to update the Security Groups on your Ansible Control box to allow outbound of everywhere (you can also lock it down to the subnets you know your instances will be on.)

OutboundAccess

16) Spin Up a Test EC2 Instance

Then spin up an EC2 instance. You can make it be whatever you want, just do one thing, and this is crucial...Tag it. Doesn't matter what..in this case, we'll use two tags of Environment = Gloria, Instance = 1 in our Ansible playbook to identify to Ansible which instance to run on.

Tagging

Tags are a crucial and easy way for Ansible to know what instances to run against. They give you a lot of control, especially in an AWS environment where you're likely to be auto-scaling instances up and down and IPs will come and go frequently. Without static inventory files, good tagging of AWS resources Is Your Friend. :)

17) Set Up Security Group Access on Test EC2 Instance

Make sure your new EC2 instance has security group access that will allow it to be managed by Ansible. On the security group page for the security group your new instance is using, edit the rules and add two new ones:

-Inbound 22 from your Ansible control box. (You can just take the subnet your Ansible control box is on, and allow inbound from that subnet.)
-Outbound rule to 0.0.0.0 so that it can reach the Yum repos.

18) Create Test Playbook

Back on your Ansible control box, create a file in the same directory as your ec2.py and ec2.ini scripts called test.yml and paste in the following code:

---
- hosts: "tag_Environment_{{ env }}:&tag_Instance_{{ instance }}"
  vars:
    env: Gloria
    instance: '1'

  become: yes

  tasks:
  - name: Install apache
    yum:
      name: httpd
      state: latest

In all fairness, I am defining the env and instance vars here in the playbook, but you can also pass them in on the command line if you know later on they'll be different and want to use one playbook with different tags.

19) Copy Private Key to Ansible Control Box

Last thing to do is copy the private key for your test EC2 instance to your Ansible control box, and chmod 600 it so that Ansible can use it to SSH to your instances. In this case, my key is named INITIAL_KEYPAIR_2018

chmod 600 INITIAL_KEYPAIR_2018

20) Run Test Playbook Against Test Instance

Now you'll go ahead and run your test playbook with the following command:

AWS_PROFILE=default ansible-playbook -i ec2.py test.yml --private-key=./INITIAL_KEYPAIR_2018 -u ec2-user

Let's break this command down a bit:

AWS_PROFILE=default
This is the profile we set up in step 12. You can name your profile in your Boto credential file whatever you want as long as you have a 'default' profile in there too...in this case, I just stuck with naming my account 'default'. You could call it "test" or "production" if you wanted, just make sure that when you run your playbook, the value of AWS_PROFILE that you pass in on the command line is the same as what you named it in your boto credential file.

ansible-playbook
This is the default command in Ansible to run a playbook.

-i ec2.py
The i flag means you are telling ansible where to get your inventory file from. In this case, since it's dynamic, we are telling Ansible to use our Boto ec2 script to pull a list dynamically from AWS of all instances with the tags we specify either in our playbooks or on the command line when we run a playbook, and then use that list as its inventory.

test.yml
Test.yml is in here to tell Ansible the name of the playbook to run. Pretty self-explanatory.

--private-key=
This is where you tell Ansible the location of the private key you are using to connect to your EC2 instances.

-u ec2-user
This is the user flag and it's very important when using Ansible with AWS instances. I spun up a RHEL instance for our EC2 test box, so the default user is ec2-user on AWS. Ansible will need to know this, so that it can use the right default user to connect to AWS with, otherwise it will not be able to work. An astute observer may have noticed in the test playbook that I added become: yes to it. This will allow you to do root user actions (like install packages), but still do the initial login as ec2-user, like AWS is expecting.

For a list of the different OS's default usernames in AWS, see here (scroll down a bit):
AWS Default User Names by OS

21) Playbook Results

Coming back to your playbook command, your playbook will run and you should see something like the below output:

AnsiblePlaybookOutput

So there you have it. You now have an Ansible control box set up and can successfully manage your remote EC2 instances via playbooks.
Have fun, and may the odds be ever in your favor >:)

© 2015-2018 Gloria Silveira. Member of the Internet Defense League.