Introduction

Hello World! At Oteemo we like to automate things. One of the tools that we use for automation is Ansible and a question that we often are asked is, “How can we develop and test our Infrastructure as Code?” In this series, we will build out a Ansible Role and discuss the strategies for developing a role with a high degree of confidence that it will perform as expected.

In this first entry, we will touch on the role itself, how we will test, and start the scaffolding.

The Role

The Ansible Role that we will develop will handle updating instances and rebooting them when necessary.

We are going to build it out from scratch with tests. This role makes a good example for role development as it uses a number of Ansible topics such as:

  • custom facts
  • blocks
  • conditionals
  • changed_when
  • async
  • delay

When we embed a role that can potentially cause reboots, we will want to make sure that we make some considerations so that reboots only happen when we expect them. Ideally, the roles we use during the provision portion of the lifecycle should be the same as the ones we use in the maintenance phase of the lifecycle. If we are doing immutable deploys, the latter does not apply; however, in many cases a base image will be created for a particular fleet and it will be promoted from lower to upper environments. It is important that we can update and reboot when necessary in a controlled fashion!

Some Thoughts on Testing

Testing does not have to be hard. There are frameworks that can assist with testing. The Molecule framework is one possible solution. I have written about Testing infrastructure with Molecule before.

Since that post, the Molecule project has released version 2. There are some good blogs out there on using the latest versions and perhaps one day I will write about it as well. However, Testing does not have to be hard. The Molecule project is promising, but I have found that it can be more time consuming than I would like. For example, all of my tests broke when we upgraded to ansible 2.5.1. I do like the project and I do continue to use it; however, there are other ways to Make testing happen (pun intended). Setting things up this way also illustrates how to test and that it is not very difficult when it is set up specific to a use case, rather than attempting to be generalized.

I also have been wanting to try out Goss, so we will use that as well.

Because we are potentially rebooting hosts, we cannot test against docker instances. We could use a vagrant solution to automate our testing against, for example, a virtualbox instance; however, I don’t have any bare metal available to run a CI server to run similar tests. We will be using Travis CI to run our tests against EC2 instances. We will want to set this up in a manner where we do not keep any secrets in our repository,

The Setup

We want our local development to look pretty similar to what our CI Server will look like. That way we can be pretty confident that we will be bringing all of the dependencies to the party. Travis CI is a Ubuntu Trusty container, so we will use a Trusty vagrantbox. Lets initialize a new repository and create one.

$ ansible-galaxy init biggsean.updates && cd $_ && git init

You will need to have vagrant installed at this point. We could do a vagrant init ubuntu/trusty64 at this point; however, I prefer just to create a simple Vagrantfile:

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure(2) do |config|
  config.vm.synced_folder ".", "/vagrant", type: "nfs"
  config.vm.define "ci" do |d|
    d.vm.box = "ubuntu/trusty64"
    d.vm.hostname = "ci"
    d.vm.network "private_network", ip: "10.20.30.40"
    d.vm.provision :shell, path: "bootstrap.sh"
    d.vm.provider "virtualbox" do |v|
      v.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
      v.memory = 2048
    end
  end
end

We will also need to create a minimal bootstrap script that we referenced in the Vagrantfile.

#!/usr/bin/env bash
sudo apt-get update
sudo apt-get -y install python-pip
sudo -H pip install virtualenv

At this point, we could do a vagrant up. We will also want to make sure that we put an entry for the vagrant metadata in our .gitignore.

Now let’s set up our test plays.

# We want to support multiple scenarios
$ mkdir -p tests/scenarios/{default,scenario-foo}

# We are using EC2, so we won't need the inventory
# created by the role scaffolding
$ rm tests/inventory

# Move the test play that was created to one of the
# scenario directories
$ mv tests/test.yml tests/scenarios/default/play.yml

We will edit the play so that it invokes the role wherever we test it. On our vagrantbox the role will be called vagrant, and on Travis it will be whatever our repository is called.

---
- hosts: localhost
  connection: local
  gather_facts: no
  roles:
    - role: "{{ lookup('env','PWD')|basename }}"

  tasks:
    - name: Do something for now
      assert:
        that: true
$ (cd tests/scenarios/scenario-foo/ && ln -s ../default/play.yml play.yml)

Test Steps

As we stated earlier, we want to be able to test this role against EC2 instances, so we will need to support the following test actions:

  • syntax check the ansible
  • lint the ansible and the Goss tests
  • create the EC2 instances
  • run the play
  • run the play again to test idempotence
  • validate the role with Goss
  • destroy the EC2 instances

Note that we do the fast tests first.

We will implement the EC2 and validation stuff later. For the initial commit and proof of concept, we will first implement syntax, lint, play, and idempotence. Fortunately for us there a tool that will facilitate this for us.

The Makefile

Let’s walk through the Makefile:

Set the shell to bash

SHELL:=bash

Set the Ansible version to a reasonable default and allow for virtualenvs that are specific to that version.

ANSIBLE_VERSION?=2.5.0
TEST_VENV:=~/.venv-ansible-$(ANSIBLE_VERSION)
ACTIVATE:=source $(TEST_VENV)/bin/activate
ANSIBLE_VERBOSITY:="-vv"
ANSIBLE_PLAYBOOK:=$(ACTIVATE) && ansible-playbook $(ANSIBLE_VERBOSITY)

Use a variable for the default version of ansible-lint and don’t allow the user to change it from the command line.

override ANSIBLE_LINT_VERSION:=3.4.23
ANSIBLE_LINT_VERBOSITY:="-vv"
ANSIBLE_LINT:=$(ACTIVATE) && ansible-lint $(ANSIBLE_LINT_VERBOSITY)

PIP:=$(ACTIVATE) && pip

Use the default scenario by default and set the order of the stages.

SCENARIO?=default
SCENARIOS_DIR:=./tests/scenarios
STAGES:=syntax lint play idempotence
PLAY:=$(SCENARIOS_DIR)/$(SCENARIO)/play.yml
RUN_PLAY:=$(ANSIBLE_PLAYBOOK) $(PLAY)

.PHONY in a Makefile just means that there isn’t actually a file target to test. While we should have all our targets as PHONY, Make does not allow implicit rules to be searched for PHONY targets. What this means for us is that we will not declare the scenario specific targets as PHONY so that we may override them later (e.g. we will need to create a rule that skips the idempotence check in a scenario where a reboot is requested). What it means is that we cannot name any files in our base directory something that matched the targets, or else Make will not do what we want.

.PHONY: test $(STAGES)
test: test-$(SCENARIO)
syntax: syntax-$(SCENARIO)
lint: lint-$(SCENARIO)
play: play-$(SCENARIO)
idempotence: idempotence-$(SCENARIO)
test-%: $(STAGES)
    @echo [email protected] Completed $?
syntax-%: install-ansible
    @echo [email protected]
    $(RUN_PLAY) --syntax-check
lint-%: install-ansible-lint
    @echo [email protected]
    $(ANSIBLE_LINT) $(PLAY)
play-%: install-ansible
    @echo [email protected]
    $(RUN_PLAY)
idempotence-%: play-%
    @echo [email protected]
    $(RUN_PLAY) | tee /dev/tty |exit $$(grep -Ec '(changed|failed)=[^0]' -)

Here are the dependencies for the different stages that we have defined.

.PHONY: install-ansible
install-ansible: create-venv ansible.cfg
    $(PIP) install ansible==$(ANSIBLE_VERSION)

.PHONY: install-ansible-lint
install-ansible-lint: create-venv install-ansible
    $(PIP) install ansible-lint==$(ANSIBLE_LINT_VERSION)

.PHONY: create-venv
create-venv:
    virtualenv $(TEST_VENV) --no-site-packages

define ANSIBLECFG
[defaults]
roles_path = ..
endef
export ANSIBLECFG
.PHONY: ansible.cfg
ansible.cfg:
    echo "$$ANSIBLECFG" > [email protected]

Finally, a clean target to remove some files that we might create.

.PHONY: clean
clean:
    rm ansible.cfg
    rm -rf ~/.venv-ansible-*

Now we have enough scaffolding to test on our vagrantbox. Let’s do a test run.

asciicast

The CI Tool

We can create a very simple .travis.yml using the same scripts that we used with our vagrantbox.

---
sudo: required

env:
  matrix:
    - ANSIBLE_VERSION=2.4.2.0
    - ANSIBLE_VERSION=2.5.1
    - ANSIBLE_VERSION=2.4.2.0 SCENARIO=scenario-foo
    - ANSIBLE_VERSION=2.5.1 SCENARIO=scenario-foo

install:
  - bash bootstrap.sh

script:
  - make test

Now we create a repository on Github, and connect it to Travis CI. Commit, push, and test.

That is enough for today. In part two we can discuss the creation and destruction of assets.