The most exciting phrase to hear in science, the one that heralds discoveries, is not ‘Eureka!’ but ‘Now that’s funny…’ – Isaac Asimov

Why we test

At Oteemo, we believe in automating to the maximum degree possible, ensuring consistent, versionable and robust infrastructure and application platforms.

In order to achieve this, we need to be able to describe our infrastructure as code, but also we must have a high level of confidence that the code we write meets the requirements to which we are developing. We like to be able to fail fast and be able to refactor our code without fear.

To automate our infrastructure, we generally look towards Ansible. In the past, to test Ansible, I would use Test Kitchen. Test Kitchen, while being an awesome tool for Chef, was not necessarily ideal for Ansible. One of the reasons we like Ansible is that it arguably puts a greater importance on idempotence than some of the other confuguration management tools. Test Kitchen, while there are work arounds, does not perform an idempotence test. In addition, since Ansible is done in python, it is not very convenient to pull in all of the ruby dependencies that Test Kitchen requires and I find it to be an easier sell to existing operations and engineering teams that are accustomed to hand-jamming golden images when we focus on one language.

Enter Molecule

The Molecule has been out since 2015 and is a testing aid written with Ansible in mind. It allows us to test our roles locally, prior to committing them. It allows us to test against multiple instances, operating systems, and virtualization providers.

Getting Started

$ # If you do not already have Ansible installed
$ pip install ansible
$ # install molecule
$ pip install molecule

Note 1: As of this writing, molecule 1.25.0 is the stable version. If you want to install the release candidate (2.0.0rc7) append --pre to the install command

Note 2: When installing on a mac, you probably need to preface the above pip commands with sudo -H

Using Molecule

When initializing molecule for either an existing role or a new role, a few decisions should be made:

  1. Driver
    • Vagrant (default)
    • Docker
    • OpenStack
  2. Providor
    • Virtualbox (default)
    • libvirt
    • VMWare Fusion
    • Parallels
  3. Verifier
    • Testinfra (default)
    • Serverspec
    • Goss (beta)

For this post, I am going to stick with the defaults. To follow along, be sure to have Vagrant and Virtualbox installed. You will need to install the python package for your driver as well:

$ pip install python-vagrant

Hello World

The Hello World in configuration management land seems to be the ntp install. We can use molecule to scaffold the role for us.

Initialize

$ molecule init --role ntp

This gives us:

$ tree ntp
ntp
├── README.md
├── defaults
│   └── main.yml
├── handlers
│   └── main.yml
├── meta
│   └── main.yml
├── molecule.yml
├── playbook.yml
├── tasks
│   └── main.yml
├── tests
│   ├── test_default.py
│   └── test_default.pyc
└── vars
    └── main.yml

First, let’s look at the contents of molecule.yml.

dependency:
  name: galaxy
driver:
  name: vagrant
vagrant:
  platforms:
    - name: ubuntu/trusty64
      box: trusty64
      box_url: https://vagrantcloud.com/ubuntu/boxes/trusty64/versions/14.04/providers/virtualbox.box
  providers:
    - name: virtualbox
      type: virtualbox
      options:
        memory: 512
        cpus: 2
  instances:
    - name: ntp
      ansible_groups:
        - group1
verifier:
  name: testinfra

The dependency section allows us to provide role dependencies with ansible-galaxy. we will not be doing that in this tutorial. The other sections should be self explanatory. Feel free to check out the docs if it is not.

To start out, I am going to make a modification to the verifier section:

verifier:
  name: testinfra
  options:
    v: True
    sudo: True

The v is for verbosity. This will have Testinfra tell us the tests that run. The sudo flag runs the tests will elevated permissions. While this is not always necessary, I often include it.

Now let’s run it to make sure it works. Then we will start to develop the role.

$ # Make sure you are in the same directory as molecule.yml
$ molecule test

If this is your first foray in to vagrant, or you have never used this particular box before then the first run will take some time as you will have to download the box. Subsequent runs will be quicker.

After vagrant spins up the box, you should see:
Molecule destroy any existing instances

--> Destroying instances...

Molecule runs an ansible syntax check

--> Checking playbook's syntax...

playbook: playbook.yml

Molecule creates the instances

--> Creating instances...
Bringing machine 'ntp' up with 'virtualbox' provider...
==> ntp: Cloning VM...
... [Vagrant output omitted]
==> ntp: Machine not provisioned because `--no-provision` is specified.

Molecule runs the provisioner

--> Starting Ansible Run...

PLAY [all] *********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [ntp]

PLAY RECAP *********************************************************************
ntp                        : ok=1    changed=0    unreachable=0    failed=0

Molecule runs an idempotence check

--> Idempotence test in progress (can take a few minutes)...
--> Starting Ansible Run...
Idempotence test passed.

Molecule runs linters on ansible and the tests

--> Executing ansible-lint...
--> Executing flake8 on *.py files found in tests/...

Molecule runs the tests

--> Executing testinfra tests found in tests/...
============================= test session starts ==============================
platform darwin -- Python 2.7.12, pytest-3.1.2, py-1.4.34, pluggy-0.4.0 -- /usr/local/opt/python/bin/python2.7
cachedir: .cache
rootdir: /Users/smcgowan/ws/oteemo/ntp, inifile:
plugins: testinfra-1.5.5
collected 1 itemss

tests/test_default.py::test_hosts_file[ansible://ntp] PASSED

=============================== warnings summary ===============================
None
  Module already imported so can not be re-written: testinfra

-- Docs: http://doc.pytest.org/en/latest/warnings.html
===================== 1 passed, 1 warnings in 2.47 seconds =====================

Molecule destroys the instance (if everything passed)

--> Destroying instances...
==> ntp: Forcing shutdown of VM...
==> ntp: Destroying VM and associated drives...

You can perform any of these steps individually. Use molecule --help to see the options.

Install ntp

Now that we have molecule working and the role scaffolded, we can create some tests for the ntp role. The general strategy is to create the tests and run them to go red and then to write the ansible code that passes the tests to go green.

# tests/test_default.py

import testinfra.utils.ansible_runner

testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
    '.molecule/ansible_inventory').get_hosts('all')


def test_ntp_is_installed(Package)
    p = Package('ntp')

    assert p.is_installed

Now run molecule test. This should not pass, since ntp is not installed. Now you are red. To go green, update your role to install the package:

---
# tasks/main.yml
- name: ensure ntp package is installed
  package:
    name: ntp
    state: present
  tags: [ntp]

Run molecule test again, and we should pass!

collected 1 itemss

tests/test_default.py::test_ntp_is_installed[ansible://ntp] PASSED

=============================== warnings summary ===============================
None
  Module already imported so can not be re-written: testinfra

-- Docs: http://doc.pytest.org/en/latest/warnings.html
===================== 1 passed, 1 warnings in 2.18 seconds =====================
--> Destroying instances...
==> ntp: Forcing shutdown of VM...
==> ntp: Destroying VM and associated drives...

Hooray!

Start/enable the service

We want to guarantee that ntp is running, so let’s add the test to tests/test_default.py and run molecule test.

import testinfra.utils.ansible_runner

testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
    '.molecule/ansible_inventory').get_hosts('all')


def test_ntp_is_installed(Package):
    p = Package('ntp')

    assert p.is_installed


def test_ntp_is_started_and_enabled(Service):
    s = Service('ntp')

    assert s.is_running
    assert s.is_enabled

Ubuntu automatically starts and enables the service, so we do not have anymore work to do. We are still green!

Multiple platforms

Let’s make our role work across multiple platforms. In molecule.yml, update the platforms section:

vagrant:
  platforms:
    - name: ubuntu/trusty64
      box: trusty64
      box_url: https://vagrantcloud.com/ubuntu/boxes/trusty64/versions/14.04/providers/virtualbox.box
    - name: bento/centos-7.3
      box: bento/centos-7.3

Unfortunately, molecule does not yet handle multiple platforms in the same run yet. Molecule v2 handles scenarios is still a release candidate. For now, we will use the –platform option.

$ molecule test --platform bento/centos-7.3

This will fail.

tests/test_default.py::test_ntp_is_installed[ansible://ntp] PASSED
tests/test_default.py::test_ntp_is_started_and_enabled[ansible://ntp] FAILED

=================================== FAILURES ===================================
________________ test_ntp_is_started_and_enabled[ansible://ntp] ________________

Service = <class 'testinfra.modules.base.SystemdService'>

    def test_ntp_is_started_and_enabled(Service):
        s = Service('ntp')

>       assert s.is_running
E       assert False
E        +  where False = <service ntp>.is_running

tests/test_default.py:16: AssertionError

We are now red again because in CentOS land, the service name is ntpd. We will need to update the test in tests/test_default.py.

def test_ntp_is_started_and_enabled(Service, SystemInfo):
    distro = SystemInfo.distribution
    if distro == 'ubuntu':
        svc = 'ntp'
    elif distro == 'centos':
        svc = 'ntpd'

    s = Service(svc)

    assert s.is_running
    assert s.is_enabled

We can now run the tests again.

$ molecule test --platform ubuntu/trusty64

The Ubuntu test is still green.

$ molecule test --platform bento/centos-7.3

The CentOS test is still red!!! This is because CentOS does not make the assumption that just because you installed a package that you want the service up and running (also because CentOS 7 uses chronyd by default, so we should probably make sure it is disabled if we want to use ntpd). We will have to update tasks/main.yml to get this to pass.

---
# tasks file for ntp
- name: ensure ntp package is installed
  package:
    name: ntp
    state: present
  tags: [ntp]

- name: set ntp service name
  set_fact:
    ntp_service: "{{ (ansible_os_family == 'Debian') | ternary('ntp','ntpd') }}"
  tags: [ntp]

- name: ensure ntp is running and enabled
  service:
    name: "{{ ntp_service }}"
    state: started
    enabled: yes
  tags: [ntp]

Now if we run both the tests they are green. Yay!

The end

Clearly we could do more things with this role. For example we could configure ntp, and perhaps I will add to the role when molecule v2 is officially released; however, this should get you started testing your infrastructure with molecule.

You can follow the code here