Editor’s note: Please check out the much newer article Configuring Travis CI on a Puppet Module Repo for the new “best practices” around setting up rspec-puppet. You are encouraged to use the newer setup, though everything on this page will still work!

We recently discussed test-driven development for puppet modules in the context of rspec-puppet. That’s a nice, simple introduction to testing, but doesn’t provide everything we need. Rspec-puppet is limited in the matchers available (notably there are no negation tests) and its inability to test dependencies (when a module includes another module), both of which will be necessary eventually. The next step is puppetlabs_spec_helper, a project by Puppet Labs that provides us with more full-fledged specification tests.

Installation

The biggest requirement for puppetlabs_spec_helper is a ruby version of 1.9 or higher. CentOS 6.5, however, only includes v1.8.7. There are numerous ways to upgrade ruby, most of which are horrible. We’ll look at using the Ruby Version Manager, or RVM, to upgrade to 1.9.3. This can be done with puppet via the maestrodev/rvm module. After adding the module to your master, create a class or modify an existing one to provide RVM and some puppet and rspec gems.

class profile::rvm {
  include '::rvm'

  $ruby_version = 'ruby-1.9.3-p551'
  rvm_system_ruby { $ruby_version:
    ensure      => present,
    default_use => true,
  }

  $rvm_gems = [
    'rspec-puppet',
    'puppet',
    'fpm',
    'puppet-lint',
    'puppetlabs_spec_helper',
  ]
  rvm_gem { $rvm_gems:
    ensure       => latest,
    ruby_version => $ruby_version,
    require      => Rvm_system_ruby[$ruby_version],
  }
}

If your distro of choice provides Ruby 1.9 or higher already, you can add puppetlabs_spec_helper as a regular package resource from the gem provider. Add the new profile to your build role.

class role::build {
  include ::profile::base  # All roles should have the base profile
  include ::profile::users::build
  include ::profile::build
  include ::profile::rvm
}

One other thing you’ll need is to open (or proxy) port 11371 from any node that uses profile::rvm, as it will require grabbing GPG keys via hkp (at L30-36). You can, alternatively, run these commands locally, as the hkp server responds on port 80, after changing to root:

[rn7284@build ~]$ sudo su -
[root@build ~]# gpg2 --keyserver hkp://keys.gnupg.net:80 --recv-keys D39DC0E3
gpg: directory `/root/.gnupg' created
gpg: new configuration file `/root/.gnupg/gpg.conf' created
gpg: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring `/root/.gnupg/secring.gpg' created
gpg: keyring `/root/.gnupg/pubring.gpg' created
gpg: requesting key D39DC0E3 from hkp server keys.gnupg.net
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key D39DC0E3: public key "Michal Papis (RVM signing) <mpapis@gmail.com>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

If the key is reported as not found, run the command once or twice more, the key server appears to inconsistently find keys.

After running the puppet agent, log out and back into your build server and ruby -v should show the version you specified and gem list should show all the specified gems and their dependencies. We’re ready to begin.

Setup

With the installation out of the way, we still need to set up our module to support puppetlabs_spec_helper. Let’s look at the existing certs module we created previously. The Rakefile and spec/spec_helper.rb need small tweaks. Add the line require ‘puppetlabs_spec_helper/rake_tasks’ to the former and require ‘puppetlabs_spec_helper/module_spec_helper’ can replace the entirety of the later file.

When we want to use puppetlabs_spec_helper with a module that hasn’t been set up yet, a good start would be to run rspec-puppet-init and copy these two files to the new module. Any modules generated with puppet module generate will already have the correct Rakefile already in place. Go ahead and do this in to your profile module, as we’ll be using that shortly.

Lastly, we will be testing additional modules, which means copying them into the working path. We do not want to track these files in our git repo. Update the .gitignore file to prevent that from happening.

[rnelson0@build certs]$ cat >> .gitignore
spec/fixtures/

If you already have any files underneath this path in your git repo, git rm them and run rspec-puppet-init again to recreate the test manifest. In the case of certs, these commands were required:

git rm spec/fixtures/manifests/site.pp
git rm spec/fixtures/modules/certs/manifests
rspec-puppet-init

Testing

Our first simple test is to run rake spec. This rake (that stands for “ruby make”) target will perform the same function as rspec did before, but provides additional features that we will configure shortly. Right now, just check that existing tests continue to work. If you have any issues, be sure to check the installation and setup directions to ensure no steps were missed. I’ll be glad to address any issues in the comments if you have them.

Technically, you can run rspec right now and it will work fine. However, the first time you run rake spec, a file that rspec relies on will be removed.

Next, we have more a more capable test structure, so we need to write more tests. Let’s start with the profile::apache class, which we could not test with rspec-puppet because the class included another module. As always, we want to write the tests first. When the class is included, it should include ::apache (not to be confused with apache, which would parse as profile::apache due to the namespace it is called in!) and the package, called httpd on CentOS. The puppetlabs-apache module requires some facts to be set to determine the package name or it will fail out completely, so in our describe section we need to specify some facts. Here’s what the full test looks like:

[rnelson0@build profile]$ cat spec/classes/apache_spec.rb
require 'spec_helper'
describe 'profile::apache', :type => :class do
  let :facts do
  {
    :id                     => 'root',
    :kernel                 => 'Linux',
    :osfamily               => 'RedHat',
    :operatingsystem        => 'RedHat',
    :operatingsystemrelease => '6',
    :concat_basedir         => '/dne',
    :path                   => '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
  }
  end

  context 'with defaults for all parameters' do
    it { is_expected.to create_class('profile::apache') }
    it { is_expected.to contain_package('httpd') }
    it { is_expected.to contain_user("apache") }
  end
end

The let :facts block specifies the key/value pairs that we want set for our spec tests. You can set these facts at the describe and context levels, but each level uses its own hash and does not inherit from the level above. If you want to just change the :operatingsystemrelease fact, you need to duplicate the rest of the hash.

Once the facts are set, we use a slightly different set of functions. Rspec-puppet’s should is equivalent to puppetlabs_spec_helper’s is_expected.to. There’s also a is_expect.not_to function, which we didn’t have before. When included, we should see our profile class, the apache package named httpd on the RedHat family, and the local user apache. Pretty simple. Try rake spec and you’ll see that we still have some work to do as our three tests all generate the same error:

Could not find class ::apache for build.nelson.va on node build.nelson.va

We are expecting that at this point. By using rspec-puppet-init, the spec/fixtures/modules/profile path is a symlink to the top of our git repo, so that when spec tests are run, the profile module is available for testing. But the apache module is not available. Now, you could make sure that the correct version of the module is checked out on your build server and create some symlinks yourself, but that’s not going to be fun. It’s also not going to be fun when the module is updated, or when you have 100 modules, or if the module is used in multiple other modules, all of which need symlinks. In other words, we need a better way, and that’s where puppetlabs_spec_helper comes in.

The helper includes support for fixtures, a method for dynamically providing the dependencies that your module requires. Unlike the symlinks insanity above, this is simple and portable. Anyone who checks out your module’s repo and has puppetlabs_spec_helper/rspec-puppet installed and working will be able to perform tests as well. We need to create a .fixtures.yml file at the top level. The format is simple, we specify one module (our own) that is loaded via symlinks and a number of repositories that are loaded via git. We won’t use a ref today for simplicity, but they are available if you need to pin to a certain module version.

fixtures:
  symlinks:
    profile: "#{source_dir}"
  repositories:
    stdlib: "git://github.com/puppetlabs/puppetlabs-stdlib"
    apache: "git://github.com/puppetlabs/puppetlabs-apache"
    concat: "git://github.com/puppetlabs/puppetlabs-concat"
    firewall: "git://github.com/puppetlabs/puppetlabs-firewall"
    ntp: "git://github.com/puppetlabs/puppetlabs-ntp"

Stdlib is listed as a dependency for most modules, even though that’s often only because it’s included in the default metadata.json when generating a new module. The next two repositories are for apache and its dependency, concat. Yep, you have to make sure that included modules have their own dependencies available. Firewall and ntp are additional dependencies of the profile module. While we’re here, take a moment to update your own module’s metadata.json‘s dependencies hash. With this file in place, perform a rake spec and you’ll be rewarded with successes:

[rnelson0@build profile]$ rake spec
Initialized empty Git repository in /home/rnelson0/git/profile/spec/fixtures/modules/stdlib/.git/
remote: Counting objects: 5903, done.
remote: Total 5903 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (5903/5903), 1.18 MiB | 1.64 MiB/s, done.
Resolving deltas: 100% (2501/2501), done.
Initialized empty Git repository in /home/rnelson0/git/profile/spec/fixtures/modules/apache/.git/
remote: Counting objects: 8038, done.
remote: Total 8038 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (8038/8038), 2.00 MiB | 1.68 MiB/s, done.
Resolving deltas: 100% (4954/4954), done.
...
Finished in 3.27 seconds (files took 0.80976 seconds to load)
5 examples, 0 failure

As you can see, the first task for our rake target is to clone the repositories. With the cloned repository in place, ::apache is a known class and therefore the tests will complete. You can go ahead and muck with a test, such as misspelling httpd, and see that the tests succeed and fail as intended.

Performance

You might notice that rake spec performs a clone every time you run it. In fact, if you look at spec/fixtures/modules, you’ll notice that it’s empty after rake complete. That’s not exactly high performance any way you look at it, but it can be absolutely devastating if any of the modules are large, like our kickstart module which contains vmware tools, or you’re on a slow or high-latency link. Let’s see what other targets are available with rake help – the help target always shows you the list of targets.

[rnelson0@build profile]$ rake help
rake beaker            # Run beaker acceptance tests
rake beaker_nodes      # List available beaker nodesets
rake build             # Build puppet module package
rake clean             # Clean a built module package
rake coverage          # Generate code coverage information
rake help              # Display the list of available rake tasks
rake lint              # Check puppet manifests with puppet-lint / Run puppet-lint
rake spec              # Run spec tests in a clean fixtures directory
rake spec_clean        # Clean up the fixtures directory
rake spec_prep         # Create the fixtures directory
rake spec_standalone   # Run spec tests on an existing fixtures directory
rake syntax            # Syntax check Puppet manifests and templates
rake syntax:hiera      # Syntax check Hiera config files
rake syntax:manifests  # Syntax check Puppet manifests
rake syntax:templates  # Syntax check Puppet templates
rake validate          # Check syntax of Ruby files and call :syntax / Validate manifests, templates, and ruby files

There are a lot of good targets available – lint, syntax(:*) and validate are helpful – but let’s stick to the spec-related targets for now. Spec_prep and spec_clean look like they could be used in tandem and spec_standalone used to run the tests. Here’s what spec_prep does when run twice:

[rnelson0@build profile]$ rake spec_prep
Initialized empty Git repository in /home/rnelson0/git/profile/spec/fixtures/modules/stdlib/.git/
remote: Counting objects: 5903, done.
remote: Total 5903 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (5903/5903), 1.18 MiB | 2.21 MiB/s, done.
Resolving deltas: 100% (2501/2501), done.
Initialized empty Git repository in /home/rnelson0/git/profile/spec/fixtures/modules/apache/.git/
remote: Counting objects: 8038, done.
remote: Total 8038 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (8038/8038), 2.00 MiB | 2.26 MiB/s, done.
Resolving deltas: 100% (4954/4954), done.
Initialized empty Git repository in /home/rnelson0/git/profile/spec/fixtures/modules/concat/.git/
remote: Counting objects: 1547, done.
remote: Total 1547 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (1547/1547), 338.49 KiB, done.
Resolving deltas: 100% (733/733), done.
Initialized empty Git repository in /home/rnelson0/git/profile/spec/fixtures/modules/firewall/.git/
remote: Counting objects: 3632, done.
remote: Total 3632 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (3632/3632), 929.90 KiB, done.
Resolving deltas: 100% (1290/1290), done.
Initialized empty Git repository in /home/rnelson0/git/profile/spec/fixtures/modules/ntp/.git/
remote: Counting objects: 1550, done.
remote: Total 1550 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (1550/1550), 307.66 KiB, done.
Resolving deltas: 100% (744/744), done.
[rnelson0@build profile]$ rake spec_prep
[rnelson0@build profile]$

If there’s a clone in place, nothing happens, so be sure to run spec_clean when upstream modules change. Now target spec_standalone and you will see the tests complete as they did before.

As an aside, targets like lint may look in the spec folders, so unless you plan on maintaining the upstream modules (most will accept PRs, so feel free to submit some!), you might want to run spec_clean first or prepare for a visual onslaught.

Summary

Today we dipped our toes in the advanced puppet spec testing world. We set up an existing and a new module to use puppetlabs_spec_helper, we added external dependencies, created some new advanced tests, and learned a bit about rake along the way. We are now in a great position to develop our modules fully with test-driven development. This is a great time to start fleshing out your existing role and profile modules to achieve 100% test coverage, and all new roles and profiles can have tests developed first. Take some time to get started on your own modules and you can see the ongoing spec test development of my profile module as well.