Continuous Integration is an important technique used in modern software development. For every change, a CI system runs a suite of tests to ensure the whole system – not just the changed portion – still “works”, or more specifically, still passes the defined tests. We are going to look at Travis CI, a cloud-based Continuous Integration service that you can connect to your GitHub repositories. This is valuable because it’s free (for “best effort” access; there are paid plans as well.) and helps you guarantee that code you check in will work with Puppet. This isn’t a substitute or replacement for rspec-puppet, this is another layer of testing that improves the quality of our work.

There are plenty of other CI systems out there – Jenkins and Bamboo are popular – but that would involve setting up the CI system as well as configuring our repo to use CI. Please feel free to investigate these CI systems, but they’ll remain beyond the scope of this blog for the time being. Please share any guides you may have in the comments, though!

Travis CI works by spinning up a VM or docker instance, cloning our git repo (using tokenized authentication), and running the command(s) we provide. Each entry in our test matrix will run on a separate node, so we can test different OSes or Ruby or Puppet versions to our heart’s content. The results of the matrix are visible through GitHub and show us red if any test failed and green if all tests passed. We’ll look at some details of how this works as we set up Travis CI.

From a workflow perspective, you’ll continue to create branches on your controlrepo and submit PRs. The only additional step is that when a PR is ready for review, you’ll want to wait for Travis CI to complete first. If it’s red, investigate the failure and remediate it. Don’t review code until everything is green because it won’t work anyway. This will mostly be a time saver, unless you’re watching your CI run which of course makes it slower!

Register for Travis CI

Travis CI works very simply through the use of a .travis.yml file in the root of your repository plus some registration options at Travis CI. Register for public repos at https://travis-ci.org/getting_started and private repos at https://magnum.travis-ci.com/getting_started. Once you’re registered, click on your avatar in the top right and choose Account Settings (URL: https://travis-ci.org/profile/<your github ID>). Find your controlrepo and move the slider to the green check box. Click on the gear and decide whether you want every push to run a CI build or just Pull Requests. I suggest using only PRs for now, especially if you may need a pay plan – the first 100 builds are free, best to limit those during setup.

Add Travis CI configuration to your repository

Now we need to add some Travis CI configuration to our repository. We need to update or create a few files: .travis.yml, Gemfile, dist/profile/Rakefile, .gitignore. If we have a dist/profile/Gemfile, we can git mv that to the root of our repository, as we won’t need it in dist/profile anymore. We’ll work with the Travis CI config last, starting with our more familiar files first.

I originally used my own or severely modified files, such as a Rakefil without the rubocop portion. In hindsight, I found the files from puppet-module-skeleton (.gitignore, Gemfile, Rakefile, .travis.yml) to be more useful in the long term. Copy the files into the appropriate location and modify as needed.

The .gitignore file needs these three entries (below), in addition to any existing entries. The new entries are based around the use of bundler by ourselves and Travis CI to install ruby gem dependencies. The lock file tracks what dependencies have been made available, vendor is the directory users will store those dependencies in, and .bundle is the directory bundler uses for its own management. If these were present in the repo, bundler would be confused:

Gemfile.lock
vendor
.bundle

Next, let’s look at the Gemfile. This is where we describe the ruby gem dependencies for the project. You can learn more about bundler, puppet and rspec dependencies in this CodingBee tutorial. This sample Gemfile is a great base for Puppet code but can always be enhanced. This ensures that the gems we need for rspec-puppet and puppet-lint are present. Remember, this Gemfile needs to be present at the root of your repository, not in dist/profile.

source 'https://rubygems.org'

group :development, :test do
  gem 'json', :require => false
  gem 'metadata-json-lint', :require => false
  gem 'puppetlabs_spec_helper', :require => false
  gem 'puppet-lint', :require => false
  gem 'rake', :require => false
  gem 'rspec-puppet', :require => false
end

if puppetversion = ENV['PUPPET_GEM_VERSION']
  gem 'puppet', puppetversion, :require => false
else
  gem 'puppet', :require => false
end

# vim:ft=ruby

Next, we need a Rakefile. This is similar to a Makefile, but for Ruby (and rake instead of make), and something we’ve been using for a while without actually calling it out. When we run rake spec, we’re using the spec target in the Rakefile. You can use the Rakefile that you get with puppet module generate, or you can use the one from Gareth’s puppet-module-skeleton, or anything in between. As a side effect, this gets us up to the latest version of the best practices rspec-puppet setup!

require 'rubygems'
require 'bundler/setup'

require 'puppetlabs_spec_helper/rake_tasks'
require 'puppet/version'
require 'puppet/vendor/semantic/lib/semantic' unless Puppet.version.to_f < 3.6
require 'puppet-lint/tasks/puppet-lint'
require 'puppet-syntax/tasks/puppet-syntax'
require 'metadata-json-lint/rake_task'

# These gems aren't always present, for instance
# on Travis with --without development
begin
  require 'puppet_blacksmith/rake_tasks'
rescue LoadError # rubocop:disable Lint/HandleExceptions
end

exclude_paths = [
  "bundle/**/*",
  "pkg/**/*",
  "vendor/**/*",
  "spec/**/*.pp",
]

Rake::Task[:lint].clear

PuppetLint.configuration.relative = true
PuppetLint.configuration.disable_80chars
PuppetLint.configuration.disable_arrow_alignment
PuppetLint.configuration.disable_class_inherits_from_params_class
PuppetLint.configuration.disable_class_parameter_defaults
PuppetLint.configuration.fail_on_warnings = true

PuppetLint::RakeTask.new :lint do |config|
  config.ignore_paths = exclude_paths
end

PuppetSyntax.exclude_paths = exclude_paths

desc "Run acceptance tests"
RSpec::Core::RakeTask.new(:acceptance) do |t|
  t.pattern = 'spec/acceptance'
end

desc "Populate CONTRIBUTORS file"
task :contributors do
  system("git log --format='%aN' | sort -u > CONTRIBUTORS")
end

desc "Run syntax, lint, and spec tests."
task :test => [
  :metadata_lint,
  :syntax,
  :lint,
  :spec,
]

With the Gemfile and Rakefile in place, we’re going to need to use bundler ourselves rather than simply calling rspec-puppet, so this is a good time to test our setup. You may need to set the environment variable PUPPET_GEM_VERSION if your code and modules will not work with the latest puppet version. I haven’t ensured that all the code is future parser compatible, so I am falling back to Puppet v3.7.3:

[rnelson0@build02 controlrepo:travisci±]$ PUPPET_GEM_VERSION=3.7.3 bundle install --path vendor --without system_tests
Resolving dependencies...
Using rake 10.4.2
Using CFPropertyList 2.2.8
Using diff-lcs 1.2.5
Using facter 2.4.4
Using json_pure 1.8.2
Using hiera 1.3.4
Using json 1.8.3
Using metaclass 0.0.4
Using spdx-licenses 1.0.0
Using metadata-json-lint 0.0.11
Using mocha 1.1.0
Using puppet 3.7.3
Using puppet-lint 1.1.0
Using puppet-syntax 2.0.0
Using rspec-support 3.3.0
Using rspec-core 3.3.2
Using rspec-expectations 3.3.1
Using rspec-mocks 3.3.2
Using rspec 3.3.0
Using rspec-puppet 2.2.0
Using puppetlabs_spec_helper 0.10.3
Using bundler 1.7.6
Your bundle is complete!
Gems in the group system_tests were not installed.
It was installed into ./vendor
[rnelson0@build02 controlrepo:travisci±]$ cd dist/profile
[rnelson0@build02 profile:travisci±]$ bundle exec rake test
ruby -c lib/facter/puppet_role.rb
Syntax OK
---> syntax:manifests
---> syntax:templates
---> syntax:hiera:yaml
metadata-json-lint metadata.json
HEAD is now at 710b3d5 Merge pull request #268 from mhaskel/1.2.0-prep
/usr/local/rvm/rubies/ruby-1.9.3-p551/bin/ruby -I/home/rnelson0/git/puppetinabox/controlrepo/vendor/ruby/1.9.1/gems/rspec-core-3.3.2/lib:/home/rnelson0/git/puppetinabox/controlrepo/vendor/ruby/1.9.1/gems/rspec-support-3.3.0/lib /home/rnelson0/git/puppetinabox/controlrepo/vendor/ruby/1.9.1/gems/rspec-core-3.3.2/exe/rspec --pattern spec/\{classes,defines,unit,functions,hosts,integration\}/\*\*/\*_spec.rb --color

profile::apache
  with defaults for all parameters
    should contain Class[profile::apache]
    should contain Package[httpd]
    should contain User[apache]

profile::base
  with defaults for all parameters
    should contain Class[profile::base]
    should contain Class[profile::linuxfw]
    should contain Class[ssh::server]
    should contain Class[ssh::client]
    should contain Class[ntp]
    should contain Ssh_authorized_key[testkey]
    should contain Yumrepo[lab]
    should contain Exec[shosts.equiv]
    should contain Class[sudo]
    should contain Sudo::Conf[wheel]
    should contain Local_user[testuser]

profile::build
  with defaults for all parameters
    should contain Class[profile::build]
    should contain Class[rvm]
    should contain Rvm_system_ruby[ruby-1.9.3-p511]
    should contain Rvm_gem[test] with require => "Rvm_system_ruby[ruby-1.9.3-p511]"
  using puppet enterprise
    should not contain Class[rvm]

profile::dhcp
  with defaults for all parameters
    should contain Class[profile::dhcp]
    should contain Package[dhcp]
    should contain Dhcp::Server::Subnet[10.0.0.0]
    should contain Dhcp::Server::Host[sample]

profile::dns
  with defaults for all parameters
    should contain Class[profile::dns]
    should contain Package[bind]
    should contain Bind::Server::Conf[/etc/named.conf]
    should contain Bind::Server::File[named.test]

profile::hiera
  with defaults for all parameters
    should contain Class[profile::hiera]
    should contain Package[hiera]
    should contain Package[hiera-puppet]

profile::linuxfw::post
  with defaults for all parameters
    should contain Class[profile::linuxfw::post]
    should contain Firewall[998 input reject]
    should contain Firewall[999 forward reject]

profile::linuxfw::pre
  with defaults for all parameters
    should contain Class[profile::linuxfw::pre]
    should contain Firewall[000 accept related established rules]
    should contain Firewall[001 accept all icmp]
    should contain Firewall[002 accept all to lo interface]
    should contain Firewall[003 accept ssh connections]

profile::linuxfw
  with defaults for all parameters
    should contain Class[profile::linuxfw]
    should contain Class[profile::linuxfw::pre]
    should contain Class[profile::linuxfw::post]
    should contain Class[firewall]

profile::mysql::client
  with defaults for all parameters
    should contain Class[profile::mysql::client]
    should contain Class[mysql::client]

profile::mysql::server
  with defaults for all parameters
    should contain Class[profile::mysql::server]
    should contain Package[policycoreutils-python]
    should contain Exec[set-mysql-selinux-context]
    should contain Lvm::Volume[lv_mysql] with pv => "/dev/sda3", fstype => "ext4" and size => "40G"
    should contain File[/data] with ensure => "directory"
    should contain Mount[/data] with ensure => "mounted", name => "/data", device => "/dev/mapper/vg_mysql-lv_mysql", fstype => "ext4" and atboot => "true"
    should contain File[/data/mysql] with ensure => "directory"
    should contain Exec[enforce-mysql-selinux-context]
    should contain Class[mysql::server]
    should contain Class[mysql::server::backup]
    should contain Firewall[100 MySQL inbound] with dport => "3306", proto => "tcp" and action => "accept"
    should contain Mysql_user[someuser@localhost]
    should contain Mysql_grant[someuser@localhost/somedb.*]
    should contain Mysql_database[somedb] with charset => "utf8"

profile::phpmyadmin
  with dbpass and cname
    should contain Class[profile::phpmyadmin]
    should contain Package[phpMyAdmin]
    should contain Selboolean[httpd_can_network_connect_db]
    should contain Selboolean[httpd_can_network_connect]
    should contain Certs::Vhost[phpmyadmin.example.com]
    should contain Apache::Vhost[phpmyadmin.example.com]
    should contain Class[apache::mod::php]
    should contain File[/etc/httpd/conf.d/phpMyAdmin.conf] with ensure => "absent"
    should contain File[/etc/phpMyAdmin] with ensure => "directory" and mode => "0755"
    should contain File[config.inc.php] with mode => "0644"

profile::puppetdb
  with defaults for all parameters
    should contain Class[profile::puppetdb]
    should contain Class[puppetdb]
    should contain Class[puppetdb::master::config]

profile::rcfiles::bash
  with defaults for all parameters
    should contain Class[profile::rcfiles::bash]
    should contain File[/etc/bashrc.puppet]
    should contain File_line[bashrc_skel_puppet_source]

profile::rcfiles::vim
  with defaults for all parameters
    should contain Class[profile::rcfiles::vim]
    should contain Package[vim-enhanced]
    should contain Vcsrepo[/usr/share/vim/puppet]
    should contain File_line[vimrc_runtimepath]
    should contain File_line[vimrc_indent]
    should contain File_line[vimrc_shiftwidth]

profile::tftp
  with defaults for all parameters
    should contain Class[profile::tftp]
    should contain Package[tftpd-hpa]
    should contain Tftp::File[pxelinux.0]
    should contain Firewall[100 tftp requests] with dport => "69", proto => "udp" and action => "accept"

profile::yumrepo
  with defaults for all parameters
    should contain Class[profile::yumrepo]
    should contain File[/repodir]
    should contain Createrepo[testrepo]
    should contain Apache::Vhost[yum.example.com]

Finished in 11.91 seconds (files took 0.55361 seconds to load)
88 examples, 0 failures

You should not see any real differences between your previous rspec results and your current results. If you do, check to make sure you used the right version of the puppet gem with bundle exec gem list. Puppet 4’s future parser can wreck havoc on older code and you’ll often see this with failures against the spec fixtures. Here’s what happens when rnelson0/local_user is parsed by Puppet 4:

  10) profile::base with defaults for all parameters should contain Sudo::Conf[wheel]
      Failure/Error: it { is_expected.to contain_sudo__conf('wheel') }
      Puppet::PreformattedError:
        Evaluation Error: Error while evaluating a Function Call, 0 is not a string.  It looks to be a Fixnum at /home/rnelson0/git/
puppetinabox/controlrepo/dist/profile/spec/fixtures/modules/local_user/manifests/init.pp:50:5 on node build02.nelson.va

If it’s not your module, you’re SOL. If it is your module, we’ll look at adding Travis CI to our modules soon enough. For today, just re-run bundle install with a prepend of the right puppet version.

Of course, if there are some issues in your controlrepo code itself, not the fixtures, and you can fix it easily, go ahead and get it done. Continue when the rake tests all complete without errors.

Finally, let’s create a .travis.yml with the tests we want to run and our support matrix:

---
sudo: false
language: ruby
bundler_args: --without system_tests
script: "cd dist/profile && bundle exec rake test"
matrix:
  fast_finish: true
  include:
  - rvm: 1.9.3
    env: PUPPET_GEM_VERSION="~> 3.0"
  - rvm: 2.1.6
    env: PUPPET_GEM_VERSION="~> 3.0"
  #- rvm: 2.1.6
  #  env: PUPPET_GEM_VERSION="~> 3.0" FUTURE_PARSER="yes"
  #- rvm: 2.1.6
  #  env: PUPPET_GEM_VERSION="~> 4.0" STRICT_VARIABLES="yes"
notifications:
  email: false

The script statement is the commands we ran for testing. You can update this as necessary. Remember that Travis CI will start in the root of your repo. If you have other modules you want to test, chain on some cd commands for those directories and tests.

The matrix setting is where we pair Ruby versions with Puppet versions. Use whatever pairings are appropriate for you. I’ve commented out the future parser versions since my code definitely does not work with Puppet 4 yet, but I certainly intend to turn it on sometime. I also don’t care about Ruby 1.8.7 or Puppet older than 3.x. This part is up to you, do whatever makes sense. There’s also a good config file in puppet-module-skeleton, unsurprisingly.

The rest of the file is defaults I’ve seen used elsewhere. Standing on the shoulders of giants and all that. You can find more details on the configuration file at Travis CI’s documentation center.

Test your changes

Next, commit your changes, push them upstream, and create a PR. Within a few seconds, you’ll see the Travis CI integration kick off. If everything went well, you’ll see green as soon as all your builds complete!

The PR I used to create this article is PR50. If you click details next to the Travis CI build, you can dive deeper. In that build, you can then dive into a matrix entry. That’s as deep as you go, this isn’t Inception. If a test fails, drill down into the specific build and check the results. You’ll find the error that resulted from the script command most of the time, but sometimes you’ll find that Travis CI had an issue. If that happens, you can go back to the build view and click the Restart Build button, a clockwise arrow icon in the upper-right area of the screen, to try again.

Once your PR status shows green for all the checks, it’s time to do your normal code review and then merge the pull request. Remember, don’t merge anything that’s red!

Enabling Travis CI on a new module

We got the hard one out of the way, so this is really simple: Install puppet-module-skeleton, generate your new module with puppet module generate, and your new module is ready out of the box to use with Travis CI! Just go into settings on Travis CI and enable it for the repo. Voila!

Summary

By enabling Travis CI, we’ve added some simple but valuable continuous integration checks to our puppet controlrepo. By setting up our test matrix, we can easily test our code in a number of environments and get rapid feedback before we even begin the code review. Hopefully this helps you with your puppet coding. We’ll look at adding CI to our modules soon!