If you maintain more than one Puppet module, you’ve probably spent some time aligning changes in your general setup by hand – rspec helpers, Gemfile, Rakefile, your travis config, etc. Once you have a third or a fourth module, you find that does not scale. Thankfully, there’s a great tool to help automate this: Modulesync.

How It Works

We’ll discuss how modulesync works first, so that we understand how to set it up. For instance, to perform a dry run, you would run msync update –noop in a modulesync configuration repo. This creates a directory modules and clones the managed modules into that directory. For each module, it then describes the diffs between the existing contents of the module and the config defined by the modulesync configuration repo. Once the changes are reviewed for accuracy, run msync update -m “Commit message here” and the diffs are applied to the default branch (master) with the commit message specified. By creating a modulesync.yml file, the default namespace and branch can be specified. The use of a different branch name allows you to create a PR to apply the changes.

You can also run it as msync update -f <string> and only apply changes against managed modules matching the string (i.e. msync update -f local_user to only apply changes to my local_user module). There are a few other options (such as –bump to bump the minor version in metadata.json on a run) that you can explore as you become proficient with modulesync.

Creating your Configuration Repository

The next step is to create a configuration repository for use with modulesync. To help, I’ve created a reference repository that provides a simple, but effective, setup. You can also review other configuration repos: puppetlabs, voxpupuli, and my own, which we’ll be reviewing here. A number of files are extremely basic and not deserving of much discussion: .gitignore, Gemfile, LICENSE, README.md, Rakefile. That leaves three files and a directory. First is modulesync.yml:

---
git_base: 'git@github.com:'
namespace: rnelson0
branch: modulesync
message: "Update from rnelson0's modulesync"

The git_base plus namespace should be the root URL for your module owner. As I use github, this turns into git@github.com:rnelson0. Modulesync will commit changes to the modulesync branch, and create it if needed, using the comment Update from rnelson0’s modulesync. Most of these are defaults that can be changed with command line arguments. That brings us to the managed_modules.yml file:

---
- puppet-certs
- puppet-domain_join
- puppet-hiera_resources
- puppet-local_user

If you combine these with git_base and namespace from above, you end up with something like git@github.com:rnelson0/puppet-certs.git, which just so happens to match the SSH clone URL on puppet-certs. That leaves the config_defaults.yml file:

.travis.yml:
  includes:
  - rvm: 1.9.3
    env: PUPPET_GEM_VERSION="~> 3.0" STRICT_VARIABLES=yes
  - rvm: 1.9.3
    env: PUPPET_GEM_VERSION="~> 3.0" STRICT_VARIABLES=yes FUTURE_PARSER=yes
  - rvm: 1.9.3
    env: PUPPET_GEM_VERSION="~> 4.0" STRICT_VARIABLES=yes
  - rvm: 2.1.0
    env: PUPPET_GEM_VERSION="~> 3.0" STRICT_VARIABLES=yes
  - rvm: 2.1.0
    env: PUPPET_GEM_VERSION="~> 3.0" STRICT_VARIABLES=yes FUTURE_PARSER=yes
  - rvm: 2.1.0
    env: PUPPET_GEM_VERSION="~> 4.0" STRICT_VARIABLES=yes
  - rvm: 2.3.0
    env: PUPPET_GEM_VERSION="~> 4.0" STRICT_VARIABLES="yes" CHECK=test
 allow_failures:
  - rvm: 2.3.0
    env: PUPPET_GEM_VERSION="~> 4.0" STRICT_VARIABLES="yes" CHECK=test

This should look somewhat familiar: it’s portions of a .travis.yml file inside of another YAML hash. To see how this works, we need to look at moduleroot/.travis.yml:

---
language: ruby
sudo: false
cache: bundler
notifications:
  email: false
branches:
  only:
    - master
bundler_args: --without development system_tests
before_install: rm Gemfile.lock || true
script: bundle exec rake test
matrix:
  fast_finish: true
<% @configs['includes'].each do |include| -%>
  - rvm: <%= include['rvm'] %>
    env: <%= include['env'] %>
<% end -%>
<% if @configs['allow_failures'] -%>
<%   @configs['allow_failures'].each do |allow_failures| -%>
  allow_failures:
    - rvm: <%= allow_failures['rvm'] %>
      env: <%= allow_failures['env'] %>
<%   end -%>
<% end -%>

Files in moduleroot/ are treating as ERB files with the contents of config_defaults.yml populating the @configs hash on a per-file basis. In other words, the top level .travis.yml hash of config_defaults.yml becomes the @configs hash when parsing moduleroot/.travis.yml. If you had a Rakefile section in the hash, it would become @configs when parsing moduleroot/Rakefile. I keep my modulesync config is pretty simple, but you can see some examples of this in the voxpupuli configs (hash and Rakefile).

Modulesyncing

With our config in place, we can try a noop run of modulesync. Here’s what that looks like when run against just my domain_join module (note that be is an alias to bundle exec):

[rnelson0@build03 modulesync_config:master]$ be msync update -f domain_join --noop
Syncing puppet-domain_join
Cloning repository fresh
Cloning from git@github.com:rnelson0/puppet-domain_join.git
Creating new branch modulesync
No config file under ./modules/puppet-domain_join/.sync.yml found, using default values
Using no-op. Files in puppet-domain_join may be changed but will not be committed.
Files changed:
diff --git a/.travis.yml b/.travis.yml
index 9648308..ab2c22c 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,18 +1,29 @@
 ---
-sudo: false
 language: ruby
-bundler_args: --without development system_tests
-before_install: rm Gemfile.lock || true
+sudo: false
+cache: bundler
+notifications:
+  email: false
 branches:
   only:
     - master
-notifications:
-  email: false
-rvm:
-  - 1.9.3
-  - 2.1.0
+bundler_args: --without development system_tests
+before_install: rm Gemfile.lock || true
 script: bundle exec rake test
-env:
-  - PUPPET_GEM_VERSION="~> 3.0" STRICT_VARIABLES=yes
-  - PUPPET_GEM_VERSION="~> 3.0" STRICT_VARIABLES=yes FUTURE_PARSER=yes
-  - PUPPET_GEM_VERSION="~> 4.0" STRICT_VARIABLES=yes
+matrix:
+  fast_finish: true
+  include:
+  - rvm: 1.9.3
+    env: PUPPET_GEM_VERSION="~> 3.0" STRICT_VARIABLES=yes
+  - rvm: 1.9.3
+    env: PUPPET_GEM_VERSION="~> 3.0" STRICT_VARIABLES=yes FUTURE_PARSER=yes
+  - rvm: 1.9.3
+    env: PUPPET_GEM_VERSION="~> 4.0" STRICT_VARIABLES=yes
+  - rvm: 2.1.0
+    env: PUPPET_GEM_VERSION="~> 3.0" STRICT_VARIABLES=yes
+  - rvm: 2.1.0
+    env: PUPPET_GEM_VERSION="~> 3.0" STRICT_VARIABLES=yes FUTURE_PARSER=yes
+  - rvm: 2.1.0
+    env: PUPPET_GEM_VERSION="~> 4.0" STRICT_VARIABLES=yes
+  - rvm: 2.3.0
+    env: PUPPET_GEM_VERSION="~> 4.0" STRICT_VARIABLES="yes" CHECK=test
+  allow_failures:
+    - rvm: 2.3.0
+      env: PUPPET_GEM_VERSION="~> 4.0" STRICT_VARIABLES="yes" CHECK=test
diff --git a/Gemfile b/Gemfile
index 7dc2945..49dad4f 100644
--- a/Gemfile
+++ b/Gemfile
@@ -4,12 +4,11 @@ group :test do
   gem "rake"
   gem "puppet", ENV['PUPPET_GEM_VERSION'] || '~> 4.0'
   gem "rspec", '< 3.2.0'
-  gem "rspec-puppet", :git => 'https://github.com/rodjek/rspec-puppet.git'
+  gem "rspec-puppet"
   gem "puppetlabs_spec_helper"
   gem "metadata-json-lint"
   gem "rspec-puppet-facts"
-  gem 'rubocop', '0.33.0'
-  gem 'simplecov', '>= 0.11.0'
+  gem 'simplecov'
   gem 'simplecov-console'

   gem "puppet-lint-absolute_classname-check"
@@ -18,7 +17,6 @@ group :test do
   gem "puppet-lint-version_comparison-check"
   gem "puppet-lint-classes_and_types_beginning_with_digits-check"
   gem "puppet-lint-unquoted_string-check"
-  gem 'puppet-lint-resource_reference_syntax'
 end

 group :development do
@@ -26,6 +24,7 @@ group :development do
   gem "travis-lint"
   gem "puppet-blacksmith"
   gem "guard-rake"
+  gem "parallel_tests"
 end

 group :system_tests do
diff --git a/Rakefile b/Rakefile
index fea73c5..57645e7 100644
--- a/Rakefile
+++ b/Rakefile
@@ -7,17 +7,16 @@ require 'puppet/vendor/semantic/lib/semantic' unless Puppet.version.to_f < 3.6
 require 'puppet-lint/tasks/puppet-lint'
 require 'puppet-syntax/tasks/puppet-syntax'
 require 'metadata-json-lint/rake_task'
-require 'rubocop/rake_task'
+require 'parallel_tests'
+require 'parallel_tests/cli'

 # These gems aren't always present, for instance
 # on Travis with --without development
 begin
   require 'puppet_blacksmith/rake_tasks'
-rescue LoadError # rubocop:disable Lint/HandleExceptions
+rescue LoadError
 end

-RuboCop::RakeTask.new
-
 exclude_paths = [
   "bundle/**/*",
   "pkg/**/*",
@@ -32,10 +31,11 @@ Rake::Task[:coverage].clear
 Rake::Task[:lint].clear

 PuppetLint.configuration.relative = true
-PuppetLint.configuration.disable_arrow_alignment
 PuppetLint.configuration.disable_80chars
 PuppetLint.configuration.disable_class_inherits_from_params_class
 PuppetLint.configuration.disable_class_parameter_defaults
+PuppetLint.configuration.disable_documentation
+PuppetLint.configuration.disable_single_quote_string_with_variables
 PuppetLint.configuration.fail_on_warnings = true

 PuppetLint::RakeTask.new :lint do |config|
@@ -54,11 +54,17 @@ task :contributors do
   system("git log --format='%aN' | sort -u > CONTRIBUTORS")
 end

+desc "Parallel spec tests"
+task :parallel_spec do
+  Rake::Task[:spec_prep].invoke
+  ParallelTests::CLI.new.run('--type test -t rspec spec/classes spec/defines spec/unit spec/functions'.split)
+  Rake::Task[:spec_clean].invoke
+end
+
 desc "Run syntax, lint, and spec tests."
 task :test => [
   :metadata_lint,
   :syntax,
   :lint,
-  :rubocop,
   :spec,
 ]
diff --git a/spec/acceptance/nodesets/centos-511-x64.yml b/spec/acceptance/nodesets/centos-511-x64.yml
index 155926d..ca14463 100644
--- a/spec/acceptance/nodesets/centos-511-x64.yml
+++ b/spec/acceptance/nodesets/centos-511-x64.yml
@@ -4,9 +4,6 @@ HOSTS:
       - master
     platform: el-5-x86_64
     box: puppetlabs/centos-5.11-64-nocm
-    box_url: https://vagrantcloud.com/puppetlabs/boxes/centos-5.11-64-nocm
     hypervisor: vagrant
-
 CONFIG:
-  log_level: verbose
   type: foss
diff --git a/spec/acceptance/nodesets/centos-66-x64.yml b/spec/acceptance/nodesets/centos-66-x64.yml
index 07843d5..214318a 100644
--- a/spec/acceptance/nodesets/centos-66-x64.yml
+++ b/spec/acceptance/nodesets/centos-66-x64.yml
@@ -4,8 +4,6 @@ HOSTS:
       - master
     platform: el-6-x86_64
     box: puppetlabs/centos-6.6-64-nocm
-    box_url: https://vagrantcloud.com/puppetlabs/boxes/centos-6.6-64-nocm
     hypervisor: vagrant
 CONFIG:
-  log_level: verbose
   type: foss
diff --git a/spec/acceptance/nodesets/debian-78-x64.yml b/spec/acceptance/nodesets/debian-78-x64.yml
index c4062fd..2baa693 100644
--- a/spec/acceptance/nodesets/debian-78-x64.yml
+++ b/spec/acceptance/nodesets/debian-78-x64.yml
@@ -4,9 +4,6 @@ HOSTS:
       - master
     platform: debian-7-amd64
     box: puppetlabs/debian-7.8-64-nocm
-    box_url: https://vagrantcloud.com/puppetlabs/boxes/debian-7.8-64-nocm
     hypervisor: vagrant
-
 CONFIG:
-  log_level: verbose
   type: foss
diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb
index f2e5985..d0c347e 100644
--- a/spec/spec_helper.rb
+++ b/spec/spec_helper.rb
@@ -9,7 +9,7 @@ require 'simplecov-console'
 SimpleCov.start do
   add_filter '/spec'
   add_filter '/vendor'
-  formatter SimpleCov::Formatter::MultiFormatter.new([
+  formatter SimpleCov::Formatter::MultiFormatter::new([
     SimpleCov::Formatter::HTMLFormatter,
     SimpleCov::Formatter::Console
   ])
Files added:
spec/acceptance/nodesets/ubuntu-server-1404-x64.yml
spec/acceptance/nodesets/debian-82-x64.yml
spec/acceptance/nodesets/centos-72-x64.yml
spec/acceptance/nodesets/ubuntu-server-1204-x64.yml
spec/acceptance/nodesets/centos-66-x64-pe.yml


--------------------------------

What you see will depend on how far off a given module is from the new standardized configuration. Review all the changes and your config before you proceed. You may need a few tweaks on your first try, especially. I won’t delve into it, but you can also use .sync.yml to add to or override some values in an individual single module, see an example here, if you have some non-standard configuration to preserve. Once your modulesync config repo is fine-tuned, you can run msync without the noop:

[rnelson0@build03 modulesync_config:master]$ be msync update -f domain_join
Syncing puppet-domain_join
Overriding any local changes to repositories in ./modules
Switching to branch modulesync
No config file under ./modules/puppet-domain_join/.sync.yml found, using default values

It’s much quieter, but if you go to your module’s page on GitHub, you’ll see that the new branch is available to be used in a PR. Create the PR using the web page or the hub gem. You can see an example in domain_join’s PR7.

Namespace Synchronization and Workflow Changes

Once you have your first module synchhornized, you are ready to synchronize your other modules by leaving off the -f <string> argument. When you make a change to the synchronized files now, you do it in the modulesync configuration repository, NOT in the module’s repository. Your code changes follow your normal workflow.

You may have noticed the modules/ directory, where each module is cloned. Be aware that these directories are, on their own, somewhat static. If you work out of those directories, you can keep the repositories up to date. If you do not, then you will want to either remove them entirely, or update them before running modulesync in the future, or you’ll receive errors because they are out of date:

[rnelson0@build03 modulesync_config:master]$ be msync update -f domain_join --noop
Syncing puppet-domain_join
Overriding any local changes to repositories in ./modules
Switching to branch modulesync
/home/rnelson0/modules/modulesync_config/vendor/ruby/gems/git-1.3.0/lib/git/lib.rb:937:in `command': git '--git-dir=/home/rnelson0/modules/modulesync_config/modules/puppet-domain_join/.git' '--work-tree=/home/rnelson0/modules/modulesync_config/modules/puppet-domain_join' pull 'origin' 'modulesync'  2>&1:fatal: Couldn't find remote ref modulesync (Git::GitExecuteError)
        from /home/rnelson0/modules/modulesync_config/vendor/ruby/gems/git-1.3.0/lib/git/lib.rb:747:in `pull'
        from /home/rnelson0/modules/modulesync_config/vendor/ruby/gems/git-1.3.0/lib/git/base.rb:351:in `pull'
        from /home/rnelson0/modules/modulesync_config/vendor/ruby/gems/modulesync-0.6.1/lib/modulesync/git.rb:56:in `block in pull'
        from /home/rnelson0/modules/modulesync_config/vendor/ruby/gems/modulesync-0.6.1/lib/modulesync/git.rb:49:in `chdir'
        from /home/rnelson0/modules/modulesync_config/vendor/ruby/gems/modulesync-0.6.1/lib/modulesync/git.rb:49:in `pull'
        from /home/rnelson0/modules/modulesync_config/vendor/ruby/gems/modulesync-0.6.1/lib/modulesync.rb:61:in `block in run'
        from /home/rnelson0/modules/modulesync_config/vendor/ruby/gems/modulesync-0.6.1/lib/modulesync.rb:57:in `each'
        from /home/rnelson0/modules/modulesync_config/vendor/ruby/gems/modulesync-0.6.1/lib/modulesync.rb:57:in `run'
        from /home/rnelson0/modules/modulesync_config/vendor/ruby/gems/modulesync-0.6.1/bin/msync:8:in `<top (required)>'
        from /home/rnelson0/modules/modulesync_config/vendor/ruby/bin/msync:23:in `load'
        from /home/rnelson0/modules/modulesync_config/vendor/ruby/bin/msync:23:in `<main>'

I suggest deciding whether you are working out of modules/ or not and ensure that you only keep one clone of your puppet modules locally to prevent confusion.

That about covers it! Let me know if you have any other tips on how to use modulesync. Thanks!