Peter Souter showed me a recent PR for the puppet-approved jenkins module where he parallelized the rspec tests. When there are a large number of tests in separate files, it can take a lot of time when run in series. Parallelizing the tests MAY offer a speed improvement; in Peter’s case, it reduced the time by almost 50%. With a small number of tests, or when an overwhelming percentage of the tests are in a single file, there may be no benefit or even a decrease in performance, so be sure to test out its effects before committing to it. It’s a pretty simple change, but let’s look at it in some detail anyway.

Gemfile

In your Gemfile, you need to add one line:

gem 'parallel_tests'

You can put it in the main group or maybe in the development group, depending on your implementation. If you don’t know which is appropriate, put it in the default group!

Rakefile

At the top of your Rakefile, add two lines to your require section:

require 'parallel_tests'
require 'parallel_tests/cli'

Somewhere below that, maybe at the end, add a new rake target or two. To create a parallel_spec target, add these lines:

desc "Parallel spec tests"
task :parallel_spec do
  Rake::Task[:spec_prep].invoke
  ParallelTests::CLI.new.run('--type test -t rspec spec/classes spec/defines spec/unit spec/functions'.split)
  Rake::Task[:spec_clean].invoke
end

This will make parallel_spec the parallel equivalent of the existing, serial spec target, which means it will run the spec_clean target afterward as well. You may wish to add a parallel_spec_standalone target without the spec_clean stanza, especially if spec_prep takes significant time in your environment.

Also note that the ParallelTest::CLI.new.run() argument includes all the possible spec test types and directories. If those directories do not exist, you should either make them or remove them from the Rakefile.

.travis.yml

Finally, change the script entry or entries in .travis.yml.

# For a controlrepo
script: cd dist/profile && bundle exec rake parallel_spec

# For a module repo
script: bundle exec rake parallel_spec

Testing Performance

Now, test out the changes before you commit! As I mentioned, it can be equal or worse performance with small or excessively confined tests (though it’s kinda funny for the user time is larger than the real time), in italics. You’ll also notice some differences in the output, in bold. This is run on a small module repo, local_user.

$ time be rake spec
Cloning into 'spec/fixtures/modules/stdlib'...
remote: Counting objects: 452, done.
remote: Compressing objects: 100% (305/305), done.
remote: Total 452 (delta 145), reused 377 (delta 131), pack-reused 0
Receiving objects: 100% (452/452), 200.73 KiB | 0 bytes/s, done.
Resolving deltas: 100% (145/145), done.
/usr/bin/ruby -I/home/rnelson0/modules/local_user/vendor/ruby/gems/rspec-core-3.1.7/lib:/home/rnelson0/modules/local_user/vendor/ruby/gems/rspec-support-3.1.2/lib /home/rnelson0/modules/local_user/vendor/ruby/gems/rspec-core-3.1.7/exe/rspec --pattern spec/\{classes,defines,unit,functions,hosts,integration,types\}/\*\*/\*_spec.rb --color

local_user
  using minimum params
    should contain User[rnelson0] with comment => "rnelson0", shell => "/bin/bash", home => "/home/rnelson0", groups => ["group1", "group2"] and password_max_age => 90
    should not contain Group[rnelson0]
    should contain Exec[set rnelson0's password]
  managing all groups
    should contain User[rnelson0] with comment => "Rob Nelson" and groups => ["group1", "group2"]
    should contain Group[rnelson0]
    should contain Group[group1]
    should contain Group[group2]
  managing gid only
    should contain User[rnelson0] with comment => "Rob Nelson" and groups => ["group1", "group2"]
    should contain Group[rnelson0]
    should not contain Group[group1]
    should not contain Group[group2]
  set manage_groups to false
    should contain User[rnelson0] with comment => "rnelson0", groups => ["group1", "group2"] and password_max_age => 90
    should not contain Group[rnelson0]
    should not contain Group[group1]
    should not contain Group[group2]
  manage_groups with invalid input
    should raise Puppet::Error
  using full params
    should contain User[rnelson0] with comment => "Rob Nelson", shell => "/bin/zsh", home => "/nfshome/rnelson0", groups => ["group1", "group2"], password_max_age => 120 and uid => 101
    should contain Local_user::Ssh_authorized_keys[ssh-rsa AAAA...zwE1 rsa-key-20141029]
    should not contain Group[rnelson0]
  with a valid date for last_change
    should contain User[rnelson0]
    should not contain Group[rnelson0]
  with an invalid date for last_change
    should raise Puppet::Error

local_user::ssh_authorized_keys
  default
    should contain Ssh_authorized_key[rnelson0_ssh-rsa_rsa-key-20141029] with user => "rnelson0", type => "ssh-rsa", key => "AAAA...zwE1" and name => "rsa-key-20141029"

local_user::windows
  with defaults
    should contain User[rnelson0] with comment => "rnelson0", groups => ["Remote Desktop Users"] and password => "Microsoft1"
  with allow_rdp => false, admin => false, no groups
    should contain User[rnelson0] with comment => "rnelson0", groups => [] and password => "Microsoft1"
  with allow_rdp => true, admin => true, no groups
    should contain User[rnelson0] with comment => "rnelson0", groups => ["Administrators", "Remote Desktop Users"] and password => "Microsoft1"
  with allow_rdp => false, admin > true, no groups
    should contain User[rnelson0] with comment => "rnelson0", groups => ["Administrators"] and password => "Microsoft1"
  with allow_rdp => false, admin => true, rnelson0 group
    should contain User[rnelson0] with comment => "rnelson0", groups => ["rnelson0", "Administrators"] and password => "Microsoft1"
  with allow_rdp => true, admin => true, rnelson0 group
    should contain User[rnelson0] with comment => "rnelson0", groups => ["rnelson0", "Administrators", "Remote Desktop Users"] and password => "Microsoft1"
  with allow_rdp => false, admin => true, rnelson0 group
    should contain User[rnelson0] with comment => "rnelson0", groups => ["rnelson0"] and password => "Microsoft1"
  fail on non-windows systems
    should fail to compile and raise an error matching /Windows support only!/

Finished in 1.9 seconds (files took 1 second to load)
31 examples, 0 failures
Coverage report generated for (1/2), (2/2), RSpec to /home/rnelson0/modules/local_user/coverage. 0.0 / 0.0 LOC (100.0%) covered.

COVERAGE: 100.00% -- 0.0/0.0 lines in 0 files


Total resources:   7
Touched resources: 7
Resource coverage: 100.00%

real    0m5.662s
user    0m4.416s
sys     0m0.406s

 

$ time be rake parallel_spec
Cloning into 'spec/fixtures/modules/stdlib'...
remote: Counting objects: 452, done.
remote: Compressing objects: 100% (305/305), done.
remote: Total 452 (delta 145), reused 377 (delta 131), pack-reused 0
Receiving objects: 100% (452/452), 200.73 KiB | 0 bytes/s, done.
Resolving deltas: 100% (145/145), done.
2 processes for 4 specs, ~ 2 specs per process

local_user
  using minimum params

local_user::ssh_authorized_keys
  default
    should contain Ssh_authorized_key[rnelson0_ssh-rsa_rsa-key-20141029] with user => "rnelson0", type => "ssh-rsa", key => "AAAA...zwE1" and name => "rsa-key-20141029"

local_user::windows
  with defaults
    should contain User[rnelson0] with comment => "rnelson0", groups => ["Remote Desktop Users"] and password => "Microsoft1"
  with allow_rdp => false, admin => false, no groups
    should contain User[rnelson0] with comment => "rnelson0", shell => "/bin/bash", home => "/home/rnelson0", groups => ["group1", "group2"] and password_max_age => 90
    should contain User[rnelson0] with comment => "rnelson0", groups => [] and password => "Microsoft1"
  with allow_rdp => true, admin => true, no groups
    should contain User[rnelson0] with comment => "rnelson0", groups => ["Administrators", "Remote Desktop Users"] and password => "Microsoft1"
  with allow_rdp => false, admin > true, no groups
    should contain User[rnelson0] with comment => "rnelson0", groups => ["Administrators"] and password => "Microsoft1"
  with allow_rdp => false, admin => true, rnelson0 group
    should contain User[rnelson0] with comment => "rnelson0", groups => ["rnelson0", "Administrators"] and password => "Microsoft1"
  with allow_rdp => true, admin => true, rnelson0 group
    should contain User[rnelson0] with comment => "rnelson0", groups => ["rnelson0", "Administrators", "Remote Desktop Users"] and password => "Microsoft1"
  with allow_rdp => false, admin => true, rnelson0 group
    should contain User[rnelson0] with comment => "rnelson0", groups => ["rnelson0"] and password => "Microsoft1"
  fail on non-windows systems
    should fail to compile and raise an error matching /Windows support only!/

Finished in 1.4 seconds (files took 1.38 seconds to load)
9 examples, 0 failures
Coverage report generated for (1/2), (2/2), RSpec to /home/rnelson0/modules/local_user/coverage. 0.0 / 0.0 LOC (100.0%) covered.

COVERAGE: 100.00% -- 0.0/0.0 lines in 0 files

    should not contain Group[rnelson0]
    should contain Exec[set rnelson0's password]
  managing all groups
    should contain User[rnelson0] with comment => "Rob Nelson" and groups => ["group1", "group2"]
    should contain Group[rnelson0]
    should contain Group[group1]
    should contain Group[group2]
  managing gid only

Total resources:   2
Touched resources: 2
Resource coverage: 100.00%
    should contain User[rnelson0] with comment => "Rob Nelson" and groups => ["group1", "group2"]
    should contain Group[rnelson0]
    should not contain Group[group1]
    should not contain Group[group2]
  set manage_groups to false
    should contain User[rnelson0] with comment => "rnelson0", groups => ["group1", "group2"] and password_max_age => 90
    should not contain Group[rnelson0]
    should not contain Group[group1]
    should not contain Group[group2]
  manage_groups with invalid input
    should raise Puppet::Error
  using full params
    should contain User[rnelson0] with comment => "Rob Nelson", shell => "/bin/zsh", home => "/nfshome/rnelson0", groups => ["group1", "group2"], password_max_age => 120 and uid => 101
    should contain Local_user::Ssh_authorized_keys[ssh-rsa AAAA...zwE1 rsa-key-20141029]
    should not contain Group[rnelson0]
  with a valid date for last_change
    should contain User[rnelson0]
    should not contain Group[rnelson0]
  with an invalid date for last_change
    should raise Puppet::Error

Finished in 2.18 seconds (files took 1.38 seconds to load)
22 examples, 0 failures
Coverage report generated for (1/2), (2/2), RSpec to /home/rnelson0/modules/local_user/coverage. 0.0 / 0.0 LOC (100.0%) covered.

COVERAGE: 100.00% -- 0.0/0.0 lines in 0 files


31 examples, 0 failures

Took 4 seconds

real    0m7.064s
user    0m9.964s
sys     0m0.964s

Because the tests are parallelized, two processes are used and the 2nd process finishes first and inserts its output (the large bold section) right in the middle of the other process’s output. This is normal, but it can be a little surprising if you’re not expecting it!

Because this module is so small small, we actually do have worse performance, 5.6s vs 7.0s. That’s pretty negligible so it’s not a big deal if you leave it in place. On a controlrepo with ~100 examples, tests ran about 5% speed improvement and against one with ~200 tests, about 30% faster. Your mileage may vary, but if you’re running up against a clock with your tests, this may be worth your effort. Thanks for the tip, Peter! Enjoy!