Peter Souter showed me a recent PR for the puppet-approved jenkins module where he parallelized the rspec tests. When there are a large number of tests in separate files, it can take a lot of time when run in series. Parallelizing the tests MAY offer a speed improvement; in Peter’s case, it reduced the time by almost 50%. With a small number of tests, or when an overwhelming percentage of the tests are in a single file, there may be no benefit or even a decrease in performance, so be sure to test out its effects before committing to it. It’s a pretty simple change, but let’s look at it in some detail anyway.
Gemfile
In your Gemfile, you need to add one line:
gem 'parallel_tests'
You can put it in the main group or maybe in the development group, depending on your implementation. If you don’t know which is appropriate, put it in the default group!
Rakefile
At the top of your Rakefile, add two lines to your require section:
require 'parallel_tests' require 'parallel_tests/cli'
Somewhere below that, maybe at the end, add a new rake target or two. To create a parallel_spec target, add these lines:
desc "Parallel spec tests" task :parallel_spec do Rake::Task[:spec_prep].invoke ParallelTests::CLI.new.run('--type test -t rspec spec/classes spec/defines spec/unit spec/functions'.split) Rake::Task[:spec_clean].invoke end
This will make parallel_spec the parallel equivalent of the existing, serial spec target, which means it will run the spec_clean target afterward as well. You may wish to add a parallel_spec_standalone target without the spec_clean stanza, especially if spec_prep takes significant time in your environment.
Also note that the ParallelTest::CLI.new.run() argument includes all the possible spec test types and directories. If those directories do not exist, you should either make them or remove them from the Rakefile.
.travis.yml
Finally, change the script entry or entries in .travis.yml.
# For a controlrepo script: cd dist/profile && bundle exec rake parallel_spec # For a module repo script: bundle exec rake parallel_spec
Testing Performance
Now, test out the changes before you commit! As I mentioned, it can be equal or worse performance with small or excessively confined tests (though it’s kinda funny for the user time is larger than the real time), in italics. You’ll also notice some differences in the output, in bold. This is run on a small module repo, local_user.
$ time be rake spec Cloning into 'spec/fixtures/modules/stdlib'... remote: Counting objects: 452, done. remote: Compressing objects: 100% (305/305), done. remote: Total 452 (delta 145), reused 377 (delta 131), pack-reused 0 Receiving objects: 100% (452/452), 200.73 KiB | 0 bytes/s, done. Resolving deltas: 100% (145/145), done. /usr/bin/ruby -I/home/rnelson0/modules/local_user/vendor/ruby/gems/rspec-core-3.1.7/lib:/home/rnelson0/modules/local_user/vendor/ruby/gems/rspec-support-3.1.2/lib /home/rnelson0/modules/local_user/vendor/ruby/gems/rspec-core-3.1.7/exe/rspec --pattern spec/\{classes,defines,unit,functions,hosts,integration,types\}/\*\*/\*_spec.rb --color local_user using minimum params should contain User[rnelson0] with comment => "rnelson0", shell => "/bin/bash", home => "/home/rnelson0", groups => ["group1", "group2"] and password_max_age => 90 should not contain Group[rnelson0] should contain Exec[set rnelson0's password] managing all groups should contain User[rnelson0] with comment => "Rob Nelson" and groups => ["group1", "group2"] should contain Group[rnelson0] should contain Group[group1] should contain Group[group2] managing gid only should contain User[rnelson0] with comment => "Rob Nelson" and groups => ["group1", "group2"] should contain Group[rnelson0] should not contain Group[group1] should not contain Group[group2] set manage_groups to false should contain User[rnelson0] with comment => "rnelson0", groups => ["group1", "group2"] and password_max_age => 90 should not contain Group[rnelson0] should not contain Group[group1] should not contain Group[group2] manage_groups with invalid input should raise Puppet::Error using full params should contain User[rnelson0] with comment => "Rob Nelson", shell => "/bin/zsh", home => "/nfshome/rnelson0", groups => ["group1", "group2"], password_max_age => 120 and uid => 101 should contain Local_user::Ssh_authorized_keys[ssh-rsa AAAA...zwE1 rsa-key-20141029] should not contain Group[rnelson0] with a valid date for last_change should contain User[rnelson0] should not contain Group[rnelson0] with an invalid date for last_change should raise Puppet::Error local_user::ssh_authorized_keys default should contain Ssh_authorized_key[rnelson0_ssh-rsa_rsa-key-20141029] with user => "rnelson0", type => "ssh-rsa", key => "AAAA...zwE1" and name => "rsa-key-20141029" local_user::windows with defaults should contain User[rnelson0] with comment => "rnelson0", groups => ["Remote Desktop Users"] and password => "Microsoft1" with allow_rdp => false, admin => false, no groups should contain User[rnelson0] with comment => "rnelson0", groups => [] and password => "Microsoft1" with allow_rdp => true, admin => true, no groups should contain User[rnelson0] with comment => "rnelson0", groups => ["Administrators", "Remote Desktop Users"] and password => "Microsoft1" with allow_rdp => false, admin > true, no groups should contain User[rnelson0] with comment => "rnelson0", groups => ["Administrators"] and password => "Microsoft1" with allow_rdp => false, admin => true, rnelson0 group should contain User[rnelson0] with comment => "rnelson0", groups => ["rnelson0", "Administrators"] and password => "Microsoft1" with allow_rdp => true, admin => true, rnelson0 group should contain User[rnelson0] with comment => "rnelson0", groups => ["rnelson0", "Administrators", "Remote Desktop Users"] and password => "Microsoft1" with allow_rdp => false, admin => true, rnelson0 group should contain User[rnelson0] with comment => "rnelson0", groups => ["rnelson0"] and password => "Microsoft1" fail on non-windows systems should fail to compile and raise an error matching /Windows support only!/ Finished in 1.9 seconds (files took 1 second to load) 31 examples, 0 failures Coverage report generated for (1/2), (2/2), RSpec to /home/rnelson0/modules/local_user/coverage. 0.0 / 0.0 LOC (100.0%) covered. COVERAGE: 100.00% -- 0.0/0.0 lines in 0 files Total resources: 7 Touched resources: 7 Resource coverage: 100.00% real 0m5.662s user 0m4.416s sys 0m0.406s
$ time be rake parallel_spec Cloning into 'spec/fixtures/modules/stdlib'... remote: Counting objects: 452, done. remote: Compressing objects: 100% (305/305), done. remote: Total 452 (delta 145), reused 377 (delta 131), pack-reused 0 Receiving objects: 100% (452/452), 200.73 KiB | 0 bytes/s, done. Resolving deltas: 100% (145/145), done. 2 processes for 4 specs, ~ 2 specs per process local_user using minimum params local_user::ssh_authorized_keys default should contain Ssh_authorized_key[rnelson0_ssh-rsa_rsa-key-20141029] with user => "rnelson0", type => "ssh-rsa", key => "AAAA...zwE1" and name => "rsa-key-20141029" local_user::windows with defaults should contain User[rnelson0] with comment => "rnelson0", groups => ["Remote Desktop Users"] and password => "Microsoft1" with allow_rdp => false, admin => false, no groups should contain User[rnelson0] with comment => "rnelson0", shell => "/bin/bash", home => "/home/rnelson0", groups => ["group1", "group2"] and password_max_age => 90 should contain User[rnelson0] with comment => "rnelson0", groups => [] and password => "Microsoft1" with allow_rdp => true, admin => true, no groups should contain User[rnelson0] with comment => "rnelson0", groups => ["Administrators", "Remote Desktop Users"] and password => "Microsoft1" with allow_rdp => false, admin > true, no groups should contain User[rnelson0] with comment => "rnelson0", groups => ["Administrators"] and password => "Microsoft1" with allow_rdp => false, admin => true, rnelson0 group should contain User[rnelson0] with comment => "rnelson0", groups => ["rnelson0", "Administrators"] and password => "Microsoft1" with allow_rdp => true, admin => true, rnelson0 group should contain User[rnelson0] with comment => "rnelson0", groups => ["rnelson0", "Administrators", "Remote Desktop Users"] and password => "Microsoft1" with allow_rdp => false, admin => true, rnelson0 group should contain User[rnelson0] with comment => "rnelson0", groups => ["rnelson0"] and password => "Microsoft1" fail on non-windows systems should fail to compile and raise an error matching /Windows support only!/ Finished in 1.4 seconds (files took 1.38 seconds to load) 9 examples, 0 failures Coverage report generated for (1/2), (2/2), RSpec to /home/rnelson0/modules/local_user/coverage. 0.0 / 0.0 LOC (100.0%) covered. COVERAGE: 100.00% -- 0.0/0.0 lines in 0 files should not contain Group[rnelson0] should contain Exec[set rnelson0's password] managing all groups should contain User[rnelson0] with comment => "Rob Nelson" and groups => ["group1", "group2"] should contain Group[rnelson0] should contain Group[group1] should contain Group[group2] managing gid only Total resources: 2 Touched resources: 2 Resource coverage: 100.00% should contain User[rnelson0] with comment => "Rob Nelson" and groups => ["group1", "group2"] should contain Group[rnelson0] should not contain Group[group1] should not contain Group[group2] set manage_groups to false should contain User[rnelson0] with comment => "rnelson0", groups => ["group1", "group2"] and password_max_age => 90 should not contain Group[rnelson0] should not contain Group[group1] should not contain Group[group2] manage_groups with invalid input should raise Puppet::Error using full params should contain User[rnelson0] with comment => "Rob Nelson", shell => "/bin/zsh", home => "/nfshome/rnelson0", groups => ["group1", "group2"], password_max_age => 120 and uid => 101 should contain Local_user::Ssh_authorized_keys[ssh-rsa AAAA...zwE1 rsa-key-20141029] should not contain Group[rnelson0] with a valid date for last_change should contain User[rnelson0] should not contain Group[rnelson0] with an invalid date for last_change should raise Puppet::Error Finished in 2.18 seconds (files took 1.38 seconds to load) 22 examples, 0 failures Coverage report generated for (1/2), (2/2), RSpec to /home/rnelson0/modules/local_user/coverage. 0.0 / 0.0 LOC (100.0%) covered. COVERAGE: 100.00% -- 0.0/0.0 lines in 0 files 31 examples, 0 failures Took 4 seconds real 0m7.064s user 0m9.964s sys 0m0.964s
Because the tests are parallelized, two processes are used and the 2nd process finishes first and inserts its output (the large bold section) right in the middle of the other process’s output. This is normal, but it can be a little surprising if you’re not expecting it!
Because this module is so small small, we actually do have worse performance, 5.6s vs 7.0s. That’s pretty negligible so it’s not a big deal if you leave it in place. On a controlrepo with ~100 examples, tests ran about 5% speed improvement and against one with ~200 tests, about 30% faster. Your mileage may vary, but if you’re running up against a clock with your tests, this may be worth your effort. Thanks for the tip, Peter! Enjoy!
If running the puppet rspec tests, there has been a “rake parallel:spec[n]” command where n is the number of threads. This speeds up the puppet rspec tests quite a bit since there are a few timeout related tests that otherwise always block the other tests from running. On my 4 core machine I always use “rake parallel::spec[3]” to still have a core to work on when the tests are running…
Hey Rob, I also wrote about this a while back http://razorconsulting.com.au/parallelising-rspec-puppet.html. Maybe this should be merged into the puppetlabs_spec_helper?
That wouldn’t be a bad idea. As Henrik says, you can run spec in parallel, but not spec_standalone. It shouldn’t be the default, though, as in many cases (particularly with a small number of tests per file) it’s demonstrably slower.