Convert a controlrepo to using the Puppet Development Kit (PDK)

I previously wrote about converting an individual puppet module’s repo to use the Puppet Development Kit. We can also convert controlrepos to use the PDK. I am starting with a “traditional” controlrepo, described here, as well as centralized tests, described here. To follow this article directly, you need to:

  • Have all hiera data and role/profile/custom modules in the /dist directory
  • Have all tests, for all modules, in the /spec directory

If your controlrepo looks different, this article can be used to migrate to the PDK, but you will have to modify some of the sections a bit.

This will be a very lengthy blog post (over 4,000 words!) and covers a very time-consuming process. It took me about 2 full days to work through this on my own controlrepo. Hopefully, this article helps you shave significant time off the effort, but don’t expect a very quick change.

Managing the PDK itself

First, let’s make sure we have a profile to install and manage the PDK. As we use the role/profile pattern, we create a class profile::pdk with the parameter version, which we can specify in hiera as profile::pdk::version: ‘1.7.1’ (current version as of this writing). This profile can then be added to an appropriate role class, like role::build for a build server, or applied directly to your laptop. I use only Enterprise Linux 7, but we could certainly flush this out to support multiple OSes:

# dist/profile/manifests/pdk.pp
class profile::pdk (
  String $version = 'present',
) {
  package {'puppet6-release':
    ensure => present,
    source => "",
  package {'pdk':
    ensure => $version,
    require => Package['puppet6-release'],

# spec/classes/profile/pdk_spec.rb
require 'spec_helper'
describe 'profile::pdk' do
  on_supported_os.each do |os, facts|
    next unless facts[:kernel] == 'Linux'
    context "on #{os}" do
      let (:facts) {
          :clientcert => 'build',

      it { compile.with_all_deps }

      it { contain_package('puppet6-release') }
      it { contain_package('pdk') }

Once this change is pushed upstream and the build server (or other target node) checks in, the PDK is available:

$ pdk --version

Now we are almost ready to go. Of course, we need to start with good, working tests! If any tests are currently failing, we need to get them to a passing state before continuing, like this:

Finished in 3 minutes 12.2 seconds (files took 1 minute 17.89 seconds to load)
782 examples, 0 failures

With everything in a known good state, we can then be sure that any failures are related to the PDK changes, and only the PDK changes.

Setting up the PDK Template

The PDK comes with a set of pre-packaged templates. It is recommended to stick with a set of templates designed for the current PDK version for stability. However, the templates are online and may updated without an accompanying PDK release. We may choose to stick with the on-disk templates, we may point to the online templates from Puppet, or we may create our own! For those working with the the on-disk templates, you can skip down to working with .sync.yml

To another template, we use the pdk convert --template-url. If this is our own template, we should make sure the latest commit is compliant with the PDK version we are using. If we point to Puppet’s templates, we essentially shift to the development track. Make sure you understand this before changing the templates. We can get back to using the on-disk template with the url file:///opt/puppetlabs/pdk/share/cache/pdk-templates.git, though, so this isn’t a decision we have to live with forever. Here’s the command to switch to the official Puppet templates:

$ pdk convert --template-url=

------------Files to be added-----------

----------Files to be modified----------


You can find a report of differences in convert_report.txt.

pdk (INFO): Module conversion is a potentially destructive action. Ensure that you have committed your module to a version control system or have a backup, and review the changes above before continuing.
Do you want to continue and make these changes to your module? Yes

------------Convert completed-----------

6 files added, 7 files modified.

Now, everyone’s setup is probably a little different and thus we cannot predict the entirety of the changes each of us must make, but there are some minimal changes everyone must make. The file .sync.yml can be created to allow each of us to override the template defaults without having to write our own templates. The layout of the YAML starts with the filename the changes will modify, followed by the appropriate config section and then the value(s) for that section. We can find the appropriate config section by looking at the template repo’s erb templates. For instance, I do not use AppVeyor, GitLab, or Travis with this controlrepo, so to have git ignore them, I made the following changes to the .gitignore‘s required hash:

$ cat .sync.yml
    - 'appveyor.yml'
    - '.gitlab-ci.yml'
    - '.travis.yml'

When changes are made to the sync file, they must be applied with the pdk update command. We can see that originally, these unused files were to be committed, but now they are properly ignored:

$ git status
# On branch pdk
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#       modified:   .gitignore
#       modified:   .rspec
#       modified:   Gemfile
#       modified:   Rakefile
#       modified:   metadata.json
#       modified:   spec/default_facts.yml
#       modified:   spec/spec_helper.rb
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#       .gitlab-ci.yml
#       .pdkignore
#       .rubocop.yml
#       .sync.yml
#       .travis.yml
#       .yardopts
#       appveyor.yml
no changes added to commit (use "git add" and/or "git commit -a")

$ cat .sync.yml
    - 'appveyor.yml'
    - '.gitlab-ci.yml'
    - '.travis.yml'

$ pdk update
pdk (INFO): Updating mss-controlrepo using the template at, from 1.7.1 to 1.7.1

----------Files to be modified----------


You can find a report of differences in update_report.txt.

Do you want to continue and make these changes to your module? Yes

------------Update completed------------

1 files modified.

$ git status
# On branch pdk
# Changes not staged for commit:
# (use "git add <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working directory)
# modified: .gitignore
# modified: .rspec
# modified: Gemfile
# modified: Rakefile
# modified: metadata.json
# modified: spec/default_facts.yml
# modified: spec/spec_helper.rb
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
# .pdkignore
# .rubocop.yml
# .sync.yml
# .yardopts
no changes added to commit (use "git add" and/or "git commit -a")

Anytime we pdk update, we will still receive new versions of the ignored files, but they won’t be committed to the repo and a git clean or a clean checkout will remove them.

After initial publication, I was made aware that you can completely delete or unmanage a file using delete:true or unmanage:true, as described here, rather than using .gitignore.

We may need to implement other overrides, except that we do not know what they would be yet, so let’s commit our changes so far. Then we can start working on validation or unit tests. It doesn’t really matter which we choose to work on first, though my preference is validation first as it does not depend on the version of Puppet we are testing.

PDK Validate

The PDK validation check, pdk validate, will check the syntax and style of metadata.json and any task json files, syntax and style of all puppet files, and ruby code style. This is roughly equivalent to our old bundle exec rake syntax task. Since the bundle setup is a wee bit old and the PDK is kept up to date, we shouldn’t be surprised if what was passing before now has failures. Here’s a sample of the errors I encountered on my first run – there were hundreds of them:

$ pdk validate
pdk (INFO): Running all available validators...
pdk (INFO): Using Ruby 2.5.1
pdk (INFO): Using Puppet 6.0.2
[✔] Checking metadata syntax (metadata.json tasks/*.json).
[✔] Checking module metadata style (metadata.json).
[✔] Checking Puppet manifest syntax (**/**.pp).
[✔] Checking Puppet manifest style (**/*.pp).
[✖] Checking Ruby code style (**/**.rb).
info: task-metadata-lint: ./: Target does not contain any files to validate (tasks/*.json).
warning: puppet-lint: dist/eyaml/manifests/init.pp:43:12: indentation of => is not properly aligned (expected in column 14, but found it in column 12)
warning: puppet-lint: dist/eyaml/manifests/init.pp:51:11: indentation of => is not properly aligned (expected in column 12, but found it in column 11)
warning: puppet-lint: dist/msswiki/manifests/init.pp:56:12: indentation of => is not properly aligned (expected in column 13, but found it in column 12)
warning: puppet-lint: dist/msswiki/manifests/init.pp:57:10: indentation of => is not properly aligned (expected in column 13, but found it in column 10)
warning: puppet-lint: dist/msswiki/manifests/init.pp:58:11: indentation of => is not properly aligned (expected in column 13, but found it in column 11)
warning: puppet-lint: dist/msswiki/manifests/init.pp:59:10: indentation of => is not properly aligned (expected in column 13, but found it in column 10)
warning: puppet-lint: dist/msswiki/manifests/init.pp:60:12: indentation of => is not properly aligned (expected in column 13, but found it in column 12)
warning: puppet-lint: dist/msswiki/manifests/rsync.pp:37:140: line has more than 140 characters
warning: puppet-lint: dist/msswiki/manifests/rsync.pp:43:140: line has more than 140 characters
warning: puppet-lint: dist/profile/manifests/access_request.pp:21:3: optional parameter listed before required parameter
warning: puppet-lint: dist/profile/manifests/access_request.pp:22:3: optional parameter listed before required parameter

We can control puppet-lint Rake settings in .sync.yml – but it only works for rake tasks. pdk validate will ignore it because puppet-lint isn’t invoked via rake. The same settings need to be put in .puppet-lint.rc in the proper format. That file is not populated via pdk, so just create it by hand. I don’t care about the arrow alignment or 140 characters checks, so I’ve added the appropriate lines to both files and re-run pdk update. We all have difference preferences, just make sure they are reflected in both locations:

$ cat .sync.yml
    - 'appveyor.yml'
    - '.gitlab-ci.yml'
    - '.travis.yml'
    - '140chars'
    - 'arrow_alignment'
$ cat .puppet-lint.rc
$ grep disable Rakefile

Now we can use pdk validate and see a lot fewer violations. We can try to automatically correct the remaining violations with pdk validate -a, which will also try to auto-fix other syntax violations, or pdk bundle exec rake lint_fix, which restricts fixes to just puppet-lint. Not all violations can be auto-corrected, so some may still need fixed manually. I also found I had a .rubocop.yml in a custom module’s directory causing rubocop failures, because apparently rubocop parses EVERY config file it finds no matter where it’s located, and had to remove it to prevent errors. It may take you numerous tries to get through this. I recommend fixing a few things and committing before moving on to the next set of violations, so that you can find your way back if you make mistakes. Here’s a command that can help you edit all the files that can’t be autofixed by puppet-lint or rubocop (assuming you’ve already completed an autofix attempt):

vi $(pdk validate | egrep "(puppet-lint|rubocop)" | awk '{print $3}' | awk -F: '{print $1}' | sort | uniq | xargs)

Alternatively, you can disable rubocop entirely if you want by adding the following to your .sync.yml. If you are only writing spec tests, this is probably fine, but if you are writing facts, types, and providers, I do not suggest it.

  selected_profile: off

We have quite a few methods to fix all the possible errors that come our way. Once we have fixed everything, we can move on to the Unit Tests. We will re-run validation again after the unit tests, to ensure any changes we make for unit tests do not introduce new violations.

Unit Tests

Previously, we used bundle exec rake spec to run unit tests. The PDK way is pdk test unit. It performs pretty much the same, but it does collect all the output before displaying it, so if you have lots of fixtures and tests, you won’t see any output for a good long while and then bam, you get it all at once. The results will probably be just a tad overwhelming at first:

$ pdk test unit
pdk (INFO): Using Ruby 2.5.1
pdk (INFO): Using Puppet 6.0.2
[✔] Preparing to run the unit tests.
[✖] Running unit tests.
  Evaluated 782 tests in 110.76110479 seconds: 700 failures, 0 pending.
failed: rspec: ./spec/classes/profile/base__junos_spec.rb:11: Evaluation Error: Error while evaluating a Resource Statement, Unknown resource type: 'cron' (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/profile/manifests/base/junos.pp, line: 15, column: 3) on node build
  profile::base::junos with defaults for all parameters should contain Cron[puppetrun]
    context 'with defaults for all parameters' do
      it { create_class('profile::base::junos') }
      it { create_cron('puppetrun') }

failed: rspec: ./spec/classes/profile/base__linux_spec.rb:12: Evaluation Error: Error while evaluating a Resource Statement, Unknown resource type: 'sshkey' (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/ssh/manifests/hostkeys.pp, line: 13, column: 5) on node build
  profile::base::linux on redhat-6-x86_64 disable openshift selinux policy should contain Selmodule[openshift-origin] with ensure => "absent"
        if (facts[:os]['family'] == 'RedHat') && (facts[:os]['release']['major'] == '6')
          context 'disable openshift selinux policy' do
            it { contain_selmodule('openshift-origin').with_ensure('absent') }
            it { contain_selmodule('openshift').with_ensure('absent') }
failed: rspec: ./spec/classes/profile/base__linux_spec.rb:162: Evaluation Error: Error while evaluating a Resource Statement, Unknown resource type: 'cron' (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/os_patching/manifests/init.pp, line: 113, column: 3) on node build
  profile::base::linux on redhat-6-x86_64 when managing OS patching should contain Class[os_patching]

          it { contain_class('os_patching') }
          if (facts[:os]['family'] == 'RedHat') && (facts[:os]['release']['major'] == '7')
            it { contain_package('yum-utils') }

failed: rspec: ./spec/classes/profile/base__linux_spec.rb:18: error during compilation: Evaluation Error: Unknown variable: '::sshdsakey'. (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/ssh/manifests/hostkeys.pp, line: 12, column: 6) on node build
  profile::base::linux on redhat-7-x86_64 with defaults for all parameters should compile into a catalogue without dependency cycles

        context 'with defaults for all parameters' do
          it { compile.with_all_deps }

          it { create_class('profile::base::linux') }

Whoa. Not cool. From 728 working tests to 700 failures is quite the explosion! And they blew up on missing resource types that are built-in to Puppet. What happened? Puppet 6, that’s what! However…

Fix Puppet 5 Tests First

When I ran pdk convert, it updated my metadata.json to specify it supported Puppet versions 4.7.0 through 6.x because I was missing any existing requirements section. The PDK defaults to using the latest Puppet version your metadata supports. Whoops! It’s okay, we can test against Puppet 5, too. I recommend that we get our existing tests working with the version of Puppet we wrote them for, just to get back to a known good state. We don’t want to be troubleshooting too many changes at once.

There are two ways to specify the version to use. There’s the CLI envvar PDK_PUPPET_VERSION that accepts a simple number like 5 or 6, which is preferred for automated systems like CI/CD, rather than humans. You can also use --puppet-version or --pe-version to set an exact version. I’m an old curmudgeon, so I’m using the non-preferred envvar setting today, but Puppet recommends using the actual program arguments! Regardless of how you specify the version, the PDK changes not just the Puppet version, but which version of Ruby it uses:

$ PDK_PUPPET_VERSION='5' pdk test unit
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.6
[✔] Preparing to run the unit tests.
[✖] Running unit tests.
  Evaluated 782 tests in 171.447295254 seconds: 519 failures, 0 pending.
failed: rspec: ./spec/classes/msswiki/init_spec.rb:24: error during compilation: Evaluation Error: Error while evaluating a Function Call, You must provide a hash of packages for wiki implementation. (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/msswiki/manifests/init.pp, line: 109, column: 5) on node build
  msswiki on redhat-6-x86_64  when using default params should compile into a catalogue without dependency cycles

          it { compile.with_all_deps }

          it { create_class('msswiki') }

failed: rspec: ./spec/classes/profile/apache_spec.rb:31: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Empty string title at 0. Title strings must have a length greater than zero. (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/concat/manifests/setup.pp, line: 59, column: 10) (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/apache/manifests/init.pp, line: 244) on node build
  profile::apache on redhat-7-x86_64 with additional listening ports should contain Firewall[100 Inbound apache listening ports] with dport => [80, 443, 8088]

          it {
   contain_firewall('100 Inbound apache listening ports').with(dport: [80, 443, 8088])

failed: rspec: ./spec/classes/profile/base__linux_spec.rb:12: Evaluation Error: Unknown variable: '::sshecdsakey'. (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/ssh/manifests/hostkeys.pp, line: 36, column: 6) on node build
  profile::base::linux on redhat-6-x86_64 disable openshift selinux policy should contain Selmodule[openshift-origin] with ensure => "absent"
        if (facts[:os]['family'] == 'RedHat') && (facts[:os]['release']['major'] == '6')
          context 'disable openshift selinux policy' do
            it { contain_selmodule('openshift-origin').with_ensure('absent') }
            it { contain_selmodule('openshift').with_ensure('absent') }

Some of us may be lucky to make it through without errors here, but I assume most of us encounter at least a few failures, like I did – “only” 519 compared to 700 before. Don’t worry, we can fix this! To help us focus a bit, we can run tests on individual spec files using pdk bundle exec rspec <filename> (remembering to specify PDK_PUPPET_VERSION or to export the variable). Everyone has different problems here, but there are some common failures, such as missing custom facts:

  69) profile::base::linux on redhat-7-x86_64 when managing OS patching should contain Package[yum-utils]
      Failure/Error: include ::ssh::hostkeys

        Evaluation Error: Unknown variable: '::sshdsakey'. (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/ssh/manifests/hostkeys.pp, line: 12, column: 6) on node build

Nate McCurdy commented that instead of calling rspec directly, you can pass a comma-separated list of files with --tests​, e.g. pdk test unit --tests path/to/spec/file.rb,path/to/spec/file2.rb

I defined my custom facts in spec/spec_helper.rb. That has definitely changed. Here’s part of the diff from running pdk convert:

$ git diff origin/production spec/spec_helper.rb
diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb
index de3e7e6..5e721b7 100644
--- a/spec/spec_helper.rb
+++ b/spec/spec_helper.rb
@@ -1,45 +1,44 @@
 require 'puppetlabs_spec_helper/module_spec_helper'
 require 'rspec-puppet-facts'

-add_custom_fact :concat_basedir, '/dne'
-add_custom_fact :is_pe, true
-add_custom_fact :root_home, '/root'
-add_custom_fact :pe_server_version, '2016.4.0'
-add_custom_fact :selinux, true
-add_custom_fact :selinux_config_mode, 'enforcing'
-add_custom_fact :sshdsakey, ''
-add_custom_fact :sshecdsakey, ''
-add_custom_fact :sshed25519key, ''
-add_custom_fact :pe_version, ''
-add_custom_fact :sudoversion, '1.8.6p3'
-add_custom_fact :selinux_agent_vardir, '/var/lib/puppet'
+include RspecPuppetFacts
+default_facts = {
+  puppetversion: Puppet.version,
+  facterversion: Facter.version,

+default_facts_path = File.expand_path(File.join(File.dirname(__FILE__), 'default_facts.yml'))
+default_module_facts_path = File.expand_path(File.join(File.dirname(__FILE__), 'default_module_facts.yml'))

Instead of modifying spec/spec_helper.rb, facts should go in spec/default_facts.yml and spec/default_module_facts.yml. As the former is modified by pdk update, it is easier to maintain the later. Review the diff of spec/spec_helper.rb and spec/default_facts.yml (if we have the latter) for our previous custom facts and their values. When a test is failing for a missing fact, we can add it to spec/default_module_facts.yml in the format factname: “factvalue”.

  1) profile::base::linux on redhat-6-x86_64 disable openshift selinux policy should contain Selmodule[openshift-origin] with ensure => "absent"
     Failure/Error: include concat::setup

       Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Empty string title at 0. Title strings must have a length greater than zero. (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/concat/manifests/setup.pp, line: 59, column: 10) (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/ssh/manifests/server/config.pp, line: 12) on node  build

This is related to an older version of puppetlabs/concat (v1.2.5). The latest is v5.1.0. After updating my Puppetfile and .fixtures.yml with the new version, I ran pdk bundle exec rake spec_prep to update the test fixtures, and this is resolved.

  2) profile::base::linux on redhat-6-x86_64 domain_join is true should contain Class[profile::domain_join]
     Failure/Error: include domain_join

       Evaluation Error: Error while evaluating a Function Call, Class[Domain_join]:
         expects a value for parameter 'domain_fqdn'
         expects a value for parameter 'domain_shortname'
         expects a value for parameter 'ad_dns'
         expects a value for parameter 'register_account'
         expects a value for parameter 'register_password' (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/profile/manifests/domain_join.pp, line: 8, column: 3) on node build

In this case, the mandatory parameters for an included class were not provided. The let (:params) block of our rspec contexts only allows us to set parameters for the current class. We have been pulling these parameters from hiera instead. This data should be coming from spec/fixtures/hieradata/default.yml, however, hiera lookup settings were also removed in spec/spec_helper.rb, breaking the existing hiera configuratioN:

 RSpec.configure do |c|
-  c.hiera_config = File.expand_path(File.join(__FILE__, '../fixtures/hiera.yaml'))
-  default_facts = {
-    puppetversion: Puppet.version,
-    facterversion: Facter.version
-  }

There is no replacement setting provided. The PDK was designed with hiera use in mind, so add this line (replacing the bold filename if yours is stored elsewhere) to .sync.yml and run pdk update. Your hiera config should start working again:

  hiera_config: 'spec/fixtures/hiera.yaml'

These are just some common issues with a PDK conversion, but we may have others to resolve. We just need to keep iterating until we get through everything.

Things are looking good! But we are just done with the Puppet 5 rodeo. Before we move on to Puppet 6, now is a good time to make sure syntax validation works, and if you need to make changes to syntax, you then run the Puppet 5 tests that you know should work. Get that all settled before moving on.

Puppet 6 Unit Tests

There’s one big change to be aware of before we even run unit tests against Puppet 6. To make updating core types easier, without requiring a brand new release of Puppet, a number of types were moved into dedicated modules. This means that for Puppet 6 testing, we need to update our Puppetfile and .fixtures.yml (though the puppet agent all-in-one package packages these modules, we do not went our tests relying on an installed puppet agent). When we update these files, we need to make sure we ONLY deploy these core modules on Puppet 6, not Puppet 5- both for the master and testing – or we will encounter issues with Puppet 5. The Puppetfile is actually ruby code, so we can check the version before loading the modules (see note below), and .fixtures.yml accepts a puppet_version parameter to modules. We can click on each module name here to get the link for the replacement module. We do not have to add all of the modules, just the ones we use, but including the ones we are likely to use or have other modules depend on can reduce friction. The changes will look like this:

# Puppetfile
require 'puppet'
# as of puppet 6 they have removed several core modules into seperate modules
if Puppet.version =~ /^6\.\d+\.\d+/
  mod 'puppetlabs-augeas_core', '1.0.3'
  mod 'puppetlabs-cron_core', '1.0.0'
  mod 'puppetlabs-host_core', '1.0.1'
  mod 'puppetlabs-mount_core', '1.0.2'
  mod 'puppetlabs-sshkeys_core', '1.0.1'
  mod 'puppetlabs-yumrepo_core', '1.0.1'

# .fixtures.yml
      repo: "puppetlabs/augeas_core"
      ref: "1.0.3"
      puppet_version: ">= 6.0.0"
      repo: "puppetlabs/cron_core"
      ref: "1.0.0"
      puppet_version: ">= 6.0.0"
      repo: "puppetlabs/host_core"
      ref: "1.0.1"
      puppet_version: ">= 6.0.0"
      repo: "puppetlabs/mount_core"
      ref: "1.0.2"
      puppet_version: ">= 6.0.0"
      repo: "puppetlabs/scheduled_task"
      ref: "1.0.0"
      puppet_version: ">= 6.0.0"
      repo: "puppetlabs/selinux_core"
      ref: "1.0.1"
      puppet_version: ">= 6.0.0"
      repo: "puppetlabs/sshkeys_core"
      ref: "1.0.1"
      puppet_version: ">= 6.0.0"
      repo: "puppetlabs/yumrepo_core"
      ref: "1.0.1"
      puppet_version: ">= 6.0.0"

Note: Having performed an actual upgrade to Puppet 6 now, I do NOT recommend adding the modules to the Puppetfile after all, unless you are specifying a newer version of the modules than is provided with the version of puppet-agent you are using, or you are not using the AIO versions, and ONLY if you have no Puppet 5 agents. Puppet 5 agents connecting to a Puppet 6 master will pluginsync these modules and throw errors instead of applying a catalog. If you do have multiple compile masters, you could conceivably keep a few running Puppet 5 and only have Puppet 5 agents connect to it, but that seems like a really specific and potentially problematic scenario, so in general, I repeat, I do NOT recommend adding the core modules to the Puppetfile. They must be placed in the .fixtures.yml file for testing, though.

Now give pdk test unit a try and see how it behaves. All the missing types will be back, so any errors we see now should be related to actual failures, or some other edge case I did not experience.

Note: I experienced the error Error: Evaluation Error: Error while evaluating a Function Call, undefined local variable or method `created' for Puppet::Pops::Loader::RubyLegacyFunctionInstantiator:Class when running my Puppet 6 tests immediately after working on the Puppet 5 tests. Nothing I found online could resolve this and when I returned to it later, it worked fine. I could not replicate the error, so I am unsure of what caused it. If you run into that error, I suggest starting a new session and running git clean -ffdx to remove unmanaged files, so that you start with a clean environment.

Updating CI/CD Integrations

Once both pdk validate and pdk test unit complete without error, we need to update the automated checks our CI/CD system uses. We all use different systems, but thankfully the PDK has many of us covered. For those who use Travis CI, Appveyor, or Gitlab, there are pre-populated .travis.yml.appveyor.yml, and .gitlab-ci.yml files, respectively. For those of us who use Jenkins, we have two options: 1) Copy one of these CI settings and integrate them into our build process (traditional or pipeline) or 2) apply profile::pdk to the Jenkins node and use the PDK for tests. Let’s look at the first option, basing it off the Travis CI config:

  - bundle -v
  - rm -f Gemfile.lock
  - gem update --system
  - gem --version
  - bundle -v
  - 'bundle exec rake $CHECK'
bundler_args: --without system_tests
  - 2.5.0
  fast_finish: true
      env: CHECK="syntax lint metadata_lint check:symlinks check:git_ignore check:dot_underscore check:test_file rubocop"
      env: CHECK=parallel_spec
      env: PUPPET_GEM_VERSION="~> 5.0" CHECK=parallel_spec
      rvm: 2.4.4
      env: PUPPET_GEM_VERSION="~> 4.0" CHECK=parallel_spec
      rvm: 2.1.9

We have to merge this into something useful for Jenkins. I am unfamiliar with Pipelines myself (I know, it’s the future, but I have $reasons!), but I have built a Jenkins server with RVM installed and configured it with a freestyle job. Here’s the current job:

[[ -s /usr/local/rvm/scripts/rvm ]] && source /usr/local/rvm/scripts/rvm
# Use the correct ruby
rvm use 2.1.9

git clean -ffdx
bundle install --path vendor --without system_tests
bundle exec rake test

The new before_install section cleans up the local directory, equivalent to git clean -ffdx, but it also spits out some version information and runs gem update. These are optional, and the latter is only helpful if you have your gems cached elsewhere (the git clean will wipe the updated gems otherwise, wasting time). The bundler_args are already part of the bundle install command. The rvm version varies by puppet version, that will need tweaked. The test command is now bundle exec rake $CHECK, with a variety of checks added in the matrix section. test used to do everything in the first 2 matrix sections; parallel_spec just runs multiple tests at once instead of in serial which can be faster. The 3rd and 4th matrix sections are for older puppet versions. We can put this together into multiple jobs, into a single job that tests multiple versions of Puppet, or into a single job testing just one Puppet version. Here’s what a Jenkins job would look like that tests Puppet 6 and 5:

[[ -s /usr/local/rvm/scripts/rvm ]] && source /usr/local/rvm/scripts/rvm

# Puppet 6
export PUPPET_GEM_VERSION="~> 6.0"
rvm use 2.5.0
bundle -v
git clean -ffdx
# Comment the next line if you do not have gems cached outside the job workspace
gem update --system
gem --version
bundle -v

bundle install --path vendor --without system_tests
bundle exec rake syntax lint metadata_lint check:symlinks check:git_ignore check:dot_underscore check:test_file rubocop
bundle exec rake parallel_spec

# Puppet 5
rvm use 2.4.4
export PUPPET_GEM_VERSION="~> 5.0"
bundle exec rake parallel_spec

This creates parity with the pre-defined CI tests for other services.

The other option is adding profile::pdk to the Jenkins server, probably through the role, and use the PDK to run tests. That Jenkins freestyle job looks a lot simpler:

echo -n "PDK Version: "
$PDK --version

# Puppet 6
git clean -ffdx
$PDK validate
$PDK test unit

# Puppet 5
git clean -ffdx

This is much simpler, and it should not need updated until removing Puppet 5 or adding Puppet 7 when it is released, whereas the RVM version in the bundle-version may need tweaked throughout the Puppet 6 lifecycle as Ruby versions change. However, the tests aren’t exactly the same. Currently, pdk validate does not run the rake target check:git_ignore, and possibly other check: tasks. In my opinion, as the pdk improves, the benefit of only having to update the PDK package version and not the git-based automation outweighs the single missing check and the maintenance of RVM on a Jenkins server. And for those of us using Travis CI/Appveyor/Gitlab-CI, it definitely makes sense to stick with the provided test setup as it requires almost no maintenance.

I used this earlier without explaining it, but the PDK also provides the ability to run pdk bundle, similar to the native bundle but using the vendored ruby and gems provided by the PDK. We can run individual tests like pdk bundle exec rake check:git_ignore, or install the PDK and modify the “bundle” Jenkins job to recreate the bundler setup using the PDK and not have to worry about RVM at all. I’ll leave that last as an exercise for the user, though.

We must be sure to review our entire Puppet pull request process and see what other integrations need updated, and of course we must update documentation for our colleagues. Remember, documentation is part of “done”, so we cannot say we are done until we update it.

Finally, with all updates in place, submit a Pull Request for your controlrepo changes. This Pull Request must go through the new process, not just to verify that it passes, but to identify any configuration steps you missed or did not document properly.


Today, we looked at converting our controlrepo to use the Puppet Development Kit for testing instead of bundler-based testing. It required lots of changes to our controlrepos, many of which the PDK handled for us via autocorrect; others involved manual updates. We reviewed a variety of changes required for CI/CD integrations such as Travis CI or Jenkins. We reviewed the specifics of our setup that others don’t share so we had a working setup top to bottom, and we updated our documentation so all of our colleagues can make use of the new setup. Finally, we opened a PR with our changes to validate the new configuration.

By using the PDK, we have leveraged the hard work of many Puppet employees and Puppet users alike who provide a combination of rigorously vetted sets of working dependencies and beneficial practices, and we can continue to improve those benefits by simply updating our version of the PDK and templates in the future. This is a drastic reduction in the mental load we all have to carry to keep up with Puppet best practices, an especially significant burden on those responsible for Puppet in their CI/CD sysytems. I would like to thank everyone involved with the PDK, including David Schmitt, Lindsey Smith, Tim Sharpe, Bryan Jen, Jean Bond, and the many other contributors inside and outside Puppet who made the PDK possible.

Linux OS Patching with Puppet Tasks

One of the biggest gaps in most IT security policies is a very basic feature, patching. Specific numbers vary, but most surveys show a majority of hacks are due to unpatched vulnerabilities. Sadly, in 2018, automatic patching on servers is still out of the grasp of many, especially those running older OSes.

While there are a number of solutions out there from OS vendors (WSUS for Microsoft, Satellite for RHEL, etc.), I manage a number of OSes and the one commonality is that they are all managed by Puppet. A single solution with central reporting of success and failure sounds like a plan. I took a look at Puppet solutions and found a module called os_patching by Tony Green. I really like this module and what it has to offer, even though it doesn’t address all my concerns at this time. It shows a lot of promise and I suspect I will be working with Tony on some features I’d like to see in the future.

Currently, os_patching only supports Red Hat/Debian-based Linux distributions. Support is planned for Windows, and I know someone is looking at contributing to provide SuSE support. The module will collect information on patching that can be used for reporting, and patching is performed through a Task, either at the CLI or using the PE console’s Task pane.


Configuring your system to use the module is pretty easy. Add the module to your Puppetfile / .fixtures.yml, add a feature flag to your profile, and include os_patching behind the feature flag. Implement your tests and you’re good to go. Your only real decision is whether you default the feature flag to enabled or disabled. In my home network, I will enable it, but a production environment may want to disable it by default and enable it as an override through hiera. Because the fact collects data from the node, it will add a few seconds to each agent’s runtime, so be sure to include that in your calculation.

Adding the module is pretty simple, Here are the Puppetfile / .fixtures.yml diffs:

# Puppetfile
mod 'albatrossflavour/os_patching', '0.3.5'

# .fixtures.yml
      repo: "albatrossflavour/os_patching"
      ref: "0.3.5"

Next, we need an update to our tests. I will be adding this to my profile::base, so I modify that spec file. Add a test for the default feature flag setting, and one for the non-default setting. Flip the to and not_to if you default the feature flag to disabled. If you run the tests now, you’ll get a failure, which is expected since there is no supporting code in the class yet.(there is more to the test, I have only included the framework plus the next tests):

require 'spec_helper'
describe 'profile::base', :type => :class do
  on_supported_os.each do |os, facts|
    let (:facts) {

    context 'with defaults for all parameters' do
      it { contain_class('os_patching') }

    context 'with manage_os_patching enabled' do
      let (:params) do {
        manage_os_patching: false,

      # Disabled feature flags
      it { is_expected.not_to contain_class('os_patching') }

Finally, add the feature flag and feature to profile::base (the additions are in italics):

class profile::base (
  Hash    $sudo_confs = {},
  Boolean $manage_puppet_agent = true,
  Boolean $manage_firewall = true,
  Boolean $manage_syslog = true,
  Boolean $manage_os_patching = true,
) {
  if $manage_firewall {
    include profile::linuxfw

  if $manage_puppet_agent {
    include puppet_agent
  if $manage_syslog {
    include rsyslog::client
  if $manage_os_patching {
    include os_patching

Your tests will pass now. That’s all it takes! For any nodes where it is enabled, you will see a new fact and some scripts pushed down on the next run:

[rnelson0@build03 controlrepo:production]$ sudo puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/os_patching.rb]/ensure: defined content as '{md5}af52580c4d1fb188061e0c51593cf80f'
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for
Info: Applying configuration version '1535052836'
Notice: /Stage[main]/Os_patching/File[/etc/os_patching]/ensure: created
Info: /Stage[main]/Os_patching/File[/etc/os_patching]: Scheduling refresh of Exec[/usr/local/bin/]
Notice: /Stage[main]/Os_patching/File[/usr/local/bin/]/ensure: defined content as '{md5}af4ff2dd24111a4ff532504c806c0dde'
Info: /Stage[main]/Os_patching/File[/usr/local/bin/]: Scheduling refresh of Exec[/usr/local/bin/]
Notice: /Stage[main]/Os_patching/Exec[/usr/local/bin/]: Triggered 'refresh' from 2 events
Notice: /Stage[main]/Os_patching/Cron[Cache patching data]/ensure: created
Notice: /Stage[main]/Os_patching/Cron[Cache patching data at reboot]/ensure: created
Notice: Applied catalog in 54.18 seconds

You can now examine a new fact, os_patching, which will shows tons of information including the pending package updates, the number of packages, which ones are security patches, whether the node is blocked (explained in a bit), and whether a reboot is required:

[rnelson0@build03 controlrepo:production]$ sudo facter -p os_patching
  package_updates => [
  package_update_count => 300,
  security_package_updates => [
  security_package_update_count => 3,
  blocked => false,
  blocked_reasons => [],
  blackouts => {},
  pinned_packages => [],
  last_run => {},
  patch_window => "",
  reboots => {
    reboot_required => "unknown"

Additional Configuration

There are a number of other settings you can configure if you’d like.

  • patch_window: a string descriptor used to “tag” a group of machines, i.e. Week3 or Group2
  • blackout_windows: a hash of datetime start/end dates during which updates are blocked
  • security_only: boolean, when enabled only the security_package_updates packages and dependencies are updated
  • reboot_override: boolean, overrides the task’s reboot flag (default: false)
  • dpkg_options/yum_options: a string of additional flags/options to dpkg or yum, respectively

You can set these in hiera. For instance, my global config has some blackout windows for the next few years:

  'End of year 2018 change freeze':
    'start': '2018-12-15T00:00:00+1000'
    'end':   '2019-01-05T23:59:59+1000'
  'End of year 2019 change freeze':
    'start': '2019-12-15T00:00:00+1000'
    'end':   '2020-01-05T23:59:59+1000'
  'End of year 2020 change freeze':
    'start': '2020-12-15T00:00:00+1000'
    'end':   '2021-01-05T23:59:59+1000'
  'End of year 2021 change freeze':
    'start': '2021-12-15T00:00:00+1000'
    'end':   '2022-01-05T23:59:59+1000'

Patching Tasks

Once the module is installed and all of your agents have picked up the new config, they will start reporting their patch status. You can query nodes with outstanding patches using PQL. A search like inventory[certname] {facts.os_patching.package_update_count > 0 and facts.clientcert !~ 'puppet'} can find all your agents that have outstanding patches (except puppet – kernel patches require reboots and puppet will have a hard time talking to itself across a reboot). You can also select against a patch_window selection with and facts.os_patching.patch_window = "Week3" or similar. You can then provide that query to the command line task:

puppet task run os_patching::patch_server --query="inventory[certname] {facts.os_patching.package_update_count > 0 and facts.clientcert !~ 'puppet'}"

Or use the Console’s Task view to run the task against the PQL selection:

Add any other parameters you want in the dialog/CLI args, like setting rebootto true, then run the task. An individual job will be created for each node, all run in parallel. If you are selecting too many nodes for simultaneous runs, use additional filters, like the aforementioned patch_window or other facts (EL6 vs EL7, Debian vs Red Hat), etc. to narrow the node selection [I blew up my home lab, which couldn’t handle the CPU/IO lab, when I ran it against all systems the first time, whooops!]. When the job is complete, you will get your status back for each node as a hash of status elements and the corresponding values, including return (success or failure), reboot, packages_updated, etc. You can extract the logs from the Console or pipe CLI logs directly to jq (json query) to analyze as necessary.


Patching for many of us requires additional automation and reporting. The relatively new puppet module os_patching provides helpful auditing and compliance information alongside orchestration tasks for patching. Applying a little Puppet Query Language allows you to update the appropriate agents on your schedule, or to pull the compliance information for any reporting needs, always in the same format regardless of the (supported) OS. Currently, this is restricted to Red Hat/Debian-based Linux distributions, but there are plans to expand support to other OSes soon. Many thanks to Tony Green for his efforts in creating this module!

Using Puppet Enterprise 2018’s new backup/restore features

I was pretty excited when I read the new features in Puppet Enterprise 2018.1. There are a lot of cool new features and fixes, but the backup/restore feature stood out for me. Even with just 5 VMs at home, I don’t want to rock the boat when rebuilding my master by losing my CA or agent certs, much less with a lot more managed nodes at work, and all the little bootstrap requirements have changed since I started using PE in 2014. Figuring out how to get everything running myself would be possible, but it would take a while and be out of date in a few months anyway. Then there is everything in PuppetDB that I do not want to lose, like collected facts/resources and run reports.

Not coincidentally, I still had a single CentOS 6 VM around because it was my all-in-one puppet master, and migrating to CentOS 7 was not something I looked forward to due to the anticipated work it would require. With the release of this feature, I decided to get off my butt and do the migration. It still took over a month to make it happen, between other work, and I want to share my experience in the hope it saves someone else a bit of pain.

Create your upgrade outline

I want to summarize the plan at a really high level, then dive in a bit deeper. Keep in mind that I have a single all-in-one master using r10k and my plan does not address multi-master or split deployments. Both of those deployment models have significantly different upgrade paths, please be careful if you try and map this outline onto those models without adjusting. For the all-in-one master, it’s pretty simple:

  • Backup old master
  • Deploy a new master VM running EL7
  • Complete any bootstrapping that isn’t part of the backup
  • Install the same version of PE
  • Restore the old master’s backup onto the new master
  • Run puppet
  • Point agents at the new master

I will cover the backup/restore steps at the end, so the first step to cover is deploying a new master. This part sounds simple, but if Puppet is currently part of your provisioning process and you only have one master, you’ve got a catch 22 situation – new deployments must talk to puppet to complete without errors, and if you deploy a new puppet master using the same process, it will either fail to communicate with itself since PE is not installed, or it will talk to a PE installation that does not reflect your production environment. We need to make sure that we have the ability to provision without puppet, or be prepared for some manual efforts in the deploy. With a single master, manual efforts aren’t that burdensome, but can still reduce accuracy, which is why I prefer a modified automated provisioning workflow.

A lot of bootstrapping – specifically hiera and r10k/code manager – should be handled by the restore. There were just a few things I needed to do:

  • Run ssh-keygen/install an existing key and attach that key to the git system. You can avoid this by managing the ssh private/public keys via file resources, but you will not be able to pull new code until puppet processes that resource.
  • SSH to your git server and accept the key. You can avoid this with the sshkey resource, with the same restriction.
  • Check your VMs default iptables/selinux posture. I suggest managing security policy via puppet, which should prevent remote agents from connecting before the first puppet run, but it’s also possible to prevent the master from communicating with itself with the wrong default policy.
  • Check the hostname matches your expectations. All of /etc/hosts, /etc/hostname, /etc/sysconfig/network should list the short and FQDN properly, and hostname; hostname -f should return the same values. /etc/resolv.conf may also need the search domain. Fix any issues before installing PE, as certs are generated during install, and having the wrong hostname result can cause cascading faults best addressed by starting over.

The restore should get the rest from the PE side of things. If your provisioning automation performs other work that you had to skip, make sure you address it now, too.

Installing PE is probably the one manual step you cannot avoid. You can go to and find links to current and past PE versions. Make sure you get the EL7 edition and not the EL6 edition. I did not check with Support, but I assume that you must restore on the same version you backed up, I would not risk even a patch release difference.

Skipping the restore brings us to running the agent, a simple puppet agent -t on the master, or waiting 30 minutes for the run to complete on its own.

The final step may not apply to your situation. In addition to refreshing the OS of the master, I switched to a new hostname. If you’re dropping your new master on top of the existing one’s hostname/IP, you can skip this step. I forked a new branch from production called mastermigration. The only change in this branch is to set the server value in /etc/puppetlabs/puppet/puppet.conf. There are a number of ways to do this, I went with a few ini_setting resources and a flag manage_puppet_conf in my profile::base::linux. The value should only be in one of the sections main or agent, so I ensured it is in main and absent elsewhere:

  if $manage_puppet_conf {
    # These settings are very useful during migration but are not needed most of the time
    ini_setting { 'puppet.conf main server':
      ensure => present,
      path => '/etc/puppetlabs/puppet/puppet.conf',
      section => 'main',
      setting => 'server',
      value => '',
    ini_setting { 'puppet.conf agent server':
      ensure => absent,
      path => '/etc/puppetlabs/puppet/puppet.conf',
      section => 'agent',
      setting => 'server',

During the migration, I can just set profile::base::linux::manage_puppet_conf: true in hiera for the appropriate hosts, or globally, and they’ll point themselves at the new master. Later, I can set it to false if I don’t want to continue managing it (while there is no reason you cannot leave the flag enabled, by leaving it as false normally you can ensure that changing the server name here does not take effect unless purposefully flip the flag; you could also parameterize the server name).

Now let’s examine the new feature that makes it go.

Backups and Restores

Puppet’s documentation on the backup/restore feature provides lots of detail. It will capture the CA and certs, all your currently deployed code, your PuppetDB contents including facts, and almost all of your PE config. About the only thing missing are some gems, which you should hopefully be managing and installing with puppet anyway.

Using the new feature is pretty simple, puppet-backup createor puppet-backup restore <filename> will suffice for this effort. There are a few options for more fine-grained control, such as backup/restore of individual scopes with --scope=<scopes>[,<additionalscopes>...], e.g. --scope=certs.


The backup will only backup the current PE edition’s files, so if you still have /etc/puppet on your old master from PE 3 days, that will not be part of the backup. However, files in directories it does back up, like /etc/puppetlabs/puppet/puppet.conf.rpmsave, will persist. This will help reduce cruft, but not eliminate it. You will still need to police on-disk content. In particular, if you accidentally placed a large file in /etc/puppetlabs, say the PE install tarball, that will end up in your backup and can inflate the size a bit. If you feel the backup is exceptionally large, you may want to search for large files in that path.

The restore docs also specify two commands to run after a restore when Code Manager is used. If you use CM, make sure not to forget this step:

puppet access login
puppet code deploy --all --wait 

The backup and restore process are mostly time-dependent on the size of your puppetdb. With ~120 agents and 14 days of reports, it took less than 10 minutes for either process and generated a ~1G tarball. Larger environments may expect the master to be offline for a bit longer, if they want to retain their full history.

Lab it up

The backup/restore process is great, but it’s new, and some of us have very ancient systems laying around. I highly recommend testing this in the lab. My test looked like this:

  • Clone the production master to a VM on another hostname/IP
  • Run puppet-backup create
  • Fully uninstall PE (sudo /opt/puppetlabs/bin/puppet-enterprise-uninstaller -p -d -y)
  • Remove any remaining directories with puppet in them, excepting the PE 2018 install files, to ensure all cruft is gone
  • Disable and uninstall any r10k webhook or puppet-related services that aren’t provided by PE itself.
  • Reboot
  • Bootstrap (from above)
  • Install PE (sudo /opt/puppetlabs/bin/puppet-enterprise-installer) only providing an admin password for the console
  • Run puppet-backup restore <backup file>
  • Run puppet agent -t
  • Make sure at least one agent can check in with puppet agent -t --server=<lab hostname> (clone an agent too if need be)
  • Reboot
  • Make sure the master and agent can still check in, Console works, etc.
  • If possible, test any systems that use puppet to make sure they work with the new master
  • Identify any missing components/errors and repeat the process until none are observed

I mentioned that I used PE3. My master had been upgraded all the way from version 3.7 to 2018.1.2. I’m glad I tested this, because there were some unexpected database settings that the restore choked on. I had to engage Puppet Support who provided the necessary commands to update the database so I could get a useful backup. This also allowed me to identify all of my bootstrap items and of course, gain familiarity and confidence with the process.

This became really important for me because, during my production migration, I ran into a bug in my provisioning system where the symptom presented itself through Puppet. Because I was very practiced with the backup/restore process, I was able to quickly determine PE was NOT the problem and correctly identify the faulty system. Though it took about 6 hours to do my “very quick” migration, only about an hour of that was actually spent on the Puppet components.

I also found a few pieces of managed files on the master where the code presumed the directory structure would already be there, which it turns out was not the case. I must have manually created some directories 4 years ago. I think the most common issues you would find at this point are dependencies and ordering, but there may be others. Either fix the code now or, if it would negatively affect the production server, prep a branch for merging just prior to the migration, with the plan to revert if you rollback.

I strongly encourage running through the process a few times and build the most complete checklist you can before moving on to production.

Putting it together

With everything I learned in the lab, my final outline looked like this:

  • Backup old master, export to another location
  • Deploy a new master VM running EL7 using an alternative workflow
  • Run ssh-keygen/install an existing key and attach that key to the git system
  • SSH to the git server and accept the key
  • Verify your VMs default iptables/selinux posture; disable during bootstrap if required
  • Validate the hostname is correct
  • Install PE
  • Restore the backup
  • [Optional] Merge any code required for the new server; run r10k/CM to ensure it’s in place on the new master
  • Run puppet
  • Point agents at the new master

Yours may look slightly different. Please, spend the time in the lab to practice and identify any missing steps, it’s well worth it.


Refreshing any system of significant age is always possible, and often fraught with manual processes that are prone to error. Puppet Enterprise 2018.1 delivered a new backup/restore process that automates much of this process. We have put together a rough outline, refined it in the lab, and then used it to perform the migration in production with high confidence, accounting for any components the backup did not include. I really appreciate this new feature and I look forward to refinements in the future. I hope that soon enough migrations should be as simple an effective as in-place upgrades.

Contributing to a Political Campaign as a Nerd

As I promised in my previous politics article, I will continue not to advocate for specific politics and remain non-partisan on my blog. I do encourage everyone, regardless of beliefs or party, to participate in politics, because it affects you whether you participate or not. Participation in our democracy can only improve it.

Over the past year or two, I have come to feel far more strongly about my politics. I live in the United States, and it’s impossible to pay attention to the news and not feel some kind of way about our Republic. This year, I decided that I needed to contribute more directly, not just passively partake in politics. If you already follow me on twitter, you know that I wear my politics on my sleeve. Like many readers, I regularly vote. Like many readers, I donate to campaigns. That’s not enough for me anymore. So I decided to do more, and reached out to some local campaigns to find out what I could offer.

I will admit that this is scary. I have spent almost 20 years working on a relatively narrow field of expertise within IT. I had no experience with politics. Going from 20 years of experience to 0 is intimidating – but if I did it, so can you. If you read this and want to contribute, I want you to know that you will do just fine. As divisive and loud and argumentative and nasty as politics seems on the evening news, every group I have worked with over the past year has been welcoming, very graceful about my lack of knowledge and mistakes, and very accommodating to how much time I have available. Please, don’t let your fear keep you out. Reach out if you have any questions!

The expertise we have is sorely needed, though, especially in political campaigns. You will quickly find out that most people involved in political campaigns are not computer experts in any way. Sure, they’re computer savvy as most people are nowadays, but there are significant gaps in that knowledge that needs filled. All you need to do is read the news and you will quickly hear about campaigns that are hacked and crowdsourced analysis of what’s on a politicians phone and overwhelming numbers of twitter and facebook shenanigans. Your help is needed and will be welcomed.

I joined a campaign and I had no idea what I was getting into. I did not find many others in tech who have shared their experiences joining their first campaign. I hope this article helps fill this gap a little bit – and if any readers are in the same situation, I would encourage you to blog about it as well! Alright, let’s get volunteering!

As I hit publish on this, there are 85 days until Election Day in the US. It is NOT too late to volunteer! Your assistance will be welcomed up until the final moments on Election Day, and there are always future elections to prepare for.

What to expect

When you click Sign Up on a campaign web site, you’ll be offered some “normal” work – canvassing, phone banking, putting a sign in your yard, etc. All things technical are notably missing from the list. To offer your technical expertise, you will have to reach out to the campaign directly. Many campaigns or candidates have a listed phone number on their website. If not, try looking at the county or state party’s website for a phone number, and inform them that you would like to get in contact with the campaign.

You will have a chance at some point to talk to the candidate or a campaign manager and make sure that’s who you want to work for. Treat it like a job interview! The campaign will ask what you can do, and you get to ask the campaign about what they will do if elected. Be honest! When I first talked to my campaign, I explained that I had 20 years of IT experience but no campaign experience. I was willing to take on things unknown, but I wanted to make sure they knew it would be new to me. I found out they had an experienced webmaster who would be providing me assistance if I joined. I also asked a lot of pointed policy questions to ensure that I would be happy if this candidate was elected. Get your questions answered and let the campaign know whether you want to join and what you can contribute.

General tech tasks

There are so many areas you can contribute to the technology side of a campaign, regardless of where in technology you work. Here’s a very short, very incomplete list of items you can help with:

  • Setting up a free Slack and teaching people how to use it (EVERYONE uses slack nowadays!)
  • Setting up a website and analytics
  • Configuring multi-factor authentication on all services
  • Setting up apps on phones and tablets
  • Answering questions about how to use a computer, application, or service, even if you’re just functioning as Google as a Service for really busy people
  • Providing a sounding board for anything technical, including how technical people and companies may respond to something

Depending on what your expertise is, you may be able to offer some very specific needs. Surely, what you know can be applied to a campaign, though I may not be able to tell you how. A lot of my expertise is in information security. Here are some examples of InfoSec advice you can provide:

  • Explain threat models. Make sure you know how they apply, too; the threat model of running for US Senator is much different than that of running for a local council seat. Everyone can benefit from making sure they don’t expose their financial details to the world, but fewer are worried about specific attacks by enemy nation-states.
  • Ensure services are registered to a well protected account that belongs to the campaign instead of a random gmail account that belongs to someone who may leave the campaign.
  • Make sure MFA is enabled everywhere possible.
  • Restrict access to services to who needs it, and at the least permission level

Remember that you are advising a campaign, not running your own business, so you will probably “lose” some arguments in areas where you are objectively the expert, and that’s okay. Make sure everyone acknowledges the trade-offs being made, and do your best to minimize the potential fallout. If there’s a realistic chance of failure, prepare remediation plans so that you are ready if they are needed – the same kinds of thing you do at work to cover your company’s butt.

Be aware of how much you can contribute. If you can only spend 2 hours a week with the campaign at odd times of the day, maybe you are not the best person to run their web site. That’s okay, just make the campaign aware that you want to help as well as your limits and surely they can find a way to make it fit. If you can spend more hours, then I encourage you to take on more significant tasks. If your circumstances change, just let the campaign know!

There are tons of small things like this that you can contribute. Don’t worry if you can’t think of something now, if you reach out to the campaign I’m sure you can come up with something together!

Larger projects

In addition to these general tasks, you may be able to contribute to higher level projects to help the campaign. If you are a data scientist, a campaign needs you! Everyone needs to know which voters to target, and they’re hopefully looking for more ethical assistance than we have seen campaigns pursue in the past. Many campaigns can determine what kinds of voters they want to target, but they may lack the skills to find those voters within the mass of voter information available. Those who are great with analytics can help get data from the web sites to the voter analysis teams. Social media experts can help leverage Twitter, Facebook, Instagram, and other services to get messaging out effectively. Online advertising needs your expertise in marketing and advertising. Larger campaigns may need custom applications like HillaryBnB (an AirBnB-style app for canvassers).

Again, I had no experience with campaigns so these are just a few of the efforts I’ve observed recently, it’s a very non-comprehensive list of options. Each campaign’s needs are different, so I suggest checking with the campaign to see what is needed, rather than trying to offer specific projects.

Tackling the unknown

Though you are volunteering because you have expertise, sometimes what a campaign needs does not line up exactly. It’s a good thing we are an industry that is constantly learning! Lean on your existing expertise to get going.

I decided to help out a campaign with Google AdWords, as the lack of such a campaign was identified when I joined. Prior to June 1st, I had never used AdWords or done anything with advertising, online or otherwise. Yeah, it was intimidating. But I believe in my candidate, so I tackled it like any other tech I learned. I found some technical articles about how AdWords works and tips for novices, scrounged up some youtube video so I could see it, and then set up an account and got to work. After almost 3 weeks, I am starting to figure it out, and the campaign is benefiting from my efforts. Find something and dig in. You can grow your technical skills and help advocate for your politics at the same time!

While I will not pretend to be authoritative on AdWords, I want to share some things I learned, beyond the simple mechanics that you can learn through the documentation and tutorials:

  • Create the AdWords account with a central campaign account. Add additional administrators with campaign-specific emails, rather than personal emails. This makes it more difficult for someone to leave the campaign and take down advertising.
  • There’s a pretty decent iPhone app for AdWords, and it’s gives you some views not available on the web site, but you cannot edit very much on it. I am sure there is one for Android as well.
  • Google will provide a $100 coupon after spending your first $25. It will be sent in an email to the account owner’s email and it’s not automatic, you need to apply the code.
  • Google will also send an email advertising a free review of your account. I would wait a few weeks before calling, so that the specialists can see some data from your campaigns.
  • Impressions (someone seeing your ad) are free. Clicks (when someone actually clicks on them) are the only thing that costs you money.
  • Campaigns are made up of Ad Groups. Each ad group can advertise a different set of text and point to a different page, but funding is allocated on a Campaign level. You can add as many ad groups to a single campaign, but budget is allocated at the campaign level. You need to balance the number of campaigns and ad groups, and keep balancing them as voters’ interests change.
  • Each Campaign can also be targeted to different locations. You can limit some AdWords Campaigns to the political campaign’s region (by district for federal offices; by using zip codes for state and local offices) to focus your advertising, such as issues-based campaigns. Others may be made open to a wider area, maybe even the whole country, such as donation campaigns.
  • Each Ad Group can be made up of keywords which receive mostly-opaque scoring. The better the score, the better the placement. Keywords that do not result in Impressions receive a lower score and can drag down the score of the entire Ad Group. Disable keywords that do not work, or replace them with more specific and helpful keywords, to keep the scoring up. All the scoring is Google magic, and the effectiveness of this will vary quite a bit for every organization. Keep an eye on it.
  • Keywords are words or phrases that you want your ad matched to. You can also add Exclusions. Combine this with Search Terms results, which show the actual search someone used and the keyword or category it matched, to filter out Impressions/Clicks that are not helpful to your campaign. I have seen some really ridiculous search terms, including the amazing imagen you are an environmentalist giving a speech environmental due to population growth in the western united states that matched the keyword environment. Whoops, way too generic, it was replaced with a more specific phrase. Another was a search including the name of another candidate in another entire state and that one click ate the entire campaign budget for the day. Keeping up with Exclusions can save your campaign a lot of money.
  • Google watches monthly trends to determine when best to spend money. Your daily budget is better thought of as the average daily spend over a 30 day period. For example, Google may determine that you won’t get much out of ads on Saturday and spend close to $0, but the 2nd Wednesday of the month get the most impact and spend far more than your budget. You will only ever be charged 2x your specified budget, even if Google “spends” 2.3x your budget (I have never observed them going past 2.0x).
  • Almost everything you do with AdWords can be changed on the fly. However, there are two things to keep in mind:
    • Any new ad or keywords (and you cannot edit ads/keywords, you actually make a new one to replace the existing one) must be approved. It can take up to 24 hours to approve. You can add ads/keywords and disable them, then enable them when needed, to ensure they are approved prior to when they are needed.
    • Significant budget changes may flag a fraud alert. If you have a significant event-related campaign coming up, set it up at least 3 days in advance, as it can take 3 days to resolve suspected fraud. If you are spending $10 a day and want to increase that to $300 for a weekend event and your account gets flagged on Friday, it may be Monday before it is unfrozen and your event will be over.
  • You won’t find many AdWords tutorials that speak to political campaigns. It’s strongly associated with businesses. A few articles and charts mentioned Social Advocacy, which is probably the closest, but…
  • Success is difficult to measure. A business may track how many people click versus how many people order something. A donation campaign can track how many people donate, but an issues or awareness campaign cannot correlate visitors to the site with votes in a primary or general election. Conversion rates on their own won’t tell you much.
    • For many candidates, awareness itself is the goal. Many voters do not fill out the whole ballot and only check non-federal boxes if they recognize the name. Responsive Ads (as opposed to Text Ads) are fairly unobtrusive but can display small graphics. Logos are very helpful to start creating brand awareness.
    • Run at least two Ads for each campaign, much like you would do A/B testing at work. Review regularly and tweak the ads over time.
  • Advertising is not an island. Coordinate with the Social Media team and the event planners. If your candidate is going to be at a local festival or state fair in a few weeks, add some keywords for the event. If Social Media is blitzing on a policy, make sure common keywords will drive voters to your candidate. When things happen in the news that affect your voters, make sure their searches will bring them to your candidate. Likewise, you can review search terms people use and inform the other teams that these are some of the things voters are searching for and make sure the candidate speaks to those concerns.
    • This can feel very ghoulish or morbid. Much of news that drives people to politicians is going to be of a negative nature. We generally don’t call our elected officials when things are looking up. You WILL probably have to capitalize on an event that resulted in harm or death. In my case, unfortunately, there was a school shooting near the district. Ugh. But voters do want to know their candidate’s policies on subjects like school shootings. Be responsible and principled and above all, caring. Do not let the need to respond compromise your integrity or the candidate’s.
  • Your candidate’s party has other candidates running. Reach out to them for assistance, for ideas, for additional eyes on problems. You can get contact info from your county/state party’s offices, usually.
  • AdWords includes a large number of reports. Working in IT, our tendency is to encourage others to run reports themselves, but everyone on a political campaign is likely already spending all the time they can on their areas of expertise. It can be a huge help if you create/tweak reports for others, and schedule them to email the requester regularly.


Just because many of us make technology a huge part of our lives, we are not one dimensional. If you feel inspired by politics, don’t hide it, become active. I’ve discussed what you might expect if you join a political campaign, some of the work and expertise technologists can offer campaigns, and my experience in joining a campaign. Whether you contribute a few hours a month or hours every day, you will be a vital part of chosen campaign. That’s awesome! Participation in democracy is what makes our Republic so strong, and that of most democratic governments.

If you have any questions about volunteering, whether it’s technical or about the experience or something else, please reach out. You can drop a comment here, or @rnelson0 on twitter.

Enjoy, and thank you!

Disabling rubocop and upgrading to PDK 1.6.0

As I lamented in my article on converting to the PDK, I really do not like Rubocop and was disappointed I could not turn it off. Thankfully, that was addressed in PDK-998 and the fix was included in time for PDK 1.6.0! Disabling it is pretty simple and though it’s strictly a fix to pdk-templates, updating the PDK won’t hurt.

First, update to PDK 1.6.0. As I use CentOS 7 and the RPM packaging, it’s as simple as sudo yum update pdk -y; follow the directions that match system. Next, we need to add the following lines to .sync.yml:

  selected_profile: off

Finally, run pdk update, or if you weren’t already using pdk-templates, run pdk convert --template-url= (I will assume the former to keep it simple). You can add --noop (or say n) and review update.txt|convert.txt to see the differences before applying, or, because you are using version control, just run a diff afterward to see the changes.

[rnelson0@build03 domain_join:pdk160]$ pdk update
pdk (INFO): Updating rnelson0-domain_join using the template at, from master@041eeb2 to 1.6.0

----------Files to be modified----------


You can find a report of differences in update_report.txt.

Do you want to continue and make these changes to your module? Yes
[✔] Installing missing Gemfile dependencies.

------------Update completed------------

7 files modified.

That’s it! Check the contents of .rubocop.yml and you will notice everything is false (just a snippet because it’s loooong):

require: rubocop-rspec
  DisplayCopNames: true
  TargetRubyVersion: '2.1'
  - "./**/*.rb"
  - bin/*
  - ".vendor/**/*"
  - "**/Gemfile"
  - "**/Rakefile"
  - pkg/**/*
  - spec/fixtures/**/*
  - vendor/**/*
  - "**/Puppetfile"
  - "**/Vagrantfile"
  - "**/Guardfile"
  Enabled: false
  Enabled: false
  Enabled: false
  Enabled: false
  Enabled: false
  Enabled: false
  Enabled: false
  Enabled: false
  Enabled: false
  Enabled: false

Running validation now finds no issues with ruby syntax no matter how much you ignore style guides:

# master, prior to updating

[rnelson0@build03 domain_join:master]$ pdk validate
[✖] Checking Ruby code style (**/**.rb).
info: task-metadata-lint: ./: Target does not contain any files to validate (tasks/*.json).
convention: rubocop: spec/spec_helper_acceptance.rb:17:27: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.
convention: rubocop: spec/spec_helper_acceptance.rb:17:49: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.
convention: rubocop: spec/spec_helper_acceptance.rb:19:66: Style/BracesAroundHashParameters: Redundant curly braces around a hash parameter.
convention: rubocop: spec/spec_helper_acceptance.rb:19:68: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.
convention: rubocop: spec/spec_helper_acceptance.rb:19:96: Layout/SpaceAfterComma: Space missing after comma.
convention: rubocop: spec/acceptance/class_spec.rb:6:9: RSpec/ExampleWording: Do not use should when describing your tests.
convention: rubocop: spec/acceptance/class_spec.rb:12:26: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.
convention: rubocop: spec/acceptance/class_spec.rb:13:26: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.
convention: rubocop: spec/classes/domain_join_spec.rb:2:25: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.

# pdk160, after updating, no code changes
[rnelson0@build03 domain_join:pdk160]$ pdk validate
pdk (INFO): Running all available validators...
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.2
[✔] Checking metadata syntax (metadata.json tasks/*.json).
[✔] Checking module metadata style (metadata.json).
[✔] Checking Puppet manifest syntax (**/**.pp).
[✔] Checking Puppet manifest style (**/*.pp).
[✔] Checking Ruby code style (**/**.rb).
info: task-metadata-lint: ./: Target does not contain any files to validate (tasks/*.json).

You may have noticed there are quite a few other files updated. The other significant change is that a :changelog task via github_changelog_generator is now included, so you can remove that from your .sync.yml if you added it and replace it with the recommended config (via the Rakefile):

      - gem: 'github_changelog_generator'
        git: ''
        ref: '20ee04ba1234e9e83eb2ffb5056e23d641c7a018'
        condition: " >='2.2.2')"

The other changes are pretty minor, in some cases cosmetic, but of course review them to make sure they’re OK. Submit a PR or equivalent and make sure the tests pass before merging. You can follow along with today’s blog post in domain_join PR35, too.


Opinion: Technology is always Political

I’m writing this opinion piece right now because of ongoing gross abuses of justice taking place in America right now. Don’t worry, as strongly as I feel about this subject, my blog will remain free of specific political advocacy (please follow me on twitter for my politics), but we absolutely need to talk about the relationship between technology and politics. I would love to see your comments here, or you can reach me on twitter if that is more comfortable.

As technical practitioners, we are often under a lot of pressure to focus on tech and minimize other subjects, especially contentious subjects like politics. These subjects can cause conflict and many of us are taught to avoid, rather than resolve, conflict. We are constantly told that talking about tech is great; maybe talk about your sports teams, music, craft beers – but never talk about politics. That’s divisive! Sure, sometimes politics may cause conflict and even alienate people, but that’s just an aspect of life, of who we are, and not something to box up and hide in a corner. We soon find that everything is political – and if it’s not, it will still be made political for us. Our politics reflect who we are, and we strive to grow and change over time, and it stands to reason that our politics grow and change with us. I find Scott Hanselman to be very eloquent on the relation of politics to self, probably because it comes up so often on his timeline:

Once you share your politics, like Scott, you may be told to stick to technology. But if everything is political, then technology must be political, too. It doesn’t exist in a vacuum. It never has and it never will. We must stop pretending that technology exists free from politics. Our community benefits from embracing this truth, not denying it.

Examination of our industry’s early history quickly shows the relationship between technology and politics. Let’s review the history of IBM during World War II. In 1933, IBM started cozying up to the Hitler regime to increase sales, starting with an innocuous sounding census. Having machine-tabulated census data allowed the German government to rapidly increase their prosecution of the Holocaust. Eventually, every Nazi concentration camp used IBM punch card technology to track prisoners – which IBM serviced under contract.

This was not an isolated act of the company or an oddity of IBM’s German subsidiary. This happened in America, too. IBM pursued and acquired the contract for the Japanese internment camps’ punch cards, at the same time its equipment was used for US Army and Navy cryptography. IBM was okay with the use of its punch card equipment to identify, round up, and track prisoners, even by a country at war with IBM’s home country, so as long as IBM got paid.

Neither of these efforts just “happened.” IBM employees developed the punch card technology. IBM employees had to contact the German and US governments to open sales channels. IBM employees  had to pursue and close sales contracts with the German and US governments. IBM employees had to provide support, spare parts, and even enter concentration camps to change the printer paper throughout World War II.

Numerous people were required to prosecute the Holocaust and the Japanese internment. Only a few were technologists, and fewer still worked for IBM. But all of them allowed their personal political and ethical views to be subsumed and harnessed to a political regime that enacted some of the worst atrocities in the history of humankind. We do not know exactly what these people intended, but history has recorded the outcome and judged it. No amount of, “Well, I didn’t mean that to happen,” or, “I didn’t want to take sides,” will ever change that.

“The only thing necessary for the triumph of evil is that good men should do nothing.” – Edmund Burke

And here we are again, in 2018, observing numerous authoritarian efforts, by both governments and public/private companies, to weaponize technology. Again, this does not just “happen.” Someone has to actively develop, sell, provide, and support these weaponized technologies. Each of us must inform ourselves of how the technologies we work on can be weaponized and used for evil. Implicit and subconscious bias will creep into products, but very frequently, efforts are consciously and overtly made to weaponize technologies. Original intent is never remembered, only the horrible outcomes.

Earlier, I stated that we should have and develop our own political views. In accepting that technology is intrinsically political, our use of technology may then become political advocacy. For example, If we are strongly in favor of a legal right, using a technology designed to interfere or suppress that right would be antithetical to our politics. Thus, our political views should drive our use of technology.

To avoid advocating for specific political views, I suggest that we all commit to a formal code of ethics that closely matches our political views. I highly recommend USENIX’s System Administrators’ Code of Ethics (more in Additional Links). It is straightforward and thorough, is compatible with most political viewpoints, and has stood the test of time within our industry. A chosen code of ethics is only helpful if we stick by it. Not just when it’s easy, but especially when it’s difficult!

So what happens if we do find ourselves involved with a system that can be weaponized, that violates our ethics and politics? How do we advocate our politics and maintain our ethical code when we have significant concerns? “It depends,” of course, on those concerns and how significant they are, on our relationship with the system, and on who we are providing for. We must each determine an inflection point where it passes from, “Can this still be saved?” to, “This cannot be saved.” Everyone’s inflection point will be different – we all have different politics and ethics, finances, health, and family situations, etc. – but we must each determine exactly where that point is for us.

Inflection point in hand, we can proceed to an action plan. We may be able to work fully above board, making cases to stakeholders and management. Or we may have to move below board, purposefully dragging our feet or working against the system. Maybe a change in vendors will address concerns or slow progress, maybe we just don’t do certain things at all. Find the appropriate monkey wrench that fits the gears of the system. We must plan our activities as we would any other technical work, laying out our goals and milestones and alternative plans for when disaster strikes.

We can also lean on each other. Reach out to your coworkers, your colleagues and peers, your friends and family, to voice your concerns and discuss remedies. We are not alone, and we can lean on or be the rock for each other. We may want to confide in just a few trusted people, to organize with our coworkers, or to become whistleblowers. There is so much nuance and possibility here that it is impossible to predict what actions will be required, but together we can determine what those actions are.

There may come a day when we find that we cannot stop the dangerous technology, when we have crossed that inflection point. We have to evaluate, honestly, whether there is still good we can do, or if we have truly passed the point of no return and need to walk away. Many of us will never be close to making these kinds of decisions, but some of us will. We must rely on our ethical codes and our planning to remain true, to see the point of no return, and to walk away, even if it costs us. To stay and actively participate knowing that we are now doing irrevocable harm would be costlier.

This is not a one-time deal. We must always keep our eyes open, evaluating our ever-changing political views against the work in front of us, applying our ethics to keep us on track, and making damn sure that we are never – metaphorically or literally – changing the printer paper in a concentration camp.

We will not build the software for concentration camps or to enable authoritarians. We will not spy on our neighbors or destroy democracies. We will commit to our ethical codes of conduct, and we will use technology to build the shining city on a hill.

Thank you.



Additional Links:

Creating your first Puppet Task for Puppet Enterprise

At PuppetConf 2017, Puppet Tasks were introduced as part of the new project Bolt. A task allows you to run a program on an arbitrary number of nodes. The program can be just about anything, it just needs to be written in a language that the target nodes can run. For Linux, that means pretty much anything – bash, python, perl, ruby, etc. On Windows, you’re a little more limited out of the box – powershell primarily. Bolt is not yet at version 1.0.0, so I suspect language support for Windows will change. You can use Bolt on its own (even without Puppet, apparently), and starting with Puppet Enterprise 2017.3, you can use Bolt at the PE Console as “tasks” in the UI.

For my first task, I simply want to run a single command on a list of nodes. While I can run arbitrary commands with the bolt command line, I want the practice of writing a task. My use case involves an external authentication system that manages users, ssh keys, and sudo configurations. When a change is made, nodes need to pull the changes. Often, a delay there does not matter – the nodes will receive the change soon enough – but sometimes I want the relevant nodes to pick it up immediately. To do so, I need it to run a single perl script, sanitized as /usr/bin/, and I want to do it on all the nodes with the profile::external_auth class.

Continue reading