If you haven’t heard, PowerShell 7 has been released! Even if you haven’t gotten emails or RSS alerts, it’s hard to miss if you use VS Code as the PowerShell plugin will remind you on startup:

Installation on Windows is as simple as selecting Yes and following the prompts. You’ll have to close VS Code – and for some reason, Slack, at least on Windows – and shortly you’ll see PowerShell 7 installed. I assume it’s similar on other OSes, but the specifics may differ.

When you relaunch VS Code, however, you’ll still be using whatever PowerShell you had prior. For me, it was 5.1. This is because the default integrated shell on Windows is the base PowerShell from your OS, and PowerShell 7 is a separate install.

Select Edit in settings.json and add this text (using the correct path if you installed to a non-default location) and save the file:

 "terminal.integrated.shell.windows": "C:\\Program Files\\PowerShell\\7\\pwsh.exe",

Because this is a completely different installation, it doesn’t inherit our existing profile, so before restarting, run notepad $profile and copy the contents. Now restart VS Code and you should have a PowerShell 7 prompt, if uncustomized.

Type notepad $profile, paste your old profile, save and exit, and restart VS Code. Now you have PowerShell 7 as your integrated console with the same customizations as before. Of course, with a new version there might be more you want to customize – and now you’re ready to do so!

If you run VS Code on macOS or Linux, the process above should be a good guideline, but will likely need some tweaks. If you’ve done this, please leave a comment with any specifics you can share, thanks!

One of the most important parts of using any configuration management software, including Puppet, is making sure that your nodes receive the correct classification. You can write all the code you want to describe the desired system state, but if you don’t attach it to a node, it doesn’t provide any value. In previous articles, I have described using hiera_include() for that classification. However, hiera functions have been deprecated since at least version 4.10.

The replacement for hiera functions is called Lookup. Automatic Parameter Lookup now directly uses lookup instead of the older hiera functions. There is an actual lookup() function. There is also a plugin for the puppet command line utility, puppet lookup. Here’s what we have now:

node default {
  hiera_include('classes')
}

What can we replace this with? If we simply swap out hiera_include with lookup, we don’t actually include the results. We can add .include at the end, which is an object-oriented function that can be called on any string or array of strings:

node default {
  lookup('classes').include
}

This works, but leaves some ugly edges. First, if there are no results for classes, it gives a static error, rather than one you can control:

C:\Windows\system32>puppet apply --noop -e 'lookup("classes").include'
Error: Function lookup() did not find a value for the name 'classes'

Second, it could return a null string, which also gives an error:

C:\Windows\system32>puppet apply --noop -e '"".include'
Error: Evaluation Error: Error while evaluating a Method call, Cannot use empty string as a class name (line: 1, column: 3) on node agent.example.com

A third issue is one of cleanliness, not an actual error: the result could have multiple instances of the same class. Inserting .unique between the lookup and the include would address that.

Fourth, we could have almost any type of data in the classes key, not just strings. If it returned a hash, some other type related error would be seen.

Finally, while rare, we could potentially want to iterate on the result, maybe for logging purposes, and as it stands now, the lookup would have to be performed again for that. We can store the result and operate on it.

With these concerns in mind, a more comprehensive result was obtained with the help of Nate McCurdy:

node default {
  # Find the first instance of `classes` in hiera data and includes unique values. Does not merge results.
  $classes = lookup('classes', Variant[String,Array[String]])
  case $classes {
    String[1]: {
      include $classes
    }
    Array[String[1],1]: {
      $classes.unique.include
    }
    default: {
      fail('This node did not receive any classification')
    }
  }
}

The lookup() call does two things different now. First, we specify a type definition. classes MUST be either a String or an Array of Strings. Any other result will trigger a type mismatch error (while I definitely encourage the “one node, one role” promise of the role/profile pattern, returning multiple classes can be useful during development – just be sure not to allow such tests to propagate to production). Second, the result is stored in $classes.

Next, we have a case statement for the result. The first case is a String with at least one character, to protect against a null string. In that case, the single class is included. The second case matches an Array of Strings with at least 1 element, and each element has at least one character, to protect against a null array or an array with null strings.

The final case matches any empty strings or arrays and throws a customized error about the lack of classification. Because this fails, instead of silently completing without applying anything, no catalog is compiled and the agent’s local state cache is not updated, both events your monitoring system can report on.

As is, this provides a flexible classification system using modern puppet language constructs. It can also be further customized for each case, if desired. I would recommend that anyone still using hiera_include() review this and implement something similar to protect against the eventual removal of the deprecated functions.

Thanks to Nate McCurdy for his assistance with the language constructs!

I previously wrote about converting an individual puppet module’s repo to use the Puppet Development Kit. We can also convert controlrepos to use the PDK. I am starting with a “traditional” controlrepo, described here, as well as centralized tests, described here. To follow this article directly, you need to:

• Have all hiera data and role/profile/custom modules in the /dist directory
• Have all tests, for all modules, in the /spec directory

If your controlrepo looks different, this article can be used to migrate to the PDK, but you will have to modify some of the sections a bit.

This will be a very lengthy blog post (over 4,000 words!) and covers a very time-consuming process. It took me about 2 full days to work through this on my own controlrepo. Hopefully, this article helps you shave significant time off the effort, but don’t expect a very quick change.

Managing the PDK itself

First, let’s make sure we have a profile to install and manage the PDK. As we use the role/profile pattern, we create a class profile::pdk with the parameter version, which we can specify in hiera as profile::pdk::version: ‘1.7.1’ (current version as of this writing). This profile can then be added to an appropriate role class, like role::build for a build server, or applied directly to your laptop. I use only Enterprise Linux 7, but we could certainly flush this out to support multiple OSes:

# dist/profile/manifests/pdk.pp
class profile::pdk (
  String $version = 'present',
) {
  package {'puppet6-release':
    ensure => present,
    source => "https://yum.puppet.com/puppet6/puppet6-release-el-7.noarch.rpm",
  }
  package {'pdk':
    ensure => $version,
    require => Package['puppet6-release'],
  }
}

# spec/classes/profile/pdk_spec.rb
require 'spec_helper'
describe 'profile::pdk' do
  on_supported_os.each do |os, facts|
    next unless facts[:kernel] == 'Linux'
    context "on #{os}" do
      let (:facts) {
        facts.merge({
          :clientcert => 'build',
        })
      }

      it { is_expected.to compile.with_all_deps }

      it { is_expected.to contain_package('puppet6-release') }
      it { is_expected.to contain_package('pdk') }
    end
  end
end

Once this change is pushed upstream and the build server (or other target node) checks in, the PDK is available:

$ pdk --version
1.7.1

Now we are almost ready to go. Of course, we need to start with good, working tests! If any tests are currently failing, we need to get them to a passing state before continuing, like this:

Finished in 3 minutes 12.2 seconds (files took 1 minute 17.89 seconds to load)
782 examples, 0 failures

With everything in a known good state, we can then be sure that any failures are related to the PDK changes, and only the PDK changes.

Setting up the PDK Template

The PDK comes with a set of pre-packaged templates. It is recommended to stick with a set of templates designed for the current PDK version for stability. However, the templates are online and may updated without an accompanying PDK release. We may choose to stick with the on-disk templates, we may point to the online templates from Puppet, or we may create our own! For those working with the the on-disk templates, you can skip down to working with .sync.yml

To another template, we use the pdk convert --template-url. If this is our own template, we should make sure the latest commit is compliant with the PDK version we are using. If we point to Puppet’s templates, we essentially shift to the development track. Make sure you understand this before changing the templates. We can get back to using the on-disk template with the url file:///opt/puppetlabs/pdk/share/cache/pdk-templates.git, though, so this isn’t a decision we have to live with forever. Here’s the command to switch to the official Puppet templates:

$ pdk convert --template-url=https://github.com/puppetlabs/pdk-templates

------------Files to be added-----------
.travis.yml
.gitlab-ci.yml
.yardopts
appveyor.yml
.rubocop.yml
.pdkignore

----------Files to be modified----------
metadata.json
Gemfile
Rakefile
spec/spec_helper.rb
spec/default_facts.yml
.gitignore
.rspec

----------------------------------------

You can find a report of differences in convert_report.txt.

pdk (INFO): Module conversion is a potentially destructive action. Ensure that you have committed your module to a version control system or have a backup, and review the changes above before continuing.
Do you want to continue and make these changes to your module? Yes

------------Convert completed-----------

6 files added, 7 files modified.

Now, everyone’s setup is probably a little different and thus we cannot predict the entirety of the changes each of us must make, but there are some minimal changes everyone must make. The file .sync.yml can be created to allow each of us to override the template defaults without having to write our own templates. The layout of the YAML starts with the filename the changes will modify, followed by the appropriate config section and then the value(s) for that section. We can find the appropriate config section by looking at the template repo’s erb templates. For instance, I do not use AppVeyor, GitLab, or Travis with this controlrepo, so to have git ignore them, I made the following changes to the .gitignore‘s required hash:

$ cat .sync.yml
---
.gitignore:
  required:
    - 'appveyor.yml'
    - '.gitlab-ci.yml'
    - '.travis.yml'

When changes are made to the sync file, they must be applied with the pdk update command. We can see that originally, these unused files were to be committed, but now they are properly ignored:

$ git status
# On branch pdk
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   .gitignore
#       modified:   .rspec
#       modified:   Gemfile
#       modified:   Rakefile
#       modified:   metadata.json
#       modified:   spec/default_facts.yml
#       modified:   spec/spec_helper.rb
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       .gitlab-ci.yml
#       .pdkignore
#       .rubocop.yml
#       .sync.yml
#       .travis.yml
#       .yardopts
#       appveyor.yml
no changes added to commit (use "git add" and/or "git commit -a")

$ cat .sync.yml
---
.gitignore:
  required:
    - 'appveyor.yml'
    - '.gitlab-ci.yml'
    - '.travis.yml'

$ pdk update
pdk (INFO): Updating mss-controlrepo using the template at https://github.com/puppetlabs/pdk-templates, from 1.7.1 to 1.7.1

----------Files to be modified----------
.gitignore

----------------------------------------

You can find a report of differences in update_report.txt.

Do you want to continue and make these changes to your module? Yes

------------Update completed------------

1 files modified.

$ git status
# On branch pdk
# Changes not staged for commit:
# (use "git add <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working directory)
#
# modified: .gitignore
# modified: .rspec
# modified: Gemfile
# modified: Rakefile
# modified: metadata.json
# modified: spec/default_facts.yml
# modified: spec/spec_helper.rb
#
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# .pdkignore
# .rubocop.yml
# .sync.yml
# .yardopts
no changes added to commit (use "git add" and/or "git commit -a")

Anytime we pdk update, we will still receive new versions of the ignored files, but they won’t be committed to the repo and a git clean or a clean checkout will remove them.

After initial publication, I was made aware that you can completely delete or unmanage a file using delete:true or unmanage:true, as described here, rather than using .gitignore.

We may need to implement other overrides, except that we do not know what they would be yet, so let’s commit our changes so far. Then we can start working on validation or unit tests. It doesn’t really matter which we choose to work on first, though my preference is validation first as it does not depend on the version of Puppet we are testing.

PDK Validate

The PDK validation check, pdk validate, will check the syntax and style of metadata.json and any task json files, syntax and style of all puppet files, and ruby code style. This is roughly equivalent to our old bundle exec rake syntax task. Since the bundle setup is a wee bit old and the PDK is kept up to date, we shouldn’t be surprised if what was passing before now has failures. Here’s a sample of the errors I encountered on my first run – there were hundreds of them:

$ pdk validate
pdk (INFO): Running all available validators...
pdk (INFO): Using Ruby 2.5.1
pdk (INFO): Using Puppet 6.0.2
[✔] Checking metadata syntax (metadata.json tasks/*.json).
[✔] Checking module metadata style (metadata.json).
[✔] Checking Puppet manifest syntax (**/**.pp).
[✔] Checking Puppet manifest style (**/*.pp).
[✖] Checking Ruby code style (**/**.rb).
info: task-metadata-lint: ./: Target does not contain any files to validate (tasks/*.json).
warning: puppet-lint: dist/eyaml/manifests/init.pp:43:12: indentation of => is not properly aligned (expected in column 14, but found it in column 12)
warning: puppet-lint: dist/eyaml/manifests/init.pp:51:11: indentation of => is not properly aligned (expected in column 12, but found it in column 11)
warning: puppet-lint: dist/msswiki/manifests/init.pp:56:12: indentation of => is not properly aligned (expected in column 13, but found it in column 12)
warning: puppet-lint: dist/msswiki/manifests/init.pp:57:10: indentation of => is not properly aligned (expected in column 13, but found it in column 10)
warning: puppet-lint: dist/msswiki/manifests/init.pp:58:11: indentation of => is not properly aligned (expected in column 13, but found it in column 11)
warning: puppet-lint: dist/msswiki/manifests/init.pp:59:10: indentation of => is not properly aligned (expected in column 13, but found it in column 10)
warning: puppet-lint: dist/msswiki/manifests/init.pp:60:12: indentation of => is not properly aligned (expected in column 13, but found it in column 12)
warning: puppet-lint: dist/msswiki/manifests/rsync.pp:37:140: line has more than 140 characters
warning: puppet-lint: dist/msswiki/manifests/rsync.pp:43:140: line has more than 140 characters
warning: puppet-lint: dist/profile/manifests/access_request.pp:21:3: optional parameter listed before required parameter
warning: puppet-lint: dist/profile/manifests/access_request.pp:22:3: optional parameter listed before required parameter

We can control puppet-lint Rake settings in .sync.yml – but it only works for rake tasks. pdk validate will ignore it because puppet-lint isn’t invoked via rake. The same settings need to be put in .puppet-lint.rc in the proper format. That file is not populated via pdk, so just create it by hand. I don’t care about the arrow alignment or 140 characters checks, so I’ve added the appropriate lines to both files and re-run pdk update. We all have difference preferences, just make sure they are reflected in both locations:

$ cat .sync.yml
---
.gitignore:
  required:
    - 'appveyor.yml'
    - '.gitlab-ci.yml'
    - '.travis.yml'
Rakefile:
  default_disabled_lint_checks:
    - '140chars'
    - 'arrow_alignment'
$ cat .puppet-lint.rc
--no-arrow_alignment-check
--no-140chars-check
$ grep disable Rakefile
PuppetLint.configuration.send('disable_relative')
PuppetLint.configuration.send('disable_140chars')
PuppetLint.configuration.send('disable_arrow_alignment')

Now we can use pdk validate and see a lot fewer violations. We can try to automatically correct the remaining violations with pdk validate -a, which will also try to auto-fix other syntax violations, or pdk bundle exec rake lint_fix, which restricts fixes to just puppet-lint. Not all violations can be auto-corrected, so some may still need fixed manually. I also found I had a .rubocop.yml in a custom module’s directory causing rubocop failures, because apparently rubocop parses EVERY config file it finds no matter where it’s located, and had to remove it to prevent errors. It may take you numerous tries to get through this. I recommend fixing a few things and committing before moving on to the next set of violations, so that you can find your way back if you make mistakes. Here’s a command that can help you edit all the files that can’t be autofixed by puppet-lint or rubocop (assuming you’ve already completed an autofix attempt):

vi $(pdk validate | egrep "(puppet-lint|rubocop)" | awk '{print $3}' | awk -F: '{print $1}' | sort | uniq | xargs)

Alternatively, you can disable rubocop entirely if you want by adding the following to your .sync.yml. If you are only writing spec tests, this is probably fine, but if you are writing facts, types, and providers, I do not suggest it.

.rubocop.yml:
  selected_profile: off

We have quite a few methods to fix all the possible errors that come our way. Once we have fixed everything, we can move on to the Unit Tests. We will re-run validation again after the unit tests, to ensure any changes we make for unit tests do not introduce new violations.

Unit Tests

Previously, we used bundle exec rake spec to run unit tests. The PDK way is pdk test unit. It performs pretty much the same, but it does collect all the output before displaying it, so if you have lots of fixtures and tests, you won’t see any output for a good long while and then bam, you get it all at once. The results will probably be just a tad overwhelming at first:

$ pdk test unit
pdk (INFO): Using Ruby 2.5.1
pdk (INFO): Using Puppet 6.0.2
[✔] Preparing to run the unit tests.
[✖] Running unit tests.
  Evaluated 782 tests in 110.76110479 seconds: 700 failures, 0 pending.
failed: rspec: ./spec/classes/profile/base__junos_spec.rb:11: Evaluation Error: Error while evaluating a Resource Statement, Unknown resource type: 'cron' (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/profile/manifests/base/junos.pp, line: 15, column: 3) on node build
  profile::base::junos with defaults for all parameters should contain Cron[puppetrun]
  Failure/Error:
    context 'with defaults for all parameters' do
      it { is_expected.to create_class('profile::base::junos') }
      it { is_expected.to create_cron('puppetrun') }
    end
  end

failed: rspec: ./spec/classes/profile/base__linux_spec.rb:12: Evaluation Error: Error while evaluating a Resource Statement, Unknown resource type: 'sshkey' (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/ssh/manifests/hostkeys.pp, line: 13, column: 5) on node build
  profile::base::linux on redhat-6-x86_64 disable openshift selinux policy should contain Selmodule[openshift-origin] with ensure => "absent"
  Failure/Error:
        if (facts[:os]['family'] == 'RedHat') && (facts[:os]['release']['major'] == '6')
          context 'disable openshift selinux policy' do
            it { is_expected.to contain_selmodule('openshift-origin').with_ensure('absent') }
            it { is_expected.to contain_selmodule('openshift').with_ensure('absent') }
          end
failed: rspec: ./spec/classes/profile/base__linux_spec.rb:162: Evaluation Error: Error while evaluating a Resource Statement, Unknown resource type: 'cron' (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/os_patching/manifests/init.pp, line: 113, column: 3) on node build
  profile::base::linux on redhat-6-x86_64 when managing OS patching should contain Class[os_patching]
  Failure/Error:
          end

          it { is_expected.to contain_class('os_patching') }
          if (facts[:os]['family'] == 'RedHat') && (facts[:os]['release']['major'] == '7')
            it { is_expected.to contain_package('yum-utils') }

failed: rspec: ./spec/classes/profile/base__linux_spec.rb:18: error during compilation: Evaluation Error: Unknown variable: '::sshdsakey'. (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/ssh/manifests/hostkeys.pp, line: 12, column: 6) on node build
  profile::base::linux on redhat-7-x86_64 with defaults for all parameters should compile into a catalogue without dependency cycles
  Failure/Error:

        context 'with defaults for all parameters' do
          it { is_expected.to compile.with_all_deps }

          it { is_expected.to create_class('profile::base::linux') }

Whoa. Not cool. From 728 working tests to 700 failures is quite the explosion! And they blew up on missing resource types that are built-in to Puppet. What happened? Puppet 6, that’s what! However…

Fix Puppet 5 Tests First

When I ran pdk convert, it updated my metadata.json to specify it supported Puppet versions 4.7.0 through 6.x because I was missing any existing requirements section. The PDK defaults to using the latest Puppet version your metadata supports. Whoops! It’s okay, we can test against Puppet 5, too. I recommend that we get our existing tests working with the version of Puppet we wrote them for, just to get back to a known good state. We don’t want to be troubleshooting too many changes at once.

There are two ways to specify the version to use. There’s the CLI envvar PDK_PUPPET_VERSION that accepts a simple number like 5 or 6, which is preferred for automated systems like CI/CD, rather than humans. You can also use --puppet-version or --pe-version to set an exact version. I’m an old curmudgeon, so I’m using the non-preferred envvar setting today, but Puppet recommends using the actual program arguments! Regardless of how you specify the version, the PDK changes not just the Puppet version, but which version of Ruby it uses:

$ PDK_PUPPET_VERSION='5' pdk test unit
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.6
[✔] Preparing to run the unit tests.
[✖] Running unit tests.
  Evaluated 782 tests in 171.447295254 seconds: 519 failures, 0 pending.
failed: rspec: ./spec/classes/msswiki/init_spec.rb:24: error during compilation: Evaluation Error: Error while evaluating a Function Call, You must provide a hash of packages for wiki implementation. (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/msswiki/manifests/init.pp, line: 109, column: 5) on node build
  msswiki on redhat-6-x86_64  when using default params should compile into a catalogue without dependency cycles
  Failure/Error:
          end

          it { is_expected.to compile.with_all_deps }

          it { is_expected.to create_class('msswiki') }

failed: rspec: ./spec/classes/profile/apache_spec.rb:31: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Empty string title at 0. Title strings must have a length greater than zero. (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/concat/manifests/setup.pp, line: 59, column: 10) (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/apache/manifests/init.pp, line: 244) on node build
  profile::apache on redhat-7-x86_64 with additional listening ports should contain Firewall[100 Inbound apache listening ports] with dport => [80, 443, 8088]
  Failure/Error:
          end

          it {
            is_expected.to contain_firewall('100 Inbound apache listening ports').with(dport: [80, 443, 8088])
          }

failed: rspec: ./spec/classes/profile/base__linux_spec.rb:12: Evaluation Error: Unknown variable: '::sshecdsakey'. (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/ssh/manifests/hostkeys.pp, line: 36, column: 6) on node build
  profile::base::linux on redhat-6-x86_64 disable openshift selinux policy should contain Selmodule[openshift-origin] with ensure => "absent"
  Failure/Error:
        if (facts[:os]['family'] == 'RedHat') && (facts[:os]['release']['major'] == '6')
          context 'disable openshift selinux policy' do
            it { is_expected.to contain_selmodule('openshift-origin').with_ensure('absent') }
            it { is_expected.to contain_selmodule('openshift').with_ensure('absent') }
          end

Some of us may be lucky to make it through without errors here, but I assume most of us encounter at least a few failures, like I did – “only” 519 compared to 700 before. Don’t worry, we can fix this! To help us focus a bit, we can run tests on individual spec files using pdk bundle exec rspec <filename> (remembering to specify PDK_PUPPET_VERSION or to export the variable). Everyone has different problems here, but there are some common failures, such as missing custom facts:

  69) profile::base::linux on redhat-7-x86_64 when managing OS patching should contain Package[yum-utils]
      Failure/Error: include ::ssh::hostkeys

      Puppet::PreformattedError:
        Evaluation Error: Unknown variable: '::sshdsakey'. (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/ssh/manifests/hostkeys.pp, line: 12, column: 6) on node build

Nate McCurdy commented that instead of calling rspec directly, you can pass a comma-separated list of files with --tests​, e.g. pdk test unit --tests path/to/spec/file.rb,path/to/spec/file2.rb

I defined my custom facts in spec/spec_helper.rb. That has definitely changed. Here’s part of the diff from running pdk convert:

$ git diff origin/production spec/spec_helper.rb
diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb
index de3e7e6..5e721b7 100644
--- a/spec/spec_helper.rb
+++ b/spec/spec_helper.rb
@@ -1,45 +1,44 @@
 require 'puppetlabs_spec_helper/module_spec_helper'
 require 'rspec-puppet-facts'

-add_custom_fact :concat_basedir, '/dne'
-add_custom_fact :is_pe, true
-add_custom_fact :root_home, '/root'
-add_custom_fact :pe_server_version, '2016.4.0'
-add_custom_fact :selinux, true
-add_custom_fact :selinux_config_mode, 'enforcing'
-add_custom_fact :sshdsakey, ''
-add_custom_fact :sshecdsakey, ''
-add_custom_fact :sshed25519key, ''
-add_custom_fact :pe_version, ''
-add_custom_fact :sudoversion, '1.8.6p3'
-add_custom_fact :selinux_agent_vardir, '/var/lib/puppet'
-
+include RspecPuppetFacts
+
+default_facts = {
+  puppetversion: Puppet.version,
+  facterversion: Facter.version,
+}

+default_facts_path = File.expand_path(File.join(File.dirname(__FILE__), 'default_facts.yml'))
+default_module_facts_path = File.expand_path(File.join(File.dirname(__FILE__), 'default_module_facts.yml'))

Instead of modifying spec/spec_helper.rb, facts should go in spec/default_facts.yml and spec/default_module_facts.yml. As the former is modified by pdk update, it is easier to maintain the later. Review the diff of spec/spec_helper.rb and spec/default_facts.yml (if we have the latter) for our previous custom facts and their values. When a test is failing for a missing fact, we can add it to spec/default_module_facts.yml in the format factname: “factvalue”.

  1) profile::base::linux on redhat-6-x86_64 disable openshift selinux policy should contain Selmodule[openshift-origin] with ensure => "absent"
     Failure/Error: include concat::setup

     Puppet::PreformattedError:
       Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Empty string title at 0. Title strings must have a length greater than zero. (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/concat/manifests/setup.pp, line: 59, column: 10) (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/ssh/manifests/server/config.pp, line: 12) on node  build

This is related to an older version of puppetlabs/concat (v1.2.5). The latest is v5.1.0. After updating my Puppetfile and .fixtures.yml with the new version, I ran pdk bundle exec rake spec_prep to update the test fixtures, and this is resolved.

  2) profile::base::linux on redhat-6-x86_64 domain_join is true should contain Class[profile::domain_join]
     Failure/Error: include domain_join

     Puppet::PreformattedError:
       Evaluation Error: Error while evaluating a Function Call, Class[Domain_join]:
         expects a value for parameter 'domain_fqdn'
         expects a value for parameter 'domain_shortname'
         expects a value for parameter 'ad_dns'
         expects a value for parameter 'register_account'
         expects a value for parameter 'register_password' (file: /home/rnelson0/puppet/controlrepo/spec/fixtures/modules/profile/manifests/domain_join.pp, line: 8, column: 3) on node build

In this case, the mandatory parameters for an included class were not provided. The let (:params) block of our rspec contexts only allows us to set parameters for the current class. We have been pulling these parameters from hiera instead. This data should be coming from spec/fixtures/hieradata/default.yml, however, hiera lookup settings were also removed in spec/spec_helper.rb, breaking the existing hiera configuratioN:

 RSpec.configure do |c|
-  c.hiera_config = File.expand_path(File.join(__FILE__, '../fixtures/hiera.yaml'))
-  default_facts = {
-    puppetversion: Puppet.version,
-    facterversion: Facter.version
-  }

There is no replacement setting provided. The PDK was designed with hiera use in mind, so add this line (replacing the bold filename if yours is stored elsewhere) to .sync.yml and run pdk update. Your hiera config should start working again:

spec/spec_helper.rb:
  hiera_config: 'spec/fixtures/hiera.yaml'

These are just some common issues with a PDK conversion, but we may have others to resolve. We just need to keep iterating until we get through everything.

Things are looking good! But we are just done with the Puppet 5 rodeo. Before we move on to Puppet 6, now is a good time to make sure syntax validation works, and if you need to make changes to syntax, you then run the Puppet 5 tests that you know should work. Get that all settled before moving on.

Puppet 6 Unit Tests

There’s one big change to be aware of before we even run unit tests against Puppet 6. To make updating core types easier, without requiring a brand new release of Puppet, a number of types were moved into dedicated modules. This means that for Puppet 6 testing, we need to update our Puppetfile and .fixtures.yml (though the puppet agent all-in-one package packages these modules, we do not went our tests relying on an installed puppet agent). When we update these files, we need to make sure we ONLY deploy these core modules on Puppet 6, not Puppet 5- both for the master and testing – or we will encounter issues with Puppet 5. The Puppetfile is actually ruby code, so we can check the version before loading the modules (see note below), and .fixtures.yml accepts a puppet_version parameter to modules. We can click on each module name here to get the link for the replacement module. We do not have to add all of the modules, just the ones we use, but including the ones we are likely to use or have other modules depend on can reduce friction. The changes will look like this:

# Puppetfile
require 'puppet'
# as of puppet 6 they have removed several core modules into seperate modules
if Puppet.version =~ /^6\.\d+\.\d+/
  mod 'puppetlabs-augeas_core', '1.0.3'
  mod 'puppetlabs-cron_core', '1.0.0'
  mod 'puppetlabs-host_core', '1.0.1'
  mod 'puppetlabs-mount_core', '1.0.2'
  mod 'puppetlabs-sshkeys_core', '1.0.1'
  mod 'puppetlabs-yumrepo_core', '1.0.1'
end

# .fixtures.yml
fixtures:
  forge_modules:
    augeas_core:
      repo: "puppetlabs/augeas_core"
      ref: "1.0.3"
      puppet_version: ">= 6.0.0"
    cron_core:
      repo: "puppetlabs/cron_core"
      ref: "1.0.0"
      puppet_version: ">= 6.0.0"
    host_core:
      repo: "puppetlabs/host_core"
      ref: "1.0.1"
      puppet_version: ">= 6.0.0"
    mount_core:
      repo: "puppetlabs/mount_core"
      ref: "1.0.2"
      puppet_version: ">= 6.0.0"
    scheduled_task:
      repo: "puppetlabs/scheduled_task"
      ref: "1.0.0"
      puppet_version: ">= 6.0.0"
    selinux_core:
      repo: "puppetlabs/selinux_core"
      ref: "1.0.1"
      puppet_version: ">= 6.0.0"
    sshkeys_core:
      repo: "puppetlabs/sshkeys_core"
      ref: "1.0.1"
      puppet_version: ">= 6.0.0"
    yumrepo_core:
      repo: "puppetlabs/yumrepo_core"
      ref: "1.0.1"
      puppet_version: ">= 6.0.0"

Note: Having performed an actual upgrade to Puppet 6 now, I do NOT recommend adding the modules to the Puppetfile after all, unless you are specifying a newer version of the modules than is provided with the version of puppet-agent you are using, or you are not using the AIO versions, and ONLY if you have no Puppet 5 agents. Puppet 5 agents connecting to a Puppet 6 master will pluginsync these modules and throw errors instead of applying a catalog. If you do have multiple compile masters, you could conceivably keep a few running Puppet 5 and only have Puppet 5 agents connect to it, but that seems like a really specific and potentially problematic scenario, so in general, I repeat, I do NOT recommend adding the core modules to the Puppetfile. They must be placed in the .fixtures.yml file for testing, though.

Now give pdk test unit a try and see how it behaves. All the missing types will be back, so any errors we see now should be related to actual failures, or some other edge case I did not experience.

Note: I experienced the error Error: Evaluation Error: Error while evaluating a Function Call, undefined local variable or method `created' for Puppet::Pops::Loader::RubyLegacyFunctionInstantiator:Class when running my Puppet 6 tests immediately after working on the Puppet 5 tests. Nothing I found online could resolve this and when I returned to it later, it worked fine. I could not replicate the error, so I am unsure of what caused it. If you run into that error, I suggest starting a new session and running git clean -ffdx to remove unmanaged files, so that you start with a clean environment.

Updating CI/CD Integrations

Once both pdk validate and pdk test unit complete without error, we need to update the automated checks our CI/CD system uses. We all use different systems, but thankfully the PDK has many of us covered. For those who use Travis CI, Appveyor, or Gitlab, there are pre-populated .travis.yml, .appveyor.yml, and .gitlab-ci.yml files, respectively. For those of us who use Jenkins, we have two options: 1) Copy one of these CI settings and integrate them into our build process (traditional or pipeline) or 2) apply profile::pdk to the Jenkins node and use the PDK for tests. Let’s look at the first option, basing it off the Travis CI config:

before_install:
  - bundle -v
  - rm -f Gemfile.lock
  - gem update --system
  - gem --version
  - bundle -v
script:
  - 'bundle exec rake $CHECK'
bundler_args: --without system_tests
rvm:
  - 2.5.0
env:
  global:
    - BEAKER_PUPPET_COLLECTION=puppet6 PUPPET_GEM_VERSION="~> 6.0"
matrix:
  fast_finish: true
  include:
    -
      env: CHECK="syntax lint metadata_lint check:symlinks check:git_ignore check:dot_underscore check:test_file rubocop"
    -
      env: CHECK=parallel_spec
    -
      env: PUPPET_GEM_VERSION="~> 5.0" CHECK=parallel_spec
      rvm: 2.4.4
    -
      env: PUPPET_GEM_VERSION="~> 4.0" CHECK=parallel_spec
      rvm: 2.1.9

We have to merge this into something useful for Jenkins. I am unfamiliar with Pipelines myself (I know, it’s the future, but I have $reasons!), but I have built a Jenkins server with RVM installed and configured it with a freestyle job. Here’s the current job:

#!/bin/bash
[[ -s /usr/local/rvm/scripts/rvm ]] && source /usr/local/rvm/scripts/rvm
# Use the correct ruby
rvm use 2.1.9

git clean -ffdx
bundle install --path vendor --without system_tests
bundle exec rake test

The new before_install section cleans up the local directory, equivalent to git clean -ffdx, but it also spits out some version information and runs gem update. These are optional, and the latter is only helpful if you have your gems cached elsewhere (the git clean will wipe the updated gems otherwise, wasting time). The bundler_args are already part of the bundle install command. The rvm version varies by puppet version, that will need tweaked. The test command is now bundle exec rake $CHECK, with a variety of checks added in the matrix section. test used to do everything in the first 2 matrix sections; parallel_spec just runs multiple tests at once instead of in serial which can be faster. The 3rd and 4th matrix sections are for older puppet versions. We can put this together into multiple jobs, into a single job that tests multiple versions of Puppet, or into a single job testing just one Puppet version. Here’s what a Jenkins job would look like that tests Puppet 6 and 5:

#!/bin/bash
[[ -s /usr/local/rvm/scripts/rvm ]] && source /usr/local/rvm/scripts/rvm

# Puppet 6
export BEAKER_PUPPET_COLLECTION=puppet6
export PUPPET_GEM_VERSION="~> 6.0"
rvm use 2.5.0
bundle -v
git clean -ffdx
# Comment the next line if you do not have gems cached outside the job workspace
gem update --system
gem --version
bundle -v

bundle install --path vendor --without system_tests
bundle exec rake syntax lint metadata_lint check:symlinks check:git_ignore check:dot_underscore check:test_file rubocop
bundle exec rake parallel_spec

# Puppet 5
rvm use 2.4.4
export PUPPET_GEM_VERSION="~> 5.0"
bundle exec rake parallel_spec

This creates parity with the pre-defined CI tests for other services.

The other option is adding profile::pdk to the Jenkins server, probably through the role, and use the PDK to run tests. That Jenkins freestyle job looks a lot simpler:

#!/bin/bash
PDK=/usr/local/bin/pdk
echo -n "PDK Version: "
$PDK --version

# Puppet 6
git clean -ffdx
$PDK validate
$PDK test unit

# Puppet 5
git clean -ffdx
PDK_PUPPET_VERSION=5 $PDK test unit

This is much simpler, and it should not need updated until removing Puppet 5 or adding Puppet 7 when it is released, whereas the RVM version in the bundle-version may need tweaked throughout the Puppet 6 lifecycle as Ruby versions change. However, the tests aren’t exactly the same. Currently, pdk validate does not run the rake target check:git_ignore, and possibly other check: tasks. In my opinion, as the pdk improves, the benefit of only having to update the PDK package version and not the git-based automation outweighs the single missing check and the maintenance of RVM on a Jenkins server. And for those of us using Travis CI/Appveyor/Gitlab-CI, it definitely makes sense to stick with the provided test setup as it requires almost no maintenance.

I used this earlier without explaining it, but the PDK also provides the ability to run pdk bundle, similar to the native bundle but using the vendored ruby and gems provided by the PDK. We can run individual tests like pdk bundle exec rake check:git_ignore, or install the PDK and modify the “bundle” Jenkins job to recreate the bundler setup using the PDK and not have to worry about RVM at all. I’ll leave that last as an exercise for the user, though.

We must be sure to review our entire Puppet pull request process and see what other integrations need updated, and of course we must update documentation for our colleagues. Remember, documentation is part of “done”, so we cannot say we are done until we update it.

Finally, with all updates in place, submit a Pull Request for your controlrepo changes. This Pull Request must go through the new process, not just to verify that it passes, but to identify any configuration steps you missed or did not document properly.

Summary

Today, we looked at converting our controlrepo to use the Puppet Development Kit for testing instead of bundler-based testing. It required lots of changes to our controlrepos, many of which the PDK handled for us via autocorrect; others involved manual updates. We reviewed a variety of changes required for CI/CD integrations such as Travis CI or Jenkins. We reviewed the specifics of our setup that others don’t share so we had a working setup top to bottom, and we updated our documentation so all of our colleagues can make use of the new setup. Finally, we opened a PR with our changes to validate the new configuration.

By using the PDK, we have leveraged the hard work of many Puppet employees and Puppet users alike who provide a combination of rigorously vetted sets of working dependencies and beneficial practices, and we can continue to improve those benefits by simply updating our version of the PDK and templates in the future. This is a drastic reduction in the mental load we all have to carry to keep up with Puppet best practices, an especially significant burden on those responsible for Puppet in their CI/CD sysytems. I would like to thank everyone involved with the PDK, including David Schmitt, Lindsey Smith, Tim Sharpe, Bryan Jen, Jean Bond, and the many other contributors inside and outside Puppet who made the PDK possible.

I was pretty excited when I read the new features in Puppet Enterprise 2018.1. There are a lot of cool new features and fixes, but the backup/restore feature stood out for me. Even with just 5 VMs at home, I don’t want to rock the boat when rebuilding my master by losing my CA or agent certs, much less with a lot more managed nodes at work, and all the little bootstrap requirements have changed since I started using PE in 2014. Figuring out how to get everything running myself would be possible, but it would take a while and be out of date in a few months anyway. Then there is everything in PuppetDB that I do not want to lose, like collected facts/resources and run reports.

Not coincidentally, I still had a single CentOS 6 VM around because it was my all-in-one puppet master, and migrating to CentOS 7 was not something I looked forward to due to the anticipated work it would require. With the release of this feature, I decided to get off my butt and do the migration. It still took over a month to make it happen, between other work, and I want to share my experience in the hope it saves someone else a bit of pain.

Create your upgrade outline

I want to summarize the plan at a really high level, then dive in a bit deeper. Keep in mind that I have a single all-in-one master using r10k and my plan does not address multi-master or split deployments. Both of those deployment models have significantly different upgrade paths, please be careful if you try and map this outline onto those models without adjusting. For the all-in-one master, it’s pretty simple:

• Backup old master
• Deploy a new master VM running EL7
• Complete any bootstrapping that isn’t part of the backup
• Install the same version of PE
• Restore the old master’s backup onto the new master
• Run puppet
• Point agents at the new master

I will cover the backup/restore steps at the end, so the first step to cover is deploying a new master. This part sounds simple, but if Puppet is currently part of your provisioning process and you only have one master, you’ve got a catch 22 situation – new deployments must talk to puppet to complete without errors, and if you deploy a new puppet master using the same process, it will either fail to communicate with itself since PE is not installed, or it will talk to a PE installation that does not reflect your production environment. We need to make sure that we have the ability to provision without puppet, or be prepared for some manual efforts in the deploy. With a single master, manual efforts aren’t that burdensome, but can still reduce accuracy, which is why I prefer a modified automated provisioning workflow.

A lot of bootstrapping – specifically hiera and r10k/code manager – should be handled by the restore. There were just a few things I needed to do:

• Run ssh-keygen/install an existing key and attach that key to the git system. You can avoid this by managing the ssh private/public keys via file resources, but you will not be able to pull new code until puppet processes that resource.
• SSH to your git server and accept the key. You can avoid this with the sshkey resource, with the same restriction.
• Check your VMs default iptables/selinux posture. I suggest managing security policy via puppet, which should prevent remote agents from connecting before the first puppet run, but it’s also possible to prevent the master from communicating with itself with the wrong default policy.
• Check the hostname matches your expectations. All of /etc/hosts, /etc/hostname, /etc/sysconfig/network should list the short and FQDN properly, and hostname; hostname -f should return the same values. /etc/resolv.conf may also need the search domain. Fix any issues before installing PE, as certs are generated during install, and having the wrong hostname result can cause cascading faults best addressed by starting over.

The restore should get the rest from the PE side of things. If your provisioning automation performs other work that you had to skip, make sure you address it now, too.

Installing PE is probably the one manual step you cannot avoid. You can go to https://support.puppet.com and find links to current and past PE versions. Make sure you get the EL7 edition and not the EL6 edition. I did not check with Support, but I assume that you must restore on the same version you backed up, I would not risk even a patch release difference.

Skipping the restore brings us to running the agent, a simple puppet agent -t on the master, or waiting 30 minutes for the run to complete on its own.

The final step may not apply to your situation. In addition to refreshing the OS of the master, I switched to a new hostname. If you’re dropping your new master on top of the existing one’s hostname/IP, you can skip this step. I forked a new branch from production called mastermigration. The only change in this branch is to set the server value in /etc/puppetlabs/puppet/puppet.conf. There are a number of ways to do this, I went with a few ini_setting resources and a flag manage_puppet_conf in my profile::base::linux. The value should only be in one of the sections main or agent, so I ensured it is in main and absent elsewhere:

  if $manage_puppet_conf {
    # These settings are very useful during migration but are not needed most of the time
    ini_setting { 'puppet.conf main server':
      ensure => present,
      path => '/etc/puppetlabs/puppet/puppet.conf',
      section => 'main',
      setting => 'server',
      value => 'puppet.example.com',
    }
    ini_setting { 'puppet.conf agent server':
      ensure => absent,
      path => '/etc/puppetlabs/puppet/puppet.conf',
      section => 'agent',
      setting => 'server',
    }
  }

During the migration, I can just set profile::base::linux::manage_puppet_conf: true in hiera for the appropriate hosts, or globally, and they’ll point themselves at the new master. Later, I can set it to false if I don’t want to continue managing it (while there is no reason you cannot leave the flag enabled, by leaving it as false normally you can ensure that changing the server name here does not take effect unless purposefully flip the flag; you could also parameterize the server name).

Now let’s examine the new feature that makes it go.

Backups and Restores

Puppet’s documentation on the backup/restore feature provides lots of detail. It will capture the CA and certs, all your currently deployed code, your PuppetDB contents including facts, and almost all of your PE config. About the only thing missing are some gems, which you should hopefully be managing and installing with puppet anyway.

Using the new feature is pretty simple, puppet-backup createor puppet-backup restore <filename> will suffice for this effort. There are a few options for more fine-grained control, such as backup/restore of individual scopes with --scope=<scopes>[,<additionalscopes>...], e.g. --scope=certs.

The backup will only backup the current PE edition’s files, so if you still have /etc/puppet on your old master from PE 3 days, that will not be part of the backup. However, files in directories it does back up, like /etc/puppetlabs/puppet/puppet.conf.rpmsave, will persist. This will help reduce cruft, but not eliminate it. You will still need to police on-disk content. In particular, if you accidentally placed a large file in /etc/puppetlabs, say the PE install tarball, that will end up in your backup and can inflate the size a bit. If you feel the backup is exceptionally large, you may want to search for large files in that path.

The restore docs also specify two commands to run after a restore when Code Manager is used. If you use CM, make sure not to forget this step:

puppet access login
puppet code deploy --all --wait 

The backup and restore process are mostly time-dependent on the size of your puppetdb. With ~120 agents and 14 days of reports, it took less than 10 minutes for either process and generated a ~1G tarball. Larger environments may expect the master to be offline for a bit longer, if they want to retain their full history.

Lab it up

The backup/restore process is great, but it’s new, and some of us have very ancient systems laying around. I highly recommend testing this in the lab. My test looked like this:

• Clone the production master to a VM on another hostname/IP
• Run puppet-backup create
• Fully uninstall PE (sudo /opt/puppetlabs/bin/puppet-enterprise-uninstaller -p -d -y)
• Remove any remaining directories with puppet in them, excepting the PE 2018 install files, to ensure all cruft is gone
• Disable and uninstall any r10k webhook or puppet-related services that aren’t provided by PE itself.
• Reboot
• Bootstrap (from above)
• Install PE (sudo /opt/puppetlabs/bin/puppet-enterprise-installer) only providing an admin password for the console
• Run puppet-backup restore <backup file>
• Run puppet agent -t
• Make sure at least one agent can check in with puppet agent -t --server=<lab hostname> (clone an agent too if need be)
• Reboot
• Make sure the master and agent can still check in, Console works, etc.
• If possible, test any systems that use puppet to make sure they work with the new master
• Identify any missing components/errors and repeat the process until none are observed

I mentioned that I used PE3. My master had been upgraded all the way from version 3.7 to 2018.1.2. I’m glad I tested this, because there were some unexpected database settings that the restore choked on. I had to engage Puppet Support who provided the necessary commands to update the database so I could get a useful backup. This also allowed me to identify all of my bootstrap items and of course, gain familiarity and confidence with the process.

This became really important for me because, during my production migration, I ran into a bug in my provisioning system where the symptom presented itself through Puppet. Because I was very practiced with the backup/restore process, I was able to quickly determine PE was NOT the problem and correctly identify the faulty system. Though it took about 6 hours to do my “very quick” migration, only about an hour of that was actually spent on the Puppet components.

I also found a few pieces of managed files on the master where the code presumed the directory structure would already be there, which it turns out was not the case. I must have manually created some directories 4 years ago. I think the most common issues you would find at this point are dependencies and ordering, but there may be others. Either fix the code now or, if it would negatively affect the production server, prep a branch for merging just prior to the migration, with the plan to revert if you rollback.

I strongly encourage running through the process a few times and build the most complete checklist you can before moving on to production.

Putting it together

With everything I learned in the lab, my final outline looked like this:

• Backup old master, export to another location
• Deploy a new master VM running EL7 using an alternative workflow
• Run ssh-keygen/install an existing key and attach that key to the git system
• SSH to the git server and accept the key
• Verify your VMs default iptables/selinux posture; disable during bootstrap if required
• Validate the hostname is correct
• Install PE
• Restore the backup
• [Optional] Merge any code required for the new server; run r10k/CM to ensure it’s in place on the new master
• Run puppet
• Point agents at the new master

Yours may look slightly different. Please, spend the time in the lab to practice and identify any missing steps, it’s well worth it.

Summary

Refreshing any system of significant age is always possible, and often fraught with manual processes that are prone to error. Puppet Enterprise 2018.1 delivered a new backup/restore process that automates much of this process. We have put together a rough outline, refined it in the lab, and then used it to perform the migration in production with high confidence, accounting for any components the backup did not include. I really appreciate this new feature and I look forward to refinements in the future. I hope that soon enough migrations should be as simple an effective as in-place upgrades.

As I lamented in my article on converting to the PDK, I really do not like Rubocop and was disappointed I could not turn it off. Thankfully, that was addressed in PDK-998 and the fix was included in time for PDK 1.6.0! Disabling it is pretty simple and though it’s strictly a fix to pdk-templates, updating the PDK won’t hurt.

First, update to PDK 1.6.0. As I use CentOS 7 and the RPM packaging, it’s as simple as sudo yum update pdk -y; follow the directions that match system. Next, we need to add the following lines to .sync.yml:

.rubocop.yml:
  selected_profile: off

Finally, run pdk update, or if you weren’t already using pdk-templates, run pdk convert --template-url=https://github.com/puppetlabs/pdk-templates (I will assume the former to keep it simple). You can add --noop (or say n) and review update.txt|convert.txt to see the differences before applying, or, because you are using version control, just run a diff afterward to see the changes.

[rnelson0@build03 domain_join:pdk160]$ pdk update
pdk (INFO): Updating rnelson0-domain_join using the template at https://github.com/puppetlabs/pdk-templates, from master@041eeb2 to 1.6.0

----------Files to be modified----------
metadata.json
.pdkignore
spec/spec_helper.rb
.gitignore
Rakefile
.rubocop.yml
Gemfile

----------------------------------------

You can find a report of differences in update_report.txt.

Do you want to continue and make these changes to your module? Yes
[✔] Installing missing Gemfile dependencies.

------------Update completed------------

7 files modified.

That’s it! Check the contents of .rubocop.yml and you will notice everything is false (just a snippet because it’s loooong):

---
require: rubocop-rspec
AllCops:
  DisplayCopNames: true
  TargetRubyVersion: '2.1'
  Include:
  - "./**/*.rb"
  Exclude:
  - bin/*
  - ".vendor/**/*"
  - "**/Gemfile"
  - "**/Rakefile"
  - pkg/**/*
  - spec/fixtures/**/*
  - vendor/**/*
  - "**/Puppetfile"
  - "**/Vagrantfile"
  - "**/Guardfile"
Bundler/DuplicatedGem:
  Enabled: false
Bundler/OrderedGems:
  Enabled: false
Layout/AccessModifierIndentation:
  Enabled: false
Layout/AlignArray:
  Enabled: false
Layout/AlignHash:
  Enabled: false
Layout/AlignParameters:
  Enabled: false
Layout/BlockEndNewline:
  Enabled: false
Layout/CaseIndentation:
  Enabled: false
Layout/ClosingParenthesisIndentation:
  Enabled: false
Layout/CommentIndentation:
  Enabled: false

Running validation now finds no issues with ruby syntax no matter how much you ignore style guides:

# master, prior to updating

[rnelson0@build03 domain_join:master]$ pdk validate
...
[✖] Checking Ruby code style (**/**.rb).
info: task-metadata-lint: ./: Target does not contain any files to validate (tasks/*.json).
convention: rubocop: spec/spec_helper_acceptance.rb:17:27: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.
convention: rubocop: spec/spec_helper_acceptance.rb:17:49: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.
convention: rubocop: spec/spec_helper_acceptance.rb:19:66: Style/BracesAroundHashParameters: Redundant curly braces around a hash parameter.
convention: rubocop: spec/spec_helper_acceptance.rb:19:68: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.
convention: rubocop: spec/spec_helper_acceptance.rb:19:96: Layout/SpaceAfterComma: Space missing after comma.
convention: rubocop: spec/acceptance/class_spec.rb:6:9: RSpec/ExampleWording: Do not use should when describing your tests.
convention: rubocop: spec/acceptance/class_spec.rb:12:26: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.
convention: rubocop: spec/acceptance/class_spec.rb:13:26: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.
convention: rubocop: spec/classes/domain_join_spec.rb:2:25: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.

# pdk160, after updating, no code changes

[rnelson0@build03 domain_join:pdk160]$ pdk validate
pdk (INFO): Running all available validators...
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.2
[✔] Checking metadata syntax (metadata.json tasks/*.json).
[✔] Checking module metadata style (metadata.json).
[✔] Checking Puppet manifest syntax (**/**.pp).
[✔] Checking Puppet manifest style (**/*.pp).
[✔] Checking Ruby code style (**/**.rb).
info: task-metadata-lint: ./: Target does not contain any files to validate (tasks/*.json).

You may have noticed there are quite a few other files updated. The other significant change is that a :changelog task via github_changelog_generator is now included, so you can remove that from your .sync.yml if you added it and replace it with the recommended config (via the Rakefile):

---
Gemfile:
  optional:
    ':development':
      - gem: 'github_changelog_generator'
        git: 'https://github.com/skywinder/github-changelog-generator'
        ref: '20ee04ba1234e9e83eb2ffb5056e23d641c7a018'
        condition: "Gem::Version.new(RUBY_VERSION.dup) >= Gem::Version.new('2.2.2')"

The other changes are pretty minor, in some cases cosmetic, but of course review them to make sure they’re OK. Submit a PR or equivalent and make sure the tests pass before merging. You can follow along with today’s blog post in domain_join PR35, too.

Enjoy!

At PuppetConf 2017, Puppet Tasks were introduced as part of the new project Bolt. A task allows you to run a program on an arbitrary number of nodes. The program can be just about anything, it just needs to be written in a language that the target nodes can run. For Linux, that means pretty much anything – bash, python, perl, ruby, etc. On Windows, you’re a little more limited out of the box – powershell primarily. Bolt is not yet at version 1.0.0, so I suspect language support for Windows will change. You can use Bolt on its own (even without Puppet, apparently), and starting with Puppet Enterprise 2017.3, you can use Bolt at the PE Console as “tasks” in the UI.

For my first task, I simply want to run a single command on a list of nodes. While I can run arbitrary commands with the bolt command line, I want the practice of writing a task. My use case involves an external authentication system that manages users, ssh keys, and sudo configurations. When a change is made, nodes need to pull the changes. Often, a delay there does not matter – the nodes will receive the change soon enough – but sometimes I want the relevant nodes to pick it up immediately. To do so, I need it to run a single perl script, sanitized as /usr/bin/external_auth.pl, and I want to do it on all the nodes with the profile::external_auth class.

Continue reading →

A few years ago, I set up my modules with a bundle-based test setup and modulesync and wrote a companion blog post. Since that was written, a lot of things have changed with puppet. One of those is the release last year of the Puppet Development Kit (PDK). The goal of the PDK is to simply development of puppet code and modules by reducing or eliminating all of the headaches of creating a mature ruby/bundler/puppet-lint/etc. setup. There is also a brand new tool called pdksync that combines the PDK with the power of modulesync. I was somewhat involved in the initial efforts toward the PDK, through my work on puppet-lint, but I have not actually used the PDK “in anger” yet, in part because of my previously working modulesync setup. This seems like a great opportunity to switch to PDK and pdksync, starting with the PDK.

Why PDK?

Before we begin, we should look at why we want to use the Puppet Development Kit. My current setup is best described as fragile. Its effectiveness varies based on what version of Ruby I use and the version of the gems I happen to download via bundler on any given day. I use CentOS 7, which is stuck on Ruby 2.0. Most of the gems in my setup require at least Ruby 2.1 or 2.2, so I have to resort to RVM to provide me with Ruby 2.3.1. Someday, I’ll need to update that to Ruby 2.4 for a gem and my setup will break until I fix it.

I am also downloading a bunch of gems that are not pinned and updated versions can bring in subtle bugs or cascading failures not related to changes in my code. Sometimes the gem is directly related to my work, like a puppet-lint version with a bug that I can downgrade and pin. Other times, it’s a very indirect dependency of a dependency of a dependency to puppet-lint, and pinning it only creates more problems for everything that depends on it. Of course, bundler also relies on rubygems.org and internet/mirror access, which sometimes go down when you need them most.

While these are surmountable issues, they always come up while I’m trying to get something done in puppet, and the minutes or hours required to fix the problem prevents me from making the changes I need, when I need.

The PDK resolves this by bundling its own version of Ruby and dependent gems. Puppet vets the setup, so I do not have to. Everything is on disk, so there’s no more required downloading of gems that can be pulled or unavailable because of network issues. This is a huge benefit to all users, whether they pay for Puppet Enterprise or use Puppet Opensource Edition for free. Less time spent worrying about dependency hell and more time getting straight to work. This is important, valuable work, but it’s not my expertise or an actual goal of my job, so I am very content to let someone else handle the setup so I can spend more time managing my systems with Puppet.

The PDK is an installable tool. Install once and you can use it with all your puppet modules and controlrepos. Upgrading the PDK is simple using your package manager. You can of course combine using the PDK on some modules and stick with the ruby/bundler setup on others. However, it will be more difficult (but not impossible) to switch between the PDK and native bundler on the same module – our CI systems will use native bundler after all – but some gem dependencies will no longer be pinned and we lose the guarantee of pinned gems that will work together.

We will see below that you can still modify the setup on each module/controlrepo to some extent, but when using the PDK, the full range of customization bundler offered is unavailable to you. I think most people will not find this to be a problem, but you should definitely read up on the PDK to make sure you understand what you gain and lose before converting to using it. If you change your mind later, switching back from PDK to a bundler-based setup is possible, but it may involve some work to find a working setup of pinned gem versions.

Installing the PDK

The very first thing I need to do is install the PDK. The following is written using PDK v1.5.0.0. The PDK is relatively new and gets frequent updates, so this may become out of date rapidly. If you run into any issues, check the version, read the release notes, and adjust accordingly.

The docs decribe how to install on various systems. I use EL7 so I will install the RPM. I also use Puppet Enterprise, not Puppet Opensource, so I have to add the Puppet repository first. rpm and yum can get me there, or I can use puppet apply:

# Manual
sudo rpm -Uvh https://yum.puppet.com/puppet5/puppet5-release-el-7.noarch.rpm
sudo yum install pdk -y

# puppet apply
cat > ~/pdk.pp < EOF
package { 'puppet5-release-el-7':
  ensure => present,
  provider => 'rpm',
  source => 'https://yum.puppet.com/puppet5/puppet5-release-el-7.noarch.rpm',
}
-> package { 'pdk':
  ensure => present,
}
EOF
sudo puppet apply ~/pdk.pp

I can now call the command pdk successfully. Be aware that it includes its own bundled ruby, so the first time you run it may take a little time to be loaded and cached, which is expected.

[rnelson0@build03 domain_join:master]$ pdk --help
NAME
    pdk - Puppet Development Kit

USAGE
    pdk command [options]

DESCRIPTION
    The shortest path to better modules.

COMMANDS
    build        Builds a package from the module that can be published to the Puppet Forge.
    bundle       (Experimental) Command pass-through to bundler
    convert      Convert an existing module to be compatible with the PDK.
    help         show help
    module       Provide CLI-backwards compatibility to the puppet module tool.
    new          create a new module, etc.
    test         Run tests.
    update       Update a module that has been created by or converted for use by PDK.
    validate     Run static analysis tests.

OPTIONS
    -d --debug                    Enable debug output.
    -f --format=           Specify desired output format. Valid
                                  formats are 'junit', 'text'. You may also
                                  specify a file to which the formatted
                                  output is sent, for example:
                                  '--format=junit:report.xml'. This option
                                  may be specified multiple times if each
                                  option specifies a distinct target file.
    -h --help                     Show help for this command.
       --version                  Show version of pdk.

If you sign up for the puppet-announce mailing list, you will be notified every time there’s a new PDK release. After reading the release notes for edge cases that may impact you, can easily upgrade to the latest version with your distro’s equivalent of yum update pdk. That is a lot easier than updating ruby/bundler setups.

Converting to the PDK

Next, my existing setup must be converted to PDK. I will walk through my efforts, but you can also review the PuppetConf 2017 video and slides about the PDK in addition to this Converting To PDK doc. I am working on my domain_join module first, starting from the release candidate for v0.5.2, if you want to recreate this effort. The module is part of my modulesync config and has 68 tests. It’s a reasonably mature module but not overly complex, perfect for testing without being too deep or shallow. I am also going to break the modulesync setup, which you can see here. Before beginning, I create a new branch on the module:

[rnelson0@build03 domain_join:master]$ git checkout -b pdk
Switched to a new branch 'pdk'

Because I use modulesync, this will not work out of the box for me, but there’s a very naive default pdk convert that can be used to update the config. It will inform you of the files that will be added/modified and prompts you to continue due to the potential for destruction. As noted, this concern is mitigated by using version control, and if you’ve read my blog before, you’re obviously using version control, right? If not, get that done first! (it’s beyond the scope of this article, but my git 101 article may help) Here’s what the naive attempt looks like:

[rnelson0@build03 domain_join:pdk]$ pdk convert

------------Files to be added-----------
.pdkignore
.project
spec/default_facts.yml
.gitlab-ci.yml
appveyor.yml

----------Files to be modified----------
metadata.json
spec/spec_helper.rb
.gitignore
.travis.yml
Rakefile
.rubocop.yml
.rspec
Gemfile

----------------------------------------

You can find a report of differences in convert_report.txt.

pdk (INFO): Module conversion is a potentially destructive action. Ensure that you have committed your module to a version control system or have a backup, and review the changes above before continuing.
Do you want to continue and make these changes to your module? Yes

------------Convert completed-----------

5 files added, 8 files modified.

[rnelson0@build03 domain_join:pdk±]$

The git diff is really, really lengthy, but you can find it here. A lot of it is simply re-arranging of stanzas in existing files (.gitignore, .rspec, .travis.yml, metadata.json) and .rubocopy.yml updates. The rest is mostly in three files: Rakefile, Gemfile and spec/spec_helper.rb. It also adds five files: .gitlab-ci.yml, .pdkignore, .project, appveyor.yml, and spec/default_facts.yml. Some info on the minor changes:

• spec/default_facts.yml: If you had default facts in your spec/spec_helper.rb file, you should move them here. The rest is mostly “housekeeping” but it removes my hiera config, which I will explore in a moment.
• .gitignore, .pdkignore: The former has been updated a bit, and the latter is the exact same thing.
• .gitlab-ci.yml, .travis.yml, appveyor.yml: Puppet provides some good defaults for a number of external systems. I am sticking with Travis CI right now, but it’s great to have defaults for other services if I branch out. The latter looks targeted at testing Windows systems, too, something that’s often problematic. These are all optional but do not hurt by being present.
• .project: Looks like some XML for use with the editor Eclipse.
• .rubocop.yml: I really don’t like rubocop but it’s included. I plan to disable rubocop as quickly as possible. However, this addresses one of my pain points – every version of rubocop changes the name of Cops, and it fails to run if it finds an unknown Cop name in its config. Since Puppet vets the config, I do not have to deal with finding all the new Cop information everytime rubocop updates. It’s not enough for me to love it, but it is significant pain reduction.

This leaves the big three files mentioned earlier, which are worthy of more detailed investigation.

Rakefile

This file is MUCH smaller now, I presume in thanks due to some pdk magic. The conversion removed a changelog task I created, so I need to get this back.

GitHubChangelogGenerator::RakeTask.new :changelog do |config|
 version = (Blacksmith::Modulefile.new).version
 config.future_release = "v#{version}"
 config.header = "# Change log\n\nAll notable changes to this project will be documented in this file.\nEach new release typically also includes the latest modulesync defaults.\nThese should not impact the
 config.exclude_labels = %w{duplicate question invalid wontfix modulesync}
end

I also have a number of puppet-lint checks I have disabled, like arrow_alignment, that I need to make sure are restored. After restoring my task and disabled checks, I will be okay with this new, slimmer default.

Gemfile

PDK includes its own version of ruby and bundler and is guaranteed to deliver a gemset with all the dependencies needed to work together. You can run pdk bundle exec gem list to see what it includes, if you are curious what those are. I will add the github_changelog_generator gem here soon, but otherwise as long as everything works, I have no need to poke at this file anymore.

spec/spec_helper.rb

Though the diff is fairly long for this file, there is nothing tricky here, it just connects the new default facts and some other common practices. It DOES remove the hiera configuration. There is a more modern version of my hiera_config that we need to add back in:

RSpec.configure do |c|
  c.hiera_config = 'spec/fixtures/hiera/hiera.yaml'
end

The naive conversion is not that bad for my setup, but it does leave me with three changes to make to keep functional parity: add the github_changelog_generator gem, the :changelog rake task, and re-enable hiera lookups.

Updating the PDK Setup

Now that I’ve identified the non-default changes needed, I can do some updates. The PDK can convert and update modules using a template system. The template it used is listed at the bottom of metadata.json. You can find the templates online, or clone that directory and examine the moduleroot contents (the moduleroot_init directory is also used when you run pdk new):

[rnelson0@build03 domain_join:pdk±]$ git diff metadata.json
diff --git a/metadata.json b/metadata.json
index 7730b90..381aff7 100644
--- a/metadata.json
+++ b/metadata.json
@@ -27,5 +27,8 @@
"name": "puppet",
"version_requirement": ">=4.0.0"
}
- ]
+ ],
+ "pdk-version": "1.5.0",
+ "template-url": "file:///opt/puppetlabs/pdk/share/cache/pdk-templates.git",
+ "template-ref": "1.5.0-0-gd1b3eca"
}

The copies on disk are from the RPM, and are almost definitely out of date. The latest templates are on GitHub. I can re-run the conversion with pdk convert --template-url=https://github.com/puppetlabs/pdk-templates. The changes for me are pretty small but will be much larger the further away in time you are from the date of the RPM build. After running it, the template info will also be updated:

+ ],
+ "pdk-version": "1.5.0",
+ "template-url": "https://github.com/puppetlabs/pdk-templates",
+ "template-ref": "heads/master-0-g7b5f6d2"

We can look at the individual templates here or clone it locally. The first thing to note is that frequently the .erb templates are dynamic data, rather than static. The simplest change is in spec/spec_helper.rb, just adding a single stanza to the Rspec.configure section, which is also dynamic:

RSpec.configure do |c|
  c.default_facts = default_facts
  <%- if @configs['hiera_config'] -%>
  c.hiera_config = "<%= @configs['hiera_config'] %>"
  <%- end -%>
  <%- if @configs['strict_level'] -%>
    c.before :each do
    # set to strictest setting for testing
    # by default Puppet runs at warning level
    Puppet.settings[:strict] = <%= @configs['strict_level'] %>
  end
  <%- end -%>
end

I highlighted the conditional that will populate the filename with the contents of configs['hiera_config']. The configs hash is populated by config_defaults.yml. The README has a lot of helpful information on the defaults. There’s just a few lines for the spec_helper.rbfile:

[rnelson0@build03 pdk-templates:master]$ tail -2 config_defaults.yml
spec/spec_helper.rb:
  strict_level: ":warning"

I need to add to this hash, but I cannot add to the templates since they are upstream. Thankfully, there’s a built in way to account for this. The contents of the configs hash are combined with the same hash taken from the local.sync.yml file!

Note: if you’d like you CAN change the templates by forking puppetlabs/pdk-templates and passing in --template-url when you call pdk new, convert, or update. You are then on the hook for updating your templates over time, though.

To make use of the sync file, I just need to add it to the root of my module directory and add the custom config. It is additive, so only differences need to be present. Here is the hiera_config value required:

[rnelson0@build03 domain_join:pdk±]$ cat .sync.yml
spec/spec_helper.rb:
  hiera_config: 'spec/fixtures/hiera.yaml'

With the use of the pdk update command, I can re-apply the templates in --noop mode and see the change:

[rnelson0@build03 domain_join:pdk±]$ pdk update --noop
pdk (INFO): Updating rnelson0-domain_join using the default template, from 1.5.0 to 1.5.0

----------Files to be modified----------
spec/spec_helper.rb

----------------------------------------

You can find a report of differences in update_report.txt.

[rnelson0@build03 domain_join:pdk±]$ cat update_report.txt
/* Report generated by PDK at 2018-05-29 20:08:11 +0000 */


--- spec/spec_helper.rb 2018-05-29 18:53:09.140882197 +0000
+++ spec/spec_helper.rb.pdknew 2018-05-29 20:08:11.819978562 +0000
@@ -28,6 +28,7 @@

 RSpec.configure do |c|
   c.default_facts = default_facts
+  c.hiera_config = spec/fixtures/hiera.yaml
   c.before :each do
   # set to strictest setting for testing
   # by default Puppet runs at warning level

Now that we have proved out the process, I need to make a few more changes. To add the github_changelog_generator, I add an array entry under Gemfile: required: ':development'. To add the task, I use Rakefile: extras:for the rake task, one entry per line (you can also use multi-line content in yaml if you prefer). This is what the file looks like as well as the pending changes:

[rnelson0@build03 domain_join:pdk±]$ cat .sync.yml
spec/spec_helper.rb:
  hiera_config: 'spec/fixtures/hiera.yaml'
Gemfile:
  required:
    ':development':
      - gem: github_changelog_generator
Rakefile:
  default_disabled_lint_checks:
    - 'arrow_alignment'
    - 'class_inherits_from_params_class'
    - 'class_parameter_defaults'
    - 'documentation'
    - 'single_quote_string_with_variables'
  extras:
    - "require 'github_changelog_generator/task'"
    - 'GitHubChangelogGenerator::RakeTask.new :changelog do |config|'
    - '  version = (Blacksmith::Modulefile.new).version'
    - '  config.future_release = "v#{version}"'
    - '  config.header = "# Change log\n\nAll notable changes to this project will be documented in this file.\nEach new release typically also includes the latest modulesync defaults.\nThese should not impact the functionality of the module."'
    - '  config.exclude_labels = %w{duplicate question invalid wontfix modulesync}'
    - 'end'
[rnelson0@build03 domain_join:pdk±]$ pdk update --noop
pdk (INFO): Updating rnelson0-domain_join using the template at https://github.com/puppetlabs/pdk-templates, from master@7b5f6d2 to 1.5.0

----------Files to be modified----------
spec/spec_helper.rb
Rakefile
Gemfile

----------------------------------------

You can find a report of differences in update_report.txt.

[rnelson0@build03 domain_join:pdk±]$ cat update_report.txt
/* Report generated by PDK at 2018-05-29 20:39:41 +0000 */


--- spec/spec_helper.rb 2018-05-29 20:33:16.488401096 +0000
+++ spec/spec_helper.rb.pdknew  2018-05-29 20:39:41.492124202 +0000
@@ -28,6 +28,7 @@

 RSpec.configure do |c|
   c.default_facts = default_facts
+  c.hiera_config = "spec/fixtures/hiera.yaml"
   c.before :each do
     # set to strictest setting for testing
     # by default Puppet runs at warning level


--- Rakefile    2018-05-29 20:33:16.489401137 +0000
+++ Rakefile.pdknew     2018-05-29 20:39:41.492832995 +0000
@@ -3,4 +3,11 @@
 require 'puppet_blacksmith/rake_tasks' if Bundler.rubygems.find_name('puppet-blacksmith').any?

 PuppetLint.configuration.send('disable_relative')
+
+require 'github_changelog_generator/task'
+GitHubChangelogGenerator::RakeTask.new :changelog do |config|
+  version = (Blacksmith::Modulefile.new).version
+  config.future_release = "v#{version}"
+  config.header = "# Change log\n\nAll notable changes to this project will be documented in this file.\nEach new release typically also includes the latest modulesync defaults.\nThese should not impact the functionality of the module."
+  config.exclude_labels = %w{duplicate question invalid wontfix modulesync}
+end


--- Gemfile     2018-05-29 20:16:10.321541394 +0000
+++ Gemfile.pdknew      2018-05-29 20:39:41.494035036 +0000
@@ -34,6 +34,7 @@
   gem "puppet-module-win-default-r#{minor_version}",   require: false, platforms: [:mswin, :mingw, :x64_mingw]
   gem "puppet-module-win-dev-r#{minor_version}",       require: false, platforms: [:mswin, :mingw, :x64_mingw]
   gem "puppet-blacksmith", '~> 3.4',                   require: false, platforms: [:ruby]
+  gem "github_changelog_generator",                    require: false
 end

 puppet_version = ENV['PUPPET_GEM_VERSION']

I now run it in yesop mode and my changes take. A quick check of rake targets confirms it. Note that all pdk bundle output is written to STDERR, not STDOUT.

[rnelson0@build03 domain_join:pdk±]$ pdk bundle exec rake -T 2>&1 | grep change
rake changelog # Generate a Change log from GitHub

I did not add the puppet-lint disable checks back in here. That is because the PDK does not use the Rakefile when running puppet-lint, it relies on the configuration file. I need to create .puppet-lint.rc at the top of the repo so that the settings are available to my CI system. That file looks like this:

[rnelson0@build03 domain_join:pdk±]$ cat .puppet-lint.rc
--no-arrow_alignment-check
--no-class_inherits_from_params_class-check
--no-documentation-check
--no-single_quote_string_with_variables-check

One difference between the Rake target and the config file is that an invalid check name in the config file can cause errors, whereas the Rake setting just doesn’t do anything. I removed the class_parameter_defaults check from the list because it is no longer a valid check.

There are a lot more things you might want to change, especially if you use CI other than Travis, but this should be enough for me to gain parity with my existing setup. Remember that you can poke at the templates online, find the default settings in config_defaults.yml, tweak in your own .sync.yml, re-run pdk update and everything should work out. If the templates cannot be wrangled as is, you can always open a ticket in the PDK project.

Make sure you commit your changes and push them up to version control, eventually to be merged into master.

First Test

Now I need to run my tests. Before I do that, I clean up everything not in git. Since I have developed in this directory, there are bundler files that don’t need to be there and may cause conflicts with the tests. Again, make sure you’ve committed changes first, or some of your uncommitted changes from the conversion will be removed:

[rnelson0@build03 domain_join:pdk±]$ git clean -ffdx
Removing .bundle/
Removing Gemfile.lock
Removing bin/
Removing convert_report.txt
Removing coverage/
Removing pkg/
Removing spec/defines/
Removing spec/fixtures/manifests/
Removing spec/fixtures/modules/
Removing spec/functions/
Removing spec/hosts/
Removing update_report.txt
Removing vendor/

PDK Tests

The first test is real simple, it’s my unit tests via pdk test unit:

[rnelson0@build03 domain_join:pdk±]$ pdk test unit
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.1
[✔] Preparing to run the unit tests.
[✔] Running unit tests.
Evaluated 68 tests in 2.321391522 seconds: 0 failures, 0 pending.
[✔] Cleaning up after running unit tests.

I also want to validate linting and syntax and whatnot with pdk validate:

[rnelson0@build03 domain_join:pdk]$ pdk validate
pdk (INFO): Running all available validators...
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.1
[✔] Checking metadata syntax (metadata.json tasks/*.json).
[✔] Checking module metadata style (metadata.json).
[✔] Checking Puppet manifest syntax (**/**.pp).
[✔] Checking Puppet manifest style (**/*.pp).
[✖] Checking Ruby code style (**/**.rb).
info: task-metadata-lint: ./: Target does not contain any files to validate (tasks/*.json).
convention: rubocop: spec/spec_helper_acceptance.rb:17:27: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.
convention: rubocop: spec/spec_helper_acceptance.rb:17:49: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.
<more rubocop results>

I have a ton of rubocop results, which I will address below. Everything else works fine, as expected.

CI Tests

The second is a little trickier. Currently, whatever CI system you use will use ruby/bundler to perform the same checks. That is planned to change (PDK-709 tracks the Travis CI setup) When I use Travis CI, it uses tests from .travis.yml. Here are the relevant portions:

bundler_args: --without system_tests
matrix:
  fast_finish: true
  include:
    -
      env: CHECK="syntax lint metadata_lint check:symlinks check:git_ignore check:dot_underscore check:test_file rubocop"
    -
      env: CHECK=parallel_spec
    -
      env: PUPPET_GEM_VERSION="~> 4.0" CHECK=parallel_spec
      rvm: 2.1.9

There are two different checks that will run. The first is all the syntax and linting, the equivalent of pdk validate. The second and third are the unit tests, run against the latest Puppet 4 and Puppet 5 independently, and equivalent to pdk test unit. Here’s what happens when I run the unit tests first:

[rnelson0@build03 domain_join:pdk]$ pdk bundle exec rake parallel_spec
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.1
Cloning into 'spec/fixtures/modules/stdlib'...
2 processes for 2 specs, ~ 1 specs per process
No examples found.

Finished in 0.00032 seconds (files took 0.07684 seconds to load)
0 examples, 0 failures


domain_join
  on redhat-6-x86_64
    with defaults for all parameters
      should not contain Package[samba-common-tools]
      should contain Package[oddjob-mkhomedir]
      should contain Package[krb5-workstation]
      should contain Package[krb5-libs]
      should contain Package[samba-common]
      should contain Package[sssd-ad]
      should contain Package[sssd-common]
      should contain Package[sssd-tools]
      should contain Package[ldb-tools]
      should contain Class[domain_join]
      should contain File[/etc/resolv.conf]
      should contain File[/etc/krb5.conf]
      should contain File[/etc/samba/smb.conf]
      should contain File[/etc/sssd/sssd.conf]
      should contain File[/usr/local/bin/domain-join]
      should contain Exec[join the domain]
    with manage_services false
      should not contain Package[sssd]
      should not contain File[/etc/sssd/sssd.conf]
      should contain File[/etc/resolv.conf]
      should contain File[/usr/local/bin/domain-join]
    with manage_services and manage_resolver false
      should not contain Package[sssd]
      should not contain File[/etc/sssd/sssd.conf]
      should not contain File[/etc/resolv.conf]
      should contain File[/usr/local/bin/domain-join]
    start script syntax
      should contain File[/usr/local/bin/domain-join] with content =~ /sssd status/
    with container
      should contain File[/usr/local/bin/domain-join] with content =~ /net ads join/
      should contain File[/usr/local/bin/domain-join] with content =~ /container_ou='container'/
    with account and password
      should contain File[/usr/local/bin/domain-join] with content =~ /register_account='service_account'/
      should contain File[/usr/local/bin/domain-join] with content =~ /register_password='open_sesame'/
    with join_domain disabled
      should not contain Exec[join the domain]
    with manage_dns disabled
      should not contain File[/usr/local/bin/domain-join] with content =~ /net ads dns register/
      should not contain File[/usr/local/bin/domain-join] with content =~ /update add /
    with manage_dns and ptr enabled
      should contain File[/usr/local/bin/domain-join] with content =~ /net ads dns register/
      should contain File[/usr/local/bin/domain-join] with content =~ /update add .+ addr show fake_interface/
  on redhat-7-x86_64
    with defaults for all parameters
      should contain Package[samba-common-tools]
      should contain Package[oddjob-mkhomedir]
      should contain Package[krb5-workstation]
      should contain Package[krb5-libs]
      should contain Package[samba-common]
      should contain Package[sssd-ad]
      should contain Package[sssd-common]
      should contain Package[sssd-tools]
      should contain Package[ldb-tools]
      should contain Class[domain_join]
      should contain File[/etc/resolv.conf]
      should contain File[/etc/krb5.conf]
      should contain File[/etc/samba/smb.conf]
      should contain File[/etc/sssd/sssd.conf]
      should contain File[/usr/local/bin/domain-join]
      should contain Exec[join the domain]
    with manage_services false
      should not contain Package[sssd]
      should not contain File[/etc/sssd/sssd.conf]
      should contain File[/etc/resolv.conf]
      should contain File[/usr/local/bin/domain-join]
    with manage_services and manage_resolver false
      should not contain Package[sssd]
      should not contain File[/etc/sssd/sssd.conf]
      should not contain File[/etc/resolv.conf]
      should contain File[/usr/local/bin/domain-join]
    start script syntax
      should contain File[/usr/local/bin/domain-join] with content =~ /status sssd.service/
    with container
      should contain File[/usr/local/bin/domain-join] with content =~ /net ads join/
      should contain File[/usr/local/bin/domain-join] with content =~ /container_ou='container'/
    with account and password
      should contain File[/usr/local/bin/domain-join] with content =~ /register_account='service_account'/
      should contain File[/usr/local/bin/domain-join] with content =~ /register_password='open_sesame'/
    with join_domain disabled
      should not contain Exec[join the domain]
    with manage_dns disabled
      should not contain File[/usr/local/bin/domain-join] with content =~ /net ads dns register/
      should not contain File[/usr/local/bin/domain-join] with content =~ /update add /
    with manage_dns and ptr enabled
      should contain File[/usr/local/bin/domain-join] with content =~ /net ads dns register/
      should contain File[/usr/local/bin/domain-join] with content =~ /update add .+ addr show fake_interface/

1 deprecation warning total

Finished in 2.34 seconds (files took 2.17 seconds to load)
68 examples, 0 failures


68 examples, 0 failures

Took 5 seconds
I, [2018-05-29T21:51:29.950455 #16602]  INFO -- : Creating symlink from spec/fixtures/modules/domain_join to /home/rnelson0/modules/domain_join
/opt/puppetlabs/pdk/share/cache/ruby/2.4.0/gems/rspec-core-3.7.1/lib/rspec/core.rb:179:in `block in const_missing': uninitialized constant RSpec::Puppet (NameError)
        from /opt/puppetlabs/pdk/share/cache/ruby/2.4.0/gems/rspec-core-3.7.1/lib/rspec/core.rb:179:in `fetch'
        from /opt/puppetlabs/pdk/share/cache/ruby/2.4.0/gems/rspec-core-3.7.1/lib/rspec/core.rb:179:in `const_missing'
        from /home/rnelson0/modules/domain_join/spec/classes/coverage_spec.rb:1:in `block in '

Deprecation Warnings:

puppetlabs_spec_helper: defaults `mock_with` to `:mocha`. See https://github.com/puppetlabs/puppetlabs_spec_helper#mock_with to choose a sensible value for you


If you need more of the backtrace for any of these deprecations to
identify where to make the necessary changes, you can configure
`config.raise_errors_for_deprecations!`, and it will turn the
deprecation warnings into errors, giving you the full backtrace.
Tests Failed

The text in bold indicates an error in spec/classes/coverage_spec.rb. The simple solution for me is to git rm it, rather than add in the right coverage gem again. It’s not particularly important to me, but if it is to you, you need to add it back to Gemfile and spec/spec_helper.rb. The important thing is that a second run does not have the error and completes successfully.

The second test is a series of rake targets and causes me some grief out of the gate:

[rnelson0@build03 domain_join:pdk]$ pdk bundle exec rake syntax lint metadata_lint check:symlinks check:git_ignore check:dot_underscore check:test_file rubocop
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.1
init.pp
---> syntax:manifests
---> syntax:templates
---> syntax:hiera:yaml
rake aborted!
.pp files present in tests folder; Move them to an examples folder following the new convention
/opt/puppetlabs/pdk/share/cache/ruby/2.4.0/gems/puppetlabs_spec_helper-2.7.0/lib/puppetlabs_spec_helper/rake_tasks.rb:231:in `block (2 levels) in '
/opt/puppetlabs/pdk/share/cache/ruby/2.4.0/gems/rake-12.3.1/exe/rake:27:in `'
/opt/puppetlabs/pdk/private/ruby/2.4.4/bin/bundle:23:in `load'
/opt/puppetlabs/pdk/private/ruby/2.4.4/bin/bundle:23:in `'
Tasks: TOP => check:test_file
(See full trace by running task with --trace)

The fix is easy – move the existing files to a new location as the convention has changed, or remove them entirely if they are not valuable – and then I can proceed without further fault:

[rnelson0@build03 domain_join:pdk]$ mkdir examples
[rnelson0@build03 domain_join:pdk]$ git mv -v tests/*pp examples/
‘tests/init.pp’ -> ‘examples/init.pp’
[rnelson0@build03 domain_join:pdk±]$ pdk bundle exec rake syntax lint metadata_lint check:symlinks check:git_ignore check:dot_underscore check:test_file rubocop
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.1
Running RuboCop...
Inspecting 0 files


0 files inspected, no offenses detected
---> syntax:manifests
---> syntax:templates
---> syntax:hiera:yaml

As I mentioned earlier, I’d like to disable RuboCop. I don’t see how right now. If I specify selected_profile: off in .sync.yml for rubocop, pdk update errors out applying the template (PDK-998). However it seems to pass just fine in that check, though the individual check fails badly (PDK-997). I’m content to let it go so long as it’s passing test and I don’t have to rewrite anything, but I will find SOME way to get rid of it if it starts causing me problems!

If you use gitlab-ci, appveyor, or some other system for testing, you will want to ensure those tests pass as well. Once done, commit everything to git again.

I am now ready to submit a pull request, and if you are following along, you may be, too. You can review and compare my pull request and the tests if you would like. You will of course notice that I merged it in spite of rubocop failures!

Summary

We have looked at what the PDK is, why we want to use it, how to install it, and how to convert a module to use it. Each module can be customized and we explored the .sync.yml file that controls customization. Once we finalized our conversion, we ran the same tests we had prior to the PDK to make sure they still work and verified the Travis CI tests, too. The next step is to find a replacement for modulesync, which allows use to push the same general configuration to multiple modules. Lucky for us, Puppet just released a potential replacement, pdksync, that I will evaluate soon.

Documentation is hard. Anyone who has been in IT long enough will have tales of chasing their tails because of incorrect or outdated docs, or even missing docs. Documentation really benefits from automation and ease of creation. For Puppet modules, there exists a tool called puppet-strings that can help with this. There are probably other tools for this, but puppet-strings is developed by Puppet and will likely be integrated into the Puppet Development Kit, so I have chosen it as my solution.

Around this time last year, November of 2016, Will Hopper wrote a blog post about how to use puppet-strings. There is also some mention of puppet-strings in the Style Guide. At the time of that blog post, puppet-strings was mostly documented in that blog post and I didn’t jump on the project, but it turns out it’s really easy to leverage. Let’s give a shot.

Converting a Module to use puppet-strings

We should be able to convert any module to use puppet-strings, whether it’s small or large, simple or complex. Find a module you’d like to convert and you can follow along with it. I am going to convert my existing module rnelson0/certs, found on GitHub. First, let’s add the new gem to our module by adding two lines to the Gemfile:

gem 'puppet-strings'
gem 'rgen'

I’ve submitted PR 149 to puppet-strings as I believe rgen should be a runtime dependency, at which point you can remove that gem from the file.

Run bundle install or bundle update. You can now run bundle exec puppet strings generate ./manifests/*.pp . It won’t do much now, since we haven’t added strings-compatible metadata to our module, but it does generate the files:

[rnelson0@build03 certs:stringsdocs±]$ bundle exec puppet strings generate ./manifests/*.pp
[warn]: Missing @param tag for parameter 'source_path' near manifests/vhost.pp:59.
[warn]: Missing @param tag for parameter 'target_path' near manifests/vhost.pp:59.
[warn]: Missing @param tag for parameter 'service' near manifests/vhost.pp:59.
Files:                    2
Modules:                  0 (    0 undocumented)
Classes:                  0 (    0 undocumented)
Constants:                0 (    0 undocumented)
Attributes:               0 (    0 undocumented)
Methods:                  0 (    0 undocumented)
Puppet Classes:           1 (    0 undocumented)
Puppet Defined Types:     1 (    0 undocumented)
Puppet Types:             0 (    0 undocumented)
Puppet Providers:         0 (    0 undocumented)
Puppet Functions:         0 (    0 undocumented)
 100.00% documented
[rnelson0@build03 certs:stringsdocs±]$ ls html
ls: cannot access html: No such file or directory
[rnelson0@build03 certs:stringsdocs±]$ ls
CONTRIBUTING.md  doc  Gemfile  Gemfile.lock  manifests  metadata.json  Rakefile  README.md  spec  tests  vendor
[rnelson0@build03 certs:stringsdocs±]$ tree doc/
doc/
├── css
│   ├── common.css
│   ├── full_list.css
│   └── style.css
├── file.README.html
├── frames.html
├── _index.html
├── index.html
├── js
│   ├── app.js
│   ├── full_list.js
│   └── jquery.js
├── puppet_classes
│   └── certs.html
├── puppet_class_list.html
├── puppet_defined_type_list.html
├── puppet_defined_types
│   └── certs_3A_3Avhost.html
└── top-level-namespace.html

4 directories, 15 files

We can view the output in a browser by pulling up doc/index.html and browsing around it. If this is on a remote machine, it needs to be served up somehow. You can also copy it to your local machine and view it in a web browser (reminder that you can download a .ZIP of a branch from GitHub). I will leave this step out in the future for brevity, but don’t forget to do it, especially if you make changes, refresh, and nothing looks different!

We can add a rake task to make this simpler. In your Rakefile, add require 'puppet-strings/tasks'. If you add the gem to your Gemfile in a group that Travis doesn’t use, you should be sure to guard against failure with something like this:

# These gems aren't always present, for instance
# on Travis with --without development
begin
  require 'puppet_blacksmith/rake_tasks'
  require 'puppet-strings/tasks'
rescue LoadError
end

There are now two new rake tasks. You can generate docs with the much shorter bundle exec rake strings:generate:

[rnelson0@build03 certs:stringsdocs±]$ be rake -T | grep strings
Could not find semantic_puppet gem, falling back to internal functionality. Version checks may be less robust.
rake strings:generate[patterns,debug,backtrace,markup,json,yard_args]  # Generate Puppet documentation with YARD
rake strings:gh_pages:update                                           # Update docs on the gh-pages branch and push to GitHub
[rnelson0@build03 certs:stringsdocs±]$ be rake strings:generate
Could not find semantic_puppet gem, falling back to internal functionality. Version checks may be less robust.
[warn]: Missing documentation for Puppet defined type 'certs::vhost' at manifests/vhost.pp:35.
[warn]: The @param tag for parameter 'title' has no matching parameter at manifests/vhost.pp:35.
Files:                    2
Modules:                  0 (    0 undocumented)
Classes:                  0 (    0 undocumented)
Constants:                0 (    0 undocumented)
Attributes:               0 (    0 undocumented)
Methods:                  0 (    0 undocumented)
Puppet Classes:           1 (    0 undocumented)
Puppet Defined Types:     1 (    0 undocumented)
Puppet Types:             0 (    0 undocumented)
Puppet Providers:         0 (    0 undocumented)
Puppet Functions:         0 (    0 undocumented)
 100.00% documented

Next, we need to make some changes to our modules to document them. We can document manifests, types, providers, and functions, but I don’t have any of my own modules with types/providers/functions and the process is pretty similar, so I will focus on just a manifest today. Here is the header for my certs::vhost defined type before I add puppet-strings metadata:

# == Define: certs::vhost
#
# SSL Certificate File Management
#
# Intended to be used in conjunction with puppetlabs/apache's apache::vhost
# definitions, to provide the ssl_cert and ssl_key files.
#
# === Parameters
#
# [name]
# The title of the resource matches the certificate's name
# e.g. 'www.example.com' matches the certificate for the hostname
# 'www.example.com'
#
# [source_path]
# The location of the certificate files. Typically references a module's files.
# e.g. 'puppet:///site_certs' wills earch $modulepath/site_certs/files on the
# master for the specified files.
#
# [target_path]
# Location where the certificate files will be stored on the managed node.
# Optional value, defaults to '/etc/ssl/certs'
#
# [service]
# Name of the web server service to notify when certificates are updated.
# Optional value, defaults to 'httpd'
#
# === Examples
#
#  Without Hiera:
#
#    $cname = www.example.com
#    certs::vhost{ $cname:
#      source_path => 'puppet:///site_certificates',
#    }
#
#  With Hiera:
#
#    server.yaml
#    ---
#    certsvhost:
#      'www.example.com':
#        source_path: 'puppet:///modules/site_certificates/'
#
#    manifest.pp
#    ---
#    certsvhost = hiera_hash('certsvhost')
#    create_resources(certs::vhost, certsvhost)
#    Certs::Vhost<| |> -> Apache::Vhost<| |>
#
# === Authors
#
# Rob Nelson <rnelson0@gmail.com>
#
# === Copyright
#
# Copyright 2014 Rob Nelson
#

And here it is afterward:

# == Define: certs::vhost
#
# SSL Certificate File Management
#
# Intended to be used in conjunction with puppetlabs/apache's apache::vhost
# definitions, to provide the ssl_cert and ssl_key files.
#
# === Parameters
#
# @param name The title of the resource matches the certificate's name # e.g. 'www.example.com' matches the certificate for the hostname # 'www.example.com'
# @param source_path The location of the certificate files. Typically references a module's files. e.g. 'puppet:///site_certs' wills earch $modulepath/site_certs/files on the master for the specified files.
# @param target_path Location where the certificate files will be stored on the managed node. Optional value, defaults to '/etc/ssl/certs'
# @param service Name of the web server service to notify when certificates are updated. Optional value, defaults to 'httpd'
#
# @example
#     Without Hiera:
#    
#     $cname = www.example.com
#     certs::vhost{ $cname:
#       source_path => 'puppet:///site_certificates',
#     }
#    
#     With Hiera:
#    
#     server.yaml
#     ---
#     certsvhost:
#       'www.example.com':
#         source_path: 'puppet:///modules/site_certificates/'
#    
#     manifest.pp
#     ---
#     certsvhost = hiera_hash('certsvhost')
#     create_resources(certs::vhost, certsvhost)
#     Certs::Vhost<| |> -> Apache::Vhost<| |>
#
# === Authors
#
# Rob Nelson <rnelson0@gmail.com>
#
# === Copyright
#
# Copyright 2014 Rob Nelson
#

We can quickly regenerate the html docs and the defined type shows up. Be sure to click the `Defined Types` link in the top left, the left-hand menu does not mix classes and types.

You can see that there’s still some other work to do. The non-strings-ified portions of the comments are left as is, rather than parsed as markdown, so that needs to change. We don’t need most of that leftover crud. The class/defined type name is already known to strings. The Authors section should come from metadata.json (though if there are multiple, I am not sure if that file accepts an array). Copyright isn’t handled by metadata.json, and may not be strictly needed depending on your jurisdiction, but if you do need to keep it, just remove the === Copyright header and leave the text (I have chosen to omit it because US copyright law automatically grants me copyright for 70 years and I’m not that worried about it anyway; I would do something different for work).

I changed some other things:

• Each  @param can take multi-line comments, as long as each trailing line maintains one space of extra indentation.
• The title of defined types should be documented using @param title (docs), though it will generate a warning like [warn]: The @param tag for parameter 'name' has no matching parameter at manifests/vhost.pp:33
• The order of metadata should go @summary > freeform text > @example > @param

Here’s the updated header and the resulting html doc:

# @summary Used in conjunction with puppetlabs/apache's apache::vhost definitions, to provide the related ssl_cert and ssl_key files for a given vhost.
#
# @example
#    Without Hiera:
#
#      $cname = www.example.com
#      certs::vhost{ $cname:
#        source_path => 'puppet:///site_certificates',
#      }
#
#    With Hiera:
#
#      server.yaml
#      ---
#      certsvhost:
#        'www.example.com':
#          source_path: 'puppet:///modules/site_certificates/'
#
#      manifest.pp
#      ---
#      certsvhost = hiera_hash('certsvhost')
#      create_resources(certs::vhost, certsvhost)
#      Certs::Vhost<| |> -> Apache::Vhost<| |>
#
# @param title
#  The title of the resource matches the certificate's name # e.g. 'www.example.com' matches the certificate for the hostname # 'www.example.com'
# @param source_path
#  Required. The location of the certificate files. Typically references a module's files. e.g. 'puppet:///site_certs' will search $modulepath/site_certs/files on the master for the specified files.
# @param target_path
#  Location where the certificate files will be stored on the managed node.
#  Default: '/etc/ssl/certs'
# @param service
#  Name of the web server service to notify when certificates are updated.
#  Default: 'http'

That’s about it! For small modules, this is probably a really simple, really quick change. For larger modules, this may take a while, but it’s tedious, not complicated.

Online Docs

There are two other things you may want to look at. First, the string docs can be a tad large (212K vs 24K for the actual manifests, for example) but more importantly, are NOT guaranteed to be in sync with the rest of your code. If you include doc/ in your git data and you change a parameter definition/use in a module and do not regenerate docs and commit them to the repo simultaneously, users may not understand and take action on the changes. If you go a long while without automatically updating them, you may confuse your users or even yourself.

You can simply add docs/ to your .gitignore file. Now, the docs are not stored in the Git repo – unless you add with `–force` or added them before updating .gitignore, at which point you will definitely want to correct that! This ensures no doc mismatch with published code and can help keep the size of the git repo a little more trim.

Second, GitHub and other providers often do not display HTML docs very well for your users, so even if you include doc/ in your repo, the contents are probably displayed as text files. Whoops! There are a few solutions for this.

1. Publish through your Git provider’s services, like gh-pages, to a per-project website. For example, GitHub provides gh-pages sites and allows you to configure the publishing source (bonus: the rake task strings:gh_pages:update will push to this easily).
2. Add a hook to your CI that generates the docs and sends them where necessary. Vox Pupuli is working on this but has not chosen an implementation yet.
3. There are a few sites that you can add your docs to, some of which automagically update for you. One of these is http://www.puppetmodule.info/. You can easily click the Add Project button in the top right to add your own project to it (voila!). Since this occurs automatically, you never have to do anything else. But, when you are making changes, your docs could get stale until the next automated run occurs.

Puppet Module, by Dominic Cleal, also offers a badge you can add to your readme. Click the About button for more info.

Style Guides

One last thing to mention. As of 12/7/2017, the Style Guide is being updated to add information about puppet-strings. Pay attention to that space! I assume that it will first start with a description of standards and then add some puppet-lint checks to help you enforce it programatically. As puppet-strings is relatively new, you can expect more changes in the immediate future as it solidifies. If you have strong opinions on documentation, please speak up in the Documents Jira project, in Slack/IRC/mailing lists, or contact me and I’ll help you get your comments to the right person.

Summary

Today we added puppet-strings to a module, replaced the existing documentation with puppet-strings-compatible documentation, and looked at some solutions for automating document updates. It’s a simple process to enable better documentation updates, something everyone needs.

Over the past year, there have been some pretty big improvements to Puppet. I am still running PE 2016.4.2 and the current version is 2017.3.2, so there’s lot of changes in there. Most of the changes are backwards-compatible, so an upgrade from last November’s version is not quite as bad as it sounds, and I definitely want to start using the new features and improvements. The big one for me is Hiera version 5 (new in Puppet 4.9 / PE 2016.4.5). It is backwards compatible, so you can start using it right now, but it does require some changes to take advantage of the new features. I have to upgrade the server, upgrade the agents, and then start implementing the new features! Why do I care about Hiera 5 in particular?

Hiera 3 was great, but you could only use one hiera setup on a server, regardless of how many environments were deployed on that server. This could cause problems when you wanted to change the hiera config and test it. You could not test it in a feature branch, it HAD to be promoted to affect the entire server. If you had multiple masters, you could change the config on just one, but that was about as flexible as hiera 3 would let you be. If it worked, awesome. If it broke, you could break a whole lot that needed undone before you could try again.

Hiera 5 introduces independent hierarchy configurations per environment and even per module! If you want to try out a new backend like hiera-eyaml, you can now create a new feature branch eyaml-test, update the configuration in that branch, push it, and ONLY nodes that use that environment will receive the new configuration. This is a huge help in testing changes to hiera without blowing up all your nodes.

The per module hierarchy also means that module authors can include defaults that use hiera, rather than the params.pp pattern. This makes it easier for module users to override settings. There are also improvements in the interface for those who want to create their own backends. And, best of all, hiera 5 means the name hiera is here to stay – no more confusion between “legacy” hiera and modern lookup, it’s all called hiera 5 now.

It does mean there are some deprecations to keep in mind, but they won’t actually go away until at least Puppet 6. You can use hiera 5 now and take some time to replace the deprecated bits. This does mean we can use our existing hiera 3 setup and worry about migrating it to hiera 5 later, too, which we will take advantage of.

I always prefer to upgrade to the latest version. If for some reason you’re upgrading to Puppet 4.9, be aware that Puppet 4.10 fixed PUP-7554, which caused failures when a hiera 3 format hiera.yaml was found at the base of a controlrepo or module. I kept a hiera.yaml in the root of my controlrepo for bootstrapping purposes for a long while, and if you do, you could hit that bug. Best to just move to the latest if you can.

I think most people will like Hiera 5, but there are a ton of other features (listed at the end) and even if nothing appeals specifically, it is good to stay up to date so you don’t get stuck with a really nasty upgrade process when you find a feature you really need. Please, don’t let yourself get a full year behind on updates like I did. Sometimes it’s really difficult to get out of that situation!

Puppet Enterprise Server Upgrade

I use Puppet Enterprise at both work and home these days, so I will go through the PE upgrade experience. The Puppet OpenSource install and upgrade instructions are on the same page of the documentation, so it seems pretty easy but your mileage may vary, of course.

First, take a look at your installation and make sure it’s in a known state – preferably a known good state all the way around, but at least a known one. If you have outstanding issues on the master, you need to resolve them. If some agents are failing to check in, you may want to take the time to fix them, or you could just keep track of the failures. After the upgrade, you don’t want to see an increase in failures. Once everything looks good, take a snapshot of your master(s) and a full application/OS backup if possible. If you have a distributed setup, perform this on all nodes as close to the same time as possible.

Second, download the latest version of PE (KB#0001) onto your master. Expand the tarball, cd into the directory, and run sudo ./puppet-enterprise-installer. You can provide a .pe.conf file with the -c option, or answer a few interactive questions to get started:

=============================================================
 Puppet Enterprise Installer
=============================================================
2017-11-08 20:38:07,432 Running command: cp /opt/puppetlabs/server/pe_build /opt/puppetlabs/server/pe_build.bak
2017-11-08 20:38:07,480 Running command: cp /opt/puppetlabs/puppet/VERSION /opt/puppetlabs/server/puppet-agent-version.bak

## We've detected an existing Puppet Enterprise 2016.5.2 install.

 Would you like to proceed with text-mode upgrade/repair? [Yn]y

## We've found a pe.conf file at /etc/puppetlabs/enterprise/conf.d/pe.conf.

 Proceed with upgrade/repair of 2016.5.2 using the pe.conf at /etc/puppetlabs/enterprise/conf.d/pe.conf? [Yn]y

The install takes a bit of time (30 minutes on my lab install). Once the upgrade is done, you’ll be directed to run puppet agent -t (with sudo of course). If you have additional compile masters or ActiveMQ hubs and spokes, also run the commands in steps 4 and 5 as well.

You should now be able to log into the Console and see the status of your environment. You will hopefully find Intentional Changes on most of your nodes and zero or no failures (if both are encountered in a run, Intentional Changes “wins” on the Console; let every node run at least twice to see if it moves back to Green or Red before continuing).

If you do encounter failures, you will have to analyze each issue to see if it’s related to the upgrade and something you can fix, or if it’s time to roll back. If you do roll back, make sure you roll back ALL the PE components, including the PuppetDB, so you don’t leave cruft somewhere. I experienced one issue in the lab described in PUP-7878, resolved by a reboot of the master after the upgrade.

If everything is good, then it is time to proceed to upgrading the Agents.

Agent Upgrades

The Puppet docs provide instructions for upgrading Agents in a variety of methods. I prefer to use the module puppetlabs/puppet_agent, as I’ve discussed before (OpenSource, PE Linux and PE Windows clients). My experience is using my profile module and hiera data in the controlrepo, Puppet’s instructions use the Console Classifier. It really does not matter how you do this, but I did find an issue with the Puppet docs (DOCUMENT-763) – after classifying, you must set the parameter puppet_agent::package_version or no upgrade occurs for agents already running Puppet 4 or 5. Set it to 5.3.3(obtained by running puppet --version on the master, which received the latest agent during the upgrade). Here’s how to do that in hiera:

puppet_agent::package_version: '5.3.3'

The next two agent runs will show changes. I ran my tests directly using ssh on a Linux host and it looked like this:

# First run upgrades Puppet 4 to 5
[rnelson0@build03 controlrepo:pe201732]$ sudo puppet agent -t --environment pe201732
Info: Using configured environment 'pe201732'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for build03.nelson.va
Info: Applying configuration version '1510241461'
Notice: /Stage[main]/Puppet_agent::Osfamily::Redhat/Yumrepo[pc_repo]/baseurl: baseurl changed 'https://yum.puppetlabs.com/el/$releasever/PC1/x86_64' to 'https://puppet.nelson.va:8140/packages/2017.3.2/el-7-x86_64'
Notice: /Stage[main]/Puppet_agent::Osfamily::Redhat/Yumrepo[pc_repo]/sslcacert: defined 'sslcacert' as '/etc/puppetlabs/puppet/ssl/certs/ca.pem'
Notice: /Stage[main]/Puppet_agent::Osfamily::Redhat/Yumrepo[pc_repo]/sslclientcert: defined 'sslclientcert' as '/etc/puppetlabs/puppet/ssl/certs/build03.nelson.va.pem'
Notice: /Stage[main]/Puppet_agent::Osfamily::Redhat/Yumrepo[pc_repo]/sslclientkey: defined 'sslclientkey' as '/etc/puppetlabs/puppet/ssl/private_keys/build03.nelson.va.pem'
Notice: /Stage[main]/Puppet_agent::Install/Package[puppet-agent]/ensure: ensure changed '1.9.2-1.el7' to '5.3.3-1.el7'
Notice: /Stage[main]/Puppet_enterprise::Mcollective::Server::Logs/File[/var/log/puppetlabs/mcollective]/mode: mode changed '0750' to '0755'
Notice: Applied catalog in 71.86 seconds

# Second run updates some PE components
[rnelson0@build03 controlrepo:pe201732]$ sudo puppet agent -t --environment pe201732
Info: Using configured environment 'pe201732'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for build03.nelson.va
Info: Applying configuration version '1510241554'
Notice: /Stage[main]/Puppet_enterprise::Pxp_agent/File[/etc/puppetlabs/pxp-agent/pxp-agent.conf]/content:
--- /etc/puppetlabs/pxp-agent/pxp-agent.conf 2017-11-08 21:16:56.713834368 +0000
+++ /tmp/puppet-file20171109-20790-1l6y5bg 2017-11-09 15:32:45.917909748 +0000
@@ -1 +1 @@
-{"broker-ws-uris":["wss://puppet.nelson.va:8142/pcp2/"],"pcp-version":"2","ssl-key":"/etc/puppetlabs/puppet/ssl/private_keys/build03.nelson.va.pem","ssl-cert":"/etc/puppetlabs/puppet/ssl/certs/build03.nelson.va.pem","ssl-ca-cert":"/etc/puppetlabs/puppet/ssl/certs/ca.pem","loglevel":"info"}
\ No newline at end of file
+{"broker-ws-uris":["wss://puppet.nelson.va:8142/pcp2/"],"pcp-version":"2","master-uris":["https://puppet.nelson.va:8140"],"ssl-key":"/etc/puppetlabs/puppet/ssl/private_keys/build03.nelson.va.pem","ssl-cert":"/etc/puppetlabs/puppet/ssl/certs/build03.nelson.va.pem","ssl-ca-cert":"/etc/puppetlabs/puppet/ssl/certs/ca.pem","loglevel":"info"}
\ No newline at end of file

Info: Computing checksum on file /etc/puppetlabs/pxp-agent/pxp-agent.conf
Info: /Stage[main]/Puppet_enterprise::Pxp_agent/File[/etc/puppetlabs/pxp-agent/pxp-agent.conf]: Filebucketed /etc/puppetlabs/pxp-agent/pxp-agent.conf to puppet with sum cad3d2db7a7a912a1734b7e8afa23037
Notice: /Stage[main]/Puppet_enterprise::Pxp_agent/File[/etc/puppetlabs/pxp-agent/pxp-agent.conf]/content: content changed '{md5}cad3d2db7a7a912a1734b7e8afa23037' to '{md5}a19b53e1586a748ba488ee4dcd7afc3c'
Info: /Stage[main]/Puppet_enterprise::Pxp_agent/File[/etc/puppetlabs/pxp-agent/pxp-agent.conf]: Scheduling refresh of Service[pxp-agent]
Notice: /Stage[main]/Puppet_enterprise::Pxp_agent::Service/Service[pxp-agent]: Triggered 'refresh' from 1 event
Notice: /Stage[main]/Puppet_enterprise::Mcollective::Server/File[/etc/puppetlabs/mcollective/server.cfg]/content: [diff redacted]
Info: Computing checksum on file /etc/puppetlabs/mcollective/server.cfg
Info: /Stage[main]/Puppet_enterprise::Mcollective::Server/File[/etc/puppetlabs/mcollective/server.cfg]: Filebucketed /etc/puppetlabs/mcollective/server.cfg to puppet with sum 7a8d59f271273738a51b4cf05ee6b33a
Notice: /Stage[main]/Puppet_enterprise::Mcollective::Server/File[/etc/puppetlabs/mcollective/server.cfg]/content: changed [redacted] to [redacted]
Info: /Stage[main]/Puppet_enterprise::Mcollective::Server/File[/etc/puppetlabs/mcollective/server.cfg]: Scheduling refresh of Service[mcollective]
Notice: /Stage[main]/Puppet_enterprise::Mcollective::Service/Service[mcollective]: Triggered 'refresh' from 1 event
Notice: Applied catalog in 6.02 seconds

I assume the PE component updates are based on facts only present with Puppet 5, facts that would not be present during the first run while the agent is still Puppet 4. Subsequent runs are stable.

I do not have a Windows agent to test with in my lab, I assume it looks similar but cannot verify. Be sure to test at least one Windows agent before releasing this change across your entire Windows fleet.

New Features

I have skipped from 2016.4 to 2017.3, which means I have missed out on new features in four major versions: 2016.5, 2017.1, 2017.2, and 2017.3. Here are some of the big features from the release notes:

• Ad-hoc Tasks (2017.3, announced at PuppetConf 2017)
• Hiera Overrides in the Console (2017.3)
• New and Improved Docs (2017.3) – update any links to puppet docs in your documentation, modules, bookmarks, anywhere you have stored them
• Orchestrator in the Console (2017.2)
• Package Inventory in the Console (2017.2)
• Hiera 5 (2017.1)
• High Availability (2016.5)

I mentioned Hiera 5 already, which I’ll discuss further in another post. I also want to immediately enable the Package Inventory. As described, I can update the Classification of the PE Agent node group to include puppet_enterprise::profile::agent with package_inventory_enabled set to true and committing the change.

While it takes affect immediately, your agents need two runs to show up: the first changes the setting so package data is collected and the second actually collects the list. Once that happens with at least one node, you’ll start seeing data populate on the Inspect > Packages page.

I do not have need for High Availability myself, but that seems really cool. In the past, it’s been quite the pain in the behind to automate yourself. I have not used Orchestrator in anger before, and I do Hiera overrides in my control repo, almost ignoring the Console Classifier otherwise, so I probably will not be exploring them very well. However, I’m really excited about Tasks, that’s something I hope to explore during the winter break, perhaps by upgrading bash across all my systems!

Summary

Today we looked at why we want to upgrade to the latest Puppet and upgraded a Puppet Enterprise monolithic master and some linux agents. It’s not that hard! We also staked out features that we want to investigate and turned on the Package Inventory. There are a lot more changes than I listed, along with tons of fixed bugs and smaller improvements, so I recommend reviewing the release notes for each version to see what interests you.

I hope to be able to look into Hiera 5 and Tasks soon, look for new blog posts on those! Let me know if there’s anything else you’d like to see discussed in the comments or on twitter. Thanks!