Happy New Year, everyone! I know that when I last blogged before the holiday, I was getting settled in with Jenkins, but I really needed to relax during the break so I had to hold off on that. I promise, I’ll get back to that soon, but in the meantime I encountered an issue that needed fixed and required a little more than the docs provided.

I ran into an issue recently with Puppet Enterprise agents that weren’t able to purge some resource types properly due to the use of Anchors. This is an issue with Puppet, not the module in question, and it was fixed in Puppet 4.4.0 but affected nodes were running older versions. Of course, rather than upgrade the agents by hand, I decided to automate it. Puppet has an Upgrading PE agents: *nix page that describes how to do this in a few ways. The latter options require manual effort, but the first option, via the module puppet_agent, can do this automatically for us. I diverged from the instructions a bit because I do my classifying with hiera rather than the PE Classifier and because there’s a missing step! (I’m trying to get that document updated, as well)

The guide says to install the module on the master and add some classification. To do this in a traditional roles and profiles setup using hiera as the classifier and with a monolithic controlrepo, we need to add the module to Puppetfile and .fixtures.yml, then add either a profile for puppet_agent or add it to a base profile. I chose the latter. Let’s add the module first:

# Puppetfile
mod 'puppetlabs-puppet_agent', '1.3.1'

# .fixtures.yml
fixtures:
  <snip>
  forge_modules:
    <snip>
    puppet_agent:
      repo: "puppetlabs/puppet_agent"
      ref: "1.3.1"

I’ve added the class to profile::base::linux (I haven’t had time to test it with Windows but assume it will work; look for a new post shortly to confirm that). We start with the spec tests first:

# spec/classes/profile/base__linux_spec.rb, added to the default context
  it { is_expected.to contain_class('puppet_agent') }

Then we update the profile class. I added a conditional flag in case we have nodes where we simply don’t want to manage this at all:

# dist/profile/manifests/base/linux.pp
class profile::base::linux (
  #...
  $manage_puppet   = true,  # Manage the puppet-agent including upgrades
) {
  if $manage_puppet {
    include puppet_agent
  }
  #...
}

If you run the tests now, you’ll get a lot of errors as puppet_agent provides a custom fact and expects it to exist, as well as a few other facts that you may not have mocked up. There are a number of ways to add these facts. I chose a global method to avoid forgetting to add the facts directly to a spec test where the class is indirectly called. To add the global facts, we need to update spec/spec_helper.rb and create spec/default_facts.yaml:

# spec/spec_helper.rb
RSpec.configure do |c|
  #...
  default_facts = {
   puppetversion: Puppet.version,
   facterversion: Facter.version
 }
 default_facts.merge!(YAML.load(File.read(File.expand_path('../default_facts.yml', __FILE__)))) if File.exist?(File.expand_path('../default_facts.yml', __FILE__))
 c.default_facts = default_facts
end

# spec/default_facts.yaml
#################
## puppet_agent #
#################
# custom fact
puppet_stringify_facts: false
# other variables
servername: 'puppet.example.com'
platform_tag: 'el-7-x86_64'

When running rspec now, the contents of spec/default_facts.yml are added to the default_facts hash, providing us with the facts that puppet_agent depends on. We can run spec tests and all our tests pass:

$ be rspec spec/classes/profile/base__linux_spec.rb

profile::base::linux
  on redhat-6-x86_64
    with defaults for all parameters
      #...
      should contain Class[puppet_agent]
  #...

Finished in 19.64 seconds (files took 4.67 seconds to load)
40 examples, 0 failures

We have one last thing to do! I mentioned a missing step in the documentation. By default, puppet_agent defaults to a version of present. The agent is already installed, so this doesn’t help us much with an upgrade unless we are coming from PE 3.8, where the puppet package was called puppet instead of puppet-agent. Since we’re already on 4.x, we need to increase that to a higher version. We have to specify a specific version as latest does not work. The version of Puppet Enterprise on my master includes puppet-agent 1.7.1, so I set that in my hiera’s least specific tier. There’s just one line you need to add:

puppet_agent::package_version: '1.7.1'

You can add this at a different tier, or have different values throughout; by default, the first match wins. Set this where you need it rather than mirroring my configuration. Also remember that we added a feature flag profile::base::linux::manage_puppet that you can set to false where you don’t want this class to be included at all.

We can commit our changes, push them upstream to a new environment, and test it on a node. Here’s the entire run’s output:

$ sudo puppet --version
4.3.2

$ sudo puppet agent -t --environment puppet_agent
Info: Using configured environment 'puppet_agent'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for example.puppet.com
Info: Applying configuration version '1483481416'
Notice: /Stage[main]/Puppet_agent::Osfamily::Redhat/File[/etc/pki/rpm-gpg/RPM-GPG-KEY-puppetlabs]/ensure: defined content as '{md5}7b4ed31e1028f921b5c965df0a42e508'
Notice: /Stage[main]/Puppet_agent::Osfamily::Redhat/File[/etc/pki/rpm-gpg/RPM-GPG-KEY-puppet]/ensure: defined content as '{md5}16e3e148bc861ee66906e475f8342f81'
Notice: /Stage[main]/Puppet_agent::Osfamily::Redhat/Exec[import-RPM-GPG-KEY-puppet]/returns: executed successfully
Notice: /Stage[main]/Puppet_agent::Osfamily::Redhat/Yumrepo[pc_repo]/ensure: created
Info: changing mode of /etc/yum.repos.d/pc_repo.repo from 600 to 644
Notice: /Stage[main]/Puppet_agent::Install/Package[puppet-agent]/ensure: ensure changed '1.3.6-1.el7' to '1.7.1-1.el7'
Warning: /Stage[main]/Ssh::Knownhosts/Sshkey[othernode.example.com_dsa]: Anchor[ssh::server::end],Anchor[ssh::client::end] still depend on me -- not purging
Warning: /Stage[main]/Ssh::Knownhosts/Sshkey[othernode.example.com_rsa]: Anchor[ssh::server::end],Anchor[ssh::client::end] still depend on me -- not purging
Notice: Applied catalog in 211.20 seconds

$ sudo puppet --version
4.7.0

It can take a while to run, over 3 minutes(!) so don’t despair if it seems like it’s hanging. You can see I’m still getting the warnings about anchors, but that’s okay because that agent run was done with puppet 4.3.2. After running it again, I can see the purged resources being cleaned up:

$ sudo puppet agent -t --environment puppet_agent
Info: Using configured environment 'puppet_agent'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for chi-build05.mss.local
Info: Applying configuration version '1483481701'
Notice: /Stage[main]/Puppet_agent::Osfamily::Redhat/Yumrepo[pc_repo]/enabled: enabled changed 'true' to 'True'
Notice: /Stage[main]/Docker::Repos/Yumrepo[docker]/gpgcheck: gpgcheck changed 'true' to 'True'
Notice: /Stage[main]/Ssh::Knownhosts/Sshkey[othernode.example.com_dsa]/ensure: removed
Info: Computing checksum on file /etc/ssh/ssh_known_hosts
Notice: /Stage[main]/Ssh::Knownhosts/Sshkey[othernode.example.com_rsa]/ensure: removed
Notice: /Stage[main]/Puppet_enterprise::Mcollective::Server/File[/etc/puppetlabs/mcollective/server.cfg]/content:
--- /etc/puppetlabs/mcollective/server.cfg      2016-11-17 02:18:50.332523423 +0000
+++ /tmp/puppet-file20170103-13188-1g2s6ai      2017-01-03 22:15:11.868383609 +0000
@@ -1,5 +1,5 @@

-# Centrally managed by Puppet version 4.3.2
+# Centrally managed by Puppet version 4.7.0
 # https://docs.puppetlabs.com/mcollective/configure/server.html

 # Connector settings (required):

Info: Computing checksum on file /etc/puppetlabs/mcollective/server.cfg
Info: /Stage[main]/Puppet_enterprise::Mcollective::Server/File[/etc/puppetlabs/mcollective/server.cfg]: Filebucketed /etc/puppetlabs/mcollective/server.cfg to puppet with sum f3dbe7e960ecd15ba6990196b4cc7d9d
Notice: /Stage[main]/Puppet_enterprise::Mcollective::Server/File[/etc/puppetlabs/mcollective/server.cfg]/content: content changed '{md5}f3dbe7e960ecd15ba6990196b4cc7d9d' to '{md5}188788f65505a9ae31bcd427cc9405c0'
Info: /Stage[main]/Puppet_enterprise::Mcollective::Server/File[/etc/puppetlabs/mcollective/server.cfg]: Scheduling refresh of Service[mcollective]
Notice: /Stage[main]/Puppet_enterprise::Mcollective::Service/Service[mcollective]: Triggered 'refresh' from 1 events
Notice: Applied catalog in 4.42 seconds

Mcollective does a little cleanup and for some reason true is converted to True in a few spots (I’m sure if I look up the release notes between 4.3.2 and 4.7.0, I’d find the cause if I were really interested) but future runs are stable, as expected.

We now have Puppet Enterprise agents that automatically receive the correct version on deployment and can be upgraded just through a single hiera update and waiting a while. When the master is upgraded in the future, you just need to bump the puppet_agent::package_version value to the new puppet-agent version. Enjoy!