I’ve been working recently on migrating to Puppet 4. All the modules I maintain have supported it for a little bit but my master and controlrepo were still on Puppet 3. I slowly hacked at this over the past month and a half when time presented itself and I learned a few things. This post is an assortment of lessons learned, more than a tutorial, but hopefully it will help others going through this effort themselves.

At a high level the process consists of:

• Make sure your code is Puppet 4 compatible via Continuous Integration.
• Find a module that manages Puppet 4 masters and agents. If your current module works with 4, this step is much easier.
• Build a new template or base image that runs Puppet 4.
• Update your controlrepo to work with a Puppet 4 master, preferably with the new puppetserver instead of apache+passenger, again using CI for testing.
• Deploy the new master and then start adding agents.

The biggest lesson I can give you is to perform these changes as 5 separate steps! I combined the last three into a single step and I paid for it. I know better and my shortcut didn’t turn out so well. Especially, do not take your Puppet 3 master down until your Puppet 4 master build tests out okay! Alright, let’s go through some of the steps.

Puppet 4 Compatibility

Puppet Labs has a good guide on Updating 3.x Manifests for Puppet 4.x. Add some Travis CI to your Controlrepo first (and make sure you get the current code passing before adding anything else!), then start updating your manifests. You’ll also need to make sure all the modules you use support Puppet 4. Hopefully this is as easy as updating to the latest version, but you may need to submit upstream PRs or use your own fork if the author is running a bit behind. This step may take a while, but is fairly easy. CI feedback is your guide here, don’t move to the next step until everything’s green.

Find a Puppet 4 Module

I’ve been using stephenrjohnson/puppet for a while, but v4 compatibility is still just an issue for now. After looking at a lot of modules, I settled on jlambert121/puppet. It works fairly well with puppetlabs/puppetdb (PR pending) with the right settings. This may work for you, it may not; you need to do your own research to ensure the module you choose meets your needs.

If you end up using jlambert121/puppet, here are some settings I found helpful to apply via Hiera (the firewall parameter means you can remove any firewall rule you have for port 8140 from your manifests). Adjust as necessary:

# puppet_master.yaml
puppet::server: true
puppet::server_version: 'latest'
puppet::dns_alt_names:
  - 'puppet'
puppet::puppetdb_server: 'puppet.example.com'
puppet::puppetdb: true
puppet::manage_puppetdb: false
puppet::manage_hiera: false
puppet::firewall: true
puppet::runmode: service

I also needed to modify my puppet master profile and rspec tests.

# spec/classes/puppet_master_spec.rb
  # after let (:facts) and before the first context
  let (:pre_condition) {
    "package{'puppetdb': ensure => present, }"
  }

Check your role/profile metadata.json file. If you are listing dependencies here (I was, but have since removed them because why would I?) be sure to replace your old module name with the new one.

Though you’ve found the new module, don’t swap it out in production, use a feature branch. We still need some changes that will hit production. Make sure you run some CI against this feature branch so you can iron out issues in your module usage before proceeding.

Build a new Template

I documented one way to build a new template here. If you use kickstart, you can also now deploy this from a module very easily, too, with danzilio/kickstart and puppet/community_kickstarts. Add them to your Puppetfile and create a profile that looks like this:

class profile::kickstart {
  include ::apache

  #::kickstart::ks_file{'el6-dhcp.ks': }
  #::kickstart::ks_file{'el6-dhcp-with-tools.ks':
  #  vmwaretools_location => 'http://10.0.0.8/',
  #}

  $el7_packages = [
    '@core',
    'ntpdate',
    'ntp',
    'wget',
    'screen',
    'git',
    'perl',
    'openssh-clients',
    'open-vm-tools',
    'man',
    'mlocate',
    'bind-utils',
    'traceroute',
    'mailx',
  ]

  ::community_kickstarts::centos7{'/var/www/html/centos7.ks':}

  firewall { '100 HTTP/S inbound':
    dport  => [80, 443],
    proto  => tcp,
    action => accept,
  }
}

Create a kickstart server that receives this profile and you can then create a new template that includes Puppet 4.x from Puppet Collection 1. The default partition layout assumes 100GB and cpu/mem is up to you. It’s very minimal, but you can add to $el7_packages to get what you need, or create your own kickstart template if you need something completely different.

Update your Controlrepo

You’ll need to grab your new puppet module and the settings for the module, which was covered above. I also found some other issues I needed to resolve.

Hiera Issues

HI-494: Sometime between puppet3/hiera1 and puppet4/hiera3, the old trick of using %%{}{realvar} in hiera data to generate %{realvar} and prevent interpolation of realvar broke. You can replace that with %%{::}{realvar}.

DOCUMENT-491: I also saw that automatic parameter lookup didn’t like %{environment} in the datadir anymore and required %{::environment}.

To address this, make these changes in your bootstrap hiera.yaml and your puppet master’s hiera data (changes in bold):

# hiera.yaml
---
:backends:
  - yaml

:logger: console

:hierarchy:
  - "clientcert/%{clientcert}"
  - "puppet_role/%{puppet_role}"
  - global

:yaml:
  :datadir: /etc/puppetlabs/code/environments/%{::environment}/hiera

# puppet_master.yaml
hiera::hierarchy:
  - 'clientcert/%%{::}{clientcert}'
  - 'puppet_role/%%{::}{puppet_role}'
  - 'global'
hiera::datadir: '/etc/puppetlabs/code/environments/%%{::}{::environment}/hiera'

You can do this work on your puppet 4 feature branch.

mcollective and R10k

There are a few changes that needed made with respect to r10k. In your bootstrap file (r10k_installation.pp), bring the gem up to the latest version and change the basedir, as Puppet 4 has a different default path.

class { 'r10k':
  version => '2.1.1',
  sources => {
    'puppet' => {
      'remote'  => 'git@github.com:username/controlrepo.git',
      'basedir' => $::settings::environmentpath,
      'prefix'  => false,
    },
  },
  manage_modulepath => false
}

Next, we need to find some new certificates to use with the webhook! There’s a request to have zack/r10k generate certificates itself, but until then, we can use some from PuppetDB. We just need to point to the new location in our module parameters, and add a relationship in our manifest to ensure the webhook doesn’t start until after the certs are there.

# puppet_master.pp
class profile::puppet_master {
  # After the r10k includes
  Package['puppetdb'] -> Service[webhook]
  # Everything else
}

# puppet.yaml
r10k::webhook::config::public_key_path: '/etc/puppetlabs/puppetdb/ssl/public.pem'
r10k::webhook::config::private_key_path: '/etc/puppetlabs/puppetdb/ssl/private.pem'

You can do this work on your puppet 4 feature branch.

Version Pinning Refresh

Review your Puppetfile and bring all the modules you can up to the latest. You may have done this already with some older versions that were not Puppet 4 compatible, but you should do this with the rest as well. Only keep an older version if it’s actually necessary. While you don’t absolutely have to do this, it’s a convenient time to do this as well, before you get to the new master. Again, develop on a feature branch and use tests to ensure everything’s okay before merging it into production. There are many tools to help bring your Puppetfile up to date.

If you do this, perform the work in a separate feature branch from your puppet 4 feature branch and merge it prior to the next step.

Deploy the new Master

Now you’re ready to take the feature branch where you started to work on the new puppet module and rebase it so you get the updated Puppetfile. This branch should have your new puppet module, r10k, and hiera changes ready to go. Your CI should tell you if you need other tweaks.

I then came up with a simple Bootstrap.md that I added to my controlrepo. Use your own controlrepo URI and this should work as a good framework for most people. During tests, checkout your feature branch and use it as the environment arg to puppet agent -t. Once everything is good, you can merge it and use no environment argument if you need to rebuild the master.

# Run ssh-keygen and attach the new key to your GitHub account

systemctl stop firewalld
mkdir /root/bootstrap
puppet module install zack/r10k --modulepath=/root/bootstrap
git clone git@github.com:username/controlrepo.git
cd controlrepo
git checkout <branch> # optional
puppet apply r10k_installation.pp --modulepath=/root/bootstrap
rm -f /etc/hiera.yaml /etc/puppetlabs/code/hiera.yaml
cp hiera.yaml /etc
cp hiera.yaml /etc/puppetlabs/code
r10k deploy environment -pv
yum install -y puppetserver
systemctl start puppetserver
puppet agent -t --environment=<branch>

It should be that simple! I would perform a snapshot before running puppet at the end, so if you need to revert for some reason, you’re a little closer to the end.

Next Steps

After building the master, the next step is to upgrade all your nodes. That’s a little more complicated and depends greatly on your environment. If you use your existing nodes, you may have to clean up some SSL certificates to talk to the new master. If you are using an immutable infrastructure model, or just rebuilding the whole lab at once, build new nodes using the updated template and they should be able to talk to the master once you sign the certs on the master, or enable autosigning.

Summary

As I said at the top, this isn’t a tutorial by any means, just a collection of lessons learned. I hope it helps you in some way! I’ve also created a Puppet 4 Lessons Learned gist. If you have comments or see room for improvement, feel free to drop a PR or comment there, and of course here or on twitter as well. Thanks!