Self-documenting Puppet modules with puppet-strings

Documentation is hard. Anyone who has been in IT long enough will have tales of chasing their tails because of incorrect or outdated docs, or even missing docs. Documentation really benefits from automation and ease of creation. For Puppet modules, there exists a tool called puppet-strings that can help with this. There are probably other tools for this, but puppet-strings is developed by Puppet and will likely be integrated into the Puppet Development Kit, so I have chosen it as my solution.

Around this time last year, November of 2016, Will Hopper wrote a blog post about how to use puppet-strings. There is also some mention of puppet-strings in the Style Guide. At the time of that blog post, puppet-strings was mostly documented in that blog post and I didn’t jump on the project, but it turns out it’s really easy to leverage. Let’s give a shot.

Converting a Module to use puppet-strings

We should be able to convert any module to use puppet-strings, whether it’s small or large, simple or complex. Find a module you’d like to convert and you can follow along with it. I am going to convert my existing module rnelson0/certs, found on GitHub. First, let’s add the new gem to our module by adding two lines to the Gemfile:

gem 'puppet-strings'
gem 'rgen'

I’ve submitted PR 149 to puppet-strings as I believe rgen should be a runtime dependency, at which point you can remove that gem from the file.

Run bundle install or bundle update. You can now run bundle exec puppet strings generate ./manifests/*.pp . It won’t do much now, since we haven’t added strings-compatible metadata to our module, but it does generate the files:

[rnelson0@build03 certs:stringsdocs±]$ bundle exec puppet strings generate ./manifests/*.pp
[warn]: Missing @param tag for parameter 'source_path' near manifests/vhost.pp:59.
[warn]: Missing @param tag for parameter 'target_path' near manifests/vhost.pp:59.
[warn]: Missing @param tag for parameter 'service' near manifests/vhost.pp:59.
Files:                    2
Modules:                  0 (    0 undocumented)
Classes:                  0 (    0 undocumented)
Constants:                0 (    0 undocumented)
Attributes:               0 (    0 undocumented)
Methods:                  0 (    0 undocumented)
Puppet Classes:           1 (    0 undocumented)
Puppet Defined Types:     1 (    0 undocumented)
Puppet Types:             0 (    0 undocumented)
Puppet Providers:         0 (    0 undocumented)
Puppet Functions:         0 (    0 undocumented)
 100.00% documented
[rnelson0@build03 certs:stringsdocs±]$ ls html
ls: cannot access html: No such file or directory
[rnelson0@build03 certs:stringsdocs±]$ ls
CONTRIBUTING.md  doc  Gemfile  Gemfile.lock  manifests  metadata.json  Rakefile  README.md  spec  tests  vendor
[rnelson0@build03 certs:stringsdocs±]$ tree doc/
doc/
├── css
│   ├── common.css
│   ├── full_list.css
│   └── style.css
├── file.README.html
├── frames.html
├── _index.html
├── index.html
├── js
│   ├── app.js
│   ├── full_list.js
│   └── jquery.js
├── puppet_classes
│   └── certs.html
├── puppet_class_list.html
├── puppet_defined_type_list.html
├── puppet_defined_types
│   └── certs_3A_3Avhost.html
└── top-level-namespace.html

4 directories, 15 files

We can view the output in a browser by pulling up doc/index.html and browsing around it. If this is on a remote machine, it needs to be served up somehow. You can also copy it to your local machine and view it in a web browser (reminder that you can download a .ZIP of a branch from GitHub). I will leave this step out in the future for brevity, but don’t forget to do it, especially if you make changes, refresh, and nothing looks different!

We can add a rake task to make this simpler. In your Rakefile, add require 'puppet-strings/tasks'. If you add the gem to your Gemfile in a group that Travis doesn’t use, you should be sure to guard against failure with something like this:

# These gems aren't always present, for instance
# on Travis with --without development
begin
  require 'puppet_blacksmith/rake_tasks'
  require 'puppet-strings/tasks'
rescue LoadError
end

There are now two new rake tasks. You can generate docs with the much shorter bundle exec rake strings:generate:

[rnelson0@build03 certs:stringsdocs±]$ be rake -T | grep strings
Could not find semantic_puppet gem, falling back to internal functionality. Version checks may be less robust.
rake strings:generate[patterns,debug,backtrace,markup,json,yard_args]  # Generate Puppet documentation with YARD
rake strings:gh_pages:update                                           # Update docs on the gh-pages branch and push to GitHub
[rnelson0@build03 certs:stringsdocs±]$ be rake strings:generate
Could not find semantic_puppet gem, falling back to internal functionality. Version checks may be less robust.
[warn]: Missing documentation for Puppet defined type 'certs::vhost' at manifests/vhost.pp:35.
[warn]: The @param tag for parameter 'title' has no matching parameter at manifests/vhost.pp:35.
Files:                    2
Modules:                  0 (    0 undocumented)
Classes:                  0 (    0 undocumented)
Constants:                0 (    0 undocumented)
Attributes:               0 (    0 undocumented)
Methods:                  0 (    0 undocumented)
Puppet Classes:           1 (    0 undocumented)
Puppet Defined Types:     1 (    0 undocumented)
Puppet Types:             0 (    0 undocumented)
Puppet Providers:         0 (    0 undocumented)
Puppet Functions:         0 (    0 undocumented)
 100.00% documented

Next, we need to make some changes to our modules to document them. We can document manifests, types, providers, and functions, but I don’t have any of my own modules with types/providers/functions and the process is pretty similar, so I will focus on just a manifest today. Here is the header for my certs::vhost defined type before I add puppet-strings metadata:

# == Define: certs::vhost
#
# SSL Certificate File Management
#
# Intended to be used in conjunction with puppetlabs/apache's apache::vhost
# definitions, to provide the ssl_cert and ssl_key files.
#
# === Parameters
#
# [name]
# The title of the resource matches the certificate's name
# e.g. 'www.example.com' matches the certificate for the hostname
# 'www.example.com'
#
# [source_path]
# The location of the certificate files. Typically references a module's files.
# e.g. 'puppet:///site_certs' wills earch $modulepath/site_certs/files on the
# master for the specified files.
#
# [target_path]
# Location where the certificate files will be stored on the managed node.
# Optional value, defaults to '/etc/ssl/certs'
#
# [service]
# Name of the web server service to notify when certificates are updated.
# Optional value, defaults to 'httpd'
#
# === Examples
#
#  Without Hiera:
#
#    $cname = www.example.com
#    certs::vhost{ $cname:
#      source_path => 'puppet:///site_certificates',
#    }
#
#  With Hiera:
#
#    server.yaml
#    ---
#    certsvhost:
#      'www.example.com':
#        source_path: 'puppet:///modules/site_certificates/'
#
#    manifest.pp
#    ---
#    certsvhost = hiera_hash('certsvhost')
#    create_resources(certs::vhost, certsvhost)
#    Certs::Vhost<| |> -> Apache::Vhost<| |>
#
# === Authors
#
# Rob Nelson <rnelson0@gmail.com>
#
# === Copyright
#
# Copyright 2014 Rob Nelson
#

And here it is afterward:

# == Define: certs::vhost
#
# SSL Certificate File Management
#
# Intended to be used in conjunction with puppetlabs/apache's apache::vhost
# definitions, to provide the ssl_cert and ssl_key files.
#
# === Parameters
#
# @param name The title of the resource matches the certificate's name # e.g. 'www.example.com' matches the certificate for the hostname # 'www.example.com'
# @param source_path The location of the certificate files. Typically references a module's files. e.g. 'puppet:///site_certs' wills earch $modulepath/site_certs/files on the master for the specified files.
# @param target_path Location where the certificate files will be stored on the managed node. Optional value, defaults to '/etc/ssl/certs'
# @param service Name of the web server service to notify when certificates are updated. Optional value, defaults to 'httpd'
#
# @example
#     Without Hiera:
#    
#     $cname = www.example.com
#     certs::vhost{ $cname:
#       source_path => 'puppet:///site_certificates',
#     }
#    
#     With Hiera:
#    
#     server.yaml
#     ---
#     certsvhost:
#       'www.example.com':
#         source_path: 'puppet:///modules/site_certificates/'
#    
#     manifest.pp
#     ---
#     certsvhost = hiera_hash('certsvhost')
#     create_resources(certs::vhost, certsvhost)
#     Certs::Vhost<| |> -> Apache::Vhost<| |>
#
# === Authors
#
# Rob Nelson <rnelson0@gmail.com>
#
# === Copyright
#
# Copyright 2014 Rob Nelson
#

We can quickly regenerate the html docs and the defined type shows up. Be sure to click the `Defined Types` link in the top left, the left-hand menu does not mix classes and types.

You can see that there’s still some other work to do. The non-strings-ified portions of the comments are left as is, rather than parsed as markdown, so that needs to change. We don’t need most of that leftover crud. The class/defined type name is already known to strings. The Authors section should come from metadata.json (though if there are multiple, I am not sure if that file accepts an array). Copyright isn’t handled by metadata.json, and may not be strictly needed depending on your jurisdiction, but if you do need to keep it, just remove the === Copyright header and leave the text (I have chosen to omit it because US copyright law automatically grants me copyright for 70 years and I’m not that worried about it anyway; I would do something different for work).

I changed some other things:

  • Each  @param can take multi-line comments, as long as each trailing line maintains one space of extra indentation.
  • The title of defined types should be documented using @param title (docs), though it will generate a warning like [warn]: The @param tag for parameter 'name' has no matching parameter at manifests/vhost.pp:33
  • The order of metadata should go @summary > freeform text > @example > @param

Here’s the updated header and the resulting html doc:

# @summary Used in conjunction with puppetlabs/apache's apache::vhost definitions, to provide the related ssl_cert and ssl_key files for a given vhost.
#
# @example
#    Without Hiera:
#
#      $cname = www.example.com
#      certs::vhost{ $cname:
#        source_path => 'puppet:///site_certificates',
#      }
#
#    With Hiera:
#
#      server.yaml
#      ---
#      certsvhost:
#        'www.example.com':
#          source_path: 'puppet:///modules/site_certificates/'
#
#      manifest.pp
#      ---
#      certsvhost = hiera_hash('certsvhost')
#      create_resources(certs::vhost, certsvhost)
#      Certs::Vhost<| |> -> Apache::Vhost<| |>
#
# @param title
#  The title of the resource matches the certificate's name # e.g. 'www.example.com' matches the certificate for the hostname # 'www.example.com'
# @param source_path
#  Required. The location of the certificate files. Typically references a module's files. e.g. 'puppet:///site_certs' will search $modulepath/site_certs/files on the master for the specified files.
# @param target_path
#  Location where the certificate files will be stored on the managed node.
#  Default: '/etc/ssl/certs'
# @param service
#  Name of the web server service to notify when certificates are updated.
#  Default: 'http'

That’s about it! For small modules, this is probably a really simple, really quick change. For larger modules, this may take a while, but it’s tedious, not complicated.

Online Docs

There are two other things you may want to look at. First, the string docs can be a tad large (212K vs 24K for the actual manifests, for example) but more importantly, are NOT guaranteed to be in sync with the rest of your code. If you include doc/ in your git data and you change a parameter definition/use in a module and do not regenerate docs and commit them to the repo simultaneously, users may not understand and take action on the changes. If you go a long while without automatically updating them, you may confuse your users or even yourself.

You can simply add docs/ to your .gitignore file. Now, the docs are not stored in the Git repo – unless you add with `–force` or added them before updating .gitignore, at which point you will definitely want to correct that! This ensures no doc mismatch with published code and can help keep the size of the git repo a little more trim.

Second, GitHub and other providers often do not display HTML docs very well for your users, so even if you include doc/ in your repo, the contents are probably displayed as text files. Whoops! There are a few solutions for this.

  1. Publish through your Git provider’s services, like gh-pages, to a per-project website. For example, GitHub provides gh-pages sites and allows you to configure the publishing source (bonus: the rake task strings:gh_pages:update will push to this easily).
  2. Add a hook to your CI that generates the docs and sends them where necessary. Vox Pupuli is working on this but has not chosen an implementation yet.
  3. There are a few sites that you can add your docs to, some of which automagically update for you. One of these is http://www.puppetmodule.info/. You can easily click the Add Project button in the top right to add your own project to it (voila!). Since this occurs automatically, you never have to do anything else. But, when you are making changes, your docs could get stale until the next automated run occurs.

Puppet Module, by Dominic Cleal, also offers a badge you can add to your readme. Click the About button for more info.

Style Guides

One last thing to mention. As of 12/7/2017, the Style Guide is being updated to add information about puppet-strings. Pay attention to that space! I assume that it will first start with a description of standards and then add some puppet-lint checks to help you enforce it programatically. As puppet-strings is relatively new, you can expect more changes in the immediate future as it solidifies. If you have strong opinions on documentation, please speak up in the Documents Jira project, in Slack/IRC/mailing lists, or contact me and I’ll help you get your comments to the right person.

Summary

Today we added puppet-strings to a module, replaced the existing documentation with puppet-strings-compatible documentation, and looked at some solutions for automating document updates. It’s a simple process to enable better documentation updates, something everyone needs.

Upgrading Puppet Enterprise from 2016.4 to 2017.3

Over the past year, there have been some pretty big improvements to Puppet. I am still running PE 2016.4.2 and the current version is 2017.3.2, so there’s lot of changes in there. Most of the changes are backwards-compatible, so an upgrade from last November’s version is not quite as bad as it sounds, and I definitely want to start using the new features and improvements. The big one for me is Hiera version 5 (new in Puppet 4.9 / PE 2016.4.5). It is backwards compatible, so you can start using it right now, but it does require some changes to take advantage of the new features. I have to upgrade the server, upgrade the agents, and then start implementing the new features! Why do I care about Hiera 5 in particular?

Hiera 3 was great, but you could only use one hiera setup on a server, regardless of how many environments were deployed on that server. This could cause problems when you wanted to change the hiera config and test it. You could not test it in a feature branch, it HAD to be promoted to affect the entire server. If you had multiple masters, you could change the config on just one, but that was about as flexible as hiera 3 would let you be. If it worked, awesome. If it broke, you could break a whole lot that needed undone before you could try again.

Hiera 5 introduces independent hierarchy configurations per environment and even per module! If you want to try out a new backend like hiera-eyaml, you can now create a new feature branch eyaml-test, update the configuration in that branch, push it, and ONLY nodes that use that environment will receive the new configuration. This is a huge help in testing changes to hiera without blowing up all your nodes.

The per module hierarchy also means that module authors can include defaults that use hiera, rather than the params.pp pattern. This makes it easier for module users to override settings. There are also improvements in the interface for those who want to create their own backends. And, best of all, hiera 5 means the name hiera is here to stay – no more confusion between “legacy” hiera and modern lookup, it’s all called hiera 5 now.

It does mean there are some deprecations to keep in mind, but they won’t actually go away until at least Puppet 6. You can use hiera 5 now and take some time to replace the deprecated bits. This does mean we can use our existing hiera 3 setup and worry about migrating it to hiera 5 later, too, which we will take advantage of.

I always prefer to upgrade to the latest version. If for some reason you’re upgrading to Puppet 4.9, be aware that Puppet 4.10 fixed PUP-7554, which caused failures when a hiera 3 format hiera.yaml was found at the base of a controlrepo or module. I kept a hiera.yaml in the root of my controlrepo for bootstrapping purposes for a long while, and if you do, you could hit that bug. Best to just move to the latest if you can.

I think most people will like Hiera 5, but there are a ton of other features (listed at the end) and even if nothing appeals specifically, it is good to stay up to date so you don’t get stuck with a really nasty upgrade process when you find a feature you really need. Please, don’t let yourself get a full year behind on updates like I did. Sometimes it’s really difficult to get out of that situation!

Puppet Enterprise Server Upgrade

I use Puppet Enterprise at both work and home these days, so I will go through the PE upgrade experience. The Puppet OpenSource install and upgrade instructions are on the same page of the documentation, so it seems pretty easy but your mileage may vary, of course.

First, take a look at your installation and make sure it’s in a known state – preferably a known good state all the way around, but at least a known one. If you have outstanding issues on the master, you need to resolve them. If some agents are failing to check in, you may want to take the time to fix them, or you could just keep track of the failures. After the upgrade, you don’t want to see an increase in failures. Once everything looks good, take a snapshot of your master(s) and a full application/OS backup if possible. If you have a distributed setup, perform this on all nodes as close to the same time as possible.

Second, download the latest version of PE (KB#0001) onto your master. Expand the tarball, cd into the directory, and run sudo ./puppet-enterprise-installer. You can provide a .pe.conf file with the -c option, or answer a few interactive questions to get started:

=============================================================
 Puppet Enterprise Installer
=============================================================
2017-11-08 20:38:07,432 Running command: cp /opt/puppetlabs/server/pe_build /opt/puppetlabs/server/pe_build.bak
2017-11-08 20:38:07,480 Running command: cp /opt/puppetlabs/puppet/VERSION /opt/puppetlabs/server/puppet-agent-version.bak

## We've detected an existing Puppet Enterprise 2016.5.2 install.

 Would you like to proceed with text-mode upgrade/repair? [Yn]y

## We've found a pe.conf file at /etc/puppetlabs/enterprise/conf.d/pe.conf.

 Proceed with upgrade/repair of 2016.5.2 using the pe.conf at /etc/puppetlabs/enterprise/conf.d/pe.conf? [Yn]y

The install takes a bit of time (30 minutes on my lab install). Once the upgrade is done, you’ll be directed to run puppet agent -t (with sudo of course). If you have additional compile masters or ActiveMQ hubs and spokes, also run the commands in steps 4 and 5 as well.

You should now be able to log into the Console and see the status of your environment. You will hopefully find Intentional Changes on most of your nodes and zero or no failures (if both are encountered in a run, Intentional Changes “wins” on the Console; let every node run at least twice to see if it moves back to Green or Red before continuing).

If you do encounter failures, you will have to analyze each issue to see if it’s related to the upgrade and something you can fix, or if it’s time to roll back. If you do roll back, make sure you roll back ALL the PE components, including the PuppetDB, so you don’t leave cruft somewhere. I experienced one issue in the lab described in PUP-7878, resolved by a reboot of the master after the upgrade.

If everything is good, then it is time to proceed to upgrading the Agents.

Agent Upgrades

The Puppet docs provide instructions for upgrading Agents in a variety of methods. I prefer to use the module puppetlabs/puppet_agent, as I’ve discussed before (OpenSource, PE Linux and PE Windows clients). My experience is using my profile module and hiera data in the controlrepo, Puppet’s instructions use the Console Classifier. It really does not matter how you do this, but I did find an issue with the Puppet docs (DOCUMENT-763) – after classifying, you must set the parameter puppet_agent::package_version or no upgrade occurs for agents already running Puppet 4 or 5. Set it to 5.3.3(obtained by running puppet --version on the master, which received the latest agent during the upgrade). Here’s how to do that in hiera:

puppet_agent::package_version: '5.3.3'

The next two agent runs will show changes. I ran my tests directly using ssh on a Linux host and it looked like this:

# First run upgrades Puppet 4 to 5
[rnelson0@build03 controlrepo:pe201732]$ sudo puppet agent -t --environment pe201732
Info: Using configured environment 'pe201732'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for build03.nelson.va
Info: Applying configuration version '1510241461'
Notice: /Stage[main]/Puppet_agent::Osfamily::Redhat/Yumrepo[pc_repo]/baseurl: baseurl changed 'https://yum.puppetlabs.com/el/$releasever/PC1/x86_64' to 'https://puppet.nelson.va:8140/packages/2017.3.2/el-7-x86_64'
Notice: /Stage[main]/Puppet_agent::Osfamily::Redhat/Yumrepo[pc_repo]/sslcacert: defined 'sslcacert' as '/etc/puppetlabs/puppet/ssl/certs/ca.pem'
Notice: /Stage[main]/Puppet_agent::Osfamily::Redhat/Yumrepo[pc_repo]/sslclientcert: defined 'sslclientcert' as '/etc/puppetlabs/puppet/ssl/certs/build03.nelson.va.pem'
Notice: /Stage[main]/Puppet_agent::Osfamily::Redhat/Yumrepo[pc_repo]/sslclientkey: defined 'sslclientkey' as '/etc/puppetlabs/puppet/ssl/private_keys/build03.nelson.va.pem'
Notice: /Stage[main]/Puppet_agent::Install/Package[puppet-agent]/ensure: ensure changed '1.9.2-1.el7' to '5.3.3-1.el7'
Notice: /Stage[main]/Puppet_enterprise::Mcollective::Server::Logs/File[/var/log/puppetlabs/mcollective]/mode: mode changed '0750' to '0755'
Notice: Applied catalog in 71.86 seconds

# Second run updates some PE components
[rnelson0@build03 controlrepo:pe201732]$ sudo puppet agent -t --environment pe201732
Info: Using configured environment 'pe201732'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for build03.nelson.va
Info: Applying configuration version '1510241554'
Notice: /Stage[main]/Puppet_enterprise::Pxp_agent/File[/etc/puppetlabs/pxp-agent/pxp-agent.conf]/content:
--- /etc/puppetlabs/pxp-agent/pxp-agent.conf 2017-11-08 21:16:56.713834368 +0000
+++ /tmp/puppet-file20171109-20790-1l6y5bg 2017-11-09 15:32:45.917909748 +0000
@@ -1 +1 @@
-{"broker-ws-uris":["wss://puppet.nelson.va:8142/pcp2/"],"pcp-version":"2","ssl-key":"/etc/puppetlabs/puppet/ssl/private_keys/build03.nelson.va.pem","ssl-cert":"/etc/puppetlabs/puppet/ssl/certs/build03.nelson.va.pem","ssl-ca-cert":"/etc/puppetlabs/puppet/ssl/certs/ca.pem","loglevel":"info"}
\ No newline at end of file
+{"broker-ws-uris":["wss://puppet.nelson.va:8142/pcp2/"],"pcp-version":"2","master-uris":["https://puppet.nelson.va:8140"],"ssl-key":"/etc/puppetlabs/puppet/ssl/private_keys/build03.nelson.va.pem","ssl-cert":"/etc/puppetlabs/puppet/ssl/certs/build03.nelson.va.pem","ssl-ca-cert":"/etc/puppetlabs/puppet/ssl/certs/ca.pem","loglevel":"info"}
\ No newline at end of file

Info: Computing checksum on file /etc/puppetlabs/pxp-agent/pxp-agent.conf
Info: /Stage[main]/Puppet_enterprise::Pxp_agent/File[/etc/puppetlabs/pxp-agent/pxp-agent.conf]: Filebucketed /etc/puppetlabs/pxp-agent/pxp-agent.conf to puppet with sum cad3d2db7a7a912a1734b7e8afa23037
Notice: /Stage[main]/Puppet_enterprise::Pxp_agent/File[/etc/puppetlabs/pxp-agent/pxp-agent.conf]/content: content changed '{md5}cad3d2db7a7a912a1734b7e8afa23037' to '{md5}a19b53e1586a748ba488ee4dcd7afc3c'
Info: /Stage[main]/Puppet_enterprise::Pxp_agent/File[/etc/puppetlabs/pxp-agent/pxp-agent.conf]: Scheduling refresh of Service[pxp-agent]
Notice: /Stage[main]/Puppet_enterprise::Pxp_agent::Service/Service[pxp-agent]: Triggered 'refresh' from 1 event
Notice: /Stage[main]/Puppet_enterprise::Mcollective::Server/File[/etc/puppetlabs/mcollective/server.cfg]/content: [diff redacted]
Info: Computing checksum on file /etc/puppetlabs/mcollective/server.cfg
Info: /Stage[main]/Puppet_enterprise::Mcollective::Server/File[/etc/puppetlabs/mcollective/server.cfg]: Filebucketed /etc/puppetlabs/mcollective/server.cfg to puppet with sum 7a8d59f271273738a51b4cf05ee6b33a
Notice: /Stage[main]/Puppet_enterprise::Mcollective::Server/File[/etc/puppetlabs/mcollective/server.cfg]/content: changed [redacted] to [redacted]
Info: /Stage[main]/Puppet_enterprise::Mcollective::Server/File[/etc/puppetlabs/mcollective/server.cfg]: Scheduling refresh of Service[mcollective]
Notice: /Stage[main]/Puppet_enterprise::Mcollective::Service/Service[mcollective]: Triggered 'refresh' from 1 event
Notice: Applied catalog in 6.02 seconds

I assume the PE component updates are based on facts only present with Puppet 5, facts that would not be present during the first run while the agent is still Puppet 4. Subsequent runs are stable.

I do not have a Windows agent to test with in my lab, I assume it looks similar but cannot verify. Be sure to test at least one Windows agent before releasing this change across your entire Windows fleet.

New Features

I have skipped from 2016.4 to 2017.3, which means I have missed out on new features in four major versions: 2016.5, 2017.1, 2017.2, and 2017.3. Here are some of the big features from the release notes:

I mentioned Hiera 5 already, which I’ll discuss further in another post. I also want to immediately enable the Package Inventory. As described, I can update the Classification of the PE Agent node group to include puppet_enterprise::profile::agent with package_inventory_enabled set to true and committing the change.

While it takes affect immediately, your agents need two runs to show up: the first changes the setting so package data is collected and the second actually collects the list. Once that happens with at least one node, you’ll start seeing data populate on the Inspect > Packages page.

I do not have need for High Availability myself, but that seems really cool. In the past, it’s been quite the pain in the behind to automate yourself. I have not used Orchestrator in anger before, and I do Hiera overrides in my control repo, almost ignoring the Console Classifier otherwise, so I probably will not be exploring them very well. However, I’m really excited about Tasks, that’s something I hope to explore during the winter break, perhaps by upgrading bash across all my systems!

Summary

Today we looked at why we want to upgrade to the latest Puppet and upgraded a Puppet Enterprise monolithic master and some linux agents. It’s not that hard! We also staked out features that we want to investigate and turned on the Package Inventory. There are a lot more changes than I listed, along with tons of fixed bugs and smaller improvements, so I recommend reviewing the release notes for each version to see what interests you.

I hope to be able to look into Hiera 5 and Tasks soon, look for new blog posts on those! Let me know if there’s anything else you’d like to see discussed in the comments or on twitter. Thanks!

Managing SSH server security with Puppet

Edit: In an earlier edition, I credited the wrong newsletter as the source. My apologies to R.I.Pienaar!

In this past week’s DevCo Newsletter, I saw the Rebex SSH Check, which reminded me that I’ve locked down the SSH server security configuration at work, but not at home. Sounds like a good opportunity to blog about the process!

Now, I’m in security, but I’m not all that about the security settings. The names vary from descriptive to really obtuse, and there’s three keys that need managed: ciphers, MACs, and KexAlgorithms (that’s Key Exchange Algorithms, which is the name I’m more familiar with). The key to security is knowing when you don’t know, and seeking out that expertise. I am very thankful for Mozilla’s really great security guidelines, including an OpenSSH guide. There are sections for Modern and Intermediate security, depending on what is available for the systems you are securing. For me, these align with the Red Hat/CentOS EL7 (Modern) and EL6/5 (Intermediate) distros that I use.

The first step is making sure we have a tier in hiera for each OS/release we support, otherwise sshd could fail to restart when it encounters a cipher set name that is unknown to the openssh version in use. That could be bad, especially if we don’t have some form of iLO console to the nodes, though if we have puppet running on a regular basis or through mcollective, we *should* be able to recover. In any case, you definitely want to check run status of your nodes after this change to make sure you don’t discover a problem when you’re trying to troubleshoot some other problem.

I define my hierarchy in hiera itself using the puppet/hiera module, so here is the yaml for hiera to parse as well as the resulting hiera.yaml, the change is in bold:

# portion of hiera/puppet_role/puppet.yaml, which applies to the puppet master
hiera::hierarchy:
  - 'clientcert/%%{::}{clientcert}'
  - 'puppet_role/%%{::}{puppet_role}'
  - 'osfamily-release/%%{::}{osfamily}-%%{::}{operatingsystemmajrelease}'
  - 'datacenter/%%{::}{datacenter}'
  - 'global'

# /etc/puppetlabs/puppet/hiera.yaml
# managed by puppet
---
:backends:
- eyaml
- yaml

:logger: console

:hierarchy:
  - "clientcert/%{clientcert}"
  - "puppet_role/%{puppet_role}"
  - "osfamily-release/%{osfamily}-%{operatingsystemmajrelease}"
  - "datacenter/%{datacenter}"
  - global

:eyaml:
  :datadir: "/etc/puppetlabs/puppet/environments/%{::environment}/hiera"
  :extension: yaml
  :pkcs7_private_key: "/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem"
  :pkcs7_public_key: "/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem"

:yaml:
  :datadir: "/etc/puppetlabs/puppet/environments/%{::environment}/hiera"

:merge_behavior: deeper

This change will need to be put in place on the master, the master service restarted, and no dissimilar configs exist in the wrong location before agents will see the changes we make below (I had a /etc/puppetlabs/code/hiera.yaml that slightly vared from /etc/puppetlabs/puppet.hiera.yaml and it kept winning out till I removed it and restarted pe-puppetserver). You can force the run now, or wait up to two full run cycles before verifying that all your agents see the changes.

The second step is to populate the two OS/release files with the specific sets you want to use. I use saz/ssh, which allows me to use the ssh::server::options parameter to free-hand some stanzas into /etc/sshd_config. These commands replicate my settings, again according to Modern for EL7 and Intermediate for EL6:

mkdir hiera/osfamily-release
cat > hiera/osfamily-release/RedHat-6.yaml << EOF
---
ssh::server::options:
  'KexAlgorithms'            : 'diffie-hellman-group-exchange-sha256'
  'Ciphers'                  : 'aes256-ctr,aes192-ctr,aes128-ctr'
  'MACs'                     : 'hmac-sha2-512,hmac-sha2-256'
EOF

cat > hiera/osfamily-release/RedHat-7.yaml << EOF
---
ssh::server::options:
  'KexAlgorithms'            : 'curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256'
  'Ciphers'                  : 'chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr'
  'MACs'                     : 'hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com'
EOF

There’s one final step: merge settings. You may have noticed the merge_behavior setting in my hiera.yaml above, but that’s defunct. Now you must set the lookup options. I do this in my least specific hiera file, hiera/global.yaml:

lookup_options:
  profile::base::linux::sudo_confs:
    merge: deep
  profile::base::linux::logrotate_rules:
    merge: deep
  ssh::server::options:
    merge: deep

If you don’t add this, then you’ll only get the first ssh::server::options values found, even for sub-keys like Ciphers that were not set at the higher tier.

Once all of these changes are in place, your agents should get the new settings and restart sshd. Any new ssh connections to the affected servers will use the specified security sets and ONLY the specified security sets. Existing connections will persist until the server or client end the sessions. We can now use curve25519-sh256@libssh.org as a KexAlgorithm with an EL6 node, but we would fail to connect to an EL7 node as only diffie-hellman-group-exchange-sha256 is available. If we re-run the Rebex SSH Test, our Modern servers show all green now. Success!

Addendum: Peter Souter notified me on twitter about his mozilla_ssh_hardening module (GitHub only at this time) that enforces the Mozilla recommendations on Ubuntu 16.04, CentOS 7, and CentOS 6. You can use that module to replace some of the work above, as long as you do not require conflicting customizations. I still hope this articles helps you understand the workings of hiera merges and the need for vetted security configurations.

Puppet 5 has arrived!

If you missed the news this past week, the Puppet 5 Platform was released! Read the announcement and the release notes for some great details. Congratulations to everyone at Puppet for getting this new release out the door. I’m looking forward to diving in with it as soon as a Puppet Enterprise release is out, since I’ve converted even my home lab away from Puppet Opensource.

There are a few things I’ve learned from the announcement thread, slack, and my own experiences with it in the last few days. It’s still early, so I am sure this will get out of date quickly, but I hope it helps others in the short term.

  • Puppet 5 AIO provides Ruby 2.4.1, so your tests should use it as well – even if you’re not using AIO puppet, it’s still helpful for any puppet modules.
  • PuppetDB requires postgresql96, but it’s not a dependency on the puppetdb package, since you can install puppetdb and postgresql on different hosts. Version 4.x works with postgresql96, so upgrade that first, then puppet. Detail here.
  • Puppet 5 includes a vendored version of the semantic_puppet gem. In Puppet 4.7 and below, it had a dependency on the external semantic_puppet getm. The gem is used by metadata-json-lint, which is often part of a puppet rspec test setup. Check out the metadata-json-lint README installation section to see how to deal with this. If your tests run against ~> 4.0 then you’re probably okay.
  • There’s a new version of puppetlabs_spec_helper that apparnetly has some issues with spec fixtures and symlinks (from slack, nothing to quote). I haven’t hit this myself, it might already be fixed, but something to be aware of if you have symlink-related issues during testing.
  • The combination of Puppet 5, rpsec-puppet, and the new puppetlabs_spec_helper are more stringent than Puppet 4 is. I’m not sure which of the three components specifically triggers it. I was testing for a resource that required another service, which was not part of the define I was testing (here). With puppet 4, this was fine, but with puppet 5, it started generating errors in this travis run. The fix is simple, through using a pre_condition to provide the service in the catalog, seen in this commit.
  • The first Puppet Enterprise release including Puppet 5 should be out sometime this fall.

That’s all I’ve run into so far. One last thing, here is a .travis.yml for testing component modules with both Puppet 4 and 5. You only need to update the matrix section, if you already have one, but I thought the whole thing might be helpful for those who don’t have tests yet.:

---
language: ruby
sudo: false
cache: bundler
notifications:
  email:
  on_failure: always
branches:
  only:
  - master
bundler_args: --without development system_tests
before_install: rm Gemfile.lock || true
script: bundle exec rake test
matrix:
  fast_finish: true
  include:
  - rvm: 2.3.1
    env: PUPPET_GEM_VERSION="~> 4.0" STRICT_VARIABLES=yes
  - rvm: 2.4.1
    env: PUPPET_GEM_VERSION="~> 5.0" STRICT_VARIABLES=yes

Where to store Puppet files and templates

I haven’t written a blog post in a while because I’ve been bogged down in work and life and not had much time in the lab. To make sure I don’t get too out of practice, I’m going to try writing some shorter tips and tricks articles. Let me know what you think.

A few days ago, someone asked a great question on the puppet-users mailing list about the location of config files in the roles/profile pattern. It’s a good question, and we can go deeper because it assumes the location of config files outside of that pattern, too. I’m going to explain where I keep my config files, and templates, in the various types of modules. There’s no single correct answer here, this is just a framework for me.

To start, let’s describe the types of modules. Component modules describe a single application/technology/thing and are designed to be consumed by end users. This is pretty much anything on the forge, such as puppet/hiera to manage a Hiera implementation or puppetlabs/apache to manage apache, vhosts, etc. There’s also a sub-type of these modules, Private Components. The line here is blurry, but think component modules that are not designed to go on the forge. This could be a module for a company’s internal application, very similar to a traditional component module, or an cluster of custom facts. Pretty much anything that’s not a Component module, or our final type: Profile modules. This last type is the collection of classes that make up your role/profile pattern implementation. They’re often simply called profile, but maybe there is more than one module if you have a lot of business groups using the puppet system. They differ from both types of component modules in that they contain the business logic of your implementation and are where you compose the collection of component modules that you use. I wrote an article on what goes in a role or profile, too.

In Component modules, the relevant configuration files or templates for the component are collected. In an ssh module, you’d have the ssh_config and sshd_config data; a sudo module would have sudoers and a template for sudoers.d/ files. Private Component modules vary quite a bit in functionality, but I treat them like regular component modules. If the module is for custom facts, there’s no need to put files or templates in it. If it’s for an internal app, the configuration files are stored in that module.

Your Role/Profile modules are a little more complicated. If you have a component module for apache, you likely have a profile class for apache, perhaps profile::apache or profile::somegroup::apache. The component module probably has its own file or template, but it may accept alternative files and templates. In this case, I create a sub-directory with the module subclass name, such as files/apache or templates/apache,  and add the file(s) there, e.g. templates/apache/vhosts.erb.

This is a pretty simple layout. The only real difficulty is when you have a private component modules and a profile for that component: do you put the file/template with the profile or the component module? I tend to lean toward the private component modules first, but I’ve done both.

I hope this helps and I’d love to hear of any other layouts you’ve had success with!

vRealize Orchestrator Workflows for Puppet Enterprise

Over the past three years, my Puppet for vSphere Admins series has meandered through a number of topics, mostly involved on the Puppet side and somewhat light on the vSphere side. That changed a bit with my article Make the Puppet vRealize Automation plugin work with vRealize Orchestrator, describing how to use the plugin’s built-in workflows to perform some actions on your VMs. However, you had to invoke the workflows one by one, and they only worked on existing VMs. That is not good enough for automation! Today, we will start to look at how to integrate the Puppet Enterprise plugin into other workflows to provide end-to-end lifecycle management for your VMs.

What is the lifecycle of a VM? This can vary quite a bit, so the lifecycle we will work with today is made to be generic enough for everyone to use, but flexible enough that everyone can expand on it. It consists of:

  • Provisioning
    • Updating ancillary systems prior to VM creation (IPAM, DNS, etc)
    • Deploying a Virtual Machine
    • Installing Puppet Enterprise on the VM
    • Using Puppet Enterprise to provision services on and configure the VM
    • Add the new VM to a vCenter tag-based backup system
  • Decommission
    • Delete the VM (removes from backups)
    • Purge the record from PE
    • Update ancillary systems after VM removal (IPAM, DNS, etc)

Continue reading

Connecting Puppetboard to Puppet Enterprise

Last week, I moved the home lab to Puppet Enterprise. One of the things I love about PE is the Console. However, I am a member of Vox Pupuli and we develop Puppetboard (the app AND the module) so it is convenient for me to use it and tie it into PE as well. Though the two overlap, each has functionality the other doesn’t. I really love the count of agent runs by status on the Puppetboard Overview page, for instance. After my migration, however, my previously-working puppetboard just gave me HTTP 500 errors. Fixing it took some wrangling. Thanks to Tim Meusel for his assistance with the cert issue.

First, let’s look at the existing manifest and hiera data for my profile::puppetboard class:

Continue reading