Last week, I moved the home lab to Puppet Enterprise. One of the things I love about PE is the Console. However, I am a member of Vox Pupuli and we develop Puppetboard (the app AND the module) so it is convenient for me to use it and tie it into PE as well. Though the two overlap, each has functionality the other doesn’t. I really love the count of agent runs by status on the Puppetboard Overview page, for instance. After my migration, however, my previously-working puppetboard just gave me HTTP 500 errors. Fixing it took some wrangling. Thanks to Tim Meusel for his assistance with the cert issue.

First, let’s look at the existing manifest and hiera data for my profile::puppetboard class:

# dist/profile/manifests/puppetboard.pp
#
class profile::puppetboard {
  include apache
  include apache::mod::wsgi

  include puppetboard
  include puppetboard::apache::vhost

  firewall { '100 Puppetboard inbound':
    dport  => 80,
    proto  => tcp,
    action => accept,
  }
}

# hiera/puppet_role/puppetboard.yaml
---
classes:
  role::puppetboard
puppetboard::offline_mode: true
puppetboard::manage_git: true
puppetboard::manage_virtualenv: true
puppetboard::apache::vhost::vhost_name: 'puppetboard.example.com'
puppetboard::apache::vhost::port: 80
puppetboard::revision: 'v0.2.1'
puppetboard::puppetdb_host: 'puppet.example.com'
puppetboard::puppetdb_port: 8080
python::dev: true
apache::mod::wsgi::wsgi_socket_prefix: '/var/run/wsgi'

This works fine with an Open Source installation of Puppet with PuppetDB listening on port 8080 (HTTP). When we install Puppet Enterprise, its PuppetDB only listens to port 8081 (HTTPS) by default. We need to change the port number and configure the SSL certs. Because this is a puppet-managed node, we already have certs in ssldir (/etc/puppetlabs/puppet/ssl) that can be used. The permissions will need to be modified so that the apache user can access them. The CA and certificate file are already readable but not the private key. The following addition to profile::puppetboard changes this:

  file {'/etc/puppetlabs/puppet/ssl/private_keys':
    ensure => directory,
    mode => '0755',
  }
  file {"/etc/puppetlabs/puppet/ssl/private_keys/${facts['clientcert']}.pem":
    ensure => file,
    mode => '0644',
  }

I don’t particularly think this is a dangerous change in permissions, but you could also create a new certificate on the master with puppet cert generate <hostname> and transfer the resulting files to a location readable by apache if you desire. Next, the hiera files need updated. As the cert name is specific to the node, I moved some data from my puppet_role hierarchy to clientcert instead:

# hiera/puppet_role/puppetboard.yaml
---
classes:
  role::puppetboard
puppetboard::offline_mode: true
puppetboard::manage_git: true
puppetboard::manage_virtualenv: true
puppetboard::apache::vhost::vhost_name: 'puppetboard.example.com'
puppetboard::apache::vhost::port: 80
puppetboard::revision: 'v0.2.1'
puppetboard::puppetdb_host: 'puppet.example.com'
puppetboard::puppetdb_port: 8081
python::dev: true
apache::mod::wsgi::wsgi_socket_prefix: '/var/run/wsgi'

# hiera/clientcert/puppetboard.example.com.yaml
---
puppetboard::puppetdb_key: "/etc/puppetlabs/puppet/ssl/private_keys/puppetboard.example.com.pem"
puppetboard::puppetdb_cert: "/etc/puppetlabs/puppet/ssl/certs/puppetboard.example.com.pem"
puppetboard::puppetdb_ssl_verify: '/etc/puppetlabs/puppet/ssl/certs/ca.pem'

You can probably use %{facts['clientcert'] and the like in hiera, but I’m a little gunshy around interpolation in hiera if I can avoid it since I’ve been burned in the past. If you do use facts, you can probably put those last three lines in the puppet_role hierarchy instead.

We also told the puppetboard node to contact the puppet server on port 8081 instead of 8080, but until we update the firewall rule on the master, we won’t get very far. This is a simple fix in profile::puppet_master (or your equivalent:

  firewall {'105 puppetdb inbound':
    dport  => 8081,
    proto  => tcp,
    action => accept,
  }

Finally, there’s one issue that I really didn’t anticipate. Regardless of setting up the key, cert, and ca location, the puppetboard node can’t connect. We can see this at the command line with curl:

$ curl 'https://puppet.example.com:8081/pdb/query/v4/nodes'   --tlsv1   --cacert /etc/puppetlabs/puppet/ssl/certs/ca.pem   --cert /etc/puppetlabs/puppe
t/ssl/certs/puppetboard.example.com.pem   --key /etc/puppetlabs/puppet/ssl/private_keys/puppetboard.example.com.pem
Permission denied: The client certificate name puppetboard.example.com doesnt appear in the certificate whitelist. Is your master's (or other PuppetDB 
client's) certname listed in PuppetDB's certificate-whitelist file?

There’s a setting called certificate-whitelist in the puppetdb section of puppet.conf. When it is not present – the default with Puppet Open Source – all nodes are allowed to connect. When it is present – the default with PE – only listed hostnames are allowed to connect. Out of the box, only the master/s)’s hostnames are added to this list. We need to add the puppetboard server. We can do this in the Console under Nodes -> Classification -> PE PuppetDB -> Classes -> whitelisted_certnames. However, I prefer to add it in hiera since that’s under version control. It takes an array (check your merge settings/lookup options if you are providing this value anywhere else):

# hiera/puppet_role/puppet.yaml
puppet_enterprise::profile::puppetdb::whitelisted_certnames:
  - 'puppetboard.example.com'

Once you run puppet on both nodes against these changes, you can use the curl command above to verify that you can connect:

$ curl 'https://puppet.example.com:8081/pdb/query/v4/nodes'   --tlsv1   --cacert /etc/puppetlabs/puppet/ssl/certs/ca.pem   --cert /etc/puppetlabs/puppe
t/ssl/certs/puppetboard.example.com.pem   --key /etc/puppetlabs/puppet/ssl/private_keys/puppetboard.example.com.pem
[{"deactivated":null,"latest_report_hash":"73b7680b782606357c08cf8d4c50619d27e7d161","facts_environment":"production","cached_catalog_status":"not_used
","report_environment":"production","latest_report_corrective_change":false,"catalog_environment":"production","facts_timestamp":"2017-03-11T23:57:02.2
89Z","latest_report_noop":false,"expired":null,"latest_report_noop_pending":false,"report_timestamp":"2017-03-11T23:57:05.653Z","certname":"jenkins.exa
mple.com","catalog_timestamp":"2017-03-11T23:57:09.3...
<GIANT BLOB OF TEXT HERE>

You can restart apache on the puppetboard server (the module does not currently notify the service when the configuration changes) and when you access puppetboard, it will connect to Puppet Enterprise. No more 500 errors! Enjoy!