Welcome back to our Puppet for vSphere Admins series. We started out deploying the puppet master and a few example manifests, then took a right turn into the land of Git and workflows. I know you’re anxious to get back to developing manifests, but we’ve got a few more things to install on the master before we worry about the manifests. PuppetDB and Hiera, and MCollective are powerful tools that most administrators will find of immense benefit. We can install these later, but we’d have to redo some of our work. Who wants to do that?

As I mentioned last week, I’ll assume you’re using r10k at some level, so I’ll mostly just reference “use r10k” unless there’s a specific gotcha. If you’re not using r10k, follow whatever workflow you’ve decided on to add modules, update manifests, and track all changes in your VCS.

PuppetDB

First, we’ll install PuppetDB. PuppetDB allows us to work with exported resources and the inventory service. All of the agents that will eventually connect to the puppet master will have facts that we care about – facts like hostname, ips, operating system, mac addresses, the kernel version, etc. You can view the facts by typing facter on the master. However, all of these facts are local to the agents. Using PuppetDB, these facts will be stored and made available to others. Exported resources allow us to mark facts that we want other agents to have access to, such as a node’s public SSH key. The inventory service is an API for services to see the fact, such as Puppet Dashboard or your own software that leverages the API.

PuppetDB can be installed from source, rpm, or via Puppet itself. We’ll use the latter. We need the module puppetlabs/puppetdb and will add the class to the master’s node definition. To get started, let’s create a new branch of the puppet repo based on the production branch (installables). If you’re using r10k, add the module (and dependencies, like puppetlabs/postgresql) to Puppetfile.

mod "puppetlabs/puppetdb"
mod "puppetlabs/postgresql"

Otherwise use puppet module install puppetlabs/puppetdb. Add these lines to the master’s node definition:

  include ::puppetdb
  include ::puppetdb::master::config

Deploy your changes and run a noop test. You should see the package installation, service configuration, postgresql installation, and a modification to use the puppetdb in puppet.conf. It will look something like this – with a lot of errors because of the noop, about failed dependencies, but they should all be tied to postgresql or puppetdb:

Notice: /Stage[main]/Postgresql::Server::Config/Postgresql::Server::Config_entry[listen_addresses]/Postgresql_conf[listen_addresses]/ensure: current_value absent, should be present (noop)
Info: /Stage[main]/Postgresql::Server::Config/Postgresql::Server::Config_entry[listen_addresses]/Postgresql_conf[listen_addresses]: Scheduling refresh of Class[Postgresql::Server::Service]
Notice: Postgresql::Server::Config_entry[listen_addresses]: Would have triggered 'refresh' from 1 events
Notice: /Stage[main]/Main/Node[puppet.nelson.va]/Notify[Generated from our notify branch]/message: current_value absent, should be Generated from our notify branch (noop)
Notice: Node[puppet.nelson.va]: Would have triggered 'refresh' from 1 events
Notice: Class[Main]: Would have triggered 'refresh' from 1 events
Notice: /Stage[main]/Postgresql::Server::Config/Concat[/var/lib/pgsql/data/pg_hba.conf]/File[/var/lib/puppet/concat/_var_lib_pgsql_data_pg_hba.conf]/ensure: current_value absent, should be directory (noop)
...

...
Notice: /Stage[main]/Puppetdb::Master::Puppetdb_conf/Ini_setting[puppetdbserver]: Dependency Exec[concat_/var/lib/pgsql/data/pg_hba.conf] has failures: true
Warning: /Stage[main]/Puppetdb::Master::Puppetdb_conf/Ini_setting[puppetdbserver]: Skipping because of failed dependencies
Notice: /Stage[main]/Puppetdb::Master::Config/Service[puppetmaster]: Dependency Exec[concat_/var/lib/pgsql/data/pg_hba.conf] has failures: true
Warning: /Stage[main]/Puppetdb::Master::Config/Service[puppetmaster]: Skipping because of failed dependencies
Notice: Class[Puppetdb::Master::Config]: Would have triggered 'refresh' from 1 events

If everything went well, run it again without the noop. This will be very noisy and take a while, there’s a LOT of moving parts here. You’ll likely see some “errors” about connection refused, but don’t fret, it just takes a few moments for the services to start up. No errors should show up in the lengthy output. Run one more agent checkin (one change expected that I’ll explain later, but we want some logs) and then you can check that everything went well by checking that the packages are installed, processes are running, and that logs are showing connections:

[root@puppet ~]# rpm -qa | grep puppetdb
puppetdb-1.6.3-1.el6.noarch
puppetdb-terminus-1.6.3-1.el6.noarch
[root@puppet ~]# rpm -qa | grep postgresql
postgresql-libs-8.4.20-1.el6_5.x86_64
postgresql-server-8.4.20-1.el6_5.x86_64
postgresql-8.4.20-1.el6_5.x86_64
[root@puppet ~]# ps -ef | grep puppetdb
puppetdb  6370     1  6 18:53 ?        00:00:39 /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d
postgres  6442  6115  0 18:53 ?        00:00:00 postgres: puppetdb puppetdb 127.0.0.1(50875) idle
postgres  6443  6115  0 18:53 ?        00:00:00 postgres: puppetdb puppetdb 127.0.0.1(50876) idle
...
[root@puppet ~]# puppet agent --test --environment=installables
Info: Applying configuration version '1398020510'
Notice: /Stage[main]/Ssh::Hostkeys/Sshkey[puppet.nelson.va_dsa]/ensure: created
Notice: /Stage[main]/Ssh::Client::Config/File[/etc/ssh/ssh_known_hosts]/mode: mode changed '0600' to '0644'
Notice: Generated from our notify branch
Notice: /Stage[main]/Main/Node[puppet.nelson.va]/Notify[Generated from our notify branch]/message: defined 'message' as 'Generated from our notify branch'
Notice: /Stage[main]/Ssh::Hostkeys/Sshkey[puppet.nelson.va_rsa]/ensure: created
Notice: Finished catalog run in 4.09 seconds
[root@puppet ~]# tail /var/log/puppetdb/puppetdb.log
2014-04-20 18:53:53,352 INFO  [pool-3-thread-1] [cli.services] Starting database garbage collection
2014-04-20 18:53:53,421 INFO  [pool-3-thread-1] [cli.services] Finished database garbage collection
2014-04-20 18:53:53,451 INFO  [pool-3-thread-1] [cli.services] Starting sweep of stale reports (threshold: 14 days)
2014-04-20 18:53:53,488 INFO  [pool-3-thread-1] [cli.services] Finished sweep of stale reports (threshold: 14 days)
2014-04-20 18:53:53,511 INFO  [clojure-agent-send-off-pool-2] [server.Server] jetty-7.x.y-SNAPSHOT
2014-04-20 18:53:54,073 INFO  [clojure-agent-send-off-pool-2] [server.AbstractConnector] Started SelectChannelConnector@localhost:8080
2014-04-20 18:53:54,434 INFO  [clojure-agent-send-off-pool-2] [ssl.SslContextFactory] Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
2014-04-20 18:53:54,466 INFO  [clojure-agent-send-off-pool-2] [server.AbstractConnector] Started SslSelectChannelConnector@puppet.nelson.va:8081
2014-04-20 19:01:50,312 INFO  [command-proc-51] [puppetdb.command] [7a2fdda7-f44a-4e57-be76-f20640775a52] [replace facts] puppet.nelson.va
2014-04-20 19:02:00,903 INFO  [command-proc-51] [puppetdb.command] [6295434a-af80-4267-9e38-9bd48dfe00f7] [replace catalog] puppet.nelson.va

Now with PuppetDB installed, you have access to exported resources. You get one for free: a resource of the type SSH::Hostkeys. We get this via the saz/sudo module we installed at the beginning of this series. You can see the Puppet DSL code used here. We’ll get into how to use the resources at a later date.

Hiera

Hiera is a different install, a little easier in some ways and more difficult in others. The first part is the easy part, installing two packages, hiera and hiera-puppet from the Puppet repos. Just add the resources to the node definition. This gives us:

node 'puppet.nelson.va' {
  include ::base
  notify { "Generated from our notify branch": }

  # PuppetDB
  include ::puppetdb
  include ::puppetdb::master::config

  # Hiera
  package { ['hiera', 'hiera-puppet']:
    ensure => present,
  }
}

Modern puppet includes the hiera functionality by default. However, it’s good to explicitly state our requirements for the puppet master. When you run puppet again, you may not notice any changes since the components are already installed. If you do rebuild the master, this will ensure Hiera is installed.

Depending on your versions, you may receive an error, though:

Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install hiera-puppet' returned 1:

Transaction Check Error:
  file /usr/bin/extlookup2hiera from install of hiera-puppet-1.0.0-1.el6.noarch conflicts with file from package puppet-3.4.3-1.el6.noarch
  file /usr/lib/ruby/site_ruby/1.8/hiera/backend/puppet_backend.rb from install of hiera-puppet-1.0.0-1.el6.noarch conflicts with file from package puppet-3.4.3-1.el6.noarch
  file /usr/lib/ruby/site_ruby/1.8/hiera/scope.rb from install of hiera-puppet-1.0.0-1.el6.noarch conflicts with file from package puppet-3.4.3-1.el6.noarch
  file /usr/lib/ruby/site_ruby/1.8/hiera_puppet.rb from install of hiera-puppet-1.0.0-1.el6.noarch conflicts with file from package puppet-3.4.3-1.el6.noarch
  file /usr/lib/ruby/site_ruby/1.8/puppet/parser/functions/hiera.rb from install of hiera-puppet-1.0.0-1.el6.noarch conflicts with file from package puppet-3.4.3-1.el6.noarch
  file /usr/lib/ruby/site_ruby/1.8/puppet/parser/functions/hiera_array.rb from install of hiera-puppet-1.0.0-1.el6.noarch conflicts with file from package puppet-3.4.3-1.el6.noarch
  file /usr/lib/ruby/site_ruby/1.8/puppet/parser/functions/hiera_hash.rb from install of hiera-puppet-1.0.0-1.el6.noarch conflicts with file from package puppet-3.4.3-1.el6.noarch
  file /usr/lib/ruby/site_ruby/1.8/puppet/parser/functions/hiera_include.rb from install of hiera-puppet-1.0.0-1.el6.noarch conflicts with file from package puppet-3.4.3-1.el6.noarch

Error Summary
-------------

That’s correct, there’s no Error Summary. This error appears to be because the hiera-puppet version has a conflict with puppet 3.4.2. Upgrading to 3.4.3 or higher resolved this issue.

The second, more difficult part, is creating a hiera.yaml file. You can take the easy way out by relying on the default file provided with Hiera with a single edit on the last line and the creation of a directory:

[root@puppet ~]# cat /etc/hiera.yaml
---
:backends:
  - yaml
:hierarchy:
  - defaults
  - "%{clientcert}"
  - "%{environment}"
  - global

:yaml:
# datadir is empty here, so hiera uses its defaults:
# - /var/lib/hiera on *nix
# - %CommonAppData%\PuppetLabs\hiera\var on Windows
# When specifying a datadir, make sure the directory exists.
  :datadir: /etc/puppet/data
[root@puppet puppet-tutorial]# mkdir /etc/puppet/data

In the long run, you’re going to need to do a bit more configuration. There are a lot of options here, you can define the hierarchy any way you want. The first match in the hierarchy is used, so the most specific settings should be found near the top, such as in the clientcert. and the least specific settings are found at the bottom, in global. If nothing matches, then puppet will move from Hiera to manifest files. This gives us the flexibility to move values from manifests to Hiera over time and not lose functionality or node definitions. Here’s an example of a very, very specific hierarchy:

:hierarchy:
  - environments/%{environment}/data/fqdn/%{fqdn}
  - environments/%{environment}/data/osfamily/%{osfamily}/%{lsbdistcodename}
  - environments/%{environment}/data/osfamily/%{osfamily}/%{lsbajdistrelease}
  - environments/%{environment}/data/osfamily/%{osfamily}/%{architecture}
  - environments/%{environment}/data/osfamily/%{osfamily}/common
  - environments/%{environment}/data/modules/%{cname}
  - environments/%{environment}/data/modules/%{caller_module_name}
  - environments/%{environment}/data/modules/%{module_name}
  - environments/%{environment}/data/common

It’s definitely up to you to define the hierarchy of your own setup. You probably don’t have enough information to make that decision now, but we’ll get to that soon enough. Just keep in mind that it’s top to bottom, first match wins, and manifests are a last resort until we come back to this.

What’s in these files? It depends on the backend specified in /etc/hiera.yaml. Since we’re using yaml, each file is YAML, has the suffix of “.yaml”, and is located in :datadir:. Let’s create the global.yaml file first. We’re going to define one key, puppetmaster, and the FQDN of our master will be the value.

[root@puppet ~]# cat > /etc/puppet/data/global.yaml
---
puppetmaster: 'puppet.nelson.va'

You can test that Hiera is working with the hiera command-line utility. Provide it a key and it will produce the matching value if it is defined and ‘nil’ otherwise:

[root@puppet ~]# hiera puppet
nil
[root@puppet ~]# hiera puppetmaster
puppet.nelson.va

There’s one last thing: the hiera command line program looks at /etc/hiera.yaml but puppet looks at $confdir/hiera.yaml. Check what $confdir evaluates to on your system and create a symlink between the two. If you skip this step, everything will look fine at the CLI but any manifests relying on hiera will fail to compile.

[root@puppet ~]# puppet config print confdir
/etc/puppet
[root@puppet ~]# ln -s /etc/hiera.yaml /etc/puppet/hiera.yaml

All we needed to do today is make sure Hiera is installed and working. We will get into the innards another day. There are other backends available – file, JSON, MySQL, and gpg – if you want to start looking ahead.

MCollective

There’s one more installable component that I want to cover, MCollective. However, the process is longer than what we already discussed today, so you’ll have to tune in next time for the completion of Installables.