Last week, we started using Hiera. We’re going to do a lot more today. First, we’ll add what we have to version control, then we’ll integrated it with r10k, and we’ll wrap up by migrating more content out of manifests into Hiera. Along the way we’ll explain how Hiera works. I also encourage you to review the Puppet Labs docs and the source code as needed.

NOTE: I have documented an improved workflow since this was written. I highly suggest following the newer repository layout, though this repository layout still works.

Version Control

Before we do anything else, we need to take the existing hiera data and put it in version control. Just like our modules and manifests, hiera data will be edited to match our feature implementation and to define the infrastructure in use. It is equally as important as our module configs. I’ve created a GitHub repository called hiera-tutorial and will reference that, but any git-based repository will work for our purposes. Create a directory for the local repo, add our existing content, and push that to the origin:

[rnelson0@puppet ~]$ cd git
[rnelson0@puppet git]$ mkdir hiera-tutorial
[rnelson0@puppet git]$ cd hiera-tutorial
[rnelson0@puppet hiera-tutorial]$ cp -r /etc/puppet/data/* ./
[rnelson0@puppet hiera-tutorial]$ ls
global.yaml  puppet_role
[rnelson0@puppet hiera-tutorial]$ git init
Initialized empty Git repository in /home/rnelson0/git/hiera/.git/
[rnelson0@puppet hiera-tutorial]$ git add .
[rnelson0@puppet hiera-tutorial]$ git commit -m 'Initial commit to hiera repo'
[master (root-commit) 4293c1c] Initial commit to hiera repo
 3 files changed, 8 insertions(+), 0 deletions(-)
 create mode 100644 global.yaml
 create mode 100644 puppet_role/puppet.yaml
[rnelson0@puppet hiera-tutorial]$ git remote add origin ssh://git@codecloud.web.att.com:7999/st_msspuppet/hiera.git
[rnelson0@puppet hiera-tutorial]$ git push origin master
Counting objects: 6, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (6/6), 451 bytes, done.
Total 6 (delta 0), reused 0 (delta 0)
To ssh://git@codecloud.web.att.com:7999/st_msspuppet/hiera.git
* [new branch]      master -> master

In a bit we’ll make a decision whether we want to use branches in our hiera data. If you decide to do that, create the production branch and push contents to it, then change the primary branch at GitHub and remove the defunct master branch. Even if you stick with a single branch, you can use this process to change the branch name, for example to data.

[rnelson0@puppet hiera-tutorial]$ git checkout -b production
Switched to a new branch 'production'
[rnelson0@puppet hiera-tutorial]$ git branch -d master
Deleted branch master (was 0e43028).
[rnelson0@puppet hiera-tutorial]$ git push origin production
Total 0 (delta 0), reused 0 (delta 0)
To git@github.com:rnelson0/hiera-tutorial.git
 * [new branch]      production -> production

[rnelson0@puppet hiera-tutorial]$ git push origin :master
To git@github.com:rnelson0/hiera-tutorial.git
 - [deleted]         master

With our existing content preserved, we can move on to the next step.

Integration with R10K.

Now we need to edit the r10k installation manifest. We have already configured r10k to populate environments with modules based off of the branches of our puppet and module repos. We will add a source for hiera, which will allow us to make changes to hiera via git and have the changes pushed around by r10k instead of manually copying files around ourselves. We will also be able to specify whether we have a monolithic hiera configuration or environment-specific configurations.

Because initial configuration of r10k was performed as root, move the r10k_installation.pp manifest from root’s home directory to ours, change permissions, and add the hiera location.

[rnelson0@puppet hiera-tutorial]$ cd
[rnelson0@puppet ~]$ sudo mv /root/r10k_installation.pp ./
[rnelson0@puppet ~]$ sudo chown rnelson0.rnelson0 r10k_installation.pp
[rnelson0@puppet ~]$ vi r10k_installation.pp

Here are the contents of the file. A source for hiera has been added – plus a comma after the existing puppet subsection – and the modulepath/manifestdir settings that were deprecated in Puppet 3.6.1 were removed. The new content is highlighted in bold:

class { 'r10k':
  version => '1.2.0',
  sources => {
    'puppet' => {
      'remote'  => 'https://github.com/rnelson0/puppet-tutorial.git',
      'basedir' => "${::settings::confdir}/environments",
      'prefix'  => false,
    },
    'hiera' => {
      'remote'  => 'https://github.com/rnelson0/hiera-tutorial.git',
      'basedir' => "${::settings::confdir}/data",
      'prefix'  => false,
    }
  },
  manage_modulepath => false,
}

Apply the manifest. You’ll see a small change to the hiera configuration file:

[rnelson0@puppet ~]$ sudo puppet apply r10k_installation.pp
Notice: Compiled catalog for puppet.nelson.va in environment production in 0.86 seconds
Notice: /Stage[main]/Main/Ini_setting[manifestdir]/ensure: created
Notice: /Stage[main]/R10k::Config/File[r10k.yaml]/content: content changed '{md5}3831ea2606d05e88804d647d56d2e12b' to '{md5}62730aa21170be02c455406043ef268e'
Notice: /Stage[main]/R10k::Config/Ini_setting[R10k Modulepath]/ensure: created
Notice: Finished catalog run in 0.57 seconds
[rnelson0@puppet ~]$ cat /etc/r10k.yaml
:cachedir: /var/cache/r10k
:sources:
  puppet:
    prefix: false
    basedir: /etc/puppet/environments
    remote: "https://github.com/rnelson0/puppet-tutorial.git"
  hiera:
    prefix: false
    basedir: /etc/puppet/data
    remote: "https://github.com/rnelson0/hiera-tutorial.git"

:purgedirs:
  - /etc/puppet/environments

We’ll also make a change to the datadir value in /etc/hiera.yaml (or /etc/puppet/hiera.yaml, as they should be symlinked). You have two choices here. The first example will allow us to create a hiera branch that matches our other feature branches, allowing a hiera configuration per environment. The disadvantage is that you MUST branch for each environment, otherwise catalog compilation failure will fail for those dynamic environments. The second examples, commented out, point directly to the production branch. This relies on defining the environment in /etc/hiera.yaml‘s :hierarchy: setting (and the commented-out line would be for a single data branch; there’s more than one way to do this). The advantage is that you don’t need to branch your hiera repo. Choose one of these settings and make it so. I’ve gone with the former in my public repo, but the last option at work.

  # One branch per environment
  :datadir: '/etc/puppet/data/%{environment}'

  # Make sure /etc/hiera.yaml's :hierarchy: includes "%{environment}" statements
  #:datadir: '/etc/puppet/data/production'
  # If you set the branch to 'data' you can tell r10k to use ${::settings::confdir} and this datadir
  #:datadir: '/etc/puppet/data'

Restart the puppetmaster service. Anytime you make changes to /etc/hiera.yaml, you will need to restart puppetmaster for the changes to take effect. You do NOT need to restart it when the contents of datadir are changed, however. Finally, re-run r10k and you’ll see the differences in the contents of /etc/puppet/data:

[rnelson0@puppet hiera-tutorial]$ ls /etc/puppet/data/puppet/role
server.yaml
[rnelson0@puppet hiera-tutorial]$ sudo r10k deploy environment -p
[rnelson0@puppet hiera-tutorial]$ ls /etc/puppet/data/puppet_role
ls: cannot access /etc/puppet/data/puppet_role: No such file or directory
[rnelson0@puppet hiera-tutorial]$ ls /etc/puppet/data/production/
.git/        global.yaml  puppet_role/

If everything went well, run puppet on the master and server01 and everything should work like it did before. Verify that before continuing. Some things to look for: the comma between the existing puppet and new hiera resource in the r10k manifest, that you did not add modulepath/manifestdir back to /etc/puppet/puppet.conf after migrating to environmentpath, and that you restarted the puppetmaster service.

Converting manifests

The next thing we need to do is convert our existing manifests/site.pp entries into hiera definitions. Last week, we converted the server role into a hiera definition. Let’s look at the definition for the puppet node:

node 'puppet.nelson.va' {
  include ::base
  notify { "Generated from our notify branch": }

  # PuppetDB
  include ::puppetdb
  include ::puppetdb::master::config

  # Hiera
  package { ['hiera', 'hiera-puppet']:
    ensure => present,
  }

  class { '::mcollective':
    client             => true,
    middleware         => true,
    middleware_hosts   => [ 'puppet.nelson.va' ],
    middleware_ssl     => true,
    securityprovider   => 'ssl',
    ssl_client_certs   => 'puppet:///modules/site_mcollective/client_certs',
    ssl_ca_cert        => 'puppet:///modules/site_mcollective/certs/ca.pem',
    ssl_server_public  => 'puppet:///modules/site_mcollective/certs/puppet.nelson.va.pem',
    ssl_server_private => 'puppet:///modules/site_mcollective/private_keys/puppet.nelson.va.pem',
  }

  user { 'root':
    ensure => present,
  } ->
  mcollective::user { 'root':
    homedir     => '/root',
    certificate => 'puppet:///modules/site_mcollective/client_certs/root.pem',
    private_key => 'puppet:///modules/site_mcollective/private_keys/root.pem',
  }

  mcollective::plugin { 'puppet':
    package => true,
  }
}

We need to create a few profiles and roles, starting with the profiles. There will be 5 of them – including a new profile that ensures the puppet-master package is installed and running and some firewall rules for it (we should have been tracking this earlier, but I forgot!). I’ve only included the comment section from the first file, but you should be sure to include the header in each file:

[rnelson0@puppet profile]$ cat manifests/puppetdb.pp
# == Class: profile::puppetdb
#
# PuppetDB profile
#
# === Parameters
#
# None
#
# === Variables
#
# None
#
# === Examples
#
#  include profile::puppetdb
#
# === Authors
#
# Rob Nelson <rnelson0@gmail.com>
#
# === Copyright
#
# Copyright 2014 Rob Nelson
#
class profile::puppetdb {
  include ::puppetdb
  include ::puppetdb::master::config
}
[rnelson0@puppet profile]$ cat manifests/hiera.pp
# Comments go here
class profile::hiera {
  package { ['hiera', 'hiera-puppet']:
    ensure => present,
  }
}
[rnelson0@puppet profile]$ cat manifests/mcollective/all.pp
# Comments go here
class profile::mcollective::all {
  class { '::mcollective':
    client             => true,
    middleware         => true,
    middleware_hosts   => [ 'puppet.nelson.va' ],
    middleware_ssl     => true,
    securityprovider   => 'ssl',
    ssl_client_certs   => 'puppet:///modules/site_mcollective/client_certs',
    ssl_ca_cert        => 'puppet:///modules/site_mcollective/certs/ca.pem',
    ssl_server_public  => 'puppet:///modules/site_mcollective/certs/puppet.nelson.va.pem',
    ssl_server_private => 'puppet:///modules/site_mcollective/private_keys/puppet.nelson.va.pem',
  }

  mcollective::plugin { 'puppet':
    package => true,
  }
}
[rnelson0@puppet profile]$ cat manifests/mcollective/users.pp
# Comments go here
class profile::mcollective::users {
  user { 'root':
    ensure => present,
  } ->
  mcollective::user { 'root':
    homedir     => '/root',
    certificate => 'puppet:///modules/site_mcollective/client_certs/root.pem',
    private_key => 'puppet:///modules/site_mcollective/private_keys/root.pem',
  }
}
[rnelson0@puppet profile]$ cat manifests/puppet_master.pp
# Comments go here
class profile::puppet_master {
  package {'puppet-server':
    ensure => present,
  }

  firewall { '100 allow agent checkins':
    dport  => 8140,
    proto  => tcp,
    action => accept,
  }

  firewall { '110 sinatra web hook':
    dport  => 80,
    proto  => tcp,
    action => accept,
  }
}

The corresponding role is very simple:

class role::puppet {
  include profile::base  # All roles should have the base profile
  include profile::puppet_master
  include profile::puppetdb
  include profile::hiera
  include profile::mcollective::users
  include profile::mcollective::all
}

We still have a definition for puppet in the site manifest. You can reduce the site.pp file to just a few lines now:

Package {
  allow_virtual => true,
}

node default {
  hiera_include('classes')
}

Commit/push changes and re-deploy with r10k. Whoops, we forgot to update hiera. That’s okay, now you know what the error message looks like when you skip this step:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item classes in
 any Hiera data file and no default supplied at /etc/puppet/environments/production/manifests/site.pp:6 on nod
e puppet.nelson.va

Hopefully, you remember how to fix this from last week. If not, navigate to your hiera repo’s puppet_role/ directory. Create a yaml file for the puppet puppet_role (or whatever puppet_role you decided upon for it) and add the role::puppet class to it:

[rnelson0@puppet puppet_role]$ cat > puppet.yaml
---
classes:
  role::puppet

Commit/push/r10k and run puppet again. You should see the firewall rules from profile::puppet_master and maybe an mcollective update; if you already had the rule in place from our earlier work then you may not see anything other than a successful checkin.

Defines – A Special Case

If you review the site.pp file, you’ll notice that it uses a function called hiera_include. This includes the defined classes, and each of those classes look up their parameters via hiera. However, there are some other hiera functions you may need. A common example is the case of defines with node-specific information. Above, in profile::mcollective::users, we use two defines, user and mcollective::user. As the values for these defines are the same for each node, we simply provided all the correct key/value pairs we needed in the module. When we need the values to be specific to a node (or environment, or network – anything more specific than “all”), we can’t populate the define’s attributes in the module.

Since we don’t have anything like that in our example setup, we’ll have to make something up. Let’s create a definition for a DHCP server, that’s probably helpful for a lab network. Here’s an example of a way to do that with profiles and the site manifest, which we will then convert to hiera/roles and profiles:

node /dhcp/ {
  include ::profile::dhcp_server

  ::dhcp::server::subnet { '10.0.0.0':
    broadcast   => '10.0.0.255',
    netmask     => '255.255.255.0',
    routers     => '10.0.0.1',
    range_begin => '10.0.0.100',
    range_end   => '10.0.0.150',
    dns_servers => ['10.0.0.1', '10.0.0.2'],
    domain_name => 'nelson.va',
    other_opts  => ['option ntp-servers 10.0.0.1'],
  }
}


[rnelson0@puppet profile]$ cat > manifests/dhcp_server.pp
class profile::dhcp_server {
  include mss::base
  include dhcp::server

  firewall { '100 Allow DHCP requests':
    proto  => 'udp',
    sport  => [67, 68],
    dport  => [67, 68],
    state  => 'NEW',
    action => 'accept',
  }
}

The define above is ::dhcp::server::subnet. You cannot hiera_include it, which means if you convert this to a role, every DHCP server will serve the same scope. If you’ve never had the pleasure of having two servers serving the same scope, I can guarantee you that you don’t want to! We’ll want to move that out of the module and into hiera, but how? We’ll use a combination of hiera_hash, which creates a hash from hiera data, and create_resources, which can instantiate resources, including defines, using a provided hash to populate the needed values. The PuppetLabs documentation shows a simple example of how this works with a flat manifest and a manifest with an array, but how would we go about doing this with hiera?

First, let’s take a look at where to put the data in hiera. If we look at the :hierarchy: portion of hiera.yaml, we’ve got a few options:

:hierarchy:
  - defaults
  - puppet_role/%{puppet_role}
  - "%{clientcert}"
  - "%{environment}"
  - global

The puppet_role will potentially apply to multiple nodes, so that’s out. Next is the clientcert value. Each node (and mcollective user!) has its own certificate, which can be seen by running puppet cert list –all on the master (optionally, use awk to only grab the important part):

[rnelson0@puppet puppet_role]$ sudo puppet cert list --all | awk '{print $2}'
"agent1.nelson.va"
"puppet.nelson.va"
"root"
"server01.nelson.va"

This should match the FQDN in most environments, but could also be the short hostname (i.e. agent1). This seems a likely choice for node-specific elements. You could also add to the hierarchy. Popular options are to provide a level that combines environment and clientcert/fqdn, as in “%{environment}/%{clientcert}”. I’ll assume the simple “%{clientcert}” level is being used with the value dhcp.nelson.va. In this yaml file, we need to create a hash called dhcp_subnet of all the values provided to the define, with a top-level hash key that matches the name of the define. In this case, that’s the subnet. All the other attributes are underneath this level. Let’s define the node yaml, plus the puppet_role yaml for dhcp:

[rnelson0@puppet hiera-tutorial]$ cat > dhcp.nelson.va.yaml
---
dhcp_subnet:
  '10.0.0.0':
    broadcast   : '10.0.0.255'
    netmask     : '255.255.255.0'
    routers     : '10.0.0.1'
    range_begin : '10.0.0.100'
    range_end   : '10.0.0.150'
    dns_servers :
      - '10.0.0.1'
      - '10.0.0.2'
    domain_name : 'nelson.va'
    other_opts  :
      - 'option ntp-servers 10.0.0.1'

[rnelson0@puppet hiera-tutorial]$ cat > puppet_role/dhcp.yaml
---
classes:
  role::dhcp

To glue everything together, we now need to create the dhcp role and import this information. We’ll use hiera_hash to import the above hash, then create_resources to instantiate a ::dhcp::server::subnet with the hash’s contents:

[rnelson0@puppet role]$ cat manifests/dhcp.pp
class role::dhcp {
  include profile::base  # All roles should have the base profile
  include profile::dhcp

  create_resources(::dhcp::server::subnet, hiera_hash('dhcp_subnet'))
}

Commit/push/r10k the changes. Since we haven’t created a DHCP node yet, you’ll have to deploy a VM from a template, as we did last week, give it a hostname of ‘dhcp’ (numbered instances are far less likely with DHCP servers, but ‘dhcp01’ works just as well – as long as the hiera yaml filename matches!), and checkin/sign/checkin with puppet. It should receive all the configuration changes required to now be a DHCP server in the network 10.0.0.0/24, assigning leases between 10.0.0.100-150.

Some other common uses with defines are to create users and manage apache vhost definitions. In both cases, you don’t want the module to contain the definitions, but hiera. This successfully abstracts the data away from the service definition and allows you to reuse your code very efficiently.

Summary

Building on last week’s roles, profiles, and hiera introduction, we’ve examined how to use version control and r10k to manage hiera, how to convert simple node definitions into roles managed via hiera, and how to provide node-specific configuration on top of the role’s classes. If you have any other node definitions in regular *.pp manifests, take the time to convert them all to hiera. From here on out, we’ll assume that’s where your data lies.

At this point, I’ll be taking a bit of a break from the blog for the summer. I’m happy to say that my wife has taken a new job and we will have closed on a new house around the time this article is published. We’ll be involved in moving and settling into our new house for a few weeks, but I’ll be bringing more puppet-ey goodness in the fall. This seems like a great time to roll up all the cahnges, so I’ve added a tag, v0.5, to all the repos associated with this series, which is a snapshot in time of the moment this article was finished (6/23/2014).

In the fall, we’ll start working on scaling up our setup. In the meantime, enjoy your ability to deploy new VMs and quickly apply policy to them. Have a great summer, everyone!