In 2014, we set up our puppet environment and we’ve spent the first half of 2015 improving the configuration. In that time, we installed hiera, were introduced to it through the role/profile pattern, focused on separating the data from the code and moving it into hiera, and most recently on an improved controlrepo that modified the hiera layout. We have been using hiera the whole time, and there’s a lot we can do to improve how we use it still.

Manage Hiera with Puppet

Our initial hiera.yaml was simple and static. With our improved controlrepo layout, the new hiera.yaml file is more dynamic. A problem still remains: we are configuring hiera manually! You may have a hiera.yaml in your controlrepo or even a bootstrap.pp file for your initial puppet master. We have also been managing the hiera package manually in profile::hiera. This addresses the problem in the short term but adds to our administrative overhead – anytime we update the hiera config, we need to do so in these files as well as on the master itself.

We can use the forge module hunner/hiera by Hunter Haugen to manage the hiera package and configuration at once. Add the latest version to your Puppetfile:

mod 'hunner/hiera', '1.2.0'

We can get rid of profile::hiera entirely and replace it with this class. Let’s update the spec tests first. Add the module to .fixtures.yml and test for the class in spec/classes/puppet_master_spec.rb.

# .fixtures.yml
    hiera: 'git://github.com/hunner/puppet-hiera'

# spec/classes/puppet_master_spec.rb
require 'spec_helper'
describe 'profile::puppet_master', :type => :class do
  let (:facts) do
  {
    :osfamily               => 'RedHat',
    :operatingsystemrelease => '6.5',
    :concat_basedir         => '/dne/',
  }
  end

  context 'with defaults for all parameters' do
    it { is_expected.to create_class('profile::puppet_master') }
    it { is_expected.to contain_class('puppet::master') }
    it { is_expected.to contain_class('hiera') }
    it { is_expected.to contain_class('epel') }
    it { is_expected.to contain_class('r10k') }
    it { is_expected.to contain_class('r10k::webhook') }
    it { is_expected.to contain_class('r10k::webhook::config') }
    it { is_expected.to contain_firewall('100 allow agent checkins') }
    it { is_expected.to contain_firewall('110 zack-r10k web hook') }
  end
end

Make sure the fixture loads with rake spec_prep. Run rake spec_standalone or rspec spec/classes/puppet_master_spec.rb and you should have your one failure.

[rnelson0@build01 profile]$ rspec spec/classes/puppet_master_spec.rb
..F......

Failures:

  1) profile::puppet_master with defaults for all parameters should contain Class[hiera]
     Failure/Error: it { is_expected.to contain_class('hiera') }
       expected that the catalogue would contain Class[hiera]
     # ./spec/classes/puppet_master_spec.rb:14:in `block (3 levels) in '

Finished in 3.45 seconds
9 examples, 1 failure

Failed examples:

rspec ./spec/classes/puppet_master_spec.rb:14 # profile::puppet_master with defaults for all parameters should contain Class[hiera]

We’re ready to touch the puppet code now. Remove the profile from role::puppet. Add ::hiera to profile::puppet.

# role/manifests/puppet.pp
class role::puppet {
  include profile::base  # All roles should have the base profile
  include profile::puppet_master
  include profile::puppetdb
}

# profile/manifests/puppet_master.pp
class profile::puppet_master {
  include epel
  class { '::puppet::master':
    storeconfigs => true,
    environments => 'directory',
  }

  include ::hiera

  ...
}

The rspec test should pass now.

[rnelson0@build01 profile]$ rspec spec/classes/puppet_master_spec.rb
.........

Finished in 3.19 seconds
9 examples, 0 failures

Hiera in Hiera

Great, we are including Hunter’s module. Next we need to provide it with some data before it wipes out our existing hiera.yaml. Hunter has a good example of using the hierarchy parameter and what you would expect the resulting configuration to look like, which we can use as a foundation for our config. We also need to set the datadir value. We need to add this to the yaml file for the puppet puppet_role, which will feed that data into hiera to spit out a hiera.yaml configuration.

Confused yet? We are using hiera’s own automatic parameter lookup (APL) to provide parameter values to the hiera class. To use APL, we get the values from the existing yaml backend. The hiera class then uses those values to build a hiera.yaml file and put it in the correct location. If this seems somewhat circular, you would be correct. In fact, we can never really drop our bootstrap setup (config or .pp file) entirely, but we do not need to keep that up to date with every change so long as on the first run, the puppet master can get the values it needs to create and install the latest configuration file.

Here’s the hierarchy parameter values, in yaml, that match my hiera.yaml. Adjust to match your needs:

hiera::hierarchy:
    - 'clientcert/%%{}{clientcert}'
    - 'puppet_role/%%{}{puppet_role}'
    - 'global'
hiera::datadir: '/etc/puppet/environments/%%{}{environment}/hiera'

You’ll notice an oddity there: %%{}. This comes about because hiera itself will try and lookup anything beginning with a percent sign and followed by a character or brackets as a variable. This sequence looks odd but properly escapes the percent sign so it shows up in your resulting hiera data.

Applying the Hiera config

Since you have a webhook that works, git add your files, commit and push. In a few moments, your new branch should be available on the master.

[rnelson0@build01 controlrepo]$ git add Puppetfile dist/profile/.fixtures.yml dist/profile/manifests/puppet_master.pp dist/profile/spec/classes/puppet_master_spec.rb dist/role/manifests/puppet.pp hiera/puppet_role/puppet.yaml
[rnelson0@build01 controlrepo]$ git push origin hierafy
Total 0 (delta 0), reused 0 (delta 0)
To git@github.com:rnelson0/controlrepo.git
 * [new branch]      hierafy -> hierafy

[rnelson0@puppet ~]$ ls /etc/puppet/environments/
hierafy  production

The last pre-check is to ensure that each version of hiera.yaml, in /etc and /etc/puppet (or /etc/puppetlabs/puppet) are actual files and not symlinks. If they are, just remove the symlink(s) and copy the real file on top. Restart the master service after changing the contents.

[root@puppet ~]# ls -l /etc/hiera.yaml /etc/puppet/hiera.yaml
lrwxrwxrwx. 1 root root 46 Apr 15 21:20 /etc/hiera.yaml -> /etc/puppet/environments/production/hiera.yaml
lrwxrwxrwx. 1 root root 46 Apr 15 21:20 /etc/puppet/hiera.yaml -> /etc/puppet/environments/production/hiera.yaml
[root@puppet ~]# rm /etc/hiera.yaml /etc/puppet/hiera.yaml
rm: remove symbolic link `/etc/hiera.yaml'? y
rm: remove symbolic link `/etc/puppet/hiera.yaml'? y
[root@puppet ~]# cp /etc/puppet/environments/production/hiera.yaml /etc/hiera.yaml
[root@puppet ~]# cp /etc/puppet/environments/production/hiera.yaml /etc/puppet/hiera.yaml
[root@puppet ~]# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd: [Wed May 13 23:39:06 2015] [warn] module passenger_module is already loaded, skipping
                                                           [  OK  ]

Run the agent on the master against the new environment. You’ll notice that hiera.yaml changes, but you shouldn’t see any effective change if your hierarchy is correct – the ERB template will leave a few blank lines that you probably didn’t have.

Notice: /Stage[main]/Hiera/File[/etc/puppet/hiera.yaml]/content:
--- /etc/puppet/hiera.yaml      2015-05-13 23:38:37.281909391 +0000
+++ /tmp/puppet-file20150513-5088-1ujh2vl-0     2015-05-13 23:44:12.576155616 +0000
@@ -1,10 +1,17 @@
+# managed by puppet
 ---
 :backends:
   - yaml
+:logger: console
 :hierarchy:
-  - clientcert/%{clientcert}
-  - puppet_role/%{puppet_role}
+  - "clientcert/%{clientcert}"
+  - "puppet_role/%{puppet_role}"
   - global

 :yaml:
-  :datadir: /etc/puppet/environments/%{environment}/hiera
+   :datadir: /etc/puppet/environments/%{environment}/hiera
+
+
+
+
+

Info: Computing checksum on file /etc/puppet/hiera.yaml
Info: /Stage[main]/Hiera/File[/etc/puppet/hiera.yaml]: Filebucketed /etc/puppet/hiera.yaml to puppet with sum 2fda98cdbf4cc3d88494da48d9ef249f
Notice: /Stage[main]/Hiera/File[/etc/puppet/hiera.yaml]/content: content changed '{md5}2fda98cdbf4cc3d88494da48d9ef249f' to '{md5}91f21a60adfcaa6be125a7843d299598'
Notice: /Stage[main]/Hiera/File[/etc/puppet/hiera.yaml]/owner: owner changed 'root' to 'puppet'
Notice: /Stage[main]/Hiera/File[/etc/puppet/hiera.yaml]/group: group changed 'root' to 'puppet'
Info: Computing checksum on file /etc/hiera.yaml
Info: FileBucket got a duplicate file {md5}2fda98cdbf4cc3d88494da48d9ef249f
Info: /Stage[main]/Hiera/File[/etc/hiera.yaml]: Filebucketed /etc/hiera.yaml to puppet with sum 2fda98cdbf4cc3d88494da48d9ef249f
Notice: /Stage[main]/Hiera/File[/etc/hiera.yaml]/ensure: ensure changed 'file' to 'link'
...
[rnelson0@puppet ~]$ sudo service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd: [Wed May 13 23:47:51 2015] [warn] module passenger_module is already loaded, skipping
                                                           [  OK  ]

Now, test to make sure lookups still work. They really should not have broken but it’s best to verify now.

[rnelson0@puppet ~]$ sudo hiera hiera::hierarchy puppet_role=puppet environment=production
nil
[rnelson0@puppet ~]$ sudo hiera hiera::hierarchy puppet_role=puppet environment=hierafy
["clientcert/%{clientcert}", "puppet_role/%{puppet_role}", "global"]

Now, when you want to update your hierarchy, you can actually do it by editing your hiera data and pushing it out. The bootstrap will get you online so that hiera can be read, then the first run will setup your desired configuration. Less duplication, more dynamic configuration.

Hiera-fy the remaining manifests

While we are on the subject, we might as well hiera-fy the rest of our manifests. We will look at profile::puppet_master together as a model for how the other profiles should be updated. Here’s the existing contents of the file:

class profile::puppet_master {
  include ::epel
  class { '::puppet::master':
    storeconfigs => true,
    environments => 'directory',
  }

  include ::hiera

  class { 'r10k':
    version => '1.5.1',
    sources => {
      'puppet' => {
        'remote'  => 'git@github.com:rnelson0/controlrepo.git',
        'basedir' => "${::settings::confdir}/environments",
        'prefix'  => false,
      },
    },
    manage_modulepath => false
  }

  class {'r10k::webhook::config':
    use_mcollective => false,
    public_key_path  => '/etc/mcollective/server_public.pem',  # Mandatory for FOSS
    private_key_path => '/etc/mcollective/server_private.pem', # Mandatory for FOSS
  }

  class {'r10k::webhook':
    user    => 'root',
    group   => '0',
  }
  Class['r10k::webhook::config'] -> Class['r10k::webhook']

  firewall { '100 allow agent checkins':
    dport  => 8140,
    proto  => tcp,
    action => accept,
  }

  firewall { '110 zack-r10k web hook':
    dport  => 8088,
    proto  => tcp,
    action => accept,
  }
}

Anything that says class should be replaced with include and all the parameters should be moved to the hiera backend. The Class statements for ordering must remain. The defined types could be imported via hiera and create_resources, but I prefer the clarity of having firewall rules in the manifest. We should write the hiera first, before we forget the values. In hiera/puppet_role/puppet.yaml, add these statements:

puppet::master::storeconfigs: 'true'
puppet::master::environments: 'directory'
r10k::version: '1.5.1'
r10k::sources:
  puppet:
    remote: 'git@github.com:rnelson0/controlrepo.git'
    basedir: '/etc/puppet/environments'
    prefix: false
r10k::manage_modulepath: false
r10k::webhook::config::use_mcollective: false
r10k::webhook::config::public_key_path: '/etc/mcollective/server_public.pem'
r10k::webhook::config::private_key_path: '/etc/mcollective/server_private.pem'
r10k::webhook::user: 'root'
r10k::webhook::group: 0

Now we can edit the profile:

class profile::puppet_master {
  include ::epel
  include ::puppet::master

  include ::hiera

  include ::r10k
  include ::r10k::webhook::config
  include ::r10k::webhook
  Class['r10k::webhook::config'] -> Class['r10k::webhook']

  firewall { '100 allow agent checkins':
    dport  => 8140,
    proto  => tcp,
    action => accept,
  }

  firewall { '110 zack-r10k web hook':
    dport  => 8088,
    proto  => tcp,
    action => accept,
  }
}

That’s a lot simpler! Commit your changes and push them upstream. If we did everything right, we can run puppet and there won’t be any effective changes. All the data comes from a different location, but has the same values. In my run, I got the semi-random re-ordering of hash keys in /etc/webhook.yaml but no changes for the values:

[rnelson0@puppet ~]$ sudo puppet agent -t --environment hierafy
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for puppet.nelson.va
Info: Applying configuration version '1431562172'
Notice: /Stage[main]/R10k::Webhook::Config/File[webhook.yaml]/content:
--- /etc/webhook.yaml   2015-05-13 23:55:49.128737079 +0000
+++ /tmp/puppet-file20150514-8165-qpsfas-0      2015-05-14 00:09:53.704582787 +0000
@@ -9,13 +9,13 @@
   enable_ssl: true
   pass: puppet
   use_mcollective: false
+  prefix: false
   public_key_path: /etc/mcollective/server_public.pem
   discovery_timeout: "10"
-  prefix: false
-  user: puppet
   private_key_path: /etc/mcollective/server_private.pem
+  user: puppet
   mco_logfile: /var/log/webhook/mco_output.log
-  use_mco_ruby: false
   prefix_command: "/bin/echo example"
+  use_mco_ruby: false
   certpath: /var/lib/peadmin/.mcollective.d
   access_logfile: /var/log/webhook/access.log

Info: Computing checksum on file /etc/webhook.yaml
Info: /Stage[main]/R10k::Webhook::Config/File[webhook.yaml]: Filebucketed /etc/webhook.yaml to puppet with sum 0aac2e252be8ce90e8ca3210e472dd75
Notice: /Stage[main]/R10k::Webhook::Config/File[webhook.yaml]/content: content changed '{md5}0aac2e252be8ce90e8ca3210e472dd75' to '{md5}cfb655aab0db12631ff913fb65f4b2af'
Info: /Stage[main]/R10k::Webhook::Config/File[webhook.yaml]: Scheduling refresh of Service[webhook]
Notice: /Stage[main]/R10k::Webhook/Service[webhook]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Profile::Base/Exec[shosts.equiv]/returns: executed successfully
Notice: Finished catalog run in 7.53 seconds

Take the time now to hiera-fy your remaining profiles before merging the branch into production.

Summary

Today we accomplished two positive outcomes. First, we hiera-fied our hiera setup, allowing us to make changes to the hierarchy very easily and without affecting our ability to bootstrap a master in a disaster recovery event. Second, we converted our existing profiles to use hiera. This creates a cleaner look for our profile classes and separates our data from our code. This is helpful if you need to engage support or share your profiles with a colleague or peer for assistance, there’s no chance you’ll give away passwords, URLs, etc. that you shouldn’t. We can further protect our data in hiera, which we will examine next time.