In 2014, we set up our puppet environment and we’ve spent the first half of 2015 improving the configuration. In that time, we installed hiera, were introduced to it through the role/profile pattern, focused on separating the data from the code and moving it into hiera, and most recently on an improved controlrepo that modified the hiera layout. We have been using hiera the whole time, and there’s a lot we can do to improve how we use it still.
Manage Hiera with Puppet
Our initial hiera.yaml was simple and static. With our improved controlrepo layout, the new hiera.yaml file is more dynamic. A problem still remains: we are configuring hiera manually! You may have a hiera.yaml in your controlrepo or even a bootstrap.pp file for your initial puppet master. We have also been managing the hiera package manually in profile::hiera. This addresses the problem in the short term but adds to our administrative overhead – anytime we update the hiera config, we need to do so in these files as well as on the master itself.
We can use the forge module hunner/hiera by Hunter Haugen to manage the hiera package and configuration at once. Add the latest version to your Puppetfile:
mod 'hunner/hiera', '1.2.0'
We can get rid of profile::hiera entirely and replace it with this class. Let’s update the spec tests first. Add the module to .fixtures.yml and test for the class in spec/classes/puppet_master_spec.rb.
# .fixtures.yml hiera: 'git://github.com/hunner/puppet-hiera' # spec/classes/puppet_master_spec.rb require 'spec_helper' describe 'profile::puppet_master', :type => :class do let (:facts) do { :osfamily => 'RedHat', :operatingsystemrelease => '6.5', :concat_basedir => '/dne/', } end context 'with defaults for all parameters' do it { is_expected.to create_class('profile::puppet_master') } it { is_expected.to contain_class('puppet::master') } it { is_expected.to contain_class('hiera') } it { is_expected.to contain_class('epel') } it { is_expected.to contain_class('r10k') } it { is_expected.to contain_class('r10k::webhook') } it { is_expected.to contain_class('r10k::webhook::config') } it { is_expected.to contain_firewall('100 allow agent checkins') } it { is_expected.to contain_firewall('110 zack-r10k web hook') } end end
Make sure the fixture loads with rake spec_prep. Run rake spec_standalone or rspec spec/classes/puppet_master_spec.rb and you should have your one failure.
[rnelson0@build01 profile]$ rspec spec/classes/puppet_master_spec.rb ..F...... Failures: 1) profile::puppet_master with defaults for all parameters should contain Class[hiera] Failure/Error: it { is_expected.to contain_class('hiera') } expected that the catalogue would contain Class[hiera] # ./spec/classes/puppet_master_spec.rb:14:in `block (3 levels) in ' Finished in 3.45 seconds 9 examples, 1 failure Failed examples: rspec ./spec/classes/puppet_master_spec.rb:14 # profile::puppet_master with defaults for all parameters should contain Class[hiera]
We’re ready to touch the puppet code now. Remove the profile from role::puppet. Add ::hiera to profile::puppet.
# role/manifests/puppet.pp class role::puppet { include profile::base # All roles should have the base profile include profile::puppet_master include profile::puppetdb } # profile/manifests/puppet_master.pp class profile::puppet_master { include epel class { '::puppet::master': storeconfigs => true, environments => 'directory', } include ::hiera ... }
The rspec test should pass now.
[rnelson0@build01 profile]$ rspec spec/classes/puppet_master_spec.rb ......... Finished in 3.19 seconds 9 examples, 0 failures
Hiera in Hiera
Great, we are including Hunter’s module. Next we need to provide it with some data before it wipes out our existing hiera.yaml. Hunter has a good example of using the hierarchy parameter and what you would expect the resulting configuration to look like, which we can use as a foundation for our config. We also need to set the datadir value. We need to add this to the yaml file for the puppet puppet_role, which will feed that data into hiera to spit out a hiera.yaml configuration.
Confused yet? We are using hiera’s own automatic parameter lookup (APL) to provide parameter values to the hiera class. To use APL, we get the values from the existing yaml backend. The hiera class then uses those values to build a hiera.yaml file and put it in the correct location. If this seems somewhat circular, you would be correct. In fact, we can never really drop our bootstrap setup (config or .pp file) entirely, but we do not need to keep that up to date with every change so long as on the first run, the puppet master can get the values it needs to create and install the latest configuration file.
Here’s the hierarchy parameter values, in yaml, that match my hiera.yaml. Adjust to match your needs:
hiera::hierarchy: - 'clientcert/%%{}{clientcert}' - 'puppet_role/%%{}{puppet_role}' - 'global' hiera::datadir: '/etc/puppet/environments/%%{}{environment}/hiera'
You’ll notice an oddity there: %%{}. This comes about because hiera itself will try and lookup anything beginning with a percent sign and followed by a character or brackets as a variable. This sequence looks odd but properly escapes the percent sign so it shows up in your resulting hiera data.
Applying the Hiera config
Since you have a webhook that works, git add your files, commit and push. In a few moments, your new branch should be available on the master.
[rnelson0@build01 controlrepo]$ git add Puppetfile dist/profile/.fixtures.yml dist/profile/manifests/puppet_master.pp dist/profile/spec/classes/puppet_master_spec.rb dist/role/manifests/puppet.pp hiera/puppet_role/puppet.yaml [rnelson0@build01 controlrepo]$ git push origin hierafy Total 0 (delta 0), reused 0 (delta 0) To git@github.com:rnelson0/controlrepo.git * [new branch] hierafy -> hierafy [rnelson0@puppet ~]$ ls /etc/puppet/environments/ hierafy production
The last pre-check is to ensure that each version of hiera.yaml, in /etc and /etc/puppet (or /etc/puppetlabs/puppet) are actual files and not symlinks. If they are, just remove the symlink(s) and copy the real file on top. Restart the master service after changing the contents.
[root@puppet ~]# ls -l /etc/hiera.yaml /etc/puppet/hiera.yaml lrwxrwxrwx. 1 root root 46 Apr 15 21:20 /etc/hiera.yaml -> /etc/puppet/environments/production/hiera.yaml lrwxrwxrwx. 1 root root 46 Apr 15 21:20 /etc/puppet/hiera.yaml -> /etc/puppet/environments/production/hiera.yaml [root@puppet ~]# rm /etc/hiera.yaml /etc/puppet/hiera.yaml rm: remove symbolic link `/etc/hiera.yaml'? y rm: remove symbolic link `/etc/puppet/hiera.yaml'? y [root@puppet ~]# cp /etc/puppet/environments/production/hiera.yaml /etc/hiera.yaml [root@puppet ~]# cp /etc/puppet/environments/production/hiera.yaml /etc/puppet/hiera.yaml [root@puppet ~]# service httpd restart Stopping httpd: [ OK ] Starting httpd: [Wed May 13 23:39:06 2015] [warn] module passenger_module is already loaded, skipping [ OK ]
Run the agent on the master against the new environment. You’ll notice that hiera.yaml changes, but you shouldn’t see any effective change if your hierarchy is correct – the ERB template will leave a few blank lines that you probably didn’t have.
Notice: /Stage[main]/Hiera/File[/etc/puppet/hiera.yaml]/content: --- /etc/puppet/hiera.yaml 2015-05-13 23:38:37.281909391 +0000 +++ /tmp/puppet-file20150513-5088-1ujh2vl-0 2015-05-13 23:44:12.576155616 +0000 @@ -1,10 +1,17 @@ +# managed by puppet --- :backends: - yaml +:logger: console :hierarchy: - - clientcert/%{clientcert} - - puppet_role/%{puppet_role} + - "clientcert/%{clientcert}" + - "puppet_role/%{puppet_role}" - global :yaml: - :datadir: /etc/puppet/environments/%{environment}/hiera + :datadir: /etc/puppet/environments/%{environment}/hiera + + + + + Info: Computing checksum on file /etc/puppet/hiera.yaml Info: /Stage[main]/Hiera/File[/etc/puppet/hiera.yaml]: Filebucketed /etc/puppet/hiera.yaml to puppet with sum 2fda98cdbf4cc3d88494da48d9ef249f Notice: /Stage[main]/Hiera/File[/etc/puppet/hiera.yaml]/content: content changed '{md5}2fda98cdbf4cc3d88494da48d9ef249f' to '{md5}91f21a60adfcaa6be125a7843d299598' Notice: /Stage[main]/Hiera/File[/etc/puppet/hiera.yaml]/owner: owner changed 'root' to 'puppet' Notice: /Stage[main]/Hiera/File[/etc/puppet/hiera.yaml]/group: group changed 'root' to 'puppet' Info: Computing checksum on file /etc/hiera.yaml Info: FileBucket got a duplicate file {md5}2fda98cdbf4cc3d88494da48d9ef249f Info: /Stage[main]/Hiera/File[/etc/hiera.yaml]: Filebucketed /etc/hiera.yaml to puppet with sum 2fda98cdbf4cc3d88494da48d9ef249f Notice: /Stage[main]/Hiera/File[/etc/hiera.yaml]/ensure: ensure changed 'file' to 'link' ... [rnelson0@puppet ~]$ sudo service httpd restart Stopping httpd: [ OK ] Starting httpd: [Wed May 13 23:47:51 2015] [warn] module passenger_module is already loaded, skipping [ OK ]
Now, test to make sure lookups still work. They really should not have broken but it’s best to verify now.
[rnelson0@puppet ~]$ sudo hiera hiera::hierarchy puppet_role=puppet environment=production nil [rnelson0@puppet ~]$ sudo hiera hiera::hierarchy puppet_role=puppet environment=hierafy ["clientcert/%{clientcert}", "puppet_role/%{puppet_role}", "global"]
Now, when you want to update your hierarchy, you can actually do it by editing your hiera data and pushing it out. The bootstrap will get you online so that hiera can be read, then the first run will setup your desired configuration. Less duplication, more dynamic configuration.
Hiera-fy the remaining manifests
While we are on the subject, we might as well hiera-fy the rest of our manifests. We will look at profile::puppet_master together as a model for how the other profiles should be updated. Here’s the existing contents of the file:
class profile::puppet_master { include ::epel class { '::puppet::master': storeconfigs => true, environments => 'directory', } include ::hiera class { 'r10k': version => '1.5.1', sources => { 'puppet' => { 'remote' => 'git@github.com:rnelson0/controlrepo.git', 'basedir' => "${::settings::confdir}/environments", 'prefix' => false, }, }, manage_modulepath => false } class {'r10k::webhook::config': use_mcollective => false, public_key_path => '/etc/mcollective/server_public.pem', # Mandatory for FOSS private_key_path => '/etc/mcollective/server_private.pem', # Mandatory for FOSS } class {'r10k::webhook': user => 'root', group => '0', } Class['r10k::webhook::config'] -> Class['r10k::webhook'] firewall { '100 allow agent checkins': dport => 8140, proto => tcp, action => accept, } firewall { '110 zack-r10k web hook': dport => 8088, proto => tcp, action => accept, } }
Anything that says class should be replaced with include and all the parameters should be moved to the hiera backend. The Class statements for ordering must remain. The defined types could be imported via hiera and create_resources, but I prefer the clarity of having firewall rules in the manifest. We should write the hiera first, before we forget the values. In hiera/puppet_role/puppet.yaml, add these statements:
puppet::master::storeconfigs: 'true' puppet::master::environments: 'directory' r10k::version: '1.5.1' r10k::sources: puppet: remote: 'git@github.com:rnelson0/controlrepo.git' basedir: '/etc/puppet/environments' prefix: false r10k::manage_modulepath: false r10k::webhook::config::use_mcollective: false r10k::webhook::config::public_key_path: '/etc/mcollective/server_public.pem' r10k::webhook::config::private_key_path: '/etc/mcollective/server_private.pem' r10k::webhook::user: 'root' r10k::webhook::group: 0
Now we can edit the profile:
class profile::puppet_master { include ::epel include ::puppet::master include ::hiera include ::r10k include ::r10k::webhook::config include ::r10k::webhook Class['r10k::webhook::config'] -> Class['r10k::webhook'] firewall { '100 allow agent checkins': dport => 8140, proto => tcp, action => accept, } firewall { '110 zack-r10k web hook': dport => 8088, proto => tcp, action => accept, } }
That’s a lot simpler! Commit your changes and push them upstream. If we did everything right, we can run puppet and there won’t be any effective changes. All the data comes from a different location, but has the same values. In my run, I got the semi-random re-ordering of hash keys in /etc/webhook.yaml but no changes for the values:
[rnelson0@puppet ~]$ sudo puppet agent -t --environment hierafy Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for puppet.nelson.va Info: Applying configuration version '1431562172' Notice: /Stage[main]/R10k::Webhook::Config/File[webhook.yaml]/content: --- /etc/webhook.yaml 2015-05-13 23:55:49.128737079 +0000 +++ /tmp/puppet-file20150514-8165-qpsfas-0 2015-05-14 00:09:53.704582787 +0000 @@ -9,13 +9,13 @@ enable_ssl: true pass: puppet use_mcollective: false + prefix: false public_key_path: /etc/mcollective/server_public.pem discovery_timeout: "10" - prefix: false - user: puppet private_key_path: /etc/mcollective/server_private.pem + user: puppet mco_logfile: /var/log/webhook/mco_output.log - use_mco_ruby: false prefix_command: "/bin/echo example" + use_mco_ruby: false certpath: /var/lib/peadmin/.mcollective.d access_logfile: /var/log/webhook/access.log Info: Computing checksum on file /etc/webhook.yaml Info: /Stage[main]/R10k::Webhook::Config/File[webhook.yaml]: Filebucketed /etc/webhook.yaml to puppet with sum 0aac2e252be8ce90e8ca3210e472dd75 Notice: /Stage[main]/R10k::Webhook::Config/File[webhook.yaml]/content: content changed '{md5}0aac2e252be8ce90e8ca3210e472dd75' to '{md5}cfb655aab0db12631ff913fb65f4b2af' Info: /Stage[main]/R10k::Webhook::Config/File[webhook.yaml]: Scheduling refresh of Service[webhook] Notice: /Stage[main]/R10k::Webhook/Service[webhook]: Triggered 'refresh' from 1 events Notice: /Stage[main]/Profile::Base/Exec[shosts.equiv]/returns: executed successfully Notice: Finished catalog run in 7.53 seconds
Take the time now to hiera-fy your remaining profiles before merging the branch into production.
Summary
Today we accomplished two positive outcomes. First, we hiera-fied our hiera setup, allowing us to make changes to the hierarchy very easily and without affecting our ability to bootstrap a master in a disaster recovery event. Second, we converted our existing profiles to use hiera. This creates a cleaner look for our profile classes and separates our data from our code. This is helpful if you need to engage support or share your profiles with a colleague or peer for assistance, there’s no chance you’ll give away passwords, URLs, etc. that you shouldn’t. We can further protect our data in hiera, which we will examine next time.
Notwithstanding the need for site wide data, what’s your opinion of MODULE specific data and how the technique you describe above can apply. We have been using the ripienaar-module_data module for a while and I really like the ability to localise both the module specific data and its hierarchy rather than try and find a single site wide hierarchy that caters for all usage patterns ??
Kind regards
Fraser.
Oh, I forgot mention, we operate in MASTERLESS mode (if that makes a significant difference )
I think I’ve seen the ripienaar module before but I have never used it. I tend to stick with the ::params module for any complex and variable defaults and I have not had the need to do that yet with a profile class. I can certainly see where it can get complex with a public module (puppetlabs-apache has a very complex params module, for instance). So I do not have an informed opinion yet, but I do see potential value in what you are doing. I will keep it in mind as I go forward though, thanks for pointing it out!