Standard

Posted by

Posted on

April 15, 2015

Posted under

DevOps, git, Puppet

Comments

8 Comments

Improved r10k deployment patterns

In previous articles, I’ve written a lot about r10k (again, again, and again), the role/profile pattern, and hiera (refactoring modules and rspec test data). I have kept each of these in a separate repository (to wit: controlrepo, role, profile, and hiera). This can also make for an awkward workflow. On the other hand, there is great separation between the various components. In some shops, granular permissions are required: the Puppet admins have access to the controlrepo and all developers have access to role/profile/hiera repos. There may even be multiple repos for different orgs. If you have a great reason to keep your repositories separate, you should continue to do so. If not, let’s take a look at how we can improve our r10k workflow by combining at least these four repositories into a single controlrepo.

Starting Point

To ensure we are all on the same page, here are the relevant portions of my Puppetfile:

# Modules from Github
mod 'custom_facts',
  :git => 'git@github.com:rnelson0/rnelson0-custom_facts'

mod 'role',
  :git => 'git@github.com:rnelson0/rnelson0-role'

mod 'profile',
  :git => 'git@github.com:rnelson0/rnelson0-profile'

mod 'home_config',
  :git => 'git@github.com:rnelson0/home_config.git'

mod 'linuxfw',
  :git => 'git@github.com:rnelson0/linuxfw'

In this case, I have five repositories for five modules. When I want to modify a role/profile, I have to:

  • Branch rnelson0-profile
  • Branch rnelson0-role
  • Branch home_config (optional)
  • Branch controlrepo
  • Branch hiera
  • Add ‘:ref => ‘ to each module in the Puppetfile
  • Go back to the Puppetfile and add a trailing comma on the ‘:git’ lines I modified (I forget about 50% of the time!)
  • Make and commit all the changes required.
  • Push all four (or five) branches to origin
  • Run r10k to deploy the new environment
  • Test/fix/push/r10k loop till everything is okay
  • Submit a PR on rnelson0-profile, merge it
  • Submit a PR on rnelson0-role, merge it
  • Submit a PR on home_config, merge it (optional)
  • Submit a PR on hiera, merge it
  • Destroy the branch on the control repo
  • Run r10k to deploy to production environment and delete the feature branch’s environment

Whew, that’s a lot of steps! There are, thankfully, webhooks and tools like reaktor that can help automate some of the pain away (ignoring testing – how you test is up to you!). You still have to create some branches, create some PRs, and merge the PRs, which is over 50% of the workflow steps. I’ve highlighted those steps in bold that cannot be easily or fully automated away.

A New Hope

Most of the steps we cannot easily automate above are caused by having multiple repos. If those repos were combined, a majority of the steps would simply go away. By combining the role, profile, and hiera data into the controlrepo, our workflow would be much simpler:

  • Branch controlrepo
  • Branch home_config (optional)
  • Make and commit all the changes required.
  • Push just one (or two) branches to origin
  • Run r10k to deploy the new environment
  • Test/fix/push/r10k loop till everything is okay
  • Submit a PR on the controlrepo, merge it
  • Submit a PR on home_config, merge it (optional)
  • Run r10k to deploy to production environment and delete the feature branch’s environment

That’s a reduction from 17 steps to just 9, nearly 50%! With webhooks, which I’ll explore in a future article, the two non-bolded steps can even be automated, reducing this to 7 manual steps. We could probably reduce it to 7 steps if we combined the home_config module with the control repo, but you will still have plenty of valid reasons for separate modules (shared modules between puppet implementations, separating it out for a forge module, or just the size of the repository) so we will use it as an example of a separate repo that you don’t want integrated with the controlrepo. Of course, the question is, how do we get from where we are now to where we want to be?

Stage 1: New Directory Layout and Contents

To get started, create a new branch on your existing controlrepo.

[rnelson0@build controlrepo]$ git checkout -b simplified_layout
Switched to a new branch 'simplified_layout'

If you don’t have an existing controlrepo, create a brand new git repository, change the default branch to production, and start populating the Puppetfile. There are a number of other articles in this series that can help you with this. Return here after you have a working Puppetfile, we’ll handle the rest of the contents in a moment. We will create three directories for hiera, the role module, and the profile module. Hiera is at the top level of the controlrepo and the modules go under the dist directory:

[rnelson0@build controlrepo]$ mkdir dist
[rnelson0@build controlrepo]$ mkdir hiera dist/role dist/profile

These new module directories are not in the basemodulepath, however.  That can be added via a single stanza in a new file called environment.conf. This file is read by the puppet master in each environment. Any changes you make here only apply to this environment – in our case, the ‘simplified_layout’ environment.

[rnelson0@build controlrepo]$ cat > environment.conf
modulepath = dist:modules:$basemodulepath
[rnelson0@build controlrepo]$ git add environment.conf

We also need to adjust hiera to look at this, which we will do via a hiera.yaml file in the controlrepo. We will have to symlink to this later, as the master always looks at the contents /etc/{puppetlabs/}puppet/hiera.yaml for its config. This should look just like your existing config file, with the exception of the :datadir: path. The value is /etc/puppet/environments/%{environment}/hiera if you use Puppet FOSS, otherwise it is /etc/puppetlabs/puppet/environments/%{environment}/hiera.

[rnelson0@build controlrepo]$ cat > hiera.yaml
---
:backends:
  - yaml
:hierarchy:
  - clientcert/%{clientcert}
  - puppet_role/%{puppet_role}
  - global

:yaml:
  :datadir: /etc/puppet/environments/%{environment}/hiera
[rnelson0@build controlrepo]$ git add hiera.yaml

You can now start over with new modules by using puppet module generate author-role –modulepath=dist/role and answering the questions, but you probably want to preserve your existing role and profile data. Change into your role/profile directory and checkout the branch you want, then use git archive piped to tar to copy the branch contents only, leaving out the .git directory but preserving other git files such as .gitignore. Do not simply throw the file contents of one repo inside another repo, the .git directories will cause problems. Repeat the process with hiera.

[rnelson0@build role]$ pwd
/home/rnelson0/puppet/role
[rnelson0@build role]$ git archive master | tar -x -C ~/puppet/controlrepo/dist/role/
[rnelson0@build role]$ cd ../profile
[rnelson0@build profile]$ git archive master | tar -x -C ~/puppet/controlrepo/dist/profile/
[rnelson0@build profile]$ ls -la ~/puppet/controlrepo/dist/role/ ~/puppet/controlrepo/dist/profile/
/home/rnelson0/puppet/controlrepo/dist/profile/:
total 44
drwxrwxr-x. 6 rnelson0 rnelson0 4096 Apr 15 14:11 .
drwxrwxr-x. 4 rnelson0 rnelson0 4096 Apr 15 13:42 ..
-rw-rw-r--. 1 rnelson0 rnelson0  673 Mar 19 00:10 .fixtures.yml
-rw-rw-r--. 1 rnelson0 rnelson0   15 Mar 19 00:10 .gitignore
drwxrwxr-x. 3 rnelson0 rnelson0 4096 Mar 19 00:10 manifests
-rw-rw-r--. 1 rnelson0 rnelson0 1100 Mar 19 00:10 metadata.json
-rw-rw-r--. 1 rnelson0 rnelson0  633 Mar 19 00:10 Rakefile
-rw-rw-r--. 1 rnelson0 rnelson0 2915 Mar 19 00:10 README.md
drwxrwxr-x. 4 rnelson0 rnelson0 4096 Mar 19 00:10 spec
drwxrwxr-x. 3 rnelson0 rnelson0 4096 Mar 19 00:10 templates
drwxrwxr-x. 2 rnelson0 rnelson0 4096 Mar 19 00:10 tests

/home/rnelson0/puppet/controlrepo/dist/role/:
total 36
drwxrwxr-x. 5 rnelson0 rnelson0 4096 Apr 15 14:11 .
drwxrwxr-x. 4 rnelson0 rnelson0 4096 Apr 15 13:42 ..
-rw-rw-r--. 1 rnelson0 rnelson0   15 Feb  2 19:12 .gitignore
drwxrwxr-x. 2 rnelson0 rnelson0 4096 Feb  2 19:12 manifests
-rw-rw-r--. 1 rnelson0 rnelson0  449 Feb  2 19:12 metadata.json
-rw-rw-r--. 1 rnelson0 rnelson0  633 Feb  2 19:12 Rakefile
-rw-rw-r--. 1 rnelson0 rnelson0 2891 Feb  2 19:12 README.md
drwxrwxr-x. 3 rnelson0 rnelson0 4096 Feb  2 19:12 spec
drwxrwxr-x. 2 rnelson0 rnelson0 4096 Feb  2 19:12 tests
[rnelson0@build profile]$ cd ../hiera
[rnelson0@build hiera]$ git archive data | tar -x -C ~/puppet/controlrepo/hiera
[rnelson0@build hiera]$ ls -l ~/puppet/controlrepo/hiera
total 12
drwxrwxr-x. 2 rnelson0 rnelson0 4096 Mar 16 13:20 clientcert
-rw-rw-r--. 1 rnelson0 rnelson0  952 Mar 16 13:20 global.yaml
drwxrwxr-x. 2 rnelson0 rnelson0 4096 Mar 16 13:20 puppet_role
[rnelson0@build hiera]$ cd ~/puppet/controlrepo
[rnelson0@build controlrepo]$ git add dist hiera

Lastly, remove the role and profile mods from the Puppetfile and the hiera source from r10k_installation.pp. If you do not use the latter file, you’ll want to modify /etc/r10k.yaml directly on your puppet master to remove the hiera source. I am also removing my custom_facts and  linuxfw modulse as I will add them to the profile module shortly.

[rnelson0@build controlrepo]$ git diff Puppetfile
diff --git a/Puppetfile b/Puppetfile
index b14f21a..82a82fc 100644
--- a/Puppetfile
+++ b/Puppetfile
@@ -46,17 +46,5 @@ mod 'zack/r10k', '1.0.2'
 mod 'zack/r10k', '1.0.2'

 # Modules from Github
-mod 'custom_facts',
-  :git => 'git@github.com:rnelson0/rnelson0-custom_facts'
-
-mod 'role',
-  :git => 'git@github.com:rnelson0/rnelson0-role'
-
-mod 'profile',
-  :git => 'git@github.com:rnelson0/rnelson0-profile'
-
 mod 'home_config',
   :git => 'git@github.com:rnelson0/home_config.git'
-
-mod 'linuxfw',
-  :git => 'git@github.com:rnelson0/linuxfw'


[rnelson0@build controlrepo]$ git diff r10k_installation.pp
diff --git a/r10k_installation.pp b/r10k_installation.pp
index 1703c3f..50708b1 100644
--- a/r10k_installation.pp
+++ b/r10k_installation.pp
@@ -6,11 +6,6 @@ class { 'r10k':
       'basedir' => "${::settings::confdir}/environments",
       'prefix'  => false,
     },
-    'hiera' => {
-      'remote'  => 'git@github.com:rnelson0/hiera_home.git',
-      'basedir' => "/etc/puppet/hiera",
-      'prefix'  => false,
-    }
   },
   manage_modulepath => false
 }

We’re going to need to do more work, but let’s commit what we have. I do like to keep a tidy commit history that contains discreet, atomic commits that are entirely usable, but this is a multi-stage process and we’re done with this stage. We can always rebase and squash the commits later. You also want to make sure that you have your pre-commit hooks in place before you do this commit, especially if you just created a new controlrepo.

[rnelson0@build controlrepo]$ git commit -m 'Simplifying controlrepo, stage 1: add role/profile/hiera'
[simplified_layout d46c386] Simplifying controlrepo, stage 1: add role/profile/hiera
 65 files changed, 1579 insertions(+), 14 deletions(-)
...

Here’s what the new and improved layout should look like:

$ tree -d -L 2
.
├── dist
│   ├── profile
│   └── role
├── hiera
│   ├── clientcert
│   └── puppet_role
├── hooks
└── manifests

8 directories

Stage 2: Merge defunct modules into the profile module

We have a few modifications to make to the profile module since we removed custom_facts and linuxfw from the Puppetfile. If you are not using custom facts or firewall rules, this stage can serve as a template for other modules you merge into profile. Otherwise, you can skip this section. Change into the profile directory and create a linuxfw directory. Copy in the init.pp file from the linuxfw module as linuxfw.pp and copy the {pre,post}.pp files into the linuxfw directory, like so:

[rnelson0@build profile]$ mkdir manifests/linuxfw
[rnelson0@build profile]$ cp ~/puppet/linuxfw/manifests/init.pp manifests/linuxfw.pp
[rnelson0@build profile]$ cp ~/puppet/linuxfw/manifests/{pre,post}.pp manifests/linuxfw/

These files will need profile:: prepended to the class names. Go ahead and update the class name and any other documentation tweaks required, like copyright date. In the base profile, the reference to ::linuxfw will need to be changed to ::profile::linuxfw. Here are the diffs:

[rnelson0@build profile]$ diff manifests/linuxfw/pre.pp ~/puppet/linuxfw/manifests/pre.pp
2c2
< class profile::linuxfw::pre { --- > class linuxfw::pre {

[rnelson0@build profile]$ diff manifests/linuxfw/post.pp ~/puppet/linuxfw/manifests/post.pp
2c2
< class profile::linuxfw::post { --- > class linuxfw::post {

[rnelson0@build profile]$ diff manifests/linuxfw.pp ~/puppet/linuxfw/manifests/init.pp
17c17
< class profile::linuxfw { --- > class linuxfw {
23,24c23,24
<     before  => Class['profile::linuxfw::post'],
<     require => Class['profile::linuxfw::pre'],
---
>     before  => Class['linuxfw::post'],
>     require => Class['linuxfw::pre'],
27c27
<   include ::profile::linuxfw::pre, ::profile::linuxfw::post --- >   include ::linuxfw::pre ::linuxfw::post

[rnelson0@build profile]$ git diff manifests/base.pp
diff --git a/dist/profile/manifests/base.pp b/dist/profile/manifests/base.pp
index 4ff7dc8..9fbf442 100644
--- a/dist/profile/manifests/base.pp
+++ b/dist/profile/manifests/base.pp
@@ -12,7 +12,7 @@
 #
 class profile::base {
   # Base firewall policy
-  include ::linuxfw
+  include ::profile::linuxfw

   # SSH server and client
   class { '::ssh::server':

Migrating the custom fact is very simple as the file requires no changes, just copy it into lib/facter. I’m taking the moment to rename the ruby file to match the fact name of puppet_role as well.

[rnelson0@build profile]$ mkdir -p lib/facter
[rnelson0@build profile]$ cp ~/puppet/custom_facts/lib/facter/roles.rb lib/facter/puppet_role.rb

If you have any other modules you want to be part of profile, or added to the dist directory, do so now. Commit and push your changes when you are done.

[rnelson0@build controlrepo]$ git commit -m 'Simplifying controlrepo, stage 2: Merge custom_facts and linuxfw into profile'
[simplified_layout 8c57cac] Simplifying controlrepo, stage 2: Merge custom_facts and linuxfw into profile
 5 files changed, 99 insertions(+), 1 deletions(-)
...

Stage 3: Correcting rspec failures

We’re almost there. Before we deploy this environment and expect it to do anything, we need to make sure rspec works fine. If you do not have rspec tests that included the linuxfw module (or your other merged modules), you may not have to change anything. I will assume you have an rspec test that includes an affected module and that it now fails:

[rnelson0@build profile]$ cat spec/classes/base_spec.rb
require 'spec_helper'
describe 'profile::base', :type => :class do
...
  context 'with defaults for all parameters' do
    it { is_expected.to create_class('linuxfw') }
    it { is_expected.to create_class('profile::base') }
    it { is_expected.to create_class('ntp') }
    it { is_expected.to create_class('ssh::server') }
    it { is_expected.to create_class('ssh::client') }
  end
end
[rnelson0@build profile]$ rspec spec/classes/base_spec.rb
F....

Failures:

  1) profile::base with defaults for all parameters should contain Class[linuxfw]
     Failure/Error: it { is_expected.to create_class('linuxfw') }
       expected that the catalogue would contain Class[linuxfw]
     # ./spec/classes/base_spec.rb:16:in `block (3 levels) in '
...

In the rspec test, change the class from linuxfw to profile::linuxfw and run the test again.

[rnelson0@build profile]$ git diff
diff --git a/dist/profile/spec/classes/base_spec.rb b/dist/profile/spec/classes/base_spec.rb
index 6b8ab34..2e882a1 100644
--- a/dist/profile/spec/classes/base_spec.rb
+++ b/dist/profile/spec/classes/base_spec.rb
@@ -13,7 +13,7 @@ describe 'profile::base', :type => :class do
   end

   context 'with defaults for all parameters' do
-    it { is_expected.to create_class('linuxfw') }
+    it { is_expected.to create_class('profile::linuxfw') }
     it { is_expected.to create_class('profile::base') }
     it { is_expected.to create_class('ntp') }
     it { is_expected.to create_class('ssh::server') }
[rnelson0@build profile]$ rspec spec/classes/base_spec.rb
.....

Finished in 1.67 seconds
5 examples, 0 failures

I only have one test to fix but you may have more. Ensure all tests pass before committing the Stage 3 set of changes.

[rnelson0@build profile]$ git commit -m 'Simplifying controlrepo, stage 3: rspec tests have 100% coverage again'
[simplified_layout ade53bf] Simplifying controlrepo, stage 3: rspec tests have 100% coverage again
 1 files changed, 1 insertions(+), 1 deletions(-)

Stage 4: Deployment

The final stage is to deploy the changes to your puppet master. This can be a one or three step process, depending on whether or not you have an existing controlrepo and are deploying in a new branch, or this is a brand new control repo and it is deployed as the production branch. When deploying a greenfield solution, skip to Symlinking hiera.yaml.

Existing Controlrepo

I have an existing controlrepo and have created a new branch simplified_layout. I will have an issue because the current /etc/hiera.yaml points to /etc/puppet/hiera.yaml with a :datadir: value of /etc/puppet/hiera/data/, my old location for hiera contents. The new :datadir: will be /etc/puppet/environments/%{environment}/hiera. There is no good way to test the new and old setup side by side. We can provide a location to the config file to the hiera CLI command for testing. Run r10k to deploy your new environment, then run hiera with the correct config location and environment. It doesn’t matter what key you look up as long as it’s valid in both environments.

[rnelson0@puppet ~]$ sudo r10k deploy environment -p
[rnelson0@puppet ~]$ hiera yumrepo_name
el-6.5
[rnelson0@puppet ~]$ hiera -c /etc/puppet/environments/simplified_layout/hiera.yaml yumrepo_name environment=simplified_layout
el-6.5

If you have any issues with this step, check the :datadir: and make sure you are using the correct location for your version of Puppet, FOSS or Enterprise. You should also double check to make sure that all the controlrepo changes were commited and pushed to origin. You can continue once the hiera CLI tool is operating properly.

Merge upstream

Now that we know the new branch works with hiera and rspec can build catalogs, it’s time to merge upstream. Submit a PR, or whatever process your group has established, and merge the branch once the code review is complete. This step is up to you as everyone has a slightly different workflow here. The important part is that you satisfy all your normal workflow requirements and don’t bypass any process. Continue once this is complete.

Symlinking hiera.yaml

The last step in this stage is to symlink hiera.yaml files to your controlrepo’s hiera.yaml. You need to do this with both /etc/hiera.yaml and /etc/{puppetlabs/}puppet/hiera.yaml so that the hiera CLI and hiera within puppet will have the same configuration file. We will symlink to the production environment, so re-run r10k and make sure the file is there before symlinking. You can copy the contents to these two files, but if you make changes in your controlrepo you will have to deploy the changes and you cannot test changes to hiera.yaml in new branches. I suggest the symlink.

[rnelson0@puppet ~]$ sudo r10k deploy environment -p
[rnelson0@puppet ~]$ tail -1 /etc/puppet/environments/production/hiera.yaml
  :datadir: /etc/puppet/environments/%{environment}/hiera
[rnelson0@puppet ~]$ sudo ln -sf /etc/puppet/environments/production/hiera.yaml /etc/hiera.yaml
[rnelson0@puppet ~]$ sudo ln -sf /etc/puppet/environments/production/hiera.yaml /etc/puppet/hiera.yaml

Note that since the datadir path involves the environment variable, you will now need to provide that on the CLI:

[rnelson0@puppet ~]$ hiera yumrepo_name
nil
[rnelson0@puppet ~]$ hiera yumrepo_name environment=production
el-6.5

You can now try a puppet agent in production and everything should work! You will see a few changes, like firewall rules receiving a new identifier and purging the old ones, but nothing that “really” changes. Things to check if it fails, in no particular order: restart puppetserver/httpd/nginx after modifying hiera.yaml, make sure ‘puppet module list –environment=production’ lists your modules, or check to see if any DSL modifications you made weren’t caught by your hook or rspec tests.

[rnelson0@puppet ~]$ sudo puppet agent -t
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for puppet.nelson.va
Info: Applying configuration version '1429134100'
Notice: /Stage[main]/Profile::Linuxfw/Firewall[9005 8eee49915b9ff9ef00b6af60d5113a66]/ensure: removed
Notice: /Stage[main]/Profile::Linuxfw/Firewall[9006 a73ef6d2c52b1ce6c5ae8d9d326b34ed]/ensure: removed
Notice: /Stage[main]/Profile::Linuxfw/Firewall[9004 dc0f1adfee77aa04ef7fdf348860a701]/ensure: removed
Notice: /Stage[main]/Profile::Linuxfw/Firewall[9001 fe701ab7ca74bd49f13b9f0ab39f3254]/ensure: removed
Notice: /Stage[main]/Profile::Linuxfw/Firewall[9002 a8eb63c76896060f20aa62621c36f77a]/ensure: removed
Notice: /Stage[main]/Profile::Linuxfw/Firewall[9003 49bcd611c61bdd18b235cea46ef04fae]/ensure: removed
Notice: /Stage[main]/Puppet::Passenger/File[/var/lib/puppet/ssl/ca]/mode: mode changed '0755' to '0770'
Notice: /File[/etc/sysconfig/iptables]/seluser: seluser changed 'unconfined_u' to 'system_u'
Notice: /Stage[main]/Puppet::Passenger/File[/var/lib/puppet/ssl/ca/requests]/mode: mode changed '0755' to '0750'
Notice: /Stage[main]/Profile::Base/Exec[shosts.equiv]/returns: executed successfully
Notice: Finished catalog run in 7.53 seconds

Stage 5: Cleanup

There are two things you need to cleanup before we call it quits. As mentioned very early on, /etc/r10k.yaml needs an update. You can remove the hiera source manually or re-apply your r10k_installation.pp file:

[rnelson0@puppet ~]$ sudo puppet apply /etc/puppet/environments/production/r10k_installation.pp
Notice: Compiled catalog for puppet.nelson.va in environment production in 0.93 seconds
Notice: /Stage[main]/R10k::Config/File[r10k.yaml]/content: content changed '{md5}8ee4702fff4902327d0a83dc4c5da3af' to '{md5}deb5acd8ed5192c8c0822aa92a35ca9a'
Notice: Finished catalog run in 0.38 seconds

With that out of the way, you can delete the old datadir:

[rnelson0@puppet ~]$ sudo rm -fR /etc/puppet/hiera/

Restart puppetmaster/httpd/nginx and do another puppet run and hiera lookup to make sure nothing blew up.

Summary

Merging four individual repos into a single controlrepo takes some effort, but the payoff is cutting your workflow in half and removing the most tedious parts, creating and merging branches. Thankfully, this is a one-time effort, so you’ll never have to do it again! Er, at least not in your current environment or job… With our single controlrepo in hand, we can start reviewing our puppet master configuration, including moving the r10k_installation.pp contents into the master’s profile class and configuring a webhook. We can also review our profiles to make sure everything is hiera-fied, without the overhead of all the branching that puts most people off from such important, but tedious, technical debt paydown.