Welcome to our last class on Git usage. Now that we have a pre-commit hook in place, we’ll finish things up with a post-receive hook.

UPDATE: On 9/17/2014, Phil released Reaktor, an early version of which much of this article is based on. I haven’t had time to investigate but it should be easier to install and far more functional!

Post-Receive Hook

I don’t know about you, but I’m already tired of having to run r10k manually. Having to ssh to the master, log in, and run commands is so droll. What can we do about it?

A post-receive event fires when a remote push is received by the repo with the hook (i.e. when I ‘git push origin branch’, the ‘origin’ server will fire the hook) and we can use a corresponding post-receive hook to deploy for us. This is slightly trickier than a pre-commit hook because of where the event is firing. We don’t want to run it on our desktop/workstation VM, because that host would be the origin repo for everyone. We can’t want run it on the puppet master, because then our puppet master is the origin. (Technically, each repository is a fully sufficient origin repository on its own, but I’m making an assumption that you have a designated origin that’s backed up and therefore you won’t be doing the same for the other repo clones.) That leaves GitHub, which is already our designated origin. Because the post-receive event will fire on the origin, we need to ensure that Github can talk to the puppet master, which is where r10k is located.

There’s no one way to do this. My design is to implement a GitHub Web Hook, which will result in Github sending a POST to a target URL (which means GitHub needs to be able to communicate with it!) that says, “Someone just committed a change, here are the details of the change.” There’s more detail on enabling the Web Hook here. If you’re using Atlassian Stash, as we do at work, hopefully the admins have installed a plugin for web hooks (ironically, I started this documentation in March and I’m still waiting on a web hook to be installed – but if I used Github…).

For this to work, of course, we have to have a web service that can listen for the POST. We’ll want that installed on the puppet master for now. At scale, you’ll likely need a stand-alone server, maybe a load-balancer, which will communicate with the master(s), but we’re starting small in our lab. When the service receives a POST, it can kick off r10k and deploy your changes. There are infinite ways to make this happen, dozens of which are documented on the internet. Let’s add another one.

Speaking of infinite ways to do this, you can read about a previous attempt I made here. tl;dr: not all of the infinite possibilities work, but they all consume your time!

With the help of Phil Zimmerman, I was provided some insight into his team’s r10k workflows. His tool uses Sinatra, a Ruby web framework, to provide the web service, and Capistrano, a ruby automation and deployment tool, to perform deployment. Phil’s tool was designed for use by his team, where they have a dedicated web service that receives the Github hook which in turn invokes Capistrano to communicate with multiple puppet masters. Phil has another trick up his sleeve which I’ll show you soon enough. I made a few edits to the Sinatra program to replace the capistrano call with a direct call to r10k, you may have to perform a few tweaks for your environment as well.

Let’s start with installation. If you’re doing this on your puppet master, like me, ruby is already present. If not, install ruby. Next, install Sinatra (optionally Capistrano) with gem, which will also install dependencies:

[root@puppet ~]# gem install sinatra
Fetching: rack-1.5.2.gem (100%)
Successfully installed rack-1.5.2
Fetching: tilt-1.4.1.gem (100%)
Successfully installed tilt-1.4.1
Fetching: rack-protection-1.5.2.gem (100%)
Successfully installed rack-protection-1.5.2
Fetching: sinatra-1.4.4.gem (100%)
Successfully installed sinatra-1.4.4
Installing ri documentation for rack-1.5.2
Installing ri documentation for rack-protection-1.5.2
Installing ri documentation for sinatra-1.4.4
Installing ri documentation for tilt-1.4.1
Done installing documentation for rack, rack-protection, sinatra, tilt after 6 seconds
4 gems installed
[root@puppet ~]# gem install capistrano
Fetching: tins-1.0.1.gem (100%)
Successfully installed tins-1.0.1
Fetching: term-ansicolor-1.3.0.gem (100%)
Successfully installed term-ansicolor-1.3.0
Fetching: net-ssh-2.8.0.gem (100%)
Successfully installed net-ssh-2.8.0
Fetching: net-scp-1.1.2.gem (100%)
Successfully installed net-scp-1.1.2
Fetching: sshkit-1.3.0.gem (100%)
Successfully installed sshkit-1.3.0
Fetching: rake-10.2.2.gem (100%)
Successfully installed rake-10.2.2
Fetching: i18n-0.6.9.gem (100%)
Successfully installed i18n-0.6.9
Fetching: capistrano-3.1.0.gem (100%)
Capistrano 3.1 has some breaking changes, like `deploy:restart` callback should be added manually to your depl
oy.rb. Please, check the CHANGELOG: http://goo.gl/SxB0lr

If you're upgrading Capistrano from 2.x, we recommend to read the upgrade guide: http://goo.gl/4536kB
Successfully installed capistrano-3.1.0
Installing ri documentation for capistrano-3.1.0
Installing ri documentation for i18n-0.6.9
Installing ri documentation for net-scp-1.1.2
Installing ri documentation for net-ssh-2.8.0
Installing ri documentation for rake-10.2.2
Installing ri documentation for sshkit-1.3.0
Installing ri documentation for term-ansicolor-1.3.0
Installing ri documentation for tins-1.0.1
Done installing documentation for capistrano, i18n, net-scp, net-ssh, rake, sshkit, term-ansicolor, tins after 18 seconds
8 gems installed

You can make sure it’s working with a one-liner. Note that rubygems must be used or you’ll get an error about ‘no such file to load — sinatra’:

[root@puppet ~]# ruby -r rubygems
require 'sinatra'
<ctrl-d>
[2014-04-04 03:41:34] INFO  WEBrick 1.3.1
[2014-04-04 03:41:34] INFO  ruby 1.8.7 (2011-06-30) [x86_64-linux]
== Sinatra/1.4.4 has taken the stage on 4567 for development with backup from WEBrick
[2014-04-04 03:41:34] INFO  WEBrick::HTTPServer#start: pid=23184 port=4567
^C== Sinatra has ended his set (crowd applauds)
[2014-04-04 03:41:38] INFO  going to shutdown ...
[2014-04-04 03:41:38] INFO  WEBrick::HTTPServer#start done.

Note: I’m not a developer, but there are some … strong thoughts … on how to use rubygems. Since that comes from one of the guys behind Sinatra, I’m going to listen to him. The choice is yours.

Next up is the script. I’ve taken the original hook and modified it slightly. Copy the contents of the file to hooks/www-r10k.rb. Let’s take a look at this by line number. I’m going to skip a few because, frankly, I don’t fully understand it all. Don’t worry, though, I’m a professional!

• 5-8: These are the only lines you need to customize. The repo at hand here is your puppet or puppetfile repo, not a module repo. Hostname has to be resolvable and you should choose an available port.
• 10-11: Bind sinatra to the correct name and port.
• 14: The webhook we configure at github must use this path, i.e. http://puppet.contoso.com/payload
• 15-18: Creates a temporary working directory. Line 18 sets the git command, which includes a reference to the temporary location, so that if we call git from outside of the repo, it can act on the correct files.
• 20: The github hook includes JSON in the POST payload. The variable push now contains this data.
• 23-29: Break the JSON apart into some various components. If you’re not very familiar with the details of git, if you’re just a user like me, you probably don’t need to know. The refs, deleted, and created sections are details of what was changed in the push that generated the hook event. The ref section then gets broken down into ref_type and branchName.
• 34-42: If the JSON includes a ‘deleted’ action for a branch type (‘heads’), call a function to delete the branch.
• 44-54: If the JSON includes a ‘created’ action for a branch type (‘heads’), call a function to create the branch, update the branch’s module ref, deploy the environment, and cleanup.
• 56-62: If it’s neither action type, then we only need to update it, deploy the module, and cleanup.
• 66-72: function deployEnv. I don’t use capistrano, so I’ve converted this to call r10k directly.
• 74-80: function deployModule. I don’t use capistrano, so I’ve converted this to call r10k directly.
• 82-999: function createPuppetfileBranch. Checks out production of the puppet repo and creates a new branch.
• 101-111, 113-118, 120-124, 126-130, 132-135: functions branchExists, deletePuppetfileBranch, clonePuppetfileRepo, checkoutBranch, and cleanup. Nothing complex here, the names describe these.
• 137-176: function updatePuppetfileModuleRef: This is the tricky bit, I’ll describe it in a bit.

After customizing the file, commit the change to the githooks branch. Of course, the web service needs to be running. We’ll run it at the command line and I’ll leave it up to you to figure out how to appropriately daemonize it. If you’re low on time, add a nohup line to the end of /etc/rc.local. I’ll save the file to /root/puppet/www-r10k.rb for now and invoke it with ruby:

[root@puppet puppet-tutorial]# mkdir /root/puppet
 [root@puppet puppet-tutorial]# cp hooks/www-r10k.rb /root/puppet/www-r10k.rb
 [root@puppet puppet-tutorial]# ruby -r rubygems /root/puppet/www-r10k.rb
 [2014-04-04 19:06:13] INFO  WEBrick 1.3.1
 [2014-04-04 19:06:13] INFO  ruby 1.8.7 (2011-06-30) [x86_64-linux]
 == Sinatra/1.4.4 has taken the stage on 80 for development with backup from WEBrick
 [2014-04-04 19:06:13] INFO  WEBrick::HTTPServer#start: pid=23486 port=80

Note: Phil was very kind and gave me preview access to his workflow when I started writing this series. He has since tweaked the hook. I have correspondingly tweaked the hook. However, there’s a discrepancy between the contents of the hook in the gist and my repo due to publishing timelines. The version in the repo works fine if you’re just cloning my repo, but you should definitely use Phil’s version as the source if you’re creating this on your own. Of course, Phil is improving his hook all the time so you should follow him on twitter to get update notifications!

You should be able to visit the specified port on your puppet master, such as http://puppet.nelson.va, and receive a 404 from Sinatra. Don’t forget to adjust/disable iptables for port 80 (since you should be managing that through puppet, and we’re setting up dynamic environments at the moment, I’d suggest disabling it and make it one of the first changes you implement when we’re done here). You’ll know you have the right service when you get there…

There’s one other thing you’ll need to do. Because this program is going to perform work against your git repos, you need to set up a service account for it and upload SSH keys. I’m confident you can figure out how to set up a service account on your own so I’ll continue to use my github user as I configured for root a few articles ago. Remember to change the global git config’s user to the service account. Generate a pair of SSH keys with ssh-keygen and attach the public key to the github account. It’s under Settings -> Deploy Keys at a URL like https://github.com/user/repo/settings/keys. Test the access with ssh git@github.com. Even if you know what you’re doing and are sure it will work, don’t skip this step as you need to accept the RSA keys at least once (also make sure your /etc/ssh/ssh_config or user config isn’t pointing UserKnownHostsFile to /dev/null – older versions of saz/ssh do just that).

[root@puppet ~]# ssh git@github.com
The authenticity of host 'github.com (192.30.252.131)' can't be established.
RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'github.com,192.30.252.131' (RSA) to the list of known hosts.
PTY allocation request failed on channel 0
Hi rnelson0/puppet-tutorial! You've successfully authenticated, but GitHub does not provide shell access.
                                                                                                         Connection to github.com closed.

If you don’t accept those keys, the first time sinatra calls r10k, it will sit there forever asking you to accept your keys. Whoops!

Adding the Web Hook

To bring it together, you need to add the Web Hook to your module repos – ALL module repos. Bring up each repo and go to Settings -> Webhooks and Services -> Add webhook. Add in a PUBLIC URL for your Puppetmaster. For instance, puppet.nelson.va won’t resolve publicly, and port 80 is used on my public system. I set up a port forward on port 8080 of my router to port 80 on the puppet master: http://rnelson0.publicdomain.com:8080/payload. When I push a commit, I see a nice notice in my github webhook page showing me the notification it sent. Notifications show green if a good response was received. A red response just means that no good response was returned – check it out to find out why! It’s likely that it’s due to timeouts. First time r10k deployments can take a while, and github only waits so long for a response. On the master, you should still see sinatra calling r10k and eventually your environments directory will be full up. There’s a lot more output than will fit here, but you should see something like this:

I, [2014-04-04T19:18:26.040802 #23486]  INFO -- : github json payload: {"deleted"=>false, "head_commit"=>{"added"=>[], "author"=>{"name"=>"Nelson, Robert",
 "email"=>"rnelson0@gmail.com", "username"=>"rnelson0"}, "committer"=>{"name"=>"Nelson, Robert", "email"=>"rnelson0@gmail.com", "username"=>"rnelson0"}, "t
imestamp"=>"2014-04-04T19:18:16Z", "modified"=>[], "message"=>"Remove test", "url"=> ... 
<and about 10 more lines> 
...}}
I, [2014-04-04T19:18:26.040893 #23486]  INFO -- : repo name = rnelson0-base
I, [2014-04-04T19:18:26.040931 #23486]  INFO -- : repo ref = refs/heads/branch
I, [2014-04-04T19:18:26.040963 #23486]  INFO -- : branch = branch
I, [2014-04-04T19:18:26.040995 #23486]  INFO -- : modify action for branch
Faraday: you may want to install system_timer for reliable timeouts
192.30.252.47 - - [04/Apr/2014 19:19:01] "POST /payload HTTP/1.1" 200 - 35.1548
192.30.252.47 - - [04/Apr/2014:19:18:26 GMT] "POST /payload HTTP/1.1" 200 0
- -> /payload

You’ll note that the logging shows all the JSON. This isn’t insecure – no copy of your keys or other authentication information – though it shows commit logs and if you’re using private repos it may give away a few details, but it IS noisy. You may want to lower logging within /etc/root/puppet/www-r10k.php or make sure logs are rotated and compressed appropriately.

Remember: add this hook to ALL module repos.

Tricksies!

Previously, I described a workflow that at a high level was:

• Feature branch in module repo
• Feature branch in puppet repo
• Modify Puppetfile to have a :ref pointing to the module’s branch
• Deploy and test
• Merge the module branch, discard the puppet branch.

If you’ve tried this out, no doubt the highlighted steps are the annoying part. So let’s get rid of them! Let’s analyze the updatePuppetfileModuleRef function, starting with line 143:

    regex = /mod (["'])#{module_name}\1(.*?)[\n]\s*:ref\s*=>\s*(['"])(\w+|\w+\.\d+\.\d+)\3/m

This complicated beastie finds and matches this portion of a Puppetfile:

mod "base",
  :git => "git://github.com/rnelson0/rnelson0-base"

Lines 141-144 take this pattern and add the ref to the branch, like so:

mod 'base',
  :git => "git://github.com/rnelson0/rnelson0-base",
  :ref => 'branch'

Lines 152-154 update the Puppetfile itself. Lines 158-166 commit the new branch to the repo. If you go back and look at the other functions, you’ll notice that this branch is auto-deleted when the feature branch is deleted as well.

This changes our workflow a bit, since we don’t have to modify the Puppetfile all that much. Here’s what’s going to happen when you need to add a new module:

• Checkout the production branch of the puppet repo, add the mod to the Puppetfile, and push the changes upstream.

That’s it! When you add a module, you’re not using it yet, so there’s no need to re-deploy with r10k. That will happen automatically when you branch one of your modules (of course, it won’t hurt if you do it manually). Here’s that workflow:

• Check out the production branch of your module repo.
• Check out a new feature branch.
• Modify your module
• Commit changes
• Github web hook fires off a POST, your sinatra receiver fires off r10k, and a few seconds later your dynamic environments are updated, including any changes from the puppet repo.
• Test
• Repeat the previous three steps until perfected.
• Merge the changes into production and delete the branch.
• Github web hook fires off a POST, your sinatra receiver fires off r10k, the dynamic environment associated with the defunct branch is deleted.

If you have multiple modules that need changed at once, you just need to create the same branch name in each. The hook will fire off from each repo, adjust the Puppetfile each time, and re-deploy. Voila!

There’s one case where you need to do some manual work. If all you do is change your site manifest, in the puppet repo, without creating or modifying a module repo, you would have to log into the master yourself and run r10k after pushing the changes, in production or a new branch, to the puppet repo. You could add the same hook to the puppet repo, but there’s another way to deal with the site manifest that we’ll investigate soon (hiera), so for now, we’ll leave site manifest updates as a manual effort.

Wrap-up

If you’ve followed the entire 200 series classes, we now have:

• Our Puppet Open Source edition master.
• Site manifests with a single node declaration.
• Our own module for roles and profiles.
• Puppet configuration and module in git repos.
• R10k for dynamic environments.
• A workflow for updating manifests and modules.
• Pre-commit hook to ensure all commits are syntactically correct, eliminating breaking commits.
• Post-receive hook to automate environment creation and deletion, reduce human involvement and error (if you don’t have Web Hooks available keep using the workflow from the 203 and 204 classes).

There’s a lot more you can do with puppet and git. I encourage you to read up elsewhere on git workflows and hooks, Jenkins CI, and other tools that people add to their Puppet workflow (see Additional Resources below). We let it sidetrack us to ensure we have some good basic familiarity with the tools and a minimum level of automation to assist us. Our ultimate goal is a fully automated Software Defined Data Center, so keep iterating and improving your workflow to reduce potential errors and slowdowns in your development and operations. From here out, unless it’s relevant to the process at hand, I’ll just say “create a feature branch” and assume you have a workflow that you know and love.

Finally, what you’ve been waiting for: Next week we’ll get back into developing the puppet installation itself.

Additional Resources

You saw this links last week, but that’s okay because they still apply.

• Puppet Conf 2013 Recorded Videos – a number of these talk about continuous integration pipelines involving Puppet + another tool, such as Jenkins
• What are git hooks?
• git-hooks: A tool to manage project, user, and global Git hooks for multiple git repositories.
• Why You Need a Git Pre-Commit Hook and Why Most Are Wrong
• Deploying Websites With a Tiny Git Hook
• Tips for using a git pre-commit hook
• Missing git hooks documentation