In an upcoming series, I’ll be writing about Puppet and Git. As part of the research, I spent a number of hours looking at existing tools for post-receive hooks that were compatible with Github and r10k. In the end, my research went a completely different way and my first effort didn’t pan out, but I did learn from the experience and thought that sharing it might help others.

I was attempting to take an integrated puppet/r10k installation supporting dynamic environments and add a post-receive hook. The current workflow finished up with having to log into the puppet master, su to root/sudo and run r10k to deploy. The primary goal of the hook was to eliminate this step. This would not only simplify the workflow, but also increase security (less people have to have root access) and eliminate mistakes (Why isn’t my change visible? Oops I forgot to run r10k). The concept of hooks is fairly simple – when certain git activities occur, programs are called – but I needed to put things together. I’m on this box, I do my git work and push it to origin, then I need origin to do … something … and tell the puppet master to do … something else.

My initial research was focused on identifying the somethings. A common solution is to install gitolite on a node and make that the origin. It can then call an external program that SSH’s to the master and runs r10k. I eliminated this option because it’s either another node to manage or another service on an existing node, plus I have to perform backups of the git repo. I’d rather use Github at home or Stash at work to foist some of those responsibilities off on others.

Another solution, which got closer to the end result, is to use a Webhook at Github.com (similar options exist for Stash, your admin just needs to install them for you) and a receiver on your puppet master. The Webhooks, by default, fire off when a push is received. You can customize the hooks to only fire off on certain events if you desire. The hook will PUT some JSON data to the specified URL. The URL obviously needs to be reachable from Github/Stash, so make sure you use a public DNS name and IP and the firewall will allow the traffic.

The next piece I needed was the receiver on the master. R10k is used by numerous people, so I figured this had to be a solved problem. My research wasn’t turning much up – I found a single post-receive hook with no commentary, a Rack app with a hook and no commentary, and finally, an app called PGSVR, with commentary! Puppet Git Sync via REST appeared to be my savior. Unfortunately, this was just the beginning of the rabbit hole.

Before I proceed, I want to say that I appreciate the work that Ventz Petkov put into pgsvr, even though it didn’t work out for me. It’s carefully noted that it’s a Beta, and regardless I learned a lot and I couldn’t have gotten to my end result without the knowledge I gained from this exercise. Please don’t take anything I say from here on out as negative. We learn a lot from failures, but we own our own failures.

The console output below is a mix of recollection and actual output – I decided to fully document this after I reverted changes, so some output was lost. If there’s no prompt in the output, it’s from memory; if a prompt is present then it’s actual output.

PGSVR

As mentioned, I chose pgsvr because it’s a utility designed specifically to address the puppet/r10k integration I need. I’ll start with the posted requirements and whether my environment met them:

• Have dynamic environments – CHECK
• Set a shell for the puppet user – NEGATIVE
• Edit your sudo-ers file – NEGATIVE
• Permissions on /etc/puppet/environments – NEGATIVE
• Perl modules Dancer and Plack – NEGATIVE

In my exploration, I found some of these pre-reqs aren’t needed, and there are a ton of missing pre-reqs (my CentOS template is a ‘minimal’ build, which is great for discovering undocumented requirements!). The perl modules need the gcc and perl-CPAN  CentOS packages installed first, then I used CPAN to install the two modules. I installed the modules with:

yum install -y gcc
perl -MCPAN -eshell
o conf prerequisites_policy follow
o conf build_requires_install_policy yes
o conf commit
install Dancer Plack

This took a while. Drink a frosty beverage a while… If you run into errors, scroll up to find the issue, exit CPAN, fix the issue, restart CPAN, and try and install the modules again. I ran into a few, mostly because I did not have gcc installed. If you run into serious issues, http://perlmonks.com/ or http://stackoverflow.com/ may be of assistance.

Next up, I installed apache, called httpd in CentOS. I cloned the pgsvr repo and moved files around as suggested – the web directory under /var/www, the vhost definition under /etc/httpd/conf.d/pgsvr.conf (must have .conf to be processed). I customized the vhost definition to match my environment. Start the httpd service and ensure I get a response – it won’t look pretty, we just want to make sure it works.

yum install -y httpd
mkdir git
cd git
git clone https://github.com/ventz/pgsvr.git
cd pgsvr
(cd app; cp -pr pgsvr /var/www)
cp apache/pgsvr /etc/httpd/conf.d/pgsvr.conf
vi pgsvr.conf                                <-- Specifically ServerName, ServerAdmin, ErrorLog, CustomLog
service httpd start
services iptables stop                       <-- Easier than a new rule during testing

I also created an executable file to do the deploy. This is called by the hook and runs as root, so if you need other commands run as root, here’s a place to put it. I enabled setuid and made it executable to only root and the apache group. I also don’t have a proxy in place, so I made sure to unset them. Adjust as necessary:

[root@puppet ~]# cat /sbin/deploy-r10k
#!/bin/bash
unset http_proxy
unset https_proxy
/usr/bin/sudo /usr/bin/r10k deploy environment -p
[root@puppet ~]# ls -la /sbin/deploy-r10k
-rwsr-xr--. 1 root apache 97 Apr 2 07:03 /sbin/deploy-r10k

The instructions next say that to modify /var/www/pgsvr/bin/app.pl to add a user and token. I created a user with a name ‘rnelson0’ and a password of ‘password’ piped through md5sum. This is a VERY simple token, it’s not doing any in-line hashing or salting so it really doesn’t matter what you use as long as it matches on both ends. You’d probably want something more secure in production – secure the code, secure the authentication, secure the traffic (or not, more later). I also needed to edit a bit more of the program. Here’s the result:

[root@puppet ~]# echo password | md5sum
286755fad04869ca523320acce0dc6a4  -
[root@puppet ~]# cat /var/www/pgsvr/bin/app.pl
#!/usr/bin/perl
# Puppet Git Sync via REST
# Ventz Petkov
# ventz@vpetkov.net

use warnings;
use strict;
use Dancer;

my $serializer = 'XML'; # or 'JSON'
my $port = 8080; # used for stand-alone app (not via Apache)


set serializer => $serializer;
set port => $port;
set startup_info => 0;
set log => 'error';
set logger => 'file';

# One way to create a token: "md5sum" and concat some initial + username
# ex: for user 'ventz', you can do: echo 'tk123ventz' | md5sum
my %users = (
    rnelson0 => "286755fad04869ca523320acce0dc6a4",
);

any ['get', 'post'] => '/' => sub {
    return {message => "PGSVR - Puppet Git Sync via REST"};
};

any ['get', 'post'] => '/sync/:user/:token' => sub {
    my $user = params->{user};
    my $token = params->{token};

    if($token eq $users{$user}) {
        `sudo /sbin/deploy-r10k`;
        my $hostname = `hostname -a`;
        chomp($hostname);
        return{message => "Syncing r10k on $hostname."};
    }
    else {
        return{message => "ERROR: Invalid user"};
    }
};


# Stand alone app (comment out for apache)
dance;

This gave me a vhost that listened for a POST to /sync/$user/$token and then kick off /sbin/deploy-r10k. I also updated /etc/sudoers:

[root@puppet ~]# tail -1 /etc/sudoers
apache ALL=(ALL) NOPASSWD: /sbin/deploy-r10k

As usual, disclaimer that this is okay in a lab but should be secured a bit better in production.I used curl at this point to test things out by providing the correct URL. Just to ensure everything is being built properly and doesn’t just look good because of existing files, I emptied out /etc/puppet/environments first then called curl. I’ve included a bad password to ensure authentication is required.

NOTE: I disabled SELinux enforcement (setenforce Permissive) . I spent a loooong time trying to develop an SELinux module that contained all the contexts and rights with audit2allow, but gave up after the 2nd hour. I suggest changing the level. You can make the change permanent in /etc/selinux/config.

[root@puppet ~]# rm -fR /etc/puppet/environments/*
[root@puppet ~]# ls -l /etc/puppet/environments/
total 0
[root@puppet ~]# curl -i http://puppet.nelson.va/sync/rnelson0/badpassword
HTTP/1.1 200 OK
Date: Wed, 02 Apr 2014 18:40:27 GMT
Server: Apache/2.2.15 (CentOS)
X-Powered-By: Perl Dancer 1.3121
Content-Length: 39
Connection: close
Content-Type: text/xml; charset=utf-8

<data message="ERROR: Invalid user" />
[root@puppet ~]# curl -i http://puppet.nelson.va/sync/rnelson0/286755fad04869ca523320acce0dc6a4
HTTP/1.1 200 OK
Date: Wed, 02 Apr 2014 18:45:33 GMT
Server: Apache/2.2.15 (CentOS)
X-Powered-By: Perl Dancer 1.3121
Content-Length: 43
Connection: close
Content-Type: text/xml; charset=utf-8

<data message="Syncing r10k on puppet." />
[root@puppet ~]# ls -l /etc/puppet/environments/ 
total 16
drwxr-xr-x. 5 root root 4096 Apr  2 18:41 master
drwxr-xr-x. 5 root root 4096 Apr  2 18:41 motd
drwxr-xr-x. 5 root root 4096 Apr  2 18:41 production
[root@puppet ~]# du -h /etc/puppet/environments/ --max-depth=1
8.6M    /etc/puppet/environments/motd
8.6M    /etc/puppet/environments/production
17M     /etc/puppet/environments/
[root@puppet ~]# puppet agent --test --noop
Info: Applying configuration version '1396464244'
Notice: /Stage[main]/Main/Node[puppet.nelson.va]/Notify[Generated from our notify branch]/message: current_value absent, should be Generated from our notify branch (noop)
Notice: Node[puppet.nelson.va]: Would have triggered 'refresh' from 1 events
Notice: Class[Main]: Would have triggered 'refresh' from 1 events
Notice: Stage[main]: Would have triggered 'refresh' from 1 events
Notice: Finished catalog run in 0.92 seconds

Note that on the first run, you may get a 502 timeout error. Don’t worry! R10k continues to run in the background, it’s just the web connection that times out. If there was a problem running r10k, you’d likely see it very early. No errors are reported to the screen, check the ErrorLog specified in your pgsvr.conf vhost file if you do run into issues.

Why This Didn’t Work

So I got working, but I did not consider it a success. Why? There are a few reasons:

• The install is a bit of a mess. Requiring perl CPAN, gcc, and a boatload of modules is… messy doesn’t begin to cover it. The perl compilations took 30 minutes alone, and that’s after I fixed some errors that cropped up and re-ran the progress. There are solutions to streamline this, but I wasn’t interested in investing the time. Maybe something like FPM would help here, but then I have a package to maintain on upgrades of my master.
• Security – GCC – I don’t put compilers on hosts unless they absolutely need it.
• Security – SELinux – I need a damn good reason to disable SELinux. This wasn’t it.
• Security – The user/token doesn’t actually do anything. You add the entire URL, including the token, into the github hook service where it’s visible to anyone with the rights there; plus it’s never going to use another user/token combination, making the username superfluous. Perhaps authentication would help if you had a gitolite server that generated an actual token and the hook is changed to perform a lookup via PAM, but as it is, it provides a false sense of security only.
• Apache – I don’t have a problem with apache, but it’s otherwise not running or used on the master, so it’s mostly increasing the attack surface area. It also introduces a potential conflict/confusion with Puppetboard or any other sort of web front end, either with port usage or number of vhosts.
• Low portability – This has to run on the webserver. If I start to split functionality as I scale Puppet upward – say, adding multiple puppetmasters and loadbalancing – this solution won’t work.

With all that in mind, this was simply not meant to be. In addition, as I found out later, the above hook would need to be in place on the puppet repo and all module repos. The eventual solution I chose doesn’t need, or really want, a hook on the puppet repo, only on the modules. That’s a story for another time.

However, I learned a lot. Some people learn well from books, I learn well from experience. “To enable a post-receive Git hook, click here in github and have a server at the destination url that runs a program,” tells me what to do but isn’t how I learn. Actually going in and configuring the hook vastly increases my understanding of and likelihood of retaining the information. I also learned from the setup of the pgsvr application and vhost. How the payload is delivered from github, what the server expects to see, how the server makes the call to the setuid binary, the problems that user ‘apache’ has trying to touch files owned by root (or ‘puppet’, or even ‘apache’!) when crossing SEL contexts – individually I understood all these components, but it was fun to see them come together and see where theory meets reality.

For instance, SEL contexts make sense. User A, Role B in Type C want to talk to User D, Role E, in Type F you have to update security to allow it. Seems easy. Except you don’t just allow A/B/C unfettered access to D/E/F, you allow the ‘read’ permission. Now you try it again, instead of erroring out on ‘read’, it errors out on ‘unlink’…then ‘chattr’, then this, then that. If you get past that, now you need to talk to D/E in Type G as well! When you’re crossing multiple contexts, it gets very complicated, very quickly, and you start thinking, “Maybe there’s a better way to do this?” That doesn’t really make “sense” to most people, and it requires either knowledge of all the contexts or knowledge of how to use tools to determine the contexts in use. As a result, I did learn how to use audit2allow to generate SELinux modules based off the logs in /var/log/audit/audit.log. Maybe someday that will come in handy!

Negative Success

Why am I sharing this with everyone else? Consider this is a reminder that we often learn more from failures than from success. If I had been able to install pgsvr with three commands and it just worked, I would have had a magic box that I waved a wand over and stuff came out. Because it failed, I was able to reshape a misbehaving magic box into a working system that I understood. I also understand the limitations, constraints, and compromises of the design. On top of it all, I had fun doing it! Thus, I had a Negative Success.

Do you have a good negative success story of your own? I’d love to hear it, either in the comments or with the twitter hashtag #NegativeSuccess.