Now that we have a unified controlrepo, we need to set up an r10k webhook. I have chosen to implement the webhook from zack/r10k. There are other webhooks out there – I’m a huge fan of Reaktor – but I chose this because I’m already using this module and because it is recommended by Puppet Labs. It’s an approved module, to boot!

Update: The zack/r10k module has migrated to puppet/r10k, which should be used instead. I’ve commented out sections that are incompatible with the most recent versions of the module, but as this article is now 2 years old, there may be other changes in surrounding modules you will become aware of, too.

Module Setup

The first step is to make sure the module is installed along with its dependencies. There are no conditional dependencies in a Puppet module’s metadata.json, so you can skip puppetlabs/pe_gem and gentoo/portage if you’d like. On the other hand, there are no ill side effects from having the modules present unless you were to use them for some reason. This is an opportunity to up the version on some pinned modules as well, such as stdlib, as long as you do not increment the major version. If the major version increases, there’s a significant chance your code will have some breakage, it’s best to do that in a separate branch.

I encountered a bug with zack/r10k v2.7.3 (#162). This bug is fixed in v2.7.4. Be sure to upgrade!

In the past, we have created an r10k_installation.pp file for bootstrapping the master. We will use this as the basis for our setup by adding it to our profile::puppet_master class. Of course, we do test-driven development around here, so we’ll start with that! We know that we need to contain the r10k class. In the Webhook Support section of the documentation, there are a number of different ways to tackle the webhook. I have chosen to run without mcollective as I only have a single master. This means we also need to test that our class contains r10k::webhook and r10k::webhook::config. Here’s what an example puppet_master class’s rspec statements look like:

require 'spec_helper'
describe 'profile::puppet_master', :type => :class do
  let (:facts) do
  {
    :osfamily               => 'RedHat',
    :operatingsystemrelease => '6.5',
    :concat_basedir         => '/dne/',
  }
  end

  context 'with defaults for all parameters' do
    it { is_expected.to create_class('profile::puppet_master') }
    # it { is_expected.to contain_package('mcollective-common') }
    it { is_expected.to contain_class('epel') }
    it { is_expected.to contain_class('r10k') }
    it { is_expected.to contain_class('r10k::webhook') }
    it { is_expected.to contain_class('r10k::webhook::config') }
    it { is_expected.to contain_firewall('100 allow agent checkins') }
    it { is_expected.to contain_firewall('110 zack-r10k web hook') }
  end
end

You also need to make sure your .fixtures.yml file is up to date with the dependencies:

    r10k: 'git@github.com:acidprime/r10k'
    pe_gem: 'git://github.com/puppetlabs/puppetlabs-pe_gem'
    ruby: 'git://github.com/puppetlabs/puppetlabs-ruby'
    gcc: 'git://github.com/puppetlabs/puppetlabs-gcc'
    inifile: 'git://github.com/puppetlabs/puppetlabs-inifile'
    vcsrepo: 'git://github.com/puppetlabs/puppetlabs-vcsrepo'
    git: 'git://github.com/puppetlabs/puppetlabs-git'
    croddy: 'git://github.com/croddy/puppet-make'
    puppetdb: 'git://github.com/puppetlabs/puppetlabs-puppetdb'

Run rake spec_prep to download the fixtures and rspec spec/classes/puppet_master_spec.rb to test (you can use rake spec_standalone as well, but it can take a while if you have a ton of tests):

[rnelson0@build01 profile]$ rake spec_prep
HEAD is now at 710b3d5 Merge pull request #268 from mhaskel/1.2.0-prep
Initialized empty Git repository in /home/rnelson0/puppet/controlrepo/dist/profile/spec/fixtures/modules/puppetdb/.git/
remote: Counting objects: 1466, done.
remote: Total 1466 (delta 0), reused 0 (delta 0), pack-reused 1466
Receiving objects: 100% (1466/1466), 413.21 KiB, done.
Resolving deltas: 100% (693/693), done.

[rnelson0@build01 profile]$ rspec spec/classes/puppet_master_spec.rb
...FFF.F

Failures:

  1) profile::puppet_master with defaults for all parameters should contain Class[r10k]
     Failure/Error: it { is_expected.to contain_class('r10k') }
       expected that the catalogue would contain Class[r10k]
     # ./spec/classes/puppet_master_spec.rb:16:in `block (3 levels) in '

Now we need to start writing code until the tests pass! Let’s start by copying from our r10k_installation.pp script, bringing the version of r10k up to the latest (1.5.1 as of this writing). If you’re managing the package r10k as a gem, remove that resource. Add the example from the zack/r10k webpage, minus the superfluous comments. Finally, add a firewall rule allowing port 8088 inbound and you should end up with a manifest that looks like this and tests that pass:

[rnelson0@build01 profile]$ cat manifests/puppet_master.pp
# == Class: profile::puppet_master
#
# Puppet Master
#
# === Authors
#
# Rob Nelson <rnelson0@gmail.com>
#
# === Copyright
#
# Copyright 2015 Rob Nelson
#
class profile::puppet_master {
  include epel
  class { '::puppet::master':
    storeconfigs => true,
    environments => 'directory',
  }

  class { 'r10k':
    version => '1.5.1',
    sources => {
      'puppet' => {
        'remote'  => 'git@github.com:rnelson0/controlrepo.git',
        'basedir' => "${::settings::confdir}/environments",
        'prefix'  => false,
      },
    },
    manage_modulepath => false
  }

  class {'r10k::webhook::config':
    use_mcollective => false,
    public_key_path  => '/etc/mcollective/server_public.pem',  # Mandatory for FOSS
    private_key_path => '/etc/mcollective/server_private.pem', # Mandatory for FOSS
  }

  class {'r10k::webhook':
    user    => 'root',
    group   => '0',
  }
  Class['r10k::webhook::config'] -> Class['r10k::webhook']

  firewall { '100 allow agent checkins':
    dport  => 8140,
    proto  => tcp,
    action => accept,
  }
  firewall { '110 zack-r10k web hook':
    dport  => 8088,
    proto  => tcp,
    action => accept,
  }

  # This is only needed with Puppet 3.
  #package { 'mcollective-common':
  #  ensure => present,
  #}
  #Package['mcollective-common'] -> Class['r10k::webhook']
}
[rnelson0@build01 profile]$ rspec spec/classes/puppet_master_spec.rb 
........ 
Finished in 3.22 seconds 8 examples, 0 failures

You may notice that we are instantiating the classes rather than including them and using hiera’s automatic parameter lookup. This is so that we have a clearer picture of what we are doing with the webhook, and because it gives us something to hiera-fy in the next article!

Now that our change is complete, it’s a good time to run rake spec_standalone. This will take a lot longer than running rspec, but we don’t want to commit any changes that do not pass the tests, much less push them upstream! There shouldn’t be any issues if your rspec tests were passing before you started this work, so push the changes upstream and run r10k to deploy them. Enjoy it while you can, we won’t be running r10k manually for much longer!

[rnelson0@build01 profile]$ git push origin webhook
Counting objects: 18, done.
Compressing objects: 100% (10/10), done.
Writing objects: 100% (10/10), 1.62 KiB, done.
Total 10 (delta 5), reused 0 (delta 0)
To git@github.com:rnelson0/controlrepo.git
   fcfea38..4f68cfb  webhook -> webhook

[rnelson0@puppet ~]$ sudo r10k deploy environment webhook

Set up the MCollective certificates

Even though we are not using mcollective, the webhook itself still relies on mcollective to be installed and having working certificates (for now, see issue 140). We can use the existing Puppet CA certs, though. This part is not scripted as it relies on the certificate generated by your puppet master on first startup. We can copy the files, though:

[root@puppet ~]# SSLDIR=`puppet config print ssldir`
[root@puppet ~]# cp $SSLDIR/ca/ca_crt.pem /etc/mcollective/server_public.pem
[root@puppet ~]# cp $SSLDIR/ca/ca_key.pem /etc/mcollective/server_private.pem

Enterprising users may add these files to their controlrepo (or a repo hosting certificatess and other files) and push them down to the master via Puppet.

Enable the Webhook

Now that our environment is ready for testing, we need to configure the webhook on the controlrepo, configure any network-level firewalls in front of the master, and run the agent on the master. The first two steps are going to vary based on your git source and firewall(s). The general principles are the same but the UI and configuration locations will vary. I am using GitHub and a Fortinet firewall. At GitHub, you configure the webhook on a per-repo basis. View the controlrepo at GitHub and click on Settings, Webhooks & Services, Add Webhook. Enter your password to continue. The defaults are: protocol https on port 8088, user:pass of ‘puppet:puppet’, path of ‘/payload’, hostname is your public DNS. This should come together in something similar to https://puppet:puppet@puppet.example.com:8088/payload. Click the button to disable SSL verification. Leave the rest of the settings alone and click the green Add webhook button at the bottom. On my Fortinet firewall, I need to configure a port forwarding rule under Policy & Objects, Objects, Virtual IPs, that forwards port 8088 to my puppet server’s IP address. This VIP is added to an existing group of VIPs, which is used in the policy rule (Policy & Objects, Policy, IPv4). I have not found a definitive source of addresses for GitHub webhooks, so the rule is all to public-vips, service all, action Accept, NAT Disable. The final step is to run the agent on the master against the new environment webhook. Stop the puppet service, to make sure it doesn’t go behind us and make other changes.

[root@puppet ~]# service puppet stop
Stopping puppet agent:                                     [  OK  ]
[root@puppet ~]# puppet agent -t --environment webhook
Info: Loading facts
Info: Caching catalog for puppet.nelson.va
Info: Applying configuration version '1430656426'
Notice: /Stage[main]/Profile::Linuxfw/Firewall[110 sinatra web hook]/ensure: removed
Notice: /Stage[main]/R10k::Config/File[r10k.yaml]/content:
--- /etc/r10k.yaml      2015-04-15 21:43:38.930235721 +0000
+++ /tmp/puppet-file20150503-417-1bzpuye-0      2015-05-03 12:34:05.930339767 +0000
@@ -1,9 +1,9 @@
 :cachedir: /var/cache/r10k
 :sources:
-  puppet:
+  puppet:
     basedir: /etc/puppet/environments
     prefix: false
-    remote: "git@github.com:rnelson0/controlrepo.git"
+    remote: git@github.com:rnelson0/controlrepo.git

 :purgedirs:
   - /etc/puppet/environments

Info: Computing checksum on file /etc/r10k.yaml
Info: /Stage[main]/R10k::Config/File[r10k.yaml]: Filebucketed /etc/r10k.yaml to puppet with sum deb5acd8ed5192c8c0822aa92a35ca9a
Notice: /Stage[main]/R10k::Config/File[r10k.yaml]/content: content changed '{md5}deb5acd8ed5192c8c0822aa92a35ca9a' to '{md5}529efc7d6f2ee9c8dbe3cfd7c39406a7'
Notice: /Stage[main]/R10k::Webhook::Config/File[webhook.yaml]/ensure: defined content as '{md5}571db2754e85c01d47cca654b223ecf6'
Info: /Stage[main]/R10k::Webhook::Config/File[webhook.yaml]: Scheduling refresh of Service[webhook]
Notice: /Stage[main]/R10k::Webhook/File[/var/run/webhook]/ensure: created
Notice: /Stage[main]/R10k::Webhook/File[/var/log/webhook]/ensure: created
Notice: /Stage[main]/R10k::Webhook/Package[webrick]/ensure: created
Notice: /File[/etc/sysconfig/iptables]/seluser: seluser changed 'unconfined_u' to 'system_u'
Notice: /Stage[main]/R10k::Install/Package[r10k]/ensure: ensure changed '1.4.01.3.41.2.01.1.0' to '1.5.1'
Notice: /Stage[main]/Profile::Puppet_master/Firewall[110 zack-r10k web hook]/ensure: created
Notice: /Stage[main]/R10k::Webhook/File[webhook_init_script]/ensure: defined content as '{md5}ccddb9342bdec0f7f48795cc092db2fe'
Notice: /Stage[main]/R10k::Webhook/File[webhook_bin]/ensure: defined content as '{md5}f5b3effc401d53459a63cb41afcb3a2e'
Info: /Stage[main]/R10k::Webhook/File[webhook_bin]: Scheduling refresh of Service[webhook]
Notice: /Stage[main]/R10k::Webhook/Service[webhook]/enable: enable changed 'false' to 'true'
Notice: /Stage[main]/R10k::Webhook/Service[webhook]: Triggered 'refresh' from 2 events
Notice: /Stage[main]/Ruby::Dev/Package[bundler]/ensure: created
Notice: /Stage[main]/Profile::Base/Exec[shosts.equiv]/returns: executed successfully
Notice: Finished catalog run in 68.42 seconds

[root@puppet ~]# netstat -apn | grep 8088
tcp        0      0 0.0.0.0:8088                0.0.0.0:*                   LISTEN      3189/ruby
[root@puppet ~]# ps -ef | grep 3189
root      3189     1  0 12:35 ?        00:00:00 /usr/bin/ruby /usr/local/bin/webhook

As you can see, the webhook is running! To see how this works, we can tail the log file, /var/log/webhook/access.log, while pushing a new branch to our controlrepo.

Important Debug Information: Go to the controlrepo’s Settings, Webhooks & Services and click the red icon by your webhook. Each delivery, it’s payload, and the response, is recorded and viewable. There is a Redeliver button visible on an expanded payload that will resend that payload, great for troubleshooting. Be careful that you don’t send payloads out of order without considering the consequences. You should probably only use it with the most recent payload.

[rnelson0@build01 controlrepo]$ git checkout -b test_webhook
Switched to a new branch 'test_webhook'
[rnelson0@build01 controlrepo]$ git push origin test_webhook
Total 0 (delta 0), reused 0 (delta 0)
To git@github.com:rnelson0/controlrepo.git
 * [new branch]      test_webhook -> test_webhook

[root@puppet ~]# tail -f /var/log/webhook/access.log
...
[2015-05-03 13:01:45] DEBUG Rack::Handler::WEBrick is mounted on /.
[2015-05-03 13:01:45] INFO  WEBrick::HTTPServer#start: pid=7185 port=8088
[2015-05-03 13:02:46] DEBUG accept: 192.30.252.46:39379
[2015-05-03 13:02:46] DEBUG Rack::Handler::WEBrick is invoked.
[2015-05-03 13:02:46] INFO  authenticated: puppet
[2015-05-03 13:02:46] INFO  message: triggered: r10k deploy environment test_webhook -pv >> /var/log/webhook/mco_output.log 2>&1 & branch: test_webhook
^C
[root@puppet ~]# ls /etc/puppet/environments
production  test_webhook  webhook

Look, ma, no hands! If you re-run the agent against the existing production, this will break, so once you’re happy with your setup – manifests, remote webhook, firewall rules, and the whole thing works – get your changes merged into production. Tail the log again and when the PR is merged or the local webhook->production merge gets pushed, you should see another webhook event fire off and push the changes to production.

[2015-05-03 13:14:09] DEBUG accept: 192.30.252.41:45404
[2015-05-03 13:14:09] DEBUG Rack::Handler::WEBrick is invoked.
[2015-05-03 13:14:09] INFO  authenticated: puppet
[2015-05-03 13:14:09] INFO  message: triggered: r10k deploy environment production -pv >> /var/log/webhook/mco_output.log 2>&1 & branch: production
[2015-05-03 13:14:39] DEBUG close: 192.30.252.41:45404

Summary

We are done with our webhook! We will revisit the configuration in our next article, but for now enjoy the fact that you don’t need to run r10k manually again! Of course, we turned off the puppet agent on the master while we were working, so don’t forget to turn it back on now.

[root@puppet ~]# service puppet start
Starting puppet agent:                                     [  OK  ]

In the next article, we will look at hiera-fying our hiera setup and refactoring our last class{} instantiations so they can be replaced with include statements.