Welcome to our last class on Git usage. Now that we have a pre-commit hook in place, we’ll finish things up with a post-receive hook.
UPDATE: On 9/17/2014, Phil released Reaktor, an early version of which much of this article is based on. I haven’t had time to investigate but it should be easier to install and far more functional!
I don’t know about you, but I’m already tired of having to run r10k manually. Having to ssh to the master, log in, and run commands is so droll. What can we do about it?
A post-receive event fires when a remote push is received by the repo with the hook (i.e. when I ‘git push origin branch’, the ‘origin’ server will fire the hook) and we can use a corresponding post-receive hook to deploy for us. This is slightly trickier than a pre-commit hook because of where the event is firing. We don’t want to run it on our desktop/workstation VM, because that host would be the origin repo for everyone. We can’t want run it on the puppet master, because then our puppet master is the origin. (Technically, each repository is a fully sufficient origin repository on its own, but I’m making an assumption that you have a designated origin that’s backed up and therefore you won’t be doing the same for the other repo clones.) That leaves GitHub, which is already our designated origin. Because the post-receive event will fire on the origin, we need to ensure that Github can talk to the puppet master, which is where r10k is located.
There’s no one way to do this. My design is to implement a GitHub Web Hook, which will result in Github sending a POST to a target URL (which means GitHub needs to be able to communicate with it!) that says, “Someone just committed a change, here are the details of the change.” There’s more detail on enabling the Web Hook here. If you’re using Atlassian Stash, as we do at work, hopefully the admins have installed a plugin for web hooks (ironically, I started this documentation in March and I’m still waiting on a web hook to be installed – but if I used Github…).
Welcome to the Puppet and Git, class 205: Git Hooks. Since we finished up talking about workflows, let’s move on and explore what a git hook is and how you can use one to improve your workflow.
What is a git hook? You can read some boring official documentation, but who does that? Instead here’s a short summary: A git hook is a program that is called when a git events are triggered. These programs are usually simple shell scripts and some common events people use them with are commits. We’ll look at commits and the event that fires before the commit is completed.
If you’ve been programming for more than, say, an hour, you’ve undoubtedly experienced the bad mojo that results from missing a semi-colon or curly brace or other piece of syntactical junk. And you may have even committed such a piece of broken code and pushed it upstream just to watch the whole thing fall apart. If we can lint our code, we can determine whether the code meets the syntactic requirements of a language. It’s important to note that linting doesn’t verify that your code does what it says it will do, it JUST verifies that the code will parse or compile. This would prevent the wonderful pattern of commits that looks something like:
We’ve installed and configured r10k, are using it for deployments, and have a workflow for new modules. More commonly, we will be working on existing modules, a slightly different workflow. We can examine this new workflow by modifying the base module only.
Workflow to Modify Existing Module
Unlike adding a new module, the Puppetfile only needs to be modified to reflect the feature branch. This is where the workflows diverge: instead of requiring a merge, commit, and push, we can create a temporary branch and just delete it when we’re done. Only the module branch needs merged. We’ll show this by making a simple change, modifying Dave’s name. We have another Dave Smith who works here, so we’ll add a middle initial and the name of Dave’s organization, to prevent confusion.
The feature branch is just called dave. We’ll work on the module repo first. Make sure you’re in master and checkout the new branch. Dave’s description should be updated to “Dave G. Smith – IT Administrator” – everything will be alright unless they hire another Dave G. Smith over there. Commit the change and push it.
Welcome back to our Puppet and Git 200-series classes. With r10k installed and configured, today we can focus on workflows. The first workflow is for a new module, either a brand new module you are creating or simply a “new to you” module, such as importing more modules from the Forge. In our classrom, we will add a single module from the forge and update the base module to make use of it. This will give us a good understanding of the workflow for r10k.
Workflow To Add A New Module
The first step in our workflow is to decided on a module to add to our setup. If you have a particular module you want to use, feel free to substitute it below. I’ve chosen saz-motd, a very simple module that is visible when installed, but will not have a material impact on your nodes. We can see right now that there is no message of the day, so we’ll know when we’re done:
[root@puppet ~]# cat /etc/motd
Note: We’ll add our module to a feature branch below. It’s a simple module, so this is fine. More complex modules, such as those that include additional facts and functions, should always be installed on the master first to ensure the plugins are synchronized, which means adding them to production. This was discussed on IRC so I don’t have a link to documentation to show how this works; this is the closest I could find. I’ll mention it again when we install such a module, but I wanted to mention it in case the module you chose provides custom facts/functions.
Welcome back! In our 201 class, we installed r10k, but we still haven’t used it. There’s two tasks we need to complete. First, the existing repo is incompatible with r10k’s dynamic management of modules, so we’ll convert its contents to the proper format. Once that is done, we can deploy dynamic environments using r10k.
Convert existing repo
Check out a clone of the existing puppet repo, rnelson0/puppet-tutorial. As mentioned previously, you can do this on the puppet master, as I will do, or you can perform it on another machine. If you’re on the master, you want to clone the repo into a different directory. After cloning it, check out a new branch called production:
I know you’re probably anxious to get started with managing your infrastructure, but we’re going to stay distracted by Git for a little longer. In the 100 series, we saw some examples of how to migrate your manifests and modules into Git and how to make changes to your manifests through branches. The setup is a little primitive, but acceptable for a lab – everything is is either done by root or involves pushing changes as a user and pulling them as root, and changes are tested in production. I’d like to introduce you to a tool called r10k that will help us create dynamic branches for testing and decouple our workflow from direct access to the puppet master. In this 201 class, we’ll work on the first half by migrating our existing repo structure into r10k.
Review and Setup
If we review the puppet-tutorial repo’s master branch, we have a standard directory layout that you should be somewhat familiar with now:
In Puppet and Git 101, we looked at how to add our existing puppet code to our repo. We’re going to take a quick look at how to create a branch, add some code, commit it, and push it to our repo.
Create a Branch
For lack of something significant to do right now, we’ll add a notify command to the node definition for puppet.nelson.va. To do so, we will checkout a new branch called, appropriately, notify. You can call your branches whatever you want, I suggest you simply be consistent in your naming scheme. At work, I use a combination of a ticket number and a one or two word description of the feature, separated by hyphens. Normally our branch is going to be short lived and only exist locally (we’re going to make an exception to that for demo purposes), so it would be a moot, but it’s still a good habit to be in.
[root@puppet puppet]# git branch
[root@puppet puppet]# git checkout -b notify
Switched to a new branch 'notify'
[root@puppet puppet]# git branch
Now that we’ve set up a puppet master and puppetized template, created a sample manifest, and started creating our own module, it’s time to take a few moments to talk about using Puppet with a version control systems (VCS). This article is mainly for those new to VCS at all or new to Git; those very familiar will want to skim or skip this article entirely.
What we have done so far is adding and removing a few lines in a couple files, and we’ve treated it as such. But it’s so much more. Writing code that represents an infrastructure state and using software to implement it is the root of two important IT movements: DevOps and the Software Defined Data Center (SDDC). You write code, puppet creates the infrastructure according to your instructions. Need something changed? Update your code, puppet takes care of the rest. What if you mess up? That’s where version control comes into play.
Version control, among other benefits, gives us the option to look at our code at points in time and to track changes over time, usually with some level of audit detail. If I make a change today and everything runs fine for a few days before blowing up, I can use version control to track the changes made to see if someone else made a change in the interval or perhaps go back to the version prior to my change. Without version control, you have no functional ability to audit your changes and revert the state of your code to a particular point in time.
There are a number of different version control systems that you can use. Subversion has been a popular VCS, though it has some long standing limitations and has been losing favor for a while. Git is a newer distributed version control system (DVCS) that has gained massive popularity by addressing some of the limitations of non-distributed VCSes and encouraging public development via Github.com and other cloud DVCS providers. We’re going to focus on Git due to its popularity, the plethora of examples of Puppet + Git available on the internet, and the ability to leverage Github.
In an upcoming series, I’ll be writing about Puppet and Git. As part of the research, I spent a number of hours looking at existing tools for post-receive hooks that were compatible with Github and r10k. In the end, my research went a completely different way and my first effort didn’t pan out, but I did learn from the experience and thought that sharing it might help others.
I was attempting to take an integrated puppet/r10k installation supporting dynamic environments and add a post-receive hook. The current workflow finished up with having to log into the puppet master, su to root/sudo and run r10k to deploy. The primary goal of the hook was to eliminate this step. This would not only simplify the workflow, but also increase security (less people have to have root access) and eliminate mistakes (Why isn’t my change visible? Oops I forgot to run r10k). The concept of hooks is fairly simple – when certain git activities occur, programs are called – but I needed to put things together. I’m on this box, I do my git work and push it to origin, then I need origin to do … something … and tell the puppet master to do … something else.
My initial research was focused on identifying the somethings. A common solution is to install gitolite on a node and make that the origin. It can then call an external program that SSH’s to the master and runs r10k. I eliminated this option because it’s either another node to manage or another service on an existing node, plus I have to perform backups of the git repo. I’d rather use Github at home or Stash at work to foist some of those responsibilities off on others.
In March, I created my PowerCLI GitHub Repo with just two cmdlets. By the end of March, I had many more cmdlets in the repo. Here are the updates:
Via http://tenthirtyam.org/per-cluster-cpu-and-memory-utilization-and-capacity-metrics-with-powercli/ (@medavamshi):
- Get-ClusterStats – A very detailed report on current Cluster resources and rough estimates of resources available after 1 or 2 cluster member failures. Useful in predicting failure scenarios as well as an eyeball view of capacity management.
Via http://hostilecoding.blogspot.com/ (@hostilecoding):
- Edit-v10VMs – An alternative GUI to vCenter that can edit VMs with vHW 10. Useful for those without vCenter or who do not like the vSphere Web Client.
Via http://myvirtualcloud.net/?p=5924 (@StevenPoitras and @andreleibovici) are a pair of cmdlets useful for stress-testing your storage (and vCenter, if you’re not careful):
- Clone-VM – Spin up a specified number of clones of the named VM, using VAAI by default.
- Unclone-VM – Provide the name of the cloned VM and stop/delete all the clones.
Via http://pelicanohintsandtips.wordpress.com/2014/03/13/creating-multiple-virtual-machines-with-powercli/ is a single cmdlet for Template deployments
- Deploy-Template – Use an existing Template and OSCustomizationSpec to deploy multiple instances of a Template into a specified Datacenter/Folder with sequential IPs.
Via http://www.shogan.co.uk/vmware/three-powercli-scripts-for-information-gathering-vms-hosts-etc/ (@shogan85) come three cmdlets that all have the option to output to CSV as well:
- Get-VMHostBIOSInfo – Report on the Model and BIOS of all VMHosts attached to your connected vCenter.
- Get-VMHostESXInfo – Report on the ESX(i) version and build of all VMHosts attached to your connected vCenter.
- Get-VMHardwareInfo – Report on the vHW version of all VMs in a specified datacenter.
Via http://hostilecoding.blogspot.com/ (@hostilecoding):