Puppet and Git, 204: r10k Workflow for Existing Module

We’ve installed and configured r10k, are using it for deployments, and have a workflow for new modules. More commonly, we will be working on existing modules, a slightly different workflow. We can examine this new workflow by modifying the base module only.

Workflow to Modify Existing Module

Unlike adding a new module, the Puppetfile only needs to be modified to reflect the feature branch. This is where the workflows diverge: instead of requiring a merge, commit, and push, we can create a temporary branch and just delete it when we’re done. Only the module branch needs merged. We’ll show this by making a simple change, modifying Dave’s name. We have another Dave Smith who works here, so we’ll add a middle initial and the name of Dave’s organization, to prevent confusion.

The feature branch is just called dave. We’ll work on the module repo first. Make sure you’re in master and checkout the new branch. Dave’s description should be updated to “Dave G. Smith – IT Administrator” – everything will be alright unless they hire another Dave G. Smith over there. Commit the change and push it.
Continue reading

Puppet 3.6.1 Updates

If you’ve been following with the Puppet series, your VMs probably started with Puppet 3.4.x or earlier. In the time since, Puppet has released up through v3.6.1 that brings a lot of improvements. However, if you simply upgrade your master and nodes, you’ll run into a few warnings about deprecations and future deprecations. Let’s take a look at the issues and how to resolve them. As always, read the release notes so that you understand the changes and test in a lab to ensure there is no negative impact.

Note: You MUST upgrade your nodes to v3.6.1 as well as the master, or you may receive fatal errors on the nodes. We haven’t gotten there yet, but if you have mcollective installed and configured, it’s a great way to upgrade your nodes at the same time.

Here’s the first item you’ll see:

[rnelson0@puppet ~]$ sudo puppet agent --test --noop
Warning: Setting modulepath is deprecated in puppet.conf. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1067:in `each')
Warning: Setting manifestdir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1071:in `each')

You can fix this by implementing environment directories. Here’s the diff I made:

[rnelson0@puppet ~]$ diff puppet.conf.org /etc/puppet/puppet.conf
15,16c15
<     modulepath = /etc/puppet/environments/$environment/modules:/opt/puppet/share/puppet/modules
<     manifestdir = /etc/puppet/environments/$environment/manifests
---
>     environmentpath = $confdir/environments

If you actually do have global modules under /opt, add a basemodulepath key and value. Now when you run another test, you may see some errors as it “fixes” itself. Run it a second time and you’ll see this:

[rnelson0@puppet ~]$ sudo puppet agent --test --noop
...
Warning: The package type's allow_virtual parameter will be changing its default value from false to true in a future release. If you do not want to allow virtual packages, please explicitly set allow_virtual to false.
   (at /usr/lib/ruby/site_ruby/1.8/puppet/type.rb:816:in `set_default')

This is a warning that something will be deprecated. You can read about the issue here. As the link says, it’s easy to fix this. In your puppet repo, add these lines to the top of manifests/site.pp:

Package {
  allow_virtual => true,
}

If you run puppet again, you’ll notice the warnings are gone!

One last note, if you get some spurious warnings, restart the puppet master service. In my lab, I didn’t need to do this, but in production I had to. I assume it’s because I did something out of order, but I couldn’t identify what that was.

Puppet and Git, 203: r10k Workflow for New Module

Welcome back to our Puppet and Git 200-series classes. With r10k installed and configured, today we can focus on workflows. The first workflow is for a new module, either a brand new module you are creating or simply a “new to you” module, such as importing more modules from the Forge. In our classrom, we will add a single module from the forge and update the base module to make use of it. This will give us a good understanding of the workflow for r10k.

Workflow To Add A New Module

The first step in our workflow is to decided on a module to add to our setup. If you have a particular module you want to use, feel free to substitute it below. I’ve chosen saz-motd, a very simple module that is visible when installed, but will not have a material impact on your nodes. We can see right now that there is no message of the day, so we’ll know when we’re done:

[root@puppet ~]# cat /etc/motd
[root@puppet ~]#

Note: We’ll add our module to a feature branch below. It’s a simple module, so this is fine. More complex modules, such as those that include additional facts and functions, should always be installed on the master first to ensure the plugins are synchronized, which means adding them to production. This was discussed on IRC so I don’t have a link to documentation to show how this works; this is the closest I could find. I’ll mention it again when we install such a module, but I wanted to mention it in case the module you chose provides custom facts/functions.

Continue reading

Puppet and Git, 202: r10k Setup – Conversion, Deployment

Welcome back! In our 201 class, we installed r10k, but we still haven’t used it. There’s two tasks we need to complete. First, the existing repo is incompatible with r10k’s dynamic management of modules, so we’ll convert its contents to the proper format. Once that is done, we can deploy dynamic environments using r10k.

Convert existing repo

Check out a clone of the existing puppet repo, rnelson0/puppet-tutorial. As mentioned previously, you can do this on the puppet master, as I will do, or you can perform it on another machine. If you’re on the master, you want to clone the repo into a different directory. After cloning it, check out a new branch called production:

Continue reading

Puppet and Git, 201: r10k Setup – Installation

I know you’re probably anxious to get started with managing your infrastructure, but we’re going to stay distracted by Git for a little longer. In the 100 series, we saw some examples of how to migrate your manifests and modules into Git and how to make changes to your manifests through branches. The setup is a little primitive, but acceptable for a lab – everything is is either done by root or involves pushing changes as a user and pulling them as root, and changes are tested in production. I’d like to introduce you to a tool called r10k that will help us create dynamic branches for testing and decouple our workflow from direct access to the puppet master. In this 201 class, we’ll work on the first half by migrating our existing repo structure into r10k.

Review and Setup

If we review the puppet-tutorial repo’s master branch, we have a standard directory layout that you should be somewhat familiar with now:

Continue reading

Puppet and Git, 102: Feature Branches

In Puppet and Git 101, we looked at how to add our existing puppet code to our repo. We’re going to take a quick look at how to create a branch, add some code, commit it, and push it to our repo.

Create a Branch

For lack of something significant to do right now, we’ll add a notify command to the node definition for puppet.nelson.va. To do so, we will checkout a new branch called, appropriately, notify. You can call your branches whatever you want, I suggest you simply be consistent in your naming scheme. At work, I use a combination of a ticket number and a one or two word description of the feature, separated by hyphens. Normally our branch is going to be short lived and only exist locally (we’re going to make an exception to that for demo purposes), so it would be a moot, but it’s still a good habit to be in.

[root@puppet puppet]# git branch
* master
[root@puppet puppet]# git checkout -b notify
Switched to a new branch 'notify'
[root@puppet puppet]# git branch
  master
* notify

Continue reading

Puppet and Git, 101: Git Basics

Now that we’ve set up a puppet master and puppetized template, created a sample manifest, and started creating our own module, it’s time to take a few moments to talk about using Puppet with a version control systems (VCS). This article is mainly for those new to VCS at all or new to Git; those very familiar will want to skim or skip this article entirely.

What we have done so far is adding and removing a few lines in a couple files, and we’ve treated it as such. But it’s so much more. Writing code that represents an infrastructure state and using software to implement it is the root of two important IT movements: DevOps and the Software Defined Data Center (SDDC). You write code, puppet creates the infrastructure according to your instructions. Need something changed? Update your code, puppet takes care of the rest. What if you mess up? That’s where version control comes into play.

Version control, among other benefits, gives us the option to look at our code at points in time and to track changes over time, usually with some level of audit detail. If I make a change today and everything runs fine for a few days before blowing up, I can use version control to track the changes made to see if someone else made a change in the interval or perhaps go back to the version prior to my change. Without version control, you have no functional ability to audit your changes and revert the state of your code to a particular point in time.

There are a number of different version control systems that you can use. Subversion has been a popular VCS, though it has some long standing limitations and has been losing favor for a while. Git is a newer distributed version control system (DVCS) that has gained massive popularity by addressing some of the limitations of non-distributed VCSes and encouraging public development via Github.com and other cloud DVCS providers. We’re going to focus on Git due to its popularity, the plethora of examples of Puppet + Git available on the internet, and the ability to leverage Github.

Continue reading

Manifest and Module Organization, Take One

In the last article, we learned how to import modules from the puppet forge. We created a very simple, but disorganized, site manifest. We need to create some organization, which will give us the ability to apply different settings to different nodes. Here’s the manifest we ended up with:

class { '::ntp':
  servers => [ '0.pool.ntp.org', '2.centos.pool.ntp.org', '1.rhel.pool.ntp.org'],
}

user { 'dave':
  ensure     => present,
  uid        => '507',
  gid        => '507',
  shell      => '/bin/bash',
  home       => '/home/dave',
  managehome => true,
}

group { 'dave':
  ensure => 'present',
  gid    => '507',
}

include ::ssh
::ssh::server::configline { 'PermitRootLogin': value => 'yes' }

The manifest includes two modules from the puppet forge and two resources managed by puppet, one user and one group. These resources, however, are going to applied to every agent that connects. As we grow the manifests, we’re going to meet some resources that are only needed on certain agents – web servers, web apps, etc. Let’s take what we have and organize it better.

Continue reading

Adding modules to your Puppet master

In the last post, we built a Puppet master. The master is ready for agents to connect, but currently it doesn’t actually do very much when they do. Our site.pp manifest is very boring:

[root@puppet ~]# cat /etc/puppet/manifests/site.pp
notify {"Agent connection is successful": }

All agents get the same manifest, and it’s just a notify statement. We want to do more than that, we want to actually manage some important configuration. Before we get into that, I want to mention a few basics that you may want to review on your own. We’ll touch on these items throughout this and future Puppet blogs, with the expectation that you’ll have read up on them already or will reference as needed.

Continue reading

Creating a Puppet Master

Puppet for vSphere Admins

Over the last four weeks, we looked at Auto Deploy, which is automation for our VMhost provisioning process. Next up, we’re going to look at Puppet, a tool to automate our VM and Guest OS provisioning.

Recently, I have been working on deploying Puppet by Puppet Labs in our work environment. Puppet is a provisioning and configuration management system. It has been made famous by its ability to simplify cloud management for those running at scale, such as the Obama for America campaign that leveraged AWS and Puppet. Manually provisioning and configuring nodes scales linearly, or worse – 5,000 nodes requires at least 1,000 times the resources as 5 nodes. Automation with a tool like Puppet scales much more gracefully. Managing 5,000 nodes is only incrementally more difficult than managing 5, and growing to 50,000 or shrinking to 500 is just as easy. There are a number of other similar products you might be interested in – Chef, Ansible, and Salt to name a few.

I am interested in Puppet for two primary reasons. First, it has a lot of mindshare and a friendly community. You can easily find numerous blogs addressing common problems, there’s an active irc channel for problems you can’t solve with the help of a search engine, there’s a gigantic public module repository (called Puppet Forge), and you’ll find many candidates who already know Puppet as you grow your team. Second, VMware has invested $30M in Puppet Labs. The increased interaction and development has already resulted in Puppet adding some VMware cloud provisioning features and should ensure those features mature. This should help round out their Software Defined DataCenter (SDDC) efforts.

Continue reading