Connecting Puppetboard to Puppet Enterprise

Last week, I moved the home lab to Puppet Enterprise. One of the things I love about PE is the Console. However, I am a member of Vox Pupuli and we develop Puppetboard (the app AND the module) so it is convenient for me to use it and tie it into PE as well. Though the two overlap, each has functionality the other doesn’t. I really love the count of agent runs by status on the Puppetboard Overview page, for instance. After my migration, however, my previously-working puppetboard just gave me HTTP 500 errors. Fixing it took some wrangling. Thanks to Tim Meusel for his assistance with the cert issue.

First, let’s look at the existing manifest and hiera data for my profile::puppetboard class:

Continue reading

What goes in a Puppet Role or Profile?

The Roles and Profiles pattern by Craig Dunn is a very common pattern used by Puppet practitioners. I’ve written about it before. One of the most common questions I see is, what goes into a Role or Profile class? Craig’s article provides some guidelines, specifically these two:

 

  • A role includes one or more profiles to define the type of server
  • A profile includes and manages modules to define a logical technical stack

 

Those are pretty helpful, but it’s not an exhaustive list, nor does it describe what is prohibited in each type of class. While the main goal of the pattern is composition, I have my own guidelines I follow that may help others:

Roles

  • No parameters
  • Includes profile classes
  • [Rarely] Ordering of resources that come from two separate profiles
  • Contains nothing else.

Here’s an example role for an application server:

role appX {
  include profile::base
  include profile::apache
  include profile::appX
  Package<| tag == 'profile::apache' |> -> Package <| tag == 'profile::appX' |>
}

Profiles

  • Optional parameters
  • Includes component modules
  • Includes basic resource types (built-in or from component modules)
  • Calls functions, include hiera_*() and lookup()
  • Traverses and manipulates variables to process their data
  • Conditionals (limited)
  • Ordering of resources, within the profile
  • May call other profiles, but should be used sparingly
  • If the code is >100 lines, consider separating the profile class into its own module, or finding an existing component module that include the functionality (100 is a very arbitrary number, feel free to adjust it to a number indicating when you want to start thinking about this option)

Here’s an example of a profile that calls other profiles:

class profile::base {

  # Include OS specific base profiles.
  case $::kernel {
    'linux': {
      include profile::base::linux
    }
    'windows': {
      include profile::base::windows
    }
    'JUNOS': {
      include profile::base::junos
    }
    default: {
      fail ("Kernel: ${::kernel} not supported in ${module_name}")
    }
  }
}

Here’s an example of a more complex profile that has parameters and includes other component modules, basic resources, functions, iteration, conditionals, and even another profile:

class profile::base::linux (
  $yumrepo_url,
  $cron_purge          = true,
  $domain_join         = false,
  $metadata_expire     = 21600, # Default value for yum is 6 hours = 21,600 seconds
  $sudo_confs          = {},
  $manage_firewall     = true,  # Manage iptables
  $manage_puppet_agent = true,  # Manage the puppet-agent including upgrades
) {
  # Manage the basics, but allow users to override some management components with flags
  if $manage_firewall {
    include profile::linuxfw
  }
  if $manage_puppet_agent {
    include puppet_agent
  }

  include ntp
  include rsyslog::client
  include motd

  # SSH server and client
  include ssh::server
  include ssh::client

  # Sudo setup
  include sudo
  $sudo_confs.each |$group, $config| {
    sudo::conf{ $group:
      * => $config,
    }
  }

  yumrepo {'local-el':
    ensure          => present,
    descr           => 'Local EL - x86_64',
    baseurl         => $yumrepo_url,
    enabled         => 1,
    gpgcheck        => 0,
    metadata_expire => $metadata_expire,
  }
  Yumrepo['local-el'] -> Package<| |>

  # Ensure unmanaged cron entries are removed
  resources { 'cron':
    purge => $cron_purge,
  }

  if $domain_join {
    include profile::domain_join
  }
}

Summary

The Roles and Profiles pattern is all about composition. The Style Guide helps you with layout and semantic choices. It doesn’t hurt to add your own rules about content and design, too. There’s no defined Best Practice here, but I hope these guidelines help you shape your own practice. Enjoy!

Migrating my home lab from Puppet OpenSource to Puppet Enterprise

I have been using Puppet Enterprise at work and Puppet OpenSource at home for a few years now. There’s a lot to love about both products, but since work uses PE and new features tend to land there first, I have been thinking about trying PE at home as well. I don’t have a large fleet or a large change velocity, so I think the conversion of the master might take some work but the agents will be easy. This will let me play with PE-only functions in my lab (specifically the PE Pipeline plugin for Jenkins) and reduces concern about drift between lab findings and work usage. It does have the downside that some of my blog articles, which I was always assured would work with the FOSS edition, may not be as foolproof in the future. However, I rarely saw that as a problem going the other way in the past, with mcollective management being the only exception. I haven’t written about mcollective much, so I think this is worth the tradeoff.

I am going to migrate my systems, rather that start fresh. This article was written as the migration happened, so if you pursue a similar migration, please read the whole article before starting – I did some things backwards or incomplete, and made a few mistakes that you could avoid. It was also written as something of a stream of consciousness. I’ve tried to edit it as best I can without losing that flow. I do hope you find it valuable.

Pre-Flight Checklist

Puppet Enterprise is a commercial product. You can use it for free with up to 10 nodes, though. This is perfect for my 9 managed-node home network. After that, it costs something around $200/node/year. Make sure you stay within that limit or pony up – we are responsible adults, we pay for the products we use. If you replace nodes, you can deactivate those nodes so they don’t continue to eat into your license count.

Continue reading

Making the Puppet vRealize Automation plugin work with vRealize Orchestrator

I’m pretty excited about this post! I’ve been building up Puppet for vSphere Admins for a few years now but the final integration aspects between Puppet and vSphere/vCenter were always a little clunky and difficult to maintain without specific dedication to those integration components. Thanks to Puppet and VMware, that’s changed now.

Puppet announced version 2.0 of their Puppet Plugin for vRealize Automation this week. There’s a nice video attached, but there’s one problem – it’s centered on vRealize Automation (vRA) and I am working with vRealize Orchestrator (vRO)! vRO is included with all licenses of vCenter, whereas vRA is a separate product that costs extra, and even though vRA requires a vRO engine to perform a lot of its work, it abstracts a lot of the configuration and implementation details away that vRO-only users need to care about. This means that much of the vRA documentation and guides you find, for the Puppet plugin or otherwise, are always missing some of the important details needed to implement the same functionality – and sometimes won’t work at all if it relies on vRA functionality not available to us.

Don’t worry, though, the Puppet plugin DOES work with vRO! We’ll look at a few workflows to install, run, and remove puppet from nodes and then discuss how we can use them within larger customized workflows. You must already have an installed vRealize Orchestrator 7.x instance configured to talk to your vCenter instance. I’m using vRO 7.0.0 with vCenter 6.0. If you’re using a newer version, some of the dialogs I show may look a little different. If you’re still on vRO 6.x, the configuration will look a LOT different (you may have to do some research to find the equivalent functionality) but the workflow examples should be accurate.

Puppet provides a User Guide for use with a reference implementation. I’ll be mostly repeating Part 2 when installing and configuring, but reality tends to diverge from reference so we’ll explore some non-reference details as well.

Continue reading

Automating Puppet tests with a Jenkins Job, version 1.1

Today, let’s build on version 1.0 of our Jenkins job. We are running builds against every commit, but when someone opens a pull request, they don’t get automated builds or feedback. If the PR submitter even knows about Jenkins, and has network access and a login, they can look at it to find out how the tests went, but most people aren’t going to have that visibility (especially if your Jenkins server is private, as in this example setup). We need to make sure Jenkins is aware of the pull request and that it updates the PR with the status. Our end goal is for each PR to start a Jenkins build and update the PR with a successful check when done:

To get there, we will install and configure a new plugin and configure our job to use the plugin.

Continue reading

Automating Puppet tests with a Jenkins Job, version 1.0

As I’ve worked through setting up Jenkins and Puppet (and remembering my password!), I created a job to automate rspec tests on my puppet controlrepo. I am sure I will go through many iterations of this as I learn more, so we’ll just call this version 1.0. The goal is that when I push a branch to my controlrepo on GitHub, Jenkins automagically runs the test. Today, we will ensure that Jenkins is notified of activity on a GitHub repo, that it spins up a clean test environment without any left over files that may inadvertently assist, and run the tests. What it will NOT do is notify anyone – it won’t work off a Pull Request and provide feedback like Travis CI does, for instance. Hopefully, I will figure that out soon.

The example below is using GitHub. You can certainly make this work with BitBucket, GitLab, Mercurial, and tons of other source control systems and platforms, but you might need some additional Jenkins Plugins. It should be pretty apparent where to change Git/GitHub to the system/platform you chose.

Creating A Job

From the main view of your Jenkins instance, click New Item. Call it whatever you want, choose Freestyle project as the type, and click OK. The next page is going to be where we set up all the parameters for the job. There are tabs across the top AND you can scroll down; you’ll see the same selection items either way. Going from the top to the bottom, the settings that we want:

Continue reading

Jenkins Tricks – Password Recovery and Job Exports

I’m finally getting back to Jenkins, which I started waaaay back in November (here and here). Unfortunately, I kind of forgot my password. Well, that’s embarrassing! I also want to redo the manifest using maestrodev/rvm which means starting over, so I need to back things up. The manual for Jenkins and the results on Google can be overwhelming sometimes, so I thought I’d share what I learned to hopefully save someone else.

Password Recovery

There’s a few ways I found to recover your password. One suggestion is to disable all security, delete your user, re-enable security and allow signups, and then recreate the same user and things should just “work”. Part of the reason you have to do this is that once you disable security, you can’t change the password for your user; only the user can. That’s … frustrating.

Disable security by editing $JENKINS_HOME/config.xml, /var/lib/jenkins/config.xml on my instance. I was able to get away with disabling it by changing <useSecurity>true</useSecurity> to false, though the article suggests removing two other lines. Restart the service with systemctl restart jenkins or equivalent and now you’re able to get in and recreate some users.

Continue reading

Upgrading Puppet OpenSource Agents with puppet_agent and jlambert121/puppet

Last week, I discussed how to upgrade your Linux and Windows Puppet Enterprise agents with puppet_agent. Today, we will look at two different ways to upgrade your Puppet OpenSource agents, first with puppet_agent and then with jlambert121/puppet. There are many other modules that you can use to manage puppet, some of which offer similar functionality that you may be able to apply some of these lessons to.

Why are we looking at two different modules today? I use jlambert121’s module to manage my puppetserver already. Unfortunately, it is puppet 4 only and I still had some puppet 3 agents hanging around according to puppetboard. I also have some other modules that are puppet 4 only, like the latest version of puppetlabs/ntp. I could use jlambert121/puppet to upgrade my agents that are already on puppet 4, but need something to help get my puppet 3 agents over the hump – that’s where puppet_agent comes in. We will start with that.

Continue reading

Tracking Puppet agent status with Puppetboard

I manage about 15 VMs in my home lab with OpenSource Puppet. Unlike Puppet Enterprise, there’s no Console to help you determine when agents stop checking in or encountering any problems applying their catalog. With only 15 agents it’s not a huge problem, but it always rears its head at the worst time. I decided to deploy Puppetboard at home to get a similar view to PE’s Console. As is always the way, we’re going to deploy that using a puppet module, Vox Pupuli’s puppet/puppetboard. Now, I’ve got Puppetboard running, but it’s served via HTTP and communicates with puppetdb via HTTP, so there’s room to improve. I’m not sure I’ll get to that, since this is on a private network, but something to keep in mind if you’re implementing this in a more sensitive deployment.

Special thanks to Tim Meusel for his assistance with some of the settings!

The first step is to add the module and its dependencies to the Puppetfile and .fixtures.yml. You may have a few of these modules already, but I’ve bumped them to the latest version.

Continue reading

Learn about Puppet module releases – via email!

We’ve all found ourselves anxiously awaiting a new module release of our favorite puppet module on the forge. It might be coming down the pipe today, next month, or who knows when, but you’re going to need it as soon as it’s released. The forge fairly recently announced an RSS feed per module. You can see this in your RSS reader, if you use one, or get email notifications with a number of services, a few of which are mentioned in that blog. I have wanted to play with If This, Then That for a while, so I decided to see if I could figure out how to get emails through it.

You’ll need an IFTTT account, at least one puppet module you want to receive alerts on, and of course an email address to receive the alert. Once you sign up for IFTTT, go to My Applets and choose New Applet. On the next page, it’s not obvious but click the +this text and it brings you to a selection page. Click on Feed, then New Feed Item. Now switch over to your target module on the forge and find the rss link. The pattern’s pretty easy, just add /rss to the module’s link.

Now click on +that and select Email, then Send me an email. If you want, adjust the contents of the email – I found the default field setup okay – and then hit Create Action. Click on the slider to the right of Receive notifications when this Applet runs and hit Finish

ifttt-fig-1

If you want to change anything, click the gear icon, adjust what you need and hit Save. Repeat for all the modules you want to receive notifications upon update. Here’s what an email will look like:

ifttt-fig-2

You can also view, update, and add additional applets using the IFTTT app for your phone or tablet. This might not be very pretty or exciting, but since I just want to get alerts without firing up an RSS reader, it’s good enough for me. Now to start updating my Puppetfile. Enjoy!