Linux OS Patching with Puppet Tasks

One of the biggest gaps in most IT security policies is a very basic feature, patching. Specific numbers vary, but most surveys show a majority of hacks are due to unpatched vulnerabilities. Sadly, in 2018, automatic patching on servers is still out of the grasp of many, especially those running older OSes.

While there are a number of solutions out there from OS vendors (WSUS for Microsoft, Satellite for RHEL, etc.), I manage a number of OSes and the one commonality is that they are all managed by Puppet. A single solution with central reporting of success and failure sounds like a plan. I took a look at Puppet solutions and found a module called os_patching by Tony Green. I really like this module and what it has to offer, even though it doesn’t address all my concerns at this time. It shows a lot of promise and I suspect I will be working with Tony on some features I’d like to see in the future.

Currently, os_patching only supports Red Hat/Debian-based Linux distributions. Support is planned for Windows, and I know someone is looking at contributing to provide SuSE support. The module will collect information on patching that can be used for reporting, and patching is performed through a Task, either at the CLI or using the PE console’s Task pane.

Setup

Configuring your system to use the module is pretty easy. Add the module to your Puppetfile / .fixtures.yml, add a feature flag to your profile, and include os_patching behind the feature flag. Implement your tests and you’re good to go. Your only real decision is whether you default the feature flag to enabled or disabled. In my home network, I will enable it, but a production environment may want to disable it by default and enable it as an override through hiera. Because the fact collects data from the node, it will add a few seconds to each agent’s runtime, so be sure to include that in your calculation.

Adding the module is pretty simple, Here are the Puppetfile / .fixtures.yml diffs:

# Puppetfile
mod 'albatrossflavour/os_patching', '0.3.5'

# .fixtures.yml
fixtures:
  forge_modules:
    os_patching:
      repo: "albatrossflavour/os_patching"
      ref: "0.3.5"

Next, we need an update to our tests. I will be adding this to my profile::base, so I modify that spec file. Add a test for the default feature flag setting, and one for the non-default setting. Flip the to and not_to if you default the feature flag to disabled. If you run the tests now, you’ll get a failure, which is expected since there is no supporting code in the class yet.(there is more to the test, I have only included the framework plus the next tests):

require 'spec_helper'
describe 'profile::base', :type => :class do
  on_supported_os.each do |os, facts|
    let (:facts) {
      facts
    }

    context 'with defaults for all parameters' do
      it { is_expected.to contain_class('os_patching') }
    end

    context 'with manage_os_patching enabled' do
      let (:params) do {
        manage_os_patching: false,
      }
      end

      # Disabled feature flags
      it { is_expected.not_to contain_class('os_patching') }
    end
  end
end

Finally, add the feature flag and feature to profile::base (the additions are in italics):

class profile::base (
  Hash    $sudo_confs = {},
  Boolean $manage_puppet_agent = true,
  Boolean $manage_firewall = true,
  Boolean $manage_syslog = true,
  Boolean $manage_os_patching = true,
) {
  if $manage_firewall {
    include profile::linuxfw
  }

  if $manage_puppet_agent {
    include puppet_agent
  }
  if $manage_syslog {
    include rsyslog::client
  }
  if $manage_os_patching {
    include os_patching
  }
  ...
}

Your tests will pass now. That’s all it takes! For any nodes where it is enabled, you will see a new fact and some scripts pushed down on the next run:

[rnelson0@build03 controlrepo:production]$ sudo puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/os_patching.rb]/ensure: defined content as '{md5}af52580c4d1fb188061e0c51593cf80f'
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for build03.nelson.va
Info: Applying configuration version '1535052836'
Notice: /Stage[main]/Os_patching/File[/etc/os_patching]/ensure: created
Info: /Stage[main]/Os_patching/File[/etc/os_patching]: Scheduling refresh of Exec[/usr/local/bin/os_patching_fact_generation.sh]
Notice: /Stage[main]/Os_patching/File[/usr/local/bin/os_patching_fact_generation.sh]/ensure: defined content as '{md5}af4ff2dd24111a4ff532504c806c0dde'
Info: /Stage[main]/Os_patching/File[/usr/local/bin/os_patching_fact_generation.sh]: Scheduling refresh of Exec[/usr/local/bin/os_patching_fact_generation.sh]
Notice: /Stage[main]/Os_patching/Exec[/usr/local/bin/os_patching_fact_generation.sh]: Triggered 'refresh' from 2 events
Notice: /Stage[main]/Os_patching/Cron[Cache patching data]/ensure: created
Notice: /Stage[main]/Os_patching/Cron[Cache patching data at reboot]/ensure: created
Notice: Applied catalog in 54.18 seconds

You can now examine a new fact, os_patching, which will shows tons of information including the pending package updates, the number of packages, which ones are security patches, whether the node is blocked (explained in a bit), and whether a reboot is required:

[rnelson0@build03 controlrepo:production]$ sudo facter -p os_patching
{
  package_updates => [
    "acl.x86_64",
    "audit.x86_64",
    "audit-libs.x86_64",
    "audit-libs-python.x86_64",
    "augeas-devel.x86_64",
    "augeas-libs.x86_64",
    ...
  ],
  package_update_count => 300,
  security_package_updates => [
    "epel-release.noarch",
    "kexec-tools.x86_64",
    "libmspack.x86_64"
  ],
  security_package_update_count => 3,
  blocked => false,
  blocked_reasons => [],
  blackouts => {},
  pinned_packages => [],
  last_run => {},
  patch_window => "",
  reboots => {
    reboot_required => "unknown"
  }
}

Additional Configuration

There are a number of other settings you can configure if you’d like.

  • patch_window: a string descriptor used to “tag” a group of machines, i.e. Week3 or Group2
  • blackout_windows: a hash of datetime start/end dates during which updates are blocked
  • security_only: boolean, when enabled only the security_package_updates packages and dependencies are updated
  • reboot_override: boolean, overrides the task’s reboot flag (default: false)
  • dpkg_options/yum_options: a string of additional flags/options to dpkg or yum, respectively

You can set these in hiera. For instance, my global config has some blackout windows for the next few years:

os_patching::blackout_windows:
  'End of year 2018 change freeze':
    'start': '2018-12-15T00:00:00+1000'
    'end':   '2019-01-05T23:59:59+1000'
  'End of year 2019 change freeze':
    'start': '2019-12-15T00:00:00+1000'
    'end':   '2020-01-05T23:59:59+1000'
  'End of year 2020 change freeze':
    'start': '2020-12-15T00:00:00+1000'
    'end':   '2021-01-05T23:59:59+1000'
  'End of year 2021 change freeze':
    'start': '2021-12-15T00:00:00+1000'
    'end':   '2022-01-05T23:59:59+1000'

Patching Tasks

Once the module is installed and all of your agents have picked up the new config, they will start reporting their patch status. You can query nodes with outstanding patches using PQL. A search like inventory[certname] {facts.os_patching.package_update_count > 0 and facts.clientcert !~ 'puppet'} can find all your agents that have outstanding patches (except puppet – kernel patches require reboots and puppet will have a hard time talking to itself across a reboot). You can also select against a patch_window selection with and facts.os_patching.patch_window = "Week3" or similar. You can then provide that query to the command line task:

puppet task run os_patching::patch_server --query="inventory[certname] {facts.os_patching.package_update_count > 0 and facts.clientcert !~ 'puppet'}"

Or use the Console’s Task view to run the task against the PQL selection:

Add any other parameters you want in the dialog/CLI args, like setting rebootto true, then run the task. An individual job will be created for each node, all run in parallel. If you are selecting too many nodes for simultaneous runs, use additional filters, like the aforementioned patch_window or other facts (EL6 vs EL7, Debian vs Red Hat), etc. to narrow the node selection [I blew up my home lab, which couldn’t handle the CPU/IO lab, when I ran it against all systems the first time, whooops!]. When the job is complete, you will get your status back for each node as a hash of status elements and the corresponding values, including return (success or failure), reboot, packages_updated, etc. You can extract the logs from the Console or pipe CLI logs directly to jq (json query) to analyze as necessary.

Summary

Patching for many of us requires additional automation and reporting. The relatively new puppet module os_patching provides helpful auditing and compliance information alongside orchestration tasks for patching. Applying a little Puppet Query Language allows you to update the appropriate agents on your schedule, or to pull the compliance information for any reporting needs, always in the same format regardless of the (supported) OS. Currently, this is restricted to Red Hat/Debian-based Linux distributions, but there are plans to expand support to other OSes soon. Many thanks to Tony Green for his efforts in creating this module!

Using Puppet Enterprise 2018’s new backup/restore features

I was pretty excited when I read the new features in Puppet Enterprise 2018.1. There are a lot of cool new features and fixes, but the backup/restore feature stood out for me. Even with just 5 VMs at home, I don’t want to rock the boat when rebuilding my master by losing my CA or agent certs, much less with a lot more managed nodes at work, and all the little bootstrap requirements have changed since I started using PE in 2014. Figuring out how to get everything running myself would be possible, but it would take a while and be out of date in a few months anyway. Then there is everything in PuppetDB that I do not want to lose, like collected facts/resources and run reports.

Not coincidentally, I still had a single CentOS 6 VM around because it was my all-in-one puppet master, and migrating to CentOS 7 was not something I looked forward to due to the anticipated work it would require. With the release of this feature, I decided to get off my butt and do the migration. It still took over a month to make it happen, between other work, and I want to share my experience in the hope it saves someone else a bit of pain.

Create your upgrade outline

I want to summarize the plan at a really high level, then dive in a bit deeper. Keep in mind that I have a single all-in-one master using r10k and my plan does not address multi-master or split deployments. Both of those deployment models have significantly different upgrade paths, please be careful if you try and map this outline onto those models without adjusting. For the all-in-one master, it’s pretty simple:

  • Backup old master
  • Deploy a new master VM running EL7
  • Complete any bootstrapping that isn’t part of the backup
  • Install the same version of PE
  • Restore the old master’s backup onto the new master
  • Run puppet
  • Point agents at the new master

I will cover the backup/restore steps at the end, so the first step to cover is deploying a new master. This part sounds simple, but if Puppet is currently part of your provisioning process and you only have one master, you’ve got a catch 22 situation – new deployments must talk to puppet to complete without errors, and if you deploy a new puppet master using the same process, it will either fail to communicate with itself since PE is not installed, or it will talk to a PE installation that does not reflect your production environment. We need to make sure that we have the ability to provision without puppet, or be prepared for some manual efforts in the deploy. With a single master, manual efforts aren’t that burdensome, but can still reduce accuracy, which is why I prefer a modified automated provisioning workflow.

A lot of bootstrapping – specifically hiera and r10k/code manager – should be handled by the restore. There were just a few things I needed to do:

  • Run ssh-keygen/install an existing key and attach that key to the git system. You can avoid this by managing the ssh private/public keys via file resources, but you will not be able to pull new code until puppet processes that resource.
  • SSH to your git server and accept the key. You can avoid this with the sshkey resource, with the same restriction.
  • Check your VMs default iptables/selinux posture. I suggest managing security policy via puppet, which should prevent remote agents from connecting before the first puppet run, but it’s also possible to prevent the master from communicating with itself with the wrong default policy.
  • Check the hostname matches your expectations. All of /etc/hosts, /etc/hostname, /etc/sysconfig/network should list the short and FQDN properly, and hostname; hostname -f should return the same values. /etc/resolv.conf may also need the search domain. Fix any issues before installing PE, as certs are generated during install, and having the wrong hostname result can cause cascading faults best addressed by starting over.

The restore should get the rest from the PE side of things. If your provisioning automation performs other work that you had to skip, make sure you address it now, too.

Installing PE is probably the one manual step you cannot avoid. You can go to https://support.puppet.com and find links to current and past PE versions. Make sure you get the EL7 edition and not the EL6 edition. I did not check with Support, but I assume that you must restore on the same version you backed up, I would not risk even a patch release difference.

Skipping the restore brings us to running the agent, a simple puppet agent -t on the master, or waiting 30 minutes for the run to complete on its own.

The final step may not apply to your situation. In addition to refreshing the OS of the master, I switched to a new hostname. If you’re dropping your new master on top of the existing one’s hostname/IP, you can skip this step. I forked a new branch from production called mastermigration. The only change in this branch is to set the server value in /etc/puppetlabs/puppet/puppet.conf. There are a number of ways to do this, I went with a few ini_setting resources and a flag manage_puppet_conf in my profile::base::linux. The value should only be in one of the sections main or agent, so I ensured it is in main and absent elsewhere:

  if $manage_puppet_conf {
    # These settings are very useful during migration but are not needed most of the time
    ini_setting { 'puppet.conf main server':
      ensure => present,
      path => '/etc/puppetlabs/puppet/puppet.conf',
      section => 'main',
      setting => 'server',
      value => 'puppet.example.com',
    }
    ini_setting { 'puppet.conf agent server':
      ensure => absent,
      path => '/etc/puppetlabs/puppet/puppet.conf',
      section => 'agent',
      setting => 'server',
    }
  }

During the migration, I can just set profile::base::linux::manage_puppet_conf: true in hiera for the appropriate hosts, or globally, and they’ll point themselves at the new master. Later, I can set it to false if I don’t want to continue managing it (while there is no reason you cannot leave the flag enabled, by leaving it as false normally you can ensure that changing the server name here does not take effect unless purposefully flip the flag; you could also parameterize the server name).

Now let’s examine the new feature that makes it go.

Backups and Restores

Puppet’s documentation on the backup/restore feature provides lots of detail. It will capture the CA and certs, all your currently deployed code, your PuppetDB contents including facts, and almost all of your PE config. About the only thing missing are some gems, which you should hopefully be managing and installing with puppet anyway.

Using the new feature is pretty simple, puppet-backup createor puppet-backup restore <filename> will suffice for this effort. There are a few options for more fine-grained control, such as backup/restore of individual scopes with --scope=<scopes>[,<additionalscopes>...], e.g. --scope=certs.

 

The backup will only backup the current PE edition’s files, so if you still have /etc/puppet on your old master from PE 3 days, that will not be part of the backup. However, files in directories it does back up, like /etc/puppetlabs/puppet/puppet.conf.rpmsave, will persist. This will help reduce cruft, but not eliminate it. You will still need to police on-disk content. In particular, if you accidentally placed a large file in /etc/puppetlabs, say the PE install tarball, that will end up in your backup and can inflate the size a bit. If you feel the backup is exceptionally large, you may want to search for large files in that path.

The restore docs also specify two commands to run after a restore when Code Manager is used. If you use CM, make sure not to forget this step:

puppet access login
puppet code deploy --all --wait 

The backup and restore process are mostly time-dependent on the size of your puppetdb. With ~120 agents and 14 days of reports, it took less than 10 minutes for either process and generated a ~1G tarball. Larger environments may expect the master to be offline for a bit longer, if they want to retain their full history.

Lab it up

The backup/restore process is great, but it’s new, and some of us have very ancient systems laying around. I highly recommend testing this in the lab. My test looked like this:

  • Clone the production master to a VM on another hostname/IP
  • Run puppet-backup create
  • Fully uninstall PE (sudo /opt/puppetlabs/bin/puppet-enterprise-uninstaller -p -d -y)
  • Remove any remaining directories with puppet in them, excepting the PE 2018 install files, to ensure all cruft is gone
  • Disable and uninstall any r10k webhook or puppet-related services that aren’t provided by PE itself.
  • Reboot
  • Bootstrap (from above)
  • Install PE (sudo /opt/puppetlabs/bin/puppet-enterprise-installer) only providing an admin password for the console
  • Run puppet-backup restore <backup file>
  • Run puppet agent -t
  • Make sure at least one agent can check in with puppet agent -t --server=<lab hostname> (clone an agent too if need be)
  • Reboot
  • Make sure the master and agent can still check in, Console works, etc.
  • If possible, test any systems that use puppet to make sure they work with the new master
  • Identify any missing components/errors and repeat the process until none are observed

I mentioned that I used PE3. My master had been upgraded all the way from version 3.7 to 2018.1.2. I’m glad I tested this, because there were some unexpected database settings that the restore choked on. I had to engage Puppet Support who provided the necessary commands to update the database so I could get a useful backup. This also allowed me to identify all of my bootstrap items and of course, gain familiarity and confidence with the process.

This became really important for me because, during my production migration, I ran into a bug in my provisioning system where the symptom presented itself through Puppet. Because I was very practiced with the backup/restore process, I was able to quickly determine PE was NOT the problem and correctly identify the faulty system. Though it took about 6 hours to do my “very quick” migration, only about an hour of that was actually spent on the Puppet components.

I also found a few pieces of managed files on the master where the code presumed the directory structure would already be there, which it turns out was not the case. I must have manually created some directories 4 years ago. I think the most common issues you would find at this point are dependencies and ordering, but there may be others. Either fix the code now or, if it would negatively affect the production server, prep a branch for merging just prior to the migration, with the plan to revert if you rollback.

I strongly encourage running through the process a few times and build the most complete checklist you can before moving on to production.

Putting it together

With everything I learned in the lab, my final outline looked like this:

  • Backup old master, export to another location
  • Deploy a new master VM running EL7 using an alternative workflow
  • Run ssh-keygen/install an existing key and attach that key to the git system
  • SSH to the git server and accept the key
  • Verify your VMs default iptables/selinux posture; disable during bootstrap if required
  • Validate the hostname is correct
  • Install PE
  • Restore the backup
  • [Optional] Merge any code required for the new server; run r10k/CM to ensure it’s in place on the new master
  • Run puppet
  • Point agents at the new master

Yours may look slightly different. Please, spend the time in the lab to practice and identify any missing steps, it’s well worth it.

Summary

Refreshing any system of significant age is always possible, and often fraught with manual processes that are prone to error. Puppet Enterprise 2018.1 delivered a new backup/restore process that automates much of this process. We have put together a rough outline, refined it in the lab, and then used it to perform the migration in production with high confidence, accounting for any components the backup did not include. I really appreciate this new feature and I look forward to refinements in the future. I hope that soon enough migrations should be as simple an effective as in-place upgrades.

Contributing to a Political Campaign as a Nerd

As I promised in my previous politics article, I will continue not to advocate for specific politics and remain non-partisan on my blog. I do encourage everyone, regardless of beliefs or party, to participate in politics, because it affects you whether you participate or not. Participation in our democracy can only improve it.

Over the past year or two, I have come to feel far more strongly about my politics. I live in the United States, and it’s impossible to pay attention to the news and not feel some kind of way about our Republic. This year, I decided that I needed to contribute more directly, not just passively partake in politics. If you already follow me on twitter, you know that I wear my politics on my sleeve. Like many readers, I regularly vote. Like many readers, I donate to campaigns. That’s not enough for me anymore. So I decided to do more, and reached out to some local campaigns to find out what I could offer.

I will admit that this is scary. I have spent almost 20 years working on a relatively narrow field of expertise within IT. I had no experience with politics. Going from 20 years of experience to 0 is intimidating – but if I did it, so can you. If you read this and want to contribute, I want you to know that you will do just fine. As divisive and loud and argumentative and nasty as politics seems on the evening news, every group I have worked with over the past year has been welcoming, very graceful about my lack of knowledge and mistakes, and very accommodating to how much time I have available. Please, don’t let your fear keep you out. Reach out if you have any questions!

The expertise we have is sorely needed, though, especially in political campaigns. You will quickly find out that most people involved in political campaigns are not computer experts in any way. Sure, they’re computer savvy as most people are nowadays, but there are significant gaps in that knowledge that needs filled. All you need to do is read the news and you will quickly hear about campaigns that are hacked and crowdsourced analysis of what’s on a politicians phone and overwhelming numbers of twitter and facebook shenanigans. Your help is needed and will be welcomed.

I joined a campaign and I had no idea what I was getting into. I did not find many others in tech who have shared their experiences joining their first campaign. I hope this article helps fill this gap a little bit – and if any readers are in the same situation, I would encourage you to blog about it as well! Alright, let’s get volunteering!

As I hit publish on this, there are 85 days until Election Day in the US. It is NOT too late to volunteer! Your assistance will be welcomed up until the final moments on Election Day, and there are always future elections to prepare for.

What to expect

When you click Sign Up on a campaign web site, you’ll be offered some “normal” work – canvassing, phone banking, putting a sign in your yard, etc. All things technical are notably missing from the list. To offer your technical expertise, you will have to reach out to the campaign directly. Many campaigns or candidates have a listed phone number on their website. If not, try looking at the county or state party’s website for a phone number, and inform them that you would like to get in contact with the campaign.

You will have a chance at some point to talk to the candidate or a campaign manager and make sure that’s who you want to work for. Treat it like a job interview! The campaign will ask what you can do, and you get to ask the campaign about what they will do if elected. Be honest! When I first talked to my campaign, I explained that I had 20 years of IT experience but no campaign experience. I was willing to take on things unknown, but I wanted to make sure they knew it would be new to me. I found out they had an experienced webmaster who would be providing me assistance if I joined. I also asked a lot of pointed policy questions to ensure that I would be happy if this candidate was elected. Get your questions answered and let the campaign know whether you want to join and what you can contribute.

General tech tasks

There are so many areas you can contribute to the technology side of a campaign, regardless of where in technology you work. Here’s a very short, very incomplete list of items you can help with:

  • Setting up a free Slack and teaching people how to use it (EVERYONE uses slack nowadays!)
  • Setting up a website and analytics
  • Configuring multi-factor authentication on all services
  • Setting up apps on phones and tablets
  • Answering questions about how to use a computer, application, or service, even if you’re just functioning as Google as a Service for really busy people
  • Providing a sounding board for anything technical, including how technical people and companies may respond to something

Depending on what your expertise is, you may be able to offer some very specific needs. Surely, what you know can be applied to a campaign, though I may not be able to tell you how. A lot of my expertise is in information security. Here are some examples of InfoSec advice you can provide:

  • Explain threat models. Make sure you know how they apply, too; the threat model of running for US Senator is much different than that of running for a local council seat. Everyone can benefit from making sure they don’t expose their financial details to the world, but fewer are worried about specific attacks by enemy nation-states.
  • Ensure services are registered to a well protected account that belongs to the campaign instead of a random gmail account that belongs to someone who may leave the campaign.
  • Make sure MFA is enabled everywhere possible.
  • Restrict access to services to who needs it, and at the least permission level

Remember that you are advising a campaign, not running your own business, so you will probably “lose” some arguments in areas where you are objectively the expert, and that’s okay. Make sure everyone acknowledges the trade-offs being made, and do your best to minimize the potential fallout. If there’s a realistic chance of failure, prepare remediation plans so that you are ready if they are needed – the same kinds of thing you do at work to cover your company’s butt.

Be aware of how much you can contribute. If you can only spend 2 hours a week with the campaign at odd times of the day, maybe you are not the best person to run their web site. That’s okay, just make the campaign aware that you want to help as well as your limits and surely they can find a way to make it fit. If you can spend more hours, then I encourage you to take on more significant tasks. If your circumstances change, just let the campaign know!

There are tons of small things like this that you can contribute. Don’t worry if you can’t think of something now, if you reach out to the campaign I’m sure you can come up with something together!

Larger projects

In addition to these general tasks, you may be able to contribute to higher level projects to help the campaign. If you are a data scientist, a campaign needs you! Everyone needs to know which voters to target, and they’re hopefully looking for more ethical assistance than we have seen campaigns pursue in the past. Many campaigns can determine what kinds of voters they want to target, but they may lack the skills to find those voters within the mass of voter information available. Those who are great with analytics can help get data from the web sites to the voter analysis teams. Social media experts can help leverage Twitter, Facebook, Instagram, and other services to get messaging out effectively. Online advertising needs your expertise in marketing and advertising. Larger campaigns may need custom applications like HillaryBnB (an AirBnB-style app for canvassers).

Again, I had no experience with campaigns so these are just a few of the efforts I’ve observed recently, it’s a very non-comprehensive list of options. Each campaign’s needs are different, so I suggest checking with the campaign to see what is needed, rather than trying to offer specific projects.

Tackling the unknown

Though you are volunteering because you have expertise, sometimes what a campaign needs does not line up exactly. It’s a good thing we are an industry that is constantly learning! Lean on your existing expertise to get going.

I decided to help out a campaign with Google AdWords, as the lack of such a campaign was identified when I joined. Prior to June 1st, I had never used AdWords or done anything with advertising, online or otherwise. Yeah, it was intimidating. But I believe in my candidate, so I tackled it like any other tech I learned. I found some technical articles about how AdWords works and tips for novices, scrounged up some youtube video so I could see it, and then set up an account and got to work. After almost 3 weeks, I am starting to figure it out, and the campaign is benefiting from my efforts. Find something and dig in. You can grow your technical skills and help advocate for your politics at the same time!

While I will not pretend to be authoritative on AdWords, I want to share some things I learned, beyond the simple mechanics that you can learn through the documentation and tutorials:

  • Create the AdWords account with a central campaign account. Add additional administrators with campaign-specific emails, rather than personal emails. This makes it more difficult for someone to leave the campaign and take down advertising.
  • There’s a pretty decent iPhone app for AdWords, and it’s gives you some views not available on the web site, but you cannot edit very much on it. I am sure there is one for Android as well.
  • Google will provide a $100 coupon after spending your first $25. It will be sent in an email to the account owner’s email and it’s not automatic, you need to apply the code.
  • Google will also send an email advertising a free review of your account. I would wait a few weeks before calling, so that the specialists can see some data from your campaigns.
  • Impressions (someone seeing your ad) are free. Clicks (when someone actually clicks on them) are the only thing that costs you money.
  • Campaigns are made up of Ad Groups. Each ad group can advertise a different set of text and point to a different page, but funding is allocated on a Campaign level. You can add as many ad groups to a single campaign, but budget is allocated at the campaign level. You need to balance the number of campaigns and ad groups, and keep balancing them as voters’ interests change.
  • Each Campaign can also be targeted to different locations. You can limit some AdWords Campaigns to the political campaign’s region (by district for federal offices; by using zip codes for state and local offices) to focus your advertising, such as issues-based campaigns. Others may be made open to a wider area, maybe even the whole country, such as donation campaigns.
  • Each Ad Group can be made up of keywords which receive mostly-opaque scoring. The better the score, the better the placement. Keywords that do not result in Impressions receive a lower score and can drag down the score of the entire Ad Group. Disable keywords that do not work, or replace them with more specific and helpful keywords, to keep the scoring up. All the scoring is Google magic, and the effectiveness of this will vary quite a bit for every organization. Keep an eye on it.
  • Keywords are words or phrases that you want your ad matched to. You can also add Exclusions. Combine this with Search Terms results, which show the actual search someone used and the keyword or category it matched, to filter out Impressions/Clicks that are not helpful to your campaign. I have seen some really ridiculous search terms, including the amazing imagen you are an environmentalist giving a speech environmental due to population growth in the western united states that matched the keyword environment. Whoops, way too generic, it was replaced with a more specific phrase. Another was a search including the name of another candidate in another entire state and that one click ate the entire campaign budget for the day. Keeping up with Exclusions can save your campaign a lot of money.
  • Google watches monthly trends to determine when best to spend money. Your daily budget is better thought of as the average daily spend over a 30 day period. For example, Google may determine that you won’t get much out of ads on Saturday and spend close to $0, but the 2nd Wednesday of the month get the most impact and spend far more than your budget. You will only ever be charged 2x your specified budget, even if Google “spends” 2.3x your budget (I have never observed them going past 2.0x).
  • Almost everything you do with AdWords can be changed on the fly. However, there are two things to keep in mind:
    • Any new ad or keywords (and you cannot edit ads/keywords, you actually make a new one to replace the existing one) must be approved. It can take up to 24 hours to approve. You can add ads/keywords and disable them, then enable them when needed, to ensure they are approved prior to when they are needed.
    • Significant budget changes may flag a fraud alert. If you have a significant event-related campaign coming up, set it up at least 3 days in advance, as it can take 3 days to resolve suspected fraud. If you are spending $10 a day and want to increase that to $300 for a weekend event and your account gets flagged on Friday, it may be Monday before it is unfrozen and your event will be over.
  • You won’t find many AdWords tutorials that speak to political campaigns. It’s strongly associated with businesses. A few articles and charts mentioned Social Advocacy, which is probably the closest, but…
  • Success is difficult to measure. A business may track how many people click versus how many people order something. A donation campaign can track how many people donate, but an issues or awareness campaign cannot correlate visitors to the site with votes in a primary or general election. Conversion rates on their own won’t tell you much.
    • For many candidates, awareness itself is the goal. Many voters do not fill out the whole ballot and only check non-federal boxes if they recognize the name. Responsive Ads (as opposed to Text Ads) are fairly unobtrusive but can display small graphics. Logos are very helpful to start creating brand awareness.
    • Run at least two Ads for each campaign, much like you would do A/B testing at work. Review regularly and tweak the ads over time.
  • Advertising is not an island. Coordinate with the Social Media team and the event planners. If your candidate is going to be at a local festival or state fair in a few weeks, add some keywords for the event. If Social Media is blitzing on a policy, make sure common keywords will drive voters to your candidate. When things happen in the news that affect your voters, make sure their searches will bring them to your candidate. Likewise, you can review search terms people use and inform the other teams that these are some of the things voters are searching for and make sure the candidate speaks to those concerns.
    • This can feel very ghoulish or morbid. Much of news that drives people to politicians is going to be of a negative nature. We generally don’t call our elected officials when things are looking up. You WILL probably have to capitalize on an event that resulted in harm or death. In my case, unfortunately, there was a school shooting near the district. Ugh. But voters do want to know their candidate’s policies on subjects like school shootings. Be responsible and principled and above all, caring. Do not let the need to respond compromise your integrity or the candidate’s.
  • Your candidate’s party has other candidates running. Reach out to them for assistance, for ideas, for additional eyes on problems. You can get contact info from your county/state party’s offices, usually.
  • AdWords includes a large number of reports. Working in IT, our tendency is to encourage others to run reports themselves, but everyone on a political campaign is likely already spending all the time they can on their areas of expertise. It can be a huge help if you create/tweak reports for others, and schedule them to email the requester regularly.

Summary

Just because many of us make technology a huge part of our lives, we are not one dimensional. If you feel inspired by politics, don’t hide it, become active. I’ve discussed what you might expect if you join a political campaign, some of the work and expertise technologists can offer campaigns, and my experience in joining a campaign. Whether you contribute a few hours a month or hours every day, you will be a vital part of chosen campaign. That’s awesome! Participation in democracy is what makes our Republic so strong, and that of most democratic governments.

If you have any questions about volunteering, whether it’s technical or about the experience or something else, please reach out. You can drop a comment here, or @rnelson0 on twitter.

Enjoy, and thank you!

Disabling rubocop and upgrading to PDK 1.6.0

As I lamented in my article on converting to the PDK, I really do not like Rubocop and was disappointed I could not turn it off. Thankfully, that was addressed in PDK-998 and the fix was included in time for PDK 1.6.0! Disabling it is pretty simple and though it’s strictly a fix to pdk-templates, updating the PDK won’t hurt.

First, update to PDK 1.6.0. As I use CentOS 7 and the RPM packaging, it’s as simple as sudo yum update pdk -y; follow the directions that match system. Next, we need to add the following lines to .sync.yml:

.rubocop.yml:
  selected_profile: off

Finally, run pdk update, or if you weren’t already using pdk-templates, run pdk convert --template-url=https://github.com/puppetlabs/pdk-templates (I will assume the former to keep it simple). You can add --noop (or say n) and review update.txt|convert.txt to see the differences before applying, or, because you are using version control, just run a diff afterward to see the changes.

[rnelson0@build03 domain_join:pdk160]$ pdk update
pdk (INFO): Updating rnelson0-domain_join using the template at https://github.com/puppetlabs/pdk-templates, from master@041eeb2 to 1.6.0

----------Files to be modified----------
metadata.json
.pdkignore
spec/spec_helper.rb
.gitignore
Rakefile
.rubocop.yml
Gemfile

----------------------------------------

You can find a report of differences in update_report.txt.

Do you want to continue and make these changes to your module? Yes
[✔] Installing missing Gemfile dependencies.

------------Update completed------------

7 files modified.

That’s it! Check the contents of .rubocop.yml and you will notice everything is false (just a snippet because it’s loooong):

---
require: rubocop-rspec
AllCops:
  DisplayCopNames: true
  TargetRubyVersion: '2.1'
  Include:
  - "./**/*.rb"
  Exclude:
  - bin/*
  - ".vendor/**/*"
  - "**/Gemfile"
  - "**/Rakefile"
  - pkg/**/*
  - spec/fixtures/**/*
  - vendor/**/*
  - "**/Puppetfile"
  - "**/Vagrantfile"
  - "**/Guardfile"
Bundler/DuplicatedGem:
  Enabled: false
Bundler/OrderedGems:
  Enabled: false
Layout/AccessModifierIndentation:
  Enabled: false
Layout/AlignArray:
  Enabled: false
Layout/AlignHash:
  Enabled: false
Layout/AlignParameters:
  Enabled: false
Layout/BlockEndNewline:
  Enabled: false
Layout/CaseIndentation:
  Enabled: false
Layout/ClosingParenthesisIndentation:
  Enabled: false
Layout/CommentIndentation:
  Enabled: false

Running validation now finds no issues with ruby syntax no matter how much you ignore style guides:

# master, prior to updating

[rnelson0@build03 domain_join:master]$ pdk validate
...
[✖] Checking Ruby code style (**/**.rb).
info: task-metadata-lint: ./: Target does not contain any files to validate (tasks/*.json).
convention: rubocop: spec/spec_helper_acceptance.rb:17:27: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.
convention: rubocop: spec/spec_helper_acceptance.rb:17:49: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.
convention: rubocop: spec/spec_helper_acceptance.rb:19:66: Style/BracesAroundHashParameters: Redundant curly braces around a hash parameter.
convention: rubocop: spec/spec_helper_acceptance.rb:19:68: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.
convention: rubocop: spec/spec_helper_acceptance.rb:19:96: Layout/SpaceAfterComma: Space missing after comma.
convention: rubocop: spec/acceptance/class_spec.rb:6:9: RSpec/ExampleWording: Do not use should when describing your tests.
convention: rubocop: spec/acceptance/class_spec.rb:12:26: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.
convention: rubocop: spec/acceptance/class_spec.rb:13:26: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.
convention: rubocop: spec/classes/domain_join_spec.rb:2:25: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.

# pdk160, after updating, no code changes
[rnelson0@build03 domain_join:pdk160]$ pdk validate
pdk (INFO): Running all available validators...
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.2
[✔] Checking metadata syntax (metadata.json tasks/*.json).
[✔] Checking module metadata style (metadata.json).
[✔] Checking Puppet manifest syntax (**/**.pp).
[✔] Checking Puppet manifest style (**/*.pp).
[✔] Checking Ruby code style (**/**.rb).
info: task-metadata-lint: ./: Target does not contain any files to validate (tasks/*.json).

You may have noticed there are quite a few other files updated. The other significant change is that a :changelog task via github_changelog_generator is now included, so you can remove that from your .sync.yml if you added it and replace it with the recommended config (via the Rakefile):

---
Gemfile:
  optional:
    ':development':
      - gem: 'github_changelog_generator'
        git: 'https://github.com/skywinder/github-changelog-generator'
        ref: '20ee04ba1234e9e83eb2ffb5056e23d641c7a018'
        condition: "Gem::Version.new(RUBY_VERSION.dup) >= Gem::Version.new('2.2.2')"

The other changes are pretty minor, in some cases cosmetic, but of course review them to make sure they’re OK. Submit a PR or equivalent and make sure the tests pass before merging. You can follow along with today’s blog post in domain_join PR35, too.

Enjoy!

Opinion: Technology is always Political

I’m writing this opinion piece right now because of ongoing gross abuses of justice taking place in America right now. Don’t worry, as strongly as I feel about this subject, my blog will remain free of specific political advocacy (please follow me on twitter for my politics), but we absolutely need to talk about the relationship between technology and politics. I would love to see your comments here, or you can reach me on twitter if that is more comfortable.

As technical practitioners, we are often under a lot of pressure to focus on tech and minimize other subjects, especially contentious subjects like politics. These subjects can cause conflict and many of us are taught to avoid, rather than resolve, conflict. We are constantly told that talking about tech is great; maybe talk about your sports teams, music, craft beers – but never talk about politics. That’s divisive! Sure, sometimes politics may cause conflict and even alienate people, but that’s just an aspect of life, of who we are, and not something to box up and hide in a corner. We soon find that everything is political – and if it’s not, it will still be made political for us. Our politics reflect who we are, and we strive to grow and change over time, and it stands to reason that our politics grow and change with us. I find Scott Hanselman to be very eloquent on the relation of politics to self, probably because it comes up so often on his timeline:

Once you share your politics, like Scott, you may be told to stick to technology. But if everything is political, then technology must be political, too. It doesn’t exist in a vacuum. It never has and it never will. We must stop pretending that technology exists free from politics. Our community benefits from embracing this truth, not denying it.

Examination of our industry’s early history quickly shows the relationship between technology and politics. Let’s review the history of IBM during World War II. In 1933, IBM started cozying up to the Hitler regime to increase sales, starting with an innocuous sounding census. Having machine-tabulated census data allowed the German government to rapidly increase their prosecution of the Holocaust. Eventually, every Nazi concentration camp used IBM punch card technology to track prisoners – which IBM serviced under contract.

This was not an isolated act of the company or an oddity of IBM’s German subsidiary. This happened in America, too. IBM pursued and acquired the contract for the Japanese internment camps’ punch cards, at the same time its equipment was used for US Army and Navy cryptography. IBM was okay with the use of its punch card equipment to identify, round up, and track prisoners, even by a country at war with IBM’s home country, so as long as IBM got paid.

Neither of these efforts just “happened.” IBM employees developed the punch card technology. IBM employees had to contact the German and US governments to open sales channels. IBM employees  had to pursue and close sales contracts with the German and US governments. IBM employees had to provide support, spare parts, and even enter concentration camps to change the printer paper throughout World War II.

Numerous people were required to prosecute the Holocaust and the Japanese internment. Only a few were technologists, and fewer still worked for IBM. But all of them allowed their personal political and ethical views to be subsumed and harnessed to a political regime that enacted some of the worst atrocities in the history of humankind. We do not know exactly what these people intended, but history has recorded the outcome and judged it. No amount of, “Well, I didn’t mean that to happen,” or, “I didn’t want to take sides,” will ever change that.

“The only thing necessary for the triumph of evil is that good men should do nothing.” – Edmund Burke

And here we are again, in 2018, observing numerous authoritarian efforts, by both governments and public/private companies, to weaponize technology. Again, this does not just “happen.” Someone has to actively develop, sell, provide, and support these weaponized technologies. Each of us must inform ourselves of how the technologies we work on can be weaponized and used for evil. Implicit and subconscious bias will creep into products, but very frequently, efforts are consciously and overtly made to weaponize technologies. Original intent is never remembered, only the horrible outcomes.

Earlier, I stated that we should have and develop our own political views. In accepting that technology is intrinsically political, our use of technology may then become political advocacy. For example, If we are strongly in favor of a legal right, using a technology designed to interfere or suppress that right would be antithetical to our politics. Thus, our political views should drive our use of technology.

To avoid advocating for specific political views, I suggest that we all commit to a formal code of ethics that closely matches our political views. I highly recommend USENIX’s System Administrators’ Code of Ethics (more in Additional Links). It is straightforward and thorough, is compatible with most political viewpoints, and has stood the test of time within our industry. A chosen code of ethics is only helpful if we stick by it. Not just when it’s easy, but especially when it’s difficult!

So what happens if we do find ourselves involved with a system that can be weaponized, that violates our ethics and politics? How do we advocate our politics and maintain our ethical code when we have significant concerns? “It depends,” of course, on those concerns and how significant they are, on our relationship with the system, and on who we are providing for. We must each determine an inflection point where it passes from, “Can this still be saved?” to, “This cannot be saved.” Everyone’s inflection point will be different – we all have different politics and ethics, finances, health, and family situations, etc. – but we must each determine exactly where that point is for us.

Inflection point in hand, we can proceed to an action plan. We may be able to work fully above board, making cases to stakeholders and management. Or we may have to move below board, purposefully dragging our feet or working against the system. Maybe a change in vendors will address concerns or slow progress, maybe we just don’t do certain things at all. Find the appropriate monkey wrench that fits the gears of the system. We must plan our activities as we would any other technical work, laying out our goals and milestones and alternative plans for when disaster strikes.

We can also lean on each other. Reach out to your coworkers, your colleagues and peers, your friends and family, to voice your concerns and discuss remedies. We are not alone, and we can lean on or be the rock for each other. We may want to confide in just a few trusted people, to organize with our coworkers, or to become whistleblowers. There is so much nuance and possibility here that it is impossible to predict what actions will be required, but together we can determine what those actions are.

There may come a day when we find that we cannot stop the dangerous technology, when we have crossed that inflection point. We have to evaluate, honestly, whether there is still good we can do, or if we have truly passed the point of no return and need to walk away. Many of us will never be close to making these kinds of decisions, but some of us will. We must rely on our ethical codes and our planning to remain true, to see the point of no return, and to walk away, even if it costs us. To stay and actively participate knowing that we are now doing irrevocable harm would be costlier.

This is not a one-time deal. We must always keep our eyes open, evaluating our ever-changing political views against the work in front of us, applying our ethics to keep us on track, and making damn sure that we are never – metaphorically or literally – changing the printer paper in a concentration camp.

We will not build the software for concentration camps or to enable authoritarians. We will not spy on our neighbors or destroy democracies. We will commit to our ethical codes of conduct, and we will use technology to build the shining city on a hill.

Thank you.

 

 

Additional Links:

Creating your first Puppet Task for Puppet Enterprise

At PuppetConf 2017, Puppet Tasks were introduced as part of the new project Bolt. A task allows you to run a program on an arbitrary number of nodes. The program can be just about anything, it just needs to be written in a language that the target nodes can run. For Linux, that means pretty much anything – bash, python, perl, ruby, etc. On Windows, you’re a little more limited out of the box – powershell primarily. Bolt is not yet at version 1.0.0, so I suspect language support for Windows will change. You can use Bolt on its own (even without Puppet, apparently), and starting with Puppet Enterprise 2017.3, you can use Bolt at the PE Console as “tasks” in the UI.

For my first task, I simply want to run a single command on a list of nodes. While I can run arbitrary commands with the bolt command line, I want the practice of writing a task. My use case involves an external authentication system that manages users, ssh keys, and sudo configurations. When a change is made, nodes need to pull the changes. Often, a delay there does not matter – the nodes will receive the change soon enough – but sometimes I want the relevant nodes to pick it up immediately. To do so, I need it to run a single perl script, sanitized as /usr/bin/external_auth.pl, and I want to do it on all the nodes with the profile::external_auth class.

Continue reading

Convert a Puppet module from Bundle-based testing to the Puppet Development Kit (PDK)

A few years ago, I set up my modules with a bundle-based test setup and modulesync and wrote a companion blog post. Since that was written, a lot of things have changed with puppet. One of those is the release last year of the Puppet Development Kit (PDK). The goal of the PDK is to simply development of puppet code and modules by reducing or eliminating all of the headaches of creating a mature ruby/bundler/puppet-lint/etc. setup. There is also a brand new tool called pdksync that combines the PDK with the power of modulesync. I was somewhat involved in the initial efforts toward the PDK, through my work on puppet-lint, but I have not actually used the PDK “in anger” yet, in part because of my previously working modulesync setup. This seems like a great opportunity to switch to PDK and pdksync, starting with the PDK.

Why PDK?

Before we begin, we should look at why we want to use the Puppet Development Kit. My current setup is best described as fragile. Its effectiveness varies based on what version of Ruby I use and the version of the gems I happen to download via bundler on any given day. I use CentOS 7, which is stuck on Ruby 2.0. Most of the gems in my setup require at least Ruby 2.1 or 2.2, so I have to resort to RVM to provide me with Ruby 2.3.1. Someday, I’ll need to update that to Ruby 2.4 for a gem and my setup will break until I fix it.

I am also downloading a bunch of gems that are not pinned and updated versions can bring in subtle bugs or cascading failures not related to changes in my code. Sometimes the gem is directly related to my work, like a puppet-lint version with a bug that I can downgrade and pin. Other times, it’s a very indirect dependency of a dependency of a dependency to puppet-lint, and pinning it only creates more problems for everything that depends on it. Of course, bundler also relies on rubygems.org and internet/mirror access, which sometimes go down when you need them most.

While these are surmountable issues, they always come up while I’m trying to get something done in puppet, and the minutes or hours required to fix the problem prevents me from making the changes I need, when I need.

The PDK resolves this by bundling its own version of Ruby and dependent gems. Puppet vets the setup, so I do not have to. Everything is on disk, so there’s no more required downloading of gems that can be pulled or unavailable because of network issues. This is a huge benefit to all users, whether they pay for Puppet Enterprise or use Puppet Opensource Edition for free. Less time spent worrying about dependency hell and more time getting straight to work. This is important, valuable work, but it’s not my expertise or an actual goal of my job, so I am very content to let someone else handle the setup so I can spend more time managing my systems with Puppet.

The PDK is an installable tool. Install once and you can use it with all your puppet modules and controlrepos. Upgrading the PDK is simple using your package manager. You can of course combine using the PDK on some modules and stick with the ruby/bundler setup on others. However, it will be more difficult (but not impossible) to switch between the PDK and native bundler on the same module – our CI systems will use native bundler after all – but some gem dependencies will no longer be pinned and we lose the guarantee of pinned gems that will work together.

We will see below that you can still modify the setup on each module/controlrepo to some extent, but when using the PDK, the full range of customization bundler offered is unavailable to you. I think most people will not find this to be a problem, but you should definitely read up on the PDK to make sure you understand what you gain and lose before converting to using it. If you change your mind later, switching back from PDK to a bundler-based setup is possible, but it may involve some work to find a working setup of pinned gem versions.

Installing the PDK

The very first thing I need to do is install the PDK. The following is written using PDK v1.5.0.0. The PDK is relatively new and gets frequent updates, so this may become out of date rapidly. If you run into any issues, check the version, read the release notes, and adjust accordingly.

The docs decribe how to install on various systems. I use EL7 so I will install the RPM. I also use Puppet Enterprise, not Puppet Opensource, so I have to add the Puppet repository first. rpm and yum can get me there, or I can use puppet apply:

# Manual
sudo rpm -Uvh https://yum.puppet.com/puppet5/puppet5-release-el-7.noarch.rpm
sudo yum install pdk -y
# puppet apply
cat > ~/pdk.pp < EOF
package { 'puppet5-release-el-7':
  ensure => present,
  provider => 'rpm',
  source => 'https://yum.puppet.com/puppet5/puppet5-release-el-7.noarch.rpm',
}
-> package { 'pdk':
  ensure => present,
}
EOF
sudo puppet apply ~/pdk.pp

I can now call the command pdk successfully. Be aware that it includes its own bundled ruby, so the first time you run it may take a little time to be loaded and cached, which is expected.

[rnelson0@build03 domain_join:master]$ pdk --help
NAME
    pdk - Puppet Development Kit

USAGE
    pdk command [options]

DESCRIPTION
    The shortest path to better modules.

COMMANDS
    build        Builds a package from the module that can be published to the Puppet Forge.
    bundle       (Experimental) Command pass-through to bundler
    convert      Convert an existing module to be compatible with the PDK.
    help         show help
    module       Provide CLI-backwards compatibility to the puppet module tool.
    new          create a new module, etc.
    test         Run tests.
    update       Update a module that has been created by or converted for use by PDK.
    validate     Run static analysis tests.

OPTIONS
    -d --debug                    Enable debug output.
    -f --format=           Specify desired output format. Valid
                                  formats are 'junit', 'text'. You may also
                                  specify a file to which the formatted
                                  output is sent, for example:
                                  '--format=junit:report.xml'. This option
                                  may be specified multiple times if each
                                  option specifies a distinct target file.
    -h --help                     Show help for this command.
       --version                  Show version of pdk.

If you sign up for the puppet-announce mailing list, you will be notified every time there’s a new PDK release. After reading the release notes for edge cases that may impact you, can easily upgrade to the latest version with your distro’s equivalent of yum update pdk. That is a lot easier than updating ruby/bundler setups.

Converting to the PDK

Next, my existing setup must be converted to PDK. I will walk through my efforts, but you can also review the PuppetConf 2017 video and slides about the PDK in addition to this Converting To PDK doc. I am working on my domain_join module first, starting from the release candidate for v0.5.2, if you want to recreate this effort. The module is part of my modulesync config and has 68 tests. It’s a reasonably mature module but not overly complex, perfect for testing without being too deep or shallow. I am also going to break the modulesync setup, which you can see here. Before beginning, I create a new branch on the module:

[rnelson0@build03 domain_join:master]$ git checkout -b pdk
Switched to a new branch 'pdk'

Because I use modulesync, this will not work out of the box for me, but there’s a very naive default pdk convert that can be used to update the config. It will inform you of the files that will be added/modified and prompts you to continue due to the potential for destruction. As noted, this concern is mitigated by using version control, and if you’ve read my blog before, you’re obviously using version control, right? If not, get that done first! (it’s beyond the scope of this article, but my git 101 article may help) Here’s what the naive attempt looks like:

[rnelson0@build03 domain_join:pdk]$ pdk convert

------------Files to be added-----------
.pdkignore
.project
spec/default_facts.yml
.gitlab-ci.yml
appveyor.yml

----------Files to be modified----------
metadata.json
spec/spec_helper.rb
.gitignore
.travis.yml
Rakefile
.rubocop.yml
.rspec
Gemfile

----------------------------------------

You can find a report of differences in convert_report.txt.

pdk (INFO): Module conversion is a potentially destructive action. Ensure that you have committed your module to a version control system or have a backup, and review the changes above before continuing.
Do you want to continue and make these changes to your module? Yes

------------Convert completed-----------

5 files added, 8 files modified.

[rnelson0@build03 domain_join:pdk±]$

The git diff is really, really lengthy, but you can find it here. A lot of it is simply re-arranging of stanzas in existing files (.gitignore, .rspec, .travis.yml, metadata.json) and .rubocopy.yml updates. The rest is mostly in three files: Rakefile, Gemfile and spec/spec_helper.rb. It also adds five files: .gitlab-ci.yml, .pdkignore, .project, appveyor.yml, and spec/default_facts.yml. Some info on the minor changes:

  • spec/default_facts.yml: If you had default facts in your spec/spec_helper.rb file, you should move them here. The rest is mostly “housekeeping” but it removes my hiera config, which I will explore in a moment.
  • .gitignore, .pdkignore: The former has been updated a bit, and the latter is the exact same thing.
  • .gitlab-ci.yml, .travis.yml, appveyor.yml: Puppet provides some good defaults for a number of external systems. I am sticking with Travis CI right now, but it’s great to have defaults for other services if I branch out. The latter looks targeted at testing Windows systems, too, something that’s often problematic. These are all optional but do not hurt by being present.
  • .project: Looks like some XML for use with the editor Eclipse.
  • .rubocop.yml: I really don’t like rubocop but it’s included. I plan to disable rubocop as quickly as possible. However, this addresses one of my pain points – every version of rubocop changes the name of Cops, and it fails to run if it finds an unknown Cop name in its config. Since Puppet vets the config, I do not have to deal with finding all the new Cop information everytime rubocop updates. It’s not enough for me to love it, but it is significant pain reduction.

This leaves the big three files mentioned earlier, which are worthy of more detailed investigation.

Rakefile

This file is MUCH smaller now, I presume in thanks due to some pdk magic. The conversion removed a changelog task I created, so I need to get this back.

GitHubChangelogGenerator::RakeTask.new :changelog do |config|
 version = (Blacksmith::Modulefile.new).version
 config.future_release = "v#{version}"
 config.header = "# Change log\n\nAll notable changes to this project will be documented in this file.\nEach new release typically also includes the latest modulesync defaults.\nThese should not impact the
 config.exclude_labels = %w{duplicate question invalid wontfix modulesync}
end

I also have a number of puppet-lint checks I have disabled, like arrow_alignment, that I need to make sure are restored. After restoring my task and disabled checks, I will be okay with this new, slimmer default.

Gemfile

PDK includes its own version of ruby and bundler and is guaranteed to deliver a gemset with all the dependencies needed to work together. You can run pdk bundle exec gem list to see what it includes, if you are curious what those are. I will add the github_changelog_generator gem here soon, but otherwise as long as everything works, I have no need to poke at this file anymore.

spec/spec_helper.rb

Though the diff is fairly long for this file, there is nothing tricky here, it just connects the new default facts and some other common practices. It DOES remove the hiera configuration. There is a more modern version of my hiera_config that we need to add back in:

RSpec.configure do |c|
  c.hiera_config = 'spec/fixtures/hiera/hiera.yaml'
end

The naive conversion is not that bad for my setup, but it does leave me with three changes to make to keep functional parity: add the github_changelog_generator gem, the :changelog rake task, and re-enable hiera lookups.

Updating the PDK Setup

Now that I’ve identified the non-default changes needed, I can do some updates. The PDK can convert and update modules using a template system. The template it used is listed at the bottom of metadata.json. You can find the templates online, or clone that directory and examine the moduleroot contents (the moduleroot_init directory is also used when you run pdk new):

[rnelson0@build03 domain_join:pdk±]$ git diff metadata.json
diff --git a/metadata.json b/metadata.json
index 7730b90..381aff7 100644
--- a/metadata.json
+++ b/metadata.json
@@ -27,5 +27,8 @@
"name": "puppet",
"version_requirement": ">=4.0.0"
}
- ]
+ ],
+ "pdk-version": "1.5.0",
+ "template-url": "file:///opt/puppetlabs/pdk/share/cache/pdk-templates.git",
+ "template-ref": "1.5.0-0-gd1b3eca"
}

The copies on disk are from the RPM, and are almost definitely out of date. The latest templates are on GitHub. I can re-run the conversion with pdk convert --template-url=https://github.com/puppetlabs/pdk-templates. The changes for me are pretty small but will be much larger the further away in time you are from the date of the RPM build. After running it, the template info will also be updated:

+ ],
+ "pdk-version": "1.5.0",
+ "template-url": "https://github.com/puppetlabs/pdk-templates",
+ "template-ref": "heads/master-0-g7b5f6d2"

We can look at the individual templates here or clone it locally. The first thing to note is that frequently the .erb templates are dynamic data, rather than static. The simplest change is in spec/spec_helper.rb, just adding a single stanza to the Rspec.configure section, which is also dynamic:

RSpec.configure do |c|
  c.default_facts = default_facts
  <%- if @configs['hiera_config'] -%>
  c.hiera_config = "<%= @configs['hiera_config'] %>"
  <%- end -%>
  <%- if @configs['strict_level'] -%>
    c.before :each do
    # set to strictest setting for testing
    # by default Puppet runs at warning level
    Puppet.settings[:strict] = <%= @configs['strict_level'] %>
  end
  <%- end -%>
end

I highlighted the conditional that will populate the filename with the contents of configs['hiera_config']. The configs hash is populated by config_defaults.yml. The README has a lot of helpful information on the defaults. There’s just a few lines for the spec_helper.rbfile:

[rnelson0@build03 pdk-templates:master]$ tail -2 config_defaults.yml
spec/spec_helper.rb:
  strict_level: ":warning"

I need to add to this hash, but I cannot add to the templates since they are upstream. Thankfully, there’s a built in way to account for this. The contents of the configs hash are combined with the same hash taken from the local.sync.yml file!

Note: if you’d like you CAN change the templates by forking puppetlabs/pdk-templates and passing in --template-url when you call pdk new, convert, or update. You are then on the hook for updating your templates over time, though.

To make use of the sync file, I just need to add it to the root of my module directory and add the custom config. It is additive, so only differences need to be present. Here is the hiera_config value required:

[rnelson0@build03 domain_join:pdk±]$ cat .sync.yml
spec/spec_helper.rb:
  hiera_config: 'spec/fixtures/hiera.yaml'

With the use of the pdk update command, I can re-apply the templates in --noop mode and see the change:

[rnelson0@build03 domain_join:pdk±]$ pdk update --noop
pdk (INFO): Updating rnelson0-domain_join using the default template, from 1.5.0 to 1.5.0

----------Files to be modified----------
spec/spec_helper.rb

----------------------------------------

You can find a report of differences in update_report.txt.

[rnelson0@build03 domain_join:pdk±]$ cat update_report.txt
/* Report generated by PDK at 2018-05-29 20:08:11 +0000 */


--- spec/spec_helper.rb 2018-05-29 18:53:09.140882197 +0000
+++ spec/spec_helper.rb.pdknew 2018-05-29 20:08:11.819978562 +0000
@@ -28,6 +28,7 @@

 RSpec.configure do |c|
   c.default_facts = default_facts
+  c.hiera_config = spec/fixtures/hiera.yaml
   c.before :each do
   # set to strictest setting for testing
   # by default Puppet runs at warning level

Now that we have proved out the process, I need to make a few more changes. To add the github_changelog_generator, I add an array entry under Gemfile: required: ':development'. To add the task, I use Rakefile: extras:for the rake task, one entry per line (you can also use multi-line content in yaml if you prefer). This is what the file looks like as well as the pending changes:

[rnelson0@build03 domain_join:pdk±]$ cat .sync.yml
spec/spec_helper.rb:
  hiera_config: 'spec/fixtures/hiera.yaml'
Gemfile:
  required:
    ':development':
      - gem: github_changelog_generator
Rakefile:
  default_disabled_lint_checks:
    - 'arrow_alignment'
    - 'class_inherits_from_params_class'
    - 'class_parameter_defaults'
    - 'documentation'
    - 'single_quote_string_with_variables'
  extras:
    - "require 'github_changelog_generator/task'"
    - 'GitHubChangelogGenerator::RakeTask.new :changelog do |config|'
    - '  version = (Blacksmith::Modulefile.new).version'
    - '  config.future_release = "v#{version}"'
    - '  config.header = "# Change log\n\nAll notable changes to this project will be documented in this file.\nEach new release typically also includes the latest modulesync defaults.\nThese should not impact the functionality of the module."'
    - '  config.exclude_labels = %w{duplicate question invalid wontfix modulesync}'
    - 'end'
[rnelson0@build03 domain_join:pdk±]$ pdk update --noop
pdk (INFO): Updating rnelson0-domain_join using the template at https://github.com/puppetlabs/pdk-templates, from master@7b5f6d2 to 1.5.0

----------Files to be modified----------
spec/spec_helper.rb
Rakefile
Gemfile

----------------------------------------

You can find a report of differences in update_report.txt.

[rnelson0@build03 domain_join:pdk±]$ cat update_report.txt
/* Report generated by PDK at 2018-05-29 20:39:41 +0000 */


--- spec/spec_helper.rb 2018-05-29 20:33:16.488401096 +0000
+++ spec/spec_helper.rb.pdknew  2018-05-29 20:39:41.492124202 +0000
@@ -28,6 +28,7 @@

 RSpec.configure do |c|
   c.default_facts = default_facts
+  c.hiera_config = "spec/fixtures/hiera.yaml"
   c.before :each do
     # set to strictest setting for testing
     # by default Puppet runs at warning level


--- Rakefile    2018-05-29 20:33:16.489401137 +0000
+++ Rakefile.pdknew     2018-05-29 20:39:41.492832995 +0000
@@ -3,4 +3,11 @@
 require 'puppet_blacksmith/rake_tasks' if Bundler.rubygems.find_name('puppet-blacksmith').any?

 PuppetLint.configuration.send('disable_relative')
+
+require 'github_changelog_generator/task'
+GitHubChangelogGenerator::RakeTask.new :changelog do |config|
+  version = (Blacksmith::Modulefile.new).version
+  config.future_release = "v#{version}"
+  config.header = "# Change log\n\nAll notable changes to this project will be documented in this file.\nEach new release typically also includes the latest modulesync defaults.\nThese should not impact the functionality of the module."
+  config.exclude_labels = %w{duplicate question invalid wontfix modulesync}
+end


--- Gemfile     2018-05-29 20:16:10.321541394 +0000
+++ Gemfile.pdknew      2018-05-29 20:39:41.494035036 +0000
@@ -34,6 +34,7 @@
   gem "puppet-module-win-default-r#{minor_version}",   require: false, platforms: [:mswin, :mingw, :x64_mingw]
   gem "puppet-module-win-dev-r#{minor_version}",       require: false, platforms: [:mswin, :mingw, :x64_mingw]
   gem "puppet-blacksmith", '~> 3.4',                   require: false, platforms: [:ruby]
+  gem "github_changelog_generator",                    require: false
 end

 puppet_version = ENV['PUPPET_GEM_VERSION']

I now run it in yesop mode and my changes take. A quick check of rake targets confirms it. Note that all pdk bundle output is written to STDERR, not STDOUT.

[rnelson0@build03 domain_join:pdk±]$ pdk bundle exec rake -T 2>&1 | grep change
rake changelog # Generate a Change log from GitHub

I did not add the puppet-lint disable checks back in here. That is because the PDK does not use the Rakefile when running puppet-lint, it relies on the configuration file. I need to create .puppet-lint.rc at the top of the repo so that the settings are available to my CI system. That file looks like this:

[rnelson0@build03 domain_join:pdk±]$ cat .puppet-lint.rc
--no-arrow_alignment-check
--no-class_inherits_from_params_class-check
--no-documentation-check
--no-single_quote_string_with_variables-check

One difference between the Rake target and the config file is that an invalid check name in the config file can cause errors, whereas the Rake setting just doesn’t do anything. I removed the class_parameter_defaults check from the list because it is no longer a valid check.

There are a lot more things you might want to change, especially if you use CI other than Travis, but this should be enough for me to gain parity with my existing setup. Remember that you can poke at the templates online, find the default settings in config_defaults.yml, tweak in your own .sync.yml, re-run pdk update and everything should work out. If the templates cannot be wrangled as is, you can always open a ticket in the PDK project.

Make sure you commit your changes and push them up to version control, eventually to be merged into master.

First Test

Now I need to run my tests. Before I do that, I clean up everything not in git. Since I have developed in this directory, there are bundler files that don’t need to be there and may cause conflicts with the tests. Again, make sure you’ve committed changes first, or some of your uncommitted changes from the conversion will be removed:

[rnelson0@build03 domain_join:pdk±]$ git clean -ffdx
Removing .bundle/
Removing Gemfile.lock
Removing bin/
Removing convert_report.txt
Removing coverage/
Removing pkg/
Removing spec/defines/
Removing spec/fixtures/manifests/
Removing spec/fixtures/modules/
Removing spec/functions/
Removing spec/hosts/
Removing update_report.txt
Removing vendor/

PDK Tests

The first test is real simple, it’s my unit tests via pdk test unit:

[rnelson0@build03 domain_join:pdk±]$ pdk test unit
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.1
[✔] Preparing to run the unit tests.
[✔] Running unit tests.
Evaluated 68 tests in 2.321391522 seconds: 0 failures, 0 pending.
[✔] Cleaning up after running unit tests.

I also want to validate linting and syntax and whatnot with pdk validate:

[rnelson0@build03 domain_join:pdk]$ pdk validate
pdk (INFO): Running all available validators...
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.1
[✔] Checking metadata syntax (metadata.json tasks/*.json).
[✔] Checking module metadata style (metadata.json).
[✔] Checking Puppet manifest syntax (**/**.pp).
[✔] Checking Puppet manifest style (**/*.pp).
[✖] Checking Ruby code style (**/**.rb).
info: task-metadata-lint: ./: Target does not contain any files to validate (tasks/*.json).
convention: rubocop: spec/spec_helper_acceptance.rb:17:27: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.
convention: rubocop: spec/spec_helper_acceptance.rb:17:49: Style/HashSyntax: Use the new Ruby 1.9 hash syntax.
<more rubocop results>

I have a ton of rubocop results, which I will address below. Everything else works fine, as expected.

CI Tests

The second is a little trickier. Currently, whatever CI system you use will use ruby/bundler to perform the same checks. That is planned to change (PDK-709 tracks the Travis CI setup) When I use Travis CI, it uses tests from .travis.yml. Here are the relevant portions:

bundler_args: --without system_tests
matrix:
  fast_finish: true
  include:
    -
      env: CHECK="syntax lint metadata_lint check:symlinks check:git_ignore check:dot_underscore check:test_file rubocop"
    -
      env: CHECK=parallel_spec
    -
      env: PUPPET_GEM_VERSION="~> 4.0" CHECK=parallel_spec
      rvm: 2.1.9

There are two different checks that will run. The first is all the syntax and linting, the equivalent of pdk validate. The second and third are the unit tests, run against the latest Puppet 4 and Puppet 5 independently, and equivalent to pdk test unit. Here’s what happens when I run the unit tests first:

[rnelson0@build03 domain_join:pdk]$ pdk bundle exec rake parallel_spec
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.1
Cloning into 'spec/fixtures/modules/stdlib'...
2 processes for 2 specs, ~ 1 specs per process
No examples found.

Finished in 0.00032 seconds (files took 0.07684 seconds to load)
0 examples, 0 failures


domain_join
  on redhat-6-x86_64
    with defaults for all parameters
      should not contain Package[samba-common-tools]
      should contain Package[oddjob-mkhomedir]
      should contain Package[krb5-workstation]
      should contain Package[krb5-libs]
      should contain Package[samba-common]
      should contain Package[sssd-ad]
      should contain Package[sssd-common]
      should contain Package[sssd-tools]
      should contain Package[ldb-tools]
      should contain Class[domain_join]
      should contain File[/etc/resolv.conf]
      should contain File[/etc/krb5.conf]
      should contain File[/etc/samba/smb.conf]
      should contain File[/etc/sssd/sssd.conf]
      should contain File[/usr/local/bin/domain-join]
      should contain Exec[join the domain]
    with manage_services false
      should not contain Package[sssd]
      should not contain File[/etc/sssd/sssd.conf]
      should contain File[/etc/resolv.conf]
      should contain File[/usr/local/bin/domain-join]
    with manage_services and manage_resolver false
      should not contain Package[sssd]
      should not contain File[/etc/sssd/sssd.conf]
      should not contain File[/etc/resolv.conf]
      should contain File[/usr/local/bin/domain-join]
    start script syntax
      should contain File[/usr/local/bin/domain-join] with content =~ /sssd status/
    with container
      should contain File[/usr/local/bin/domain-join] with content =~ /net ads join/
      should contain File[/usr/local/bin/domain-join] with content =~ /container_ou='container'/
    with account and password
      should contain File[/usr/local/bin/domain-join] with content =~ /register_account='service_account'/
      should contain File[/usr/local/bin/domain-join] with content =~ /register_password='open_sesame'/
    with join_domain disabled
      should not contain Exec[join the domain]
    with manage_dns disabled
      should not contain File[/usr/local/bin/domain-join] with content =~ /net ads dns register/
      should not contain File[/usr/local/bin/domain-join] with content =~ /update add /
    with manage_dns and ptr enabled
      should contain File[/usr/local/bin/domain-join] with content =~ /net ads dns register/
      should contain File[/usr/local/bin/domain-join] with content =~ /update add .+ addr show fake_interface/
  on redhat-7-x86_64
    with defaults for all parameters
      should contain Package[samba-common-tools]
      should contain Package[oddjob-mkhomedir]
      should contain Package[krb5-workstation]
      should contain Package[krb5-libs]
      should contain Package[samba-common]
      should contain Package[sssd-ad]
      should contain Package[sssd-common]
      should contain Package[sssd-tools]
      should contain Package[ldb-tools]
      should contain Class[domain_join]
      should contain File[/etc/resolv.conf]
      should contain File[/etc/krb5.conf]
      should contain File[/etc/samba/smb.conf]
      should contain File[/etc/sssd/sssd.conf]
      should contain File[/usr/local/bin/domain-join]
      should contain Exec[join the domain]
    with manage_services false
      should not contain Package[sssd]
      should not contain File[/etc/sssd/sssd.conf]
      should contain File[/etc/resolv.conf]
      should contain File[/usr/local/bin/domain-join]
    with manage_services and manage_resolver false
      should not contain Package[sssd]
      should not contain File[/etc/sssd/sssd.conf]
      should not contain File[/etc/resolv.conf]
      should contain File[/usr/local/bin/domain-join]
    start script syntax
      should contain File[/usr/local/bin/domain-join] with content =~ /status sssd.service/
    with container
      should contain File[/usr/local/bin/domain-join] with content =~ /net ads join/
      should contain File[/usr/local/bin/domain-join] with content =~ /container_ou='container'/
    with account and password
      should contain File[/usr/local/bin/domain-join] with content =~ /register_account='service_account'/
      should contain File[/usr/local/bin/domain-join] with content =~ /register_password='open_sesame'/
    with join_domain disabled
      should not contain Exec[join the domain]
    with manage_dns disabled
      should not contain File[/usr/local/bin/domain-join] with content =~ /net ads dns register/
      should not contain File[/usr/local/bin/domain-join] with content =~ /update add /
    with manage_dns and ptr enabled
      should contain File[/usr/local/bin/domain-join] with content =~ /net ads dns register/
      should contain File[/usr/local/bin/domain-join] with content =~ /update add .+ addr show fake_interface/

1 deprecation warning total

Finished in 2.34 seconds (files took 2.17 seconds to load)
68 examples, 0 failures


68 examples, 0 failures

Took 5 seconds
I, [2018-05-29T21:51:29.950455 #16602]  INFO -- : Creating symlink from spec/fixtures/modules/domain_join to /home/rnelson0/modules/domain_join
/opt/puppetlabs/pdk/share/cache/ruby/2.4.0/gems/rspec-core-3.7.1/lib/rspec/core.rb:179:in `block in const_missing': uninitialized constant RSpec::Puppet (NameError)
        from /opt/puppetlabs/pdk/share/cache/ruby/2.4.0/gems/rspec-core-3.7.1/lib/rspec/core.rb:179:in `fetch'
        from /opt/puppetlabs/pdk/share/cache/ruby/2.4.0/gems/rspec-core-3.7.1/lib/rspec/core.rb:179:in `const_missing'
        from /home/rnelson0/modules/domain_join/spec/classes/coverage_spec.rb:1:in `block in '

Deprecation Warnings:

puppetlabs_spec_helper: defaults `mock_with` to `:mocha`. See https://github.com/puppetlabs/puppetlabs_spec_helper#mock_with to choose a sensible value for you


If you need more of the backtrace for any of these deprecations to
identify where to make the necessary changes, you can configure
`config.raise_errors_for_deprecations!`, and it will turn the
deprecation warnings into errors, giving you the full backtrace.
Tests Failed

The text in bold indicates an error in spec/classes/coverage_spec.rb. The simple solution for me is to git rm it, rather than add in the right coverage gem again. It’s not particularly important to me, but if it is to you, you need to add it back to Gemfile and spec/spec_helper.rb. The important thing is that a second run does not have the error and completes successfully.

The second test is a series of rake targets and causes me some grief out of the gate:

[rnelson0@build03 domain_join:pdk]$ pdk bundle exec rake syntax lint metadata_lint check:symlinks check:git_ignore check:dot_underscore check:test_file rubocop
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.1
init.pp
---> syntax:manifests
---> syntax:templates
---> syntax:hiera:yaml
rake aborted!
.pp files present in tests folder; Move them to an examples folder following the new convention
/opt/puppetlabs/pdk/share/cache/ruby/2.4.0/gems/puppetlabs_spec_helper-2.7.0/lib/puppetlabs_spec_helper/rake_tasks.rb:231:in `block (2 levels) in '
/opt/puppetlabs/pdk/share/cache/ruby/2.4.0/gems/rake-12.3.1/exe/rake:27:in `'
/opt/puppetlabs/pdk/private/ruby/2.4.4/bin/bundle:23:in `load'
/opt/puppetlabs/pdk/private/ruby/2.4.4/bin/bundle:23:in `'
Tasks: TOP => check:test_file
(See full trace by running task with --trace)

The fix is easy – move the existing files to a new location as the convention has changed, or remove them entirely if they are not valuable – and then I can proceed without further fault:

[rnelson0@build03 domain_join:pdk]$ mkdir examples
[rnelson0@build03 domain_join:pdk]$ git mv -v tests/*pp examples/
‘tests/init.pp’ -> ‘examples/init.pp’
[rnelson0@build03 domain_join:pdk±]$ pdk bundle exec rake syntax lint metadata_lint check:symlinks check:git_ignore check:dot_underscore check:test_file rubocop
pdk (INFO): Using Ruby 2.4.4
pdk (INFO): Using Puppet 5.5.1
Running RuboCop...
Inspecting 0 files


0 files inspected, no offenses detected
---> syntax:manifests
---> syntax:templates
---> syntax:hiera:yaml

As I mentioned earlier, I’d like to disable RuboCop. I don’t see how right now. If I specify selected_profile: off in .sync.yml for rubocop, pdk update errors out applying the template (PDK-998). However it seems to pass just fine in that check, though the individual check fails badly (PDK-997). I’m content to let it go so long as it’s passing test and I don’t have to rewrite anything, but I will find SOME way to get rid of it if it starts causing me problems!

If you use gitlab-ci, appveyor, or some other system for testing, you will want to ensure those tests pass as well. Once done, commit everything to git again.

I am now ready to submit a pull request, and if you are following along, you may be, too. You can review and compare my pull request and the tests if you would like. You will of course notice that I merged it in spite of rubocop failures!

Summary

We have looked at what the PDK is, why we want to use it, how to install it, and how to convert a module to use it. Each module can be customized and we explored the .sync.yml file that controls customization. Once we finalized our conversion, we ran the same tests we had prior to the PDK to make sure they still work and verified the Travis CI tests, too. The next step is to find a replacement for modulesync, which allows use to push the same general configuration to multiple modules. Lucky for us, Puppet just released a potential replacement, pdksync, that I will evaluate soon.