Edit: In an earlier edition, I credited the wrong newsletter as the source. My apologies to R.I.Pienaar!

In this past week’s DevCo Newsletter, I saw the Rebex SSH Check, which reminded me that I’ve locked down the SSH server security configuration at work, but not at home. Sounds like a good opportunity to blog about the process!

Now, I’m in security, but I’m not all that about the security settings. The names vary from descriptive to really obtuse, and there’s three keys that need managed: ciphers, MACs, and KexAlgorithms (that’s Key Exchange Algorithms, which is the name I’m more familiar with). The key to security is knowing when you don’t know, and seeking out that expertise. I am very thankful for Mozilla’s really great security guidelines, including an OpenSSH guide. There are sections for Modern and Intermediate security, depending on what is available for the systems you are securing. For me, these align with the Red Hat/CentOS EL7 (Modern) and EL6/5 (Intermediate) distros that I use.

The first step is making sure we have a tier in hiera for each OS/release we support, otherwise sshd could fail to restart when it encounters a cipher set name that is unknown to the openssh version in use. That could be bad, especially if we don’t have some form of iLO console to the nodes, though if we have puppet running on a regular basis or through mcollective, we *should* be able to recover. In any case, you definitely want to check run status of your nodes after this change to make sure you don’t discover a problem when you’re trying to troubleshoot some other problem.

I define my hierarchy in hiera itself using the puppet/hiera module, so here is the yaml for hiera to parse as well as the resulting hiera.yaml, the change is in bold:

# portion of hiera/puppet_role/puppet.yaml, which applies to the puppet master
hiera::hierarchy:
  - 'clientcert/%%{::}{clientcert}'
  - 'puppet_role/%%{::}{puppet_role}'
  - 'osfamily-release/%%{::}{osfamily}-%%{::}{operatingsystemmajrelease}'
  - 'datacenter/%%{::}{datacenter}'
  - 'global'

# /etc/puppetlabs/puppet/hiera.yaml
# managed by puppet
---
:backends:
- eyaml
- yaml

:logger: console

:hierarchy:
  - "clientcert/%{clientcert}"
  - "puppet_role/%{puppet_role}"
  - "osfamily-release/%{osfamily}-%{operatingsystemmajrelease}"
  - "datacenter/%{datacenter}"
  - global

:eyaml:
  :datadir: "/etc/puppetlabs/puppet/environments/%{::environment}/hiera"
  :extension: yaml
  :pkcs7_private_key: "/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem"
  :pkcs7_public_key: "/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem"

:yaml:
  :datadir: "/etc/puppetlabs/puppet/environments/%{::environment}/hiera"

:merge_behavior: deeper

This change will need to be put in place on the master, the master service restarted, and no dissimilar configs exist in the wrong location before agents will see the changes we make below (I had a /etc/puppetlabs/code/hiera.yaml that slightly vared from /etc/puppetlabs/puppet.hiera.yaml and it kept winning out till I removed it and restarted pe-puppetserver). You can force the run now, or wait up to two full run cycles before verifying that all your agents see the changes.

The second step is to populate the two OS/release files with the specific sets you want to use. I use saz/ssh, which allows me to use the ssh::server::options parameter to free-hand some stanzas into /etc/sshd_config. These commands replicate my settings, again according to Modern for EL7 and Intermediate for EL6:

mkdir hiera/osfamily-release
cat > hiera/osfamily-release/RedHat-6.yaml << EOF
---
ssh::server::options:
  'KexAlgorithms'            : 'diffie-hellman-group-exchange-sha256'
  'Ciphers'                  : 'aes256-ctr,aes192-ctr,aes128-ctr'
  'MACs'                     : 'hmac-sha2-512,hmac-sha2-256'
EOF

cat > hiera/osfamily-release/RedHat-7.yaml << EOF
---
ssh::server::options:
  'KexAlgorithms'            : 'curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256'
  'Ciphers'                  : 'chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr'
  'MACs'                     : 'hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com'
EOF

There’s one final step: merge settings. You may have noticed the merge_behavior setting in my hiera.yaml above, but that’s defunct. Now you must set the lookup options. I do this in my least specific hiera file, hiera/global.yaml:

lookup_options:
  profile::base::linux::sudo_confs:
    merge: deep
  profile::base::linux::logrotate_rules:
    merge: deep
  ssh::server::options:
    merge: deep

If you don’t add this, then you’ll only get the first ssh::server::options values found, even for sub-keys like Ciphers that were not set at the higher tier.

Once all of these changes are in place, your agents should get the new settings and restart sshd. Any new ssh connections to the affected servers will use the specified security sets and ONLY the specified security sets. Existing connections will persist until the server or client end the sessions. We can now use curve25519-sh256@libssh.org as a KexAlgorithm with an EL6 node, but we would fail to connect to an EL7 node as only diffie-hellman-group-exchange-sha256 is available. If we re-run the Rebex SSH Test, our Modern servers show all green now. Success!

Addendum: Peter Souter notified me on twitter about his mozilla_ssh_hardening module (GitHub only at this time) that enforces the Mozilla recommendations on Ubuntu 16.04, CentOS 7, and CentOS 6. You can use that module to replace some of the work above, as long as you do not require conflicting customizations. I still hope this articles helps you understand the workings of hiera merges and the need for vetted security configurations.

If you missed the news this past week, the Puppet 5 Platform was released! Read the announcement and the release notes for some great details. Congratulations to everyone at Puppet for getting this new release out the door. I’m looking forward to diving in with it as soon as a Puppet Enterprise release is out, since I’ve converted even my home lab away from Puppet Opensource.

There are a few things I’ve learned from the announcement thread, slack, and my own experiences with it in the last few days. It’s still early, so I am sure this will get out of date quickly, but I hope it helps others in the short term.

• Puppet 5 AIO provides Ruby 2.4.1, so your tests should use it as well – even if you’re not using AIO puppet, it’s still helpful for any puppet modules.
• PuppetDB requires postgresql96, but it’s not a dependency on the puppetdb package, since you can install puppetdb and postgresql on different hosts. Version 4.x works with postgresql96, so upgrade that first, then puppet. Detail here.
• Puppet 5 includes a vendored version of the semantic_puppet gem. In Puppet 4.7 and below, it had a dependency on the external semantic_puppet getm. The gem is used by metadata-json-lint, which is often part of a puppet rspec test setup. Check out the metadata-json-lint README installation section to see how to deal with this. If your tests run against ~> 4.0 then you’re probably okay.
• There’s a new version of puppetlabs_spec_helper that apparnetly has some issues with spec fixtures and symlinks (from slack, nothing to quote). I haven’t hit this myself, it might already be fixed, but something to be aware of if you have symlink-related issues during testing.
• The combination of Puppet 5, rpsec-puppet, and the new puppetlabs_spec_helper are more stringent than Puppet 4 is. I’m not sure which of the three components specifically triggers it. I was testing for a resource that required another service, which was not part of the define I was testing (here). With puppet 4, this was fine, but with puppet 5, it started generating errors in this travis run. The fix is simple, through using a pre_condition to provide the service in the catalog, seen in this commit.
• The first Puppet Enterprise release including Puppet 5 should be out sometime this fall.

That’s all I’ve run into so far. One last thing, here is a .travis.yml for testing component modules with both Puppet 4 and 5. You only need to update the matrix section, if you already have one, but I thought the whole thing might be helpful for those who don’t have tests yet.:

---
language: ruby
sudo: false
cache: bundler
notifications:
  email:
  on_failure: always
branches:
  only:
  - master
bundler_args: --without development system_tests
before_install: rm Gemfile.lock || true
script: bundle exec rake test
matrix:
  fast_finish: true
  include:
  - rvm: 2.3.1
    env: PUPPET_GEM_VERSION="~> 4.0" STRICT_VARIABLES=yes
  - rvm: 2.4.1
    env: PUPPET_GEM_VERSION="~> 5.0" STRICT_VARIABLES=yes

I haven’t written a blog post in a while because I’ve been bogged down in work and life and not had much time in the lab. To make sure I don’t get too out of practice, I’m going to try writing some shorter tips and tricks articles. Let me know what you think.

A few days ago, someone asked a great question on the puppet-users mailing list about the location of config files in the roles/profile pattern. It’s a good question, and we can go deeper because it assumes the location of config files outside of that pattern, too. I’m going to explain where I keep my config files, and templates, in the various types of modules. There’s no single correct answer here, this is just a framework for me.

To start, let’s describe the types of modules. Component modules describe a single application/technology/thing and are designed to be consumed by end users. This is pretty much anything on the forge, such as puppet/hiera to manage a Hiera implementation or puppetlabs/apache to manage apache, vhosts, etc. There’s also a sub-type of these modules, Private Components. The line here is blurry, but think component modules that are not designed to go on the forge. This could be a module for a company’s internal application, very similar to a traditional component module, or an cluster of custom facts. Pretty much anything that’s not a Component module, or our final type: Profile modules. This last type is the collection of classes that make up your role/profile pattern implementation. They’re often simply called profile, but maybe there is more than one module if you have a lot of business groups using the puppet system. They differ from both types of component modules in that they contain the business logic of your implementation and are where you compose the collection of component modules that you use. I wrote an article on what goes in a role or profile, too.

In Component modules, the relevant configuration files or templates for the component are collected. In an ssh module, you’d have the ssh_config and sshd_config data; a sudo module would have sudoers and a template for sudoers.d/ files. Private Component modules vary quite a bit in functionality, but I treat them like regular component modules. If the module is for custom facts, there’s no need to put files or templates in it. If it’s for an internal app, the configuration files are stored in that module.

Your Role/Profile modules are a little more complicated. If you have a component module for apache, you likely have a profile class for apache, perhaps profile::apache or profile::somegroup::apache. The component module probably has its own file or template, but it may accept alternative files and templates. In this case, I create a sub-directory with the module subclass name, such as files/apache or templates/apache,  and add the file(s) there, e.g. templates/apache/vhosts.erb.

This is a pretty simple layout. The only real difficulty is when you have a private component modules and a profile for that component: do you put the file/template with the profile or the component module? I tend to lean toward the private component modules first, but I’ve done both.

I hope this helps and I’d love to hear of any other layouts you’ve had success with!

Last week, I moved the home lab to Puppet Enterprise. One of the things I love about PE is the Console. However, I am a member of Vox Pupuli and we develop Puppetboard (the app AND the module) so it is convenient for me to use it and tie it into PE as well. Though the two overlap, each has functionality the other doesn’t. I really love the count of agent runs by status on the Puppetboard Overview page, for instance. After my migration, however, my previously-working puppetboard just gave me HTTP 500 errors. Fixing it took some wrangling. Thanks to Tim Meusel for his assistance with the cert issue.

First, let’s look at the existing manifest and hiera data for my profile::puppetboard class:

Continue reading →

The Roles and Profiles pattern by Craig Dunn is a very common pattern used by Puppet practitioners. I’ve written about it before. One of the most common questions I see is, what goes into a Role or Profile class? Craig’s article provides some guidelines, specifically these two:

• A role includes one or more profiles to define the type of server
• A profile includes and manages modules to define a logical technical stack

Those are pretty helpful, but it’s not an exhaustive list, nor does it describe what is prohibited in each type of class. While the main goal of the pattern is composition, I have my own guidelines I follow that may help others:

Roles

• No parameters
• Includes profile classes
• [Rarely] Ordering of resources that come from two separate profiles
• Contains nothing else.

Here’s an example role for an application server:

role appX {
  include profile::base
  include profile::apache
  include profile::appX
  Package<| tag == 'profile::apache' |> -> Package <| tag == 'profile::appX' |>
}

Profiles

• Optional parameters
• Includes component modules
• Includes basic resource types (built-in or from component modules)
• Calls functions, include hiera_*() and lookup()
• Traverses and manipulates variables to process their data
• Conditionals (limited)
• Ordering of resources, within the profile
• May call other profiles, but should be used sparingly
• If the code is >100 lines, consider separating the profile class into its own module, or finding an existing component module that include the functionality (100 is a very arbitrary number, feel free to adjust it to a number indicating when you want to start thinking about this option)

Here’s an example of a profile that calls other profiles:

class profile::base {

  # Include OS specific base profiles.
  case $::kernel {
    'linux': {
      include profile::base::linux
    }
    'windows': {
      include profile::base::windows
    }
    'JUNOS': {
      include profile::base::junos
    }
    default: {
      fail ("Kernel: ${::kernel} not supported in ${module_name}")
    }
  }
}

Here’s an example of a more complex profile that has parameters and includes other component modules, basic resources, functions, iteration, conditionals, and even another profile:

class profile::base::linux (
  $yumrepo_url,
  $cron_purge          = true,
  $domain_join         = false,
  $metadata_expire     = 21600, # Default value for yum is 6 hours = 21,600 seconds
  $sudo_confs          = {},
  $manage_firewall     = true,  # Manage iptables
  $manage_puppet_agent = true,  # Manage the puppet-agent including upgrades
) {
  # Manage the basics, but allow users to override some management components with flags
  if $manage_firewall {
    include profile::linuxfw
  }
  if $manage_puppet_agent {
    include puppet_agent
  }

  include ntp
  include rsyslog::client
  include motd

  # SSH server and client
  include ssh::server
  include ssh::client

  # Sudo setup
  include sudo
  $sudo_confs.each |$group, $config| {
    sudo::conf{ $group:
      * => $config,
    }
  }

  yumrepo {'local-el':
    ensure          => present,
    descr           => 'Local EL - x86_64',
    baseurl         => $yumrepo_url,
    enabled         => 1,
    gpgcheck        => 0,
    metadata_expire => $metadata_expire,
  }
  Yumrepo['local-el'] -> Package<| |>

  # Ensure unmanaged cron entries are removed
  resources { 'cron':
    purge => $cron_purge,
  }

  if $domain_join {
    include profile::domain_join
  }
}

Summary

The Roles and Profiles pattern is all about composition. The Style Guide helps you with layout and semantic choices. It doesn’t hurt to add your own rules about content and design, too. There’s no defined Best Practice here, but I hope these guidelines help you shape your own practice. Enjoy!

I have been using Puppet Enterprise at work and Puppet OpenSource at home for a few years now. There’s a lot to love about both products, but since work uses PE and new features tend to land there first, I have been thinking about trying PE at home as well. I don’t have a large fleet or a large change velocity, so I think the conversion of the master might take some work but the agents will be easy. This will let me play with PE-only functions in my lab (specifically the PE Pipeline plugin for Jenkins) and reduces concern about drift between lab findings and work usage. It does have the downside that some of my blog articles, which I was always assured would work with the FOSS edition, may not be as foolproof in the future. However, I rarely saw that as a problem going the other way in the past, with mcollective management being the only exception. I haven’t written about mcollective much, so I think this is worth the tradeoff.

I am going to migrate my systems, rather that start fresh. This article was written as the migration happened, so if you pursue a similar migration, please read the whole article before starting – I did some things backwards or incomplete, and made a few mistakes that you could avoid. It was also written as something of a stream of consciousness. I’ve tried to edit it as best I can without losing that flow. I do hope you find it valuable.

Pre-Flight Checklist

Puppet Enterprise is a commercial product. You can use it for free with up to 10 nodes, though. This is perfect for my 9 managed-node home network. After that, it costs something around $200/node/year. Make sure you stay within that limit or pony up – we are responsible adults, we pay for the products we use. If you replace nodes, you can deactivate those nodes so they don’t continue to eat into your license count.

Continue reading →