Edit: In an earlier edition, I credited the wrong newsletter as the source. My apologies to R.I.Pienaar!

In this past week’s DevCo Newsletter, I saw the Rebex SSH Check, which reminded me that I’ve locked down the SSH server security configuration at work, but not at home. Sounds like a good opportunity to blog about the process!

Now, I’m in security, but I’m not all that about the security settings. The names vary from descriptive to really obtuse, and there’s three keys that need managed: ciphers, MACs, and KexAlgorithms (that’s Key Exchange Algorithms, which is the name I’m more familiar with). The key to security is knowing when you don’t know, and seeking out that expertise. I am very thankful for Mozilla’s really great security guidelines, including an OpenSSH guide. There are sections for Modern and Intermediate security, depending on what is available for the systems you are securing. For me, these align with the Red Hat/CentOS EL7 (Modern) and EL6/5 (Intermediate) distros that I use.

The first step is making sure we have a tier in hiera for each OS/release we support, otherwise sshd could fail to restart when it encounters a cipher set name that is unknown to the openssh version in use. That could be bad, especially if we don’t have some form of iLO console to the nodes, though if we have puppet running on a regular basis or through mcollective, we *should* be able to recover. In any case, you definitely want to check run status of your nodes after this change to make sure you don’t discover a problem when you’re trying to troubleshoot some other problem.

I define my hierarchy in hiera itself using the puppet/hiera module, so here is the yaml for hiera to parse as well as the resulting hiera.yaml, the change is in bold:

# portion of hiera/puppet_role/puppet.yaml, which applies to the puppet master
hiera::hierarchy:
  - 'clientcert/%%{::}{clientcert}'
  - 'puppet_role/%%{::}{puppet_role}'
  - 'osfamily-release/%%{::}{osfamily}-%%{::}{operatingsystemmajrelease}'
  - 'datacenter/%%{::}{datacenter}'
  - 'global'

# /etc/puppetlabs/puppet/hiera.yaml
# managed by puppet
---
:backends:
- eyaml
- yaml

:logger: console

:hierarchy:
  - "clientcert/%{clientcert}"
  - "puppet_role/%{puppet_role}"
  - "osfamily-release/%{osfamily}-%{operatingsystemmajrelease}"
  - "datacenter/%{datacenter}"
  - global

:eyaml:
  :datadir: "/etc/puppetlabs/puppet/environments/%{::environment}/hiera"
  :extension: yaml
  :pkcs7_private_key: "/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem"
  :pkcs7_public_key: "/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem"

:yaml:
  :datadir: "/etc/puppetlabs/puppet/environments/%{::environment}/hiera"

:merge_behavior: deeper

This change will need to be put in place on the master, the master service restarted, and no dissimilar configs exist in the wrong location before agents will see the changes we make below (I had a /etc/puppetlabs/code/hiera.yaml that slightly vared from /etc/puppetlabs/puppet.hiera.yaml and it kept winning out till I removed it and restarted pe-puppetserver). You can force the run now, or wait up to two full run cycles before verifying that all your agents see the changes.

The second step is to populate the two OS/release files with the specific sets you want to use. I use saz/ssh, which allows me to use the ssh::server::options parameter to free-hand some stanzas into /etc/sshd_config. These commands replicate my settings, again according to Modern for EL7 and Intermediate for EL6:

mkdir hiera/osfamily-release
cat > hiera/osfamily-release/RedHat-6.yaml << EOF
---
ssh::server::options:
  'KexAlgorithms'            : 'diffie-hellman-group-exchange-sha256'
  'Ciphers'                  : 'aes256-ctr,aes192-ctr,aes128-ctr'
  'MACs'                     : 'hmac-sha2-512,hmac-sha2-256'
EOF

cat > hiera/osfamily-release/RedHat-7.yaml << EOF
---
ssh::server::options:
  'KexAlgorithms'            : 'curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256'
  'Ciphers'                  : 'chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr'
  'MACs'                     : 'hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com'
EOF

There’s one final step: merge settings. You may have noticed the merge_behavior setting in my hiera.yaml above, but that’s defunct. Now you must set the lookup options. I do this in my least specific hiera file, hiera/global.yaml:

lookup_options:
  profile::base::linux::sudo_confs:
    merge: deep
  profile::base::linux::logrotate_rules:
    merge: deep
  ssh::server::options:
    merge: deep

If you don’t add this, then you’ll only get the first ssh::server::options values found, even for sub-keys like Ciphers that were not set at the higher tier.

Once all of these changes are in place, your agents should get the new settings and restart sshd. Any new ssh connections to the affected servers will use the specified security sets and ONLY the specified security sets. Existing connections will persist until the server or client end the sessions. We can now use curve25519-sh256@libssh.org as a KexAlgorithm with an EL6 node, but we would fail to connect to an EL7 node as only diffie-hellman-group-exchange-sha256 is available. If we re-run the Rebex SSH Test, our Modern servers show all green now. Success!

Addendum: Peter Souter notified me on twitter about his mozilla_ssh_hardening module (GitHub only at this time) that enforces the Mozilla recommendations on Ubuntu 16.04, CentOS 7, and CentOS 6. You can use that module to replace some of the work above, as long as you do not require conflicting customizations. I still hope this articles helps you understand the workings of hiera merges and the need for vetted security configurations.