In the last post, we built a Puppet master. The master is ready for agents to connect, but currently it doesn’t actually do very much when they do. Our site.pp manifest is very boring:

[root@puppet ~]# cat /etc/puppet/manifests/site.pp
notify {"Agent connection is successful": }

All agents get the same manifest, and it’s just a notify statement. We want to do more than that, we want to actually manage some important configuration. Before we get into that, I want to mention a few basics that you may want to review on your own. We’ll touch on these items throughout this and future Puppet blogs, with the expectation that you’ll have read up on them already or will reference as needed.

• There’s a Puppet Style Guide. Follow it. Your coworkers, and possibly your future self, will thank you.
• Learning Puppet is a great tutorial. The second chapter is about manifests and moves onto other components we will talk about. If you’re not familiar with Puppet, pull it up beside this post and flip back and forth.
• The Puppet Forge has over 2,000 modules that we will make heavy use of. Always check here first.
• Use of Git with Puppet is highly encouraged. If you aren’t familiar with VCS in general, or Git in particular, find some tutorials. I recommend CodeSchool‘s interactive course.

Importing modules from the Forge

At this point, most tutorials will teach you how to write your own manifest to manage ssh, ntp, or another common service. You’ll then be told later that there’s already a module for common services and configurations on the Forge. You’ll eventually want to write your own modules, but to me it makes more sense to stand on the shoulders of others until we have a good foundation. With that in mind, we’ll skip straight to importing from the Forge and our service of choice will be ntp, a service nearly everyone needs.

You can search for modules in two ways. At the CLI, use puppet module search <string>. You will get a very plain listing of results:

[root@puppet ~]# puppet module search ntp
Notice: Searching https://forge.puppetlabs.com ...
NAME                      DESCRIPTION                                 AUTHOR             KEYWORDS
DavidSchmitt-ntp                                                      @DavidSchmitt      ntp time
a2tar-ntp                 Install ntp on ubuntu                       @a2tar             ubuntu, ntp
aageyev-ntp               Install ntp on ubuntu                       @aageyev           ubuntu, ntp
adenning-winntp           Manage the win32time service.               @adenning          ntp windows win32time
csail-ntp                 Configures NTP servers and clients          @csail             ntp ntpd debian
dhoppe-ntp                NTP Module                                  @dhoppe            debian ubuntu ntp
erwbgy-ntp                configure and manage ntpd                   @erwbgy            ntp time CentOS rhel
erwbgy-system             Manage Linux system resources and servi...  @erwbgy            rhel cron exec ntp
evenup-time               Manages the timezone and ntp.               @evenup            ntp
example42-ntp             Puppet module for ntp                       @example42         ntp example42
example42-openntpd        Puppet module for openntpd                  @example42         openntpd ntp
ghoneycutt-ntp            Manage NTP                                  @ghoneycutt        ntp time sync
hunner-mcollective        MCollective module for managing the MC ...  @hunner            mcollective ntp
kickstandproject-ntp      UNKNOWN                                     @kickstandproject  ntp
mmitchell-puppetlabs_ntp  UNKNOWN                                     @mmitchell
mthibaut-ntp              NTP Module                                  @mthibaut          ntp hiera
oppegaard-ntpd            OpenNTP module for OpenBSD                  @oppegaard         ntpd ntp openbsd
puppetlabs-ntp            NTP Module                                  @puppetlabs        ntp time aix rhel
saz-ntp                   UNKNOWN                                     @saz               ntp time SuSE OEL
thias-ntp                 Network Time Protocol module                @thias             ntp ntpd centos rhel
warriornew-ntp            ntp setup                                   @warriornew        ntp
xucchini-ntp                                                          @xucchini          ubuntu ntp time kvm

The ordering is alphanumeric based on the Name field. The name consists of the author’s alias and the module’s name. You have to decide based on these results which to install, but there’s not a lot to go on. It does look like there’s a vendor module (puppetlabs-ntp) but there’s also a module by some mmitchell with puppetlabs in the name. Kind of confusing and doesn’t tell us what is the correct choice. Our next option is searching at the forge.

The forge’s results are ordered by number of downloads. This gives us some idea of the popularity of a module, but certainly not the correctness of it. With ntp, the dropoff from the 1st to 2nd result is dramatic, 75k to 6k at the time of writing. Searching for other modules, like apache, provide less clear results – 94k to 79k to 1k. The 3rd is probably not what you want, but the 1st and 2nd results are very close. You can click on the module name to get a lot more information on each module, including a README, the Changelog and Dependencies, Types provided, and License, if applicable to the module. Use this information to determine what module is appropriate for your environment and use case.

After reviewing the forge results, puppetlabs-ntp is definitely the module we want. Install the module at the CLI with puppet module install <name>. Remember that the name is the combination of author alias and module name:

[root@puppet etc]# puppet module install puppetlabs-ntp
Notice: Preparing to install into /etc/puppet/modules ...
Notice: Downloading from https://forge.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
/etc/puppet/modules
└─┬ puppetlabs-ntp (v3.0.1)
  └── puppetlabs-stdlib (v4.1.0)

Notice that puppet was nice enough to add the dependency of puppetlabs-stdlib for us. Sweet, less work for us to do. If at some point you want to uninstall this or any other module, use puppet module uninstall <name>, again specifying the author-module combination. You can also see the modules installed with puppet module list:

[root@puppet etc]# puppet module list
/etc/puppet/modules
├── puppetlabs-ntp (v3.0.1)
└── puppetlabs-stdlib (v4.1.0)
/usr/share/puppet/modules (no modules installed)

Adding our module to site.pp

With an ntp module in place, we’re ready to proceed. Go back to the module’s forge page and review the README. Because puppetlabs does a great job with documentation, everything you need to use the module is documented here. Some modules do not have such great documentation. In that case, near the top of the page is a Project URL link. This link will get you access to the source code for the module. You can also use that link to review and fix bugs. There’s a good contribution guide applicable to puppet itself and puppetlabs modules, though the general workflow (clone, patch, commit, push, pull request) should serve well for most modules.

In the Usage section, there’s a minimum config which is one line. Replace the contents of your site.pp file with this:

[root@puppet etc]# cat > /etc/puppet/manifests/site.pp
include '::ntp'
^D

Any agent that connects should now receive some ntp settings from the master.

Performing an agent run

Let’s check a few things first, so we can see exactly what kind of magic puppet provides. Do we have either the package for ntp or the config file /etc/ntp.conf on our system?

[root@puppet etc]# rpm -qa | grep -i ntp
ntpdate-4.2.6p5-1.el6.centos.x86_64
[root@puppet etc]# ls /etc/ntp.conf
ls: cannot access /etc/ntp.conf: No such file or directory

This is not part of the minimal install, so of course it is not there. Now, if I have the local agent talk to itself, we can see what will happen by adding the ‘-noop’, or no operation, flag to our agent statement:

[root@puppet etc]# puppet agent --test --noop
Info: Retrieving plugin
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/puppet_vardir.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/pe_version.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/root_home.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/facter_dot_d.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
Info: Caching catalog for puppet.nelson.va
Info: Applying configuration version '1391107572'
Notice: /Stage[main]/Ntp::Install/Package[ntp]/ensure: current_value absent, should be present (noop)
Notice: Class[Ntp::Install]: Would have triggered 'refresh' from 1 events
Notice: /Stage[main]/Ntp::Config/File[/etc/ntp.conf]/ensure: current_value absent, should be file (noop)
Notice: Class[Ntp::Config]: Would have triggered 'refresh' from 1 events
Info: Class[Ntp::Config]: Scheduling refresh of Class[Ntp::Service]
Notice: Class[Ntp::Service]: Would have triggered 'refresh' from 1 events
Info: Class[Ntp::Service]: Scheduling refresh of Service[ntp]
Notice: /Stage[main]/Ntp::Service/Service[ntp]/ensure: current_value stopped, should be running (noop)
Info: /Stage[main]/Ntp::Service/Service[ntp]: Unscheduling refresh on Service[ntp]
Notice: Class[Ntp::Service]: Would have triggered 'refresh' from 1 events
Notice: Stage[main]: Would have triggered 'refresh' from 3 events
Notice: Finished catalog run in 0.33 seconds

Look at the lines that include the string ‘noop’. First is “/Stage[main]/Ntp::Install/Package[ntp]/ensure: current_value absent, should be present (noop)”. Let’s parse this. In the stage called ‘main’, the Ntp::Install module looks at a Package resource called ‘ntp’ and its attribute ensure. Ensure currently has a value of absent, which should be present according to the manifest. That doesn’t look very much like our “include ‘::ntp'” statement. If you go back to the documentation, scroll down to the Reference section. There are three classes as part of our module, one of which is ntp::install. The ntp class includes ntp::install, hence our reference to Ntp::Install. The description says that it handles the packages, hence the reference to Package[ntp]. Because the basic usage of this module is to install ntp, the ensure attribute is set to present, which gives us the tail end of the log statement.

Side note: You will notice some peculiarities in capitalization. The output from an agent does not match the manifest. I have not found a specific explanation for this (please comment if you have one), but capitalization appears to be lowercase for language declaration and Firstupper for reports, by design.

The next two ‘noop’ statements are similar. The 2nd is where ntp::config would have added the /etc/ntp.conf file and the 3rd is where ntp::service would have started the service. There are two other messages of importance in there. “Class[Ntp::Config]: Scheduling refresh of Class[Ntp::Service]” indicates that when /etc/ntp.conf is created, it will alert ntp::service of the change. The next statement is “Class[Ntp::Service]: Would have triggered ‘refresh’ from 1 events” which shows that the service would have been refreshed, or restarted. This pattern is repeated a few lines later after the service “should be running”. Of course, none of this actually happened because of –noop mode. As this all looks good, you can run the command without the noop flag. It looks remarkably similar, except that actions were taken. The output below is abbreviated:

[root@puppet etc]# puppet agent --test
...
Info: Applying configuration version '1391107572'
Notice: /Stage[main]/Ntp::Install/Package[ntp]/ensure: created
Notice: /Stage[main]/Ntp::Config/File[/etc/ntp.conf]/content:
--- /etc/ntp.conf       2013-07-15 09:18:47.000000000 +0000
+++ /tmp/puppet-file20140130-2499-9jewwd-0      2014-01-30 19:05:38.020962469 +0000
@@ -1,53 +1,22 @@
-# For more information about this file, see the man pages
-# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).
+# ntp.conf: Managed by puppet.
+#
+# Keep ntpd from panicking in the event of a large clock skew
+# when a VM guest is suspended and resumed.
+tinker panic 0

-driftfile /var/lib/ntp/drift
-
-# Permit time synchronization with our time source, but do not
-# permit the source to query or modify the service on this system.
+# Permit time synchronization with our time source, but do not'
+# permit the source to query or modify the service on this system.'
...
Info: /Stage[main]/Ntp::Config/File[/etc/ntp.conf]: Filebucketed /etc/ntp.conf to puppet with sum 7fda24f62b1c7ae951db0f746dc6e0cc
Notice: /Stage[main]/Ntp::Config/File[/etc/ntp.conf]/content: content changed '{md5}7fda24f62b1c7ae951db0f746dc6e0cc' to '{md5}af2c4d9dba34bb187ddfbf76bbf74815'
Info: Class[Ntp::Config]: Scheduling refresh of Class[Ntp::Service]
Info: Class[Ntp::Service]: Scheduling refresh of Service[ntp]
Notice: /Stage[main]/Ntp::Service/Service[ntp]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Ntp::Service/Service[ntp]: Unscheduling refresh on Service[ntp]
Notice: Finished catalog run in 9.99 seconds

This does look different than the noop run. All the changes marked noop are actually made, so we receive additional information on the changes themselves. After the package ntp is installed via the ‘ensure’ attribute, /etc/ntp.conf is updated. What follows is diff output. We didn’t have this file before, so why is there a diff? When ntp is installed, the package includes a conf file. The new conf file is applied afterward, in series, so the diff is between the package’s config and the one provided by puppet. After the diff follows a statement referencing a filebucket. This is where files are recycled, rather than deleted, so you can restore them. The default installation provides this on each agent, or you can create a remote filebucket. This helps reduce the number of “ohno seconds” in your life.

Following that everything looks like it did during the noop run. The ntp service receives its notification to refresh and changes its state to running. Let’s check – see if ntp is installed, see if it is running, and see if it will be running after a reboot:

[root@puppet etc]# rpm -qa | grep ntp
ntp-4.2.6p5-1.el6.centos.x86_64
ntpdate-4.2.6p5-1.el6.centos.x86_64
[root@puppet etc]# ps -ef | grep ntp
ntp       2688     1  0 19:14 ?        00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
root      2822  1666  0 19:36 pts/0    00:00:00 grep ntp
[root@puppet etc]# chkconfig | grep ntp
ntpd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
ntpdate         0:off   1:off   2:off   3:off   4:off   5:off   6:off

What does idempotent and declarative really mean?

You may have seen the word “idempotent” thrown around while researching puppet. You’ve probably also heard the puppet language described as a declarative language. What does this mean to a puppet user? Idempotent means that multiple applications of a single operation will not change the resultant state beyond the first application. Put another way, doing the same thing two, three, or a hundred times with no other changes always results in the same state. We can see this by performing another agent test. Nothing will change now that ntp is installed, configured, and running:

[root@puppet etc]# puppet agent --test
Info: Retrieving plugin
...
Info: Caching catalog for puppet.nelson.va
Info: Applying configuration version '1391107572'
Notice: Finished catalog run in 0.27 seconds

Puppet’s declarative language is also very nice. If you were programming a utility to install and start ntp, it would be a much more complex program than it might appear at first blush. “Step 1: Install ntp. Step 2: Start ntp” is simple, right? What if ntp is already installed? What if it’s installed but has no configuration? What if ntp won’t start? What I’m running on some strange operating system that has no ntp? Your code goes from simple to convoluted in a hurry. Check this, check that, do this if that, etc. Plus, you have to be a programmer. With puppet’s declarative language, your manifests state what you want, not how to get there. You leave that up to puppet itself – it gets to deal with screwed up configs, weird operating systems, and whatever that klutz before you broke (who probably also happens to be you). And that’s okay if you’re a klutz and not a programmer, that’s why you’re using puppet. Let’s test this out. Remove /etc/ntp.conf and run the agent test to see what happens.

[root@puppet etc]# puppet agent --test
Info: Retrieving plugin
...
Info: Caching catalog for puppet.nelson.va
Info: Applying configuration version '1391107572'
Notice: /Stage[main]/Ntp::Config/File[/etc/ntp.conf]/ensure: defined content as '{md5}af2c4d9dba34bb187ddfbf76bbf74815'
Info: Class[Ntp::Config]: Scheduling refresh of Class[Ntp::Service]
Info: Class[Ntp::Service]: Scheduling refresh of Service[ntp]
Notice: /Stage[main]/Ntp::Service/Service[ntp]: Triggered 'refresh' from 1 events
Notice: Finished catalog run in 0.57 seconds
[root@puppet etc]# ps -ef | grep ntp
ntp       3132     1  0 19:46 ?        00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
root      3138  1666  0 19:47 pts/0    00:00:00 grep ntp

Nicely done, you klutz! You broke ntp! Thankfully, puppet is not a klutz. It redeploys ntp.conf, which results in a notification to the ntp service object which restarts ntp. We can see this because the ntpd PID is now different.

Configuring our module

All that’s great. We have a module from the forge and we’re going to force all agents to run ntp. Except maybe we don’t like something about the module. It was cool when you found it, but now it doesn’t want to leave the house anymore, it just sits there and eats crackers on your couch, leaving crumbs everywhere. You want it to dress up a little bit and try and impress you. Maybe use a different ntp server for variety. Thankfully, that’s easy to do without modifying the module (Note: puppetlabs makes highly modular modules. Not everyone does. If you write a module for the forge, you should, too. Choose wisely).

Back at the module’s documentation page, we see there’s a lot more under Usage that we ignored before. Let’s say that we do want to change the ntp servers in use. Let’s see what is used now:

[root@puppet etc]# grep server /etc/ntp.conf
server 0.centos.pool.ntp.org
server 1.centos.pool.ntp.org
server 2.centos.pool.ntp.org

Hmmm, that seems reasonable. But, for the sake of learning, we’ll make some changes. Let’s use 0.pool.ntp.org, 2.centos.pool.ntp.org, and 1.rhel.pool.ntp.org, just for grins. Looking at the 2nd usage example, we need to declare a class, rather than include it, and a list of servers. I wonder why that is. It turns out that when you include a class, you cannot pass any parameters to it. Whatever default parameters are provided by the module’s author are used. To use parameters, you must define the class and pass the parameters as key/value pairs (servers/array of servers in our case). Update your site.pp to follow the usage example:

[root@puppet etc]# cat > /etc/puppet/manifests/site.pp
class { '::ntp':
  servers => [ '0.pool.ntp.org', '2.centos.pool.ntp.org', '1.rhel.pool.ntp.org'],
}

Perform another agent run, first with –noop to see what the changes will be:

[root@puppet etc]# puppet agent --test --noop
Info: Retrieving plugin
...
Info: Applying configuration version '1391112473'
Notice: /Stage[main]/Ntp::Config/File[/etc/ntp.conf]/content:
--- /etc/ntp.conf       2014-01-30 19:46:15.212754769 +0000
+++ /tmp/puppet-file20140130-3155-yx1hld-0      2014-01-30 20:07:54.006242951 +0000
@@ -12,9 +12,9 @@
...
-server 0.centos.pool.ntp.org
-server 1.centos.pool.ntp.org
+server 0.pool.ntp.org
 server 2.centos.pool.ntp.org
+server 1.rhel.pool.ntp.org
...
Notice: /Stage[main]/Ntp::Config/File[/etc/ntp.conf]/content: current_value {md5}af2c4d9dba34bb187ddfbf76bbf74815, should be {md5}070f524f44040d25f72329d46dc80e7a (noop)
Notice: Class[Ntp::Config]: Would have triggered 'refresh' from 1 events
Info: Class[Ntp::Config]: Scheduling refresh of Class[Ntp::Service]
Notice: Class[Ntp::Service]: Would have triggered 'refresh' from 1 events
Info: Class[Ntp::Service]: Scheduling refresh of Service[ntp]
Notice: /Stage[main]/Ntp::Service/Service[ntp]: Would have triggered 'refresh' from 1 events
Notice: Class[Ntp::Service]: Would have triggered 'refresh' from 1 events
Notice: Stage[main]: Would have triggered 'refresh' from 2 events
Notice: Finished catalog run in 0.37 seconds

If you made a mistake, fix it now. Otherwise run without the noop. Check your /etc/ntp.conf and you will find the server list has changed.

[root@puppet ~]# cat /etc/ntp.conf | grep server
server 0.pool.ntp.org
server 2.centos.pool.ntp.org
server 1.rhel.pool.ntp.org

Perfect. We’ve got puppet to configure our server the way we specified, even it’s a non-optimal configuration.

Adding more modules

First, a user

Now is a perfect time to add some additional modules. As we mentioned in the prior article, root is allowed to ssh to the server. That’s bad policy. We should create a local user and then disable root’s ability to log in. We could create a local user, but a user is just another resource for puppet to manage so we’ll let it do its job. We don’t need a module, and indeed if you check the forge you won’t find a good module for this. However, Learning Puppet has an example of a user resource definition. Let’s copy it with two wrinkles, an gid of ‘507’ and a shell ‘/bin/bash’, to add the user ‘dave’ from its example documentation and test the site manifest:

[root@puppet ~]# cat >> /etc/puppet/manifests/site.pp

user { 'dave':
  ensure     => present,
  uid        => '507',
  gid        => '507',
  shell      => '/bin/bash',
  home       => '/home/dave',
  managehome => true,
}
^D
[root@puppet ~]# puppet agent --test --noop
Info: Retrieving plugin
...
Info: Applying configuration version '1391117535'
Notice: /Stage[main]/Main/User[dave]/ensure: current_value absent, should be present (noop)
Notice: Class[Main]: Would have triggered 'refresh' from 1 events
Notice: Stage[main]: Would have triggered 'refresh' from 1 events
Notice: Finished catalog run in 0.23 seconds

As everything looks good, run the agent without noop and ‘dave’ is now an account on our server. Except it isn’t. What went wrong?

Error: Could not create user dave: Execution of '/usr/sbin/useradd -g 507 -d /home/dave -s /bin/zsh -u 507 -m dave' returned 6: useradd: group '507' does not exist
Error: /Stage[main]/Main/User[dave]/ensure: change from absent to present failed: Could not create user dave: Execution of '/usr/sbin/useradd -g 507 -d /home/dave -s /bin/zsh -u 507 -m dave' returned 6: useradd: group '507' does not exist

This is a reminder that the -noop flag is not flawless. Just because the command returns no errors does not mean the compiled catalog will successfully apply a node. We’ll talk about using environments to test changes later, but for now just be cautious to test your manifests on one agent before letting other agents pick up a catalog that may fail to apply. We can fix this issue by defining the group. The Learning Puppet page doesn’t show us a group definition, but it shows us how to get it, with puppet resource <type> <name>. Examine the group wheel. We can take this group definition and use it to construct a group called dave with an id of 507 and add that to our manifest:

[root@puppet files]# puppet resource group wheel
group { 'wheel':
  ensure => 'present',
  gid    => '10',
}
[root@puppet files]# cat >> /etc/puppet/manifests/site.pp

group { 'dave':
  ensure => 'present',
  gid    => '507',
}
^D

Now if you force another run you’ll see success, both with –noop and without.

[root@puppet files]# puppet agent --test
Info: Retrieving plugin
...
Info: Caching catalog for puppet.nelson.va
Info: Applying configuration version '1391121769'
Notice: /Stage[main]/Main/Group[dave]/ensure: created
Notice: /Stage[main]/Main/User[dave]/ensure: created
Notice: Finished catalog run in 0.82 seconds

With the user and group created, use passwd dave and set the password. Ensure you can log in as dave and su to root before continuing.

Manage SSH

It’s time to manage ssh. If you check the forge, you’ll find a nice module for ssh by author saz. Go ahead and install that. Looking at the usage, it suggests adding include ssh::server to our site manifest. However, if you read the style guide, you’ll notice that section 11.6 suggests explicitly specifying the empty namespace to prevent scoping issues, so we’ll use ::ssh::server instead. Perform a noop test and see what is going to happen:

[root@puppet files]# cat >> /etc/puppet/manifests/site.pp

include ::ssh::server
^D

Run a –noop and you’ll see what is planned for your agent.

[root@puppet files]# puppet agent --test --noop
Info: Retrieving plugin
...
Info: Caching catalog for puppet.nelson.va
Info: Applying configuration version '1391121972'
Notice: Finished catalog run in 0.47 seconds

Well, that seems odd! It’s not – remember idempotency? We already have ssh server running, that’s how we got logged in as root and dave. The config file appears to be the same, though, which isn’t going to fix our security issue. We need to tweak the config, but how? The forge page doesn’t really provide a lot of clues, nor does the project URL. You can snoop through the code and some readers may stumble on the answer on their own. I found part of the answer in a bug reported against the project. We can use ::ssh::server::configline to set PermitRootLogin to no. We will also need to add a forge module yguenane-augeas (if you don’t have it installed you’ll get a missing dependency error, but you get to benefit from my mistakes). Install the module, add the line to your manifest, and perform a noop.

[root@puppet files]# puppet module install yguenane-augeas
Notice: Preparing to install into /etc/puppet/modules ...
Notice: Downloading from https://forge.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
/etc/puppet/modules
└─┬ yguenane-augeas (v0.1.1)
  └── yguenane-ygrpms (v0.1.0)
[root@puppet files]# cat >> /etc/puppet/manifests/site.pp
::ssh::server::configline { 'PermitRootLogin': value => 'no' }
^D
[root@puppet files]# puppet agent --test --noop
Info: Retrieving plugin
...
Info: Caching catalog for puppet.nelson.va
Info: Applying configuration version '1391123634'
Notice: Augeas[sshd_config_PermitRootLogin](provider=augeas):
--- /etc/ssh/sshd_config        2014-01-30 23:09:41.132809605 +0000
+++ /etc/ssh/sshd_config.augnew 2014-01-30 23:13:55.953215192 +0000
@@ -136,4 +136,4 @@
 #      X11Forwarding no
 #      AllowTcpForwarding no
 #      ForceCommand cvs server
+PermitRootLogin no

Notice: /Stage[main]/Main/Ssh::Server::Configline[PermitRootLogin]/Augeas[sshd_config_PermitRootLogin]/returns: current_value need_to_run, should be 0 (noop)
Info: /Stage[main]/Main/Ssh::Server::Configline[PermitRootLogin]/Augeas[sshd_config_PermitRootLogin]: Scheduling refresh of Class[Ssh::Server::Service]
Notice: Ssh::Server::Configline[PermitRootLogin]: Would have triggered 'refresh' from 1 events
Notice: Class[Ssh::Server::Service]: Would have triggered 'refresh' from 1 events
Info: Class[Ssh::Server::Service]: Scheduling refresh of Service[sshd]
Notice: /Stage[main]/Ssh::Server::Service/Service[sshd]: Would have triggered 'refresh' from 1 events
Notice: Class[Ssh::Server::Service]: Would have triggered 'refresh' from 1 events
Notice: Class[Main]: Would have triggered 'refresh' from 1 events
Notice: Stage[main]: Would have triggered 'refresh' from 2 events
Notice: Finished catalog run in 0.72 seconds

Now you can see a diff where PermitRootLogin is set to no. As we saw with ntp, when the sshd config file is changed, a notification is sent to the service object so that it knows to refresh. Run a normal –test and this will apply. Ssh to localhost and test logging in as root. You’ll notice an error, there is no ssh client; this is not included in the minimal install of CentOS.

If you go back to the saz-sudo page, you’ll notice there are two other use cases, one for client only and one for client and server. We want the client and server both, so it looks like we can change our site.pp a tad to have a single instance of include ::ssh instead of include ::ssh::server. Run a –noop and make sure everything is okay before running a –test.

[root@puppet files]# ssh localhost
-bash: ssh: command not found
[root@puppet files]# vi /etc/puppet/manifests/site.pp
[root@puppet files]# tail -2 /etc/puppet/manifests/site.pp
include ::ssh
::ssh::server::configline { 'PermitRootLogin': value => 'no' }
[root@puppet files]# puppet agent --test --noop
Info: Retrieving plugin
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/puppet_vardir.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/pe_version.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/root_home.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/facter_dot_d.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
Info: Caching catalog for puppet.nelson.va
Info: Applying configuration version '1391124320'
Notice: /Stage[main]/Ssh::Client::Install/Package[openssh-clients]/ensure: current_value absent, should be present (noop)
Notice: Class[Ssh::Client::Install]: Would have triggered 'refresh' from 1 events
Notice: /Stage[main]/Ssh::Client::Config/File[/etc/ssh/ssh_config]/ensure: current_value absent, should be file (noop)
Notice: Class[Ssh::Client::Config]: Would have triggered 'refresh' from 1 events
Notice: Stage[main]: Would have triggered 'refresh' from 2 events
Notice: Finished catalog run in 0.71 seconds
[root@puppet files]# puppet agent --test
Info: Retrieving plugin
...
Info: Applying configuration version '1391124320'
Notice: /Stage[main]/Ssh::Client::Install/Package[openssh-clients]/ensure: created
Notice: /Stage[main]/Ssh::Client::Config/File[/etc/ssh/ssh_config]/content:
--- /etc/ssh/ssh_config 2013-11-22 22:40:03.000000000 +0000
+++ /tmp/puppet-file20140130-8665-1fn3nrr-0     2014-01-30 23:26:14.508641140 +0000
@@ -1,4 +1,3 @@
-#      $OpenBSD: ssh_config,v 1.25 2009/02/17 01:28:32 djm Exp $

 # This is the ssh client system-wide configuration file.  See
 # ssh_config(5) for more information.  This file provides defaults for
@@ -17,9 +16,10 @@
 # list of available options, their meanings and defaults, please see the
 # ssh_config(5) man page.

-# Host *
+Host *
 #   ForwardAgent no
 #   ForwardX11 no
...
Info: /Stage[main]/Ssh::Client::Config/File[/etc/ssh/ssh_config]: Filebucketed /etc/ssh/ssh_config to puppet with sum 69908a2d0c5ca21a2b1685defbf4cf8b
Notice: /Stage[main]/Ssh::Client::Config/File[/etc/ssh/ssh_config]/content: content changed '{md5}69908a2d0c5ca21a2b1685defbf4cf8b' to '{md5}34815c31785be0c717f766e8d2c8d4d7'
Notice: /Stage[main]/Ssh::Client::Config/File[/etc/ssh/ssh_config]/mode: mode changed '0644' to '0444'
Notice: Finished catalog run in 8.48 seconds

Now you can ssh to localhost. Try and login as root and you will be denied. Ssh as dave and you can get in fine. You’ve now plugged the security hole.

Declarations only do what you declare

This is our lab, however, and we want to be able to log in as root because it’s easy and, well, you’re not named dave. You could comment out the PermitRootLogin statement, but that won’t revert anything:

[root@puppet files]# tail -2 /etc/puppet/manifests/site.pp
include ::ssh
#::ssh::server::configline { 'PermitRootLogin': value => 'no' }
[root@puppet files]# puppet agent --test --noop
Info: Retrieving plugin
...
Info: Applying configuration version '1391124640'
Notice: Finished catalog run in 0.37 seconds

You did not declare a value for PermitRootLogin, so puppet does not manage that value. Whatever value it has, puppet ignores it entirely. With that in mind, you have three ways to fix it. If you don’t care what the value is set to in your environment, simply leave it commented out. This may lead to inconsistent settings across servers. You could leave this server the way it is and on your next server set the value to yes. Some of your servers will be insecure and some will be secured, but you won’t know which unless you try. Perhaps that’s okay.

A second way to fix it is to uninstall the sshd package. On the next run, puppet will reinstall sshd with the default configuration file. In the long run, this isn’t much different than the first fix – tomorrow, 6 months, or a year down the road, sshd.conf could be modified and puppet will never notice. You could be very surprised one day when you try and log in as root and are prohibited. Different fix, same result.

In our lab, we decidedly want to be insecure, and we want all nodes to be that way. We can implement this by going back to our manifest, uncommenting the configline statement, and setting the value to yes. This will ensure that all nodes that receive this manifest WILL have a PermitRootLogin value of yes and ensures consistency across the lab.

[root@puppet files]# vi /etc/puppet/manifests/site.pp
[root@puppet files]# tail -2 /etc/puppet/manifests/site.pp
include ::ssh
::ssh::server::configline { 'PermitRootLogin': value => 'yes' }
[root@puppet files]# puppet agent --test --noop
Info: Retrieving plugin
...
Info: Caching catalog for puppet.nelson.va
Info: Applying configuration version '1391125206'
Notice: Augeas[sshd_config_PermitRootLogin](provider=augeas):
--- /etc/ssh/sshd_config        2014-01-30 23:24:42.093815233 +0000
+++ /etc/ssh/sshd_config.augnew 2014-01-30 23:40:07.376308793 +0000
@@ -136,4 +136,4 @@
 #      X11Forwarding no
 #      AllowTcpForwarding no
 #      ForceCommand cvs server
-PermitRootLogin no
+PermitRootLogin yes
...

Voila! Perform a regular run and now you can ssh to localhost as root.

This is an important lesson. If you add something to a node via puppet and then remove the resource management from the manifest, that resource is still present on the node. If you install an smtp relay on the wrong host, don’t just remove it from the manifest, remove it from the host! Otherwise, you have a potential spam relay on your network. Same applies to anything else you manage. This is why our template was a ‘minimal’ install – puppet isn’t saying “only have these things installed and configured like this,” it’s saying, “I only install a manage these resources.” This will also help you flesh out your dependencies. If you already have every package installed, how will you know which ones a specific resource requires and which ones can be removed?

Conclusion

Last week, you built a master. This week, you started to create a manifest that contains settings you want on all agents. Here’s the final site.pp:

class { '::ntp':
  servers => [ '0.pool.ntp.org', '2.centos.pool.ntp.org', '1.rhel.pool.ntp.org'],
}

user { 'dave':
  ensure     => present,
  uid        => '507',
  gid        => '507',
  shell      => '/bin/bash',
  home       => '/home/dave',
  managehome => true,
}

group { 'dave':
  ensure => 'present',
  gid    => '507',
}

include ::ssh
::ssh::server::configline { 'PermitRootLogin': value => 'yes' }

As you can imagine, this is a very basic manifest that needs some more work. In the next article, we’ll look at manifest organization and how to create our own module.