In our ongoing Puppet series, we just completed installing PuppetDB and Hiera. There’s one other installable that’s a bit more complicated than those two.

MCollective

Since writing this article, the module puppetlabs/mcollective has been updated. In particular, the “middleware => true” parameter is not valid in the base class. I have not had time to revisit this and update the solution. If you have, please drop me a line in the comments!

The last component we’re going to install today is MCollective. While developed by Puppet Labs, MCollective isn’t directly related to Puppet, as PuppetDB and Hiera are. It’s not a Configuration Management tool, it’s an Orchestration API. It does integrate quite well with Puppet and Facter, among other sources. Some things you can do with MCollective might be to query how many systems have 32GB of RAM, how many systems are running a version of OpenSSL vulnerable to Heartbleed, or to restart Apache on all servers in the Development environment. This installation is trickier than either PuppetDB or Hiera.

Note: As usual, keep in mind that we’re installing in a lab environment. When you move Puppet to production, everything will need to handle a larger scale. MCollective specifically is a good candidate for separating the service out among various servers with at least a middleware, server, and client node. Today we will settle for installing it all on the master.

As usual, we will install MCollective through puppet, specifically the puppetlabs/mcollective module. If you’re using r10k, beware, there’s a lot of dependencies, many of which have their own dependencies, to add here. Use puppet module install puppetlabs/mcollective or deploy with r10k. As a nifty trick, if you are using r10k, you can install with puppet module add it to Puppetfile, then re-deploy. The module install will only install the module and dependencies you don’t have already.

[root@puppet puppet-tutorial]# puppet module install puppetlabs/mcollective
Notice: Preparing to install into /etc/puppet/environments/production/modules ...
Notice: Downloading from https://forge.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
/etc/puppet/environments/production/modules
└─┬ puppetlabs-mcollective (v1.1.3)
  ├─┬ garethr-erlang (v0.3.0)
  │ ├── puppetlabs-apt (v1.4.2)
  │ └── stahnma-epel (v0.0.6)
  ├─┬ puppetlabs-activemq (v0.2.0)
  │ └── puppetlabs-java (v1.1.0)
  ├── puppetlabs-java_ks (v1.2.3)
  ├── puppetlabs-rabbitmq (v3.1.0)
  └── richardc-datacat (v0.4.3)
[root@puppet puppet-tutorial]# vi Puppetfile
[root@puppet puppet-tutorial]# git diff
diff --git a/Puppetfile b/Puppetfile
index 11d979c..1775720 100644
--- a/Puppetfile
+++ b/Puppetfile
@@ -12,6 +12,15 @@ mod "yguenane/ygrpms", "0.1.0"
 mod "saz/motd"
 mod "puppetlabs/puppetdb"
 mod "puppetlabs/postgresql"
+mod "puppetlabs/mcollective"
+mod "garethr/erlang"
+mod "puppetlabs/apt"
+mod "stahnma/epel"
+mod "puppetlabs/activemq"
+mod "puppetlabs/java"
+mod "puppetlabs/java_ks"
+mod "puppetlabs/rabbitmq"
+mod "richardc/datacat"

 # For our r10k installer
 mod "zack/r10k", "1.0.2"
[root@puppet puppet-tutorial]# git commit -a -m 'Add mcollective module to puppetmaster'
### Checking puppet syntax, for science! ###

### Checking if puppet manifests are valid ###

### Checking if ruby template syntax is valid ###

Everything looks good.
[installables 7a49f85] Add mcollective module to puppetmaster
 1 files changed, 9 insertions(+), 0 deletions(-)
[root@puppet puppet-tutorial]# git push origin installables
Counting objects: 5, done.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 509 bytes, done.
Total 3 (delta 1), reused 0 (delta 0)
To https://rnelson0@github.com/rnelson0/puppet-tutorial
   f635c44..7a49f85  installables -> installables
[root@puppet puppet-tutorial]# r10k deploy environment -p
Faraday: you may want to install system_timer for reliable timeouts

Next, we need to add a few classes to the master. We need a middleware node first. We’re going to use ActiveMQ, which is the default middleware, though you can use RabbitMQ if you’d like.

  class { '::mcollective': 
    client             => true,
    middleware         => true,
    middleware_hosts   => [ 'puppet.nelson.va' ],
    middleware_ssl     => true,
    securityprovider   => 'ssl',
    ssl_client_certs   => 'puppet:///modules/site_mcollective/client_certs',
    ssl_ca_cert        => 'puppet:///modules/site_mcollective/certs/ca.pem',
    ssl_server_public  => 'puppet:///modules/site_mcollective/certs/puppet.nelson.va.pem',
    ssl_server_private => 'puppet:///modules/site_mcollective/private_keys/puppet.nelson.va.pem',
  }

You’ll notice there are some certs to add to a module called site_mcollective. This module does not exist yet, you’ll need to create it and edit it. After creating the repo, use puppet module generate –modulepath path site_mcollective to create the module in our source tree, rename it, and initialize it as a git repo. Push the changes when you’re done. Alternatively, you can clone mine and start from there.

[root@puppet puppet-tutorial]# cd ..
[root@puppet git]# puppet module generate --modulepath `pwd` rnelson0-site_mcollective
Notice: Generating module at /root/git/rnelson0-site_mcollective
rnelson0-site_mcollective
rnelson0-site_mcollective/manifests
rnelson0-site_mcollective/manifests/init.pp
rnelson0-site_mcollective/Modulefile
rnelson0-site_mcollective/README
rnelson0-site_mcollective/spec
rnelson0-site_mcollective/spec/spec_helper.rb
rnelson0-site_mcollective/tests
rnelson0-site_mcollective/tests/init.pp
[root@puppet git]# mv rnelson0-site_mcollective/ site_mcollective/
[root@puppet git]# cd site_mcollective/
[root@puppet site_mcollective]# touch README.md
[root@puppet site_mcollective]# git init
Initialized empty Git repository in /root/git/site_mcollective/.git/
[root@puppet site_mcollective]# git add .
[root@puppet site_mcollective]# git commit -m "first commit"
[master (root-commit) b2b4026] first commit
 5 files changed, 97 insertions(+), 0 deletions(-)
 create mode 100644 Modulefile
 create mode 100644 README
 create mode 100644 README.md
 create mode 100644 manifests/init.pp
 create mode 100644 spec/spec_helper.rb
 create mode 100644 tests/init.pp
[root@puppet site_mcollective]# git remote add origin git@github.com:rnelson0/site_mcollective.git
[root@puppet site_mcollective]# git push -u origin master
Counting objects: 11, done.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (11/11), 1.90 KiB, done.
Total 11 (delta 0), reused 0 (delta 0)
To git@github.com:rnelson0/site_mcollective.git
 * [new branch]      master -> master
Branch master set up to track remote branch master from origin.

With the module created, we need to store some certificates in it.

Note: You should NEVER store certificates, public keys, or other sensitive information in a public repository. Anyone could grab that information and use it to impersonate your devices, attack them, or run up your credit card bill. I am ignoring my own advice because I’ve mitigated the vulnerability by having my puppet system in a protected lab, and it’s very difficult to write a tutorial and show what the end result is otherwise. This changes nothing, it’s still a really, really, REALLY bad practice. Use a private repo or an internal server with appropriate security controls!

Let me repeat: NEVER store certificates, public keys, or other sensitive information in a public repository!
Never do this. Never. Ever. NEVER! EVAR!

Using the puppet cert generate <user> command, generate a cert for the root user. In production, you’ll want to use your non-priv user.

[root@puppet site_mcollective]# puppet cert generate root
Notice: root has a waiting certificate request
Notice: Signed certificate request for root
Notice: Removing file Puppet::SSL::CertificateRequest root at '/var/lib/puppet/ssl/ca/requests/root.pem'
Notice: Removing file Puppet::SSL::CertificateRequest root at '/var/lib/puppet/ssl/certificate_requests/root.pem'

Combined with the keys Puppet has already generated, this gives us user and agent public keys ($ssldir/certs) and private keys ($ssldir/private_keys) that we need to copy into the files directory of the module.

[root@puppet site_mcollective]# mkdir -p files/certs
[root@puppet site_mcollective]# mkdir -p files/client_certs
[root@puppet site_mcollective]# mkdir -p files/private_keys
[root@puppet site_mcollective]# cp /var/lib/puppet/ssl/certs/puppet.nelson.va.pem files/certs
[root@puppet site_mcollective]# cp /var/lib/puppet/ssl/certs/ca.pem files/certs
[root@puppet site_mcollective]# cp /var/lib/puppet/ssl/certs/root.pem files/client_certs
[root@puppet site_mcollective]# cp /var/lib/puppet/ssl/private_keys/root.pem files/private_keys
[root@puppet site_mcollective]# cp /var/lib/puppet/ssl/private_keys/puppet.nelson.va.pem files/private_keys

After you commit/push the changes to site_mcollective, add the module to Puppetfile in the puppet repo.

[root@puppet site_mcollective]# cd ..
[root@puppet git]# cd puppet-tutorial/
[root@puppet puppet-tutorial]# vi Puppetfile
[root@puppet puppet-tutorial]# git diff
diff --git a/Puppetfile b/Puppetfile
index 1775720..461bac0 100644
--- a/Puppetfile
+++ b/Puppetfile
@@ -36,3 +36,6 @@ mod "puppetlabs/vcsrepo", "0.2.0"
 # Modules from Github
 mod "base",
   :git => "git://github.com/rnelson0/rnelson0-base"
+
+mod "site_mcollective",
+  :git => "git://github.com/rnelson0/site_mcollective"

Lastly you’ll need to add the root user’s cert to the root user’s directory. Let’s add that in in the node definition.

[root@puppet puppet-tutorial]# git diff
diff --git a/manifests/site.pp b/manifests/site.pp
index 846ae00..4a02d21 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -21,4 +21,12 @@ node 'puppet.nelson.va' {
     ssl_server_public  => 'puppet:///modules/site_mcollective/certs/puppet.nelson.va.pem',
     ssl_server_private => 'puppet:///modules/site_mcollective/private_keys/puppet.nelson.va/pem',
   }
+
+  user { 'root':
+    ensure => present,
+  } ->
+  mcollective::user { 'root':
+    homedir     => '/root',
+    certificate => 'puppet:///modules/site_mcollective/client_certs/root.pem',
+    private_key => 'puppet:///modules/site_mcollective/private_keys/root.pem',
+  }
 }

Update the files, commit/push your changes, and deploy. Run puppet and you MCollective should be available on the master. Assuming no errors, it’s time to install the puppet agent plug-in. As I mentioned earlier, PuppetLabs develops MCollective, but it’s not just a puppet component. There are a number of plug-ins available, which can be installed with the mcollective::plugin definition that the mcollective module provides. Add this to the master’s node definition, push/commit, and run puppet again.

[root@puppet puppet-tutorial]# git diff
diff --git a/manifests/site.pp b/manifests/site.pp
index 869dbea..f077727 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -30,4 +31,8 @@ node 'puppet.nelson.va' {
     certificate => 'puppet:///modules/site_mcollective/client_certs/root.pem',
     private_key => 'puppet:///modules/site_mcollective/private_keys/root.pem',
   }
+
+  mcollective::plugin { 'puppet':
+    package => true,
+  }
 }

Finally, let’s test everything. We’ll use mco to ping, get an inventory, and view the puppet plug-in help, just to make sure we have base functionality. (We’ll worry about the connection_headers warning later)

[root@puppet puppet-tutorial]# mco ping
warn 2014/04/23 14:15:27: activemq.rb:274:in `connection_headers' Connecting without STOMP 1.1 heartbeats, if you are using ActiveMQ 5.8 or newer consider setting plugin.activemq.heartbeat_interval
puppet                                   time=175.73 ms


---- ping statistics ----
1 replies max: 175.73 min: 175.73 avg: 175.73
[root@puppet puppet-tutorial]# mco inventory puppet
warn 2014/04/23 14:15:35: activemq.rb:274:in `connection_headers' Connecting without STOMP 1.1 heartbeats, if you are using ActiveMQ 5.8 or newer consider setting plugin.activemq.heartbeat_interval
Inventory for puppet:

   Server Statistics:
                      Version: 2.4.1
                   Start Time: Wed Apr 23 14:11:59 +0000 2014
                  Config File: /etc/mcollective/server.cfg
                  Collectives: mcollective
              Main Collective: mcollective
                   Process ID: 6200
               Total Messages: 9
...
[root@puppet puppet-tutorial]# mco help puppet

Schedule runs, enable, disable and interrogate the Puppet Agent

Usage: mco puppet [OPTIONS] [FILTERS] <ACTION> [CONCURRENCY|MESSAGE]
Usage: mco puppet <count|enable|status|summary>
Usage: mco puppet disable [message]
Usage: mco puppet runonce [PUPPET OPTIONS]
Usage: mco puppet resource type name property1=value property2=value
Usage: mco puppet runall [--rerun SECONDS] [PUPPET OPTIONS]

The ACTION can be one of the following:
...

Some other interesting plugins are service, package, and nrpe. You can use this to manipulate packages on your nodes, such as upgrading openssl, restarting a service like apache, or running Nagios plugins remotely to verify status of devices after a network issue. There’s plenty you can do with MCollective and though I will cover it as we go, you may want to read ahead, especially if you have some mass updates to perform in the near-term.

As you scale up, you’re going to want to spit things up into middleware, client, and server nodes. See here, under MCollective Terminology, to understand what nodes get what roles. Here are some example node definitions that use our site_mcollective module and the nelson.va domain; adjust accordingly.

# middleware
node 'puppet.nelson.va' {
  class { '::mcollective':
    middleware         => true,
    middleware_hosts   => [ 'puppet.nelson.va' ],
    middleware_ssl     => true,
    securityprovider   => 'ssl',
    ssl_client_certs   => 'puppet:///modules/site_mcollective/client_certs',
    ssl_ca_cert        => 'puppet:///modules/site_mcollective/certs/ca.pem',
    ssl_server_public  => 'puppet:///modules/site_mcollective/certs/puppet.nelson.va.pem',
    ssl_server_private => 'puppet:///modules/site_mcollective/private_keys/puppet.nelson.va.pem',
  }
}

node 'mco-client.nelson.va'
  class { '::mcollective':
    client             => true,
    middleware         => true,
    middleware_hosts   => [ 'puppet.nelson.va' ],
    middleware_ssl     => true,
    securityprovider   => 'ssl',
    ssl_client_certs   => 'puppet:///modules/site_mcollective/client_certs',
    ssl_ca_cert        => 'puppet:///modules/site_mcollective/certs/ca.pem',
    ssl_server_public  => 'puppet:///modules/site_mcollective/certs/puppet.nelson.va.pem',
    ssl_server_private => 'puppet:///modules/site_mcollective/private_keys/puppet.nelson.va.pem',
  }

  mcollective::user { 'mcollective-user':
    certificate => 'puppet:///modules/site_mcollective/client_certs/mcollective-user.pem',
    private_key => 'puppet:///modules/site_mcollective/private_keys/mcollective-user.pem',
  }

  mcollective:plugin { 'puppet':
    package => true,
  }
}

node 'mc-server.nelson.va' {
  class { '::mcollective':
    middleware_hosts   => [ 'puppet.nelson.va' ],
    middleware_ssl     => true,
    securityprovider   => 'ssl',
    ssl_client_certs   => 'puppet:///modules/site_mcollective/client_certs',
    ssl_ca_cert        => 'puppet:///modules/site_mcollective/certs/ca.pem',
    ssl_server_public  => 'puppet:///modules/site_mcollective/certs/puppet.nelson.va.pem',
    ssl_server_private => 'puppet:///modules/site_mcollective/private_keys/puppet.nelson.va.pem',
  }

  mcollective::actionpolicy { 'nrpe':
    default => 'deny',
  }

  mcollective::actionpolicy::rule { 'vagrant user can use nrpe agent':
    agent    => 'nrpe',
    callerid => 'cert=vagrant',
  }
}

In such a deployment, you would log into the node mco-client.nelson.va and run mco from there. However, until we get into working on scaling up, I’ll assume that you’re using mco on the master.

Other Installables

Now that you have PuppetDB, Hiera, and MCollective installed, you’ve got all the tools you really need for a good Puppet system, plus the workflows we defined. We’ll start to use these tools next week, when we get back to defining manifests. These are also the most intrusive tools to add at a later date as they require some changes to your configuration and workflow. You’ll probably be comfortable with just these tools, but if you’re in the installing mode, there’s a few other tools to look at.

There are a number of Consoles for Puppet. These give some sort of graphical view into your Puppet system. The Foreman is a very popular OSS console that includes an ENC along with the GUI. Puppetboard is an OSS reporting tool that is starting to flesh out and add features from Puppet Dashboard, an older OSS tool that lost “official” PuppetLabs support but is still in somewhat active development. And of course, Puppet Enterprise includes the Puppet Enterprise Console. Any of these consoles will be helpful but aren’t something I plan to cover at this time.

We’ll also be discussing scaling up Puppet, but in the future. If you can’t wait, there’s a lot of information in Pro Puppet 2nd Ed., as well as plenty of internet articles, that deal with this subject.

I can’t think of any other tools that would have mass appeal. If I missed anything, drop me a line and I’ll update this article. Otherwise, tune in next week as we start to define manifests for our first agent.