Puppet Scale Up with Apache/Passenger

Welcome back! I hope everyone had a good summer and recharged their batteries. Bonus points if you found time to play with puppet, too! Now that we’ve had a healthy break, let’s get back to it!

When we left the series in July, we had a Puppet master, a few nodes, were implementing the roles and profiles pattern, and used r10k to manage it all. However, we didn’t address scalability. Today, we’ll take a look at addressing this by using Apache and Passenger.

Scaling Up

There are two ways to scale – out and up. If we were to scale out, we’d be concerned with running multiple masters and synchronizing all data between them. That’s something we might look at eventually, but first we want to scale up, which is the process of providing more resources to our master. Since we are vSphere admins, we can easily increase the resources provided to the VM. For instance, our VM has 1 vCPU and 2GB of RAM. It would be easy, and helpful, to increase that, perhaps to 2×4 or 4×8 vCPUxRAM.

Unfortunately, system resources are not the only limitation in our system. Out of the box, Puppet uses WEBrick and scales to about 10 nodes. More than one nodes trying to talk at the same time will generate conflicts and cause some or all nodes to fail to receive a catalog. No matter the resources available, these limitations persist. The answer is to use a dedicated web server with a Rack-based application stack. While any server will work, if you don’t have a preference, then PuppetLabs suggests you use Apache with the Passenger mod. There is a lot of information on Puppet’s site about the limitations and the remedy.

Take a gander at those instructions. You’ve got to install Apache, install Passenger, install the Rack application, enable the Apache vhost, and enable Apache. Or… you can use a module to manage the setup. You could, of course, create your own module, but the Forge has a number of existing modules already. I would recommend using Stephen R. Johnson’s puppet module, found at https://forge.puppetlabs.com/stephenrjohnson/puppet. Like all modules, we’ll want to install it on the master in the main branch first so it can be synced properly. Here’s the diff you need to commit:

[rnelson0@puppet puppet-tutorial]$ git diff ...
diff --git a/Puppetfile b/Puppetfile
index da1fc4a..4296927 100644
--- a/Puppetfile
+++ b/Puppetfile
@@ -21,6 +21,7 @@ mod "puppetlabs/java"
 mod "puppetlabs/java_ks"
 mod "puppetlabs/rabbitmq"
 mod "richardc/datacat"
+mod "stephenrjohnson/puppet"

 # For our r10k installer
 mod "zack/r10k"

Commit, push this upstream, and use r10k to deploy. Run puppet agent –test on the master. You won’t see any changes to the catalog, but the new puppet module will be synchronized. The new r10k deployment may also install some updated versions of existing modules, and by checking in the logs will show up now, rather than clouding our upcoming edits.

Creating a Passenger branch

Now that we have the module installed, we’ll need to create a passenger branch. We’ll do this in the Puppetfile, profile, and hiera repos. If your hiera.yaml file doesn’t point to a dynamic directory, you can skip that step. Otherwise, create the branch for hiera and push it upstream without any other changes.

[rnelson0@puppet hiera-tutorial]$ git checkout -b passenger
Switched to a new branch 'passenger'
[rnelson0@puppet hiera-tutorial]$ git push origin passenger
Total 0 (delta 0), reused 0 (delta 0)
To git@github.com:rnelson0/hiera-tutorial.git
 * [new branch]      passenger -> passenger

Use the same checkout commands in the Puppetfile and profile repos. Here’s the Puppetfile diff:

[rnelson0@puppet puppet-tutorial]$ git diff ...
diff --git a/Puppetfile b/Puppetfile
index da1fc4a..4296927 100644
--- a/Puppetfile
+++ b/Puppetfile
@@ -44,7 +45,8 @@ mod "role",
   :git => "git://github.com/rnelson0/rnelson0-role"

 mod "profile",
-  :git => "git://github.com/rnelson0/rnelson0-profile"
+  :git => "git://github.com/rnelson0/rnelson0-profile",
+  :ref => 'passenger'

 mod "site_mcollective",
   :git => "git://github.com/rnelson0/site_mcollective"

All we’ve done is add the new ref and the comma on the preceding line. Commit the change and push it to origin. Next up is the profile repo. Don’t forget to create the new branch.

[rnelson0@puppet profile]$ git diff ...
diff --git a/manifests/puppet_master.pp b/manifests/puppet_master.pp
index b126b8d..73a1aa9 100644
--- a/manifests/puppet_master.pp
+++ b/manifests/puppet_master.pp
@@ -23,8 +23,9 @@
 # Copyright 2014 Rob Nelson
 class profile::puppet_master {
-  package {'puppet-server':
-    ensure => present,
+  include epel
+  class { '::puppet::master':
+    storeconfigs => true,
+    environments => 'directory',

   firewall { '100 allow agent checkins':

Now that we’re moving to the new puppet module, we don’t need to track the puppet-server package directly. The class ::puppet::master replaces this, and it adds the environments value directory to use the modern dynamic directory setup. Last, we have the statement include epel. We’ll need the RedHat EPEL repository, which I’ll show later, and thankfully we already have stahnma/epel as a dependency for other modules. If you don’t have it in your Puppetfile, now’s a good time to add it. Again, commit and push the changes upstream.

As usual, deploy your environments with r10k:

[rnelson0@puppet profile]$ sudo r10k deploy environment -p
Faraday: you may want to install system_timer for reliable timeouts

Do not run the puppet agent yet! If you read the puppet module page, you may have problems applying changes to the puppet master while it’s running. We can avoid this potential issue by applying the passenger configuration up front. Because the mod_passenger package is not in the standard CentOS repositories but in the EPEL repository, let’s make sure the EPEL repository is available first.

[rnelson0@puppet profile]$ sudo puppet apply -e "include epel"
[rnelson0@puppet profile]$ sudo puppet apply -e "class{'puppet::repo::puppetlabs': } \
Class['puppet::repo::puppetlabs'] -> Package <| |> \
class { 'puppetdb': } \
class { 'puppet::master': storeconfigs => true, environments => 'directory' }"

You’ll see a LOT of dialog scroll past. I won’t quote most of it because of the size. You should not see any warnings or errors. If you do see any errors, try and address them. You can run into some sequencing issues, even with a desired-state engine, so the first troubleshooting step should be to run it a second time. After the apply is complete without errors, run the puppet agent and make sure it works with the passenger environment:

[rnelson0@puppet hiera-tutorial]$ sudo puppet agent --test --environment=passenger                                                                                                                   
Notice: Finished catalog run in 8.67 seconds

After it completes, run it once more. The modulepath and manifest errors should go away. This is related to the environments => ‘directory’ statement in the profile.

Once the puppet run is finished, apache should be configured and running. We can check that quickly with ps:

[rnelson0@puppet profile]$ ps -ef | grep http
root     20484     1  0 00:43 ?        00:00:00 /usr/sbin/httpd
apache   20501 20484  0 00:43 ?        00:00:00 /usr/sbin/httpd
apache   20502 20484  0 00:43 ?        00:00:00 /usr/sbin/httpd
apache   20503 20484  0 00:43 ?        00:00:00 /usr/sbin/httpd
apache   20504 20484  0 00:43 ?        00:00:00 /usr/sbin/httpd
apache   20505 20484  0 00:43 ?        00:00:00 /usr/sbin/httpd
apache   20506 20484  0 00:43 ?        00:00:00 /usr/sbin/httpd
apache   20507 20484  0 00:43 ?        00:00:00 /usr/sbin/httpd
apache   20508 20484  0 00:43 ?        00:00:00 /usr/sbin/httpd

You can also check the logs, which have the pattern /var/log/httpd/puppet-${certname}*log. Run tail in one window and another checkin in another:

[rnelson0@puppet profile]$ sudo tail -f /var/log/httpd/puppet-puppet.nelson.va_access_ssl.log
<start of a new agent run> - - [01/Jul/2014:02:48:13 +0000] "GET /production/node/puppet.nelson.va?fail_on_404=true&transaction_uuid=7e2290d3-64a6-4798-bd21-0259c59e6c10 HTTP/1.1" 200 4132 "-" "-" - - [01/Jul/2014:02:48:13 +0000] "GET /production/file_metadatas/pluginfacts?links=manage&recurse=true&checksum_type=md5&ignore=.svn&ignore=CVS&ignore=.git HTTP/1.1" 200 307 "-" "-" - - [01/Jul/2014:02:48:14 +0000] "GET /production/file_metadatas/plugins?links=manage&recurse=true&checksum_type=md5&ignore=.svn&ignore=CVS&ignore=.git HTTP/1.1" 200 85605 "-" "-" - - [01/Jul/2014:02:48:16 +0000] "POST /production/catalog/puppet.nelson.va HTTP/1.1" 200 461481 "-" "-" - - [01/Jul/2014:02:48:34 +0000] "GET /production/file_metadata/modules/epel/RPM-GPG-KEY-EPEL-6?links=manage&source_permissions=use HTTP/1.1" 200 329 "-" "-" - - [01/Jul/2014:02:48:35 +0000] "GET /production/file_metadata/modules/postgresql/validate_postgresql_connection.sh?links=manage&source_permissions=use HTTP/1.1" 200 350 "-" "-" - - [01/Jul/2014:02:48:35 +0000] "GET /production/file_metadata/modules/site_mcollective/private_keys/root.pem?links=manage&source_permissions=use HTTP/1.1" 200 344 "-" "-" - - [01/Jul/2014:02:48:35 +0000] "GET /production/file_metadata/modules/site_mcollective/client_certs/root.pem?links=manage&source_permissions=use HTTP/1.1" 200 344 "-" "-" - - [01/Jul/2014:02:48:35 +0000] "GET /production/file_metadata/modules/site_mcollective/certs/ca.pem?links=manage&source_permissions=use HTTP/1.1" 200 335 "-" "-" - - [01/Jul/2014:02:48:36 +0000] "GET /production/file_metadata/modules/concat/concatfragments.sh?links=manage&source_permissions=use HTTP/1.1" 200 331 "-" "-" - - [01/Jul/2014:02:48:36 +0000] "GET /production/file_metadata/modules/site_mcollective/certs/puppet.nelson.va.pem?links=manage&source_permissions=use HTTP/1.1" 200 349 "-" "-" - - [01/Jul/2014:02:48:36 +0000] "GET /production/file_metadatas/modules/mcollective/site_libdir?links=manage&recurse=true&checksum_type=md5 HTTP/1.1" 200 674 "-" "-" - - [01/Jul/2014:02:48:37 +0000] "GET /production/file_metadatas/modules/mcollective/plugins/actionpolicy?links=manage&recurse=true&checksum_type=md5 HTTP/1.1" 200 1800 "-" "-" - - [01/Jul/2014:02:48:37 +0000] "GET /production/file_metadata/modules/site_mcollective/certs/ca.pem?links=manage&source_permissions=use HTTP/1.1" 200 335 "-" "-" - - [01/Jul/2014:02:48:38 +0000] "GET /production/file_metadata/modules/site_mcollective/certs/puppet.nelson.va.pem?links=manage&source_permissions=use HTTP/1.1" 200 349 "-" "-" - - [01/Jul/2014:02:48:39 +0000] "GET /production/file_metadata/modules/site_mcollective/private_keys/puppet.nelson.va.pem?links=manage&source_permissions=use HTTP/1.1" 200 356 "-" "-" - - [01/Jul/2014:02:48:41 +0000] "GET /production/file_metadata/modules/site_mcollective/certs/puppet.nelson.va.pem?links=manage&source_permissions=use HTTP/1.1" 200 349 "-" "-" - - [01/Jul/2014:02:48:41 +0000] "GET /production/file_metadata/modules/site_mcollective/certs/ca.pem?links=manage&source_permissions=use HTTP/1.1" 200 335 "-" "-" - - [01/Jul/2014:02:48:41 +0000] "GET /production/file_metadata/modules/site_mcollective/private_keys/puppet.nelson.va.pem?links=manage&source_permissions=use HTTP/1.1" 200 356 "-" "-" - - [01/Jul/2014:02:48:41 +0000] "GET /production/file_metadatas/modules/site_mcollective/client_certs?links=manage&recurse=true&checksum_type=md5 HTTP/1.1" 200 685 "-" "-" - - [01/Jul/2014:02:48:42 +0000] "GET /production/file_metadata/modules/puppetdb/routes.yaml?links=manage&source_permissions=use HTTP/1.1" 200 326 "-" "-" - - [01/Jul/2014:02:48:43 +0000] "PUT /production/report/puppet.nelson.va HTTP/1.1" 200 9 "-" "-"

One other thing to note – puppetmaster runs WEBrick, so the service is disabled and stopped. Passenger runs in httpd, so that service is enabled and started. If you want to restart Puppet now, you must restart httpd.


Assuming everything worked, the next thing to do is to clean up. In the profile repo, we need to merge the changes:

[rnelson0@puppet profile]$ git checkout master
Switched to branch 'master'
[rnelson0@puppet profile]$ git merge passenger master
Fast-forwarding to: passenger
Already up-to-date with master
Merge made by octopus.
 manifests/puppet_master.pp |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)
[rnelson0@puppet profile]$ git push origin master
Counting objects: 1, done.
Writing objects: 100% (1/1), 204 bytes, done.
Total 1 (delta 0), reused 0 (delta 0)
To git@github.com:rnelson0/rnelson0-profile.git
   33d5e8c..2d587cd  master -> master
[rnelson0@puppet profile]$ git branch -D passenger
Deleted branch passenger (was fcc010a).
[rnelson0@puppet profile]$ git push origin :passenger
To git@github.com:rnelson0/rnelson0-profile.git
 - [deleted]         passenger

In the hiera and Puppetfile repos, there were no persistent changes, so just delete the branches locally and at origin:

[rnelson0@puppet hiera-tutorial]$ git checkout production
Switched to branch 'production'
[rnelson0@puppet hiera-tutorial]$ git branch -D passenger
Deleted branch passenger (was a57a742).
[rnelson0@puppet hiera-tutorial]$ git push origin :passenger
cTo git@github.com:rnelson0/hiera-tutorial.git
 - [deleted]         passenger
[rnelson0@puppet puppet-tutorial]$ git checkout production
Switched to branch 'production'
[rnelson0@puppet puppet-tutorial]$ git diff passenger
diff --git a/Puppetfile b/Puppetfile
index 4296927..c1dec39 100644
--- a/Puppetfile
+++ b/Puppetfile
@@ -45,8 +45,7 @@ mod "role",
   :git => "git://github.com/rnelson0/rnelson0-role"

 mod "profile",
-  :git => "git://github.com/rnelson0/rnelson0-profile",
-  :ref => 'passenger'
+  :git => "git://github.com/rnelson0/rnelson0-profile"

 mod "site_mcollective",
   :git => "git://github.com/rnelson0/site_mcollective"
[rnelson0@puppet puppet-tutorial]$ git branch -D passenger
Deleted branch passenger (was ddfbee3).
[rnelson0@puppet puppet-tutorial]$ git push origin :passenger
To git@github.com:rnelson0/puppet-tutorial
 - [deleted]         passenger

With everything cleaned up, redeploy with r10k. You’ll see the passenger environment, and the hiera copy, go away.

[rnelson0@puppet puppet-tutorial]$ sudo r10k deploy environment -p
Faraday: you may want to install system_timer for reliable timeouts

One last time, run the puppet agent without specifying an environment and you shouldn’t see any effective changes. If you have multiple nodes out there, you may already benefit from scaling up. If you only have a few nodes, you can run the agent on multiple nodes at the same time and watch them connect instead of fail and die!

There’s more that can be done with Passenger to tune your install. If you only have a dozen hosts, any optimizing might be totally wasted time. If you are supporting thousands of nodes, you’re going to want to spend some time tuning things now. If you’re in the middle, keep an eye on utilization and start tuning when resources are still available, but low. The Rack application stack isn’t specific to Puppet, so you can find tuning information throughout the web. I recommend ramindk’s tuning suggestions. We’re mostly vSphere admins, so if you’re not familiar with this, you may want to ask your developer and sysadmin friends for some assistance.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s