Welcome back! I hope everyone had a good summer and recharged their batteries. Bonus points if you found time to play with puppet, too! Now that we’ve had a healthy break, let’s get back to it!
When we left the series in July, we had a Puppet master, a few nodes, were implementing the roles and profiles pattern, and used r10k to manage it all. However, we didn’t address scalability. Today, we’ll take a look at addressing this by using Apache and Passenger.
Scaling Up
There are two ways to scale – out and up. If we were to scale out, we’d be concerned with running multiple masters and synchronizing all data between them. That’s something we might look at eventually, but first we want to scale up, which is the process of providing more resources to our master. Since we are vSphere admins, we can easily increase the resources provided to the VM. For instance, our VM has 1 vCPU and 2GB of RAM. It would be easy, and helpful, to increase that, perhaps to 2×4 or 4×8 vCPUxRAM.
Unfortunately, system resources are not the only limitation in our system. Out of the box, Puppet uses WEBrick and scales to about 10 nodes. More than one nodes trying to talk at the same time will generate conflicts and cause some or all nodes to fail to receive a catalog. No matter the resources available, these limitations persist. The answer is to use a dedicated web server with a Rack-based application stack. While any server will work, if you don’t have a preference, then PuppetLabs suggests you use Apache with the Passenger mod. There is a lot of information on Puppet’s site about the limitations and the remedy.
Take a gander at those instructions. You’ve got to install Apache, install Passenger, install the Rack application, enable the Apache vhost, and enable Apache. Or… you can use a module to manage the setup. You could, of course, create your own module, but the Forge has a number of existing modules already. I would recommend using Stephen R. Johnson’s puppet module, found at https://forge.puppetlabs.com/stephenrjohnson/puppet. Like all modules, we’ll want to install it on the master in the main branch first so it can be synced properly. Here’s the diff you need to commit:
[rnelson0@puppet puppet-tutorial]$ git diff ... diff --git a/Puppetfile b/Puppetfile index da1fc4a..4296927 100644 --- a/Puppetfile +++ b/Puppetfile @@ -21,6 +21,7 @@ mod "puppetlabs/java" mod "puppetlabs/java_ks" mod "puppetlabs/rabbitmq" mod "richardc/datacat" +mod "stephenrjohnson/puppet" # For our r10k installer mod "zack/r10k"
Commit, push this upstream, and use r10k to deploy. Run puppet agent –test on the master. You won’t see any changes to the catalog, but the new puppet module will be synchronized. The new r10k deployment may also install some updated versions of existing modules, and by checking in the logs will show up now, rather than clouding our upcoming edits.
Creating a Passenger branch
Now that we have the module installed, we’ll need to create a passenger branch. We’ll do this in the Puppetfile, profile, and hiera repos. If your hiera.yaml file doesn’t point to a dynamic directory, you can skip that step. Otherwise, create the branch for hiera and push it upstream without any other changes.
[rnelson0@puppet hiera-tutorial]$ git checkout -b passenger Switched to a new branch 'passenger' [rnelson0@puppet hiera-tutorial]$ git push origin passenger Total 0 (delta 0), reused 0 (delta 0) To git@github.com:rnelson0/hiera-tutorial.git * [new branch] passenger -> passenger
Use the same checkout commands in the Puppetfile and profile repos. Here’s the Puppetfile diff:
[rnelson0@puppet puppet-tutorial]$ git diff ... diff --git a/Puppetfile b/Puppetfile index da1fc4a..4296927 100644 --- a/Puppetfile +++ b/Puppetfile @@ -44,7 +45,8 @@ mod "role", :git => "git://github.com/rnelson0/rnelson0-role" mod "profile", - :git => "git://github.com/rnelson0/rnelson0-profile" + :git => "git://github.com/rnelson0/rnelson0-profile", + :ref => 'passenger' mod "site_mcollective", :git => "git://github.com/rnelson0/site_mcollective"
All we’ve done is add the new ref and the comma on the preceding line. Commit the change and push it to origin. Next up is the profile repo. Don’t forget to create the new branch.
[rnelson0@puppet profile]$ git diff ...
diff --git a/manifests/puppet_master.pp b/manifests/puppet_master.pp
index b126b8d..73a1aa9 100644
--- a/manifests/puppet_master.pp
+++ b/manifests/puppet_master.pp
@@ -23,8 +23,9 @@
# Copyright 2014 Rob Nelson
#
class profile::puppet_master {
- package {'puppet-server':
- ensure => present,
+ include epel
+ class { '::puppet::master':
+ storeconfigs => true,
+ environments => 'directory',
}
firewall { '100 allow agent checkins':
Now that we’re moving to the new puppet module, we don’t need to track the puppet-server package directly. The class ::puppet::master replaces this, and it adds the environments value directory to use the modern dynamic directory setup. Last, we have the statement include epel. We’ll need the RedHat EPEL repository, which I’ll show later, and thankfully we already have stahnma/epel as a dependency for other modules. If you don’t have it in your Puppetfile, now’s a good time to add it. Again, commit and push the changes upstream.
As usual, deploy your environments with r10k:
[rnelson0@puppet profile]$ sudo r10k deploy environment -p Faraday: you may want to install system_timer for reliable timeouts
Do not run the puppet agent yet! If you read the puppet module page, you may have problems applying changes to the puppet master while it’s running. We can avoid this potential issue by applying the passenger configuration up front. Because the mod_passenger package is not in the standard CentOS repositories but in the EPEL repository, let’s make sure the EPEL repository is available first.
[rnelson0@puppet profile]$ sudo puppet apply -e "include epel"
[rnelson0@puppet profile]$ sudo puppet apply -e "class{'puppet::repo::puppetlabs': } \
Class['puppet::repo::puppetlabs'] -> Package <| |> \
class { 'puppetdb': } \
class { 'puppet::master': storeconfigs => true, environments => 'directory' }"
You’ll see a LOT of dialog scroll past. I won’t quote most of it because of the size. You should not see any warnings or errors. If you do see any errors, try and address them. You can run into some sequencing issues, even with a desired-state engine, so the first troubleshooting step should be to run it a second time. After the apply is complete without errors, run the puppet agent and make sure it works with the passenger environment:
[rnelson0@puppet hiera-tutorial]$ sudo puppet agent --test --environment=passenger ... Notice: Finished catalog run in 8.67 seconds
After it completes, run it once more. The modulepath and manifest errors should go away. This is related to the environments => ‘directory’ statement in the profile.
Once the puppet run is finished, apache should be configured and running. We can check that quickly with ps:
[rnelson0@puppet profile]$ ps -ef | grep http root 20484 1 0 00:43 ? 00:00:00 /usr/sbin/httpd apache 20501 20484 0 00:43 ? 00:00:00 /usr/sbin/httpd apache 20502 20484 0 00:43 ? 00:00:00 /usr/sbin/httpd apache 20503 20484 0 00:43 ? 00:00:00 /usr/sbin/httpd apache 20504 20484 0 00:43 ? 00:00:00 /usr/sbin/httpd apache 20505 20484 0 00:43 ? 00:00:00 /usr/sbin/httpd apache 20506 20484 0 00:43 ? 00:00:00 /usr/sbin/httpd apache 20507 20484 0 00:43 ? 00:00:00 /usr/sbin/httpd apache 20508 20484 0 00:43 ? 00:00:00 /usr/sbin/httpd
You can also check the logs, which have the pattern /var/log/httpd/puppet-${certname}*log. Run tail in one window and another checkin in another:
[rnelson0@puppet profile]$ sudo tail -f /var/log/httpd/puppet-puppet.nelson.va_access_ssl.log ... <start of a new agent run> 10.0.0.35 - - [01/Jul/2014:02:48:13 +0000] "GET /production/node/puppet.nelson.va?fail_on_404=true&transaction_uuid=7e2290d3-64a6-4798-bd21-0259c59e6c10 HTTP/1.1" 200 4132 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:13 +0000] "GET /production/file_metadatas/pluginfacts?links=manage&recurse=true&checksum_type=md5&ignore=.svn&ignore=CVS&ignore=.git HTTP/1.1" 200 307 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:14 +0000] "GET /production/file_metadatas/plugins?links=manage&recurse=true&checksum_type=md5&ignore=.svn&ignore=CVS&ignore=.git HTTP/1.1" 200 85605 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:16 +0000] "POST /production/catalog/puppet.nelson.va HTTP/1.1" 200 461481 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:34 +0000] "GET /production/file_metadata/modules/epel/RPM-GPG-KEY-EPEL-6?links=manage&source_permissions=use HTTP/1.1" 200 329 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:35 +0000] "GET /production/file_metadata/modules/postgresql/validate_postgresql_connection.sh?links=manage&source_permissions=use HTTP/1.1" 200 350 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:35 +0000] "GET /production/file_metadata/modules/site_mcollective/private_keys/root.pem?links=manage&source_permissions=use HTTP/1.1" 200 344 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:35 +0000] "GET /production/file_metadata/modules/site_mcollective/client_certs/root.pem?links=manage&source_permissions=use HTTP/1.1" 200 344 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:35 +0000] "GET /production/file_metadata/modules/site_mcollective/certs/ca.pem?links=manage&source_permissions=use HTTP/1.1" 200 335 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:36 +0000] "GET /production/file_metadata/modules/concat/concatfragments.sh?links=manage&source_permissions=use HTTP/1.1" 200 331 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:36 +0000] "GET /production/file_metadata/modules/site_mcollective/certs/puppet.nelson.va.pem?links=manage&source_permissions=use HTTP/1.1" 200 349 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:36 +0000] "GET /production/file_metadatas/modules/mcollective/site_libdir?links=manage&recurse=true&checksum_type=md5 HTTP/1.1" 200 674 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:37 +0000] "GET /production/file_metadatas/modules/mcollective/plugins/actionpolicy?links=manage&recurse=true&checksum_type=md5 HTTP/1.1" 200 1800 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:37 +0000] "GET /production/file_metadata/modules/site_mcollective/certs/ca.pem?links=manage&source_permissions=use HTTP/1.1" 200 335 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:38 +0000] "GET /production/file_metadata/modules/site_mcollective/certs/puppet.nelson.va.pem?links=manage&source_permissions=use HTTP/1.1" 200 349 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:39 +0000] "GET /production/file_metadata/modules/site_mcollective/private_keys/puppet.nelson.va.pem?links=manage&source_permissions=use HTTP/1.1" 200 356 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:41 +0000] "GET /production/file_metadata/modules/site_mcollective/certs/puppet.nelson.va.pem?links=manage&source_permissions=use HTTP/1.1" 200 349 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:41 +0000] "GET /production/file_metadata/modules/site_mcollective/certs/ca.pem?links=manage&source_permissions=use HTTP/1.1" 200 335 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:41 +0000] "GET /production/file_metadata/modules/site_mcollective/private_keys/puppet.nelson.va.pem?links=manage&source_permissions=use HTTP/1.1" 200 356 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:41 +0000] "GET /production/file_metadatas/modules/site_mcollective/client_certs?links=manage&recurse=true&checksum_type=md5 HTTP/1.1" 200 685 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:42 +0000] "GET /production/file_metadata/modules/puppetdb/routes.yaml?links=manage&source_permissions=use HTTP/1.1" 200 326 "-" "-" 10.0.0.35 - - [01/Jul/2014:02:48:43 +0000] "PUT /production/report/puppet.nelson.va HTTP/1.1" 200 9 "-" "-"
One other thing to note – puppetmaster runs WEBrick, so the service is disabled and stopped. Passenger runs in httpd, so that service is enabled and started. If you want to restart Puppet now, you must restart httpd.
Cleanup
Assuming everything worked, the next thing to do is to clean up. In the profile repo, we need to merge the changes:
[rnelson0@puppet profile]$ git checkout master Switched to branch 'master' [rnelson0@puppet profile]$ git merge passenger master Fast-forwarding to: passenger Already up-to-date with master Merge made by octopus. manifests/puppet_master.pp | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) [rnelson0@puppet profile]$ git push origin master Counting objects: 1, done. Writing objects: 100% (1/1), 204 bytes, done. Total 1 (delta 0), reused 0 (delta 0) To git@github.com:rnelson0/rnelson0-profile.git 33d5e8c..2d587cd master -> master [rnelson0@puppet profile]$ git branch -D passenger Deleted branch passenger (was fcc010a). [rnelson0@puppet profile]$ git push origin :passenger To git@github.com:rnelson0/rnelson0-profile.git - [deleted] passenger
In the hiera and Puppetfile repos, there were no persistent changes, so just delete the branches locally and at origin:
[rnelson0@puppet hiera-tutorial]$ git checkout production Switched to branch 'production' [rnelson0@puppet hiera-tutorial]$ git branch -D passenger Deleted branch passenger (was a57a742). [rnelson0@puppet hiera-tutorial]$ git push origin :passenger cTo git@github.com:rnelson0/hiera-tutorial.git - [deleted] passenger [rnelson0@puppet puppet-tutorial]$ git checkout production Switched to branch 'production' [rnelson0@puppet puppet-tutorial]$ git diff passenger diff --git a/Puppetfile b/Puppetfile index 4296927..c1dec39 100644 --- a/Puppetfile +++ b/Puppetfile @@ -45,8 +45,7 @@ mod "role", :git => "git://github.com/rnelson0/rnelson0-role" mod "profile", - :git => "git://github.com/rnelson0/rnelson0-profile", - :ref => 'passenger' + :git => "git://github.com/rnelson0/rnelson0-profile" mod "site_mcollective", :git => "git://github.com/rnelson0/site_mcollective" [rnelson0@puppet puppet-tutorial]$ git branch -D passenger Deleted branch passenger (was ddfbee3). [rnelson0@puppet puppet-tutorial]$ git push origin :passenger To git@github.com:rnelson0/puppet-tutorial - [deleted] passenger
With everything cleaned up, redeploy with r10k. You’ll see the passenger environment, and the hiera copy, go away.
[rnelson0@puppet puppet-tutorial]$ sudo r10k deploy environment -p Faraday: you may want to install system_timer for reliable timeouts
One last time, run the puppet agent without specifying an environment and you shouldn’t see any effective changes. If you have multiple nodes out there, you may already benefit from scaling up. If you only have a few nodes, you can run the agent on multiple nodes at the same time and watch them connect instead of fail and die!
There’s more that can be done with Passenger to tune your install. If you only have a dozen hosts, any optimizing might be totally wasted time. If you are supporting thousands of nodes, you’re going to want to spend some time tuning things now. If you’re in the middle, keep an eye on utilization and start tuning when resources are still available, but low. The Rack application stack isn’t specific to Puppet, so you can find tuning information throughout the web. I recommend ramindk’s tuning suggestions. We’re mostly vSphere admins, so if you’re not familiar with this, you may want to ask your developer and sysadmin friends for some assistance.
rnelson0 