For the past few weeks, we have been working on packaging our own software and deploying it with puppet. Before that, we touched on refactoring modules to use hiera. In fact, I grandiosely referred to it as Hiera, R10K, and the end of manifests as we know them. I included a very simple example of how to refactor a per-node manifest into the role/profile pattern and use hiera to assign it to the node. Today, we’ll look at more features of hiera and how you would refactor an existing class to use hiera.
In a legacy implementation of puppet, you’ll likely find plenty of existing modules whose classes have static assignment or lots of conditionals to determine the necessary values to be applied. Even in a greenfield implementation of puppet, you may find yourself writing straight Puppet DSL for your classes before refactoring them to use hiera. Figuring out how to refactor efficiently isn’t always obvious.
First, let’s take a look at Gary Larizza’s When to Hiera and The Problem with Separating Data from Puppet Code articles. Gary covers the when and why much better than I could, so please, go read his article and then come back here. Gary also covers the common pre-hiera pattern and a few patterns that can be used when refactoring to hiera. There is another pattern that is documented indirectly by Gary (under Hiera data bindings in Puppet 3.x.x) and in the Hiera Complete Example at docs.puppetlab.com. I’m going to explain this pattern and document this directly, adding another Hiera pattern to Gary’s list.
Class defaults
Lets look at the way class defaults are defined. The doc tells us to look at the init.pp of the ntp module. We see this at the top:
class ntp (
$autoupdate = $ntp::params::autoupdate,
$config = $ntp::params::config,
$config_template = $ntp::params::config_template,
$disable_monitor = $ntp::params::disable_monitor,
$driftfile = $ntp::params::driftfile,
$logfile = $ntp::params::logfile,
$iburst_enable = $ntp::params::iburst_enable,
$keys_enable = $ntp::params::keys_enable,
$keys_file = $ntp::params::keys_file,
$keys_controlkey = $ntp::params::keys_controlkey,
$keys_requestkey = $ntp::params::keys_requestkey,
$keys_trusted = $ntp::params::keys_trusted,
$package_ensure = $ntp::params::package_ensure,
$package_name = $ntp::params::package_name,
$panic = $ntp::params::panic,
$preferred_servers = $ntp::params::preferred_servers,
$restrict = $ntp::params::restrict,
$interfaces = $ntp::params::interfaces,
$servers = $ntp::params::servers,
$service_enable = $ntp::params::service_enable,
$service_ensure = $ntp::params::service_ensure,
$service_manage = $ntp::params::service_manage,
$service_name = $ntp::params::service_name,
$udlc = $ntp::params::udlc
) inherits ntp::params {
The ntp class accepts a number of parameters when it is instantiated. If the parameter is not provided, a default value is called from the ntp::params class, found here. A short excerpt shows us some of these assignments:
class ntp::params {
$autoupdate = false
$config_template = 'ntp/ntp.conf.erb'
$disable_monitor = false
$keys_enable = false
$keys_controlkey = ''
$keys_requestkey = ''
$keys_trusted = []
$logfile = undef
...
When you include the class in a pre-hiera world as include ntp, the value $ntp::autoupdate is set to the value of $ntp::params::autoupdate of false. If you instead instantiate the class as:
class {'ntp':
autoupdate => true,
}
Then $ntp::autoupdate has a value of true.
Hiera lookups
In Gary’s patterns, there’s a con that shows up everywhere – Hiera needs to be configured. If hiera is not configured, any hiera_include() call will result in an error during catalog compilation. There is no graceful fallback to another value (Gary’s closest related example, under Roles and Profiles, does this on purpose, so that failure to define the parameters generates a failure). If your use case allows for sane, non-destructive default values, Hiera allows us to do things slightly differently way. Any parameter to a class that is not provided a value during instantiation, such as when include‘ing the class, will first look in hiera for a value, then fall back to the default in the class definition. In other words, if you do not provide the autoupdate parameter as above, hiera will look for ntp::autoupdate in your defined hierarchy. If a value is found, it is used. If no value is found, then the default $ntp::params::autoupdate value is used. This graceful fallback eliminates the most problematic con, how to behave when hiera is not available.
Let me repeat myself: If your use case allows for sane, non-destructive default values, Hiera allows us to do things slightly differently way. We’ve removed one con and added another: must allow use of sane, non-destructive default values.
If you want to change the autoupdate default from false to true, add the last line to your global (or common, default, etc.) file in your hiera hierarchy:
[rnelson0@puppet ~]$ cat /etc/puppet/data/production/global.yaml --- puppetmaster: 'puppet.nelson.va' ntp::autoupdate: false
You can now include the class, rather than using the class{} format. Hiera will find ntp::autoupdate in hiera and use it for the value of $ntp::autoupdate. Note that the dollar sign does NOT go in your hiera file. You do want to use the same quoting rules that you would use in Puppet DSL. You can test the value with the hiera command line utility:
[rnelson0@puppet ~]$ hiera ntp::autoupdate environment='production' true
Like other values in hiera, the hierarchy is followed to obtain it. You can override this global value in more specific matches, or only define it in more specific matches and remove it from the global level.
Refactoring a class
Great, we have a way to find default values with hiera and fail back gracefully when they’re not present! The example above looks at using a class that is already ready for this usage. If you have a class that isn’t friendly, how do you get there from here? Let’s look at the steps required, then we’ll look at an example:
- Determine what is data and what is code. Check out Gary’s first article if you need some assistance here.
- Branch! Don’t affect your production setup.
- Create one parameter for each data item in the class.
- Set a default value for each parameter. You may need to create a new class for this (params is a common one), but in some cases the defaults can be in-line.
- Modify the class to make use of the data via parameters. Replace hard-coded values with the appropriate variable, or variables if one value was split into multiple data parameters.
- Update your hiera data to make use of the modified class.
- Test in your branch. Iterate steps 3-6 till satisfied.
- Promote up to production.
Here’s a sample class, profile::wiki, that we will refactor to use hiera.
You won’t be able to review this example in my github repos. This is taken from a production setup and has been slightly anonymized. If you find any inconsistencies or typos, please use the comments to let me know!
[rnelson0@puppet profile]$ cat manifests/wiki.pp
class profile::wiki {
## Hiera Lookups
$docroot = hiera('profile::wiki::docroot')
$wikienvironment = hiera('profile::wiki::environment')
# SELinux and NFS settings
$selbooleans = ['httpd_can_network_connect', 'httpd_use_nfs',]
selboolean { $selbooleans:
value => on,
persistent => true,
}
class { '::nfs::idmap':
idmap_domain => 'localdomain',
}
$cname = "wikidev.contoso.com"
file { "${cname}.crt":
ensure => file,
path => "/etc/ssl/certs/${cname}.crt",
source => "puppet:///modules/certs/${cname}.crt",
} ->
file { "${cname}.key":
ensure => file,
path => "/etc/ssl/certs/${cname}.key",
source => "puppet:///modules/certs/${cname}.key",
} ->
apache::vhost {"${cname}":
port => 443,
docroot => $docroot,
ssl => true,
ssl_cert => "/etc/ssl/certs/${cname}.crt",
ssl_key => "/etc/ssl/certs/${cname}.key",
serveradmin => 'distro@contoso.com',
directories => [
{ 'path' => $docroot,
'AllowOverride' => 'All',
'Order' => 'allow,deny',
'Allow' => 'from all',
},
{ 'path' => "${docroot}/misc",
'Options' => 'Indexes FollowSymlinks',
'AllowOverride' => 'All',
},
],
}
include apache::mod::php
include ::epel
# Packages
Yumrepo['epel'] -> Package<| |>
$packages = [
'mediawiki',
'mediawiki-extensions',
'gatekeeper',
"mediawiki-${wikienvironment}-config",
'nfs-utils',
]
package { $packages:
ensure => latest,
}
# NFS mountpoints
mount { "${docroot}/wiki/DeletedFileStore":
ensure => 'mounted',
device => '192.168.1.32:/nfs/share/wiki/DeletedFileStoreDev',
fstype => 'nfs',
options => 'defaults',
target => '/etc/fstab',
}
mount { "${docroot}/wiki/images":
ensure => 'mounted',
device => '192.168.1.32:/nfs/share/wiki/ImagesDev',
fstype => 'nfs',
options => 'defaults',
target => '/etc/fstab',
}
# The mount point directories rely on the wiki package
Package['mediawiki'] -> Mount<| |>
# crontab
cron { 'wikibackups':
ensure => 'present',
command => '/root/bin/backup > /root/backup.log 2>&1',
hour => 1,
minute => 0,
user => 'root',
}
cron { 'sphinx_main':
ensure => 'present',
command => '/usr/bin/indexer --quiet --config /opt/sphinx/sphinx.conf wiki_main --rotate >/dev/null 2>&1',
hour => 3,
minute => 0,
user => 'sphinx',
}
cron { 'sphinx_incrementals':
ensure => 'present',
command => '/usr/bin/indexer --quiet --config /opt/sphinx/sphinx.conf wiki_incremental --rotate >/dev/null 2>&1',
minute => 5,
user => 'sphinx',
}
# Enable connection to Sphinx for search
firewall { '105 searchd':
proto => 'tcp',
source => '127.0.0.1',
dport => 9312,
state => 'NEW',
action => 'accept',
}
}
This is a pretty involved class. We’ve got some SELinux booleans, an NFS domain name, cert files, an apache vhost, a yumrepo (and another inherited from base that isn’t visible), some internally developed packages, NFS mounts, some crontab entries, and a firewall rule. At the top, there are two hiera lookups, so you can see that some data/code decisions were already made. We just need to keep going and identify all the other data elements. Let’s take a look at a few decisions and how I made them (step #1):
- docroot – In reality, it turns out that the packages for mediawiki are predicated on a specific docroot. We’ll set this, as it’s used in a few places, but there’s no need to pull it from hiera. Code.
- wikienvironment – There will be at least production and development, and that’s going to be determined on a per-node basis. Data.
- cname – You can tell I was developing in development (wikidev.contoso.com) but clearly we’ll need to specify this per-node. Data.
- NFS mount points – Again, it shows the development environment (DeletedFileStoreDev and ImagesDev). But we also see there’s an NFS server IP in here. This results in TWO pieces of Data.
- A few other decisions I won’t bore you with.
Here’s the resulting refactored class parameters:
class profile::wiki (
$environment = 'development',
$cname = 'wikidev.contoso.com',
$serveradmin = 'distro@contoso.com',
$idmap_domain = 'localdomain',
$nfs_server = 'localhost',
$DeletedFileStoreMount = '/nfs/share/wiki/deleted',
$ImagesMount = '/nfs/share/wiki/images',
$database = 'wikidev',
) {
# Global settings
$docroot = '/srv/www' # Built into the packages, cannot be changed.
Here are all the data items (step #3) and defaults (step #4). As I mentioned earlier, this can go in a params class. In this case, we don’t have any logic decisions to make, so putting the defaults inline works as well (we could create a profile::wiki::params class, but it would just contain these vars and no logic – is that worth the overhead?). The defaults all point to development or the localhost (it is preferable to use an actual NFS server in your environment). This meets our requirement of sane, non-destructive defaults. You wouldn’t want an errant wiki node to stomp on production, would you?
Now we need to make sure the class supports the changes (step #5). You can see where I made changes, for instance in combining the new $nfs_server and $*Mount variables for the mount resources, in this diff. Don’t forget to switch from single to double quotes if you need to interpolate!
[rnelson0@puppet profile]$ git diff 4cce680fdcb4cee65a4acc12edbcbf2d5eef6387
diff --git a/manifests/wiki.pp b/manifests/wiki.pp
index 2ccfd50..0183fed 100644
--- a/manifests/wiki.pp
+++ b/manifests/wiki.pp
@@ -22,11 +22,18 @@
#
# Copyright 2014
#
-class profile::wiki {
-
- ## Hiera Lookups
- $docroot = hiera('profile::wiki::docroot')
- $wikienvironment = hiera('profile::wiki::environment')
+class profile::wiki (
+ $environment = 'development',
+ $cname = 'wikidev.contoso.com',
+ $serveradmin = 'distro@contoso.com',
+ $idmap_domain = 'localdomain',
+ $nfs_server = 'localhost',
+ $DeletedFileStoreMount = '/nfs/share/wiki/deleted',
+ $ImagesMount = '/nfs/share/wiki/images',
+ $database = 'wikidev',
+) {
+ # Global settings
+ $docroot = '/var/www' # Built into the packages, cannot be changed.
# SELinux and NFS settings
$selbooleans = ['httpd_can_network_connect', 'httpd_use_nfs',]
@@ -35,10 +42,10 @@ class profile::wiki {
persistent => true,
}
class { '::nfs::idmap':
- idmap_domain => 'localdomain',
+ idmap_domain => $idmap_domain,
}
- $cname = "wikidev.contoso.com"
+ # Certificates are based on the cname
file { "${cname}.crt":
ensure => file,
path => "/etc/ssl/certs/${cname}.crt",
@@ -55,7 +62,7 @@ class profile::wiki {
ssl => true,
ssl_cert => "/etc/ssl/certs/${cname}.crt",
ssl_key => "/etc/ssl/certs/${cname}.key",
- serveradmin => 'distro@contoso.com',
+ serveradmin => $serveradmin,
directories => [
{ 'path' => $docroot,
'AllowOverride' => 'All',
@@ -78,7 +85,7 @@ class profile::wiki {
'mediawiki',
'mediawiki-extensions',
'gatekeeper',
- "mediawiki-${wikienvironment}-config",
+ "mediawiki-${environment}-config",
'nfs-utils',
]
package { $packages:
@@ -88,14 +95,14 @@ class profile::wiki {
# NFS mountpoints
mount { "${docroot}/wiki/DeletedFileStore":
ensure => 'mounted',
- device => '192.168.1.32:/nfs/share/wiki/DeletedFileStoreDev',
+ device => "${nfs_server}:${DeletedFileStoreMount}",
fstype => 'nfs',
options => 'defaults',
target => '/etc/fstab',
}
mount { "${docroot}/wiki/images":
ensure => 'mounted',
- device => '192.168.1.32:/nfs/share/wiki/ImagesDev',
+ device => "${nfs_server}:${ImagesMount}",
fstype => 'nfs',
options => 'defaults',
target => '/etc/fstab',
@@ -114,14 +121,14 @@ class profile::wiki {
}
cron { 'sphinx_main':
ensure => 'present',
- command => '/usr/bin/indexer --quiet --config /opt/sphinx/sphinx.conf wiki_main --rotate >/dev/null 2>&1'
+ command => "/usr/bin/indexer --quiet --config /opt/sphinx/sphinx.conf ${database}_main --rotate >/dev/nul
hour => 3,
minute => 0,
user => 'sphinx',
}
cron { 'sphinx_incrementals':
ensure => 'present',
- command => '/usr/bin/indexer --quiet --config /opt/sphinx/sphinx.conf wiki_incremental --rotate >/dev/nul
+ command => "/usr/bin/indexer --quiet --config /opt/sphinx/sphinx.conf ${database}_incremental --rotate >/
minute => 5,
user => 'sphinx',
}
@@ -141,5 +148,4 @@ class profile::wiki {
state => 'NEW',
action => 'accept',
}
-
-}
+}
\ No newline at end of file
Last, we need some hiera data (step #6). Here are the changes to my role and global data, where I had defined the earlier hiera lookup values, and the new node data:
[rn7284@puppet hiera]$ git diff 48c3d42b078597f8c294cefe6a929dd04865b1a0 diff --git a/wikidev01.contoso.com.yaml b/wikidev01.contoso.com.yaml new file mode 100644 index 0000000..6f5d5ab --- /dev/null +++ b/wikidev01.contoso.com.yaml @@ -0,0 +1,9 @@ +--- +profile::wiki::environment : 'development' +profile::wiki::cname : 'wikidev.contoso.com' +profile::wiki::serveradmin : 'distro@contoso.com' +profile::wiki::idmap_domain : 'localdomain' +profile::wiki::nfs_server : '192.168.1.32' +profile::wiki::DeletedFileStoreMount : '/nfs/share/wiki/DeletedFileStoreDev' +profile::wiki::ImagesMount : '/nfs/share/wiki/ImagesDev' +profile::wiki::database : 'wikidev' diff --git a/global.yaml b/global.yaml index 4a3f657..0dc383b 100644 --- a/global.yaml +++ b/global.yaml @@ -1,3 +1,2 @@ --- puppetmaster: 'puppet.mss.local' -profile::wiki::docroot: '/srv/www' diff --git a/puppet_role/wikidev.yaml b/puppet_role/wikidev.yaml index bbbef51..360ac85 100644 --- a/puppet_role/wikidev.yaml +++ b/puppet_role/wikidev.yaml @@ -1,4 +1,3 @@ --- classes: - 'role::wiki' -profile::wiki::environment: 'development'
I can’t show the testing (step #7) here very easily. Using r10k, I redeployed my environments and tested against the new environment (puppet agent -t –noop –environment=wiki_hiera) until the only changes I saw were related to differences in the module versions between environments (long story short, the newer environment gets newer versions of all the dependency modules, which can cause what looks like very large deltas in config, but which have no actual affect) and changes I wanted to effect, such as using the database name in the crontab jobs for sphinxsearch. I then merged the changes (step #8) into production. Now, if I need more wiki servers, it’s a matter of populating some data in hiera and applying the role that includes profile::wiki to the new node(s).
The above process is very subjective and specific to your environment. If you had multiple NFS servers in your organization, you could end up with the images and DeletedFileStore mounts on different NFS servers, so you wouldn’t want an $nfs_server parameter. Maybe the list of packages might vary per node, so you would need another parameter. The good news is that nothing you do here is written in stone. You can always revisit your decisions later, though there will be some cost associated with the effort. Try and get it right the first time, but don’t worry if it requires multiple iterations, especially as you begin your hiera refactoring journey.
Summary
There are many ways to refactor a class to support data in hiera. Today, we’ve discussed another pattern that can do this.
Pros:
- Gracefully downgrades to using defaults without errors.
- Data resides in Hiera.
- Code resides in the class.
- The class can be include’ed without risk of multiple-declaration errors.
- Backwards compatible to Puppet 2.6.
- Does NOT rely on Hiera!
- Modules contain a default values but allow overrides, making them Forge friendly.
- Eases conversion of legacy modules.
Cons:
- Data values can come from Hiera or classes, which may affect troubleshooting.
- Must support default values; will continue building a catalog when “important” parameters are not defined via hiera/class{} instantiation.
This pattern has worked successfully for me, but I’d love to hear what patterns work for other people. Please drop me a line in the comments, or find me on twitter @rnelson0. Thanks!
rnelson0 
Hi –
This article is very helpful. I am a beginner to puppet. Can you plesae help me understand what is this syntax referring to?
@@ -114,14 +121,14 @@ class what are those number what do they tell you?
Veera, that is output from the `git diff` command and shows the line number and characters changed in the files.