Recently, we looked at converting a module to be hiera friendly. Another task you may have to look forward to is tracking configuration files for different hosts providing the same service. You could have a config for each node, network, environment, etc., all of which need updated if some common element changes. Or, you could use a Puppet Template to have a single template that is populated by node-specific elements from node facts and puppet variables. Each node would receive a personalized copy and any common element changes would be reflected across the board immediately.

As an example, I run some mediawiki servers at work. Each one points to a different database but is otherwise very similar. The search engine is SphinxSearch and it relies on the Sphinx config file /etc/sphinx/sphinx.conf. The config includes the database connection information, which varies from device to device, and a number of other settings standardized across the wikis (minimum search term length, wildcards, and other search settings). Keeping the database connection information accurate across three wikis would normally require three config files. Let’s simplify that with a template.

Puppet templates are written in ERB, a templating language that is part of the Ruby standard library. ERB commands are interpolated into the config where needed and puppet feeds facts and variables to ERB, which determines what values to populate the config with. We have a few good sources of information on the templates: the Ruby docs, a Puppet Labs Using Puppet Templates article, or the Learning Puppet chapter on Templates. I’ll be picking out some highlights, reference them as needed as we work on our template.

Original Config File

Before we do anything else, let’s look at the original config file that belongs at /etc/sphinx/sphinx.conf. This file was based on the sample provided by the SphinxSearch extension. The part we care about is the top block; the rest of the config file will be the same for every wiki that uses SphinxSearch:

source src_wiki_main
{
    # data source
    type            = mysql
    sql_host        = 192.168.1.15
    sql_user        = wikiuser
    sql_pass        = password
    sql_db          = wikidb

    # pre-query, executed before the main fetch query
    sql_query_pre   = SET NAMES utf8

    # main document fetch query - change the table names if you are using a prefix
    sql_query = SELECT page_id, page_title, page_namespace, page_is_redirect, old_id, old_text FROM page, revision, text WHERE rev_id=page_latest AND old_id=rev_text_id

    # attribute columns
    sql_attr_uint   = page_namespace
    sql_attr_uint   = page_is_redirect
    sql_attr_uint   = old_id

    # uncomment next line to collect all category ids for a category filter
    #sql_attr_multi  = uint category from query; SELECT cl_from, page_id AS category FROM categorylinks, page WHERE page_title=cl_to AND page_namespace=14

    # optional - used by command-line search utility to display document information
    sql_query_info  = SELECT page_title, page_namespace FROM page WHERE page_id=$id
}

Lines 17-21 are the variable part of our config, where the sql_* settings for a data source are specified. The first three attributes may be shared among multiple nodes, but the sql_db attribute is unique to each one. You don’t want the wiki serving up the infosecdb to be presenting search results from wikidb. Even though some of those attributes are shared, we’ll make them all variables in the template for the most flexibility. You probably don’t want to revisit this every time.

Template Syntax

An ERB template mixes plain text with ruby code. The ruby code is delimited by paired bracket/percent tags, <% like this %>. A plain tag, as before, runs ruby code but does not display it in the interpolated template. The use of a <%= printing tag %> will print out the results of the ruby code. Within the tags, facts, global, and current scope variables are available by prepending them with an at sign, i.e. <%= @fqdn %> will print the fully qualified domain name of the node. For variables outside of the current scope, the scope.lookupvar method is used, i.e. <%= scope.lookup(‘wiki::sql_db’) %> will print the value of $wiki::sql_db. Note that no dollar sign is used and the leading :: is left off (I have not found a style guide explanation for this, but it’s pretty consistent in documentation).

You can also access hiera. You would do this with scope.function, i.e. <%= scope.function_hiera([“hiera::var”]) %>, but this is not recommended by puppet. It does clutter up the template, making it fairly unreadable and reliant on hiera. Assigning a local variable the value of a hiera result and using that variable in your template is a much better pattern.

We won’t use them today, but there are two other tags. <%# comments start with a pound %>, and if you have multiline code, using a hyphen in your tags will strip out leading or trailing space. This is great for conditionals and loops:

<% if @something -%>
server  <%= @server %>
<% end -%>

This would put ‘server @server’ on its own line without blank lines for the if and end lines.

ERB is itself fairly simple. Let’s take a look at the converted lines 17-21:

    # data source
    type = mysql
    sql_host = <%= @sql_host %>
    sql_user = <%= @sql_user %>
    sql_pass = <%= @sql_pass %>
    sql_db = <%= @sql_db %>

We’re simply going to take the value of the $sql_* variables from the calling class and interpolate them. If we set $sql_host to ‘192.168.1.15’, we’ll have the same value as our original config file had, and so on for the other values. So, where do we do that? In the module. In this case, I have a class profile::wiki in the profile module that will put this file on the node.

This is again taken from a production environment, so the content is not available on github in its entirety. I’ll post the relevant config snippets with some anonymization, you can see more in a previous article.

Module Modification

The most relevant portion is the package list:

  # Packages
  Yumrepo['epel'] -> Package<| |>
  $packages = [
    'mediawiki',
    'mediawiki-extensions',
    'gatekeeper',
    "mediawiki-${wikienvironment}-config",
    'nfs-utils',
  ]
  package { $packages:
    ensure => latest,
  }

In this list of packages is mediawiki-extensions, which deploys the SphinxSearch extension and the prerequisite of sphinx. After that is installed, we will overwrite the config file (if we do it beforehand, the package could overwrite it or fail to install). Let’s add a file resource with a template for the content attribute and ensure it follows the mediawiki-extensions package. We’ll go ahead and manage the searchd service and make sure it is notified.

  file { 'sphinx.conf':
    path    => '/etc/sphinx/sphinx.conf',
    ensure  => file,
    require => Package['mediawiki-extensions'],
    notify  => Service['searchd'],
    content => template('profile/wiki/sphinx.conf.erb'),
  }
  service { 'searchd':
    ensure => running,
  }

The template() function parses profile/wiki/sphinx.conf.erb and looks for the file in the location $environmentpath/profile/templates/wiki/sphinx.conf.erb. Create that file and populate it. Deploy the environment and run the agent. You’ll notice that it works… kinda. When you look at the changes to the file, you end up with this (just showing the adds, not the deletes):

+       # data source
+       type            = mysql
+       sql_host        =
+       sql_user        =
+       sql_pass        =
+       sql_db          =

We don’t have an sql_host variable in our manifest, so we got a null string. Whoops! Let’s go back to the top of our wiki profile and add some variables to the class. As we did last time, we set a local variable with a default in the local class. Here’s the diff:

$ git diff
diff --git a/manifests/wiki.pp b/manifests/wiki.pp
index 58514cc..7c819b5 100644
--- a/manifests/wiki.pp
+++ b/manifests/wiki.pp
@@ -30,8 +30,16 @@ class profile::wiki (
   $nfs_server            = 'localhost',
   $DeletedFileStoreMount = '/nfs/share/wiki/deleted',
   $ImagesMount           = '/nfs/share/wiki/images',
-  $database              = 'wikidev',
+  $sql_host              = undef,
+  $sql_user              = 'wikiuser',
+  $sql_pass              = 'password',
+  $sql_db                = 'wikidev',
 ) {
+  # Verify required values
+  if ($sql_host == undef) {
+    fail('No sql_host has been specified. Provide the name/ip of the database server hosting the specified wiki db.')
+  }
+
   # Global settings
   $docroot  = '/srv/www' # Built into the packages, cannot be changed.

@@ -132,14 +140,14 @@ class profile::wiki (
   }
   cron { 'sphinx_main':
     ensure  => 'present',
-    command => "/usr/bin/indexer --quiet --config /opt/sphinx/sphinx.conf ${database}_main --rotate >/dev/null 2>&1",
+    command => "/usr/bin/indexer --quiet --config /opt/sphinx/sphinx.conf wiki_main --rotate >/dev/null 2>&1",
     hour    => 3,
     minute  => 0,
     user    => 'sphinx',
   }
   cron { 'sphinx_incrementals':
     ensure  => 'present',
-    command => "/usr/bin/indexer --quiet --config /opt/sphinx/sphinx.conf ${database}_incremental --rotate >/dev/null 2>&1",
+    command => "/usr/bin/indexer --quiet --config /opt/sphinx/sphinx.conf wiki_incremental --rotate >/dev/null 2>&1",
     minute  => 5,
     user    => 'sphinx',
   }

$database has been replaced with $sql_db, which allows us to also fix a subtle bug with the cronjobs for sphinx (the config file in the package was static and only held ‘wiki_main’, so any other database name broke it… silently). The other variables required in the ERB template are provided along with defaults. We set $sql_host to a default of undef and then check to see if it has a value. If it remains undef, we error out and fail the catalog compilation. Deploy this change and you’ll see the error message:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: No sql_host has been specified. Provide the name/ip of the database server hosting the 
specified wiki db. at /etc/puppet/environments/v1_23_5/modules/profile/manifests/wiki.pp:40 on node wikidev01.nelson.va

Let’s go back to the YAML for this host. This could be done at another hierarchy level, if you have multiple servers connecting to the same database. Add in the three missing values and rename the one existing values. IPs and passwords are anonymized of course:

-profile::wiki::database              : 'wikitesting'
+profile::wiki::sql_host              : '192.168.1.15'
+profile::wiki::sql_user              : 'wikiuser'
+profile::wiki::sql_pass              : 'password'
+profile::wiki::sql_db                : 'wikitesting'

Let’s re-run the agent again.

+       # data source
+       type            = mysql
+       sql_host        = 192.168.1.15
+       sql_user        = wikiuser
+       sql_pass        = password
+       sql_db          = wikitesting

You should also see some entries about searchd having a scheduled refresh.Congratulations, you’ve just created a template.

Summary

Alongside a module that we’ve converted from hardcoded values to values from hiera, we’ve taken some hardcoded config files and created a template that can populate the config with hiera data. This article went through one file for one profile. You’ll probably find more files that would benefit from template conversion (for example: mediawiki’s LocalSettings.php also has data source information). If these config files were in packages (hint: like my mediawiki-${environment}-config package), then the process might be fairly lengthy, stripping files out of packages to turn them into templates populated from hiera, but you now have all the tools you need. Go forth and template!