Why Puppet?

As we near the end of my schedule puppet content, I’ve asked the Twitterverse for any other topics people want to see discussed. Jason Shiplett asked a great question: Why Puppet?

This is essentially a two-fold question. First, you must understand what Configuration Management (CM) is and why you need it. Second, of all the CM tools out there, why would you choose Puppet?

Configuration Management

In spite of my telling Jason that the world doesn’t need another “Why CM?” post, here we go :)

Plenty of other people have done a great job explaining what Configuration Management is and why you need it. Chief among these is Information Technology Infrastructure Library, or ITIL, a framework for IT Service Management. In the Service Transition volume, Configuration Management is described. We can simplify the meaning to describing and managing the state of a configuration through a service’s lifecycle.

Continue reading

Refactoring a Puppet class for use with Hiera

For the past few weeks, we have been working on packaging our own software and deploying it with puppet. Before that, we touched on refactoring modules to use hiera. In fact, I grandiosely referred to it as Hiera, R10K, and the end of manifests as we know them. I included a very simple example of how to refactor a per-node manifest into the role/profile pattern and use hiera to assign it to the node. Today, we’ll look at more features of hiera and how you would refactor an existing class to use hiera.

In a legacy implementation of puppet, you’ll likely find plenty of existing modules whose classes have static assignment or lots of conditionals to determine the necessary values to be applied. Even in a greenfield implementation of puppet, you may find yourself writing straight Puppet DSL for your classes before refactoring them to use hiera. Figuring out how to refactor efficiently isn’t always obvious.

First, let’s take a look at Gary Larizza’s When to Hiera  and The Problem with Separating Data from Puppet Code articles. Gary covers the when and why much better than I could, so please, go read his article and then come back here. Gary also covers the common pre-hiera pattern and a few patterns that can be used when refactoring to hiera. There is another pattern that is documented indirectly by Gary (under Hiera data bindings in Puppet 3.x.x) and in the Hiera Complete Example at docs.puppetlab.com. I’m going to explain this pattern and document this directly, adding another Hiera pattern to Gary’s list.

Continue reading

FPM and Build Automation

Having created one or more build servers, the next logical step is to start building software. We touched on this briefly a few weeks ago, and with a proper development station, it’s time to expand on it.

If you’re a developer by trade, you can probably skim or skip this article. Remember, this series is aimed at vSphere Admins, not devs. I’d certainly appreciate your insights in the comments or on twitter, though!

Modifying software build processes for FPM

We’ve used FPM in the past to take a directory and turn it into a package. This works very well when /some/long/path belongs entirely to your application. What if your application drops a binary in /bin, a manpage in /usr/share/man/man5, a config file in /etc, or even just a few files in a directory that’s shared with other packages? Let’s take a look at an extension for mediawiki. This is very simple, we have a legacy Makefile and two useful targets, dev and prod:

Continue reading

Puppetize a Build server

The Puppet series so far has really focused on VM builds and just started to touch on software packaging. We need an appropriate place to do this work, and what better way to set that than via Puppet itself? Today, we’ll create some roles and profiles for a build server, which could be permanent and share amongst developers, spun up as needed for the team, or spun up per developer.

Build Profile and Role

The last few examples we have done with FPM were on our “production” servers. That’s less than ideal for a few reasons. You wouldn’t want to mess up the publicly available service while packaging, whether by overwriting a file, exhausting resources, or the brief outage when services restart. It is not a good idea to add compilers and development libraries to any server unnecessarily as it increases the attack surface (additional security risks, additional packages to patch, additional items for auditors to flag, etc). You also probably do not want your build servers in the same environment as your production servers (unless, as is the case in these examples, your “production” environment is your lab – so just pretend it’s a different environment). Let’s assume that we do not have a good reason to violate these best practices, so our goal is set up a dedicated build server. It will require all the software we have been using so far, and we will throw on a local user. If you have LDAP or another directory service in your lab, I would suggest using it, but this is a good example as sometimes the build network is restricted.

We have two profiles to create, then. The first is the local users. We’ll call this class ::profile::build_users, in case we create another grouping of users later. The second profile is for our build software, and we will call it ::profile::build. Here are the two class files, located at profile/manifests/users/build.pp and profile/manifests/build.pp, respectively.

Continue reading

Deploying your custom application with Puppet

In the past two weeks, we learned how to create packages for our own applications and how to host them in a repository. The next step is to puppetize the application so that you can deploy your application to nodes through automation. We’ll need to add the repo to our base profile, so all nodes receive it, define a profile that requires the application, and a role class and corresponding hiera yaml to apply the configuration to a specified node. Let’s get started!

Add the repo to the base profile

This step is fairly simple. Last week, we defined the repo and applied it manually with:

  yumrepo {'el-6.5':
    descr    => 'rnelson0 El 6.5 - x86_64',
    baseurl  => 'http://yum.nelson.va/el-6.5/',
    enabled  => 'true',
    gpgcheck => 'false',
  }

Add that to your base profile. It should look something like this now:

Continue reading

Create a Yum Repo

In last week’s article, we learned how to build a package with FPM, specifically an RPM. Today, we’ll look at creating a Yum repository to host your packages. The repo can be built with puppet, which can also distribute settings so all your managed nodes can use the repo. By adding the package to the repo, it becomes available to install, again via puppet. This is the first step on the road to the automated software packing and delivery that is vital for continuous integration.

A repo has a few components.

  • Webserver – Content is served up over http.
  • createrepo – A piece of software that manages the repo’s catalog.
  • RPMs – What it’s serving up.

We don’t need to know how the pieces work, though. We’ll rely on palli/createrepo to manage the repo itself. We just make sure a webserver is available, the directories are there, and that there’s some content available.

Configure a host

I’m going to start moving faster now, because we’ve done this part a few times already. Please let me know on twitter if I’m going to fast and I can update the page and make sure future articles keep the same pace.

The first thing is to choose a node. Since our series so far relies on the hostname for a role, spin up a new VM called ‘yumrepo01′ or something similar. You don’t need to configure anything afterward, except possibly update Puppet (as I’m writing this, v3.7.0 was just released). Of course, you should run the agent and accept the cert so that the node can communicate with puppet before continuing.

Next up is the manifest. Start with the profile.

[rnelson0@puppet profile]$ cat manifests/yumrepo.pp
class profile::yumrepo {
  include '::profile::apache'

  apache::vhost {'yum.nelson.va':
    docroot    => '/var/www/html/puppetrepo',
  }
}

Our existing ::profile::apache ensures the web server is installed and running and creates firewall rules to allow traffic in. The apache::vhost definition creates the virtual host for our web server. We will place our repo underneath /var/www/html/puppetrepo but you can choose anywhere you want. The next piece is the role.

[rnelson0@puppet role]$ cat manifests/yumrepo.pp
class role::yumrepo {
  include profile::base  # All roles should have the base profile
  include profile::yumrepo

  $repodirs = hiera('repodirs')
  file { $repodirs :
    ensure => 'directory',
  }
  create_resources('::createrepo', hiera_hash('yumrepos'), {require => File[$repodirs]} )
}

After adding the base and yumrepo commands, we need to create the directories for the repo itself. The vhost doesn’t actually create the specified directory, and there’s also a cache directory. We also create some ::createrepo defines. All of this content comes from hiera:

[rnelson0@puppet hiera-tutorial]$ cat puppet_role/yumrepo.yaml
---
classes:
  - role::yumrepo
repodirs:
  - '/var/www/html/puppetrepo'
  - '/var/cache/puppetrepo'
yumrepos:
  'el-6.5':
    repository_dir       : '/var/www/html/puppetrepo/el-6.5'
    repo_cache_dir       : '/var/cache/puppetrepo/el-6.5'
    suppress_cron_stdout : true

Our repo has the name el-6.5 (based on our CentOS 6.5 base image) and exists one level underneath our docroot and cache directory (there’s a good reason why). There’s a cron job that runs every minute, rebuilding the index. Unless you like getting new mail every minute, I suggest suppressing output. Here’s the cron job it will create:

# Puppet Name: update-createrepo-el-6.5
*/1 * * * * /usr/bin/createrepo --cachedir /var/cache/puppetrepo/el-6.5 --changelog-limit 5 --update /var/www/html/puppetrepo/el-6.5 1>/dev/null

There are other options you can feed ::createrepo, but this will suffice to start. With the role and profile manifest and hiera yaml in place on the master, you can run the agent on yumrepo01 and everything should install properly. You can test access by visiting http://yum.nelson.va/ afterward. You should see the el-6.5/repodata directory structure, and the bottom dir will have a few files. Great success!

Populating the repo

This step is easy – copy files to the repo. Where packages go is up to you. You can toss them in the top level, you can create an RPMS or Packages directory; wherever you through it, createrepo will find it. We’ll put our files at the top level (/var/www/html/puppetrepo/el-6.5/ in this case). The crontab job runs once a minute, so after a brief wait it will be ready. You can again browse to the repo at http://yum.nelson.va/el-6.5/, using the correct hostname for your environment.

If you do use subdirectories, ::createrepo does not create or manage this subdirectory, so you would have to. Be aware that by doing so, you’ll be managing a directory that is managed by the define and you’ll have to ensure you don’t manage the same directory twice, once via ::createrepo and twice via a File resource. You could set manage_repo_dirs to false in your ::createrepo definition if you’d rather manage all the directories yourself.

Using the repo

Finally, we need a way to make use of the repo. A yumrepo resource type is native to puppet, so let’s look at the definition for our new repo:

  yumrepo {'el-6.5':
    descr    => 'rnelson0 El 6.5 - x86_64',
    baseurl  => 'http://yum.nelson.va/el-6.5/',
    enabled  => 'true',
    gpgcheck => 'false',
  }

We are defining the repo el-6.5, giving it a description and the URL to use. Adding a repo will just create the repo file but it defaults to disabled, so we set enabled to true. The default for gpgcheck, which ensures signatures on RPMs are valid, is true. Since we haven’t signed our package, we’ll set that to false. I’ll leave configuration of gpg, or lack thereof, up to you. Let’s go back to server01 and use puppet apply to attach the repo. Afterward, search for helloworld.

[rnelson0@server01 ~]$ yum repolist
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: bay.uchicago.edu
 * extras: mirror.steadfast.net
 * updates: mirror.anl.gov
repo id                                   repo name                                                     status
base                                      CentOS-6 - Base                                               6,367
extras                                    CentOS-6 - Extras                                                15
puppetlabs-deps                           Puppet Labs Dependencies El 6 - x86_64                           68
puppetlabs-products                       Puppet Labs Products El 6 - x86_64                              422
updates                                   CentOS-6 - Updates                                            1,467
repolist: 8,339
[rnelson0@server01 ~]$ sudo puppet apply
  yumrepo {'el-6.5':
    descr    => 'rnelson0 El 6.5 - x86_64',
    baseurl  => 'http://yum.nelson.va/el-6.5/',
    enabled  => 'true',
    gpgcheck => 'false',
  }
Notice: Compiled catalog for server01.nelson.va in environment production in 0.11 seconds
Notice: /Stage[main]/Main/Yumrepo[el-6.5]/ensure: created
Notice: Finished catalog run in 0.07 seconds
[rnelson0@server01 ~]$ yum repolist
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: bay.uchicago.edu
 * extras: mirror.steadfast.net
 * updates: mirror.anl.gov
el-6.5                                                                                 | 2.9 kB     00:00
el-6.5/primary_db                                                                      | 2.3 kB     00:00
repo id                                   repo name                                                     status
base                                      CentOS-6 - Base                                               6,367
el-6.5                                    rnelson0 El 6.5 - x86_64                                          1
extras                                    CentOS-6 - Extras                                                15
puppetlabs-deps                           Puppet Labs Dependencies El 6 - x86_64                           68
puppetlabs-products                       Puppet Labs Products El 6 - x86_64                              422
updates                                   CentOS-6 - Updates                                            1,467
repolist: 8,342
[rnelson0@server01 ~]$ yum search helloworld
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: bay.uchicago.edu
 * extras: mirror.steadfast.net
 * updates: mirror.anl.gov
========================================== N/S Matched: helloworld ===========================================
helloworld.x86_64 : no description given

  Name and summary matches only, use "search all" for everything.

Go ahead and try and install it with yum (abbreviated output):

[rnelson0@server01 ~]$ sudo yum install helloworld
...
Resolving Dependencies
--> Running transaction check
---> Package helloworld.x86_64 0:1.0-1 will be installed
--> Finished Dependency Resolution
...
Is this ok [y/N]: y
Downloading Packages:
http://yum.nelson.va/el-6.5/helloworld-1.0-1.x86_64.rpm: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403 Forbidden"

If you check the permissions on the file, you might wonder why a world readable file isn’t available. Our CentOS template enforces selinux, so let’s take a look at the context:

[root@yumrepo01 el-6.5]# ls -laZ /var/www/html/puppetrepo/el-6.5/helloworld-1.0-1.x86_64.rpm
-rw-rw-r--. rnelson0 rnelson0 unconfined_u:object_r:user_home_t:s0 /var/www/html/puppetrepo/el-6.5/helloworld-1.0-1.x86_64.rpm
[root@yumrepo01 el-6.5]# ls -laZ /var/www/html/puppetrepo/el-6.5/repodata/repomd.xml
-rw-r--r--. root root unconfined_u:object_r:httpd_sys_content_t:s0 /var/www/html/puppetrepo/el-6.5/repodata/repomd.xml

Here’s our issue: the new rpm is not in the httpd_sys_content_t context but is in the user_home_t context. This occurred because I scp’ed the file from server01 as rnelson0 to yumrepo01 as rnelson0, then on yumrepo01 moved the file to /var/www/html/puppetrepo as root. Let’s move the file back, then try copying it as root:

[root@yumrepo01 el-6.5]# mv /var/www/html/puppetrepo/el-6.5/helloworld-1.0-1.x86_64.rpm ~rnelson0/
[root@yumrepo01 el-6.5]# cp ~rnelson0/helloworld-1.0-1.x86_64.rpm /var/www/html/puppetrepo/el-6.5/
[root@yumrepo01 el-6.5]# ls -laZ /var/www/html/puppetrepo/el-6.5/helloworld-1.0-1.x86_64.rpm
-rw-r--r--. root root unconfined_u:object_r:httpd_sys_content_t:s0 /var/www/html/puppetrepo/el-6.5/helloworld-1.0-1.x86_64.rpm

That’s better! You can also use the restorecon command to restore the default context on files, i.e. restorecon /var/www/html/puppetrepo/el-6.5/helloworld-1.0-1.x86_64.rpm. Now try and install it again:

[rnelson0@server01 ~]$ sudo yum install helloworld
...
Is this ok [y/N]: y
Downloading Packages:
helloworld-1.0-1.x86_64.rpm                                                            | 1.7 kB     00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : helloworld-1.0-1.x86_64                                                                    1/1
  Verifying  : helloworld-1.0-1.x86_64                                                                    1/1

Installed:
  helloworld.x86_64 0:1.0-1

Complete!
[rnelson0@server01 ~]$ rpm -qa | grep helloworld
helloworld-1.0-1.x86_64
[rnelson0@server01 ~]$ rpm -ql helloworld
/var/www/html/index.php

Summary

You now have a package, a repository that hosts it, and a client that can install the packages hosted by the repo. Go ahead and spin up server02-server10 and you should be able to easily define the repo and install the rpm. As we haven’t created any dependencies yet, the lack of httpd will prevent our app from working. If you install apache manually or via puppet, your hello world app will work. We’ll work on some more details of FPM, including dependencies, and integration of packages and repos with our puppet manifests next time.

References

Creating packages with FPM

In my exploration of Puppet, I’ve found a lot of oblique references to managing software deployments with it, but very little solid guides on how to do so. I need to tackle this for work, so I figured I should start at the top – creating a software deployment. To be clear, I’m speaking of internally developed software, or modifications to public software, not something you can find in your distribution’s packages and install with the puppet package resource type.

Creating Some Software

Going back even further, we need to create some software. I’d wager that most already have something laying around – perhaps a few scripts in a directory along with a Makefile that lets you run “make install” to put them in the final destination, or a tarball and config file that are “installed” by a script that untars the software and copies your customized config in place. If you don’t have something like that, let’s make something. How about a simple PHP application? It’s just a Hello World, nothing special, so you don’t need to know PHP for this.

Spin up a new VM, or requisition one of your existing dev VMs. I’m going to use server01 from the Puppet series. Make sure apache and php are installed, and if this node isn’t managed via our web server role, iptables may block connections so we will stop it:

[rnelson0@server01 ~]$ sudo puppet apply -e "package {['httpd', 'php']: ensure => present}"
Notice: Compiled catalog for server01.nelson.va in environment production in 0.33 seconds
Notice: /Stage[main]/Main/Package[php]/ensure: created
Notice: Finished catalog run in 20.70 seconds
[rnelson0@server01 ~]$ sudo service httpd restart
Stopping httpd:                                            [FAILED]
Starting httpd:                                            [  OK  ]
[rnelson0@server01 ~]$ sudo service iptables stop
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Unloading modules:                               [  OK  ]

Continue reading