Having created one or more build servers, the next logical step is to start building software. We touched on this briefly a few weeks ago, and with a proper development station, it’s time to expand on it.

If you’re a developer by trade, you can probably skim or skip this article. Remember, this series is aimed at vSphere Admins, not devs. I’d certainly appreciate your insights in the comments or on twitter, though!

Modifying software build processes for FPM

We’ve used FPM in the past to take a directory and turn it into a package. This works very well when /some/long/path belongs entirely to your application. What if your application drops a binary in /bin, a manpage in /usr/share/man/man5, a config file in /etc, or even just a few files in a directory that’s shared with other packages? Let’s take a look at an extension for mediawiki. This is very simple, we have a legacy Makefile and two useful targets, dev and prod:

[rnelson0@build ilink]$ cat Makefile.org
WEBROOT = /var/www
PROD = ${WEBROOT}/wiki/extensions
DEV = ${WEBROOT}/wikidev/extensions
INSTALL = install -m 0644
OBJECTS = ilink.php
WARNING = Make prod or dev individually. Too many accidents have occurred by pushing to both during dev cycles

all:
        $(error ${WARNING})

prod:
        install -d ${PROD}
        ${INSTALL} ${OBJECTS} -C ${PROD}

dev:
        install -d ${DEV}
        ${INSTALL} ${OBJECTS} -C ${DEV}

There’s nothing complex about this at all. However, the target directory /var/www/wiki/extensions will have files that belong to other extensions. With such a simple install of one file, we could cherry pick ilink.php. This wouldn’t work if there were more files – and we might be surprised if later on, someone adds another file to the OBJECTS variable. That’s okay, because with a few changes, we can install to a temporary directory and use that as our source for FPM. Add a $DESTDIR variable to installed components. In this example, just change the PROD and DEV lines at the top:

PROD = ${DESTDIR}${WEBROOT}/wiki/extensions
DEV = ${DESTDIR}${WEBROOT}/wikidev/extensions

Now, when you make the target, just create and provide a DESTDIR. It’s good practice to wipe out the temporary directory to make sure no cruft is hanging around:

[rnelson0@build ilink]$ rm -fR /tmp/ilink
[rnelson0@build ilink]$ mkdir /tmp/ilink
[rnelson0@build ilink]$ make prod DESTDIR=/tmp/ilink
install -d /tmp/ilink/var/www/wiki/extensions
install -m 0644 ilink.php -C /tmp/ilink/var/www/wiki/extensions
[rnelson0@build ilink]$ ls -l /tmp/ilink/var/www/wiki/extensions/ilink.php
-rw-r--r--. 1 rnelson0 rnelson0 643 Sep 23 03:13 /tmp/ilink/var/www/wiki/extensions/ilink.php

If you make similar changes to all your build systems – DESTDIR for make/Makefile and the equivalent for other systems like Maven or rake – installing a copy on a build server is as simple as make a fresh directory and install into it.

More FPM Goodness

Now it’s time to take a look at some more FPM use cases. There are quite a few documented at the github project’s wiki page. We’ll start by finishing up our example above, at the PackageMakeInstall page. There are a few options here that we glossed over before:

• -v <version>: Versions number and…
• –iteration <num>: RedHat’s ‘release’ and Debian’s ‘debian_revision’. The version plus iteration generates the unique version number for your target package. If a package exists with that version number, fpm will error out, so you want to increase one or both on every build.
• –description “<string>”: The package description, i.e. what you see from rpm -qi <package>
• -m <email>: Maintainer’s email. For the love of all that’s holy, use an email distro that will persist when you have moved on.
• –after-install <program>: A program to run after the package is installed, such as enabling and starting a service
• –before-remove <program>: As above, programs to run before a package is uninstalled.
• -d <package>: Create a dependency. For instance, our wiki extension should probably rely on apache (httpd), php, and mediawiki.
• –rpm-user <group> and –rpm-group <group>: Set the ownership of files in the package. With RPMs, this seems to always translate to root/root anyway, but I prefer to be explicit.
• -C <directory>: fpm will change to this directory (the CHDIR) and track files relative to this. Pair with DESTDIR to grab just the files for your package.
• <path>: Don’t forget the path. When using -C, it can be a simple period ‘.’ as everything underneath CHDIR applies.

If we put all these together, we can take our ilink file above and package it as mediawiki-extension-ilink:

[rnelson0@build ilink]$ fpm -s dir -t rpm -n mediawiki-extension-ilink \
>   --description "Mediawiki extension ilink" \
>   -m rnelson0@gmail.com \
>   -d httpd \
>   --rpm-user root \
>   --rpm-group root \
>   -v 0.9.0 \
>   --iteration 1 \
>   -C /tmp/ilink \
>   .
no value for epoch is set, defaulting to nil {:level=>:warn}
no value for epoch is set, defaulting to nil {:level=>:warn}
Created package {:path=>"mediawiki-extension-ilink-0.9.0-1.x86_64.rpm"}
[rnelson0@build ilink]$ rpm -qpil mediawiki-extension-ilink-0.9.0-1.x86_64.rpm
Name        : mediawiki-extension-ilink    Relocations: /
Version     : 0.9.0                             Vendor: rnelson0@build
Release     : 1                             Build Date: Tue 23 Sep 2014 05:38:12 AM GMT
Install Date: (not installed)               Build Host: build.nelson.va
Group       : default                       Source RPM: mediawiki-extension-ilink-0.9.0-1.src.rpm
Size        : 643                              License: unknown
Signature   : (none)
Packager    : rnelson0@gmail.com
URL         : http://example.com/no-uri-given
Summary     : Mediawiki extension ilink
Description :
Mediawiki extension ilink
/var/www/wiki/extensions/ilink.php

There are a number of other options available – vendor, license, url – that are visible from fpm –help, way too many to get into here. Suffice to say that you should provide as much detail as possible for every package you make. Many of the options, in particular vendor and license, are probably shared between packages you and your team develop, which should simplify things.

In addition to the method above, where a package is installed to a temporary directory and then CHDIR is set to pull all files underneath it, you can use the –prefix command to prepend a path to the files. With our one file application, this is definitely feasible. Here’s what that would look like:

[rnelson0@build ilink]$ fpm -s dir -t rpm -n mediawiki-extension-ilink \
>   --description "Mediawiki extension ilink" \
>   -m rnelson0@gmail.com \
>   -d httpd \
>   --rpm-user root \
>   --rpm-group root \
>   -v 0.9.0 \
>   --iteration 2 \
>   --prefix /var/www/wiki/extensions \
>   ilink.php
no value for epoch is set, defaulting to nil {:level=>:warn}
no value for epoch is set, defaulting to nil {:level=>:warn}
Created package {:path=>"mediawiki-extension-ilink-0.9.0-2.x86_64.rpm"}
[rnelson0@build ilink]$ rpm -qpl mediawiki-extension-ilink-0.9.0-2.x86_64.rpm
/var/www/wiki/extensions/ilink.php

The generated rpm looks the same except for the iteration number. This is very useful for such single-directory packages. You can even put it in your Makefile:

[rnelson0@build ilink]$ cat Makefile
WEBROOT = /var/www
PROD = ${DESTDIR}${WEBROOT}/wiki/extensions
DEV = ${DESTDIR}${WEBROOT}/wikidev/extensions
INSTALL = install -m 0644
OBJECTS = ilink.php
WARNING = Make prod or dev individually. Too many accidents have occurred by pushing to both during dev cycles
VERSION = 0.9.0
ITERATION = 3

package:
        fpm -s dir -t rpm -n mediawiki-extension-ilink \
          --description "Mediawiki extension ilink" \
          -m rnelson0@gmail.com \
          -d httpd \
          --rpm-user root \
          --rpm-group root \
          -v ${VERSION} \
          --iteration ${ITERATION} \
          --prefix /var/www/wiki/extensions \
          ilink.php

all:
        $(error ${WARNING})

prod:
        install -d ${PROD}
        ${INSTALL} ${OBJECTS} -C ${PROD}

dev:
        install -d ${DEV}
        ${INSTALL} ${OBJECTS} -C ${DEV}
[rnelson0@build ilink]$ make package
fpm -s dir -t rpm -n mediawiki-extension-ilink \
          --description "Mediawiki extension ilink" \
          -m rnelson0@gmail.com \
          -d httpd \
          --rpm-user root \
          --rpm-group root \
          -v 0.9.0 \
          --iteration 3 \
          --prefix /var/www/wiki/extensions \
          ilink.php
no value for epoch is set, defaulting to nil {:level=>:warn}
no value for epoch is set, defaulting to nil {:level=>:warn}
Created package {:path=>"mediawiki-extension-ilink-0.9.0-3.x86_64.rpm"}
[rnelson0@build ilink]$ rpm -qpl mediawiki-extension-ilink-0.9.0-3.x86_64.rpm
/var/www/wiki/extensions/ilink.php

Let’s try one more trick. The version and iteration above are hardcoded. The iteration, especially, is a pain, because now every build requires a modification to the Makefile and a new commit. There are a number of ways to get around this problem that are a combination of preference and what your build system allows. If your IDE or tooling auto-increments this, you’ve got one less problem to solve! If not, here’s a poor man’s way of making those numbers dynamic. We’ll reference some external files for the version and iteration, AND we’ll make the iteration value auto-increment on a completed build (in this case if ‘fpm’ does not return an error). I’ve removed the ‘dev’ target for simplicity and gone back to using DESTDIR with ‘-C’, which you’ll likely use more often.

Here’s how NOT to do this:

[rnelson0@build ilink]$ cat Makefile
WEBROOT = /srv/www
PROD = ${DESTDIR}${WEBROOT}/wiki/extensions
INSTALL = install -m 0644
OBJECTS = ilink.php
VERSION = $(shell if [ -f version ]; then cat version; else echo 0.0; fi)
ITERATION = $(shell if [ -f iteration ]; then cat iteration; else echo 1; fi)

package: install
        fpm -s dir -t rpm -n mediawiki-extension-ilink \
          --description "Mediawiki extension ilink" \
          -m rnelson0@gmail.com \
          -d httpd \
          --rpm-user root \
          --rpm-group root \
           -v ${VERSION} \
          --iteration ${ITERATION} \
          -C /tmp/ilink/var/www/wiki/extensions \
          .
        @expr ${ITERATION} + 1 > iteration

install:
        rm -fR ${DESTDIR}
        install -d ${PROD}
        ${INSTALL} ${OBJECTS} -C ${PROD}
[rnelson0@build ilink]$ echo 0.9 > version
[rnelson0@build ilink]$ echo 10 > iteration
[rnelson0@build ilink]$ DESTDIR=/tmp/ilink make package
rm -fR /tmp/ilink
install -d /tmp/ilink/srv/www/wiki/extensions
install -m 0644 ilink.php -C /tmp/ilink/srv/www/wiki/extensions
fpm -s dir -t rpm -n mediawiki-extension-ilink \
          --description "Mediawiki extension ilink" \
          -m rnelson0@gmail.com \
          -d httpd \
          -v 0.9 \
          --iteration 10 \
          -C /tmp/ilink/var/www/wiki/extensions \
          .
no value for epoch is set, defaulting to nil {:level=>:warn}
no value for epoch is set, defaulting to nil {:level=>:warn}
Created package {:path=>"mediawiki-extension-ilink-0.9-10.x86_64.rpm"}
[rnelson0@build ilink]$ cat version
0.9
[rnelson0@build ilink]$ cat iteration
11

First, let’s look at we did properly. The VERSION and ITERATION values come from the files version and iteration. version should go in source control; iteration gets auto-incremented so may be a bad file to track. Since it starts at 1 if it’s not present, lack of the file just means you start over. We use some simple tests and expr to increment the number, it only looks tricky because make isn’t exactly like your shell. If you take the auto-increment out, tracking the file in your repo is more appropriate and less frustrating. The package target relies on the install target to make sure everything is up to date, then builds the package with the write version-iteration. Looks pretty good. Notice the problem? Imagine if a user with the right (wrong?) privileges did something like this, maliciously or on purpose:

[rnelson0@build ilink]$ DESTDIR=/ make -n package
rm -fR /
...

Not only do you not have the RPM you were expecting, but you also have a long day ahead of you. It could also be more subtle, like setting DESTDIR to /var/log or /usr/share/bin, where the damage isn’t immediately apparent. Clearly, we don’t want to leave that in place. How do you fix it, though?

As we noted, make isn’t bash, so your options are a bit limited. A good solution seems to be letting make do its thing, including supporting DESTDIR, and creating a build script to do the packaging. The script sets DESTDIR to the name of a temporary directory we create, does a make install using that DESTDIR, checks for success before incrementing the iteration and deleting the temp directory, or alerts the user to check for errors without incrementing the iteration. There’s also a check to make sure that DESTDIR matches the expected pattern so that any in operando malfeasance with the variable does not impede the system; in the worst case, some other temporary directory is deleted and a coworker’s build needs to re-run.

Special thanks to Jordan Sissel for taking the time to work this out with me, the day after PuppetConf no less! You, sir, rock.

[rnelson0@build ilink]$ cat Makefile
WEBROOT = /srv/www
PROD = ${DESTDIR}${WEBROOT}/wiki/extensions
INSTALL = install -m 0644
OBJECTS = ilink.php

install:
        rm -fR ${DESTDIR}
        install -d ${PROD}
        ${INSTALL} ${OBJECTS} -C ${PROD}
[rnelson0@build ilink]$ cat build
VERSION=`if [ -f version ]; then cat version; else echo 0.0; fi`
ITERATION=`if [ -f iteration ]; then cat iteration; else echo 1; fi`
PACKAGE="mediawiki-extension-ilink"

export DESTDIR=`mktemp -d`
echo "Creating build in $DESTDIR"
make install
if [ $? -ne 0 ]
then
  echo ""
  echo "Make failed. Exiting. Check and clean $DESTDIR as needed."
  exit 1
fi

fpm -s dir -t rpm -n ${PACKAGE} \
  --description "Mediawiki extension ilink" \
  -m rnelson0@gmail.com \
  -d httpd \
  --rpm-user root \
  --rpm-group root \
   -v ${VERSION} \
  --iteration ${ITERATION} \
  -C ${DESTDIR} \
  .

if [ $? -ne 0 ]
then
  echo ""
  echo "Build failed. Exiting. Check and clean $DESTDIR as needed."
  exit 2
fi
expr $ITERATION + 1 > iteration

echo ""
echo "Build successful, removing $DESTDIR."
echo "$DESTDIR" | grep '^/tmp/tmp\.' >/dev/null && rm -rf "$DESTDIR"
exit 0
[rnelson0@build ilink]$ chmod +x build
[rnelson0@build ilink]$ ./build
Creating build in /tmp/tmp.QDu2VSN5gz
rm -fR /tmp/tmp.QDu2VSN5gz
install -d /tmp/tmp.QDu2VSN5gz/srv/www/wiki/extensions
install -m 0644 ilink.php -C /tmp/tmp.QDu2VSN5gz/srv/www/wiki/extensions
no value for epoch is set, defaulting to nil {:level=>:warn}
no value for epoch is set, defaulting to nil {:level=>:warn}
Created package {:path=>"mediawiki-extension-ilink-0.9-11.x86_64.rpm"}

Build successful, removing /tmp/tmp.QDu2VSN5gz.
[rnelson0@build ilink]$ echo 11 > iteration
[rnelson0@build ilink]$ ./build
Creating build in /tmp/tmp.wTfhHInCyR
rm -fR /tmp/tmp.wTfhHInCyR
install -d /tmp/tmp.wTfhHInCyR/srv/www/wiki/extensions
install -m 0644 ilink.php -C /tmp/tmp.wTfhHInCyR/srv/www/wiki/extensions
no value for epoch is set, defaulting to nil {:level=>:warn}
File already exists, refusing to continue: mediawiki-extension-ilink-0.9-11.x86_64.rpm {:level=>:fatal}

Build failed. Exiting. Check and clean /tmp/tmp.wTfhHInCyR as needed.

Again, this can be somewhat simplistic. You can see areas where the build script is going to be shared between applications, so that’s a place to look at providing more abstraction in the future. We’ll revisit this as we modernize and puppetize more processes.

There are a ton of other examples on the wiki – how to package gems, perl CPAN modules, even Solaris packages. Be sure to add to the wiki documentation if you work on anything unique.

Summary

Today we expanded our usage of FPM by looking at additional methods to build applications, in particular in a method that can be automated. We already have a package repository to store the applications. We’re getting closer to have an automated pipeline from software development to implementation in production.

I don’t have a topic for next week, yet, so please let me know what you’d like to see covered via twitter. Thanks!