Welcome back! In our 201 class, we installed r10k, but we still haven’t used it. There’s two tasks we need to complete. First, the existing repo is incompatible with r10k’s dynamic management of modules, so we’ll convert its contents to the proper format. Once that is done, we can deploy dynamic environments using r10k.

Convert existing repo

Check out a clone of the existing puppet repo, rnelson0/puppet-tutorial. As mentioned previously, you can do this on the puppet master, as I will do, or you can perform it on another machine. If you’re on the master, you want to clone the repo into a different directory. After cloning it, check out a new branch called production:

[root@puppet ~]# cd git
[root@puppet git]# git clone https://rnelson0@github.com/rnelson0/puppet-tutorial
Initialized empty Git repository in /root/git/puppet-tutorial/.git/
remote: Reusing existing pack: 775, done.
remote: Total 775 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (775/775), 385.69 KiB, done.
Resolving deltas: 100% (162/162), done.
[root@puppet git]# cd puppet-tutorial/
[root@puppet puppet-tutorial]# git checkout -b production
Switched to a new branch 'production'
[root@puppet puppet-tutorial]# git branch
  master
* production

Last week, we discussed the changes required for the repo. To recap:

The directory structure of an r10k repo is much simpler:

.
|-- manifests
  |-- site.pp
|-- Puppetfile
|-- dist
|-- site
|-- .gitignore

The puppet configuration files are removed. The site manifest stays where it is, the modules directory is replaced by the Puppetfile, and we have two new directories, dist for internally developed generic modules (where roles/profiles may reside) and site for modules deploying custom services.

To accomplish this, we need to add the file Puppetfile and remove some tracked objects. Let’s start with removing the files and directories.

[root@puppet puppet-tutorial]# git rm -r modules auth.conf fileserver.conf puppet.conf
rm 'auth.conf'
rm 'fileserver.conf'
rm 'modules/apache/CHANGELOG.md'
rm 'modules/apache/CONTRIBUTING.md'
<snip>
rm 'modules/ygrpms/tests/init.pp'
rm 'puppet.conf'
[root@puppet puppet-tutorial]#

Next we have to add the Puppetfile. You can find the details of the format on the librarian-puppet github page. You set the forge location, then include a list of mods on the forge, optionally with version, or via git checkout, also with optional version/tag. Unlike librarian-puppet, however, r10k requires you to track dependencies. Let me repeat that: r10k does NOT track dependencies, all dependencies must be stated in your Puppetfile! Up above, we performed two puppet module list commands, or you can run the command yourself. Capture that output and add it to a file. You’ll need to convert it to Puppetfile format. If you toss that all in vi, here are the commands to modify it very quickly:

:%s/\s*[└┬├─]*\s/mod "/
:%s/ (v/", "/
:%s/)/"/
:%s/-/\//g

We need to remove the custom module from the list (rnelson0-base) and add a module from git for it. Be warned, modules from git do NOT have the username before the name of the module, so rnelson0-base becomes simply base. Converting the module lists, replacing rnelson0-base with base from git/rnelson0-base, adding the forge header at the top, and some descriptions, gives us this Puppetfile:

forge "http://forge.puppetlabs.com"

# r10k requirements
mod "zack/r10k", "1.0.2"
mod "gentoo/portage", "2.1.0"
mod "mhuffnagle/make", "0.0.2"
mod "puppetlabs/gcc", "0.1.0"
mod "puppetlabs/git", "0.0.3"
mod "puppetlabs/inifile", "1.0.3"
mod "puppetlabs/pe_gem", "0.0.1"
mod "puppetlabs/ruby", "0.1.0"
mod "puppetlabs/vcsrepo", "0.2.0"

# Additional modules from the Puppet Forge
mod "puppetlabs/apache", "0.11.0"
mod "puppetlabs/concat", "1.0.1"
mod "puppetlabs/firewall", "1.0.0"
mod "puppetlabs/ntp", "3.0.1"
mod "puppetlabs/stdlib", "4.1.0"
mod "saz/ssh", "1.4.0"
mod "yguenane/augeas", "0.1.1"
mod "yguenane/ygrpms", "0.1.0"

# Modules from Github
mod "base",
  :git => "git://github.com/rnelson0/rnelson0-base"

You’ll notice that I included versions for every module, but the librarian-puppet example doesn’t list them. I’m not going to enter a holy war of whether you should always tie to specific versions or not – do whatever is standard for your shop.

Create the Puppetfile, add it to your local repo, commit, and push the production branch upstream

[root@puppet puppet-tutorial]# cat > Puppetfile
forge "http://forge.puppetlabs.com"

# Modules from the Puppet Forge
mod "puppetlabs/apache", "0.11.0"
mod "puppetlabs/concat", "1.0.1"
mod "puppetlabs/firewall", "1.0.0"
mod "puppetlabs/ntp", "3.0.1"
mod "puppetlabs/stdlib", "4.1.0"
mod "saz/ssh", "1.4.0"
mod "yguenane/augeas", "0.1.1"
mod "yguenane/ygrpms", "0.1.0"

# For our r10k installer
mod "zack/r10k", "1.0.2"
mod "gentoo/portage", "2.1.0"
mod "mhuffnagle/make", "0.0.2"
mod "puppetlabs/gcc", "0.1.0"
mod "puppetlabs/git", "0.0.3"
mod "puppetlabs/inifile", "1.0.3"
mod "puppetlabs/pe_gem", "0.0.1"
mod "puppetlabs/ruby", "0.1.0"
mod "puppetlabs/vcsrepo", "0.2.0"

# Modules from Github
mod "base",
  :git => "git://github.com/rnelson0/rnelson0-base"
[root@puppet puppet-tutorial]# git add Puppetfile
[root@puppet puppet-tutorial]# git commit -m 'Conversion to r10k format'
[production bef7066] Conversion to r10k format
 649 files changed, 26 insertions(+), 38106 deletions(-)
 create mode 100644 Puppetfile
 delete mode 100644 auth.conf
 
 delete mode 100644 modules/ygrpms/tests/init.pp
 delete mode 100644 puppet.conf
[root@puppet puppet-tutorial]# git push origin production
Counting objects: 4, done.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 631 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
To https://rnelson0@github.com/rnelson0/puppet-tutorial
 * [new branch]      production -> production

If you refresh your github page now, you should see the new branch production, and it should look markedly different from the master branch.

Deploying using r10k

Now that we have created a repo for our module, installed r10k, and converted our main repo to use r10k, we need to use r10k to deploy everything. This step is important because, after installing r10k with zack/r10k, r10k started managing our modules directory. It’s very empty and catalogs will fail to compile. Switch over to the puppet master and perform a noop agent run.

[root@puppet puppet-tutorial]# puppet agent --test --noop
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve information from environment production source(s) puppet://puppet.nelson.va/plugins
Notice: /File[/var/lib/puppet/lib/puppet]: Dependency File[/var/lib/puppet/lib] has failures: true

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class ::base for puppet.nelson.va on node puppet.nelson.va
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
[root@puppet puppet-tutorial]# puppet config print modulepath
/etc/puppet/environments/production/modules:/opt/puppet/share/puppet/modules
[root@puppet puppet-tutorial]# ls /etc/puppet/environments/production/modules
ls: cannot access /etc/puppet/environments/production/modules: No such file or directory
[root@puppet puppet-tutorial]# ls /opt/puppet/share/puppet/modules
ls: cannot access /opt/puppet/share/puppet/modules: No such file or directory

Don’t worry! r10k will take care of this if we let it. We can deploy our dynamic environments with the command r10k deploy environment -p:

[root@puppet puppet-tutorial]# r10k deploy environment -pv
Faraday: you may want to install system_timer for reliable timeouts
[R10K::Task::Deployment::DeployEnvironments - INFO] Loading environments from all sources
[R10K::Task::Environment::Deploy - NOTICE] Deploying environment master
[R10K::Task::Puppetfile::Sync - INFO] Loading modules from Puppetfile into queue
[R10K::Task::Environment::Deploy - NOTICE] Deploying environment production
[R10K::Task::Puppetfile::Sync - INFO] Loading modules from Puppetfile into queue
[R10K::Task::Module::Sync - INFO] Deploying base into /etc/puppet/environments/production/modules
[R10K::Task::Module::Sync - INFO] Deploying vcsrepo into /etc/puppet/environments/production/modules
[R10K::Task::Module::Sync - INFO] Deploying ruby into /etc/puppet/environments/production/modules
[R10K::Task::Module::Sync - INFO] Deploying pe_gem into /etc/puppet/environments/production/modules
[R10K::Task::Module::Sync - INFO] Deploying inifile into /etc/puppet/environments/production/modules
[R10K::Task::Module::Sync - INFO] Deploying git into /etc/puppet/environments/production/modules
[R10K::Task::Module::Sync - INFO] Deploying gcc into /etc/puppet/environments/production/modules
[R10K::Task::Module::Sync - INFO] Deploying make into /etc/puppet/environments/production/modules
[R10K::Task::Module::Sync - INFO] Deploying portage into /etc/puppet/environments/production/modules
[R10K::Task::Module::Sync - INFO] Deploying r10k into /etc/puppet/environments/production/modules
[R10K::Task::Module::Sync - INFO] Deploying ygrpms into /etc/puppet/environments/production/modules
[R10K::Task::Module::Sync - INFO] Deploying augeas into /etc/puppet/environments/production/modules
[R10K::Task::Module::Sync - INFO] Deploying ssh into /etc/puppet/environments/production/modules
[R10K::Task::Module::Sync - INFO] Deploying stdlib into /etc/puppet/environments/production/modules
[R10K::Task::Module::Sync - INFO] Deploying ntp into /etc/puppet/environments/production/modules
[R10K::Task::Module::Sync - INFO] Deploying firewall into /etc/puppet/environments/production/modules
[R10K::Task::Module::Sync - INFO] Deploying concat into /etc/puppet/environments/production/modules
[R10K::Task::Module::Sync - INFO] Deploying apache into /etc/puppet/environments/production/modules
[R10K::Task::Deployment::PurgeEnvironments - INFO] Purging stale environments from /etc/puppet/environments
[R10K::Task::Puppetfile::Purge - INFO] Purging stale modules from /etc/puppet/environments/master/modules
[root@puppet puppet-tutorial]# ls /etc/puppet/environments/production/modules
apache  base    firewall  git      make  pe_gem   r10k  ssh     vcsrepo
augeas  concat  gcc       inifile  ntp   portage  ruby  stdlib  ygrpms

If you run the agent now, you’ll see it’s much, much happier:

[root@puppet puppet-tutorial]# puppet agent --test --noop
<snip>
Info: Applying configuration version '1395849183'
Notice: /Stage[main]/Main/Node[puppet.nelson.va]/Notify[Generated from our notify branch]/message: current_value absent, should be Generated from our notify branch (noop)
Notice: Node[puppet.nelson.va]: Would have triggered 'refresh' from 1 events
Notice: Class[Main]: Would have triggered 'refresh' from 1 events
Notice: Stage[main]: Would have triggered 'refresh' from 1 events
Notice: Finished catalog run in 2.02 seconds

Congratulations, you’ve successfully configured your repo for dynamic environments! Let’s run through some of the steps above that you’ll be repeating, though:

• git checkout -b <feature>: Create a new branch. Make edits, commit, push to origin. This step can be done anywhere
• r10k deploy environment -p[v]: Deploy the environments, one per branch of your repo. Any branches without a Puppetfile, like our master, will be empty. The -v is for verbose and is optional. This must be run on the puppet master (we’ll look at automating it later).
• puppet agent –test [–environment <name>]: Run a puppet agent against the production environment, or the optional environment specified. This dynamic environment name would be the name of your new feature branch.

Cleanup

We have a little cleanup to do from our pre-r10k setup.

[root@puppet puppet-tutorial]# tree /etc/puppet -L 1
/etc/puppet
├── auth.conf
├── environments
├── fileserver.conf
├── manifests
├── modules
├── puppet.conf
└── README.md

We need the conf files, but modules are now referenced under environments/$environment/modules. Remove the unused modules and you’ll see catalog compilation is unaffected:

[root@puppet puppet-tutorial]# rm -fR /etc/puppet/modules/
[root@puppet puppet-tutorial]# puppet agent --test --noop
Info: Applying configuration version '1395849183'
Notice: /Stage[main]/Main/Node[puppet.nelson.va]/Notify[Generated from our notify branch]/message: current_value absent, should be Generated from our notify branch (noop)
Notice: Node[puppet.nelson.va]: Would have triggered 'refresh' from 1 events
Notice: Class[Main]: Would have triggered 'refresh' from 1 events
Notice: Stage[main]: Would have triggered 'refresh' from 1 events
Notice: Finished catalog run in 0.97 seconds

The same goes for manifests which are now under /etc/puppet/environments/$environment/manifests. Get rid of the original copy of this as well:

[root@puppet puppet-tutorial]# rm -fR /etc/puppet/manifests/
[root@puppet puppet-tutorial]# puppet agent --test --noop
Info: Applying configuration version '1396323811'
Notice: /Stage[main]/Main/Node[puppet.nelson.va]/Notify[Generated from our notify branch]/message: current_value absent, should be Generated from our notify branch (noop)
Notice: Node[puppet.nelson.va]: Would have triggered 'refresh' from 1 events
Notice: Class[Main]: Would have triggered 'refresh' from 1 events
Notice: Stage[main]: Would have triggered 'refresh' from 1 events
Notice: Finished catalog run in 0.99 seconds

We also have two branches in the puppet-tutorial repo. When the environment is unspecified, production is used, which is why we created the new branch. There’s no need for the master branch anymore. Because of my tutorial articles, I won’t delete the branch, but in a production environment I would. This would leave one main branch, production, plus the feature branches. You could easily do this through the github interface, or you could delete the branch locally and push it upstream. You’ll also need to go to Settings for the repo and set the default branch to production or you’ll get the error “deletion of the current branch prohibited”. You can make the change even if you keep master around, that way anyone doing a git clone will receive production as the starting branch. Here’s an example of me deleting a branch master2 from the CLI rather than github:

[root@puppet puppet-tutorial]# git pull origin master2
From https://github.com/rnelson0/puppet-tutorial
 * branch master2 -> FETCH_HEAD
Already up-to-date.
[root@puppet puppet-tutorial]# git branch
 master
* production
[root@puppet puppet-tutorial]# git branch master2
[root@puppet puppet-tutorial]# git branch
 master
 master2
* production
[root@puppet puppet-tutorial]# git checkout master
Switched to branch 'master'
[root@puppet puppet-tutorial]# git branch -D master2
Deleted branch master2 (was bef7066).
[root@puppet puppet-tutorial]# git branch
* master
 production
[root@puppet puppet-tutorial]# git push origin :master2
To https://rnelson0@github.com/rnelson0/puppet-tutorial
 - [deleted] master2

With our filesystem and github repo cleaned up, we’re all set for dynamic environments without any cruft of the previous configuration.

In the next class, we’ll define a new workflow that utilizes r10k, step one toward a goal of not logging into the master as root every time you make a change.