Automating Puppet tests with a Jenkins Job, version 1.1

Today, let’s build on version 1.0 of our Jenkins job. We are running builds against every commit, but when someone opens a pull request, they don’t get automated builds or feedback. If the PR submitter even knows about Jenkins, and has network access and a login, they can look at it to find out how the tests went, but most people aren’t going to have that visibility (especially if your Jenkins server is private, as in this example setup). We need to make sure Jenkins is aware of the pull request and that it updates the PR with the status. Our end goal is for each PR to start a Jenkins build and update the PR with a successful check when done:

To get there, we will install and configure a new plugin and configure our job to use the plugin.

Continue reading

Automating Puppet tests with a Jenkins Job, version 1.0

As I’ve worked through setting up Jenkins and Puppet (and remembering my password!), I created a job to automate rspec tests on my puppet controlrepo. I am sure I will go through many iterations of this as I learn more, so we’ll just call this version 1.0. The goal is that when I push a branch to my controlrepo on GitHub, Jenkins automagically runs the test. Today, we will ensure that Jenkins is notified of activity on a GitHub repo, that it spins up a clean test environment without any left over files that may inadvertently assist, and run the tests. What it will NOT do is notify anyone – it won’t work off a Pull Request and provide feedback like Travis CI does, for instance. Hopefully, I will figure that out soon.

The example below is using GitHub. You can certainly make this work with BitBucket, GitLab, Mercurial, and tons of other source control systems and platforms, but you might need some additional Jenkins Plugins. It should be pretty apparent where to change Git/GitHub to the system/platform you chose.

Creating A Job

From the main view of your Jenkins instance, click New Item. Call it whatever you want, choose Freestyle project as the type, and click OK. The next page is going to be where we set up all the parameters for the job. There are tabs across the top AND you can scroll down; you’ll see the same selection items either way. Going from the top to the bottom, the settings that we want:

Continue reading

Jenkins Tricks – Password Recovery and Job Exports

I’m finally getting back to Jenkins, which I started waaaay back in November (here and here). Unfortunately, I kind of forgot my password. Well, that’s embarrassing! I also want to redo the manifest using maestrodev/rvm which means starting over, so I need to back things up. The manual for Jenkins and the results on Google can be overwhelming sometimes, so I thought I’d share what I learned to hopefully save someone else.

Password Recovery

There’s a few ways I found to recover your password. One suggestion is to disable all security, delete your user, re-enable security and allow signups, and then recreate the same user and things should just “work”. Part of the reason you have to do this is that once you disable security, you can’t change the password for your user; only the user can. That’s … frustrating.

Disable security by editing $JENKINS_HOME/config.xml, /var/lib/jenkins/config.xml on my instance. I was able to get away with disabling it by changing <useSecurity>true</useSecurity> to false, though the article suggests removing two other lines. Restart the service with systemctl restart jenkins or equivalent and now you’re able to get in and recreate some users.

Continue reading

Scheduling regular Travis CI builds with Cron Jobs

If you have a project of any complexity that is using Travis CI for testing, you’ve inevitably run into an issue where you make a minor change – maybe even just to a markdown file – and the tests that were passing before completely bomb. If this hasn’t happened to you, just wait, it will! This typically happens because some dependency, such as a newer rubygem you depend on indirectly, is no longer compatible with your test environment. As an example, I updated my .travis.yml in a puppet module project, opened PR16, and suddenly all the previously-green tests went red. The error NoMethodError: undefined method 'last_comment' for #<Rake::Application:0x000000015849f8> was observed because the dependent gem rake was updated from v11 to v12, conflicting with the pinned version of rspec. Until this problem is fixed, every PR, no matter how simple or complex, is going to fail its test.

This puts a lot of burden on both the person submitting the PR, who just wants their simple change to be approved, and the project maintainer, who wants their build status to work. Neither is satisfied until the problem is tracked down and remediated, often in a separate PR. By that time, the contributor may not have the interest to update their PR and the maintainer has to decide if they want to fix it up or discard it. No-one is happy. All because some upstream dependency changed and we weren’t aware of it until the moment a PR was submitted, even though it may have happened days or months ago.

Continue reading

Installing Jenkins and RVM

Update: Contrary to the module readme as of 12/1/2016, this WILL result in a VM running Jenkins 2, rather than version 1.

It’s time to get started installing Jenkins for use with Puppet. I’ll be installing this in my home lab on a vm called jenkins (so inventive, I know) as an all-in-one server for simplicity. I don’t know Jenkins real well so I don’t know what I should be doing for masters and build servers, but my home lab needs are small anyway. I gave it 2 cores and 1 GB of RAM, which I think is on the low end for Jenkins. My initial exploration led me to believe I’ll need to install Jenkins as well as RVM, since my base OS image is CentOS 7 (Ruby 2.0.0) and the current Puppet 4 AIO build is against Ruby 2.1. I’m using Puppet and the rtyler/jenkins module to set up Jenkins, since I don’t know anything about its innards. Down below, you’ll see that I installed RVM by hand, after which it occurred to me that there’s a perfectly cromulent maestrodev/rvm module that I’ve used in the past – something I may change out later, but I already wrote my manifest so I’m going to share it anyway!

I used Jenkins jobs to experiment a lot with this setup, such as enabling RVM on a job and seeing what errors showed up in the log, but I’m going to hold off on showing that magic until the next article. I will still explain where portions of the manifest came from, just without the actual errors.

Before we go any further, make sure you have the jenkins module and its dependencies added to your Puppetfile, .fixtures.yml, and wherever else you might need to track it.

Continue reading

Getting started with Jenkins and Puppet

If you’ve followed my blog, you’ve seen that I enjoy using Travis CI to run my puppet rspec tests on the controlrepo or against component modules. When you create a PR, Travis starts a build and adds some links to the PR notes to the builds. When it’s complete, the PR is updated to let you know whether the build was successful or not. You can also configure your repo to prevent merges unless the build succeeds. The best part is that it “just works” – you never have to worry about upgrading Travis or patching it or really anything other than making sure you enabled it. It’s a pretty awesome system, especially since it is absolutely free for open source projects. I do love Travis!

But, Travis isn’t always available and isn’t always the answer. Travis only runs against PRs or (typically) merges into the default branch. It won’t run on a schedule, even if your code never changed, which can help test any dynamic dependencies you might have (whether you should have any is a different topic!). It runs on a limited subset of OSes that Travis supports. You have to use Github and public repos to hit the free tier – if your controlrepo is private, there’s a different travis site to use and there is no free plan, though there is a limited number of trial builds to get you started. After that, it can be pretty pricey, starting at 1 concurrent builds for $69/month. This is great for a business, but most of us can’t afford $840 a year for the home network. It’s also, by itself, somewhat limited in how you can integrate it. It can be part of a pipeline, but it just receives a change and sends a status back to Github, it won’t notify another system itself. You have to build that yourself.

There are a ton of other Continuous Integration and Continuous Delivery systems out there, though. They can be cheaper, have better integrations, run on your own hardware/cloud, and don’t have to require Github. Among the myriad options available, I chose to look at Jenkins, with the rtyler/jenkins puppet module. I can run this on my own VM in my home lab (hardware/cloud), it can integrate with GitHub private repos ($0) or BitBucket (doesn’t require Github). CloudBees also discussed a new Puppet Enterprise Pipeline plugin at PuppetConf 2016, which is really appealing to me (better integrations). I can also have it run builds on a schedule, so if dependencies change out from underneath me, I’m alerted when that happens, not when I run my first test a month after the underlying change.

I’m very new to Jenkins and I’m going to do a lot of things wrong while testing it, but I’m going to start blogging about my experiences as I go, right or wrong, and then try and do a wrapup article once I learn more and have a fully working solution. I’ve been inspired to do this by Julia Evans and her wonderful technical articles that are more about the exploration of technology. That goes against my normal style of figuring it all out first, but I so love to see the exploration of others and hope you will, too! As we go on this journey, please feel free to educate or correct me in the comments and on twitter, as always.

Create a Least-Privilege account to perform domain joins

I’ve been working to automate joining linux machines to an Active Directory domain lately and I was surprised to find little documentation on creating an AD account just for the domain joins. I was able to piece things together by looking at umpteen documents and lots of trial and error. I’ve compiled what I found in the hopes that others do not have to struggle so much. This is just what I was able to find out – please let me know if you have a better way!


In a Windows Active Directory domain, it’s important to join computers to the domain. When the computer is joined, the computer account is created and lets it do things like send user authentication requests/receive responses, update it’s IP/name in DNS, and otherwise participate in the AD domain. This is a pretty fundamental and vital requirement, so Microsoft has made it easy for users to perform domain joins, but with some limits.

If a user is a enabled, a member of Domain Users, a member of the localĀ Administrators group, and the correct authentication information is used, they are granted the ability to join any given computer to a domain. The key ms-ds-MachineAccountQuota is defined at the domain level with a default value of 10. Due to the quota, any random user can join 10 computers to the domain after which they will no longer be able to do so. Enabled members of Domain Administrators are exempt from both local Administrators membership and the quota and can join unlimited computers so long as the correct authentication information is used.

This works great with Windows machines, but presents a slightly different problem when joining non-Windows machines to a domain. In these cases, there is likely no local Administrators group, so regular users are never able to satisfy all of the requirements to join a machine to the domain. Domain Administrators can, but that violates the principle of least privilege and is not the best option for production environments. We want a non-administrator account who can join as many computers to the domain as is required. I found two steps were required to create this account.

Active Directory Delegation

In the Active Directory Users and Computers MMC snap-in, you can use the Delegate Control wizard to delegate the ability to create computer accounts to a user account. Unfortunately, I have not found scriptable commands that are an equivalent to this wizard, so we need to describe the GUI process.

  • Create an account, ex: domainjoin, in the appropriate hierarchy of your Active Directory. It is recommend that User cannot change password and Password never expires are selected so the account is always available. It will not have ability to log into a server or any elevated privilege.
  • Delegate the ability to manage computer objects to the user with the Active Directory Users and Computers snap in (from JSI Tip 8144 with tweaks).
    • Open the Active Directory Users and Computers snap-in.
    • Right click the container under which you want the computers added (ex: Computers) and choose Delegate Control.
    • Click Next.
    • Click Add and supply your user account(s), e.g domainjoin. Click Next when complete.
    • Select Create custom task to delegate and click Next.
    • Select Only the following objects in the folder and then check Computer objects and Create selected objects in this folder. Click Next.
    • Under Permissions, check Create All Child Objects and Write All Properties. Click Next.
    • Click Finish

Increase the MachineAccountQuota value

The second step is to increase the quota from the default value of 10. This appears to be done domain-wide, so all users will get the new quota. I somehow doubt that will be a problem, but if it is, you will have to do further research on how to proceed. To increase the quota, we just need a single command entered in an Administrative PowerShell terminal.

Set-ADDomain -Replace @{"ms-ds-MachineAccountQuota"="10000"}

I used 10,000 because we have less than 100 nodes ready to join the domain. You can increase the value if your scale is a bit higher. I’m sure there’s a way to reset the quota, too, I just haven’t found it yet.

Joining your node to the domain

You’re now ready to join your node to the domain with your new least-privilege account domainjoin. I have created a puppet module, domain_join, to meet my personal needs. I’d love to hear how you’re tackling this issue, especially if the solution is better than mine!