vRealize Orchestrator Workflows for Puppet Enterprise

Over the past three years, my Puppet for vSphere Admins series has meandered through a number of topics, mostly involved on the Puppet side and somewhat light on the vSphere side. That changed a bit with my article Make the Puppet vRealize Automation plugin work with vRealize Orchestrator, describing how to use the plugin’s built-in workflows to perform some actions on your VMs. However, you had to invoke the workflows one by one, and they only worked on existing VMs. That is not good enough for automation! Today, we will start to look at how to integrate the Puppet Enterprise plugin into other workflows to provide end-to-end lifecycle management for your VMs.

What is the lifecycle of a VM? This can vary quite a bit, so the lifecycle we will work with today is made to be generic enough for everyone to use, but flexible enough that everyone can expand on it. It consists of:

  • Provisioning
    • Updating ancillary systems prior to VM creation (IPAM, DNS, etc)
    • Deploying a Virtual Machine
    • Installing Puppet Enterprise on the VM
    • Using Puppet Enterprise to provision services on and configure the VM
    • Add the new VM to a vCenter tag-based backup system
  • Decommission
    • Delete the VM (removes from backups)
    • Purge the record from PE
    • Update ancillary systems after VM removal (IPAM, DNS, etc)

Continue reading

Automating Puppet tests with a Jenkins Job, version 1.1

Today, let’s build on version 1.0 of our Jenkins job. We are running builds against every commit, but when someone opens a pull request, they don’t get automated builds or feedback. If the PR submitter even knows about Jenkins, and has network access and a login, they can look at it to find out how the tests went, but most people aren’t going to have that visibility (especially if your Jenkins server is private, as in this example setup). We need to make sure Jenkins is aware of the pull request and that it updates the PR with the status. Our end goal is for each PR to start a Jenkins build and update the PR with a successful check when done:

To get there, we will install and configure a new plugin and configure our job to use the plugin.

Continue reading

Automating Puppet tests with a Jenkins Job, version 1.0

As I’ve worked through setting up Jenkins and Puppet (and remembering my password!), I created a job to automate rspec tests on my puppet controlrepo. I am sure I will go through many iterations of this as I learn more, so we’ll just call this version 1.0. The goal is that when I push a branch to my controlrepo on GitHub, Jenkins automagically runs the test. Today, we will ensure that Jenkins is notified of activity on a GitHub repo, that it spins up a clean test environment without any left over files that may inadvertently assist, and run the tests. What it will NOT do is notify anyone – it won’t work off a Pull Request and provide feedback like Travis CI does, for instance. Hopefully, I will figure that out soon.

The example below is using GitHub. You can certainly make this work with BitBucket, GitLab, Mercurial, and tons of other source control systems and platforms, but you might need some additional Jenkins Plugins. It should be pretty apparent where to change Git/GitHub to the system/platform you chose.

Creating A Job

From the main view of your Jenkins instance, click New Item. Call it whatever you want, choose Freestyle project as the type, and click OK. The next page is going to be where we set up all the parameters for the job. There are tabs across the top AND you can scroll down; you’ll see the same selection items either way. Going from the top to the bottom, the settings that we want:

Continue reading

Jenkins Tricks – Password Recovery and Job Exports

I’m finally getting back to Jenkins, which I started waaaay back in November (here and here). Unfortunately, I kind of forgot my password. Well, that’s embarrassing! I also want to redo the manifest using maestrodev/rvm which means starting over, so I need to back things up. The manual for Jenkins and the results on Google can be overwhelming sometimes, so I thought I’d share what I learned to hopefully save someone else.

Password Recovery

There’s a few ways I found to recover your password. One suggestion is to disable all security, delete your user, re-enable security and allow signups, and then recreate the same user and things should just “work”. Part of the reason you have to do this is that once you disable security, you can’t change the password for your user; only the user can. That’s … frustrating.

Disable security by editing $JENKINS_HOME/config.xml, /var/lib/jenkins/config.xml on my instance. I was able to get away with disabling it by changing <useSecurity>true</useSecurity> to false, though the article suggests removing two other lines. Restart the service with systemctl restart jenkins or equivalent and now you’re able to get in and recreate some users.

Continue reading

Scheduling regular Travis CI builds with Cron Jobs

If you have a project of any complexity that is using Travis CI for testing, you’ve inevitably run into an issue where you make a minor change – maybe even just to a markdown file – and the tests that were passing before completely bomb. If this hasn’t happened to you, just wait, it will! This typically happens because some dependency, such as a newer rubygem you depend on indirectly, is no longer compatible with your test environment. As an example, I updated my .travis.yml in a puppet module project, opened PR16, and suddenly all the previously-green tests went red. The error NoMethodError: undefined method 'last_comment' for #<Rake::Application:0x000000015849f8> was observed because the dependent gem rake was updated from v11 to v12, conflicting with the pinned version of rspec. Until this problem is fixed, every PR, no matter how simple or complex, is going to fail its test.

This puts a lot of burden on both the person submitting the PR, who just wants their simple change to be approved, and the project maintainer, who wants their build status to work. Neither is satisfied until the problem is tracked down and remediated, often in a separate PR. By that time, the contributor may not have the interest to update their PR and the maintainer has to decide if they want to fix it up or discard it. No-one is happy. All because some upstream dependency changed and we weren’t aware of it until the moment a PR was submitted, even though it may have happened days or months ago.

Continue reading

Installing Jenkins and RVM

Update: Contrary to the module readme as of 12/1/2016, this WILL result in a VM running Jenkins 2, rather than version 1.

It’s time to get started installing Jenkins for use with Puppet. I’ll be installing this in my home lab on a vm called jenkins (so inventive, I know) as an all-in-one server for simplicity. I don’t know Jenkins real well so I don’t know what I should be doing for masters and build servers, but my home lab needs are small anyway. I gave it 2 cores and 1 GB of RAM, which I think is on the low end for Jenkins. My initial exploration led me to believe I’ll need to install Jenkins as well as RVM, since my base OS image is CentOS 7 (Ruby 2.0.0) and the current Puppet 4 AIO build is against Ruby 2.1. I’m using Puppet and the rtyler/jenkins module to set up Jenkins, since I don’t know anything about its innards. Down below, you’ll see that I installed RVM by hand, after which it occurred to me that there’s a perfectly cromulent maestrodev/rvm module that I’ve used in the past – something I may change out later, but I already wrote my manifest so I’m going to share it anyway!

I used Jenkins jobs to experiment a lot with this setup, such as enabling RVM on a job and seeing what errors showed up in the log, but I’m going to hold off on showing that magic until the next article. I will still explain where portions of the manifest came from, just without the actual errors.

Before we go any further, make sure you have the jenkins module and its dependencies added to your Puppetfile, .fixtures.yml, and wherever else you might need to track it.

Continue reading

Getting started with Jenkins and Puppet

If you’ve followed my blog, you’ve seen that I enjoy using Travis CI to run my puppet rspec tests on the controlrepo or against component modules. When you create a PR, Travis starts a build and adds some links to the PR notes to the builds. When it’s complete, the PR is updated to let you know whether the build was successful or not. You can also configure your repo to prevent merges unless the build succeeds. The best part is that it “just works” – you never have to worry about upgrading Travis or patching it or really anything other than making sure you enabled it. It’s a pretty awesome system, especially since it is absolutely free for open source projects. I do love Travis!

But, Travis isn’t always available and isn’t always the answer. Travis only runs against PRs or (typically) merges into the default branch. It won’t run on a schedule, even if your code never changed, which can help test any dynamic dependencies you might have (whether you should have any is a different topic!). It runs on a limited subset of OSes that Travis supports. You have to use Github and public repos to hit the free tier – if your controlrepo is private, there’s a different travis site to use and there is no free plan, though there is a limited number of trial builds to get you started. After that, it can be pretty pricey, starting at 1 concurrent builds for $69/month. This is great for a business, but most of us can’t afford $840 a year for the home network. It’s also, by itself, somewhat limited in how you can integrate it. It can be part of a pipeline, but it just receives a change and sends a status back to Github, it won’t notify another system itself. You have to build that yourself.

There are a ton of other Continuous Integration and Continuous Delivery systems out there, though. They can be cheaper, have better integrations, run on your own hardware/cloud, and don’t have to require Github. Among the myriad options available, I chose to look at Jenkins, with the rtyler/jenkins puppet module. I can run this on my own VM in my home lab (hardware/cloud), it can integrate with GitHub private repos ($0) or BitBucket (doesn’t require Github). CloudBees also discussed a new Puppet Enterprise Pipeline plugin at PuppetConf 2016, which is really appealing to me (better integrations). I can also have it run builds on a schedule, so if dependencies change out from underneath me, I’m alerted when that happens, not when I run my first test a month after the underlying change.

I’m very new to Jenkins and I’m going to do a lot of things wrong while testing it, but I’m going to start blogging about my experiences as I go, right or wrong, and then try and do a wrapup article once I learn more and have a fully working solution. I’ve been inspired to do this by Julia Evans and her wonderful technical articles that are more about the exploration of technology. That goes against my normal style of figuring it all out first, but I so love to see the exploration of others and hope you will, too! As we go on this journey, please feel free to educate or correct me in the comments and on twitter, as always.