I Survived #VirtualDesignMaster Challenge 1!

This week has been pretty exciting. It’s getting closer to the move and things are starting to seem real – which means more time is involved in it. Somehow, in the midst of all that, I managed to complete my design proposal for Virtual Design Master’s first challenge, a whopping 30 minutes before the due date. On Thursday night, all the contestants defended their design. To my surprise, I survived! I am thankful of some critical reviews from Jason Shiplett and some friends on IRC. We lost a few competitors, as is the nature of the challenge, but everyone’s designs are amazing. Check them out at http://www.virtualdesignmaster.com/.

This week’s challenge is about constraints. We have some physical constraints – have to use the same vendors, and needs to fit in 21U, oh and by the way, it’s on the moon – plus a unique requirement I haven’t seen anywhere else: IPv6 only. That’s going to be tough. But they weren’t done with the constraints yet. We have to use someone else’s design from challenge 1! Everyone on Team Beta has to work off the design by Daemon Behr (@VMUG_Vancouver). I’m very honored that my design (@rnelson0) was chosen as the design that Team Alpha has to work from.

If you are available next Thursday at 9PM Eastern, tune in at http://www.virtualdesignmaster.com/live/ to see the results of challenge 2!

Intro to Roles and Profiles with Puppet and Hiera

If you’ve been following along with the Puppet series, our next task is to start using roles and profiles. If you’re just visiting, feel free to review the series to get caught up. Today, we will discuss the roles and profiles pattern, start implementing it as well as a custom fact, and deploy a webserver on a node managed by puppet. Finally, we’ll move some of our configuration from the site manifest into Hiera.

NOTE: A small note on security. I’ve been running through this series as ‘root’ and earlier said, “Well, just be more secure in production.” That’s lame. This blog covers security as well as virtualization and automation so I’m going to live up to that. For now, I’ve added a local user with useradd, updated sudoers, and cloned all the repos so that I can show best practices, which will include doing most work as my user and then sudo/su to run a few commands as root. Later, we’ll manage local users via puppet.

Continue reading

#vExpert, #VirtualDesignMaster, and other Stuff

My summer has been exciting. On Wednesday, I received notification that I was accepted as a vExpert for 2014! That’s pretty awesome, both as confirmation that hard work has payed off and encouragement to keep it up in the future. On Thursday night, the Virtual Design Master competition kicked off. This will hopefully keep me busy throughout the summer. I haven’t even gotten started on it, though, as I am on-call this week and things blew up right after the live start. Here’s hoping it settles down so I can work this weekend!

As if that wasn’t busy enough, my wife accepted a new job in June with a start date in August. We’ll be moving in support of that around the end of the month. With that in mind, I’m taking a summer break from the blog (but certainly not a vacation!). I have a few scheduled articles that will take me through the end of July and I’m hoping to have a guest author to cover August until VMworld. I plan to get back to blogging in early September.

Until then, here is a mix of the most popular articles and the ones I really enjoyed writing.

  • Puppet – There are two more articles to complete the intro portion. Now that you’re familiar with Puppet, we’ll look at closer integration with vSphere in the Fall.
  • Auto Deploy Deep Dive – I was hoping to present this at VMworld but it wasn’t meant to be! Check out the #vBrownBag presentation, too. They’re in the middle of a Cisco track and will be covering Docker on 7/23, good stuff.
  • The Philosophy of Ender’s Game – Now that the movie’s out on DVD and cable, it’s a good time to watch it again and do some critical analysis. Preferably on your tablet while piloting a quadcopter drone, both ideas that can be traced back to this novel. This wasn’t very popular, but it was one of my favorite articles to write. It’s always fun to wax philosophical.
  • Snapshots Management – Surprisingly, this recent article seems very popular. I shouldn’t be surprised, snapshots continue to be a wildly misunderstood tool that cause problems for even veteran vSphere admins.
  • InfoSec and Social Media – This article was a result of attending CPX 2014 and comparing it to VMworld 2013. I had fun writing it, soliciting feedback, and working to do the things I said I would.
  • Synology Multi-VLAN Setup – This remains a very popular article. I hope Synology makes VLAN configuration a little smoother in future DSM revisions, but until then, this will get you going.

Have a great summer!

Snapshots and Automated Emails

A common problem in virtualization is snapshots. The name “snapshot” makes us (novice or otherwise!) think of a picture in time, which sometimes leads to the belief that the snapshot is “taken” and then stored somewhere, though that’s not how snapshots really work.

In reality, snapshots create a psuedo-consistent state of the virtual disk at that point in time. Subsequent writes in a snapshotted state are redirected to delta files. If you are performing an upgrade, a snapshot is helpful, allowing you to restore the prior system state if there are problems. After a few days, the snapshot loses its value as a restore becomes increasingly unlikely because you would lose the application changes as well. Snapshots also play a role in backups, where they are used temporarily to provide the psuedo-consistent state for the backup utility before the snapshot is deleted.

When a snapshot is deleted, that delta is applied to the base virtual disk(s), playing back through the transactions. Large snapshots take a long time to delete and affect system performance until the consolidation is complete. They can also affect the VM during normal operation as the delta file size increases.
Continue reading

Puppet Installables – MCollective

In our ongoing Puppet series, we just completed installing PuppetDB and Hiera. There’s one other installable that’s a bit more complicated than those two.

MCollective

Since writing this article, the module puppetlabs/mcollective has been updated. In particular, the “middleware => true” parameter is not valid in the base class. I have not had time to revisit this and update the solution. If you have, please drop me a line in the comments!

The last component we’re going to install today is MCollective. While developed by Puppet Labs, MCollective isn’t directly related to Puppet, as PuppetDB and Hiera are. It’s not a Configuration Management tool, it’s an Orchestration API. It does integrate quite well with Puppet and Facter, among other sources. Some things you can do with MCollective might be to query how many systems have 32GB of RAM, how many systems are running a version of OpenSSL vulnerable to Heartbleed, or to restart Apache on all servers in the Development environment. This installation is trickier than either PuppetDB or Hiera.
Continue reading

What is a systems administrator?

I’ve seen a few topics recently where people seem to misunderstand what a systems administrator is. The cause for this is likely due to a combination of factors including HR job classifications, some misunderstandings, and also a simple lack of other terms to use. Here’s my definition:

Systems Administration requires that you administer a group of interconnected objects, i.e. a system. One the important components of a modern system is the network, something that actual systems administrators know about. This does not mean that they are experts on each system component, but they are familiar with the components and can perform basic and some intermediate troubleshooting without requiring assistance.

In many cases, it appears that what people are talking about is actually a server administrator, maybe even just a computer operator. If we had an IT union, the only thing I’d want from them is to prevent people (and HR!) from misrepresenting their actual responsibilities.

Is that a good definition of systems administration, and are there better terms we can use? What do you think?

Puppet Installables – PuppetDB and Hiera

Welcome back to our Puppet for vSphere Admins series. We started out deploying the puppet master and a few example manifests, then took a right turn into the land of Git and workflows. I know you’re anxious to get back to developing manifests, but we’ve got a few more things to install on the master before we worry about the manifests. PuppetDB and Hiera, and MCollective are powerful tools that most administrators will find of immense benefit. We can install these later, but we’d have to redo some of our work. Who wants to do that?

As I mentioned last week, I’ll assume you’re using r10k at some level, so I’ll mostly just reference “use r10k” unless there’s a specific gotcha. If you’re not using r10k, follow whatever workflow you’ve decided on to add modules, update manifests, and track all changes in your VCS.

Continue reading

Puppet and Git, 206: Git Hooks – Post-Receive

Welcome to our last class on Git usage. Now that we have a pre-commit hook in place, we’ll finish things up with a post-receive hook.

UPDATE: On 9/17/2014, Phil released Reaktor, an early version of which much of this article is based on. I haven’t had time to investigate but it should be easier to install and far more functional!

Post-Receive Hook

I don’t know about you, but I’m already tired of having to run r10k manually. Having to ssh to the master, log in, and run commands is so droll. What can we do about it?

A post-receive event fires when a remote push is received by the repo with the hook (i.e. when I ‘git push origin branch’, the ‘origin’ server will fire the hook) and we can use a corresponding post-receive hook to deploy for us. This is slightly trickier than a pre-commit hook because of where the event is firing. We don’t want to run it on our desktop/workstation VM, because that host would be the origin repo for everyone. We can’t want run it on the puppet master, because then our puppet master is the origin. (Technically, each repository is a fully sufficient origin repository on its own, but I’m making an assumption that you have a designated origin that’s backed up and therefore you won’t be doing the same for the other repo clones.) That leaves GitHub, which is already our designated origin. Because the post-receive event will fire on the origin, we need to ensure that Github can talk to the puppet master, which is where r10k is located.

There’s no one way to do this. My design is to implement a GitHub Web Hook, which will result in Github sending a POST to a target URL (which means GitHub needs to be able to communicate with it!) that says, “Someone just committed a change, here are the details of the change.” There’s more detail on enabling the Web Hook here. If you’re using Atlassian Stash, as we do at work, hopefully the admins have installed a plugin for web hooks (ironically, I started this documentation in March and I’m still waiting on a web hook to be installed – but if I used Github…).
Continue reading

Puppet and Git, 205: Git Hooks – Pre-Commit

Welcome to the Puppet and Git, class 205: Git Hooks. Since we finished up talking about workflows, let’s move on and explore what a git hook is and how you can use one to improve your workflow.

Git Hooks

What is a git hook? You can read some boring official documentation, but who does that? Instead here’s a short summary: A git hook is a program that is called when a git events are triggered. These programs are usually simple shell scripts and some common events people use them with are commits. We’ll look at commits and the event that fires before the commit is completed.

Pre-Commit Hook

If you’ve been programming for more than, say, an hour, you’ve undoubtedly experienced the bad mojo that results from missing a semi-colon or curly brace or other piece of syntactical junk. And you may have even committed such a piece of broken code and pushed it upstream just to watch the whole thing fall apart. If we can lint our code, we can determine whether the code meets the syntactic requirements of a language. It’s important to note that linting doesn’t verify that your code does what it says it will do, it JUST verifies that the code will parse or compile. This would prevent the wonderful pattern of commits that looks something like:
Continue reading

Puppet and Git, 204: r10k Workflow for Existing Module

We’ve installed and configured r10k, are using it for deployments, and have a workflow for new modules. More commonly, we will be working on existing modules, a slightly different workflow. We can examine this new workflow by modifying the base module only.

Workflow to Modify Existing Module

Unlike adding a new module, the Puppetfile only needs to be modified to reflect the feature branch. This is where the workflows diverge: instead of requiring a merge, commit, and push, we can create a temporary branch and just delete it when we’re done. Only the module branch needs merged. We’ll show this by making a simple change, modifying Dave’s name. We have another Dave Smith who works here, so we’ll add a middle initial and the name of Dave’s organization, to prevent confusion.

The feature branch is just called dave. We’ll work on the module repo first. Make sure you’re in master and checkout the new branch. Dave’s description should be updated to “Dave G. Smith – IT Administrator” – everything will be alright unless they hire another Dave G. Smith over there. Commit the change and push it.
Continue reading