Puppet Scale Up with Apache/Passenger

Welcome back! I hope everyone had a good summer and recharged their batteries. Bonus points if you found time to play with puppet, too! Now that we’ve had a healthy break, let’s get back to it!

When we left the series in July, we had a Puppet master, a few nodes, were implementing the roles and profiles pattern, and used r10k to manage it all. However, we didn’t address scalability. Today, we’ll take a look at addressing this by using Apache and Passenger.

Scaling Up

There are two ways to scale – out and up. If we were to scale out, we’d be concerned with running multiple masters and synchronizing all data between them. That’s something we might look at eventually, but first we want to scale up, which is the process of providing more resources to our master. Since we are vSphere admins, we can easily increase the resources provided to the VM. For instance, our VM has 1 vCPU and 2GB of RAM. It would be easy, and helpful, to increase that, perhaps to 2×4 or 4×8 vCPUxRAM.

Unfortunately, system resources are not the only limitation in our system. Out of the box, Puppet uses WEBrick and scales to about 10 nodes. More than one nodes trying to talk at the same time will generate conflicts and cause some or all nodes to fail to receive a catalog. No matter the resources available, these limitations persist. The answer is to use a dedicated web server with a Rack-based application stack. While any server will work, if you don’t have a preference, then PuppetLabs suggests you use Apache with the Passenger mod. There is a lot of information on Puppet’s site about the limitations and the remedy.

Continue reading

Summer Wrap-Up – #VirtualDesignMaster, #HyperV, and more

With summer coming to a close, there’s an unavoidable temptation to look book on what was. This year has been pretty busy, so I thought I’d take some time to look back on my accomplishments.

In February, I started this blog. I set out with a goal of one article per week on the blog. In the last seven months, I’ve met that average for the week even if I don’t write another article (52 articles in 29 weeks), but I also learned that it’s not feasible for me to actually write one article every week. I wrote three series, on PowerCLI, Auto Deploy, and Puppet for the vSphere Admin (ongoing) and managed over 14,000 page views in ~7 months. I have been lucky enough to convince Jason Crichton (@hawkbox) to contribute articles on Hyper-V while I took a summer break, and there’s really no point in trying to advertise new content during VMworld next week. I count the blog as a success and I learned some lessons along the way.

Speaking of which, I am unable to attend VMworld this year. Last year was my first VMworld and I sorely miss the opportunity to go back and to meet all my tweeps. I’ll be watching online, however!

In the area of personal growth, I acquired my VCP5-DCV certification in April and became a vExpert in July. Both were exciting, the certification because it validated my technical knowledge and the vExpert because it validated my community activities. The fact that both came with some free licenses didn’t hurt, either. I plan to put those to good use in the coming months to continue my education! I’m looking forward to being part of these two communities and learning from the other members.

In July, Season 2 of Virtual Design Master (VDM) began. I was one of 11 contestants. I survived round 1 and round 2, but I had to fold on round 3 of 4 due to time commitments. Congratulations to Timothy Patterson (@PcProfessionals) for winning Season 2! I learned a LOT during this competition – taking all the skills I use on a day to day basis as a vSphere admin, what I studied for my VCP, and a number of vSphere design books and applying it to design and engineering, which is not part of my day job in operations. I haven’t done any actual design work in years, and then it was always on a much smaller scale. VDM was challenging, exciting, and scarey. It was also very rewarding. In addition to the technical growth, there was a great sense of community and co-opetition between the contestants and the viewers of the competition. Check out #VirtualDesignMaster on Twitter for some great discussions. There’s some good content about IPv6 and Openstack in particular.

As I mentioned above, I was able to convince Jason to become a contributing editor on my blog. He is in the middle of his own Hyper-V series, which now has its own page. As a vSphere admin myself, I find the content very helpful – the world isn’t all VMware – and I hope that others find it useful as well. We would both love to hear any comments you have on the series and what direction it should go in.

In addition to all of the above, my wife changed jobs in August and we moved from Richmond, VA to Indianapolis, IN. We’re really excited about the new job and home, but we’re both very exhausted, too. If you’re in the Indianapolis area and I didn’t meet you at the IndyVMUG, drop me a line.

Have a great summer, and enjoy VMworld if you are attending!

Basic Sudo Access

A few months back I noted that I’d be more concerned with security going forward. Among other things, this means using sudo for accounting and authorization. A mature system would configure this via Puppet or another CM tool. If you haven’t gotten there with your CM tool, or you are working on a standalone system, you can still enable basic sudo access with a few steps:

[root@puppet git]# useradd rnelson0 -c "Rob Nelson"
[root@puppet git]# passwd rnelson0
Changing password for user rnelson0.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.

[root@puppet ~]# cat > /etc/sudoers.d/sudoadmins
rnelson0        ALL=(ALL)       ALL
  1. Add a local user.
  2. Set a password for the local user.
  3. Allow the user to use sudo for all commands.

This is a good start for a lab. In production, more granular sudo permissions should be used, in addition to a central user management system, like LDAP or ADS. And if you have to do this in more than one place, automation is a good idea.

Introducing Jason Crichton, aka @hawkbox

With summer upon us, I’ve taken a break from the blog. You’ll still see a few of my small posts pop out every so often, but no lengthy technical posts from me for a while. That doesn’t mean the blog is taking the summer off, though!

I’m proud to introduce a colleague and fellow Arsian, Jason Crichton, as a contributing author on my blog! Jason is going to write some articles over the summer about Hyper-V. For those of us (myself included!) who are only familiar with vSphere, Jason will help us compare the analogous features from each product with articles most Wednesdays this summer. Here’s a little background those of you who have not met Jason before:

My name is Jason Crichton, I’m an IT professional like Rob, just crossing over the 10 year mark of System Administration this summer. I started in the trenches of help desk and through a bit of luck and a lot of hard work now work as a Senior Systems Analyst for a relatively small multinational corporation. I tend to end up heavily involved in the virtualization, security, and operations aspects of the business. Recently I have moved into Powershell tool development for improving the lives of our help desk staff. I find the willingness of people like Rob to put the time and energy into sites like this incredibly valuable, so when he asked me to contribute I was thrilled at the opportunity to give back myself.
When I’m not working with tech, I tend to be motorbiking with my wife Christina on whatever random trip we’ve been able to organize.
My professional experience can be viewed on LinkedIn. Additionally you can follow me on twitter through @hawkbox.
Please give Jason a warm welcome to the blogosphere! If you have any requests for Hyper-V topics, please let myself or Jason know what you’d like to see covered. Thanks!

Saving the moon, #VirtualDesignMaster style

This week was another nail biter in the Virtual Design Master competition. Challenge 2 required us to save the moon while using someone else’s design plus a few constraints: must fit in 21U, have to use the same vendors (but can use different product lines) as the provided design used, and the big one, the moon base only has IPv6 networking. I understand IPv6 but certainly haven’t designed an IPv6-only network, so this was pretty scary and very time consuming for the research.

There were a lot of great designs presented by the VDM competitors. Three of us had to work off of Daemon Behr’s and six of us had to work off of my design from the previous challenge. It was fun to see how other people managed the same base project and morphed it into a project that had their fingerprints on it. Watch the results show and check out the designs (here’s mine). During the design and the judging, I learned a few things in no particular order:
Continue reading

Hiera, R10K, and the end of manifests as we know them

Last week, we started using Hiera. We’re going to do a lot more today. First, we’ll add what we have to version control, then we’ll integrated it with r10k, and we’ll wrap up by migrating more content out of manifests into Hiera. Along the way we’ll explain how Hiera works. I also encourage you to review the Puppet Labs docs and the source code as needed. Continue reading

I Survived #VirtualDesignMaster Challenge 1!

This week has been pretty exciting. It’s getting closer to the move and things are starting to seem real – which means more time is involved in it. Somehow, in the midst of all that, I managed to complete my design proposal for Virtual Design Master’s first challenge, a whopping 30 minutes before the due date. On Thursday night, all the contestants defended their design. To my surprise, I survived! I am thankful of some critical reviews from Jason Shiplett and some friends on IRC. We lost a few competitors, as is the nature of the challenge, but everyone’s designs are amazing. Check them out at http://www.virtualdesignmaster.com/.

This week’s challenge is about constraints. We have some physical constraints – have to use the same vendors, and needs to fit in 21U, oh and by the way, it’s on the moon – plus a unique requirement I haven’t seen anywhere else: IPv6 only. That’s going to be tough. But they weren’t done with the constraints yet. We have to use someone else’s design from challenge 1! Everyone on Team Beta has to work off the design by Daemon Behr (@VMUG_Vancouver). I’m very honored that my design (@rnelson0) was chosen as the design that Team Alpha has to work from.

If you are available next Thursday at 9PM Eastern, tune in at http://www.virtualdesignmaster.com/live/ to see the results of challenge 2!

Intro to Roles and Profiles with Puppet and Hiera

If you’ve been following along with the Puppet series, our next task is to start using roles and profiles. If you’re just visiting, feel free to review the series to get caught up. Today, we will discuss the roles and profiles pattern, start implementing it as well as a custom fact, and deploy a webserver on a node managed by puppet. Finally, we’ll move some of our configuration from the site manifest into Hiera.

NOTE: A small note on security. I’ve been running through this series as ‘root’ and earlier said, “Well, just be more secure in production.” That’s lame. This blog covers security as well as virtualization and automation so I’m going to live up to that. For now, I’ve added a local user with useradd, updated sudoers, and cloned all the repos so that I can show best practices, which will include doing most work as my user and then sudo/su to run a few commands as root. Later, we’ll manage local users via puppet.

Continue reading

#vExpert, #VirtualDesignMaster, and other Stuff

My summer has been exciting. On Wednesday, I received notification that I was accepted as a vExpert for 2014! That’s pretty awesome, both as confirmation that hard work has payed off and encouragement to keep it up in the future. On Thursday night, the Virtual Design Master competition kicked off. This will hopefully keep me busy throughout the summer. I haven’t even gotten started on it, though, as I am on-call this week and things blew up right after the live start. Here’s hoping it settles down so I can work this weekend!

As if that wasn’t busy enough, my wife accepted a new job in June with a start date in August. We’ll be moving in support of that around the end of the month. With that in mind, I’m taking a summer break from the blog (but certainly not a vacation!). I have a few scheduled articles that will take me through the end of July and I’m hoping to have a guest author to cover August until VMworld. I plan to get back to blogging in early September.

Until then, here is a mix of the most popular articles and the ones I really enjoyed writing.

  • Puppet – There are two more articles to complete the intro portion. Now that you’re familiar with Puppet, we’ll look at closer integration with vSphere in the Fall.
  • Auto Deploy Deep Dive – I was hoping to present this at VMworld but it wasn’t meant to be! Check out the #vBrownBag presentation, too. They’re in the middle of a Cisco track and will be covering Docker on 7/23, good stuff.
  • The Philosophy of Ender’s Game – Now that the movie’s out on DVD and cable, it’s a good time to watch it again and do some critical analysis. Preferably on your tablet while piloting a quadcopter drone, both ideas that can be traced back to this novel. This wasn’t very popular, but it was one of my favorite articles to write. It’s always fun to wax philosophical.
  • Snapshots Management – Surprisingly, this recent article seems very popular. I shouldn’t be surprised, snapshots continue to be a wildly misunderstood tool that cause problems for even veteran vSphere admins.
  • InfoSec and Social Media – This article was a result of attending CPX 2014 and comparing it to VMworld 2013. I had fun writing it, soliciting feedback, and working to do the things I said I would.
  • Synology Multi-VLAN Setup – This remains a very popular article. I hope Synology makes VLAN configuration a little smoother in future DSM revisions, but until then, this will get you going.

Have a great summer!

Snapshots and Automated Emails

A common problem in virtualization is snapshots. The name “snapshot” makes us (novice or otherwise!) think of a picture in time, which sometimes leads to the belief that the snapshot is “taken” and then stored somewhere, though that’s not how snapshots really work.

In reality, snapshots create a psuedo-consistent state of the virtual disk at that point in time. Subsequent writes in a snapshotted state are redirected to delta files. If you are performing an upgrade, a snapshot is helpful, allowing you to restore the prior system state if there are problems. After a few days, the snapshot loses its value as a restore becomes increasingly unlikely because you would lose the application changes as well. Snapshots also play a role in backups, where they are used temporarily to provide the psuedo-consistent state for the backup utility before the snapshot is deleted.

When a snapshot is deleted, that delta is applied to the base virtual disk(s), playing back through the transactions. Large snapshots take a long time to delete and affect system performance until the consolidation is complete. They can also affect the VM during normal operation as the delta file size increases.
Continue reading