Why not Puppet?

Alternatively: Common mistakes made when adopting Puppet.

I love me some Puppet, and anyone who knows me will tell you I’ll talk about it and configuration management as long as you let me. However, sometimes it’s not the answer people expect it to be. Is it even the right tool? As a counterpoint to Why Puppet?, let’s look at some potential use cases and see whether they are a good fit. These use cases have been gathered from my own usage, ask.puppetlabs.com, #puppet on IRC, and some user stories recounted to me and are presented in no specific order. Special thanks to Ryan McKern for some additional stories and editing.

Is it possible to run something only if the file/user/package/whatever is present? (IRC, nearly every day)

The situation is often presented as, “$Thing won’t install without me answering some questions or providing an answer file, can I get Puppet to manage it only if the package is installed?” Yes, but also no.

Continue reading

On Karōjisatsu And Avoiding Burnout

Recently, John Willis (@botchagalupe) wrote an excellent article about Karōjisatsu, one who commits suicide due to mental stress, often work-related. It’s a very sad, emotional tale that is relevant in many industries, but one that speaks particularly to high-pressure, high-stress STEM jobs, including IT. If you have not read this article, please take a few moments to go read it now.

The core idea of nearly overwhelming burnout is probably one that you recognize. John’s article spoke very eloquently on the need to reach out if you feel overwhelmed, that you’re not alone, that there are many people who are willing to help you, and that suicide is not an option. I would like to add that if I can ever be of any assistance to anyone reading this, don’t hesitate to reach out. If you ever feel truly overwhelmed, reach out to the National Suicide Hotline at (800) 273-8255 as well. You do matter!

John describes some causes of Karōshi, including, “Stress accumulated due to frustration at not being able to achieve the goals set by the company.” There is always pressure to do more with less and in IT, we tend to feel this pressure very heavily. Systems and their associated problems always seem to come and rarely to go, giving even stable, growth-restricted companies an increasing IT burden. Every day, there is an increasing amount of systems knowledge – often of the tribal and oral history varieties – for each of us to remember and maintain. When things go wrong – and they always do – we have to drop what we are doing to put out the fires, delaying our schedule and often without the ability to adjust the delivery dates on the schedule. We often feel that we must work harder and longer to make up for these delays and maintain the schedule in order to hit the company’s goals. The mental and physical stress of something going wrong combined with the mental and physical stress of working harder and longer accumulates in a vicious cycle that must be broken before it leads to karōshi.

I know this feeling. I have found myself looking at the clock near midnight, telling myself that I’ll put the computer down in 10 minutes and go to bed, only to blink and the clock reads 3AM. I have gotten up early on a Sunday to fix something broken that I could not get to on Friday. I’ve even found myself getting up early to “fix” something that’s not broken! The pressure of needing to resolve an issue, ship a product, or address a customer’s question keeps my brain running at night when it should be resting and recuperating so that I can do good work the next day. Sometimes it’s not even a company goal that keeps me working on an issue, just my stubborn pride. Whatever the cause, I know the feeling of overwhelming pressure that affects all of us from time to time.

Burnout of any sort, whether it puts you on the edge of suicide or the edge of your career, is dangerous. We must all develop coping strategies to deal with these feelings. I have been fortunate to have some wonderful mentors in my career. I credit my first two bosses for giving me two great coping strategies to deal with this pressure, and I would like to share those strategies with you.

The first coping strategy is courtesy of Bob at Centerline, my first “real world” job. We were the IT Operations staff at an engineering firm. His advice was simple: “Sometimes, you let it burn.” It’s very easy to hear users scream and think that world really is ending. What the users are saying is important, but we must evaluate what we hear carefully and prioritize accordingly. Are we reacting because a single person is struggling with an issue or because the company is negatively affected by a problem more than they are positively affected by whatever you are currently doing? If you’re off shift when the issue occurs, must it really be taken care of immediately by you, or can it wait or be handled by someone else? Most of us have been taught repeatedly that the answer is always, “Fix it now!” but is that truly the case?

When issues have a low severity or affect a low number of users, particularly if you’re treating symptoms and not causes, let them “burn”. While things are burning, put your effort toward fixing the underlying causes in order to prevent future fires. You will often find that your environment is not as flammable as everyone thought and that a little fire and smoke won’t destroy the company. It’s still hot, and it still hurts, but it is a different kind of hurt. This is an especially great way to deal with chronic issues. Rather than dropping everything for, say, a single user who complains about a broken report that they need RIGHT NOW, fix the underlying bug in the reporting system. If you can pick just one “burn day” a month and spend that time on underlying causes, you will find yourself in a much better position in a few months. If you can do it more frequently, or cherry-pick some chronic issues to let burn, you may see results in just a few weeks.

Regardless of the frequency with which you have burn days, you’ll notice one thing very quickly: your stress levels will go down. When you do encounter a chronic issue that you cannot let burn, you know that someday soon you will be able to make that issue go away forever. Your time will be freed up to work on improvements and innovation rather than just outages, lowering the pressure put upon you and enabling you to meet the company’s goals.

The second coping strategy was taught to me by Scott from RBA Systems. This is a consulting firm where we provided both development and operations to our customers. I was a 21 year old kid who just dropped out of college and was out to prove myself in IT. In my first few weeks, Scott often had to tell me, “pace yourself.” I wish I could say I thought nothing of it, but as the young smartass I was, I thought it was something a jaded old guy would say. I’m tough and there’s no way I’ll let him slow me down! Instead, in just a few months, the blistering pace I had coming out the gate had to falter and Scott, who also had to manage a few other people at the same time, started lapping me.

There’s simply no way you can keep up a lightning pace forever. Going 110% seems great until your body and mind start to fall apart due to the constant pressure they are under. Even going 100% cannot be maintained. You might find yourself flagging at the end of the day or your typing rate going to shit or constantly typing the wrong commands in the wrong windows. This is especially dangerous with ‘reboot’, ‘write erase’, or ‘rm’ style commands! None of these actions help you, your company, or your customers. Find out what your 100% looks like, pull back a bit from it until you find your pace you can maintain that balances speed, efficiency, and accuracy. Keep adjusting that pace over time as your skills improve and your work/life demands shift to maintain the balance. You may be making adjustments every day, and that’s okay – no-one’s perfect.

I credit my ability to successfully maintain a high level of performance and avoid burnout in IT over the past fifteen years to the valuable lessons from my early mentors, burn days and pacing myself. I hope these tools can help others with this ongoing struggle.

Do these things because you have pride in your work, because you want to be able to continue contributing to IT for decades, because they’re the right things to do. Do it because you matter. Do it because you love life.

Updated vSphere Upgrade Order

On March 12, 2015, VMware released vSphere 6 for General Availability. I thought it would be a good time to recap, and pretty up, my older upgrade post. The previous post was based on vSphere 5.5 and the specifics of the software upgrades have changed, but the general order has not.

vSphere Upgrade OrderCheck Compatibility

Read all of the Hardware Compatibility List, Interoperability Matrix, and other similar documents for all your components. Make sure all hardware is supported with ESXi 6 and that all software solutions support both ESXi and vCenter 6. Repeat with all other vSphere suite components such as SRM and the vRealize products. Contact vendors of incompatible solutions and find out when their v6 support is expected. If anything in your list does NOT support the latest version, make a decision on whether to remove or replace the components or to halt the upgrade until they are available.

Continue reading

February #IndyVMUG LiveTweeting / Future Meetings

IndyVMUG held their February meeting last week and I live tweeted most of it. You can check out the storify here.

I did not capture the dates for the next IndyVMUG meetings. If you’re close to Indianapolis on any of these dates, you owe it to yourself to try and make it here. We have great leadership and sponsors who make sure that every event is hosted someplace nice and has great content. The user conference in July will bring in over 1000 people, so if there’s one event to make, that’s the one. There should also be another 1-2 monthly meetings in April, May, and June, but dates aren’t set yet. Follow @IndyVMUG to keep up!

  • March 17, 2015 – Agenda includes PernixData and DataGravity, plus some undisclosed March Madness fun.
  • July 22, 2015 – A VCDX workshop hosted by Chris Colotti. Vital if you’re working toward your VCDX.
  • July 23, 2015 – Yearly User Conference. Check out the agenda and sign up when it becomes available.

Deploying MySQL with Puppet, without disabling SELinux

I’ve been vocal in the past about the need to not disable SELinux. Very vocal. However, SELinux can be difficult to work with. I was reminded of how difficult while deploying MySQL recently. Let’s take a look at how to iron out the SELinux configuration for MySQL, and how to deploy it with Puppet I will be using CentOS 6.6 in this article. The package names and SELinux information may vary if you use another distribution.

MySQL Design

Let’s review the design of the MySQL installation before continuing. For the most part, it’s a standard install, we’re not doing any elaborate tuning or anything. All the passwords will be ‘password’ (clearly you should change this in production!). All the anonymous users (@localhost, root@localhost, etc.) will have a password set. An additional ‘replication’ user is created so multiple databases can be replicated and example replication settings are included. The test databases are removed and a single user/database pair of wikiuser/wikidb will be created. We won’t do anything with the database, it’s just an example that can be duplicated as needed.

Continue reading

Rob Nelson’s 2015 Goals

On January 19th, I graded my efforts in 2014 and promised to document my technical goals for this year.

Learn Ruby

I spend a lot of time with Puppet and R10k and while I’m a decent enough programmer to pick up on what’s going on in general, I really need to grok Ruby at an advanced level. I bought a copy of Eloquent Ruby for the theoretical and I’ve eyeballed a few projects to work on for the practical.

Blog more about Security

One of my primary functions at work is related to security, but I rarely blog about it. That needs to change.

Home Network

1) After moving, I have my servers in a temporary space. Complete the finished space and migrate all the hardware there. Parts are on order and I’m going to add to the breaker box soon!

2) Stand up a complete environment. I’m missing vCO, an IPAM solution, and monitoring. Yes, it’s just my home network, but if I can’t monitor a handful of devices, I can’t monitor hundreds or thousands.

Expand Puppetinabox

Two weeks ago I released my first solo software project, puppetinabox. I’ve laid out some enhancements for it and I’m sure there are deficiencies I’m not aware of. I need to practice some good software development patterns both for myself, and if I want it to be used by others.

Propose a VMworld Talk

Last year, I proposed an Auto Deploy talk. It was not accepted. I won’t make it a goal to have an accepted talk – I have no control over it, after all – but I’d like to put a proposal in again. Topic undecided.


Last year, I obtained by VCP5-DCV. This year, I need to at least get my VCAP-DCA. That means a new study guide, some time in the lab, and a plan to prep, study, take, and pass the exam.


These are goals that are important to me. I’m documenting the goals so I can hold myself accountable come next January and see how I progressed.

2014 Recap: How did I do?

Inspired by Scott Lowe’s goals for 2015, I’ve decided to be more rigorous about my own work-related goals. In this post, I’ll give a recap of my unofficial plans for 2014. I never wrote these down or otherwise formalized them, so they are a bit rough.

  • Start a blog

This one I achieved pretty early in the year. I started collecting material during the holiday and started publishing content on February. Grade: Pass

  • Post a Monday morning blog article every week

I had mixed success here. My initial aim really was to have one article a week and I made it for about 10 months, with the help of Jason (@hawkbox), who contributed some Hyper-V content during August. Things started to fall apart in November, though. I participated in the 30in30 challenge for November and didn’t quite make the mark, and it got me away from the weekly post. In December, I didn’t post weekly. So I failed, only hitting this about 70% of the time. Grade: C

However, I consider the results a success. I learned a few things. One, how difficult it is to get content ready every week without fail. I would have done much poorer if Jason had not helped me in August, as I was reaching burnout. Learning new things can be fun, but the pressure to do it continually does suck some of the enjoyment out. It also has a high cost on your free time, you need to spend a lot of it on the blog and it can intrude upon time with family and friends (you’ll note that blog content has been almost nil since the holiday season began, for this reason). Second, a weekly article isn’t needed. I managed to have 100 blog articles in 11 months, which is more than one per week. Some were really short, some very lengthy technical articles, and some were opinion pieces. People found the articles and I get a note from readers once in a while. Thank you all for reading, it really makes it worthwhile.

Even though I missed my original mark, I’ll regrade it as a B as I met the real, unwritten goal – create helpful content and post it on my blog frequently.

  • Learn Puppet

On this goal, I did very well. I wrote some 35+ articles, created three forge modules, and it spawned a project ‘Puppetinabox’. It also had some unrecognized synergy with using and learning Git, which I now feel adept with. I still have a lot of things to learn about Puppet, but my efforts clearly paid off. Grade: A

  • Present publicly

A goal I added midway through the year was to present something publicly. It didn’t matter what, but it had to be in front of real live human beings. I had done an Auto Deploy Deep Dive on the vBrownBag podcast earlier in the year and loved it. A podcast can make you nervous, but no-one can see you and you can’t see the attendees. I wanted to really push myself as I know I’m not good at public speaking. I ended up giving two public presentations. The first was a Virtual Design Master followup at the annual Indianapolis VMUG conference in July and the second was about DevOps for SysAdmins in November at a regular VMUG meeting. I had a lot of fun with both and I think I got past much of my stage fright. I also presented with Byron Schaller both times, who really helped with getting the presentations together. Of course, both presentations were to small groups. I had applied to give my Auto Deploy presentation at VMworld and presenting to (hopefully!) a few hundred people would have been a much different experience. There’s always next year! Grade: B+

For someone without a solid set of goals for the year, I think I did fairly well! I’m going to try and improve on this for 2015 by documenting the goals publicly and posting them here. Stay tuned.