Category Archives: DevOps

Standard

Posted by

Posted on

November 24, 2014

Posted under

DevOps, vDM30in30

Comments

2 Comments

DevOps for the SysAdmin

Last Thursday, I was proud to present the following slidedeck with Byron Schaller at the Indianapolis VMUG meeting. There was one edit I wanted to make afterward, which of course is when I have the best editing thoughts 🙂

The Dev in DevOps stands for software development, not software developers.

It’s a small and subtle difference that has a huge impact.

If you haven’t spoken at your local VMUG, please, give it a shot. It’s incredibly rewarding in the growth of your speaking abilities, your ability to internalize and then vocalize a subject, and in participation in the VMUG. Most VMUGs struggle to find speakers. Please volunteer and give back to the community..

Standard

Posted by

Posted on

November 17, 2014

Posted under

DevOps, Puppet

Comments

2 Comments

Introduction to rspec-puppet

Editor’s note: Please check out the much newer article Configuring Travis CI on a Puppet Module Repo for the new “best practices” around setting up rspec-puppet. You are encouraged to use the newer setup, though everything on this page will still work!

Over the course of the Puppet series, one thing I’ve ignored is testing. As vSphere admins, many of us are comfortable with programming but probably not as well versed in some practices as full-time developers. Today we’ll look at an introduction to some test-driven-development with puppet.

Test Driven Development

What is this Test Driven Development, or TDD, that everyone speaks so highly of? In essence, you write tests that fail before you write any code, then you write code to satisfy the tests. Each test typically looks at a specific unit of functionality of a program, such as whether a file is created or has contents, and are called “unit tests.” By testing a specific function, when you have a failure, you can typically narrow down the problem domain to a few lines of code. When all unit tests generate successes, your code works (in theory!). In addition, when you modify the code in the future, these unit tests help ensure that you haven’t broken something that was previously working, also known as a “regression.”

TDD depends, of course, on writing tests that both provide coverage of all your code and that map to the requirements of the program. If you forget to provide a test that covers a vital portion of your code, all your tests can be successful but leave you with a broken program. If you have not been practicing TDD on an existing program, you can still add tests. However, you will not have 100% test coverage (the percent of code that is covered by unit tests) initially, or possibly ever, as all of the existing code was written prior to the unit tests. To keep things simple today, we’ll start writing some new code.

Continue reading →

Standard

Posted by

Posted on

November 12, 2014

Posted under

DevOps, Puppet, vDM30in30

Comments

Leave a comment

Don’t Disable SELinux

When developing new web-based solutions on modern Linux distros, inevitably you’ll run into a fun issue – like your webserver throwing database errors when there’s not even any traffic making it to out of the server toward the database – and bang your head against the desk a bit. If you google for the error, you’ll run into the worst advise possible: “If your problem is not solved then disable your SELinux.” That’s right, just disable the whole thing because one part bothers you. The only positive part of this advise is that you may not have even though to look at SELinux before that.

You can verify that SELinux is the issue by taking a look at the audit log (tail -f /var/log/audit/audit.log) and using your web application. You’ll see a ton of crap that is simply undecipherable to human beings. What you’re looking for is the word denied and the application, file, or user that is having an issue. Here’s a deny for the application httpd when trying to talk to that remote database:

type=AVC msg=audit(1415813628.801:628): avc:  denied  { name_connect } for  pid=11911 comm="httpd"
 dest=3306 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:mysqld_port_t:s0
 tclass=tcp_socket

The next step is to narrow the issue down. There are a large number of settings for SELinux, known as SELinux Booleans, that may be affecting your application. Take a quick gander at them, find the most likely boolean, set the value to on, and try your application again. If it doesn’t work, set it to off and try another. Here’s a Tips and Trick page that describes the process in more detail and provides a pretty thorough list of booleans. Can’t access files on an NFS share via httpd? Set httpd_use_nfs to true. Talking to a remote database as above? That’s httpd_can_network_connect_db. This is just as simple and more beneficial than disabling SELinux altogether.

Of course, I’d be remiss if I just told you to use setsebool as root. You need to including this setting in your application definition. For example, integrate the setting into your puppet manifests with the selboolean type. Set the value to on and persistent to true. Apply your manifest and getsebool will show the new value. Here’s an example of a manifest I built for the phpMyAdmin application, specifically lines 25-28 where the selbooleans are set. If you’re using a different configuration management tool, you’ll have to do this part yourself, the important part is that you capture the setting.

Take a few minutes to learn how to use SELinux, so you’re aware of when you’re barking up the wrong tree and how to resolve issues, and integrate your findings into your application’s state definition. You’ll benefit by leaving the protection in place.

Standard

Posted by

Posted on

November 7, 2014

Posted under

DevOps, PowerCLI, vDM30in30

Comments

Leave a comment

Use existing definitions as a baseline

Sometimes we spend way too long trying to define things in our head when we can get existing configurations from the system. It’s vital to have a full service definition or any promotion of the service through environments will turn up missing components and make your life hell. If you’re building a new service that looks similar to an old one, or evolves the old service, steal the old service’s definition and then modify it.

vSphere

There are a number of ways to gather existing service definitions. If you’re building a new host and you have Enterprise Plus licenses, use Host Profiles. Export an existing host’s config to a host profile, uncheck the irrelevant portions, change what’s relevant but different, and apply to the new host. It might take a few tweaks, but you’ll get it right soon. Then export the new host’s config to a host profile and you’re good to go.

If you don’t have Enterprise Plus, take a look at PowerCLI. It will take more legwork, but there are a ton of cmdlets available to capture networking, storage, and other service definitions from existing hosts which you can then apply elsewhere.

Continue reading →

Standard

Posted by

Posted on

November 3, 2014

Posted under

DevOps, Linux, Puppet

Comments

2 Comments

Deploying your SSH Authorized Key via Puppet

Update: I have since published a forge module rnelson0-local_user that can be used to distribute keys as well. If you are using keys with local users, I highly recommend using the forge module. If you are not managing the users directly (say, for domain-joined nodes), continue to use the solution presented belwo.

Today, let’s look at deploying SSH Authorized Keys via puppet. An authorized key is a public key used for public key authentication (not to be confused with an ssh key, which is the unique key identifier of a host that verifies the server is who it says it is). By attaching an authorized key to a user, any login attempt for that user that presents the corresponding private key will be authenticated successfully, giving you the ability to log in without a password. This is commonly used for automation, where no user is present to enter a password, or for a user with a private key to access systems without additional steps.

Authorized keys are typically considered more secure than a password, but they rely on protecting the private key. If the private key is not secured, anyone who obtains the private key can impersonate the account. If a non-privileged user’s key is lost, only that user’s access and files are at immediate risk. An attacker would still need to escalate privileges to damage the system. If a privileged user’s key (no-one reading this logs in as a privileged user, such as root, right? RIGHT?) or an automation account’s key is lost, the immediate risk is much higher. An attacker might gain access to the entire system or be able to attack other systems. You must absolutely secure private keys and ensure you follow the principle of least privilege for all users, especially automation accounts.

Let’s look at an example of how to use a properly secured authorized key. In past articles, we’ve built a yum repository and a build server. You may be logging into these servers frequently and transfering files between the two. Every time, you need to enter your passwords. That gets old, quickly. If you had an authorized key in place, you can ssh to both servers and present your private key, no password. If you copy the private key to the build server or create a new key, you could scp files from the build server to the yumrepo the same way. This should make life a lot easier for you.

There are lots of ways to generate keys depending on your OS and applications. My workflow is to use Putty on a Windows 7 laptop to connect to linux VMs, then use the linux openssh client to ssh to other linux VMs. I’ll cover generating and configuring keys with Putty and openssh.

Continue reading →

Standard

Posted by

Posted on

October 27, 2014

Posted under

DevOps, Puppet

Comments

2 Comments

Creating a Puppet ERB Template

Recently, we looked at converting a module to be hiera friendly. Another task you may have to look forward to is tracking configuration files for different hosts providing the same service. You could have a config for each node, network, environment, etc., all of which need updated if some common element changes. Or, you could use a Puppet Template to have a single template that is populated by node-specific elements from node facts and puppet variables. Each node would receive a personalized copy and any common element changes would be reflected across the board immediately.

As an example, I run some mediawiki servers at work. Each one points to a different database but is otherwise very similar. The search engine is SphinxSearch and it relies on the Sphinx config file /etc/sphinx/sphinx.conf. The config includes the database connection information, which varies from device to device, and a number of other settings standardized across the wikis (minimum search term length, wildcards, and other search settings). Keeping the database connection information accurate across three wikis would normally require three config files. Let’s simplify that with a template.

Puppet templates are written in ERB, a templating language that is part of the Ruby standard library. ERB commands are interpolated into the config where needed and puppet feeds facts and variables to ERB, which determines what values to populate the config with. We have a few good sources of information on the templates: the Ruby docs, a Puppet Labs Using Puppet Templates article, or the Learning Puppet chapter on Templates. I’ll be picking out some highlights, reference them as needed as we work on our template.

Continue reading →

Standard

Posted by

Posted on

October 23, 2014

Posted under

DevOps, Opinion, Puppet

Comments

1 Comment

Why Puppet?

As we near the end of my schedule puppet content, I’ve asked the Twitterverse for any other topics people want to see discussed. Jason Shiplett asked a great question: Why Puppet?

This is essentially a two-fold question. First, you must understand what Configuration Management (CM) is and why you need it. Second, of all the CM tools out there, why would you choose Puppet?

Configuration Management

In spite of my telling Jason that the world doesn’t need another “Why CM?” post, here we go 🙂

Plenty of other people have done a great job explaining what Configuration Management is and why you need it. Chief among these is Information Technology Infrastructure Library, or ITIL, a framework for IT Service Management. In the Service Transition volume, Configuration Management is described. We can simplify the meaning to describing and managing the state of a configuration through a service’s lifecycle.

Continue reading →

Standard

Posted by

Posted on

October 20, 2014

Posted under

DevOps, Puppet

Comments

4 Comments

Refactoring a Puppet class for use with Hiera

For the past few weeks, we have been working on packaging our own software and deploying it with puppet. Before that, we touched on refactoring modules to use hiera. In fact, I grandiosely referred to it as Hiera, R10K, and the end of manifests as we know them. I included a very simple example of how to refactor a per-node manifest into the role/profile pattern and use hiera to assign it to the node. Today, we’ll look at more features of hiera and how you would refactor an existing class to use hiera.

In a legacy implementation of puppet, you’ll likely find plenty of existing modules whose classes have static assignment or lots of conditionals to determine the necessary values to be applied. Even in a greenfield implementation of puppet, you may find yourself writing straight Puppet DSL for your classes before refactoring them to use hiera. Figuring out how to refactor efficiently isn’t always obvious.

First, let’s take a look at Gary Larizza’s When to Hiera and The Problem with Separating Data from Puppet Code articles. Gary covers the when and why much better than I could, so please, go read his article and then come back here. Gary also covers the common pre-hiera pattern and a few patterns that can be used when refactoring to hiera. There is another pattern that is documented indirectly by Gary (under Hiera data bindings in Puppet 3.x.x) and in the Hiera Complete Example at docs.puppetlab.com. I’m going to explain this pattern and document this directly, adding another Hiera pattern to Gary’s list.

Continue reading →

Standard

Posted by

Posted on

October 13, 2014

Posted under

DevOps, Linux, Puppet

Comments

1 Comment

FPM and Build Automation

Having created one or more build servers, the next logical step is to start building software. We touched on this briefly a few weeks ago, and with a proper development station, it’s time to expand on it.

If you’re a developer by trade, you can probably skim or skip this article. Remember, this series is aimed at vSphere Admins, not devs. I’d certainly appreciate your insights in the comments or on twitter, though!

Modifying software build processes for FPM

We’ve used FPM in the past to take a directory and turn it into a package. This works very well when /some/long/path belongs entirely to your application. What if your application drops a binary in /bin, a manpage in /usr/share/man/man5, a config file in /etc, or even just a few files in a directory that’s shared with other packages? Let’s take a look at an extension for mediawiki. This is very simple, we have a legacy Makefile and two useful targets, dev and prod:

Continue reading →

Standard

Posted by

Posted on

October 6, 2014

Posted under

DevOps, Linux, Puppet

Comments

1 Comment

Puppetize a Build server

The Puppet series so far has really focused on VM builds and just started to touch on software packaging. We need an appropriate place to do this work, and what better way to set that than via Puppet itself? Today, we’ll create some roles and profiles for a build server, which could be permanent and share amongst developers, spun up as needed for the team, or spun up per developer.

Build Profile and Role

The last few examples we have done with FPM were on our “production” servers. That’s less than ideal for a few reasons. You wouldn’t want to mess up the publicly available service while packaging, whether by overwriting a file, exhausting resources, or the brief outage when services restart. It is not a good idea to add compilers and development libraries to any server unnecessarily as it increases the attack surface (additional security risks, additional packages to patch, additional items for auditors to flag, etc). You also probably do not want your build servers in the same environment as your production servers (unless, as is the case in these examples, your “production” environment is your lab – so just pretend it’s a different environment). Let’s assume that we do not have a good reason to violate these best practices, so our goal is set up a dedicated build server. It will require all the software we have been using so far, and we will throw on a local user. If you have LDAP or another directory service in your lab, I would suggest using it, but this is a good example as sometimes the build network is restricted.

We have two profiles to create, then. The first is the local users. We’ll call this class ::profile::build_users, in case we create another grouping of users later. The second profile is for our build software, and we will call it ::profile::build. Here are the two class files, located at profile/manifests/users/build.pp and profile/manifests/build.pp, respectively.

Continue reading →