Kickstart your CentOS Template, EL7 Edition

I wrote an article on kickstarting your CentOS Template in early 2014 that focused on Enterprise Linux 6. Later in the year, RHEL7 was announced and CentOS 7 soon followed. It’s well past time to refresh the kickstart article. To keep this more of a “moving target”, I’ve created a github repo to host the kickstart files at puppetinabox/centos-kickstart, so you can turn there for updates or submit your own PRs. I’m also toying with an existing puppet module danzilio/kickstart that generates kickstart files, and I plan to contribute some PRs to it to manage the kickstart service itself. In the meantime, I’ll show a small profile that will do the same thing, since it’s just apache and a few files.

Kickstart Configuration

The new EL7 file was based off the EL6 version. I simply changed the package list as some were no longer available and the open-vm-tools are now the preferred method of VMware tools management. That section was removed from the bottom. In the additional steps section, I changed the yum repo for puppet from Puppet 3 to Puppet Collections 1 for Puppet 4. I also removed the banner setup, that’s easy enough to add in if you like.

Kickstart Service Management

The kickstart service itself is pretty simple. You can use puppetlabs-apache to install apache and then place your files in it’s default root of /var/www/html. Take the kickstart files and add them to dist/profile/files with any modifications you require. Then create a profile that includes apache plus the kickstart files. That would look something like this:

Continue reading

NFS Export Settings for vSphere

Over the past few days, I’ve had to set up an NFS export for use with vSphere. I found myself frustrated with the vSphere docs because they only seem to cover the vSphere side of the house – when discussing the export that vSphere will use, the docs simply said, “Ask your vendor.” If you’re enabling NFS on your SAN and they have a one-click setup for you or a document with settings to enable, great, but if you’re setting up on a bare metal server or VM running Linux like me, you don’t have a vendor to ask.

Since I didn’t have a guide, I took a look at what happens when you enable NFS on a Synology. I don’t know if this is optimal, but this works for many people with the defaults. You can replicate this in Control Panel -> Shared Folders -> Highlight and Edit -> NFS Permissions. Add a new rule, add a hostname or IP entry, and hit OK. Here’s what the defaults look like:

NFS Exports fig 1

Let’s take a look at what happened in the busybox shell. SSH to your Synology as root with the admin password. Take a look at the permissions on the mount path (e.g. /volume1/rnelson0) and the contents of /etc/exports.

NFS Exports fig 2

(there is no carriage return at the end of the file, the ‘ds214>’ is part of the prompt not the exports contents)

A working mode for the directory is ‘0777’ and there’s a long string of nfs options. They are described in detail in the exports(5) man page. Here’s a high-level summary of each:

  • rw: Enable writes to the export (default is read-only).
  • async: Allow the NFS server process to accept additional writes before the current writes are written to disk. This is very much a preference and has potential for lost data.
  • no_wdelay: Do not delay writes if the server suspects (how? I don’t know) that another write is coming. This is a default with async so actually has no specific benefit here unless you remove async. This can have performance impacts, check whether wdelay is more appropriate.
  • no_root_squash: Do not map requests from uid/gid 0 (typically root) to the anonymous uid/gid.
  • insecure_locks: Do not require authentication of locking requests.
  • sec=sys: There are a number of modes, sys means no cryptographic security is used.
  • anonuid/anongid: The uid/gid for the anonymous user. On my Synology these are 1025/100 and match the guest account. Most Linux distros use 99/99 for the nobody account. vSphere will be writing as root so this likely has no actual effect.

I changed the netmask and anon(u|g)id values, as it’s most likely that a linux box with a nobody user would be the only non-vSphere client. Those should be the only values you need to change; async and no_wdelay are up to your preference.

If you use Puppet, you can automate the setup of your NFS server and exports. I use the echocat/nfs module from the forge (don’t forget the dependencies!). With the assumption that you already have a /nfs mount of sufficient size in place, the following manifest will create a share with the correct permissions and export flags for use with vSphere:

node server {
  include ::nfs::server

  file{ '/nfs/vsphere':
    ensure => directory,
    mode   => '0777',
  ::nfs::server::export{ '/nfs/vsphere':
    ensure  => 'mounted',
    nfstag  => 'vsphere',
    clients => ',async,no_wdelay,no_root_squash,insecure_locks,sec=sys,anonuid=99,anongid=99)',

To add your new datastore to vSphere, launch the vSphere Web Client. Go to the Datastores page, select the Datacenter you want to add the export to, and click on the Related Objects tab. Select Datastores and the first icon is to add a new datastore. Select NFS and the NFS version (new to vSphere 6), give your datastore a name, and provide the name of the export (/nfs/vsphere) and the hostname/IP of the server ( Continue through the wizard, checking each host that will need the new datastore.

NFS Exports fig 3

Click Finish and you have a datastore that uses NFS! You may need to tweak the export flags for your environment, but this should be a good starting point. If you are aware of a better set of export flags, please let me know in the comments. Thanks!

Deploying MySQL with Puppet, without disabling SELinux

I’ve been vocal in the past about the need to not disable SELinux. Very vocal. However, SELinux can be difficult to work with. I was reminded of how difficult while deploying MySQL recently. Let’s take a look at how to iron out the SELinux configuration for MySQL, and how to deploy it with Puppet I will be using CentOS 6.6 in this article. The package names and SELinux information may vary if you use another distribution.

MySQL Design

Let’s review the design of the MySQL installation before continuing. For the most part, it’s a standard install, we’re not doing any elaborate tuning or anything. All the passwords will be ‘password’ (clearly you should change this in production!). All the anonymous users (@localhost, root@localhost, etc.) will have a password set. An additional ‘replication’ user is created so multiple databases can be replicated and example replication settings are included. The test databases are removed and a single user/database pair of wikiuser/wikidb will be created. We won’t do anything with the database, it’s just an example that can be duplicated as needed.

Continue reading

Don’t Disable SELinux, Part 2

Yesterday I warned everyone not to disable SELinux because the fix is almost always a quick one. But, what do you do if there is no selboolean that fixes your problem with a simple one liner?

After yesterday’s article, Tim Meusel shared a message he receives in his audit log when running nginx on his puppet master with SELinux in enforce mode:

type=AVC msg=audit(1415871389.171:787): avc:  denied  { name_connect }
 for  pid=2228 comm="nginx" dest=8080
 tcontext=system_u:object_r:http_cache_port_t:s0 tclass=tcp_socket
type=SYSCALL msg=audit(1415871389.171:787): arch=c000003e syscall=42
 success=no exit=-13 a0=19 a1=259e2b0 a2=10 a3=7fffdac559d0 items=0
 ppid=2227 pid=2228 auid=4294967295 uid=996 gid=995 euid=996 suid=996
 fsuid=996 egid=995 sgid=995 fsgid=995 tty=(none) ses=4294967295
 comm="nginx" exe="/usr/sbin/nginx" subj=system_u:system_r:httpd_t:s0

That’s…. that’s ugly. The important parts have been highlighted. Nginx cannot talk to the tcp_socket at /var/run/puppet/puppetmaster_unicorn.sock. There doesn’t appear to be a selboolean that matches the issue. You could try flipping semi-relevant booleans for hours till you stumble upon some combination that may work, undoubtedly with side effects, and possibly never find the right combination. That could end up being a LOT of time wasted without any guarantee of success.

Instead, use audit2allow. By providing the tool with portions of an audit log, it will build an SELinux policy that will allow everything marked as “denied”. Here’s an example of generating a policy for review, then generating and applying that policy:

grep nginx /var/log/audit/audit.log | audit2allow > nginx.te
more nginx.te
grep nginx /var/log/audit/audit.log | audit2allow -M nginx
semodule -i nginx.pp

You can find more detail on the tool on the web, particularly this article where another nginx user is struggling with SELinux. You may have to repeat this process a few times – nginx stopped running when it failed to attach to the socket, so there could be other SELinux permission issues it would encounter if it had not failed. You won’t see those in the audit.log until it gets past the socket. Keep at it until audit2allow is building the same policy file on consecutive runs, at which point there are no new failures to discover. Your application should be fully working now and encounter no more SELinux permission issues.

Update: Tim continued to struggle after he performed the above steps until he moved the unicorn socket out of /var/run (which is admittedly not the recommended location!) even though he wasn’t seeing any more failures in the audit log. This command forces SELinux to log all failure events and then the new failures showed up and were processed by audit2allow:

semodule --disable_dontaudit --build

See Tim’s blog for more info.

You can apply the policy via puppet using the selmodule type, plus a file resource to put the .pp file in the correct location.

While this takes a lot longer to resolve than touching some selbooleans, you should only have to do it once. This ensures you still have the protections of SELinux and a well defined policy state for your application. If, and only if, this doesn’t resolve your issue, should you even entertain the thought of disabling SELinux, as a temporary resolution until a permanent solution is found.

Deploying your SSH Authorized Key via Puppet

Update: I have since published a forge module rnelson0-local_user that can be used to distribute keys as well. If you are using keys with local users, I highly recommend using the forge module. If you are not managing the users directly (say, for domain-joined nodes), continue to use the solution presented belwo.

Today, let’s look at deploying SSH Authorized Keys via puppet. An authorized key is a public key used for public key authentication (not to be confused with an ssh key, which is the unique key identifier of a host that verifies the server is who it says it is). By attaching an authorized key to a user, any login attempt for that user that presents the corresponding private key will be authenticated successfully, giving you the ability to log in without a password. This is commonly used for automation, where no user is present to enter a password, or for a user with a private key to access systems without additional steps.

Authorized keys are typically considered more secure than a password, but they rely on protecting the private key. If the private key is not secured, anyone who obtains the private key can impersonate the account. If a non-privileged user’s key is lost, only that user’s access and files are at immediate risk. An attacker would still need to escalate privileges to damage the system. If a privileged user’s key (no-one reading this logs in as a privileged user, such as root, right? RIGHT?) or an automation account’s key is lost, the immediate risk is much higher. An attacker might gain access to the entire system or be able to attack other systems. You must absolutely secure private keys and ensure you follow the principle of least privilege for all users, especially automation accounts.

Let’s look at an example of how to use a properly secured authorized key. In past articles, we’ve built a yum repository and a build server. You may be logging into these servers frequently and transfering files between the two. Every time, you need to enter your passwords. That gets old, quickly. If you had an authorized key in place, you can ssh to both servers and present your private key, no password. If you copy the private key to the build server or create a new key, you could scp files from the build server to the yumrepo the same way. This should make life a lot easier for you.

There are lots of ways to generate keys depending on your OS and applications. My workflow is to use Putty on a Windows 7 laptop to connect to linux VMs, then use the linux openssh client to ssh to other linux VMs. I’ll cover generating and configuring keys with Putty and openssh.

Continue reading

FPM and Build Automation

Having created one or more build servers, the next logical step is to start building software. We touched on this briefly a few weeks ago, and with a proper development station, it’s time to expand on it.

If you’re a developer by trade, you can probably skim or skip this article. Remember, this series is aimed at vSphere Admins, not devs. I’d certainly appreciate your insights in the comments or on twitter, though!

Modifying software build processes for FPM

We’ve used FPM in the past to take a directory and turn it into a package. This works very well when /some/long/path belongs entirely to your application. What if your application drops a binary in /bin, a manpage in /usr/share/man/man5, a config file in /etc, or even just a few files in a directory that’s shared with other packages? Let’s take a look at an extension for mediawiki. This is very simple, we have a legacy Makefile and two useful targets, dev and prod:

Continue reading

Puppetize a Build server

The Puppet series so far has really focused on VM builds and just started to touch on software packaging. We need an appropriate place to do this work, and what better way to set that than via Puppet itself? Today, we’ll create some roles and profiles for a build server, which could be permanent and share amongst developers, spun up as needed for the team, or spun up per developer.

Build Profile and Role

The last few examples we have done with FPM were on our “production” servers. That’s less than ideal for a few reasons. You wouldn’t want to mess up the publicly available service while packaging, whether by overwriting a file, exhausting resources, or the brief outage when services restart. It is not a good idea to add compilers and development libraries to any server unnecessarily as it increases the attack surface (additional security risks, additional packages to patch, additional items for auditors to flag, etc). You also probably do not want your build servers in the same environment as your production servers (unless, as is the case in these examples, your “production” environment is your lab – so just pretend it’s a different environment). Let’s assume that we do not have a good reason to violate these best practices, so our goal is set up a dedicated build server. It will require all the software we have been using so far, and we will throw on a local user. If you have LDAP or another directory service in your lab, I would suggest using it, but this is a good example as sometimes the build network is restricted.

We have two profiles to create, then. The first is the local users. We’ll call this class ::profile::build_users, in case we create another grouping of users later. The second profile is for our build software, and we will call it ::profile::build. Here are the two class files, located at profile/manifests/users/build.pp and profile/manifests/build.pp, respectively.

Continue reading