Fortigate user permissions peculiarities

While working with a customer on their Fortigate firewalls, I was introduced to a peculiarity of how FortiOS interprets user’s diag commands. I suspect this affects multiple versions, but I don’t have the ability to test this.

  • FortiOS: 4.2.x
  • User: wild-card (TACACS)
  • Profile: super_admin_readonly

TACACS users whose permissions elevate them to the super_admin profile are unaffected. They can run diag commands unrestricted as they have full access.

TACACS users whose permissions remain at super_admin_readonly were finding that they could not run diag commands that accessed an interface, such as diag sniff packet any “icmp”. Upon further investigation, the issue was related to the IP the user connected to and the interface (“any” in the example) used in the command. As a readonly user, the any interface is off-limits. The interfaces configured for the VDOM that the user connected to are available to the readonly users.

In other words, if a firewall had two VDOMs, Common and DMZ, and the user connected to any interface connected to the Common interface, only those interfaces would be useable. For instance, diag sniff packet common-outside “icmp” would work, as well as common-inside. Interfaces connected to other VDOMs are off-limits, so diag sniff packet dmz-outside “icmp would fail. By providing the end user a list of the IP addresses and interface names, and the VDOM they belonged to, the user was able to perform all required diagnostic commands.

I hope this is fixed in more recent versions, but at least there’s a workaround that makes some logical sense.

Thinking TechX

A word we hear too much of these days is ‘disrupt’. When it’s not overused, it means that you’re trying to change the way you do things in some dramatic fashion. Instead of doing things by hand, you use some tool to automate some or all of it. Or you switch from Linux everywhere to Window everywhere, or vice versa. Whatever the change is, the point is that you’re changing how you do things.

Something that frequently appears to be forgotten during disruption is to change how you think about doing things. When you were doing things on Windows, you probably did a lot of mouse clicking and typing. Now you’ve moved to Linux. Was the change really about the OS? Probably not. The change was about not having to click the mouse and type. So stop it! Start “thinking Linux”, or whatever technology you’re using.

This has two advantages. First, it becomes really disruptive, because it was the thought process holding you back the whole time. If you only change the technology, you’ve just hidden the problem for a while. That buys you a bit of runway but no real solution. Applying an entirely new thought process will help you get out of the rut of “the way we’ve always done it.”

Second, if you are using idiomatic patterns of the chosen technology – such as using camelCase in Powershell but snake_case in Ruby – you’re going to find it much easier to attract and retain coworkers who already think that way. If your Ruby code looks like PowerShell, most Ruby devs will just run away. Even if your team has low turnover, it will make everyone on the team better able to receive new team members and allow the team to better contribute back to the community, especially via open source projects.

Take the time to approach your problems in a new manner from top to bottom and you’ll reap the benefits.

The Goal: Throughput and Efficiency

One of the most important concepts of The Goal is to increase throughput. Throughput is the rate at which the system generates money through sales. That is, when your company takes raw materials, processes them into a finished good, and sells it, the measured rate of that activity is your throughput. Severe emphasis on sales. Throughput is not the same as efficiency. Today, we will look at throughput vs. efficiency and how these concepts apply to IT.

Though we are focusing on throughput, we must state the descriptions of the two other measurements. Inventory is all the money that the system has invested in purchasing things which it intends to sell. Operational expense is all the money the system spends in order to turn inventory into throughput. I list the three definitions together because the definitions are precise and interconnected. Changing even a single word in one requires the other two be adjusted as well.

Another important concept in throughput is that it measures the entire system, not a locality. Whether you work in your garage or in a giant auto plant, you can not measure throughput locally, it must be measured over the entire system. This conflicts with most companies’ measurements of local efficiency. Employers naturally want to keep all their employees busy and employees like to see their coworkers pull their own weight. Why should Jane get to twiddle her thumbs at the Fob machine when Jill is busy pushing pallets of Fob parts around the floor? Is it fair to George to watch Jeff read the newspaper while he has to investigate hundreds of parts for quality control? And shouldn’t Jane and Jeff be worried that they might be reprimanded or fired for not being efficient, or draw the ire of their coworkers?

Continue reading

Introduction to rspec-puppet

Over the course of the Puppet series, one thing I’ve ignored is testing. As vSphere admins, many of us are comfortable with programming but probably not as well versed in some practices as full-time developers. Today we’ll look at an introduction to some test-driven-development with puppet.

Test Driven Development

What is this Test Driven Development, or TDD, that everyone speaks so highly of? In essence, you write tests that fail before you write any code, then you write code to satisfy the tests. Each test typically looks at a specific unit of functionality of a program, such as whether a file is created or has contents, and are called “unit tests.” By testing a specific function, when you have a failure, you can typically narrow down the problem domain to a few lines of code. When all unit tests generate successes, your code works (in theory!). In addition, when you modify the code in the future, these unit tests help ensure that you haven’t broken something that was previously working, also known as a “regression.”

TDD depends, of course, on writing tests that both provide coverage of all your code and that map to the requirements of the program. If you forget to provide a test that covers a vital portion of your code, all your tests can be successful but leave you with a broken program. If you have not been practicing TDD on an existing program, you can still add tests. However, you will not have 100% test coverage (the percent of code that is covered by unit tests) initially, or possibly ever, as all of the existing code was written prior to the unit tests. To keep things simple today, we’ll start writing some new code.

Rspec-puppet

The tool we will use today is called rspec-puppet. Rspec is a testing tool for Ruby and rspec-puppet is an implementation created specifically to test puppet modules. Before we work on the module, we need to install rspec-puppet. Using puppet, this is very simple. Add these statements to your build profile or use it in a puppet apply. Make sure to apply it to your build server before continuing.

  package {'rspec-puppet':
    ensure   => present,
    provider => gem,
  }

You can find some more documentation and examples of using rspec-puppet on its github page. Rspec tests are generally of the format it { should test(‘something’) }, a very human-readable testing framework. Let’s create a module that we want to test.

Certs module

I have a particular need for a module to easily create two SSL-related files for webservers based on existing content. This module is called certs, and this article will step through generating it fully. If you have your own need for a module, go ahead and create it now, but you will have to map the article to your module on your own. Here’s the command to generate a module called certs by rnelson0 in the current path. You can hit enter to all the questions it asks.

puppet module generate rnelson0-certs --modulepath=./

Cd into certs. The framework for a puppet module already exists. Run git init and add an upstream with git remote add origin <git url>. Before we go any further, let’s look at what our module needs to provide – remember, we start with tests, not code! Here’s part of an instance of an apache::vhost used by one of my profile classes:

  apache::vhost {$cname:
    ...
    ssl_cert    => "/etc/ssl/certs/${cname}.crt",
    ssl_key     => "/etc/ssl/certs/${cname}.key",
    ...
  }

The cert and key files must be present on the local system. You can achieve this with a File resource for the crt that comes before a File resource for the key which comes before the apache::vhost (file -> file -> apache::vhost). That’s great when you have one profile that uses apache::vhost, but not so awesome when you have multiple such profiles, it’s a lot of repeated code which means a lot of room for errors. The goal is to simplify the code. Here’s how I envision my module working:

  certs::vhost {$cname: } ->
  apache::vhost {$cname:
    ...
    ssl_cert    => "/etc/ssl/certs/${cname}.crt",
    ssl_key     => "/etc/ssl/certs/${cname}.key",
    ...
  }

The module can put the files where we want them. Hrm, what if someone wants the modules someplace other than /etc/ssl/certs? We should probably allow that with a target_path parameter. We’ll also need to let the caller specify where the files are coming from. The files will have to come from somewhere else – we can’t add them to the certs module if we want to share the module with others – perhaps another puppet module or an NFS mount. We’ll call that parameter source_path and we’ll suggest a puppet module called othermodule (maybe site_certificates or domainX_certificates). Here’s what the modified statements might look like:

  certs::vhost {$cname:
    source_path => 'puppet:///othermodule',
    target_path => '/etc/ssl/certs',
  } ->
  apache::vhost {$cname:
    ...
    ssl_cert    => "/etc/ssl/certs/${cname}.crt",
    ssl_key     => "/etc/ssl/certs/${cname}.key",
    ...
  }

We’re not worried about how it happens yet, just defining what the module needs to do. You can’t test something unless you know what it is.

Creating a unit test

It’s almost time to create a unit test. If we look at the rspec-puppet home page, we’ll see that we need to run rspec-puppet-init. You should see something like this:

[rnelson0@build certs]$ rspec-puppet-init
 + spec/defines/
 + spec/functions/
 + spec/hosts/
 + spec/fixtures/
 + spec/fixtures/manifests/
 + spec/fixtures/modules/
 + spec/fixtures/modules/certs/
 + spec/fixtures/manifests/site.pp
 + spec/fixtures/modules/certs/manifests
!! spec/spec_helper.rb already exists and differs from template
!! Rakefile already exists and differs from template

Whoops, there are two errors. I haven’t explored Rakefile yet, but spec/spec_helper.rb will definitely cause us problems. Remove the file and re-run rspec-puppet-init and you’ll see a much different file:

[rnelson0@build certs]$ cat spec/spec_helper.rb
require 'puppetlabs_spec_helper/module_spec_helper'
[rnelson0@build certs]$ rm spec/spec_helper.rb
[rnelson0@build certs]$ rspec-puppet-init
 + spec/spec_helper.rb
!! Rakefile already exists and differs from template
[rnelson0@build certs]$ cat spec/spec_helper.rb
require 'rspec-puppet'

fixture_path = File.expand_path(File.join(__FILE__, '..', 'fixtures'))

RSpec.configure do |c|
  c.module_path = File.join(fixture_path, 'modules')
  c.manifest_dir = File.join(fixture_path, 'manifests')
end

You can also follow a manual process to set up the spec files, but this gets you there a lot faster. You can now type rspec at the root of your module and the default test – that include <modulename> results in including the module in a catalog – will succeed.  There are a TON of ‘deprecated’ warnings that will show up. PUP-3594 explains where the messages come from, a workaround, and where possible fixes may come from. I’m just going to ignore the warnings until there is a permanent fix, as it doesn’t obscure the last line, which is what we really care about:

[rnelson0@build certs]$ rspec
.

Deprecation Warnings:

Filtering by an `:example_group` subhash is deprecated. Use the subhash to filter directly instead. Called from /usr/lib/ruby/gems/1.8/gems/rspec-puppet-1.0.1/lib/rspec-puppet/example.rb:12.
Filtering by an `:example_group` subhash is deprecated. Use the subhash to filter directly instead. Called from /usr/lib/ruby/gems/1.8/gems/rspec-puppet-1.0.1/lib/rspec-puppet/example.rb:16.
Filtering by an `:example_group` subhash is deprecated. Use the subhash to filter directly instead. Called from /usr/lib/ruby/gems/1.8/gems/rspec-puppet-1.0.1/lib/rspec-puppet/example.rb:20.
Too many uses of deprecated 'Filtering by an `:example_group` subhash'. Pass `--deprecation-out` or set `config.deprecation_stream` to a file for full output.

--------------------------------------------------------------------------------
RSpec::Puppet::ManifestMatchers::CreateGeneric implements a legacy RSpec matcher
protocol. For the current protocol you should expose the failure messages
via the `failure_message` and `failure_message_when_negated` methods.
(Used from /home/rnelson0/git/certs/spec/classes/init_spec.rb:5)
--------------------------------------------------------------------------------


If you need more of the backtrace for any of these deprecations to
identify where to make the necessary changes, you can configure
`config.raise_errors_for_deprecations!`, and it will turn the
deprecation warnings into errors, giving you the full backtrace.

5 deprecation warnings total

Finished in 0.03549 seconds (files took 0.38404 seconds to load)
1 example, 0 failures

With no failures, we know that we have working code according to our tests. That’s great, except we haven’t really defined a test. So, let’s look at creating one. When we call certs::vhost with the name “www.example.com” and a source_path of “puppet:///othermodule”, we expect it to create /etc/ssl/certs/www.example.com.{crt,key} files. That’s the first thing we want to test. The rspec-puppet tutorial will help us generate the test.

Certs::vhost is a define, so tests will go in spec/defines/<thing to be tested>_spec.rb, where the thing to be tested has the module namespace stripped off, unless it’s the top level class certs. Create the file now with a basic outline of a unit test:

require 'spec_helper'

describe 'certs::vhost' do
  # Your tests go in here
end

The next bits of code go between the describe and end lines, in the order they are introduced.

First, we define what values are passed to the define, namely the title of the define and the source_path. This is done with the use of let. The title can be set on its own, but source_path is part of the params hash. You’ll notice that tests can be read as English with only a few articles missing. We let the keys have certain values like this:

  let(:title) { 'www.example.com' }
  let(:params) { {
    :source_path => 'puppet:///othermodule'
  } }

It should contain two file resources with known path and source values. Everything relates to either puppet type definitions or those provided by other modules. We can create the tests using almost-English commands and methods:

  it do
    should contain_file('www.example.com.crt').with({
      :path   => '/etc/ssl/certs/www.example.com.crt',
      :source => 'puppet:///othermodule/www.example.com.crt'
    })
  end
  it do
    should contain_file('www.example.com.key').with({
      :path   => '/etc/ssl/certs/www.example.com.key',
      :source => 'puppet:///othermodule/www.example.com.key'
    })
  end

The could should be pretty explanatory. Keep track of your paren, curly brace, and do/end pairs. Save the spec file and run rspec from the top directory of the module:

[rnelson0@build certs]$ rspec
.FF

Failures:

  1) certs::vhost
     Failure/Error: })
     Puppet::Error:
       Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type certs::vhost at line 4 on node build.nelson.va
     # ./spec/defines/vhosts_spec.rb:13

  2) certs::vhost
     Failure/Error: })
     Puppet::Error:
       Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type certs::vhost at line 4 on node build.nelson.va
     # ./spec/defines/vhosts_spec.rb:19
...
3 examples, 2 failures

We expect this to fail, because we haven’t written any code for the actual module. It’s time to start on the module.

Certs::vhost definition

I’m going to skip over documentation of your module, except to say that you should have some, because it’s simply too lengthy for this article. You can see mine at GitHub and plenty of modules, especially the Puppet Approved Modules, have much better documentation (and more rspec tests as well!).

Create manifests/vhost.pp and the skeleton of our define. Remember that we have already determined two parameters, one of which has a default value.

define certs::vhost (
  $source_path = undef,
  $target_path        = '/etc/ssl/certs',
) {

}

If we save our file and  run rspec now, we will see slightly more helpful failure messages:

  1) certs::vhost should contain File[www.example.com.crt] with source => "puppet:///othermodule/www.example.com.crt" and path => "/etc/ssl/certs/www.example.com.crt"
     Failure/Error: })
       expected that the catalogue would contain File[www.example.com.crt]
     # ./spec/defines/vhosts_spec.rb:13

  2) certs::vhost should contain File[www.example.com.key] with source => "puppet:///othermodule/www.example.com.key" and path => "/etc/ssl/certs/www.example.com.key"
     Failure/Error: })
       expected that the catalogue would contain File[www.example.com.key]
     # ./spec/defines/vhosts_spec.rb:19

By defining the tests first, we have described what the define is required to do. Specifically, create two File resources, “title.{crt,key}”, with source and path values of “source_path/title.{crt.key}” and “target_path/title.{crt,key}”, respectively. We are also testing that it’s a file, not a directory, so the ensure value should be “file”. That’s easy enough to define. Remember that the title attribute is also known as name within the define. If you use title, it can get confusing as each resource we use has its own title. Here’s the full content of manifests/vhost.pp:

define certs::vhost (
  $source_path = undef,
  $target_path        = '/etc/ssl/certs',
) {

  file {"$name.crt":
    ensure => file,
    source => "${source_path}/${name}.crt",
    path   => "${target_path}/${name}.crt",
  }
  file {"$name.key":
    ensure => file,
    source => "${source_path}/${name}.key",
    path   => "${target_path}/${name}.key",
  }
}

If we run rspec now, we have 3 successes and 0 failures. Congratulations, we’ve built a simple module using test-driven development!

Bonus Points

There are a lot more tests you should have for our simple module, both in rspec and in the module itself. You can see those tests at GitHub, but see if you can sleuth them out on your own before looking at the answers. I’m new to TDD as well, so I’m sure I missed a few tests. Drop me a comment, or open an issue/PR on GitHub, and let me know what you find!

Summary

We looked at Test-Driven Development today and how we can use it with puppet modules. We built a small module using TDD practices (tests first, then code!). The value of TDD may seem low at the moment, but as complexity rises, the value grows. We are also comfortable that if we tweak our module – even just to adjust formatting – we can run rspec and ensure we didn’t accidentally touch the syntax and break it. Tests can be added to our existing modules – including roles and profiles! – to start providing test coverage until our entire puppet codebase has full coverage. And perhaps most importantly, we can tell our developer coworkers that we understand TDD now!

Sometimes We Break Things

Today’s a no-deploy Friday for me, like it is for many. However, also like many others, here I am deploying things. Small, minor things, but it would ruin my weekend if they broke anyway. Sometimes the worst does happen and we break things. Don’t worry, we’re professionals!

So, what happens if you do break something? First, don’t panic. Everyone’s broken something before, and that includes everyone above you in the food chain. The second step is to notify those above you according to your internal processes. In most cases, that means stopping what you are doing and giving your boss a paragraph summary of the issue, what it affects, and what you’re doing about it, then getting back to work. Third, don’t panic! I know I already said that, but since you’ve now gone and told your boss, they may have induced some panic – let it pass. The only way you’ll recover is if you don’t panic. Breath.

Fourth, fix it! Use your mind to decide what was supposed to happen, what you did, and where things went wrong. Identify the steps required to either back things out or repair the situation so you can proceed. Document the steps and follow them. If you have a maintenance window you are operating under, put some time estimates down and set an alarm for when you need to make the go/no-go call. Though the situation is urgent, taking a few moments now to prepare will make you more efficient as you proceed. Give your management chain short updates throughout the event until it is cleared, and don’t let rising panic get to you.

I love the overnight shift!

Above, I said that everyone has broken something. I’d like to share two stories of some of my most significant failures so you can see that mistakes aren’t only natural, but are learning experiences.

The first situation was in 2001. My boss and I were doing an upgrade to a Novell Netware 5.0 server, either to 5.1 or some patch for 5.0. We had planned the window a few days ago, had made a checklist, read the release notes, opened a pre-emptive support case with Novell, etc. – everything you’re supposed to do to prepare for such a significant change. We started the upgrade at 5:30PM on the dot as per the maintenance notice provided to the users. Up popped the first screen – but it wasn’t what the release notes said would happen. It was some warning about something something could do something to the something else and maybe you’d like to… and I have no idea what else it said, because I gleefully mashed the ‘Y’ key to continue.

My boss’s sole acknowledgement of my impertinence was to turn his head slowly at me, say, “I was reading that.”

We turned back to the server and stared at it for a few minutes, willing the dots to move faster across the screen. When the whole screen was full of dots and there was no sign of when it would finish, we stepped out and went back to our PCs to surf the internet, checking back every 10 minutes. At around 6:30PM, it finished! There was no warning text this time, so I gleefully mashed the ‘Y’ key to reboot the server.

The only problem is, Netware never booted. I don’t remember if it hung or if some error finally displayed, but there was no joy to be found. I think we tried a second reboot and probably a cold boot, but by 7:00PM it was clear, there was no easy fix to be had. Our window was supposed to end at 8:30PM.

“I told you that I was reading that screen,” my boss said as he started putting on his coat. “It was warning us about something, but we don’t know what because you clicked past it. We shouldn’t have done this, but you decided you really wanted to, so I guess you’ll have it fixed by the time I come in tomorrow, or you won’t come in at all.” Then he was gone.

The next 9 hours of my life went by in a blur. Aside from calling my wife and relating what my boss said, I don’t know what I did exactly. But by 4AM, the server was booted and it had the latest patch. I went home, got some sleep, and rolled in around 10AM with shadows under my eyes. No-one said anything about my late arrival, and my boss never said a word to me about my decision again. I remained there for a few years and received great ratings the entire time.

My inbox is empty. That never happened before!

A few years later, we were moving, so I put in my two week notice. On my last Friday at the company, I decided to help my coworkers by running the email maintenance program. We were using a DOS-based predecessor to GroupWise and it was now 2002. A lot of users had large mailboxes (some of you may see where this is going already) and sometimes things got slow. I kicked off the maintenance program and walked away.

A couple of minutes later, the phones started ringing, with a common theme. People were having issues with their email. Suddenly, a light bulb went on in my head. I ran to the console where the maintenance program was running and starting mashing ctrl-C as hard as I could. See, there was a slight problem with this old DOS-based application: It didn’t like mailboxes over 2G in size. The email client ran fine, but that was it. The maintenance script would chug along on mailboxes just fine, cleaning up broken references and reclaiming empty space from each mailbox’s database file, until it ran into a database file over 2G. Once that happened, you were in uncharted waters since the program didn’t support that. Here be dragons! Sometimes it would just muck up the database and clean some extra emails, othertimes, it would delete EVERYTHING. I never saw such pristine inboxes! The worst part is … this was the second time we encountered it. I had just forgotten about it.

Thankfully, we had great backups and I caught the program relatively early, meaning we didn’t have to restore that much – and the tapes were still on-site so we didn’t have to wait, either. Later in the day, the big boss called me into his office to give me my recommendation letters. I must have been nervous, because he asked me to review it and I told him it looked great without really reading it. He asked me a few times if I was sure before letting out a big belly laugh. He had added a little flavor to his recommendation. It took a little bit before he could compose himself and hand me the real letters :)

recommendation letter

As you can see, I still keep the letter around. It’s hilarious (now!), but it’s also a great reminder to pay more attention to what you’re doing, especially on your last day at a company! These two learning experiences taught me not only how to better plan to avoid such “teachable moments”, but also that I could recover from them with confidence.

Please, share your own “learning experiences” in the comments!

Don’t Disable SELinux, Part 2

Yesterday I warned everyone not to disable SELinux because the fix is almost always a quick one. But, what do you do if there is no selboolean that fixes your problem with a simple one liner?

After yesterday’s article, Tim Meusel shared a message he receives in his audit log when running nginx on his puppet master with SELinux in enforce mode:

type=AVC msg=audit(1415871389.171:787): avc:  denied  { name_connect }
 for  pid=2228 comm="nginx" dest=8080
 scontext=system_u:system_r:httpd_t:s0
 tcontext=system_u:object_r:http_cache_port_t:s0 tclass=tcp_socket
type=SYSCALL msg=audit(1415871389.171:787): arch=c000003e syscall=42
 success=no exit=-13 a0=19 a1=259e2b0 a2=10 a3=7fffdac559d0 items=0
 ppid=2227 pid=2228 auid=4294967295 uid=996 gid=995 euid=996 suid=996
 fsuid=996 egid=995 sgid=995 fsgid=995 tty=(none) ses=4294967295
 comm="nginx" exe="/usr/sbin/nginx" subj=system_u:system_r:httpd_t:s0
 key=(null)

That’s…. that’s ugly. The important parts have been highlighted. Nginx cannot talk to the tcp_socket at /var/run/puppet/puppetmaster_unicorn.sock. There doesn’t appear to be a selboolean that matches the issue. You could try flipping semi-relevant booleans for hours till you stumble upon some combination that may work, undoubtedly with side effects, and possibly never find the right combination. That could end up being a LOT of time wasted without any guarantee of success.

Instead, use audit2allow. By providing the tool with portions of an audit log, it will build an SELinux policy that will allow everything marked as “denied”. Here’s an example of generating a policy for review, then generating and applying that policy:

grep nginx /var/log/audit/audit.log | audit2allow > nginx.te
more nginx.te
grep nginx /var/log/audit/audit.log | audit2allow -M nginx
semodule -i nginx.pp

You can find more detail on the tool on the web, particularly this article where another nginx user is struggling with SELinux. You may have to repeat this process a few times – nginx stopped running when it failed to attach to the socket, so there could be other SELinux permission issues it would encounter if it had not failed. You won’t see those in the audit.log until it gets past the socket. Keep at it until audit2allow is building the same policy file on consecutive runs, at which point there are no new failures to discover. Your application should be fully working now and encounter no more SELinux permission issues.

Update: Tim continued to struggle after he performed the above steps until he moved the unicorn socket out of /var/run (which is admittedly not the recommended location!) even though he wasn’t seeing any more failures in the audit log. This command forces SELinux to log all failure events and then the new failures showed up and were processed by audit2allow:

semodule --disable_dontaudit --build

See Tim’s blog for more info.

You can apply the policy via puppet using the selmodule type, plus a file resource to put the .pp file in the correct location.

While this takes a lot longer to resolve than touching some selbooleans, you should only have to do it once. This ensures you still have the protections of SELinux and a well defined policy state for your application. If, and only if, this doesn’t resolve your issue, should you even entertain the thought of disabling SELinux, as a temporary resolution until a permanent solution is found.

Don’t Disable SELinux

When developing new web-based solutions on modern Linux distros, inevitably you’ll run into a fun issue – like your webserver throwing database errors when there’s not even any traffic making it to out of the server toward the database – and bang your head against the desk a bit. If you google for the error, you’ll run into the worst advise possible: “If your problem is not solved then disable your SELinux.” That’s right, just disable the whole thing because one part bothers you. The only positive part of this advise is that you may not have even though to look at SELinux before that.

You can verify that SELinux is the issue by taking a look at the audit log (tail -f /var/log/audit/audit.log) and using your web application. You’ll see a ton of crap that is simply undecipherable to human beings. What you’re looking for is the word denied and the application, file, or user that is having an issue. Here’s a deny for the application httpd when trying to talk to that remote database:

type=AVC msg=audit(1415813628.801:628): avc:  denied  { name_connect } for  pid=11911 comm="httpd"
 dest=3306 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:mysqld_port_t:s0
 tclass=tcp_socket

The next step is to narrow the issue down. There are a large number of settings for SELinux, known as SELinux Booleans, that may be affecting your application. Take a quick gander at them, find the most likely boolean, set the value to on, and try your application again. If it doesn’t work, set it to off and try another. Here’s a Tips and Trick page that describes the process in more detail and provides a pretty thorough list of booleans. Can’t access files on an NFS share via httpd? Set httpd_use_nfs to true. Talking to a remote database as above? That’s httpd_can_network_connect_db. This is just as simple and more beneficial than disabling SELinux altogether.

Of course, I’d be remiss if I just told you to use setsebool as root. You need to including this setting in your application definition. For example, integrate the setting into your puppet manifests with the selboolean type. Set the value to on and persistent to true. Apply your manifest and getsebool will show the new value. Here’s an example of a manifest I built for the phpMyAdmin application (filenames in the header since you can’t put slashes in gist filenames; provide your own profile::base class), specifically lines 13-16 where the selbooleans are set. If you’re using a different configuration management tool, you’ll have to do this part yourself, the important part is that you capture the setting.

Take a few minutes to learn how to use SELinux, so you’re aware of when you’re barking up the wrong tree and how to resolve issues, and integrate your findings into your application’s state definition. You’ll benefit by leaving the protection in place.