Allowing expected failures with Travis CI

Recently, I started setting up my puppet modules with Travis CI. One thing that bothered me was that once I added a new version to the matrix, it had to pass with that version or the PR would have a red mark. Today, I found out how to ignore those new versions with the allow_failures key in the matrix hash of .travis.yml. For instance, if I am currently testing up to Puppet 3.7.0 and I to start testing with 3.7.0 future parser and 4.0.0, I add the new versions to the env hash and the matrix‘s allow_failures key. The new lines are in bold:

---
sudo: false
language: ruby
branches:
  only:
    production
bundler_args: --without development system_tests
script: "cd dist/profile && bundle exec rake test"
notifications:
  email: false
rvm:
  - 1.9.3
  - 2.0.0
  - 2.1.0
env:
  - PUPPET_GEM_VERSION="~> 3.3.0"
  - PUPPET_GEM_VERSION="~> 3.4.0"
  - PUPPET_GEM_VERSION="~> 3.5.0" STRICT_VARIABLES=yes
  - PUPPET_GEM_VERSION="~> 3.6.0" STRICT_VARIABLES=yes
  - PUPPET_GEM_VERSION="~> 3.7.0" STRICT_VARIABLES=yes
  - PUPPET_GEM_VERSION="~> 3.7.0" STRICT_VARIABLES=yes FUTURE_PARSER=yes
  - PUPPET_GEM_VERSION="~> 4.0.0" STRICT_VARIABLES=yes
matrix:
  exclude:
  # Ruby 2.1.0
  - rvm: 2.1.0
    env: PUPPET_GEM_VERSION="~> 3.2.0"
  - rvm: 2.1.0
    env: PUPPET_GEM_VERSION="~> 3.3.0"
  - rvm: 2.1.0
    env: PUPPET_GEM_VERSION="~> 3.4.0"
  allow_failures:
    - env: PUPPET_GEM_VERSION="~> 4.0.0" STRICT_VARIABLES=yes
    - env: PUPPET_GEM_VERSION="~> 3.7.0" STRICT_VARIABLES=yes FUTURE_PARSER=yes

Now, when Travis CI runs, it will add the new versions to the match and will mark the designated versions separately as Allowed Failures. You can see this in action with puppetinabox/controlrepo build #22, which contains expected failures and so is considered a passing build. I can now develop against my supported versions and test against new versions, without affecting my test results. This has solved a pretty big frustration for me. Hopefully, it helps you as well!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s