Recently, I started setting up my puppet modules with Travis CI. One thing that bothered me was that once I added a new version to the matrix, it had to pass with that version or the PR would have a red mark. Today, I found out how to ignore those new versions with the allow_failures key in the matrix hash of .travis.yml. For instance, if I am currently testing up to Puppet 3.7.0 and I to start testing with 3.7.0 future parser and 4.0.0, I add the new versions to the env hash and the matrix‘s allow_failures key. The new lines are in bold:

---
sudo: false
language: ruby
branches:
  only:
    production
bundler_args: --without development system_tests
script: "cd dist/profile && bundle exec rake test"
notifications:
  email: false
rvm:
  - 1.9.3
  - 2.0.0
  - 2.1.0
env:
  - PUPPET_GEM_VERSION="~> 3.3.0"
  - PUPPET_GEM_VERSION="~> 3.4.0"
  - PUPPET_GEM_VERSION="~> 3.5.0" STRICT_VARIABLES=yes
  - PUPPET_GEM_VERSION="~> 3.6.0" STRICT_VARIABLES=yes
  - PUPPET_GEM_VERSION="~> 3.7.0" STRICT_VARIABLES=yes
  - PUPPET_GEM_VERSION="~> 3.7.0" STRICT_VARIABLES=yes FUTURE_PARSER=yes
  - PUPPET_GEM_VERSION="~> 4.0.0" STRICT_VARIABLES=yes
matrix:
  exclude:
  # Ruby 2.1.0
  - rvm: 2.1.0
    env: PUPPET_GEM_VERSION="~> 3.2.0"
  - rvm: 2.1.0
    env: PUPPET_GEM_VERSION="~> 3.3.0"
  - rvm: 2.1.0
    env: PUPPET_GEM_VERSION="~> 3.4.0"
  allow_failures:
    - env: PUPPET_GEM_VERSION="~> 4.0.0" STRICT_VARIABLES=yes
    - env: PUPPET_GEM_VERSION="~> 3.7.0" STRICT_VARIABLES=yes FUTURE_PARSER=yes

Now, when Travis CI runs, it will add the new versions to the match and will mark the designated versions separately as Allowed Failures. You can see this in action with puppetinabox/controlrepo build #22, which contains expected failures and so is considered a passing build. I can now develop against my supported versions and test against new versions, without affecting my test results. This has solved a pretty big frustration for me. Hopefully, it helps you as well!