#Puppetinabox moving to v4 in 2016

I recently used Travis CI to help me get all my puppet modules and my controlrepo ready for Puppet v4. I have one dependent module (ajjahn/dchp) that needs a few polishing touches (issue #7) and then I plan to start moving PuppetInABox to version 4 as well. There are many moving parts but I would like to get this done in the first quarter of 2016.

One thing I will need to do is convert the master’s service from apache/passenger to puppetserver. Unfortunately, stephenrjohnson/puppet does NOT support puppetserver or puppet v4 yet. There are a number of forge modules that provide some level of support for puppetserver and I could use your help in finding the right one. It would seem the clear winner, at least by downloads, is camptocamp/puppetserver. Maybe one of the others is better. Perhaps the stephenrjohnson/puppet module is nearing readiness for version 4.

What are you using, and what tips do you have for someone converting from version 3 to 4? Drop me a line in the comments or reach out on twitter. Thanks!

Modern rspec-puppet practices

I’ve written a bit about rspec-puppet in the past (directly here, here, and here, and indirectly here and here). The state of rspec-puppet has changed over the past year and change, though. Let’s see if we can collect the current practices in one place.

First, there’s a better way to deploy rspec-puppet than I wrote about earlier. If you’re implementing rspec-puppet brand new today, I suggest starting with puppet-module-skeleton as your base. If you’re upgrading an existing setup, take a closer look at the skeleton’s Gemfile, Rakefile, .rspec, .rubocop, spec/spec_helper.rb, /spec/spec_helper_acceptance.rb.erb, spec/classes/coverage_spec.rb, and spec/classes/example_spec.rb.erb and bring in the updates that are relevant. If you want to use hiera for your testing, I suggest adding these lines to spec/spec_helper.rb (PR submitted).

By using the Gemfile from puppet-module-skeleton, you can now use bundler to manage your gems instead of installing them globally on your nodes. Run bundle install first. You can then run the full suite of tests with bundle exec rake test, or simply run rspec with bundle exec rake spec_prep and rake spec_standalone. If you get into trouble, you can use git clean -fdx to remove all the untracked files, mostly from rspec fixtures.

Next is writing the tests. Writing rspec tests can be done with a few different matchers, but it { is_expected.[not_]to contain_<resource>(‘<resourcename>’) } is a common format. Of course, the officially documented matchers work just fine, though you may not see them in the wild very much. As for the tests themselves, always be sure that you are testing your code, and only your code. If you’re including a component module in your profile module, test that the component module is installed, but don’t test the component module’s resources – the module should have its own rspec tests. If it does not, add those tests to the component, not your profile. And of course, do not test Puppet itself – it also has rspec tests!

I do suggest using hiera instead of the :params symbol, though. This can help you design your hiera data while you work on rspec tests, to make sure you know what the hiera data should look like, not just what you think it should look like. Second, I think it is easier to set an additional top-level fact once, in the top describe block, than to set numerous params in each context block, though I believe you have to for defined types.

For general rspec guidance, check out Better Specs. It’s very helpful whether you’re writing tests for puppet or something else. I’ve also created a repository to track great examples of puppet module tests, broken down by area, called puppet-reference_modules. If you see a great example, please submit a PR!

These are the common practices as I understand and observe them. Is there anything I’ve missed or gotten wrong? Let me know!

Troubleshooting Hiera from the CLI

Sometimes it’s very difficult to see how hiera works and what values it might return. To help with troubleshooting, you can use the hiera cli to do lookups yourself. You obviously need to do this on a puppet master, or a node that’s configured to have all the hiera data in the same place the masters do. Here’s the hiera.yaml configuration:

# managed by puppet
---
:backends:
  - yaml
:logger: console
:hierarchy:
  - "clientcert/%{clientcert}"
  - "puppet_role/%{puppet_role}"
  - global

:yaml:
   :datadir: /etc/puppet/environments/%{environment}/hiera

Tier 1: global

Let’s look up the ntp::servers key from the global tier. One thing to note here is that the datadir includes the variable environment, so we will need to provide that tuple to the cli, like so:

Continue reading

Update your Puppet modules on the Forge quickly with Blacksmith

This weekend, I’ve been playing around with getting my puppet modules up to par with puppet v4. I’ve also recently covered a lot of ground with Rakefiles/Travis CI and excluding files from the build. Building on that, I learned about Blacksmith and how it can let you quickly push new versions of your modules to the forge. It’s really simple.

First, you need an auth file. This belongs to your user, not the module repo, so there’s no chance it will make it into your git repo. You should still be sure it’s not readable to anyone but you, though. The syntax is very simple, just drop this at ~/.puppetforge.yml and replace the defaults with your user/password:

---
url: https://forgeapi.puppetlabs.com
username: myuser
password: mypassword

Next, make sure your module’s Rakefile and Gemfile are set up properly.

$ cat Rakefile
# Relevant snippet
# These gems aren't always present, for instance
# on Travis with --without development
begin
  require 'puppet_blacksmith/rake_tasks'
rescue LoadError # rubocop:disable Lint/HandleExceptions
end

$ cat Gemfile
# Relevant snippet
group :development do
  gem "travis"
  gem "travis-lint"
  gem "puppet-blacksmith"
  gem "guard-rake"
end

Once you use bundler to install the gems locally, you will find some new rake tests available:

$ be rake -T | grep module
rake build                                   # Build puppet module package
rake clean                                   # Clean a built module package
rake module:bump                             # Bump module version to the next patch
rake module:bump:major                       # Bump module version to the next MAJOR version
rake module:bump:minor                       # Bump module version to the next MINOR version
rake module:bump:patch                       # Bump module version to the next PATCH version
rake module:bump_commit                      # Bump version and git commit
rake module:clean                            # Runs clean again
rake module:dependency[module_name,version]  # Set specific module dependency version
rake module:push                             # Push module to the Puppet Forge
rake module:release                          # Release the Puppet module, doing a clean, build, tag, push, bump_commit and git push
rake module:tag                              # Git tag with the current module versio

If your module is ready to go and your authentication works, you just need to do a rake module:release. This will run the following commands (useful if one errors out and you need to replay from there):

bundle exec rake module:clean
bundle exec rake build
bundle exec rake module:tag
bundle exec rake module:push
bundle exec rake module:bump_commit
git push origin

You can of course use the target module:release, or use the independent targets. For instance, I wanted to make sure my build wasn’t 21M again, so I did the bump_commit/clean/build steps first, then the tag/push/git push:

[rnelson0@build02 local_user:master]$ be rake module:bump_commit
Bumping version from 1.0.1 to 1.0.2
[rnelson0@build02 local_user:master]$ be rake build
[rnelson0@build02 local_user:master]$ ls -lh pkg
total 12K
drwxrwxr-x. 5 rnelson0 rnelson0 4.0K Nov 15 17:21 rnelson0-local_user-1.0.2
-rw-rw-r--. 1 rnelson0 rnelson0 7.1K Nov 15 17:21 rnelson0-local_user-1.0.2.tar.gz
[rnelson0@build02 local_user:master]$ be rake module:tag
[rnelson0@build02 local_user:master]$ be rake module:push
Uploading to Puppet Forge rnelson0/local_user
[rnelson0@build02 local_user:master]$ git push origin master
Counting objects: 5, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 317 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
To git@github.com:rnelson0/puppet-local_user
   87a1957..13a66bb  master -> master

For verification, I visit my module on the forge and the latest version now says 1.0.2. Perfect! I know how it works, in case something breaks in the future, and I’m confident that I can release the next version using the module:release target. Thank you to the people at maestrodev and the many  contributors for this awesome tool!

Exclude files from a puppet module release

I ran into a funny problem updating my rnelson0/certs puppet module today. I tried to use puppet-blacksmith to create and push a new build to the Forge. It continually failed with a strange error:

[rnelson0@build02 certs:master]$ be rake module:release
Cleaning for module build
Uploading to Puppet Forge rnelson0/certs
rake aborted!
Errno::ECONNRESET: Connection reset by peer

Well crap, that doesn’t make any sense! Let’s do a build manually and take a look at it:

[rnelson0@build02 certs:master]$ puppet module build
Notice: Building /home/rnelson0/modules/certs for release
Module built: /home/rnelson0/modules/certs/pkg/rnelson0-certs-0.7.0.tar.gz
[rnelson0@build02 certs:master]$ ls -lh pkg
total 21M
drwxrwxr-x. 6 rnelson0 rnelson0 4.0K Nov 15 04:16 rnelson0-certs-0.7.0
-rw-rw-r--. 1 rnelson0 rnelson0  21M Nov 15 04:16 rnelson0-certs-0.7.0.tar.gz
[rnelson0@build02 certs:master]$ tar tzvf pkg/rnelson0-certs-0.7.0.tar.gz | more
drwxrwxr-x rnelson0/rnelson0 0 2015-11-15 04:16 rnelson0-certs-0.7.0/
drwxrwxr-x rnelson0/rnelson0 0 2015-11-15 04:16 rnelson0-certs-0.7.0/tests/
-rw-rw-r-- rnelson0/rnelson0 509 2015-10-24 19:27 rnelson0-certs-0.7.0/tests/init.pp
-rw-rw-r-- rnelson0/rnelson0 911468 2015-11-15 04:16 rnelson0-certs-0.7.0/checksums.json
drwxrwxr-x rnelson0/rnelson0      0 2015-11-15 04:16 rnelson0-certs-0.7.0/manifests/
-rw-rw-r-- rnelson0/rnelson0    310 2015-10-24 17:27 rnelson0-certs-0.7.0/manifests/init.pp
-rw-rw-r-- rnelson0/rnelson0   2208 2015-10-24 17:27 rnelson0-certs-0.7.0/manifests/vhost.pp
drwxrwxr-x rnelson0/rnelson0      0 2015-11-15 04:16 rnelson0-certs-0.7.0/spec/
drwxrwxr-x rnelson0/rnelson0      0 2015-11-15 04:16 rnelson0-certs-0.7.0/spec/classes/
-rw-rw-r-- rnelson0/rnelson0    141 2015-10-24 19:27 rnelson0-certs-0.7.0/spec/classes/init_spec.rb
drwxrwxr-x rnelson0/rnelson0      0 2015-11-15 04:16 rnelson0-certs-0.7.0/spec/defines/
-rw-rw-r-- rnelson0/rnelson0   1433 2015-10-24 19:27 rnelson0-certs-0.7.0/spec/defines/vhost_spec.rb
-rw-rw-r-- rnelson0/rnelson0     52 2015-10-24 17:27 rnelson0-certs-0.7.0/spec/spec_helper.rb
drwxrwxr-x rnelson0/rnelson0      0 2015-11-15 04:16 rnelson0-certs-0.7.0/spec/fixtures/
drwxrwxr-x rnelson0/rnelson0      0 2015-11-15 04:16 rnelson0-certs-0.7.0/spec/fixtures/manifests/
drwxrwxr-x rnelson0/rnelson0      0 2015-11-15 04:16 rnelson0-certs-0.7.0/spec/fixtures/modules/
-rw-rw-r-- rnelson0/rnelson0   1409 2015-10-24 19:27 rnelson0-certs-0.7.0/Rakefile
-rw-rw-r-- rnelson0/rnelson0   2208 2015-11-15 02:58 rnelson0-certs-0.7.0/README.md
drwxrwxr-x rnelson0/rnelson0      0 2015-11-15 04:16 rnelson0-certs-0.7.0/vendor/
drwxrwxr-x rnelson0/rnelson0      0 2015-11-15 04:16 rnelson0-certs-0.7.0/vendor/ruby/

Can you see the problem? The vendor directory used by bundler is part of the package. That’s not right! It’s causing the forge connection to timeout because of the size or transmission time. Plus, no-one’s going to download a 21M module. Thankfully, there’s an easy way to fix this. In addition to the .gitignore that we use to prevent files from being made part of the git repository, we also need a .pmtignore, but we only need to exclude the vendors path:

[rnelson0@build02 certs:master]$ cat > .pmtignore
vendor/
[rnelson0@build02 certs:master±]$ puppet module build
Notice: Building /home/rnelson0/modules/certs for release
Module built: /home/rnelson0/modules/certs/pkg/rnelson0-certs-0.7.0.tar.gz
[rnelson0@build02 certs:master±]$ ls -lh pkg
total 12K
drwxrwxr-x. 5 rnelson0 rnelson0 4.0K Nov 15 04:23 rnelson0-certs-0.7.0
-rw-rw-r--. 1 rnelson0 rnelson0 7.0K Nov 15 04:23 rnelson0-certs-0.7.0.tar.gz
[rnelson0@build02 certs:master±]$ tar tzvf pkg/rnelson0-certs-0.7.0.tar.gz
drwxrwxr-x rnelson0/rnelson0 0 2015-11-15 04:23 rnelson0-certs-0.7.0/
drwxrwxr-x rnelson0/rnelson0 0 2015-11-15 04:23 rnelson0-certs-0.7.0/tests/
-rw-rw-r-- rnelson0/rnelson0 509 2015-10-24 19:27 rnelson0-certs-0.7.0/tests/init.pp
-rw-rw-r-- rnelson0/rnelson0 631 2015-11-15 04:23 rnelson0-certs-0.7.0/checksums.json
-rw-rw-r-- rnelson0/rnelson0 10143 2015-11-15 03:42 rnelson0-certs-0.7.0/Gemfile.lock
drwxrwxr-x rnelson0/rnelson0     0 2015-11-15 04:23 rnelson0-certs-0.7.0/manifests/
-rw-rw-r-- rnelson0/rnelson0   310 2015-10-24 17:27 rnelson0-certs-0.7.0/manifests/init.pp
-rw-rw-r-- rnelson0/rnelson0  2208 2015-10-24 17:27 rnelson0-certs-0.7.0/manifests/vhost.pp
drwxrwxr-x rnelson0/rnelson0     0 2015-11-15 04:23 rnelson0-certs-0.7.0/spec/
drwxrwxr-x rnelson0/rnelson0     0 2015-11-15 04:23 rnelson0-certs-0.7.0/spec/classes/
-rw-rw-r-- rnelson0/rnelson0   141 2015-10-24 19:27 rnelson0-certs-0.7.0/spec/classes/init_spec.rb
drwxrwxr-x rnelson0/rnelson0     0 2015-11-15 04:23 rnelson0-certs-0.7.0/spec/defines/
-rw-rw-r-- rnelson0/rnelson0  1433 2015-10-24 19:27 rnelson0-certs-0.7.0/spec/defines/vhost_spec.rb
-rw-rw-r-- rnelson0/rnelson0    52 2015-10-24 17:27 rnelson0-certs-0.7.0/spec/spec_helper.rb
drwxrwxr-x rnelson0/rnelson0     0 2015-11-15 04:23 rnelson0-certs-0.7.0/spec/fixtures/
drwxrwxr-x rnelson0/rnelson0     0 2015-11-15 04:23 rnelson0-certs-0.7.0/spec/fixtures/manifests/
drwxrwxr-x rnelson0/rnelson0     0 2015-11-15 04:23 rnelson0-certs-0.7.0/spec/fixtures/modules/
-rw-rw-r-- rnelson0/rnelson0  1409 2015-10-24 19:27 rnelson0-certs-0.7.0/Rakefile
-rw-rw-r-- rnelson0/rnelson0  2208 2015-11-15 02:58 rnelson0-certs-0.7.0/README.md
-rw-rw-r-- rnelson0/rnelson0   861 2015-10-24 19:27 rnelson0-certs-0.7.0/Gemfile
-rw-rw-r-- rnelson0/rnelson0   506 2015-11-15 04:23 rnelson0-certs-0.7.0/metadata.json
[rnelson0@build02 certs:master±]$

I’m going to submit a PR against puppet-module-skeleton to get that file added for the future (PR95).

Adding a gem to your project with bundler

Now that we know the basics of how to use bundler, let’s put our new knowledge to good use by adding a gem to a project. We’ll look at how to add generate-puppetfile to a puppet Controlrepo, but you can add any gem to any project you’d like. It’s very simple.

Start by cloning your project and checking out a new branch:

[rnelson0@build02 controlrepo:production]$ git checkout -b generate-puppetfile
Switched to a new branch 'generate-puppetfile'

Open up the existing Gemfile. If you don’t have one, you just need the source statement followed by one or more gems.

[rnelson0@build02 controlrepo:generate-puppetfile]$ cat Gemfile
source 'https://rubygems.org'

group :development, :test do
  gem 'json', :require => false
  gem 'metadata-json-lint', :require => false
  gem 'puppetlabs_spec_helper', :require => false
  gem 'puppet-lint', :require => false
  gem 'rake', :require => false
  gem 'rspec-puppet', :require => false
end

if puppetversion = ENV['PUPPET_GEM_VERSION']
  gem 'puppet', puppetversion, :require => false
else
  gem 'puppet', :require => false
end

# vim:ft=ruby

Add the gem (generate-puppetfile) to the file:

[rnelson0@build02 controlrepo:generate-puppetfile]$ cat Gemfile
source 'https://rubygems.org'

group :development, :test do
  gem 'json', :require => false
  gem 'metadata-json-lint', :require => false
  gem 'puppetlabs_spec_helper', :require => false
  gem 'puppet-lint', :require => false
  gem 'rake', :require => false
  gem 'rspec-puppet', :require => false
  gem 'generate-puppetfile'
end

if puppetversion = ENV['PUPPET_GEM_VERSION']
  gem 'puppet', puppetversion, :require => false
else
  gem 'puppet', :require => false
end

# vim:ft=ruby

Now run bundle install:

[rnelson0@build02 controlrepo:generate-puppetfile]$ bundle install --path vendor --without system_tests
Fetching gem metadata from https://rubygems.org/.........
Fetching version metadata from https://rubygems.org/..
Installing rake 10.4.2
Installing diff-lcs 1.2.5
Installing facter 2.4.4
Installing generate-puppetfile 0.9.6
Installing json_pure 1.8.2
Installing hiera 1.3.4
Installing json 1.8.3 with native extensions
Installing metaclass 0.0.4
Installing spdx-licenses 1.0.0
Installing metadata-json-lint 0.0.11
Installing mocha 1.1.0
Installing puppet 3.7.3
Installing puppet-lint 1.1.0
Installing puppet-syntax 2.0.0
Installing rspec-core 2.99.2
Installing rspec-expectations 2.99.2
Installing rspec-mocks 2.99.4
Installing rspec 2.99.0
Installing rspec-puppet 2.2.0
Installing puppetlabs_spec_helper 0.10.3
Using bundler 1.10.6
Bundle complete! 8 Gemfile dependencies, 21 gems now installed.
Gems in the group system_tests were not installed.
Bundled gems are installed into ./vendor.

And now you can use the gem in your project with bundle exec, without installing it globally:

[rnelson0@build02 controlrepo:generate-puppetfile]$ bundle exec generate-puppetfile -v
generate-puppetfile v0.9.6

At this point, it may be worth adding an alias to our shell:

[rnelson0@build02 controlrepo:generate-puppetfile]$ alias be
alias be='bundle exec'

Enjoy!

Introducing generate-puppetfile, or Creating a ruby program to update your Puppetfile and .fixtures.yml

About a month ago, I whipped up simple shell script that would download some puppet modules and then generate a valid snippet of code you could insert into your Puppetfile. This was helpful when you wanted to add a single module but it had dependencies, and then those dependencies had dependencies, and then 30 minutes go by and you have no idea what it was you set out to do in the beginning. As I started using it, I realized this was only half the battle – I might already have some of those dependencies in my Puppetfile and adding the same modules again doesn’t work.

So I started adding to the script and quickly realized a shell script was not sufficient. About three weeks ago, I decided to convert it to a ruby program and add cli arguments to support all the new features I wanted and that some users were requesting. I had a few problems I knew I needed to solve, namely how to parse an existing Puppetfile and pull out the existing forge modules, how to combine that and any non-forge module data with the new module list and generate a new file, and how to generate a .fixtures.yml file. I also ended up with a boatload of problems that I didn’t know I needed to solve.

Continue reading

Configuring Travis CI on a Puppet Module Repo

Recently we looked at enabling Travis CI on the Controlrepo. Today, we’re going to do the same for a module repo. We’re going to use much of the same logic and files, just tweaking things a bit to fit the slightly different file layout and perhaps changing the test matrix a bit. If you have not registered for Travis CI yet, go ahead and take care of that (public or private) before continuing.

The first challenge is to decide if you’re going to enable Travis CI with an existing module, or a new module. Since a new module is probably easier, let’s get the hard stuff out of the way.

Set up an existing module

I have an existing module rnelson0/certs which has no CI but does have working rspec tests, a great candidate for today’s efforts. Let’s make sure the tests actually work, it’s easy to make incorrect assumptions:

modules travis ci fig 1

Continue reading

Configuring Travis CI on your Puppet Controlrepo

Continuous Integration is an important technique used in modern software development. For every change, a CI system runs a suite of tests to ensure the whole system – not just the changed portion – still “works”, or more specifically, still passes the defined tests. We are going to look at Travis CI, a cloud-based Continuous Integration service that you can connect to your GitHub repositories. This is valuable because it’s free (for “best effort” access; there are paid plans as well.) and helps you guarantee that code you check in will work with Puppet. This isn’t a substitute or replacement for rspec-puppet, this is another layer of testing that improves the quality of our work.

There are plenty of other CI systems out there – Jenkins and Bamboo are popular – but that would involve setting up the CI system as well as configuring our repo to use CI. Please feel free to investigate these CI systems, but they’ll remain beyond the scope of this blog for the time being. Please share any guides you may have in the comments, though!

Travis CI works by spinning up a VM or docker instance, cloning our git repo (using tokenized authentication), and running the command(s) we provide. Each entry in our test matrix will run on a separate node, so we can test different OSes or Ruby or Puppet versions to our heart’s content. The results of the matrix are visible through GitHub and show us red if any test failed and green if all tests passed. We’ll look at some details of how this works as we set up Travis CI.

From a workflow perspective, you’ll continue to create branches on your controlrepo and submit PRs. The only additional step is that when a PR is ready for review, you’ll want to wait for Travis CI to complete first. If it’s red, investigate the failure and remediate it. Don’t review code until everything is green because it won’t work anyway. This will mostly be a time saver, unless you’re watching your CI run which of course makes it slower!

Continue reading

#PuppetConf 2015 Wrap-Up

I mentioned over the spring/summer that I was headed to PuppetConf 2015, which happened last week. It was a blast! I highly recommend that if you use Puppet, you find a way to make it to PuppetConf 2016 which will be held in San Diego.

There were a lot of great events, official and unofficial, throughout the week. I met a ton of people, way too many to mention individually, and made a lot of friends. I live tweeted three of the event days, which are storified, and here are some highlights:

Contributor’s Summit: This is a great opportunity to become involved in the community. You can contribute docs, code, or commentary. I’m serious about the last, lots more time was spent on designing than coding things. A few of us – Henrik, Felix, Vanessa, and myself – sat down to attack HI-118 and created something. Plenty of other people and groups created their own things. I saw lots of ways to do the same things and also the awesome puppet-retrospec, which creates rspec-puppet tests for all .pp code in a module. It’s very naive at this point, but it’s better than not having tests!

Sessions, Day One: At the keynote, Puppet’s new Application Orchestration was unleashed. This is seriously awesome. Define your application’s microservices, then assign nodes to provide the services. Need multiple nodes for a service? Assign more than one. Want a node to provide more than one service – say, a single SQL server that serves more than one database? Assign multiple services to that node. It’s pretty simple but pretty powerful. Of course, we only got to check out some demos in the demo section of the Exhibit Floor, but it’s very promising.

I attended a number of sessions, of course:

  • State of the Puppet Community – Kara Sowles & Meg Hartley, Puppet Labs
  • 200,000 Lines Later: Our Journey to Manageable Puppet Code – David Danzilio, Constant Contact
  • Infrastructure Security: How Hard Could it Be, Right? – Ben Hughes, Etsy
  • Identity: LGBTQ in Tech – Daniele Sluijters, Spotify
  • Hacking Types and Providers – Introduction and Hands-On – Felix Frank, mpex GmbH

Sessions, Day Two: Today’s keynote showed a bit more of the Application Orchestrator but also focused on the speed and capabilities of some C++ prototypes for facter and puppet. They’re blazing fast. I also spoke on Puppetizing Your Organization! That was terrifying but rewarding. If you have something to share, PuppetConf is the place, it’s extremely friendly and rewarding. Here are the sessions I attended:

  • Thriving in Bureaucratic Environments – Ashley Hathaway, IBM Watson
  • Application Modeling Patterns – David Lutterkort & Ryan Coleman, Puppet Labs
  • Building Communities – Byron Miller, HomeAway.com

After the last session, I had to head home immediately and take a red-eye. I missed out on the pub crawl and some of the after activities, but I had a great time while I was there. Hello to everyone I met there, thanks to everyone who contributed to my presentation and made it that much better, and especially thanks to everyone who showed up to my talk! Hopefully I’ll see you all in San Diego next October!

Update: I forgot to mention how great the Oregon Conference Center was. By far one of the most organized conferences I’ve been to and absolutely the best catered foods.