RSpec Example Filtering for Multiple Version Testing

Authoring a client library for an upstream service is often challenging, but testing and preventing regressions against upstream API changes is sometimes impossible. This post discusses using RSpec example metadata filtering as a way to test against different client libraries on Travis CI.

The Problem

I am one of the maintainers of the HashiCorp Vault Ruby API client library. The library tries to maintain compatibility with past versions of Vault (within reason), while supporting the new features of upcoming clients.

At first, I configured the .travis.yml to accept a list of VAULT_VERSION environment variables, which established a build matrix:

env:
  - VAULT_VERSION=0.6.1
  - VAULT_VERSION=0.5.3
  - VAULT_VERSION=0.4.1
  - VAULT_VERSION=0.3.1

rvm:
  - 2.1
  - 2.2.5
  - 2.3.1

before_install:
  - wget -O vault.zip -q https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip
  - unzip vault.zip
  - mkdir ~/bin
  - mv vault ~/bin
  - export PATH="~/bin:$PATH"

This matrix allows me to easily test against multiple Ruby versions and multiple Vault versions simultaneously, and it worked great for a fairly long time. But then the inevitable happened - Vault added a new API. This API was backwards compatible since it was additive, but it was not present in earlier versions of Vault.

Without thinking, I add the new feature, write a test, and submit a Pull Request, but everything is red except the latest version of Vault. After pontificating, I decided this made sense, since that API path did not exist in prior version of Vault.

Novice me would have done something like this:

it "does the thing that only exists in the new API" do
  if vault_version >= "0.1.0" do
    expect(client.new_method).to eq("awesome")
  else
    skip "not supported on earlier versions of Vault"
  end
end

But this pattern is not very DRY and would have propagated itself throughout the codebase. I knew there had to be a better way.

The Solution

As with all things computer science, my search for an answer started at Google. I consider myself a pretty good Rubyist, and I have been using RSpec for many years. I am not going to get into the testing wars, but RSpec is my go-to, it works really well for me, and that is my preference. As part of my RSpec use, I regularly use filter groups, in particular the magic :focus tag, which allows me to focus on a specific test or context of tests to isolate runs to the features I am currently working on. In RSpec 3+, the following two things are equivalent:

it "tests something", :focus do
  # ...
end

it "tests something", focus: true do
  # ...
end

This is part of the treat_symbols_as_metadata thing, but that is not super relevant to this post. The relevant part is that the value of that hash key can actually be anything, including a version constraint.

I very quickly decided on the API I wanted. I think this API is very succinct, clearly describing the test and its requirements, without bloated conditionals or comments:

it "tests something", vault: ">= 0.5.3" do
  # ...
end

Even without looking at the code, it is probably clear that this particular test requires Vault version 0.5.3 or higher. Now that I decided on the API, I had to figure out how to make it work. After all, RSpec metadata is just that - metadata. There is no way RSpec could just magically interpret that for me.

If you take a look at the RSpec docs for inclusion filters and exclusion filters, this might look impossible. Most of the examples show a single key begin filtered:

RSpec.configure do |c|
  c.filter_run_excluding broken: true
end

But I really want something like this:

# This is not real code and does not work.
RSpec.configure do |c|
  c.filter_run_excluding do |example|
    # Exclude the tests where the given Vault constraint is not satisfied by the
    # current Vault version.
    !Gem::Constraint.new(example[:vault]).satisfied_by?(vault_version)
  end
end

Looking through the examples on the RSpec Relish docs, this appears to be impossible, but I was determined. So I went code spelunking! It turns out that, deep in some Ruby in the RSpec code that tests RSpec, I found my hint:

RSpec.configure do |c|
  c.filter_run_excluding :ruby => lambda {|version|
    case version.to_s
    when "!jruby"
      RUBY_ENGINE == "jruby"
    when /^> (.*)/
      !(RUBY_VERSION.to_s > $1)
    else
      !(RUBY_VERSION.to_s =~ /^#{version.to_s}/)
    end
  }
end

It looks like we can pass a proc/lambda/block to the filter, and that filter gets evaluated each time. Yay! So we refactor our earlier code to something like this:

RSpec.configure do |config|
  config.filter_run_excluding vault: ->(v) {
    !Gem::Requirement.new(v).satisfied_by?(TEST_VAULT_VERSION)
  }
end

This tells RSpec to evaluate the Vault constraint and see if it satisfies the currently running Vault version... but where does that come from? the TEST_VAULT_VERSION environment variable is set outside of the Proc for performance reasons (remember, that Proc will be evaluated on each metadata with a defined key of vault):

TEST_VAULT_VESRION = Gem::Version.new(ENV["VAULT_VERSION"] || "100")

On Travis CI, the environment variable is set as part of our testing matrix, but locally we might be testing against some random version. I chose an arbitrarily high number to make sure all tests run locally.

Conclusion

I think this is a really elegant solution, but I would be interested in hearing other patterns folks are taking when testing a client library against multiple versions of an upstream. Additionally, I would love to add an example like this to the RSpec core documentation, since I could not find an example of using a Proc for filtering anywhere in the docs!