Chef recipe code coverage

How do you do code coverage with Chef?

This seems like a simple question and should have a really simple answer. Chef recipes and resources are just Ruby code afterall, so a tool like Simplecov should do the trick...

Wrong!

During a chef-client run, Chef dynamically loads (and reloads) resources at runtime. Tools like Simplecov rely on parsing Ruby's AST tree to determine when a particular line of code is executed. But, if that file is reloaded, the AST tree is entirely rebuilt in the Ruby VM (YARV in most cases), invalidating the cache and rendering tools like Simplecov unusable.

Across conferences, meetups, Twitter and IRC, "code coverage" (or "resource reporting") is one of the most common feature requests I receive. I'm happy to announce that ChefSpec v3.1.0 now includes a resource coverage reporter!

It's important to note: this is not a code coverage analysis. ChefSpec is only aware of resources in a recipe. ChefSpec cannot examine your libraries or custom resources. Fortunately you can still use Simplecov and plain-old-RSpec to test those.

How do I use it?

The resource coverage reporter is not enabled by default, since it does come with a slight performance cost. To enable coverage reporting, just generate the report at the end of your spec_helper:

# spec_helper.rb

# Existing setup, helpers, etc

at_exit { ChefSpec::Coverage.report! }

That's it!

Now when you run your specs, ChefSpec will print a resource coverage summary. For example, suppose I have a recipe that declares a package installation and a template render:

package 'foo'
template 'bacon'

And an associated ChefSpec test:

require 'spec_helper'

describe 'bacon::default' do
  let(:chef_run) { ChefSpec::Runner.new.converge(described_recipe) }

  it 'installs foo' do
    expect(chef_run).to install_package('foo')
  end
end

Notice that I am only testing the package installation, not the template. The resulting coverage report looks like this:

ChefSpec Coverage report generated at './.coverage/results.json':

  Total Resources:   2
  Touched Resources: 1
  Touch Coverage:    50.0%

Untouched Resources:

  template[bacon] ./recipes/default.rb:3

What about machine-parsable code?

While I know how much you all love to use sed and awk, ChefSpec actually generates a machine-friendly result file in .coverage/results.json. As the extension implies, this file is machine-parsable JSON (Javascript Object Notation) and all modern programming languages can easily turn this file into a usable object. Here's the machine-parsable coverage report from the previous example:

{
  "total": 2,
  "touched": 1,
  "untouched": 1,
  "coverage": 50.0,
  "detailed": {
    "package[foo]": {
      "source": {
        "file": "./recipes/default.rb",
        "line": 1
      },
      "touched": true
    },
    "template[bacon]": {
      "source": {
        "file": "./recipes/default.rb",
        "line": 3
      },
      "touched": false
    }
  }
}

With machine-parsable output, the possibilities are endless. Here are some fantastic ideas:

• Configure your continuous integration server to automatically reject recipe additions that do not include tests
• Create a dashboard with pretty colors indicating code coverage across all your organization's cookbooks
• Build a [insert your editor here] plugin that automatically highlights untested resources in recipes (paging Doug Ireton)
• Have a party!

Can I get a pretty [format] report?

This feature is relative easy to implement given the machine-parsable output. I have some design ideas around "outputters" (like XML, HTML, CSV, custom, etc) that you will likely see in the next release. But for now, the only output formats are console and JSON.

Limitations

• ChefSpec coverage is entirely limited to recipe in the resource collection. It cannot cover resources or libraries.
• The reporter is a single-touch test (also known as statement coverge). So the coverage report can report a resource was tested even if all branches of logic for that resource were not exhausted.
• There is no way to test specific attributes of a resource were tested at this time.

I really hope you enjoy the new code coverage feature of ChefSpec. It has been a long time in the making and I am really excited for the roadmap moving forward.

Happy cookbook testing!