Moving to Individual Cookbooks

If you've been working with Chef in your organization for awhile, you've probably accumulated a bunch of cookbooks in a giant repository. With great tools like Berkshelf out there, it's become customary to rely on external community cookbooks, git repositories, and your own Chef Server for cookbooks. Essentially cookbooks have been extracted as a first-class object.

With Berkshelf, you can manage cookbooks as "dependencies", much like bundler does with a Gemfile. But that raises a big question - how do I move to Berkshelf? Joshua Timberman wrote up a nice blog post on moving to Berkshelf, and he listed techniques for moving to a community-managed cookbook.

In this blog post, I'll show how to extract your non-community cookbooks (like your proprietary organization cookbooks) into their own (private) git repositories.

Move Over to Community Cookbooks

Joshua Timberman wrote up a nice blog post on this topic, but I have a slightly different spin.

First, add berkshelf to your Gemfile:

gem 'berkshelf'

And run the bundle command to install.

Here's little Ruby script I wrote to move all your cookbooks into Berkshelf. It respects your current metadata.rb version, so you shouldn't have any compatibility issues.

require 'chef'
require 'fileutils'

def loader
  @loader ||= ::Chef::CookbookLoader.new('cookbooks')
end

community_cookbooks = `git branch | grep 'chef-vendor' | awk -F- '{print $3}'`.split("\n")
community_cookbooks.collect!{ |cookbook| loader[cookbook.to_s] }

File.open('Berksfile', 'w') do |file|
  file.write "site :opscode\n"
  file.write "\n"
  community_cookbooks.sort.each do |cookbook|
    file.write "cookbook '#{cookbook.name}', '#{cookbook.version}'\n"
  end
end

community_cookbooks.each do |cookbook|
  FileUtils.rm_rf File.join('cookbooks', cookbook)
end

`bundle exec berks install`

You can copy-paste that in an IRB session run from the root of your chef repository.

Move Over Private Cookbooks

What about your organizational cookbooks? How do you move those into Berkshelf? Well, with some magical git and shell script of course!

First, we made an assumption that you've already moved your non-organizational cookbooks to Berkshelf (see previous part).

Here's the "English" process:

1. Get a list of all current organizational cookbooks
2. For each cookbook:
   1. Create a new private repository on github
   2. Use git to rewrite the version history for that path
   3. Push to github
   4. Reset master
   5. Add it to Berkshelf

We will be leveraging the hub gem to create the remote repositories for us.

Make sure you have pushed and committed everything in your chef repository before continuing!

Install hub:

brew install hub

We need to run a hub command so that it will store your credentials:

hub create foo

Now go delete that repository. This is the only manual step. If you've used hub before, you do not need to do this step.

Here's a tiny bash snippet to get you started:

export USER="FILL THIS IN" # The github user or organization to create the repos

for cookbook in `ls cookbooks`
do
  # Create the remote repository
  hub create $USER/$cookbook -p -d "Private repository for $cookbook"
  git remote add $cookbook git@github.com:$USER/$cookbook.git

  # Filter out anything relating to this cookbook - this will save the git history related
  # to this cookbook
  git filter-branch --prune-empty --subdirectory-filter cookbooks/$cookbook master

  # Push to the new remote
  git push $cookbook master

  # Remove the remote (we don't need it anymore)
  git remote rm $cookbook

  # Reset to master for the next cookbook
  git reset --hard HEAD

  # Remove the cookbook
  git rm -rf cookbooks/$cookbook

  # Add this cookbook to the Berksfile
  echo "cookbook '$cookbook', github: '$USER/$cookbook'" >> Berksfile

  # Save master with the new changes
  git add Berksfile
  git commit -m "Moved $cookbook to Berkshelf"
  git push origin master
done

bundle exec berks install

As an added bonus, we are rewriting the git history, so you still have the entire history relating to that cookbook in the new repository.