I was inspired to write this post when I noticed that half of the people on my team were fighting with Chef. Opscode Chef is an amazingly flexible tool and it’s very powerful. It’s this flexibility that can cause problems if you don’t know what you are doing. Manipulating attributes is just one example of how Chef’s flexibility can cause problems. Attributes can be modified in multiple places, recipes, data_bags, roles and environments and there is a complex precedence hierarchy describing how you finally end up with the attribute value that will finally applied to the node. It is very easy to get unexpected results.
As professional developers we are learning things all the time, and constantly polishing our skills. Personally I focus my educational efforts on the programming languages that I use on a daily basis in my job and I tend to give ancillary tools like git and Chef lower priority. I’ve found however, that Chef and it’s DSL has a learning curve and requires the kind of attention that you’d apply to a regular programming language like C++, Ruby, or Java. You can write code that gets that job done but even so is unmaintainable crap. Sadly, I’ve found that unmaintainable crap is common in the many of the Chef resources I’ve seen written. I think that developers don’t tend to apply the same rigor to their Chef development that they do with writing the code that Chef is used to deploy. The problem with doing this is that Chef ends up being a huge time suck instead of a time saver. The rest of this article contains some suggestions I have to help keep your team focused on working on things like writing code instead of fighting with Chef.
Suggestion: Chef resources (data_bags, cookbooks, roles, and environments) should be treated the same as any other source code. Chef resources must be stored in git (or other version control). If Chef resources are associated with a project, they should be located in version control under the project root. Let’s say we have a project called ‘foo’ then the Chef artifacts for foo would be located below the project root like so:
foo/chef foo/chef/cookbooks foo/chef/site-cookbooks foo/chef/data_bags foo/chef/roles foo/chef/environments
If there are any changes to be made to the Chef server they must be stored in version control before being pushed to the Chef server. If you use feature branches when developing code, you’d do the same thing when make changes to your Chef artifacts. You want to have a change history for your Chef artifacts and to my knowledge, Chef server does not provide this. The changes to the Chef server should be treated the same as a production release unless the changes are explicitly restricted to a non-production environment. Even if changes are going to be tested on a non-production environment, Chef changes should be vetted using Vagrant and chef-solo prior to pushing changes to QA.
Suggestion: Use Roles appropriately. From Opscode Chef
A role is a way to define certain patterns and processes that exist across nodes in a Chef organization as belonging to a single job function.
Examples for roles would be names like persistance, or web-admin. Roles shouldn’t change very often, they are not versioned, and they should exist across environments rather than define an environment. We should use….. wait for it…. Chef environments to deal with different deployment environments. I would say that once in place, a role should never change. For example, lets say we have a persistance role that is already in production and lets say we want to change that role somehow. We’ll want to test the role in QA and we’ll further want to make sure that the changed role isn’t inadvertently propagated to our production environment. I’d suggest naming the role under test something new like persistance-1. That way the new role must be explicitly assigned to a node in order for the changes to be propagated to an environment.
Suggestion: Use Environments appropriately. See snarky comment in the Roles suggestion. Attributes that are different in environments should be overridden in a Chef environment. The environment should be named such that it is easy to associate with what it belongs to i.e. our QA3 environment should be named QA3. Chef environments should contain cookbook version contraints for the cookbooks used in that particular environment. See cookbook suggestion.
Suggestion: Chef cookbooks should always be versioned and associated with Chef environments. Work on source code within a recipe should be performed on a feature branch, AND, the cookbook should get a new version before it gets released to the Chef Server. If you don’t have an environment with versioned recipes associated with a node that node will get the latest version of a cookbook by default. This can lead to unpleasant surprises.
Well that’s it. I hope you find these suggestions helpful. One caveat, I’m a Chef newbie so if there if you know of better ways of doing things than the ones I’ve mentioned here, please feel free to pipe in with your own comments.