Scripted infrastructure, or Infrastructure as Code, has become a necessity to allow operations teams to keep up with the increasing rate of change of application development and sprawling data centers. Tools like Chef, Puppet and Ansible have helped turn previously untouchable servers into disposable commodities that can be recreated on demand.
Although scripted infrastructure has simplified many scalability and repeatability problems, it has introduced operations teams to coding practices that were often the domain of application developers.
The following is an overview of some development best-practices that will help you ensure repeatable and reliable deployment of your infrastructure code using Chef.
I hope that most of us are well versed in the benefits of version control, but it is always good to hammer home this point: version control is NOT just a backup for your code. Version control provides the following benefits:
- A central location where everyone can find the definitions of your infrastructure
- Automatic audits of code changes
- Authorization and authentication over who can make changes (in this case, to your infrastructure)
- Collaboration via *shared* code
- Simple roll-back of harmful changes
From personal experience, I would recommend using Git as your source control tool. Git has become the standard in version control and will integrate easily with some of the other tools discussed later in this article.
A major benefit of a widely used tool, like Chef, is the ability to re-use and build upon code that has been written and open-sourced by others. Why re-write a cookbook for managing iptables if there is already a great one out there? It is often tempting to just use the “latest” version of that code, but that can sometimes introduce instability into your code that relies on that software. Luckily, Chef has a great built-in tool to help you manage dependencies: Berkshelf!
At its core, you simply specify which version of a cookbook you want to use and Berkshelf will do the heavy lifting of retrieving that cookbook and placing it somewhere your code can find it. It will also grab any additional cookbooks that it may depend on.
An example ‘Berksfile’ for your cookbook to use MySQL and nginx 2.6:
source "https://supermarket.chef.io" metadata cookbook "mysql" cookbook "nginx", "~> 2.6"
Additionally, Berkshelf can also help you maintain best practices in versioning your own cookbooks. I highly suggest perusing the documentation. Berkshelf is part of the Chef Development Kit(1).
Static Code Analysis
Syntax errors can often find their way into deployed code before they are caught, causing tools like Chef to halt action and spit out a nasty error! Static code analysis can help prevent these problems by analyzing your code before it is actually run anywhere.
Chef provides Foodcritic, as part of the Chef Development Kit(1), to inspect your code and let you know of any gotchas prior to running it. Foodcritic will help check for the following issues(2):
- Best practices
- Common mistakes
Foodcritic will help you write cleaner, more maintainable code while adhering to best practices from the Chef community!
Testing is possibly my favorite subject to discuss in software development. It is the most often overlooked, yet has the greatest benefit of almost any coding practice. Testing your code will provide the following benefits:
- Ensure your code performs exactly as expected. No more, no less.
- Provide a safety net for future changes to the code
- Act as pseudo-documentation for other developers
Well-tested code will ensure that the changes you want to make are the ONLY changes that happen to your infrastructure.
Unit testing Chef code will allow you to test that your cookbooks run as expected without having to set up expensive infrastructure to verify. Chef provides ChefSpec to run your specified set of cookbooks solely in a mock environment to allow you to test assertions against expected output. For example, deleting a file:
it 'deletes a file with an explicit action' do expect(chef_run).to delete_file('/tmp/explicit_action') expect(chef_run).to_not delete_file('/tmp/not_explicit_action') end
When run against your cookbook, the above test will verify that one file, ‘/tmp/explicit_action’ was deleted, while ‘/tmp/not_explicit_action’ was not. You can imagine that verifying file deletions (and non-deletions) are a smart thing to test. ChefSpec is part of the Chef Development Kit(1).
Integration testing is a slightly more realistic version of testing where you run your cookbook against an actual virtual machine or other representation of your server. The tests then verify that everything was applied as expected.
Once again, Chef has you covered with a best-of-breed testing tool called Test Kitchen. Test Kitchen will spin up a virtual machine, apply your cookbook(s), and then test any assertions you have given it to verify your cookbook code. At the end of the test you can even have Test Kitchen clean up for you!
An important aspect of integration testing is to ensure that whatever VM or container you test against is as close as possible to the real server where the cookbooks will be run. For instance, don’t test against an Ubuntu 12.04 VM if your production environment is Ubuntu 14.04. The closer you can get everything to match, the more confident you can be that your cookbooks will work as expected.
Test Kitchen is part of the Chef Development Kit (1).
A Final Word on Testing
I’m sure we’ve all heard the saying “There isn’t enough time for testing” or “We’ll write tests later.” I can tell you from experience that both couldn’t be further from the truth. Writing tests while writing code ensures you are in the proper context for testing the code. Do you think you will be able to remember how that code works a month from now? How about six months from now? Let the tests explain it for you and provide you some confidence that you can change your code without breaking anything! Testing WILL save you time in the long run. I always advocate for testing.
Continuous Delivery is the process whereby a developer’s code is pushed from the developers computer and, via a build pipeline, moves into the final destination environment. In scripted infrastructure terms, it is the process where code moves from a developer’s or admin’s workstation and eventually configures a server or servers in the infrastructure.
The goal of this process is that server administrators no longer configure any servers “by hand.” Tools like Jenkins, a continuous integration server, will help you to define this pipeline and ensure code is deployed in a repeatable way. An example pipeline would be:
- Developer checks cookbook into source control
- CI server detects change and pulls latest code
- CI server runs any static analysis and tests and stops the build on any failure
- If no failures, CI server will apply the cookbook to intended servers or wait for manual indication that cookbooks can be applied
The idea behind this process is that any time a change is needed in an environment, it is first developed locally and sent through the pipeline for verification prior to deployment. This process prevents the many ad-hoc and undocumented changes that can occur in a non-scripted infrastructure. It also ensures traceability and facilitates auditing the system for changes.
I hope this was a good introduction to best practices to use while developing your scripted infrastructure with Chef. I have utilized these practices in my own development to achieve a highly secure and reliable infrastructure development and deployment process. Hopefully you can take advantage of these recommendations and work toward your automation goals!
(1) Chef Development Kit — https://downloads.chef.io/chef-dk/
(2) From — https://docs.chef.io/foodcritic.html