March 3, 2017

Chef Solo and Vagrant to Rapidly Prototype Virtual Machines

Eagle makes very heavy use of virtual machines (VMs) both on public and private clouds. This is done within the context of providing data and software to customer directly or in a work flow. Many of these VMs require complex and protracted installs that can take weeks manually and the installs are often repeated several times a year as new databases and data are released. To reduce this work load of this we have been using Chef for some time on our most demanding installs. This has proved to be successful and recently we have begun to push the use of Chef from our most demanding VM installs to all but the most trivial. In addition to the speed and simplicity of installing via Chef there is also a very useful aspect of detailed documentation.

I had not used Chef before and our Chef expert recommended using Chef-Solo and Vagrant as the best way to get started on converting installs procedures. The install of Chef proved to be very simple either the latest version from Opscode:

curl -L | sudo bash

or standard via apt-get:

apt-get install chef

The vagrant install was also simple using either a debian package downloaded from the web:

> vagrant_1.3.5_x86_64.deb
dpkg -i vagrant_1.3.5_x86_64.deb

or apt-get:

apt-get install vagrant

Once the installs are complete to get a VM running is simple. Download the relevant vagrant file using vagrant init:

vagrant init ubuntu12.04

and the run vagrant using the vagrant up command. At this point a VM is running. You can find the IP address using vagrant ssh-config or login simply using vagrant ssh.

Up until this point everything is accepted as being the correct way to install and run things. The next step in integrating Chef and vagrant seems to have at least 3 different ways of being done. The simplest and what seemed to be the oldest way on integrating the 2 technologies is using a 2 simple shell scripts to install Chef and then run the chosen recipe. This worked and was quick to set up for a simple recipe but on further reading and discussions with colleagues it seemed it might not be adequate for more complex ones. This led me to investigate using knife solo in conjunction Chef Librarian and Berkshelf. They both seemed very good but after a detailed discussion and some experimentation we went with Berkshelf. Knife solo will work with either but it will use Berkshelf if both are installed. Using knife solo is very easy and only need you to set up provision in your vagrant file. You can then install your recipe and reset you VM to its vanilla state with a few knife solo commands.

knife solo init mychefrepo
knife solo prepare ubuntu@
knife solo cook ubuntu@
knife solo clean ubuntu@

Init sets up the repository, prepare will install chef solo, cook will run the recipe on VM and clean will reset the VM back to its original state. This simple cycle of commands will enable you to iterate round when developing a chef recipe quickly only using your local machine and with a VM. Even though I had limited Chef experience it enabled me to quickly develop a useful recipe. If you are thinking of using Chef I can thoroughly recommend using Vagrant, Chef solo, Berkshelf and knife solo.

Topics: Bioinformatics