Automated deployment and configuration with Chef and Vagrant

As we like to work smart and not hard most of the time we are always looking to improve how we develop software and reduce the pain of integration and deployment into different environments. While working with a current client this has become a real need to due the scale of the project and the lack of developers and system administrators assigned to the project.
Even though there are many products that can do automated deployment and configuration the client choose Chef. The client is also leveraging lots of the Amazon Web Services but to keep costs down we have been using Vagrant to do the initial local development work before starting up EC2 instances. This also allows developers get the same baseline environment up and running, usually, very quickly before they start adding and changing the code.
Therefore we thought that we would share our observations and initial experience of Vagrant and Chef.
Starting point?
Once we got passed the initial setup instruction provide on the Vagrant site that was great as we was able to get the basic virtual server up and running within an hour but we found the documentation provide on the Chef a little confusing but as it was a great reference.

We found http://www.jedi.be/blog/2011/03/28/using-vagrant-as-a-team/  a really useful article and soon came up with the understanding that recipes are useful but it's actually the definition of the node that provides the starting point, this is the chef.run_list within the VagrantFile, that should contain a few basic baseline recipes but we would recommend it mainly contains roles. It is within a role that recipes should be used in conjunction with attribute overrides.

The following is our baseline VagrantFile that we have used a few times for defining development environment relies on chef-solo provisioning. This gets around some chef-solo mode issues and also updates the OS to use 0.10.04 version of Chef so documentation found online becomes a good reference:

#

# Start of file

#

USER = ENV['OPSCODE_USER'] || ENV['USER']
BASE_BOX = ENV['VAGRANT_BOX'] || 'lucid32'
CHEF_CONFIG_PATH = "../../chef"

Vagrant::Config.run do |config|

  config.vm.box = BASE_BOX  config.vm.box_url = "http://files.vagrantup.com/#{BASE_BOX}.box"
  config.vm.host_name = "#{USER}-vagrant"

  config.vm.forward_port("web", 80, 8080)
  config.vm.forward_port("web-secure", 443, 8443)
  config.vm.forward_port("database", 5432, 85432) # PostgreSQL

  config.vm.provision :shell, :inline => "gem update chef"
  config.vm.provision :chef_solo do |chef|
    chef.node_name = "#{USER}-vagrant"
    chef.cookbooks_path = "#{CHEF_CONFIG_PATH}/cookbooks"
    chef.data_bags_path = "#{CHEF_CONFIG_PATH}/data_bags"
    chef.roles_path = "#{CHEF_CONFIG_PATH}/roles"
    chef.provisioning_path = "/etc/chef"
    chef.log_level = :debug

    # adjust the run list to suit your testing needs
    chef.run_list = [
      "recipe[apt]",
      "role[something]"
    ]

    # You may also specify custom JSON attributes:
    #chef.json= {}
  end
end

#
# End Of File
#

I use a folder structure as follows as a project just for deployment (a very simple approach):
(project name)
  + chef
  | + cookbooks
  | | + apt (recipe downloaded from http://community.opscode.com/cookbooks/apt)
  | | + chef-solo-search (recipe downloaded from https://github.com/edelight/chef-solo-search/downloads)
  | + data_bags
  | + roles
  | + something.json
  + vagrant
    + (server name)
      + VagrantFileThe key points to make chef-solo run like the chef client/server mode are:

• The shell provision to up grade the base lucid32 to use the latest chef. 
• Adding the basic recipes and roles to the chef.run_list that would part of the node definition 
• Having the chef-solo-search/library as a recipe 

I am not a Ruby programmer
Chef is based on Ruby and therefore you use this to help structure the recipes, definitions and templetes used. This was okay but you need to be aware that JSON is used as part of the chef-solo mode when defining the roles and data_bags.
Useful Commands
Creating a virtual instance:
vagrant up
Updating the virtual instance with the latest chef provisioning:
vagrant provision
Removing a virtual instance:
vagrant destroy

What makes a great or good software developer?

We have often wondered and have been asked this a number of times, could it be:

• The three great virtues of a programmer: laziness, impatience, and hubris - Lary Wall, Tom Christiansen, Jon Orwat, (2000) Programming Perl, O'Reilly Media Inc.

Which the Oxford dictionary gives the above words meaning as:

• Laziness, a derivative of lazy a. & v 1. a. unwilling to work; appropriate to or inducing indolence; (of river etc) slow moving.
• Impatience, a derivative of impatient (-shent) a. not enduring with composure (at, with); (of action) indicating such feeling; intolerant of; restless desirous (for anything, to do); hence or cog.
• Hubris n. insolent pride or presumption; (Gk Tragedy) overweening pride towards the gods, leading to nemesis; so ~i'stic a. [Gk]

The book gives the words meaning as:

• laziness The quality that makes you go to great effort to reduce overall energy expenditure. It makes you write labor-saving programs that other people will find useful, and document what you wrote so you don't have to answer so many questions about it. Hence, the first great virtue of a programmer. Also hence, this book. See also impatience and hubris.
  
• impatience The anger you feel when the computer is being lazy. This makes you write programs that don't just react to your needs, but actually anticipate them. Or at least that pretend to. Hence, the second great virtue of a programmer. See also laziness and hubris.
  
• hubris Excessive pride, the sort of thing Zeus zaps you for. Also the quality that makes you write (and maintain) programs that other people won't want to say bad things about. Hence, the third great virtue of a programmer. See also laziness and impatience.

I think these are good virtues but how can you apply this to software projects of various sizes and with organizational structures ranging from the evolving chaos to the quality checking every decision and artifact. The most successful software developers I have met have another virtue that is being pragmatic and is captured beautifully in the book by Andrew Hunt and David Thomas (2000), The Pragmatic Programmer,  Addison Wesley.

For me the high lights are the following:

• Continuous Testing.
• Design and implementing software over just implementing software.
• Being flexible towards the business needs or requirements.

Continuous Testing

This is the idea of having small unit tests associated with each module of code that proves the behavior of the code. This can be use to ensure all changes have not adverse effect and also Test Driven Development changes the way developers approach a problem as they have to think a little about what they are going to do and create a skeleton of a test and not worry about the detail until they come to develop that behavior.

Once a project as a number of code modules and associated tests this can be controlled by a build system and help the automation of the project allowing anyone to get a release from the Source Version Control System, verify the code passes the test and deploy to the appropriate environment.

Design and implementing software over just implementing software

This is one we really believe in but for some reason is the hardest to convince people to adhere to has people believe that is takes to much time or cost to much money. When I say design I mean fully documented so as the team scales or changes everyone can read the design specification and get fully up to speed.

This does vary depending on the industry that the software is being developed for but I would hope that the software engineering industry would pride itself in doing design before implementing the code. There are lots of tools that can help in doing software design, our favorite is Enterprise Architect by Sparx Systems.

Being flexible towards the business needs or requirements

This is the main reason we insist our development teams we work with to design their software and unit test the software. This allows us and the companies we work with to say yes to our clients when they change their minds or a deal relies on the System working in a particular way.

That only gets you half way to achieving this goal. You also need a software architecture design that the developers are will to follow and a good process to identify the impact of the change to know how much it going to cost.

If the cost is to much it can always be put into the softwares roadmap so your clients know you have listened. The main reason I like to develop flexible and configurable software is to be able to say yes to our clients.

Business developments and a web product

We have been busy since the last post. Helping the London Sports & Social Club define their initial digital communication strategy. Also we have  been consulting for a number of companies, i.e. miBuys Ltd, British Sky Broadcasting Ltd, Streamingtank Ltd, Smarter Health, Wazoku Ltd and Imagination Ltd, to ensure that we have a cash flow to cover our costs and help with the development of unosh. This is the implementation of the 'Foodplanner' idea that has been mentioned in previous entries.

Follow @unosh_news via twitter for development updates and we hope you will support us even if it's just liking a recipe and thanks in advance.