Automated deployment and configuration with Chef and Vagrant

As we like to work smart and not hard most of the time we are always looking to improve how we develop software and reduce the pain of integration and deployment into different environments. While working with a current client this has become a real need to due the scale of the project and the lack of developers and system administrators assigned to the project.
Even though there are many products that can do automated deployment and configuration the client choose Chef. The client is also leveraging lots of the Amazon Web Services but to keep costs down we have been using Vagrant to do the initial local development work before starting up EC2 instances. This also allows developers get the same baseline environment up and running, usually, very quickly before they start adding and changing the code.
Therefore we thought that we would share our observations and initial experience of Vagrant and Chef.
Starting point?
Once we got passed the initial setup instruction provide on the Vagrant site that was great as we was able to get the basic virtual server up and running within an hour but we found the documentation provide on the Chef a little confusing but as it was a great reference.

We found http://www.jedi.be/blog/2011/03/28/using-vagrant-as-a-team/  a really useful article and soon came up with the understanding that recipes are useful but it's actually the definition of the node that provides the starting point, this is the chef.run_list within the VagrantFile, that should contain a few basic baseline recipes but we would recommend it mainly contains roles. It is within a role that recipes should be used in conjunction with attribute overrides.

The following is our baseline VagrantFile that we have used a few times for defining development environment relies on chef-solo provisioning. This gets around some chef-solo mode issues and also updates the OS to use 0.10.04 version of Chef so documentation found online becomes a good reference:

#

# Start of file

#

USER = ENV['OPSCODE_USER'] || ENV['USER']
BASE_BOX = ENV['VAGRANT_BOX'] || 'lucid32'
CHEF_CONFIG_PATH = "../../chef"

Vagrant::Config.run do |config|

  config.vm.box = BASE_BOX  config.vm.box_url = "http://files.vagrantup.com/#{BASE_BOX}.box"
  config.vm.host_name = "#{USER}-vagrant"

  config.vm.forward_port("web", 80, 8080)
  config.vm.forward_port("web-secure", 443, 8443)
  config.vm.forward_port("database", 5432, 85432) # PostgreSQL

  config.vm.provision :shell, :inline => "gem update chef"
  config.vm.provision :chef_solo do |chef|
    chef.node_name = "#{USER}-vagrant"
    chef.cookbooks_path = "#{CHEF_CONFIG_PATH}/cookbooks"
    chef.data_bags_path = "#{CHEF_CONFIG_PATH}/data_bags"
    chef.roles_path = "#{CHEF_CONFIG_PATH}/roles"
    chef.provisioning_path = "/etc/chef"
    chef.log_level = :debug

    # adjust the run list to suit your testing needs
    chef.run_list = [
      "recipe[apt]",
      "role[something]"
    ]

    # You may also specify custom JSON attributes:
    #chef.json= {}
  end
end

#
# End Of File
#

I use a folder structure as follows as a project just for deployment (a very simple approach):
(project name)
  + chef
  | + cookbooks
  | | + apt (recipe downloaded from http://community.opscode.com/cookbooks/apt)
  | | + chef-solo-search (recipe downloaded from https://github.com/edelight/chef-solo-search/downloads)
  | + data_bags
  | + roles
  | + something.json
  + vagrant
    + (server name)
      + VagrantFileThe key points to make chef-solo run like the chef client/server mode are:

• The shell provision to up grade the base lucid32 to use the latest chef. 
• Adding the basic recipes and roles to the chef.run_list that would part of the node definition 
• Having the chef-solo-search/library as a recipe 

I am not a Ruby programmer
Chef is based on Ruby and therefore you use this to help structure the recipes, definitions and templetes used. This was okay but you need to be aware that JSON is used as part of the chef-solo mode when defining the roles and data_bags.
Useful Commands
Creating a virtual instance:
vagrant up
Updating the virtual instance with the latest chef provisioning:
vagrant provision
Removing a virtual instance:
vagrant destroy

What makes a great or good software developer?

We have often wondered and have been asked this a number of times, could it be:

• The three great virtues of a programmer: laziness, impatience, and hubris - Lary Wall, Tom Christiansen, Jon Orwat, (2000) Programming Perl, O'Reilly Media Inc.

Which the Oxford dictionary gives the above words meaning as:

• Laziness, a derivative of lazy a. & v 1. a. unwilling to work; appropriate to or inducing indolence; (of river etc) slow moving.
• Impatience, a derivative of impatient (-shent) a. not enduring with composure (at, with); (of action) indicating such feeling; intolerant of; restless desirous (for anything, to do); hence or cog.
• Hubris n. insolent pride or presumption; (Gk Tragedy) overweening pride towards the gods, leading to nemesis; so ~i'stic a. [Gk]

The book gives the words meaning as:

• laziness The quality that makes you go to great effort to reduce overall energy expenditure. It makes you write labor-saving programs that other people will find useful, and document what you wrote so you don't have to answer so many questions about it. Hence, the first great virtue of a programmer. Also hence, this book. See also impatience and hubris.
  
• impatience The anger you feel when the computer is being lazy. This makes you write programs that don't just react to your needs, but actually anticipate them. Or at least that pretend to. Hence, the second great virtue of a programmer. See also laziness and hubris.
  
• hubris Excessive pride, the sort of thing Zeus zaps you for. Also the quality that makes you write (and maintain) programs that other people won't want to say bad things about. Hence, the third great virtue of a programmer. See also laziness and impatience.

I think these are good virtues but how can you apply this to software projects of various sizes and with organizational structures ranging from the evolving chaos to the quality checking every decision and artifact. The most successful software developers I have met have another virtue that is being pragmatic and is captured beautifully in the book by Andrew Hunt and David Thomas (2000), The Pragmatic Programmer,  Addison Wesley.

For me the high lights are the following:

• Continuous Testing.
• Design and implementing software over just implementing software.
• Being flexible towards the business needs or requirements.

Continuous Testing

This is the idea of having small unit tests associated with each module of code that proves the behavior of the code. This can be use to ensure all changes have not adverse effect and also Test Driven Development changes the way developers approach a problem as they have to think a little about what they are going to do and create a skeleton of a test and not worry about the detail until they come to develop that behavior.

Once a project as a number of code modules and associated tests this can be controlled by a build system and help the automation of the project allowing anyone to get a release from the Source Version Control System, verify the code passes the test and deploy to the appropriate environment.

Design and implementing software over just implementing software

This is one we really believe in but for some reason is the hardest to convince people to adhere to has people believe that is takes to much time or cost to much money. When I say design I mean fully documented so as the team scales or changes everyone can read the design specification and get fully up to speed.

This does vary depending on the industry that the software is being developed for but I would hope that the software engineering industry would pride itself in doing design before implementing the code. There are lots of tools that can help in doing software design, our favorite is Enterprise Architect by Sparx Systems.

Being flexible towards the business needs or requirements

This is the main reason we insist our development teams we work with to design their software and unit test the software. This allows us and the companies we work with to say yes to our clients when they change their minds or a deal relies on the System working in a particular way.

That only gets you half way to achieving this goal. You also need a software architecture design that the developers are will to follow and a good process to identify the impact of the change to know how much it going to cost.

If the cost is to much it can always be put into the softwares roadmap so your clients know you have listened. The main reason I like to develop flexible and configurable software is to be able to say yes to our clients.

Business developments and a web product

We have been busy since the last post. Helping the London Sports & Social Club define their initial digital communication strategy. Also we have  been consulting for a number of companies, i.e. miBuys Ltd, British Sky Broadcasting Ltd, Streamingtank Ltd, Smarter Health, Wazoku Ltd and Imagination Ltd, to ensure that we have a cash flow to cover our costs and help with the development of unosh. This is the implementation of the 'Foodplanner' idea that has been mentioned in previous entries.

Follow @unosh_news via twitter for development updates and we hope you will support us even if it's just liking a recipe and thanks in advance. 

Focused on action

Short post this time as we have been focused on a couple of items in the last few weeks, mainly an iPhone Application that has been create with Tim Rottach. It has reached a point that we are now looking for iPhone and iPod Touch testers, please complete this questionnaire to become a tester. We will let you discover what has been created once you have signed up.

We also attend a Roger Hamilton's Wealth Dynamics seminar in London. This was great event as we met lot of interesting people with ideas that we could help with and are looking forward to creating some partnerships. Wealth Dynamics is great for identify each team members strengths and what each persons ideal role is in the team, which would maximize the productivity and satisfaction of the team as everyone gets into their flow.

Nuts and Bolts!

They say that time passes quicker as you get older. This seems to be true for New Edge Engineering as the last few weeks have pasted so quickly. We have been putting the nuts and bolts of the business systems in place and starting to act on them, such as:

• Bank account set up (picking the best deal from which4U list).
• Time sheets helping to track the use of time and to focus on the most important items needing to be done.
• Expense sheets to track costs.
• Marketing plan to determine how we intend to attract customers.
• Business plan for a new service (with the working title 'Food Planner') that New Edge Engineering is incubating.
• Market research at the British Library.
• Market research questionnaire created for the 'Food Planner' (click here to help).

As part of the marketing and sales process, we realised that New Edge Engineering had no face. As an initial start to this we decided to get some professional photographs taken to reflect the brand of New Edge Engineering. Lucky for us, we knew Tun Shwe at Go Logic who was willing to help out and had a fun day out and about London. Tun actually took some photographs on film, which he prefers over digital. Go Logic is providing a photography service and are able to help promote products, services or companies via the web. Check out Tun's personal Flickr Stream and if you contact Tun please mention New Edge Engineering.

We have discovered another interesting networking site 'Meet The Boss', which was signing people up at the IPTV World Forum a few weeks ago. Also attended the London Facebook Developers Garage evening that was about how people are monetizing their games, there were some interesting ideas, which could be used as part of the Marketing Funnel to start the spending.

The main focus for the last few weeks has been on the business and marketing plans that are starting to take shape, especially after reading the business link information. Also learning the marketing vocabulary and strategies to increase and attract customers. As part of the discovery we came across Marketing Spiral that seems to indicate a change in ideas, luckily some marketing friends were willing to talk and explain the basic ideas of what marketing aims and objectives. Thanks for that.

This new knowledge has been used to focus on the 'Food Planner' service, which lead me to the British Library to read about ABC1 consumers, Cooking & Eating, Telecommunications and E-Commerce: The Internet Grocery Market to help understand the current market that the 'Food Planner' will be helping. I had forgotten was amazing reference library it is. For people that have never been in or used the British Library, it free but you do need a Reading Pass to get into the various reading rooms. The usual id and proof of address needed to get your free Reading Pass. For us the Business and Intellectual Property area is great as they have all the marketing reports and assessments from all the major research companies providing information about many industries and is open till 8 pm most nights but check.

So after reading a lot about the internet grocery market I have put together an initial questionnaire to test the 'Food Planner' idea and get a better understanding of what people want. So please help, pass along http://research.newedgeengineering.com/foodplanner and thanks in advance.

Lead Generation?

Time flies! The week started (and that feels like yesterday) thinking about how well the sales and marketing teams should be doing. Some planning had been done last week to support the sales process:

• Basic logo design.
• Business cards.
• Digital stationary that includes letterhead, fax, sales invoice, technical specification, compliments slip.
• Identified some freelance sites that could use to pitch for work such as guru and elance could be used.

Also started last week was:

• The design and creation of the logo (more because it was possible to do it in house but we need to change this habit).
• The submission of the business cards order to Overnight Prints that delivered all the way from Germany this week. It's great to finally have something to give to people when they ask. It had been annoying to miss out on some of the opportunities that have appeared in the last couple of months.

Putting the sales and marketing hat on, which was unusual to wear, we started to think about how to promote the company and also how to measure the performance. The first step was to add the following to this site:

• Updated that blog to have a few more gadgets, including the logo and te!! a frend.
• Signed up and added Google Analytics, Quantcast, Alexa and Woopra (but be warned this is still in beta and even tho you can get an account it may take a while to get your sites approved and start receiving real time statistics).

The marketing plan was started to help the sales, communications, marketing and public relations of New Edge Engineering. Part of that plan was to get in touch with a few people that had asked for help from New Edge Engineering in the past (but no business cards to give to them at the time). This has started some good conversation but lead time and the accounting lag caused havoc with the cash flow, therefore started to think of other ways to create a cash flow. The obvious way was to sign up to a number of freelance websites and start pitching for work straight away but the process of filling in profile information, which is a necessary evil and the Linked In profile has served so well, slowed the progress. It would be great to find a tool that could populate other profiles based on the Linked In profile.

The other great thing that has been happening this week is the generation of ideas, especially other people sharing theirs. Great stuff but the ideas need to prove there is actually a market to sell to and determine if investment is needed.

Also had the chance to talk to a few experienced business owners about New Edge Engineering's ambitions and direction. Thanks to Judith who gave me, and everyone who attended the Money Gym property weekend, a chance of a free wealth consultation. Also thanks to Jon and Nick from Nativ who answered many questions about what they would do differently and gave some great insights.

The beginning of something fun and great I hope.

Wow, what an exciting experience - actually owning a company that I am responsible for. This may seem trivial to some people but I have been involved in a few start-ups in the past which had great ideas, which seemed to spark the imagination of investors but unfortunately failed.

So this is the end of week one for New Edge Engineering and a brand new and a brave new world for me personally. I am a technologist who has always been in charge of the creation of products, leading the development team through the long nights making sure they have fun along the way. When previous start-ups have failed they have been able to sell on the technology that I have been responsible for - a bonus for me!

So now it is all up to me with hopefully support from like-minded people that are interested in working on cutting or bleeding edge ideas and making money through innovation. So what is my vision of New Edge Engineering? Good question, I have been working on the basics of getting simple basic things together, logo, business cards, where the money is coming from, talking to people about joint ventures. So lots of idea but the business needs a clear focus and that is creating software for the media industry.

I personally have over 10 years experience in creating software for interactive TV, IPTV/WebTV, iPhone Applications, content management systems, publishing platforms and websites using a number of different techniques to deliver the software as product, i.e. waterfall, JAD/RAD, agile, JFDI, using the appropriate technologies to solve the business problems. Therefore I believe in producing high quality software that can flex with the ever changing media industry and constantly delivering to the business requirements.

So if you have some amazing ideas but don't have the technical skill please contact matthew@newedgeengineering.com to see if we can help.