Chef Quick-start

Introduction

Sometimes, people make something complex very complicated and put-offish indeed and perhaps not intentionally. Or, they suffer from a complex that makes them want to wear pointy black hats with moons and stars on them. Whatever the case, I found the Chef learning curve particularly steep and this is me flattening it albeit with a few caveats.

Quickest steps to setting up and using Chef

1. Setting up the Chef server

In the very recent past (May 2013), this has gotten easier. It is two downloads. It's supposed that disparate versions of Chef can be mixed. (I run a 10.x server with 11.x client nodes.)

2. Setting up a Chef client node

Put this installation on the back burner...

We'll have to leave off setting up our Chef client node just a bit to set up our development host. We could do it using the Chef server web interface,^* but it's not how I want to show it. First, we have to set up the developer host we're going to use to administer Chef. After that we'll finish this step.

^* With Chef, there are always half a dozen ways to do any one thing.

3. Choose and set up a development host for Chef administration

I leave the choosing to you with the caveat that I use Linux here.

Please install a Chef client on the development host you've chosen just as you did for the VM node you chose to install in step 2, but don't run the knife configure client command yet.

Once the Chef client is installed on your development host, here are the steps to setting it up.

At this point, you should be able to get some warm fuzzies from typing

    $ knife cookbook list

It won't return anything, but the fact that it doesn't display an error means you've contacted your Chef server and it has replied that you've not created any cookbooks.

What you should know about knife.rb

This file controls a lot of things you're about to use such as:

• Whether errors are verbose; log_level set to :debug instead of :info.
• Log location can be stdout or a file.
• The name of your development host, which is also a Chef node.
• The path to your client private key: Chef simply won't work if this is bad or the key itself is (or goes) bad.
• The Chef server URL.
• The path to your cookbooks; this affects knife cookbook. Elsewhere, you may see this as a list of paths. Apparently, this is a concept Opscode wishes to obsolete.
• Your e-mail address (I've so far never received e-mail from Chef).
• Other stuff as is.

Note: You can create as many instances of this chef-repos filesystem on your development host as you like and these instances can be named anything you like. If you're a developer operations persons working with more than product team, this might come in handy.

2. (continued) Setting up a Chef node

4. Setting up a client, a role and a cookbook/recipe

Unless you don't wish to log into the Chef server web interface as admin, there aren't many reasons to create another user. However, we do want to create a client, just as we did for the client node we want to manage, which is really crucial to finishing our development host set-up.

Administration client (you)

Chef documentation is slightly confusing with its use of the word client. To help disambiguate, I like to keep this term for the up-coming concept and, if I use it with nodes, I make sure I never say only "client" when I mean "client node". Here, we're talking about your Chef client that does real work, i.e.: the administrating development host and not the working node you're going to set up (on which you'll only ever run the command, chef-client).

While you will usually log into the Chef server web interface as admin, you'll use the Chef knife utility as yourself. This is what we're setting up here and Chef knows you as the .pem file which contains your private key when you use knife. Here, I'm russ.pem.

We're only going to create one client. If you've got a colleague that's going to help you administer Chef, he or she will need a separate one (all the while probably consuming the chef-repos contents from common Git repository).

Roles

Roles are an artificial designation in Chef to help organize who does what in the setting up of a Chef client node. For example, ensuring the installation of Tomcat or your database might be considered a role.

You can make a role to be as powerful and run as many recipes as you like, for example, set up an entire application or database server node, or as granualar and pointed, such as only to install Tomcat or MySQL as part of what a node needs. As you chef more and more, it will dawn on you just how granular you should be. If in doubt, be granular. This will make Legos™ you can use to set up different types of nodes.

For our purposes, we're going to create one role, application_node. We could use knife to set up our role, which we'd immediately see in the Chef server web interface, but we want a written record of what we're doing so we don't have to look in the web interface, which might go missing if the Chef server ever died.

You should give thought here to commiting everything to Git or a suitable version-control system that we do here for recoverability.

1. Under the roles subdirectory, create a new Ruby file, application_node, with these contents:

       name "application_node"
       description "Role for managing web application server nodes"
   
       for_application_servers = %w{
   	    recipe[tomcat6]
   	    recipe[xyz]
       }
   
       run_list for_application_servers
   

2. Use knife and this file to create this new role:

       ~/chef-repos $ knife role from file application_node.rb
       Updated Role application_node!
   

This installed a new role, application_node, into Chef which you can see in the web interface by clicking the Roles tab. It also told Chef that this role knows of and uses two recipes (out of two cookbooks), which we haven't written yet.

Nodes

Recipes and cookbooks

Cookbooks are another area in which Chef terminology gets flaky if in a different way. You may become confused about the difference between recipes and cookbooks.

Cookbooks are collections of recipes. Your cookbook here, in its subdirectory, is one and it will hold one or more recipes. You could create others. I'm not going to illustrate that so all you'll see is a bunch of cookbooks under the Cookbooks tab of the web interface, but no recipes by name.

Much of Chef documentation uses "recipe" for "cookbook" when a cookbook contains only one recipe. It's confusing. Just beware.

We'll create two cookbooks with one recipe each here.

1. xyz
2. tomcat6

Cookbook/recipe xyz

This is a recipe for an imaginary web application, xyz.

1. Under subdirectory cookbooks, issue the following command:

   ~/chef-repos/cookbooks $ mkdir -p xyz xyz/files/default xyz/recipes templates/default
   

2. Change your current working directory to xyz and add the following files:

   metadata.rb:

       maintainer       "Despicable Me"
       maintainer_email "[email protected]"
       license          "Apache 2.0"
       description      "Fun web application (xyz) for application node (server) cluster"
       long_description IO.read(File.join(File.dirname(__FILE__), 'README.rdoc'))
       version          "1.0.0"
   

   README.rdoc:

       = DESCRIPTION:
       Chef recipe for fun XYZ web application.
   
       = REQUIREMENTS:
       Tomcat
   

   Note: When you grow up in Chef, these files, especially the second, will tend to acquire more and better information as you'll see soon when we create the tomcat6 recipe.

3. Copy any harmless Debian package to the path files/default.

   Admittedly, this is where my Chef quick-start tutorial goes south: I'm not going to go the trouble of creating your web application. Also, this recipe will install a Debian package. I'll try to come back later and add another recipe that copies a WAR file to /var/lib/tomcat6/webapps instead. I use a Debian package since I need to populate the filesystem under /opt with data.

4. Change to the recipes subdirectory and add the following file including the comment which you can remove, but will be a cheat to give you some information that transcends what I have in mind for this quick-start tutorial.

   default.rb:

       #
       # Cookbook Name::
       # Recipe:: xyz
       #
       # The purpose of this recipe is ONLY to deploy xyz.deb.
       #
       # The way this works is:
       # 1. A new xyz.deb is copied into cookbooks/xyz/files/default.
       # 2. The updated recipe is committed to Git.
       # 3. A Jenkins project notices that there's something new in Git.
       # 4. Jenkins runs a script that knifes the new recipe up to the Chef server.
       # 5. The next time an application node runs chef-client, which is controlled
       #      by a cron job, the new xyz.deb is deployed by this recipe
       #      using dpkg -i.
       #
       # Possible failures:
       # 1. xyz.deb doesn't have a later version than the existing one. This
       #      results in dpkg -i failing to install it.
       # 2. Something else we'll learn by experience and document here.
       #
       # Copyright 2013, Etretat Logiciels, LLC and Russell Bateman
       #
   
       # Copy the Debian package to /tmp. This package is put into the
       # recipe before replacing/updating the recipe on the Chef server.
       cookbook_file "/tmp/xyz.deb" do
   	    source "xyz.deb"
   	    mode   "0644"
   	    owner  "root"
   	    group  "root"
       end
   
       # Install the Debian package...
       dpkg_package "xyz" do
   	    source "/tmp/xyz.deb"
   	    action :install
       end
   

5. Return to the root of your chef-repos subdirectory and knife this cookbook up using this command:

       ~/chef-repos $ knife cookbook upload xyz
       Uploading xyz            [1.0.0]
       Uploaded 1 cookbook.
   

6. If you like, examine this cookbook in the Chef server web interface under the tab, Cookbooks. Follow the links down and you should see the text of the actual recipe, default.rb.

Cookbook/recipe tomcat6

There are two reasons I'm illustration the creation of a recipe that installs Tomcat 6 on a Chef node. First, the example of a web application simply requires it. Second, and more important, it illustrates how you can steal your way to Chef glory and win fame among your peers without much effort (sometimes).

The Internet is full of great, useful Chef cookbooks/recipes to do all sorts of things. Pop up your favorite browser and Google for "opscode chef recipe for tomcat". (When I Google for Chef stuff, I often add "opscode" to keep myself from getting distracted by food offerings.)

Ready to go!

At this point, we've finished our quick-start tutorial except for forcing one node (or as many as we want) to install itself with Tomcat 6 and our web application. We use ssh to get into our node, get root, then use the chef-client command to load our recipes, which is another way of saying "install software and configure" our client node:

    ~/chef-repos $ ssh app01
    Welcome to Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-48-virtual x86_64)

     * Documentation:  https://help.ubuntu.com/

      System information as of Fri Jun 28 15:43:59 MDT 2013

      System load:    0.0             Processes:           97
      Usage of /home: 6.1% of 937MB   Users logged in:     1
      Memory usage:   5%              IP address for eth0: 16.86.192.114
      Swap usage:     0%

      Graph this data and manage this system at https://landscape.canonical.com/

    Last login: Wed Jun 26 14:31:30 2013 from kalevala.site
    russ@app01:~$ sudo bash
    root@app01:~# chef-client

Oops! I get an error...

If you fail to create a client in the Chef server web interface, copy its private key and paste it into /etc/chef/client.pem, you'll get an error such as:

    root@app01:/etc/chef# chef-client
    ...
    Chef encountered an error attempting to create the client "app01"
    ...

If you get the following error...

    Authentication Error:
    ---------------------
    Failed to authenticate to the chef server (http 401).
    The request failed because your clock has drifted by more than 15 minutes.
    Syncing your clock to an NTP Time source should resolve the issue.

...or something like:

    [2013-07-08T18:29:08+00:00] INFO: HTTP Request Returned 401 Unauthorized: Failed to authenticate. Please synchronize the clock on your client

    ================================================================================
    Chef encountered an error attempting to load the node data for "app01"
    ================================================================================

Trouble-shooting

Try these steps to trouble-shoot:

1. Is the Chef client node name identical to the hostname on which it's going to run? (This isn't totally crucial, but differences between the two can lead to trouble sometimes.)
2. Did you run knife configure client ./?
3. Is /etc/chef/client.pem the right/latest private key for this client in the Chef server web interface?
4. In /etc/chef/client.rb, is chef_server_url correctly set (i.e.: points to Chef server)? (Note, each time you run knife configure client, this will be clobbered.)
5. Is ntp installed?
6. Is /etc/ntp.conf updated to reflect the time server? (Hint: if you're behind a firewall, this is most likely what's biting you.)
7. Does /etc/timezone match what's on your Chef server's host? If this is already set up on other nodes and the Chef server, edit this file by hand and copy what's in those configurations (should be identical) to the host you're trouble-shooting.
8. Have you bounced ntp?
9. Did you try bouncing the host? Surprisingly more effective than the previous suggestion.
10. Have you compared the output of date against the same on your Chef server or another, working node? Sometimes, you just need to wait awhile (only a minute or two) and reboot before it aligns with the time server. Reboot a couple of times; the output of date is the dead give-away this is working.

Appendix: Finished filesystem (illustration)

At the end of this tutorial, here's the /home/russ/chef-repos filesystem:

    ~ $ tree chef-repos
    chef-repos
     |— .chef
     |   |— knife.rb
     |   `— russ.pem
     |— cookbooks
     |   |— tomcat6
     |   |   |— attributes
     |   |   |   `— default.rb
     |   |   |— files
     |   |   |   `— default
     |   |   |       `— apache-tomcat-6.0.37.tar.gz
     |   |   |— metadata.rb
     |   |   |— README.rdoc
     |   |   |— recipes
     |   |   |   `— default.rb
     |   |   `— templates
     |   |       `— default
     |   |           |— context.xml.erb
     |   |           `— server.xml.erb
     |   `— xyz
     |       |— files
     |       |   `— default
     |       |— metadata.rb
     |       |— README.rdoc
     |       `— recipes
     |— nodes
     |   `— app01.json
     `— roles
         `— application_node.rb

More...

Appendix: Files changed in the Tomcat recipe

attributes/default.rb:

The biggest changes are here. If you are familiar with Tomcat, they are easily made. Otherwise, examine these and adapt to your environment. In particular, the ones in green almost certainly need to be different for you.

Of course, this is all overkill. When you install Tomcat, you don't usually need to modify the default server.xml. However, quibbling about this is beyond the scope of this document.

    node.default[:installPath] = "/usr/share/tomcat6/"
    node.default[:tomcatPath] = "/var/lib/tomcat6/"
    node.default[:serverPort] = 8005
    node.default[:connectorPort] = 8080
    node.default[:ajpPort] = 8009
    node.default[:hostIP] = "16.86.192.114"
    node.default[:multicastIP] = "228.0.0.4"
    node.default[:multicastPort] = 45564
    node.default[:nioAddress] = "16.86.192.114"
    node.default[:nioPort] = 4000
    node.default[:tempDir] = "/var/lib/tomcat6/webapps/"
    node.default[:deployDir] = "/var/lib/tomcat6/webapps/"
    node.default[:watchDir] = "/tmp/war-listen/"

metadata.rb:

The values in green are what I changed.

    maintainer       "Despicable Me"
    maintainer_email "[email protected]"
    license          "Apache 2.0"
    description      "Apache Tomcat 6 configuration for cluster"
    long_description IO.read(File.join(File.dirname(__FILE__), 'README.rdoc'))
    version          "1.0.0"

README.rdoc:

This wouldn't have to be changed, but I changed it a bit to make it more relevant to my use.

    = DESCRIPTION:
    Chef cookbook for Apache Tomcat in clustered environment

    = REQUIREMENTS:
    Ubuntu

    = ATTRIBUTES:
    node.default[:installPath] - path to install tomcat
    node.default[:tomcatPath] - full tomcat path after installation
    node.default[:serverPort] - by default 8005
    node.default[:connectorPort] - by default 8080
    node.default[:ajpPort] - by default 8009
    node.default[:hostIP] - use the host address
    node.default[:multicastIP] - by default "228.0.0.4" (per Apache)
    node.default[:multicastPort] - by default 45564 (per Apache)
    node.default[:nioAddress] - use the host address
    node.default[:nioPort] - by default 4000
    node.default[:tempDir] - temp dir for deploying
    node.default[:deployDir] - dir to deploy WARs
    node.default[:watchDir] - dir to watch WARs

    For more information see opscode tutorials.

recipes/default.rb:

Remember, we want to use this for Tomcat 6, not Tomcat 7. So, lots of stuff has to change. We picked the latest Tomcat 6 build. Also, as I plan to use this on an Ubuntu platform, I had to change from "debian" because, even though Ubuntu is Debian, I got an error out of Ruby land if I didn't change this.

    #
    # Cookbook Name:: apache-tomcat
    # Recipe:: default
    #
    # Copyright 2011
    #
    # Licensed under the Apache License, Version 2.0 (the "License");
    # you may not use this file except in compliance with the License.
    # You may obtain a copy of the License at
    #
    #     http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.

    # TODO extract version to attributes

    # currently tested just with Debian
    if not node['platform'] == "ubuntu" then
      raise RuntimeError, "Unsupported platform: #{node['platform']}"
    end

    # make install folder if it doesn't exist
    execute "mkdir" do
      command "mkdir -p #{node[:installPath]}"
      action :run
    end

    # copy tomcat distrib to install folder
    cookbook_file "#{node[:installPath]}/apache-tomcat-6.0.37.tar.gz" do
      source "apache-tomcat-6.0.37.tar.gz"
      mode "0644"
    end

    # extract tomcat
    execute "tar" do
      command "tar -xvf #{node[:installPath]}/apache-tomcat-6.0.37.tar.gz --directory #{node[:installPath]}/"
      action :run
    end

    # remove archive from install folder
    execute "rm" do
      command "rm -i #{node[:installPath]}/apache-tomcat-6.0.37.tar.gz"
      action :run
    end

    # configure context.xml
    template "#{node[:installPath]}/apache-tomcat-6.0.37/conf/context.xml" do
      source "context.xml.erb"
      mode "0644"
    end

    # configure server.xml
    template "#{node[:installPath]}/apache-tomcat-6.0.37/conf/server.xml" do
      source "server.xml.erb"
      mode "0644"
    end

Appendix: A more concise list of steps and commands

The steps below are concrete, an attempt to condense them in order to make the whole visible, but assumptions about platform (Ubuntu Precise 12.04), Chef version (11.x), etc. have to be made. This appendix has worth mostly to remind and orient. Some steps are too complex to enumerate simply here and, accordingly, links are given to the deeper details.