Debian download for Ubuntu

(Please note that when I began these notes, Jenkins had been set up by a colleague getting its WAR and merely dropping it into Tomcat. The second time, I did it myself and used a more modern way, described here.)

$ wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
$ deb http://pkg.jenkins-ci.org/debian binary/
$ sudo apt-get update
$ sudo apt-get install jenkins

Now it's installed, where is it?

...and what do I do with it before setting up my job/project?

Git plug-in

Go to Manage Plugins.

Check the list of plug-ins Jenkins already has to ensure the Git plug-in isn't already there by clicking the Installed tab.

To install it, click the Available tab and type "git" into the Filter. Look for "Jenkins GIT plugin" and click the checkbox, then Install without restart. (You can restart later if need be.)

(Note: the last time I did this, I found the following useful plug-ins for Git and Jenkins: Git Plugin, Git Client Plugin, GitHub Plugin, GitHub pull request builder plugin.)

Setting up a new job

Job is synonymous with project when speaking Jenkins.

In my case, I've set Jenkins up to monitor my Git remotes and kick off a build every time someone pushes something back to remote master branch. Before the build will happen, though, I have to set up a job.

Jenkins failing to clone repository

Remember, it's not Jenkins that does the cloning. Jenkins is really user tomcat6. Here we're on a server named tuonela where Jenkins is running. We "log in" as tomcat6 (shown below) and do the other things in this illustration:

root@tuonela:~# sudo -s -H -u tomcat6
tomcat6@tuonela:~$ alias ll='ls -l'
tomcat6@tuonela:~$ mkdir .ssh
tomcat6@tuonela:~$ chmod a-rwx .ssh
tomcat6@tuonela:~$ chmod u+rwx .ssh
tomcat6@tuonela:~$ ls -dal .ssh
drwx------ 2 tomcat6 tomcat6 4096 2011-08-10 12:01 .ssh
tomcat6@tuonela:~$ cd .ssh
tomcat6@tuonela:~/.ssh$ which ssh-keygen
/usr/bin/ssh-keygen
tomcat6@tuonela:~/.ssh$ ssh-keygen -t rsa -C "Jenkins uses Tomcat"
tomcat6@tuonela:~/.ssh$ cp id_rsa.pub [email protected]
tomcat6@tuonela:~/.ssh$ vim README.txt
In this subdirectory, there is a key pair generated for Jenkins to be able to
use Tomcat to clone a Git repository. Don't erase or regenerate the private key,
id_rsa, or this will cease working. If you do mess this up, you'll have to
regenerate a new pair, rename id_rsa.pub to [email protected], replace the one in
in gitolite-admin/keydir, and commit it.

    -rw------- 1 tomcat6 tomcat6 1679 2011-08-10 11:15 id_rsa
    -rw-r--r-- 1 tomcat6 tomcat6  401 2011-08-10 11:15 id_rsa.pub
    -rw-r--r-- 1 tomcat6 tomcat6  401 2011-08-10 11:16 [email protected]

Also, do NOT muck with the permissions on this subdirectory. They must remain:

    drwx------   2 tomcat6 4096 2011-08-10 11:21 .ssh

or most applications will not honor the private key here.
tomcat6@tuonela:~/.ssh$ ll
total 28
drwx------ 2 tomcat6 tomcat6 4096 2011-08-12 08:38 .
drwxrwxrwx 7 root    root    4096 2011-08-12 08:38 ..
-rw------- 1 tomcat6 tomcat6 1679 2011-08-10 11:15 id_rsa
-rw-r--r-- 1 tomcat6 tomcat6  401 2011-08-10 11:15 id_rsa.pub
-rw-r--r-- 1 tomcat6 tomcat6  222 2011-08-10 12:01 known_hosts
-rw-r--r-- 1 tomcat6 tomcat6  751 2011-08-12 08:38 README.txt
-rw-r--r-- 1 tomcat6 tomcat6  401 2011-08-10 11:16 [email protected]
tomcat6@tuonela:~/.ssh$ cd ..
tomcat6@tuonela:~$ which git
tomcat6@tuonela:~$ git clone git@localhost:myproject
Cloning into myproject...
remote: Counting objects: 269, done.
remote: Compressing objects: 100% (229/229), done.
remote: Total 269 (delta 83), reused 0 (delta 0)
Receiving objects: 100% (269/269), 14.06 MiB | 8.42 MiB/s, done.
Resolving deltas: 100% (83/83), done.
tomcat6@tuonela:~$ rm -rf myproject/

Failed to access database

Similarly, for MySQL, Jenkins has to be able to reach it. Because a MySQL user need not be an actual user on the host, we'll postulate user jenkins. Initially, we don't have such a user. We'll create it and give it privileges because, for our application, Jenkins needs to run database scripts.

root@tuonela:~# mysql -u jenkins -pjenkins
ERROR 1045 (28000): Access denied for user 'jenkings'@'localhost' (using password: YES)
root@tuonela:~# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 79
Server version: 5.1.54-1ubuntu4 (Ubuntu)

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL v2 license

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> GRANT ALL PRIVILEGES ON acme.* TO 'jenkins'@'localhost' IDENTIFIED BY 'jenkins';
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
mysql> quit
Bye

Failed to find "Default" target

If you see this in the Console output of your Jenkins build, ...

BUILD FAILED
Target "Default" does not exist in the project "project-name".

...it's because you failed to specify your default target, i.e.: the ant target you want Jenkins to build. To fix this, go to your project in Jenkins, Configure -> Build -> Targets and type in the name of the target you'd like it to build.

How to update Jenkins

You might see something like this:

To update Jenkins (clicking on this will not usually work), get to the host running it, get root and use Aptitude:

    $ apt-get update
    $ apt-get install jenkins

How to fix e-mail notification list

The actual configuration file is on the path /usr/share/tomcat6/.jenkins/jobs/<project-name>/config.xml. I tried to use the following method, which did not appear immediately to modify the file on the path above (so I also modified that file, but maybe I just wasn't patient enough or, I needed to bounce Tomcat):

1. Go to the Dashboard page.
2. Find the representation of a job, named <project-name>.
3. Look for and click a hypertext link by the name of the project.
4. You get a page entitled <project-name>.
5. Look for an click a link, Configure.
6. You get a page with options for <project-name> jobs.
7. Scrolling down, see configuration option "E-mail Notification".
8. Modify the list of recipients, white-spaced, in "Recipients".

How to set up e-mail from Jenkins

You'll need a service for Jenkins to send e-mail notifications when the project build is broken. This doesn't come for free. If it's not already done, you'll see something like...

BUILD FAILED
...

Total time: 0 seconds
Build step 'Invoke Ant' marked build as failure
Sending e-mails to: [email protected]
ERROR: Could not connect to SMTP host: localhost, port: 25
javax.mail.MessagingException: Could not connect to SMTP host: localhost, port: 25;
  nested exception is:
	java.net.ConnectException: Connection refused
	at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1934)
	at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:638)
	at javax.mail.Service.connect(Service.java:295)
	at javax.mail.Service.connect(Service.java:176)
	at javax.mail.Service.connect(Service.java:125)
	at javax.mail.Transport.send0(Transport.java:194)
	at javax.mail.Transport.send(Transport.java:124)
	at hudson.tasks.MailSender.execute(MailSender.java:115)
	at hudson.tasks.Mailer.perform(Mailer.java:115)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:717)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:692)
	at hudson.model.Build$BuildExecution.post2(Build.java:183)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:639)
	at hudson.model.Run.execute(Run.java:1527)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:236)
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
	at java.net.Socket.connect(Socket.java:546)
	at com.sun.mail.util.SocketFetcher.createSocket(SocketFetcher.java:286)
	at com.sun.mail.util.SocketFetcher.getSocket(SocketFetcher.java:231)
	at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1900)
	... 17 more
Finished: FAILURE

First, ensure Jenkins has the e-mail plug-in. You can install this via Manage Jenkins -> Manage Plugins -> Available. Search for Email-ext plugin and click Download now and install after restart (at the very bottom of the very long page. You can go to Jenkins' plugins subdirectory to ensure it's there:

drwxr-xr-x  6 jenkins users     4096 Oct  9 19:34 email-ext/
-rw-r--r--  1 jenkins users   174730 Oct  9 19:33 email-ext.jpi

Bounce Jenkins and you have it.

Next, choose what service will be used to send e-mail. There are posts Google-able that show you how to do it via Yahoo, Google, etc. I chose to use Linux/Unix sendmail.

root@acme-buildserver:/var/lib/jenkins/plugins# apt-get install sendmail

Last, ensure your Jenkins settings are ready. Go to Manage Jenkins -> Configure System. Scroll all the way down to E-mail Notification and set these values (for sendmail).

SMTP Server                   localhost
Default user e-mail suffix    (nothing)
Sender E-mail Address         [email protected]

There's also a checkbox to Test configuration by sending test e-mail to see if it works or you can launch a build you know fails.

How to debug a build

What's sweet about Jenkins is that you can go to where it does the build and launch it yourself to see what goes wrong in its environment. This is particularly useful for early builds while you're shaking the bugs out of your Jenkins set-up. This is where you go:

/usr/share/tomcat6/.jenkins/jobs/acme/workspace

Ownership and permission issues

Whatever you do as you screw around with stuff to get it working, be super careful that all the subdirectories and files belong to tomcat6 or jenkins (which of these depends on how you installed it and how it's running) have the right ownership and permissions or you'll break normal usage.

Appendix: Links

Appendix: Sample job configuration

Appendix: Deploying a WAR to remote Tomcat

In the post-action section of the application project configuration in Jenkins,

1. Check box next to Publish artifacts to SCP repository.
2. Select appropriate scp site (other host) in drop-down.
3. Specify artifact to copy (the newly built WAR file) relative to root of Jenkins (Eclipse) project, e.g.: build/thing.war.
4. Specify destination relative to path set up configuring scp plug-in. If you already said /var/lib/tomcat/webapps, then there is no need to specify anything here.

Typically, Tomcat running on the target host will already know it's to explode and deploy WARs as soon as they arrive.

The problem with this deploy method...

Solution for multiple deployments by Jankins

The definitive solution appears to be:

1. Do not use the Jenkins scp plug-in.
2. Set up a user named jenkins user on the deployment target hosts.
3. Write a script called from build.xml in the final Jenkins target that copies the newly built WAR to the target hosts.

Here's a script example. It copies myapp.war to two application server hosts, app-1 and app-2:

deploy.sh:

    #!/bin/sh
    # Deploy our WAR to app-1 and app-2.
    scp ./build/deploy/myapp.war jenkins@app-1:/var/lib/tomcat6/webapps
    scp ./build/deploy/myapp.war jenkins@app-2:/var/lib/tomcat6/webapps

And, here's a sample ant invocation:

build.xml:

    .
    .
    .
    <property name="deploy.script"  value="deploy.sh" />
    .
    .
    .
    <!-- == j e n k i n s - b u i l d - a n d - d e p l o y ================ -->
    <target name="jenkins-build"
            depends="test-run,build-war"
            description="Build WAR (Jenkins-only target).">
        <echo message="Jenkins building for deployment..." />
    </target>

    <target name="jenkins-deploy"
            depends="jenkins-build"
            description="Deploy built WAR (Jenkins-only target).">
        <echo message="Jenkins deploying built WAR..." />
        <exec executable="/bin/bash">
            <arg value="${deploy.script}" />
        </exec>
    </target>
    .
    .
    .

Appendix: Random troubles

Jenkins update

One day in February, 2013, we updated Jenkins to get some new features via Aptitude on Ubuntu Precise Server. We had Jenkins running as a user with originally, as I recall from having installed it (but I could be mistaken), everything underneath /home/jenkins.

Immediately, all our builds began failing. It appears that two things were awry as a result of the update, which put everything under /var/lib/jenkins instead of the aforementioned user.

1. The RSA key used with GitHub was missing from the path /var/lib/jenkins/.ssh. In fact, there was no .ssh subdirectory. We copied that subdirectory, including the key, from /home/jenkins/.ssh.
2. Then, the build began failing with a note to the effect that Git didn't know who we were. The solution was to do this:

       $  git config --global user.email "[email protected]"*
       $  git config --global user.name jenkins
   		

^* You may ask why this e-mail address? Uh, what other since Jenkins is at best a user on that Linux host and implies no specific e-mail identity.

Jenkins date and time issues

I finally became annoyed enough with having to translate out of UTC which Jenkins defaults to that I went in and added this line to /etc/defaults/jenkins:

    JAVA_ARGS="-Dorg.apache.commons.jelly.tags.fmt.timeZone=America/Denver"

And bounced Jenkins from /etc/init.d—the GUI won't cut it—I don't know if this sticks or not then next I bounce from the GUI.

Appendix: Migrating Jenkins to a new server

If you need to move your existing Jenkins set-up to a new server, this is possible and easier to do than setting it up all over again by hand:

Appendix: Additional software installations

...besides sendmail, here are some other that may be useful:

Java (revisited)

—As already discussed, a JDK is mandatory. I usually download a proper Sun JDK to leave nothing to chance and put it on the path, e.g.: /usr/java/jdk1.6.0_45, then link to it thus (both 6 and 7 shown):

    root@acme-jenkins:/usr/java # ln -s ./jdk1.6.0_45 ./java6
    root@acme-jenkins:/usr/java # ln -s ./jdk1.7.0_25 ./java7

...but it's important to note that you'll want to set JAVA_HOME in Jenkin's profile if not already done:

/var/lib/jenkins/.profile:

    export JAVA_HOME=/usr/java/java7

Gradle

Sonar

JaCoCo

Appendix: Some illustrations of workflow

What someone has termed, "the Jenkins spotless house." This delays deployment until after all test phases have run—possibly taking hours. Unit tests only take seconds or minutes whereas functional and, especially, load testing takes potentially much longer.

The "clean-enough house" conceived as a way to speed up the workflow by accepting that the load testing, which takes a very long time, will only come in at a later point, probably long after actual deployment.