Using Git submodules for unified build scripts

I have recently worked on unifying the build scripts for many of our software components at Glasses Direct.

At each repository for a component, we typically have two files build.sh (for developer builds, relying on an existing virtualenv) and jenkins-build.sh (for Jenkins builds, where the Jenkins job re-creates the virtualenv if it does not exist or if the requirements files have changed since last build – we rely on the md5 checksums for that).

We already had sort-of-standardised scripts, but they were replicated in many repos. It defeated the point of having a single source for these. I decided to give git submodules a try, as a way to make code from one repo (our build scripts) easily accessible from others.

Adding the submodule

Our build scripts are (far than ideally) in a specific branch buildtools of a repo called libraries.

Here is how we add that submodule:

git submodule add git@github.com:glassesdirect/libraries.git

If we run git status we will see git has added a .gitmodules and a libraries directory.

Now we want to point to the desired branch:

cd libraries
git checkout buildtools
cd ..
git add libraries  # for the branch change

Once that is done, we can commit and add the changes.

The resulting scripts

Now the build scripts in each repo can look as simple as… build.sh:

#!/bin/bash
source libraries/buildtools/build-functions.sh
do_build lab_export_service

And jenkins-build.sh:

#!/bin/bash
source libraries/buildtools/build-functions.sh
do_jenkins_build $*

The bulk of our build scripts is now in libraries/buildtools/build-functions.sh. There we can have the basic building blocks as bash functions, such as:

function check_if_requirements_changed() {
# Verify whether any of the PIP requirements file has changed.
# This may save us the creation of a virtual environment if all is equal.

   MD5FILE=".requirements.md5"
   newMD5=`md5sum *requirements.txt | md5sum`
   oldMD5=`cat $MD5FILE`

   if [ "${oldMD5}" == "${newMD5}" ]
   then
      REQS_CHANGED=0
   else
      echo "Requirements files changed."
      REQS_CHANGED=1
   fi
}

etc.

Remote-Control Jenkins from Python

Motivation

I just recently changed jobs: my new development team is larger, it’s a very dynamic environment with multiple branches being created in git repositories all the time, and people are creating Jenkins build jobs in order to run automated builds and tests for these branches on a daily basis.

At some point the manual copy-pasting of Jenkins jobs that was the general practice was beginning to annoy me and I decided to give a try to Jenkins’ HTTP API. The goal was to be able to script the creation of jobs, from consistent updateable descriptions. The result has been the creation of the autojenkins package written in Python, that will allow us to query the status of Jenkins jobs, trigger manual builds, create new ones, and cleanup by removing old jobs.

Using Autojenkins

Autojenkins is published on PyPI, you install it simply with:
pip install autojenkins

Once installed, it provides an easy to use Python client for the Jenkins HTTP API. Here’s a taste of the API with some sample usage:

from autojenkins import Jenkins

j = Jenkins('http://jenkins.pe.local')

# trigger a manual build and check results
j.build('my-job')
j.last_result('my-job')

# get only the result string (one of 'SUCCESS', 'UNSTABLE', 'FAILURE'):
j.last_result('my-job')['result']

# get the configuration file for a job:
j.get_config_xml('template')

# Create a new job from a job named 'template', replacing variables
j.create_copy('my-new-job', 'template',
              repo='my-repo',
              branch='my-branch')

# build
j.build('my-new-job')

# check result and delete if successful:
result = j.last_result('my-new-job')['result']
if result == 'SUCCESS':
    j.delete('my-new-job')

Publishing Django Test Coverage Reports in Jenkins

This post is built on some assumptions.

First, I assume that you already know that writing unit tests is good for you. Well, to be honest, if you are not systematically writing tests for your code, you shouldn’t be calling yourself a software engineer anyway. No excuses.

In consequence I also assume that your latest Django project includes its dose of unit testing. But do you have a clear idea of which parts of your Django site are not being tested? Are you taking action to improve on that area? In other words, are you already obtaining and analysing coverage data for your project?

If so, lucky you. I didn’t, decided it was about time, and set out to the task.

I will try to demistify the process, since it takes very little effort and you can reap substantial benefits from it – provided, of course, that you take a look at the coverage reports on a regular basis, and add tests for the uncovered methods… but you promise you will do that, won’t you? Great!

We will start by generating the reports manually, and will then move on to automating them into Jenkins, our friendly build butler.