Publishing Django Test Coverage Reports in Jenkins

This post is built on some assumptions.

First, I assume that you already know that writing unit tests is good for you. Well, to be honest, if you are not systematically writing tests for your code, you shouldn’t be calling yourself a software engineer anyway. No excuses.

In consequence I also assume that your latest Django project includes its dose of unit testing. But do you have a clear idea of which parts of your Django site are not being tested? Are you taking action to improve on that area? In other words, are you already obtaining and analysing coverage data for your project?

If so, lucky you. I didn’t, decided it was about time, and set out to the task.

I will try to demistify the process, since it takes very little effort and you can reap substantial benefits from it – provided, of course, that you take a look at the coverage reports on a regular basis, and add tests for the uncovered methods… but you promise you will do that, won’t you? Great!

We will start by generating the reports manually, and will then move on to automating them into Jenkins, our friendly build butler.

Generating Coverage Reports with coverage.py

Precondition: you have a more or less running Django project, that already includes some tests (somewhere where you can run manage.py test).

First you will need to install coverage.py:

$ pip install coverage

After installation, you just have to run coverage somescript.py where you used to run python somescript.py. Yup, that simple. Coverage data will be collected during execution, and you will be able to generate nice reports afterwards.

Since I’m too lazy for repeatedly typing the same long command lines, or searching through shell history, I wrote a script cover.sh at the root directory of my project that calls coverage with all the options I want:

#!/bin/bash

# clean up previous results:
rm htmlcov/*
rm .coverage

# I do not want coverage data for my South migrations:
PARMS=--omit='*migrations*'

# run the tests and collect coverage, only for our applications
coverage run --source=license,customer,updates,ism sites/ismbackend/manage.py test

# generate plaintext and HTML report
echo "----------------------------"
echo "Coverage results:"
coverage report $PARMS
coverage html $PARMS
echo "HTML report generated in htmlcov/index.html"

# optionally display an HTML report
if [ "$1" == "-f" ]
then
  firefox htmlcov/index.html
fi

With --source you can control for which modules you want to collect coverage data (in my case, a number of django apps in my project), and with --omit you can set file patterns which you want to exclude from your collection and report (e.g. South migrations).

Now cover.sh will generate text output and HTML files, and cover.sh -f will additionally open firefox and display the HTML reports. The HTML reports will look like this. Click on a file name and you will see the source code, with green for lines that have been run, while the uncovered lines will be highlighted in red.

Collecting Coverage Data within Jenkins

Now that we can collect coverage data from the command line, we are ready to go one step further and automate this within our favorite continuous integration server.

Pre-requisites for this step: a running Jenkins install.

In Jenkins, I set up my project’s build as a free-form project. The build command is this shell script:

#!/bin/bash

# select the virtual environment
cd ${WORKSPACE}/ism-backend
export WORKON_HOME=/home/isigma/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh
workon ismbackend

# update dependencies and run the tests
pip install -r requirements.txt
export DJANGO_SETTINGS_MODULE="sites.ismbackend.settings"
export PYTHONPATH=.
./cover.sh

With this build script, the test coverage report will be generated at the end. Now we only need to tell Jenkins to publish a link to it after build. For that, we will go to Jenkins’ plugin management page and install the Jenkins HTML Publisher Plugin. This is a simple but effective plugin that allows you to publish a link to any HTML file generated within your build.

Following the instructions from the plugin page, for this project I checked the option to publish an HTML report after the build. Set the fields to:

  • HTML Directory to archive: ism-backend/htmlcov
  • Index page[s]: index.html
  • Report title: Test Coverage Report

And that’s all there is to it. After the next build, Jenkins will display in your job’s page a report icon and a link with the title “Test Coverage Report”, pointing to the nice HTML reports generated by coverage.py

Take a look at the classes with a poor coverage (calculated as the percentage of lines of code that were run), and you will easily see what your tests are not covering.

Enjoy!

About these ads

, , , , ,

  1. #1 by lidaobing on 2011/12/12 - 09:00

    not enough, we need draw a chart on how coverage ratio changed

    • #2 by Carles Barrob├ęs on 2011/12/13 - 08:08

      Since I wrote this article I learnt of a different way that people use to integrate coverage reports with Jenkins, and consists of using Cobertura-style reports (XML) and the Cobertura plugin.

      This will give you a graph of coverage statistics over time, as you desire.

      In short:

      • Use the option –with-xunit in coverage run
      • Generate an XML report with coverage xml
      • Install the cobertura plugin, and activate it for your job by checking “Publish Cobertura Coverage Report” and setting “cobertura XML report pattern” to “**/coverage.xml”

      The plugin: https://wiki.jenkins-ci.org/display/JENKINS/Cobertura+Plugin
      (forget about all of the maven-specific steps).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: