Multiplying Python Unit Test Cases with Different Sets of Data

…or Data-Driven Tests

Often I find myself in the situation of wanting to run some unit test for a part of my system with different data values. I want to verify that the system works correctly with some combinations of data – and not just have a single good case test with a specific combination of parameters (which is often what people are contented with).

I do not want to create a test that loops over the test data, exercises the code, and verifies the correct execution with some assertions… that would make for a single test case that would fail at the first data combination that doesn’t work. I want it to be run as one test case per test data value.

The typical naive approach to this is to write a method that runs the actual test and performs assertions on the results (say verify_xxx_works), and create some test_xxx_* methods that call that one with different data values. Boy is that lame.

Fortunately, nose includes the concept of test functions as generators. If you write your test as a generator that spits out tuples of (callable, argument...), nose will call it and run one test case per yielded value, thus effectively multiplying your number of test cases. You will see OKs for values that pass, and one failure per each set of arguments that fails. Great!

Oops, wait, not so great. If you read the docs carefully (I didn’t on my first try) you will find the small print: Please note that method generators are not supported in unittest.TestCase subclasses

Meaning that if your tests are written using unittest.TestCase you’re on your own again.

Unhappy with the situation of not being able to run one TestCase method with different sets of data in a non-clumsy way, I’ve been playing around and the result is a small library that I’ve called “DDT” (which could either stand for Data-Driven or Data-Decorated Tests). It allows developers to create multiple test cases from a single test method, feeding it different data using a decorator.

So let’s say you want to test a function you just wrote named larger_than_two. You could start with the following test:

# tests.py
import unittest
from ddt import ddt, data
from mycode import larger_than_two

class FooTestCase(unittest.TestCase):

    def test_larger_than_two(self):
        self.assertTrue(larger_than_two(3))

    def test_not_larger_than_two(self):
        self.assertFalse(larger_than_two(1))

DDT consists of a class decorator ddt (for your TestCase subclass) and a method decorator data (for your tests that want to be multiplied).
This allows you to change your tests to:

# tests.py
import unittest
from ddt import ddt, data
from mycode import larger_than_two

@ddt
class FooTestCase(unittest.TestCase):

    @data(3, 4, 12, 23)
    def test_larger_than_two(self, value):
        self.assertTrue(larger_than_two(value))

    @data(1, -3, 2, 0)
    def test_not_larger_than_two(self, value):
        self.assertFalse(larger_than_two(value))

And then run them with:

 % nosetests tests.py
........
----------------------------------------------------------------------
Ran 8 tests in 0.002s

OK

2 test methods + some magic decorator = 8 test cases.

Boy do I love Python.

How does it work?

Here is a little insight into how DDT works. (For the full code take a look at ddt.py in GitHub).

• The data method decorator just adds an extra magic attribute to the test method, which contains the data values it must be run with.
• The ddt class decorator takes all methods that have been decorated with @data (i.e. those that include the mentioned extra magic attribute) and replaces them with as many tweaked copies of themselves as arguments were passed to data. Each tweaked copy is in fact a decorated version of the original method being fed the corresponding test data value.

Get DDT

• DDT is in PyPI, you can install it with: pip install ddt
• You can also find DDT source code in GitHub, feel free to fork me.
• Documentation can be found at Read the Docs

• nose
• python
• test
• unittest