Multiplying Python Unit Test Cases with Different Sets of Data

…or Data-Driven Tests

Often I find myself in the situation of wanting to run some unit test for a part of my system with different data values. I want to verify that the system works correctly with some combinations of data – and not just have a single good case test with a specific combination of parameters (which is often what people are contented with).

I do not want to create a test that loops over the test data, exercises the code, and verifies the correct execution with some assertions… that would make for a single test case that would fail at the first data combination that doesn’t work. I want it to be run as one test case per test data value.

The typical naive approach to this is to write a method that runs the actual test and performs assertions on the results (say verify_xxx_works), and create some test_xxx_* methods that call that one with different data values. Boy is that lame.

Fortunately, nose includes the concept of test functions as generators. If you write your test as a generator that spits out tuples of (callable, argument...), nose will call it and run one test case per yielded value, thus effectively multiplying your number of test cases. You will see OKs for values that pass, and one failure per each set of arguments that fails. Great!

Oops, wait, not so great. If you read the docs carefully (I didn’t on my first try) you will find the small print: Please note that method generators are not supported in unittest.TestCase subclasses

Meaning that if your tests are written using unittest.TestCase you’re on your own again.

Unhappy with the situation of not being able to run one TestCase method with different sets of data in a non-clumsy way, I’ve been playing around and the result is a small library that I’ve called “DDT” (which could either stand for Data-Driven or Data-Decorated Tests). It allows developers to create multiple test cases from a single test method, feeding it different data using a decorator.

So let’s say you want to test a function you just wrote named larger_than_two. You could start with the following test:

# tests.py
import unittest
from ddt import ddt, data
from mycode import larger_than_two

class FooTestCase(unittest.TestCase):

    def test_larger_than_two(self):
        self.assertTrue(larger_than_two(3))

    def test_not_larger_than_two(self):
        self.assertFalse(larger_than_two(1))

DDT consists of a class decorator ddt (for your TestCase subclass) and a method decorator data (for your tests that want to be multiplied).
This allows you to change your tests to:

# tests.py
import unittest
from ddt import ddt, data
from mycode import larger_than_two

@ddt
class FooTestCase(unittest.TestCase):

    @data(3, 4, 12, 23)
    def test_larger_than_two(self, value):
        self.assertTrue(larger_than_two(value))

    @data(1, -3, 2, 0)
    def test_not_larger_than_two(self, value):
        self.assertFalse(larger_than_two(value))

And then run them with:

 % nosetests tests.py
........
----------------------------------------------------------------------
Ran 8 tests in 0.002s

OK

2 test methods + some magic decorator = 8 test cases.

Boy do I love Python.

How does it work?

Here is a little insight into how DDT works. (For the full code take a look at ddt.py in GitHub).

  • The data method decorator just adds an extra magic attribute to the test method, which contains the data values it must be run with.
  • The ddt class decorator takes all methods that have been decorated with @data (i.e. those that include the mentioned extra magic attribute) and replaces them with as many tweaked copies of themselves as arguments were passed to data. Each tweaked copy is in fact a decorated version of the original method being fed the corresponding test data value.

Get DDT

  • DDT is in PyPI, you can install it with: pip install ddt
  • You can also find DDT source code in GitHub, feel free to fork me.
  • Documentation can be found at Read the Docs

, , ,

  1. #1 by Mark Simpson on 2012/05/15 - 16:38

    Nice one. Possibly a bit hacky (total python noob here, so there may well be a much more elegant way to achieve this), but I’ve done the following to pseudo extend it, too:

    # via http://stackoverflow.com/a/1123054/83891

    class TestData:
        def __init__(self, **entries):
            self.__dict__.update(entries)
    

    Then, rather than using tuples/dictionaries when passing multiple arguments to a test, you can do something like

    @data(TestData(input=3, multiply_by=2, expected_output=6),
                TestData(input=2, multiply_by=1, expected_output=2))
    def test_multiply(self, test_case):
        result = test_case.input * test_case.multiply_by
        self.assertEquals(result, test_case.expected_result)
    

    Is there a better way to achieve this? Cheers again.

    • #2 by Carles Barrobés on 2012/05/16 - 08:47

      It is a question of taste, whether one would prefer this to using dictionary instances. E.g. test_case.input vs test_case['input'] may be considered a marginal gain for the extra required boilerplate.

      I came to Python after a long time in Java, and one of the things I found is that Python people seem to be confortable using dictionaries where Java people would have been creating custom classes. I think part of the reason is because Java is really bad at dictionaries, the syntax is too verbose, and also because Java gets you into the habit of strong typing. After using Python for a while, I find myself with a growing allergy to the amount of boilerplate required by other languages.

      That said, since with ddt you are free to pass whatever objects as test data, this is a perfectly valid way of using it. It is not something I would add as a builtin part of the library though.

    • #3 by Carles Barrobés on 2014/02/21 - 22:30

      BTW we have recently implemented an @unpack decorator that does what you need.

      • #4 by Mark Simpson on 2014/02/24 - 10:18

        Thanks! We’ve been using ddt on our python project quite extensively, so I’ll let you know how we get on with @unpack.

  2. #5 by Mark Simpson on 2012/05/16 - 09:21

    I thought that might be the case. I’m more of a .NET guy day to day, so that’s probably why I felt the need to avoid a dictionary (probably unnecessarily)

    I think a slightly better format might be something like the way NUnit does data-driven testing for .NET — it uses reflection on the data contained in the attribute and then figures out the arguments to pass to the test function like this: http://nunit.org/?p=testCase&r=2.5

    It could look something like this:
    https://gist.github.com/2705947

    I’m possibly conflating two entirely different things though, as I know decorators aren’t an analogue of attributes. When I have a moment I’ll do some experimentation, though.

    Anyway, cheers for the package, it’s really helpful!

  3. #6 by Stéphane Ancelot on 2013/01/23 - 16:59

    If you want the tests ordered in the way you wrote the TestData lines , modify ddt.py line 48 as follow :

    test_name = getattr(v, “__name__”, “{0}{2:06d}_{1}”.format(name, v,i))

  4. #7 by waqasabdul on 2014/01/15 - 22:13

    Does this support setUp and tearDown methods as well? i.e. will it call setUp and tearDown for each test data value?
    As you described the working above, it looks like it doesn’t. I suppose, as a workaround, we can simulate setUp and tearDown using try/catch inside the test method.

    • #8 by Carles Barrobés on 2014/01/15 - 22:33

      If you have tried it by now, you will have realised that it does. Effectively the way it works, for each data value you get a test method. So TestCase works as it would normally do if you had written each of these methods by hand.

      • #9 by waqasabdul on 2014/01/15 - 22:58

        setUp and tearDown are called but they don’t get test data. Test method is getting test data as additional parameter, but setUp and tearDown are not. So they cannot do custom setup or teardown based on value of test data.

      • #10 by Carles Barrobés on 2014/02/21 - 22:33

        Oh I see what you mean. Feel free to submit an issue to https://github.com/txels/ddt/issues describing how you would like this to work.

        I think this can be extremely tricky to handle without quite some black magic, due to how unittest.TestCase works. I don’t know how you would like to write the method signatures for setUp and tearDown.

  5. #11 by Michael Cookson on 2014/02/17 - 17:40

    This is awesome. Does anyone know if it’s possible to give each generated test it’s own set of @attr tags? These are the tags that nose uses to select the tests to run. I love using DDT as much as possible, but one problem is that they always have to share a set of tags.

    • #12 by Carles Barrobés on 2014/02/21 - 22:28

      You can submit an issue to https://github.com/txels/ddt/issues with a suggestion of how you think this should look/work, and I’ll see what can be done.

      A pull request would be even better :)

  1. Data-Driven Testing with Python | Mark's Devblog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: