Multiplying Python Unit Test Cases with Different Sets of Data

…or Data-Driven Tests

Often I find myself in the situation of wanting to run some unit test for a part of my system with different data values. I want to verify that the system works correctly with some combinations of data – and not just have a single good case test with a specific combination of parameters (which is often what people are contented with).

I do not want to create a test that loops over the test data, exercises the code, and verifies the correct execution with some assertions… that would make for a single test case that would fail at the first data combination that doesn’t work. I want it to be run as one test case per test data value.

The typical naive approach to this is to write a method that runs the actual test and performs assertions on the results (say verify_xxx_works), and create some test_xxx_* methods that call that one with different data values. Boy is that lame.

Fortunately, nose includes the concept of test functions as generators. If you write your test as a generator that spits out tuples of (callable, argument...), nose will call it and run one test case per yielded value, thus effectively multiplying your number of test cases. You will see OKs for values that pass, and one failure per each set of arguments that fails. Great!

Oops, wait, not so great. If you read the docs carefully (I didn’t on my first try) you will find the small print: Please note that method generators are not supported in unittest.TestCase subclasses

Meaning that if your tests are written using unittest.TestCase you’re on your own again.

Unhappy with the situation of not being able to run one TestCase method with different sets of data in a non-clumsy way, I’ve been playing around and the result is a small library that I’ve called “DDT” (which could either stand for Data-Driven or Data-Decorated Tests). It allows developers to create multiple test cases from a single test method, feeding it different data using a decorator.

So let’s say you want to test a function you just wrote named larger_than_two. You could start with the following test:

import unittest
from ddt import ddt, data
from mycode import larger_than_two

class FooTestCase(unittest.TestCase):

    def test_larger_than_two(self):

    def test_not_larger_than_two(self):

DDT consists of a class decorator ddt (for your TestCase subclass) and a method decorator data (for your tests that want to be multiplied).
This allows you to change your tests to:

import unittest
from ddt import ddt, data
from mycode import larger_than_two

class FooTestCase(unittest.TestCase):

    @data(3, 4, 12, 23)
    def test_larger_than_two(self, value):

    @data(1, -3, 2, 0)
    def test_not_larger_than_two(self, value):

And then run them with:

 % nosetests
Ran 8 tests in 0.002s


2 test methods + some magic decorator = 8 test cases.

Boy do I love Python.

How does it work?

Here is a little insight into how DDT works. (For the full code take a look at in GitHub).

  • The data method decorator just adds an extra magic attribute to the test method, which contains the data values it must be run with.
  • The ddt class decorator takes all methods that have been decorated with @data (i.e. those that include the mentioned extra magic attribute) and replaces them with as many tweaked copies of themselves as arguments were passed to data. Each tweaked copy is in fact a decorated version of the original method being fed the corresponding test data value.


  • DDT is in PyPI, you can install it with: pip install ddt
  • You can also find DDT source code in GitHub, feel free to fork me.
  • Documentation can be found at Read the Docs

22 thoughts on “Multiplying Python Unit Test Cases with Different Sets of Data

  1. Nice one. Possibly a bit hacky (total python noob here, so there may well be a much more elegant way to achieve this), but I’ve done the following to pseudo extend it, too:

    # via

    class TestData:
        def __init__(self, **entries):

    Then, rather than using tuples/dictionaries when passing multiple arguments to a test, you can do something like

    @data(TestData(input=3, multiply_by=2, expected_output=6),
                TestData(input=2, multiply_by=1, expected_output=2))
    def test_multiply(self, test_case):
        result = test_case.input * test_case.multiply_by
        self.assertEquals(result, test_case.expected_result)

    Is there a better way to achieve this? Cheers again.

    • It is a question of taste, whether one would prefer this to using dictionary instances. E.g. test_case.input vs test_case['input'] may be considered a marginal gain for the extra required boilerplate.

      I came to Python after a long time in Java, and one of the things I found is that Python people seem to be confortable using dictionaries where Java people would have been creating custom classes. I think part of the reason is because Java is really bad at dictionaries, the syntax is too verbose, and also because Java gets you into the habit of strong typing. After using Python for a while, I find myself with a growing allergy to the amount of boilerplate required by other languages.

      That said, since with ddt you are free to pass whatever objects as test data, this is a perfectly valid way of using it. It is not something I would add as a builtin part of the library though.

  2. I thought that might be the case. I’m more of a .NET guy day to day, so that’s probably why I felt the need to avoid a dictionary (probably unnecessarily)

    I think a slightly better format might be something like the way NUnit does data-driven testing for .NET — it uses reflection on the data contained in the attribute and then figures out the arguments to pass to the test function like this:

    It could look something like this:

    I’m possibly conflating two entirely different things though, as I know decorators aren’t an analogue of attributes. When I have a moment I’ll do some experimentation, though.

    Anyway, cheers for the package, it’s really helpful!

  3. Pingback: Data-Driven Testing with Python | Mark's Devblog
  4. If you want the tests ordered in the way you wrote the TestData lines , modify line 48 as follow :

    test_name = getattr(v, “__name__”, “{0}{2:06d}_{1}”.format(name, v,i))

  5. Does this support setUp and tearDown methods as well? i.e. will it call setUp and tearDown for each test data value?
    As you described the working above, it looks like it doesn’t. I suppose, as a workaround, we can simulate setUp and tearDown using try/catch inside the test method.

    • If you have tried it by now, you will have realised that it does. Effectively the way it works, for each data value you get a test method. So TestCase works as it would normally do if you had written each of these methods by hand.

      • setUp and tearDown are called but they don’t get test data. Test method is getting test data as additional parameter, but setUp and tearDown are not. So they cannot do custom setup or teardown based on value of test data.

      • Oh I see what you mean. Feel free to submit an issue to describing how you would like this to work.

        I think this can be extremely tricky to handle without quite some black magic, due to how unittest.TestCase works. I don’t know how you would like to write the method signatures for setUp and tearDown.

  6. This is awesome. Does anyone know if it’s possible to give each generated test it’s own set of @attr tags? These are the tags that nose uses to select the tests to run. I love using DDT as much as possible, but one problem is that they always have to share a set of tags.

  7. How to put a list variable in @data, and test should run for each item of list, for example:
    list1 = [1, 2, 3, 4, 5]

    def test_method(self, value):
    result = value > 0

    Above test should run for each item of list1.

    Please suggest me how to do this.

  8. Pingback: » Python:How to generate dynamic (parametrized) unit tests in python?
  9. Pingback: Connecting an Analog Joystick to the Raspberry Pi (and using it with an RGB LED to simulate a color wheel) – Grant Winney
  10. Thanks for making this library. I had been meaning to make my own, but since one already exists, there’s no reason to. Pretty much the only difference between what you did and what I planned to do was that I intended to have the “unpack” style be the default.
    The other big difference was that I didn’t plan to create new methods for each iteration of the data; it would loop through, run the data, collect any failures, then pump out all the failures as one failure with multiple times it failed. Doing it that way even gets around the subclassing problem you’re having, so maybe you want to look into that for version 2 (then you can completely skip over the idea of customizing the names generated by the function maker)? I won’t mind if you throw out the idea; it doesn’t quite mesh with how yours works currently.
    Anyway, again, I’d like to say thanks, so I don’t have to write it :)

  11. Its possible to apply the data decorator to all the class? I have a class with different method and I want to run all the test for each data input

    list1 = [1, 2, 3, 4, 5]

    import unittest
    from ddt import ddt, data
    from mycode import larger_than_two

    class FooTestCase(unittest.TestCase):
    def test_larger_than_two(self, value):

    def test_not_larger_than_two(self, value):

    • also what about the setup/teardown function???

      how can I write setup and teardown with the @ddt library that use the data list and “initialize” it?

      • This is not possible. setUp and tearDown are instance methods that run at test execution time, whereas the `ddt` decorators run at class load time. Meaning that the data must be available then.

        `ddt` is rather simplistic because it’s not a test runner. If you are looking for more flexible solutions to feed data to your tests, you may want to take a look at py.test and its implementation of *fixtures*. Since py.test is a test runner, it can do much more than `ddt`.

    • This is not possible at the moment, `data` is not implemented as a class decorator.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s