How to use subtests¶

Added in version 9.0.

Note

This feature is experimental. Its behavior, particularly how failures are reported, may evolve in future releases. However, the core functionality and usage are considered stable.

pytest allows for grouping assertions within a normal test, known as subtests.

Subtests are an alternative to parametrization, particularly useful when the exact parametrization values are not known at collection time.

# content of test_subtest.py


def test(subtests):
    for i in range(5):
        with subtests.test(msg="custom message", i=i):
            assert i % 2 == 0

Each assertion failure or error is caught by the context manager and reported individually:

$ pytest -q test_subtest.py
uuuuuF                                                               [100%]
================================= FAILURES =================================
_______________________ test [custom message] (i=1) ________________________

subtests = <_pytest.subtests.Subtests object at 0xdeadbeef0001>

    def test(subtests):
        for i in range(5):
            with subtests.test(msg="custom message", i=i):
>               assert i % 2 == 0
E               assert (1 % 2) == 0

test_subtest.py:6: AssertionError
_______________________ test [custom message] (i=3) ________________________

subtests = <_pytest.subtests.Subtests object at 0xdeadbeef0001>

    def test(subtests):
        for i in range(5):
            with subtests.test(msg="custom message", i=i):
>               assert i % 2 == 0
E               assert (3 % 2) == 0

test_subtest.py:6: AssertionError
___________________________________ test ___________________________________
contains 2 failed subtests
========================= short test summary info ==========================
SUBFAILED[custom message] (i=1) test_subtest.py::test - assert (1 % 2) == 0
SUBFAILED[custom message] (i=3) test_subtest.py::test - assert (3 % 2) == 0
FAILED test_subtest.py::test - contains 2 failed subtests
3 failed, 3 subtests passed in 0.12s

In the output above:

Subtest failures are reported as SUBFAILED.
Subtests are reported first and the “top-level” test is reported at the end on its own.

Note that it is possible to use subtests multiple times in the same test, or even mix and match with normal assertions outside the subtests.test block:

def test(subtests):
    for i in range(5):
        with subtests.test("stage 1", i=i):
            assert i % 2 == 0

    assert func() == 10

    for i in range(10, 20):
        with subtests.test("stage 2", i=i):
            assert i % 2 == 0

Note

See How to parametrize fixtures and test functions for an alternative to subtests.

Verbosity¶

By default, only subtest failures are shown. Higher verbosity levels (-v) will also show progress output for passed subtests.

It is possible to control the verbosity of subtests by setting verbosity_subtests.

Typing¶

pytest.Subtests is exported so it can be used in type annotations:

def test(subtests: pytest.Subtests) -> None: ...

Parametrization vs Subtests¶

While traditional pytest parametrization and subtests are similar, they have important differences and use cases.

Parametrization¶

Happens at collection time.
Generates individual tests.
Parametrized tests can be referenced from the command line.
Plays well with plugins that handle test execution, such as --last-failed.
Ideal for decision table testing.

Subtests¶

Happen during test execution.
Are not known at collection time.
Can be generated dynamically.
Cannot be referenced individually from the command line.
Plugins that handle test execution cannot target individual subtests.
An assertion failure inside a subtest does not interrupt the test, letting users see all failures in the same report.

Note

This feature was originally implemented as a separate plugin in pytest-subtests, but since 9.0 has been merged into the core.

The core implementation should be compatible to the plugin implementation, except it does not contain custom command-line options to control subtest output.