Testing & Assertions

Running a function once and eyeballing the answer checks it on that input, on that day. A test checks it on every run, forever, and says out loud what “correct” meant. Those are very different guarantees, and only one of them survives you changing the code.

The suite on this page is real, and it contains one test that fails on purpose, because a passing suite proves nothing about the suite. That failure documents an actual bug in the function under test, and it is a bug worth meeting.

Assertions come first

A guard lives inside the function and runs on every call. No framework, no second file, no test runner.

def cpm(counts):
    """Convert counts to counts per million.

    Args:
        counts: Numeric read counts.

    Returns:
        Counts scaled to one million reads.
    """
    counts = np.asarray(counts, dtype=float)
    if np.any(counts < 0):
        raise ValueError("counts must not be negative")
    return counts / counts.sum() * 1e6


cpm([-5, 10])

cpm([-5, 10]) ->  ValueError: counts must not be negative

Raise ValueError or TypeError for input a caller can get wrong. Keep bare assert for internal invariants, because python -O strips every assert in the file and a guard that vanishes under optimisation is not a guard.

The difference from a test: a guard checks every future call, including the ones you never thought to try. A test checks one case you did think of. If you write one thing after finishing a function, write the guard.

What is worth testing in analysis code

Not every line. Four things pay for themselves:

	what it catches
Shape	returns the wrong length, the wrong dtype, a matrix transposed
Known answer	a tiny input you can work out by hand, so the maths is checked once
Invariant	something true of any output, whatever the input
Edge case	empty, one element, NaN, zero, negative, all-identical

The invariant is the one people miss and the one that keeps working as the code changes. For counts per million it writes itself: whatever goes in, the output sums to a million.

The edge case is where bugs concentrate, because the happy path is the part you already checked by running it once.

A real suite, with a real failure

test_cpm.py:

def test_returns_one_value_per_input_count():
    out = cpm([10, 20, 70])
    assert out.shape == (3,)
    assert np.issubdtype(out.dtype, np.floating)


def test_scales_to_a_million():
    # The invariant: whatever the input, the output sums to 1e6.
    assert cpm([10, 20, 70]).sum() == pytest.approx(1e6)


def test_answer_you_can_work_out_by_hand():
    # 25 out of a library of 100 is a quarter, so 250,000 per million.
    np.testing.assert_allclose(cpm([25, 75]), [250_000, 750_000])


def test_rejects_input_it_cannot_handle():
    with pytest.raises(ValueError, match="must not be negative"):
        cpm([-1, 10])


def test_survives_an_nan_count():
    """FAILS: documents a real bug."""
    out = cpm([10, np.nan, 70])
    assert not np.all(np.isnan(out))

Run it with pytest -v:

test_cpm.py::test_returns_one_value_per_input_count PASSED               [ 16%]
test_cpm.py::test_scales_to_a_million PASSED                             [ 33%]
test_cpm.py::test_answer_you_can_work_out_by_hand PASSED                 [ 50%]
test_cpm.py::test_rejects_input_it_cannot_handle PASSED                  [ 66%]
test_cpm.py::test_handles_a_single_count PASSED                          [ 83%]
test_cpm.py::test_survives_an_nan_count FAILED                           [100%]
...
========================= 1 failed, 5 passed in 0.03s ==========================

Note pytest.approx and np.testing.assert_allclose. Comparing floats with == is the mistake variables and data types covers, and it is why every test framework ships a tolerance helper.

pytest reports the failure by re-running the assertion and showing every intermediate value, which is why a plain assert is enough and you rarely need a special matcher.

The bug the failing test found

np.sum propagates NaN. So one missing count turns the entire output array into NaN, not just its own entry:

cpm([10, nan, 70])       -> [nan nan nan]
cpm_fixed([10, nan, 70]) -> [125000.     nan 875000.]

The function does not raise. It does not warn. It returns the right shape and the right dtype with every value wrong. That is the shape of bug that survives review, gets into a figure, and is found six months later by someone else.

The repair is np.nansum, and then the missing value stays in its own slot where it belongs. The point is that nothing except a test was ever going to tell you the difference.

An unseeded test is not a test

rng = np.random.default_rng(42)
counts = rng.poisson(100, size=5)  # -> [109 112 108  87  96], every time

Prefer a Generator you create and pass around to the legacy np.random.seed(), which sets a single global that any library can reset underneath you.

Seed inside each test, not once at module import. A single seed at the top makes every test depend on the order the others ran in, so adding a test at line 10 breaks a test at line 90 and the failure looks like it came from nowhere.

Seeding is not only for tests. Any analysis with a random component needs one to be reproducible: bootstraps, permutations, k-means starts, train and test splits, UMAP layouts.

Testing figures and models

You cannot assert that a plot looks right, and you should not try. Test the numbers that go into the figure, which is where the bugs are, and look at the picture yourself.

Same for a model. Assert the shapes, assert the ranges, assert that a known-signal input is recovered. Do not assert an exact accuracy: that pins your suite to a library version and it will fail on an upgrade for reasons that have nothing to do with your code.

Quick reference

Task	Call
Guard an input	raise ValueError("message")
A test	any def test_*() with a plain assert
Compare floats	pytest.approx(x), np.testing.assert_allclose(a, b)
Expect a failure	with pytest.raises(ValueError, match="..."):
Run the suite	pytest -v
Run one test	pytest test_cpm.py::test_name
Stop at first failure	pytest -x
One test, many inputs	@pytest.mark.parametrize
Deterministic randomness	np.random.default_rng(42)

Next steps

• Debugging & Reading Errors, the other half: a guard catches a bad input, a test catches a bad implementation, and debugging is what you do when neither fired.
• Documentation and Style, for the Google docstrings, Sphinx configuration, and package layout that turn a tested module into a documented API.
• The pytest documentation, especially fixtures and parametrize, which is how one test becomes twenty.

The runnable scripts, the suite, and the container are in the companion code repo under guides/programming/python/testing/.