Debugging & Reading Errors

An error message is not an obstacle between you and the result. It is the most specific information you will ever get about what your code actually did, and most people throw it away by reading only the last line.

Every traceback quoted below is real, produced by a script that raises it on purpose and catches it. Nothing here is written from memory, which matters on this page more than most: an early draft printed a hand-typed TypeError next to a real traceback that said something different.

Read it bottom-up

A count column arrives as text, which is what pandas gives you when a cell says n/a:

import pandas as pd


def normalise(counts, total):
    """Scale counts by a requested total.

    Args:
        counts: Numeric counts to scale.
        total: Target total for scaling.

    Returns:
        Scaled counts.
    """
    return counts / total  # <- this is where it will raise


def cpm(counts):
    """Convert counts to counts per million.

    Args:
        counts: Numeric read counts.

    Returns:
        Counts scaled to one million reads.
    """
    return normalise(counts, counts.size) * 1e6


counts = pd.Series(["120", "455", "89"])  # object dtype, not numeric
cpm(counts)

The traceback ends with the answer:

TypeError: unsupported operand type(s) for /: 'str' and 'int'

The last line is what went wrong. The lines above it are the path that got there, most recent call last. Read the bottom line first, always.

Almost none of the traceback is yours

The full traceback for that failure is 40 lines of pandas internals. The script counts what actually belongs to you:

That traceback has 10 frames. 3 of them are in your file:
    line 36   in <module>     cpm(counts)
    line 27   in cpm          return normalise(counts, counts.size) * 1e6
    line 23   in normalise    return counts / total

Ten frames, three yours. Scan for your own filename, ignore the rest, and start at the last one of yours.

Then work upward. The error is raised in normalise, and nothing in normalise is wrong: it was handed a bad value. cpm passed it along. The bug is on the line that built counts, which appears in no frame at all. The deepest frame is rarely the culprit.

To see the values rather than the lines, walk the frames and print their locals. That is what a debugger shows you, and you can do it without one:

  cpm() line 27
      counts   Series     0    120 1    455 2     89 dtype: object
  normalise() line 23
      counts   Series     0    120 1    455 2     89 dtype: object
      total    int        3

That is an excerpt: the real dump has all ten frames, and the seven pandas ones below these carry the same object array down through arithmetic_op and _masked_arith_op until one of them gives up. Your two frames are the ones that can be fixed.

e does not survive the except block

Python deletes the except ... as e binding when the block exits, so touching e afterwards is a NameError. Capture what you need inside the block: frames = traceback.extract_tb(e.__traceback__).

The dtype that does not raise

The error above was the lucky case. The dangerous version returns a number.

df = pd.DataFrame({"gene": ["A", "B", "C"], "count": ["120", "455", "89"]})

It looks completely normal:

  gene count
0    A   120
1    B   455
2    C    89

Until you ask for the dtypes:

gene     object
count    object
dtype: object

object means str, and adding strings concatenates:

df['count'].sum() = '12045589'

No exception. A result of the right shape and the wrong value, which will travel a long way through a pipeline before anyone notices. When something looks off, print .dtypes before you print anything else. See NumPy Essentials for the same trap one layer down.

A warning is more dangerous than an exception

An exception stops. A warning lets the wrong number reach your figure:

WARNING: RuntimeWarning: divide by zero encountered in divide
result: [10. 10. inf]
mean:   inf

NumPy warns once per location by default. A loop that divides by zero on the thousandth iteration warns there and stays silent for the rest of the run.

While debugging, promote them:

warnings.simplefilter("error")
np.seterr(all="raise")  # numpy specifically

Fail early with a guard

Move the failure to the front door of the function, with a message that names the broken assumption:

def cpm(counts):
    """Convert counts to counts per million.

    Args:
        counts: Numeric read counts.

    Returns:
        Counts scaled to one million reads.
    """
    counts = np.asarray(counts)
    assert np.issubdtype(counts.dtype, np.number), \
        f"counts must be numeric, got {counts.dtype}"
    assert (counts >= 0).all(), "counts must not be negative"
    return counts / counts.sum() * 1e6

AssertionError: counts must be numeric, got <U3

python -O strips assert, so use it for programmer errors and raise ValueError for anything a user can trigger.

except what you expected, not everything

Worth looking for in any code you are reviewing, including your own, because a broad catch is a tempting way to make a script stop complaining:

try:
    result = run_analysis(df)
except Exception:
    result = None

The script demonstrates the cost. A function whose body contains a typo, a NameError, returns None and reports nothing:

broad() returns None, and the NameError is gone forever.
A typo now looks exactly like a legitimately empty result.

Catch the specific failure you know how to recover from and let the rest crash. A crash is information; a None you cannot explain is not.

def narrow(path):
    """Read a CSV file and report a missing path.

    Args:
        path: CSV input path.

    Returns:
        Parsed table, or None when the file is absent.
    """
    try:
        return pd.read_csv(path)
    except FileNotFoundError as e:
        print(f"missing input: {e.filename}")
        return None

The interactive debugger

pdb needs a human at the console, so the script behind this page cannot run it.

Task	How
Stop at a line	breakpoint()
Run the whole script under the debugger	python -m pdb script.py
Inspect the frame after a crash	pdb.post_mortem(), or %debug in IPython

Inside pdb: l lists, n steps over, s steps in, u and d move up and down frames, p prints, c continues. Moving up a frame is the one people forget, and it is usually where the bug is.

Quick reference

Task	Call
What is this really	type(x), x.dtype, df.dtypes
Full traceback as text	traceback.format_exc()
Just your frames	traceback.extract_tb(e.__traceback__)
Stop at a line	breakpoint()
Inspect after a crash	pdb.post_mortem()
Turn warnings into errors	warnings.simplefilter("error")
Turn numpy warnings into errors	np.seterr(all="raise")
Handle one failure	except FileNotFoundError:

Next steps

• Testing & Assertions, the other half of this: a guard catches a bad input, a test catches a bad implementation.
• NumPy Essentials, where dtype, axis and broadcasting bugs come from.
• Fundamentals You Still Need, if you work with a coding agent: what to check in what it produces, and why this page is on that list.

The runnable script and the container are in the companion code repo under guides/programming/python/debugging/.