NumPy Essentials

Pandas, polars, scikit-learn, scipy and every deep learning framework are built on the NumPy array. You can go a long way without meeting it directly, right up until a number comes out wrong and the explanation is one layer down.

This page is that layer. It is short on API surface and long on the four behaviours that produce silently wrong answers: dtype, axis, broadcasting, and the fact that a slice of an array is not a copy.

Every number below comes from a real run in the python-base container.

Creating arrays

import numpy as np

counts = np.array([120, 455, 89, 1200, 33])

counts        [ 120  455   89 1200   33]
shape         (5,)
dtype         int64
ndim          1

A gene-by-sample matrix, the shape almost every expression analysis uses:

expr = np.array([
    [5.1, 6.3, 2.2, 2.8],
    [0.4, 0.9, 7.7, 8.1],
    [3.3, 3.1, 3.4, 3.2],
])
# expr.shape -> (3, 4), meaning 3 genes x 4 samples

Common constructors: np.zeros(3), np.arange(0, 10, 2), np.linspace(0, 1, 5).

dtype: where the silent bugs live

An array holds one dtype for every element. That is what makes it fast, and it is what makes it lie to you if you are not looking.

np.array([1, 2, 3])        dtype int64
Assigning a float into an int array TRUNCATES, silently:
  after ints[0] = 9.99  ->  [9 2 3]   (not 9.99)

No warning. The value you assigned is simply not the value stored.

Integer overflow does not raise either. int8 tops out at 127:
  int8(120) + int8(10) = [-126]   (wrapped to negative)

And mixing types promotes the entire array:

np.array([1, 2.5, 3])      dtype float64
np.array([1, '2', 3])      dtype <U21  <- everything is a string now

That last one is the CSV case. One stray text cell and the whole column becomes strings, arithmetic starts concatenating, and nothing raises. When a result looks wrong, print the dtype before you print anything else.

Vectorized operations replace the loop

An operation applies to every element with no loop written:

counts          [ 120  455   89 1200   33]
counts * 2      [ 240  910  178 2400   66]
np.log2(counts + 1)  [ 6.919  8.833  6.492 10.23   5.087]

Counts per million in one line:

cpm = counts / counts.sum() * 1e6

[ 63257.8 239852.4  46916.2 632577.8  17395.9]
sums to 1000000.0

This is not only about speed. A loop with an index is where off-by-one errors live, and vectorized code has no index to get wrong.

Axes: the argument people get backwards

axis is the dimension you collapse, not the one you keep.

expr.mean(axis=0)  [2.933 3.433 4.433 4.7  ]   per SAMPLE (4 values)
expr.mean(axis=1)  [4.1   4.275 3.25 ]         per GENE   (3 values)

If the length of the result is not what you expected, the axis is backwards. Checking the length is the fastest possible test, and it catches the transposed-matrix bug that otherwise runs to completion and produces a plausible figure.

Boolean masks

counts             [ 120  455   89 1200   33]
counts > 100       [ True  True False  True False]
counts[mask]       [ 120  455 1200]
mask.sum()         3

mask.sum() is a tally, because True counts as 1. It is the shortest way to answer “how many pass this threshold”.

Combine with & and |, and parenthesise. Python’s and does not work elementwise and raises on an array:

(counts > 100) & (counts < 1000)  # -> [120, 455]
np.where(counts > 100, "high", "low")

Broadcasting

Arrays of different shapes are stretched to match, compared right to left, where a dimension is 1 or missing. Centring each gene on its own mean:

gene_means = expr.mean(axis=1, keepdims=True)  # shape (3, 1)
centred = expr - gene_means

expr shape        (3, 4)
gene_means shape  (3, 1)  <- keepdims=True keeps it 2-D
row means after centring: [ 0. -0.  0.]

Drop keepdims and the mean has shape (3,), which broadcasts along the wrong axis:

ValueError: operands could not be broadcast together with shapes (3,4) (3,)

Here it errored, which is lucky. On a square matrix it would not have. It would have subtracted sample means from genes and returned a full matrix of plausible numbers. This is the strongest argument for keeping non-square test fixtures: a 3-by-4 matrix catches an axis bug that a 4-by-4 matrix hides.

NaN spreads

vals              [10. nan 70.]
vals.sum()        nan     <- one NaN poisons the total
np.nansum(vals)   80.0
np.isnan(vals)    [False  True False]

NaN is never equal to anything, including itself:

np.nan == np.nan   ->  False

So test with np.isnan(), never with ==. This is the bug the Testing page catches with a failing test.

Views versus copies

after modifying a slice, the ORIGINAL changed:  [  1 999   3   4   5]
with .copy(), it did not:                       [1 2 3 4 5]

Slicing a list copies. Slicing an array does not: you get a view onto the same memory. This is the most common surprise for someone arriving from base Python, and it is how a function quietly modifies its caller’s data.

When you mean a copy, say .copy().

Reproducible randomness

rng = np.random.default_rng(42)
rng.poisson(100, 5)  # -> [109 112 108  87  96]

Use np.random.default_rng(seed) and pass the generator around. The legacy np.random.seed() sets a single global that any library can reset underneath you.

Quick reference

Task	Call
Shape, type, dimensions	a.shape, a.dtype, a.ndim
Force a dtype	np.array(x, dtype=float), a.astype(float)
Per-column / per-row stat	a.mean(axis=0) / a.mean(axis=1)
Keep dimensions for broadcasting	a.mean(axis=1, keepdims=True)
Filter	a[a > threshold]
Count matches	(a > threshold).sum()
Elementwise choice	np.where(cond, x, y)
Missing-safe stats	np.nansum, np.nanmean, np.isnan
Real copy	a.copy()
Seeded generator	np.random.default_rng(42)

Next steps

• Pandas Essentials, which puts labels and mixed types on top of these arrays.
• Debugging & Reading Errors, where the dtype trap shows up as a real traceback.
• Testing & Assertions, including the NaN bug above.

The runnable script and the container are in the companion code repo under guides/programming/python/numpy-essentials/.