Missing Values

Real data has holes: a well fails QC, a probe drops out, a join leaves a gap. How you handle them changes the result, so it is worth doing on purpose rather than letting a function decide for you. This page uses expression_qc.csv, the same counts table with four measurements missing. R reads a blank as NA, Polars as null.

Every code block runs in a container in the companion repository, and the two languages agree on the counts and the imputed value.

Detect

Before anything else, see how many values are missing and where.

R

qc <- read.csv("../fixtures/expression_qc.csv")

sum(is.na(qc$counts))
qc |> filter(is.na(counts))

total missing counts: 4

  sample_id condition  gene counts
1       S02   control GAPDH     NA
2       S03   control   MYC     NA
3       S06   treated  EGFR     NA
4       S08   treated BRCA1     NA

Python

qc = pl.read_csv("../fixtures/expression_qc.csv")

print(qc["counts"].null_count())
qc.filter(pl.col("counts").is_null())

total missing counts: 4
shape: (4, 4)
┌───────────┬───────────┬───────┬────────┐
│ sample_id ┆ condition ┆ gene  ┆ counts │
│ ---       ┆ ---       ┆ ---   ┆ ---    │
│ str       ┆ str       ┆ str   ┆ i64    │
╞═══════════╪═══════════╪═══════╪════════╡
│ S02       ┆ control   ┆ GAPDH ┆ null   │
│ S03       ┆ control   ┆ MYC   ┆ null   │
│ S06       ┆ treated   ┆ EGFR  ┆ null   │
│ S08       ┆ treated   ┆ BRCA1 ┆ null   │
└───────────┴───────────┴───────┴────────┘

Drop

The simplest choice: remove rows with a missing count. Safe when the gaps are few and random, wasteful otherwise.

R

dropped <- qc |> drop_na(counts)
nrow(dropped)   # 48 - 4 = 44

rows after drop: 44

Python

dropped = qc.drop_nulls("counts")
dropped.height   # 48 - 4 = 44

rows after drop: 44

Fill with a constant

Replace missing with a fixed value, for example 0. Convenient, but be careful: a zero count is a real claim about the biology, so filling counts with 0 is rarely correct.

R

filled <- qc |> mutate(counts = replace_na(counts, 0))
filled |> filter(sample_id == "S03", gene == "MYC")

  sample_id condition gene counts
1       S03   control  MYC      0

Python

filled = qc.with_columns(pl.col("counts").fill_null(0))
filled.filter((pl.col("sample_id") == "S03") & (pl.col("gene") == "MYC"))

shape: (1, 4)
┌───────────┬───────────┬──────┬────────┐
│ sample_id ┆ condition ┆ gene ┆ counts │
│ str       ┆ str       ┆ str  ┆ i64    │
╞═══════════╪═══════════╪══════╪════════╡
│ S03       ┆ control   ┆ MYC  ┆ 0      │
└───────────┴───────────┴──────┴────────┘

Impute with a group mean

A better default for expression data: fill each gap with that gene’s mean across the samples that do have data. Same grouped pattern as a computed column. MYC’s missing value in S03 becomes the mean of its seven observed counts, about 540.7.

R

imputed <- qc |>
  group_by(gene) |>
  mutate(counts = ifelse(is.na(counts), mean(counts, na.rm = TRUE), counts)) |>
  ungroup()
imputed |> filter(sample_id == "S03", gene == "MYC")

# A tibble: 1 × 4
  sample_id condition gene  counts
  <chr>     <chr>     <chr>  <dbl>
1 S03       control   MYC     541.

Python

imputed = qc.with_columns(
    pl.col("counts").fill_null(pl.col("counts").mean().over("gene"))
)
imputed.filter((pl.col("sample_id") == "S03") & (pl.col("gene") == "MYC"))

shape: (1, 4)
┌───────────┬───────────┬──────┬────────────┐
│ sample_id ┆ condition ┆ gene ┆ counts     │
│ str       ┆ str       ┆ str  ┆ f64        │
╞═══════════╪═══════════╪══════╪════════════╡
│ S03       ┆ control   ┆ MYC  ┆ 540.714286 │
└───────────┴───────────┴──────┴────────────┘

The tibble rounds to 541 for display; both compute 540.714. Imputation is a modelling choice, not a cleanup step: the gene mean is a reasonable default here, but on real data you would justify it against the alternatives.