Reshaping

The same data has two shapes. Long form has one row per measurement, which is tidy and easy to filter and group. Wide form is a matrix, genes down the side and samples across the top, which is what most expression tools and heatmaps expect. Reshaping moves between the two. tidyr calls the moves pivot_wider and pivot_longer; Polars calls them pivot and unpivot.

Every code block runs in a container in the companion repository, on the shared long counts table. The round trip is lossless: the totals match after going wide and back.

Long to wide: build the matrix

Turn the long table into a gene-by-sample matrix. Each gene becomes a row, each sample a column, and the value in each cell is that gene’s count in that sample.

R

library(tidyr)

wide <- expr |>
  select(gene, sample_id, counts) |>
  pivot_wider(names_from = sample_id, values_from = counts)
wide

# A tibble: 6 × 9
  gene    S01   S02   S03   S04   S05   S06   S07   S08
  <chr> <int> <int> <int> <int> <int> <int> <int> <int>
1 TP53    393   416   432   402   619   598   605   568
2 EGFR    294   290   304   296   757   638   666   857
3 MYC     247   243   228   266   810   531   854   834
4 BRCA1   195   183   158   149   204   184   174   206
5 GAPDH  3939  4112  4118  3850  4376  3560  3959  4312
6 ACTB   3117  4331  4398  4024  3292  4199  3666  4018

Python

wide = expr.pivot("sample_id", index="gene", values="counts")
wide

shape: (6, 9)
┌───────┬──────┬──────┬──────┬───┬──────┬──────┬──────┬──────┐
│ gene  ┆ S01  ┆ S02  ┆ S03  ┆ … ┆ S05  ┆ S06  ┆ S07  ┆ S08  │
│ ---   ┆ ---  ┆ ---  ┆ ---  ┆   ┆ ---  ┆ ---  ┆ ---  ┆ ---  │
│ str   ┆ i64  ┆ i64  ┆ i64  ┆   ┆ i64  ┆ i64  ┆ i64  ┆ i64  │
╞═══════╪══════╪══════╪══════╪═══╪══════╪══════╪══════╪══════╡
│ TP53  ┆ 393  ┆ 416  ┆ 432  ┆ … ┆ 619  ┆ 598  ┆ 605  ┆ 568  │
│ EGFR  ┆ 294  ┆ 290  ┆ 304  ┆ … ┆ 757  ┆ 638  ┆ 666  ┆ 857  │
│ MYC   ┆ 247  ┆ 243  ┆ 228  ┆ … ┆ 810  ┆ 531  ┆ 854  ┆ 834  │
│ BRCA1 ┆ 195  ┆ 183  ┆ 158  ┆ … ┆ 204  ┆ 184  ┆ 174  ┆ 206  │
│ GAPDH ┆ 3939 ┆ 4112 ┆ 4118 ┆ … ┆ 4376 ┆ 3560 ┆ 3959 ┆ 4312 │
│ ACTB  ┆ 3117 ┆ 4331 ┆ 4398 ┆ … ┆ 3292 ┆ 4199 ┆ 3666 ┆ 4018 │
└───────┴──────┴──────┴──────┴───┴──────┴──────┴──────┴──────┘

Six genes by eight samples, plus the gene column: a six-row, nine-column matrix. Polars abbreviates the middle columns in its display; all eight samples are present.

Wide to long: back to tidy

Reverse it. Every column except gene collapses back into two columns, one holding the old column name (the sample) and one holding the value (the count).

R

long <- wide |>
  pivot_longer(cols = -gene, names_to = "sample_id", values_to = "counts")
head(long)

# A tibble: 6 × 3
  gene  sample_id counts
  <chr> <chr>      <int>
1 TP53  S01          393
2 TP53  S02          416
3 TP53  S03          432
4 TP53  S04          402
5 TP53  S05          619
6 TP53  S06          598

Python

long = wide.unpivot(index="gene", variable_name="sample_id", value_name="counts")
long.head()

shape: (5, 3)
┌───────┬───────────┬────────┐
│ gene  ┆ sample_id ┆ counts │
│ ---   ┆ ---       ┆ ---    │
│ str   ┆ str       ┆ i64    │
╞═══════╪═══════════╪════════╡
│ TP53  ┆ S01       ┆ 393    │
│ EGFR  ┆ S01       ┆ 294    │
│ MYC   ┆ S01       ┆ 247    │
│ BRCA1 ┆ S01       ┆ 195    │
│ GAPDH ┆ S01       ┆ 3939   │
└───────┴───────────┴────────┘

The two long results order the rows differently, tidyr keeps a gene together while Polars walks sample by sample, but they hold the same 48 measurements. The counts sum to 76872 either way, the same total as the original long table, so the round trip lost nothing.

Transpose: flip the matrix

Transposing is different from pivoting. A pivot reshapes long data using a key column; a transpose swaps the axes of a matrix you already have, turning rows into columns. Here it flips the gene-by-sample matrix into a sample-by-gene one, which is the layout most machine-learning tools want (samples as rows, features as columns). R does it with t() on a matrix; Polars has a transpose method that can name the new columns from an existing one.

R

library(tibble)

gene_mat <- as.matrix(wide[, -1])   # drop the gene column -> numeric matrix
rownames(gene_mat) <- wide$gene
transposed <- rownames_to_column(as.data.frame(t(gene_mat)), "sample_id")
transposed

  sample_id TP53 EGFR MYC BRCA1 GAPDH ACTB
1       S01  393  294 247   195  3939 3117
2       S02  416  290 243   183  4112 4331
3       S03  432  304 228   158  4118 4398
4       S04  402  296 266   149  3850 4024
5       S05  619  757 810   204  4376 3292
6       S06  598  638 531   184  3560 4199
7       S07  605  666 854   174  3959 3666
8       S08  568  857 834   206  4312 4018

Python

transposed = wide.transpose(
    include_header=True, header_name="sample_id", column_names="gene"
)
transposed

shape: (8, 7)
┌───────────┬──────┬──────┬─────┬───────┬───────┬──────┐
│ sample_id ┆ TP53 ┆ EGFR ┆ MYC ┆ BRCA1 ┆ GAPDH ┆ ACTB │
│ ---       ┆ ---  ┆ ---  ┆ --- ┆ ---   ┆ ---   ┆ ---  │
│ str       ┆ i64  ┆ i64  ┆ i64 ┆ i64   ┆ i64   ┆ i64  │
╞═══════════╪══════╪══════╪═════╪═══════╪═══════╪══════╡
│ S01       ┆ 393  ┆ 294  ┆ 247 ┆ 195   ┆ 3939  ┆ 3117 │
│ S02       ┆ 416  ┆ 290  ┆ 243 ┆ 183   ┆ 4112  ┆ 4331 │
│ S03       ┆ 432  ┆ 304  ┆ 228 ┆ 158   ┆ 4118  ┆ 4398 │
│ S04       ┆ 402  ┆ 296  ┆ 266 ┆ 149   ┆ 3850  ┆ 4024 │
│ S05       ┆ 619  ┆ 757  ┆ 810 ┆ 204   ┆ 4376  ┆ 3292 │
│ S06       ┆ 598  ┆ 638  ┆ 531 ┆ 184   ┆ 3560  ┆ 4199 │
│ S07       ┆ 605  ┆ 666  ┆ 854 ┆ 174   ┆ 3959  ┆ 3666 │
│ S08       ┆ 568  ┆ 857  ┆ 834 ┆ 206   ┆ 4312  ┆ 4018 │
└───────────┴──────┴──────┴─────┴───────┴───────┴──────┘

Now samples are the rows and genes the columns: an eight-by-seven table (eight samples plus the sample_id column). MYC in S07 is still 854, just read down a column instead of across a row.