Distinct & Duplicates

Two related questions: what are the unique values, and are there duplicate rows? The first is routine, how many genes, how many conditions. The second is a data-quality check: a file merged twice, a sample listed under two labels. dplyr uses distinct and n_distinct; Polars uses unique and n_unique.

Every code block runs in a container in the companion repository. One function differs between the two, noted below.

Distinct values

The unique values in a column, and a count of them.

R

distinct(expr, gene)
nrow(distinct(expr, condition))   # 2
n_distinct(expr$sample_id)        # 8

   gene
1  TP53
2  EGFR
3   MYC
4 BRCA1
5 GAPDH
6  ACTB

distinct conditions: 2
n_distinct sample_id: 8

Python

expr.select("gene").unique(maintain_order=True)
expr.select("condition").n_unique()   # 2
expr["sample_id"].n_unique()          # 8

shape: (6, 1)
┌───────┐
│ gene  │
│ ---   │
│ str   │
╞═══════╡
│ TP53  │
│ EGFR  │
│ MYC   │
│ BRCA1 │
│ GAPDH │
│ ACTB  │
└───────┘

distinct conditions: 2
n_unique sample_id: 8

Distinct works on combinations too. distinct(expr, gene, condition) gives 12 rows, the six genes times two conditions.

Detect and drop duplicate rows

Here the first annotation row is duplicated, giving eight rows where seven are unique. This is where the two libraries differ in a way worth knowing.

R

with_dup <- bind_rows(annotation, annotation[1, ])
sum(duplicated(with_dup))                    # 1: R flags only the repeat
nrow(with_dup) - nrow(distinct(with_dup))    # 1: rows removed

rows including the duplicate: 8
duplicated() flags: 1
rows removed by distinct(): 1

Python

with_dup = pl.concat([annotation, annotation.head(1)])
with_dup.is_duplicated().sum()               # 2: Polars flags every copy
with_dup.height - with_dup.n_unique()        # 1: rows removed

rows including the duplicate: 8
is_duplicated() flags: 2
rows removed by unique(): 1

The trap: R’s duplicated() marks only the second and later copies of a row, so it reports 1, while Polars is_duplicated() marks every row that has a duplicate, both copies, so it reports 2. They are answering slightly different questions. To count how many rows a dedup removes, the safe formula in both is the number of rows minus the number of unique rows, which is 1 either way. distinct() and unique() themselves agree: both leave seven rows.