Polars Essentials

What is Polars

Polars is a DataFrame library written in Rust. It is designed for speed and can handle large datasets much faster than pandas. Polars uses an expression system instead of method chaining. It also supports lazy evaluation, which lets Polars optimize your query before running it. Unlike pandas, Polars has no index column.

If you work with gene expression matrices or variant call tables with millions of rows, Polars will feel noticeably faster.

Creating a DataFrame

Start by importing Polars and creating a DataFrame of differentially expressed genes.

import polars as pl

degs = pl.DataFrame({
    "gene": ["TP53", "BRCA1", "EGFR", "KRAS", "MYC", "PTEN", "RB1", "APC"],
    "log2fc": [2.5, -1.8, 3.2, 0.3, -2.1, 1.5, -0.2, 0.8],
    "pvalue": [0.001, 0.003, 0.0001, 0.45, 0.002, 0.01, 0.67, 0.12],
    "biotype": ["protein_coding"] * 8,
})
print(degs)

shape: (8, 4)
┌───────┬────────┬────────┬────────────────┐
│ gene  ┆ log2fc ┆ pvalue ┆ biotype        │
│ ---   ┆ ---    ┆ ---    ┆ ---            │
│ str   ┆ f64    ┆ f64    ┆ str            │
╞═══════╪════════╪════════╪════════════════╡
│ TP53  ┆ 2.5    ┆ 0.001  ┆ protein_coding │
│ BRCA1 ┆ -1.8   ┆ 0.003  ┆ protein_coding │
│ EGFR  ┆ 3.2    ┆ 0.0001 ┆ protein_coding │
│ KRAS  ┆ 0.3    ┆ 0.45   ┆ protein_coding │
│ MYC   ┆ -2.1   ┆ 0.002  ┆ protein_coding │
│ PTEN  ┆ 1.5    ┆ 0.01   ┆ protein_coding │
│ RB1   ┆ -0.2   ┆ 0.67   ┆ protein_coding │
│ APC   ┆ 0.8    ┆ 0.12   ┆ protein_coding │
└───────┴────────┴────────┴────────────────┘

Each column has a strict type. Polars infers these types automatically from the data you provide.

Basic inspection

Check the shape of the DataFrame. This returns a tuple of rows and columns.

print(degs.shape)

(8, 4)

View the data types for each column.

print(degs.dtypes)

[String, Float64, Float64, String]

Preview the first few rows with head().

print(degs.head(3))

shape: (3, 4)
┌───────┬────────┬────────┬────────────────┐
│ gene  ┆ log2fc ┆ pvalue ┆ biotype        │
│ ---   ┆ ---    ┆ ---    ┆ ---            │
│ str   ┆ f64    ┆ f64    ┆ str            │
╞═══════╪════════╪════════╪════════════════╡
│ TP53  ┆ 2.5    ┆ 0.001  ┆ protein_coding │
│ BRCA1 ┆ -1.8   ┆ 0.003  ┆ protein_coding │
│ EGFR  ┆ 3.2    ┆ 0.0001 ┆ protein_coding │
└───────┴────────┴────────┴────────────────┘

Selecting columns

Use select() to pick specific columns. Pass column names as strings.

print(degs.select("gene", "log2fc"))

shape: (8, 2)
┌───────┬────────┐
│ gene  ┆ log2fc │
│ ---   ┆ ---    │
│ str   ┆ f64    │
╞═══════╪════════╡
│ TP53  ┆ 2.5    │
│ BRCA1 ┆ -1.8   │
│ EGFR  ┆ 3.2    │
│ KRAS  ┆ 0.3    │
│ MYC   ┆ -2.1   │
│ PTEN  ┆ 1.5    │
│ RB1   ┆ -0.2   │
│ APC   ┆ 0.8    │
└───────┴────────┘

Filtering rows

Use filter() with pl.col() expressions to select rows that match a condition. Here we keep only genes with a significant p-value.

sig = degs.filter(pl.col("pvalue") < 0.05)
print(sig)

shape: (5, 4)
┌───────┬────────┬────────┬────────────────┐
│ gene  ┆ log2fc ┆ pvalue ┆ biotype        │
│ ---   ┆ ---    ┆ ---    ┆ ---            │
│ str   ┆ f64    ┆ f64    ┆ str            │
╞═══════╪════════╪════════╪════════════════╡
│ TP53  ┆ 2.5    ┆ 0.001  ┆ protein_coding │
│ BRCA1 ┆ -1.8   ┆ 0.003  ┆ protein_coding │
│ EGFR  ┆ 3.2    ┆ 0.0001 ┆ protein_coding │
│ MYC   ┆ -2.1   ┆ 0.002  ┆ protein_coding │
│ PTEN  ┆ 1.5    ┆ 0.01   ┆ protein_coding │
└───────┴────────┴────────┴────────────────┘

Combine multiple conditions with & for AND. Wrap each condition in parentheses. This filters for significantly upregulated genes.

sig_up = degs.filter(
    (pl.col("pvalue") < 0.05) & (pl.col("log2fc") > 1)
)
print(sig_up)

shape: (3, 4)
┌──────┬────────┬────────┬────────────────┐
│ gene ┆ log2fc ┆ pvalue ┆ biotype        │
│ ---  ┆ ---    ┆ ---    ┆ ---            │
│ str  ┆ f64    ┆ f64    ┆ str            │
╞══════╪════════╪════════╪════════════════╡
│ TP53 ┆ 2.5    ┆ 0.001  ┆ protein_coding │
│ EGFR ┆ 3.2    ┆ 0.0001 ┆ protein_coding │
│ PTEN ┆ 1.5    ┆ 0.01   ┆ protein_coding │
└──────┴────────┴────────┴────────────────┘

Adding columns with with_columns

Use with_columns() to create new columns from existing ones. The original DataFrame is not modified. Polars returns a new DataFrame with the added columns.

Here we compute the negative log10 p-value and label each gene as “up” or “down”.

result = degs.with_columns(
    (-pl.col("pvalue").log(base=10)).alias("neg_log10p"),
    pl.when(pl.col("log2fc") > 0)
      .then(pl.lit("up"))
      .otherwise(pl.lit("down"))
      .alias("direction"),
)
print(result.select("gene", "log2fc", "pvalue", "neg_log10p", "direction"))

shape: (8, 5)
┌───────┬────────┬────────┬────────────┬───────────┐
│ gene  ┆ log2fc ┆ pvalue ┆ neg_log10p ┆ direction │
│ ---   ┆ ---    ┆ ---    ┆ ---        ┆ ---       │
│ str   ┆ f64    ┆ f64    ┆ f64        ┆ str       │
╞═══════╪════════╪════════╪════════════╪═══════════╡
│ TP53  ┆ 2.5    ┆ 0.001  ┆ 3.0        ┆ up        │
│ BRCA1 ┆ -1.8   ┆ 0.003  ┆ 2.522879   ┆ down      │
│ EGFR  ┆ 3.2    ┆ 0.0001 ┆ 4.0        ┆ up        │
│ KRAS  ┆ 0.3    ┆ 0.45   ┆ 0.346787   ┆ up        │
│ MYC   ┆ -2.1   ┆ 0.002  ┆ 2.69897    ┆ down      │
│ PTEN  ┆ 1.5    ┆ 0.01   ┆ 2.0        ┆ up        │
│ RB1   ┆ -0.2   ┆ 0.67   ┆ 0.173925   ┆ down      │
│ APC   ┆ 0.8    ┆ 0.12   ┆ 0.920819   ┆ up        │
└───────┴────────┴────────┴────────────┴───────────┘

The pl.when().then().otherwise() pattern works like an if/else statement. You can chain multiple .when() calls to handle more conditions.

Sorting

Sort by p-value to see the most significant genes first.

print(degs.sort("pvalue").select("gene", "pvalue"))

shape: (8, 2)
┌───────┬────────┐
│ gene  ┆ pvalue │
│ ---   ┆ ---    │
│ str   ┆ f64    │
╞═══════╪════════╡
│ EGFR  ┆ 0.0001 │
│ TP53  ┆ 0.001  │
│ MYC   ┆ 0.002  │
│ BRCA1 ┆ 0.003  │
│ PTEN  ┆ 0.01   │
│ APC   ┆ 0.12   │
│ KRAS  ┆ 0.45   │
│ RB1   ┆ 0.67   │
└───────┴────────┘

Use descending=True to sort from highest to lowest. Here we find the top 3 upregulated genes.

print(degs.sort("log2fc", descending=True).select("gene", "log2fc").head(3))

shape: (3, 2)
┌──────┬────────┐
│ gene ┆ log2fc │
│ ---  ┆ ---    │
│ str  ┆ f64    │
╞══════╪════════╡
│ EGFR ┆ 3.2    │
│ TP53 ┆ 2.5    │
│ PTEN ┆ 1.5    │
└──────┴────────┘

group_by

Use group_by() to split data into groups and compute summary statistics with agg(). This is useful for comparing conditions in an experiment.

sample_counts = pl.DataFrame({
    "sample": ["S1", "S1", "S2", "S2", "S3", "S3"],
    "gene": ["TP53", "BRCA1", "TP53", "BRCA1", "TP53", "BRCA1"],
    "condition": ["control", "control", "treated", "treated", "treated", "treated"],
    "count": [100, 250, 450, 120, 380, 95],
})

grouped = sample_counts.group_by("condition").agg(
    pl.col("count").mean().alias("mean_count"),
    pl.col("count").count().alias("n_samples"),
)
print(grouped.sort("condition"))

shape: (2, 3)
┌───────────┬────────────┬───────────┐
│ condition ┆ mean_count ┆ n_samples │
│ ---       ┆ ---        ┆ ---       │
│ str       ┆ f64        ┆ u32       │
╞═══════════╪════════════╪═══════════╡
│ control   ┆ 175.0      ┆ 2         │
│ treated   ┆ 261.25     ┆ 4         │
└───────────┴────────────┴───────────┘

Note that group_by() does not guarantee row order. Always chain .sort() if you need consistent output.

Lazy evaluation

Lazy mode is one of the biggest advantages of Polars. When you call .lazy(), Polars does not execute anything right away. Instead it builds a query plan. Polars then optimizes this plan before running it. For example, it may push filters before sorts to reduce the number of rows processed.

Call .collect() at the end to execute the plan and get a DataFrame back.

result = (
    degs.lazy()
    .filter(pl.col("pvalue") < 0.05)
    .with_columns(
        pl.when(pl.col("log2fc") > 1).then(pl.lit("up"))
        .when(pl.col("log2fc") < -1).then(pl.lit("down"))
        .otherwise(pl.lit("ns"))
        .alias("direction")
    )
    .sort("pvalue")
    .select("gene", "log2fc", "pvalue", "direction")
    .collect()
)
print(result)

shape: (5, 4)
┌───────┬────────┬────────┬───────────┐
│ gene  ┆ log2fc ┆ pvalue ┆ direction │
│ ---   ┆ ---    ┆ ---    ┆ ---       │
│ str   ┆ f64    ┆ f64    ┆ str       │
╞═══════╪════════╪════════╪═══════════╡
│ EGFR  ┆ 3.2    ┆ 0.0001 ┆ up        │
│ TP53  ┆ 2.5    ┆ 0.001  ┆ up        │
│ MYC   ┆ -2.1   ┆ 0.002  ┆ down      │
│ BRCA1 ┆ -1.8   ┆ 0.003  ┆ down      │
│ PTEN  ┆ 1.5    ┆ 0.01   ┆ up        │
└───────┴────────┴────────┴───────────┘

For small datasets the speed difference is negligible. For millions of rows, lazy evaluation can be significantly faster.

Reshaping data

Gene expression data often needs to be converted between wide and long formats. Wide format has one column per sample. Long format has one row per measurement.

unpivot

The unpivot() method converts wide data to long format. This is equivalent to melt() in pandas.

expression_wide = pl.DataFrame({
    "gene": ["TP53", "BRCA1", "EGFR"],
    "Sample1": [100, 250, 50],
    "Sample2": [120, 300, 75],
    "Sample3": [90, 280, 60],
})
print(expression_wide)

shape: (3, 4)
┌───────┬─────────┬─────────┬─────────┐
│ gene  ┆ Sample1 ┆ Sample2 ┆ Sample3 │
│ ---   ┆ ---     ┆ ---     ┆ ---     │
│ str   ┆ i64     ┆ i64     ┆ i64     │
╞═══════╪═════════╪═════════╪═════════╡
│ TP53  ┆ 100     ┆ 120     ┆ 90      │
│ BRCA1 ┆ 250     ┆ 300     ┆ 280     │
│ EGFR  ┆ 50      ┆ 75      ┆ 60      │
└───────┴─────────┴─────────┴─────────┘

expression_long = expression_wide.unpivot(
    on=["Sample1", "Sample2", "Sample3"],
    index="gene",
    variable_name="sample",
    value_name="count",
)
print(expression_long)

shape: (9, 3)
┌───────┬─────────┬───────┐
│ gene  ┆ sample  ┆ count │
│ ---   ┆ ---     ┆ ---   │
│ str   ┆ str     ┆ i64   │
╞═══════╪═════════╪═══════╡
│ TP53  ┆ Sample1 ┆ 100   │
│ BRCA1 ┆ Sample1 ┆ 250   │
│ EGFR  ┆ Sample1 ┆ 50    │
│ TP53  ┆ Sample2 ┆ 120   │
│ BRCA1 ┆ Sample2 ┆ 300   │
│ EGFR  ┆ Sample2 ┆ 75    │
│ TP53  ┆ Sample3 ┆ 90    │
│ BRCA1 ┆ Sample3 ┆ 280   │
│ EGFR  ┆ Sample3 ┆ 60    │
└───────┴─────────┴───────┘

Long format is useful for plotting and for group_by operations across samples.

pivot

The pivot() method converts long data back to wide format.

back_to_wide = expression_long.pivot(
    on="sample",
    index="gene",
    values="count",
)
print(back_to_wide)

shape: (3, 4)
┌───────┬─────────┬─────────┬─────────┐
│ gene  ┆ Sample1 ┆ Sample2 ┆ Sample3 │
│ ---   ┆ ---     ┆ ---     ┆ ---     │
│ str   ┆ i64     ┆ i64     ┆ i64     │
╞═══════╪═════════╪═════════╪═════════╡
│ TP53  ┆ 100     ┆ 120     ┆ 90      │
│ BRCA1 ┆ 250     ┆ 300     ┆ 280     │
│ EGFR  ┆ 50      ┆ 75      ┆ 60      │
└───────┴─────────┴─────────┴─────────┘

Reading and writing files

Polars can read and write CSV, TSV, Parquet, and other formats. Here we write a TSV and read it back.

degs.select("gene", "log2fc", "pvalue").write_csv("/tmp/degs_polars.tsv", separator="\t")
read_back = pl.read_csv("/tmp/degs_polars.tsv", separator="\t")
print(read_back)

shape: (8, 3)
┌───────┬────────┬────────┐
│ gene  ┆ log2fc ┆ pvalue │
│ ---   ┆ ---    ┆ ---    │
│ str   ┆ f64    ┆ f64    │
╞═══════╪════════╪════════╡
│ TP53  ┆ 2.5    ┆ 0.001  │
│ BRCA1 ┆ -1.8   ┆ 0.003  │
│ EGFR  ┆ 3.2    ┆ 0.0001 │
│ KRAS  ┆ 0.3    ┆ 0.45   │
│ MYC   ┆ -2.1   ┆ 0.002  │
│ PTEN  ┆ 1.5    ┆ 0.01   │
│ RB1   ┆ -0.2   ┆ 0.67   │
│ APC   ┆ 0.8    ┆ 0.12   │
└───────┴────────┴────────┘

For large files, consider using Parquet format with write_parquet() and read_parquet(). Parquet files are compressed and much faster to read than CSV.

Pandas vs Polars quick comparison

If you already know pandas, this table maps common operations to their Polars equivalents.

Operation	pandas	Polars
Select columns	df[["gene", "log2fc"]]	df.select("gene", "log2fc")
Filter rows	df[df["pvalue"] < 0.05]	df.filter(pl.col("pvalue") < 0.05)
Add column	df["new"] = values	df.with_columns(expr.alias("new"))
Sort	df.sort_values("col")	df.sort("col")
Group + aggregate	df.groupby("col").agg(...)	df.group_by("col").agg(...)
Wide to long	df.melt(...)	df.unpivot(...)
Long to wide	df.pivot(...)	df.pivot(...)
Read CSV	pd.read_csv(...)	pl.read_csv(...)

The biggest difference is that Polars uses expressions like pl.col("gene") instead of bracket indexing. This takes some getting used to, but it makes complex queries more readable.

Next steps

Continue to OOP with Classes to learn how to organize your bioinformatics code into reusable classes.