Pandas Essentials

Pandas is the most widely used Python library for tabular data. It provides the DataFrame, a two-dimensional labeled data structure. Think of it as a spreadsheet you can program. Every step in a bioinformatics pipeline involves tables: gene expression matrices, variant calls, sample metadata. Pandas handles all of them.

Creating a DataFrame

You can build a DataFrame from a Python dictionary. Each key becomes a column name. Each value is a list of entries for that column.

Here we create a small table of differentially expressed genes. Each row has a gene name, a log2 fold change, a p-value, and a biotype label.

import pandas as pd

degs = pd.DataFrame({
    "gene": ["TP53", "BRCA1", "EGFR", "KRAS", "MYC", "PTEN", "RB1", "APC"],
    "log2fc": [2.5, -1.8, 3.2, 0.3, -2.1, 1.5, -0.2, 0.8],
    "pvalue": [0.001, 0.003, 0.0001, 0.45, 0.002, 0.01, 0.67, 0.12],
    "biotype": ["protein_coding"] * 8,
})
print(degs)

    gene  log2fc  pvalue         biotype
0   TP53     2.5  0.0010  protein_coding
1  BRCA1    -1.8  0.0030  protein_coding
2   EGFR     3.2  0.0001  protein_coding
3   KRAS     0.3  0.4500  protein_coding
4    MYC    -2.1  0.0020  protein_coding
5   PTEN     1.5  0.0100  protein_coding
6    RB1    -0.2  0.6700  protein_coding
7    APC     0.8  0.1200  protein_coding

The numbers on the left are the index. By default, pandas assigns integer indices starting at 0.

Basic inspection

Before analyzing any dataset, inspect its size and types. The .shape attribute returns rows and columns as a tuple.

print(degs.shape)

(8, 4)

This tells us we have 8 genes and 4 columns.

Use .dtypes to check the data type of each column.

print(degs.dtypes)

gene           str
log2fc     float64
pvalue     float64
biotype        str
dtype: object

Use .head() to preview the first few rows. Pass an integer to control how many rows you see.

print(degs.head(3))

    gene  log2fc  pvalue         biotype
0   TP53     2.5  0.0010  protein_coding
1  BRCA1    -1.8  0.0030  protein_coding
2   EGFR     3.2  0.0001  protein_coding

Use .describe() to get summary statistics for numeric columns.

print(degs.describe())

         log2fc    pvalue
count  8.000000  8.000000
mean   0.525000  0.157012
std    1.888121  0.258758
min   -2.100000  0.000100
25%   -0.600000  0.001750
50%    0.550000  0.006500
75%    1.750000  0.202500
max    3.200000  0.670000

This is useful for spotting outliers or unexpected ranges in expression data.

Selecting columns

Use bracket notation with a column name to select a single column. This returns a Series.

print(degs["gene"])

0     TP53
1    BRCA1
2     EGFR
3     KRAS
4      MYC
5     PTEN
6      RB1
7      APC
Name: gene, dtype: str

To select multiple columns, pass a list of column names inside the brackets.

print(degs[["gene", "log2fc"]])

    gene  log2fc
0   TP53     2.5
1  BRCA1    -1.8
2   EGFR     3.2
3   KRAS     0.3
4    MYC    -2.1
5   PTEN     1.5
6    RB1    -0.2
7    APC     0.8

Note the double brackets. The outer brackets are the selection operator. The inner brackets define the list.

Filtering rows

Filtering is one of the most common operations in bioinformatics. You often need to keep only genes that meet a significance threshold.

To filter, place a boolean condition inside brackets.

sig = degs[degs["pvalue"] < 0.05]
print(sig)

    gene  log2fc  pvalue         biotype
0   TP53     2.5  0.0010  protein_coding
1  BRCA1    -1.8  0.0030  protein_coding
2   EGFR     3.2  0.0001  protein_coding
4    MYC    -2.1  0.0020  protein_coding
5   PTEN     1.5  0.0100  protein_coding

Notice that rows 3, 6, and 7 are gone. Their p-values were above 0.05.

You can combine conditions with & for AND and | for OR. Wrap each condition in parentheses.

sig_up = degs[(degs["pvalue"] < 0.05) & (degs["log2fc"] > 1)]
print(sig_up)

   gene  log2fc  pvalue         biotype
0  TP53     2.5  0.0010  protein_coding
2  EGFR     3.2  0.0001  protein_coding
5  PTEN     1.5  0.0100  protein_coding

This gives us significantly upregulated genes only.

loc and iloc

Pandas provides two indexers for precise row and column selection.

loc selects by label. You provide row labels and column names.
iloc selects by integer position. You provide row and column indices as numbers.

print(degs.loc[0, "gene"])

TP53

print(degs.iloc[0:3, 0:2])

    gene  log2fc
0   TP53     2.5
1  BRCA1    -1.8
2   EGFR     3.2

With iloc, the slice 0:3 means rows 0, 1, and 2. The upper bound is excluded, just like standard Python slicing.

Adding columns

You can create new columns from existing ones. This is useful for derived metrics like negative log10 p-values, which are standard in volcano plots.

import numpy as np
degs["neg_log10p"] = -np.log10(degs["pvalue"])
degs["direction"] = np.where(degs["log2fc"] > 0, "up", "down")
print(degs[["gene", "log2fc", "pvalue", "neg_log10p", "direction"]])

    gene  log2fc  pvalue  neg_log10p direction
0   TP53     2.5  0.0010    3.000000        up
1  BRCA1    -1.8  0.0030    2.522879      down
2   EGFR     3.2  0.0001    4.000000        up
3   KRAS     0.3  0.4500    0.346787        up
4    MYC    -2.1  0.0020    2.698970      down
5   PTEN     1.5  0.0100    2.000000        up
6    RB1    -0.2  0.6700    0.173925      down
7    APC     0.8  0.1200    0.920819        up

np.where() works like an if/else applied to every row. If log2fc > 0, the value is "up". Otherwise, it is "down".

Sorting

Sorting by p-value puts the most significant genes at the top.

print(degs.sort_values("pvalue")[["gene", "pvalue"]])

    gene  pvalue
2   EGFR  0.0001
0   TP53  0.0010
4    MYC  0.0020
1  BRCA1  0.0030
5   PTEN  0.0100
7    APC  0.1200
3   KRAS  0.4500
6    RB1  0.6700

Use ascending=False to sort in descending order. This is handy for finding the most upregulated genes.

print(degs.sort_values("log2fc", ascending=False)[["gene", "log2fc"]].head(3))

   gene  log2fc
2  EGFR     3.2
0  TP53     2.5
5  PTEN     1.5

groupby

Grouping lets you compute summary statistics per category. This is essential when comparing conditions like treated vs. control.

sample_counts = pd.DataFrame({
    "sample": ["S1", "S1", "S2", "S2", "S3", "S3"],
    "gene": ["TP53", "BRCA1", "TP53", "BRCA1", "TP53", "BRCA1"],
    "condition": ["control", "control", "treated", "treated", "treated", "treated"],
    "count": [100, 250, 450, 120, 380, 95],
})

grouped = sample_counts.groupby("condition")["count"].agg(["mean", "count"])
print(grouped)

             mean  count
condition
control    175.00      2
treated    261.25      4

The .agg() method accepts a list of function names. Here we compute the mean and number of observations per condition.

Reshaping data

Bioinformatics tools often expect data in a specific shape. Some need one row per gene per sample. Others need a gene-by-sample matrix. Pandas provides melt and pivot to convert between these formats.

melt

melt converts wide format to long format. Wide format has one column per sample. Long format has one row per observation.

expression_wide = pd.DataFrame({
    "gene": ["TP53", "BRCA1", "EGFR"],
    "Sample1": [100, 250, 50],
    "Sample2": [120, 300, 75],
    "Sample3": [90, 280, 60],
})
print(expression_wide)

    gene  Sample1  Sample2  Sample3
0   TP53      100      120       90
1  BRCA1      250      300      280
2   EGFR       50       75       60

expression_long = expression_wide.melt(
    id_vars="gene",
    var_name="sample",
    value_name="count",
)
print(expression_long)

    gene   sample  count
0   TP53  Sample1    100
1  BRCA1  Sample1    250
2   EGFR  Sample1     50
3   TP53  Sample2    120
4  BRCA1  Sample2    300
5   EGFR  Sample2     75
6   TP53  Sample3     90
7  BRCA1  Sample3    280
8   EGFR  Sample3     60

Long format is required by most plotting libraries and statistical tools.

pivot

pivot does the reverse. It converts long format back to wide format.

back_to_wide = expression_long.pivot(
    index="gene",
    columns="sample",
    values="count",
)
print(back_to_wide)

sample  Sample1  Sample2  Sample3
gene
BRCA1       250      300      280
EGFR         50       75       60
TP53        100      120       90

Note that the rows are now sorted alphabetically by gene name. The gene column has become the index.

Reading and writing files

Most bioinformatics data lives in CSV or TSV files. Pandas can read and write both formats.

Use .to_csv() to write a DataFrame to a file. Set sep="\t" for tab-separated output. Set index=False to avoid writing the row numbers.

degs[["gene", "log2fc", "pvalue"]].to_csv("/tmp/degs.tsv", sep="\t", index=False)
read_back = pd.read_csv("/tmp/degs.tsv", sep="\t")
print(read_back)

    gene  log2fc  pvalue
0   TP53     2.5  0.0010
1  BRCA1    -1.8  0.0030
2   EGFR     3.2  0.0001
3   KRAS     0.3  0.4500
4    MYC    -2.1  0.0020
5   PTEN     1.5  0.0100
6    RB1    -0.2  0.6700
7    APC     0.8  0.1200

Pandas can also read Excel files with pd.read_excel() and compressed files directly. For example, pd.read_csv("data.tsv.gz", sep="\t") will decompress and read in one step.

Quick reference

Operation	Code
Create DataFrame	`pd.DataFrame({"col": [values]})`
Check shape	`df.shape`
Check types	`df.dtypes`
Preview rows	`df.head(n)`
Summary stats	`df.describe()`
Select one column	`df["col"]`
Select multiple columns	`df[["col1", "col2"]]`
Filter rows	`df[df["col"] > value]`
Combine filters	`df[(cond1) & (cond2)]`
Select by label	`df.loc[row, col]`
Select by position	`df.iloc[row, col]`
Add column	`df["new"] = expression`
Sort	`df.sort_values("col")`
Group and aggregate	`df.groupby("col").agg(["mean"])`
Wide to long	`df.melt(id_vars="col")`
Long to wide	`df.pivot(index, columns, values)`
Read TSV	`pd.read_csv("file.tsv", sep="\t")`
Write TSV	`df.to_csv("file.tsv", sep="\t", index=False)`

Next steps

Pandas is the standard for data manipulation in Python. However, it can be slow on large datasets like whole-genome variant files. Polars Essentials introduces a faster alternative that uses the same concepts you learned here.