Tidyverse Essentials

What is the tidyverse?

The tidyverse is a collection of R packages that share a consistent grammar for data manipulation. The core packages include dplyr for transforming data, tidyr for reshaping data, and readr for reading files. All tidyverse functions are designed to work together. They accept data frames as input and return data frames as output.

Load the tidyverse with a single command:

library(tidyverse)

The pipe operator

The pipe operator passes the result of one function into the next. This lets you read code from left to right instead of from the inside out.

Without the pipe, nested function calls are hard to read:

# Without pipe
head(sort(degs$pvalue), 3)

[1] 1e-04 1e-03 2e-03

With the native pipe |>, each step is on its own line:

# With native pipe
degs$pvalue |> sort() |> head(3)

[1] 1e-04 1e-03 2e-03

The |> operator takes the output from the left side and feeds it as the first argument to the function on the right side. R introduced the native pipe |> in version 4.1. You will also see %>% from the magrittr package in older code. Both work the same way for most use cases. We recommend the native pipe |> because it requires no extra packages.

Sample data

We will use a data frame of differentially expressed genes throughout this page. Create it now:

degs <- data.frame(
  gene = c("TP53", "BRCA1", "EGFR", "KRAS", "MYC", "PTEN", "RB1", "APC"),
  log2fc = c(2.5, -1.8, 3.2, 0.3, -2.1, 1.5, -0.2, 0.8),
  pvalue = c(0.001, 0.003, 0.0001, 0.45, 0.002, 0.01, 0.67, 0.12),
  biotype = c("protein_coding", "protein_coding", "protein_coding",
              "protein_coding", "protein_coding", "protein_coding",
              "protein_coding", "protein_coding")
)

This represents output from a typical differential expression analysis. Each row is a gene with its fold change, p-value, and biotype annotation.

filter

filter() keeps rows that match a condition. Think of it as subsetting your results to genes of interest.

Keep only genes with a significant p-value:

degs |> filter(pvalue < 0.05)

   gene log2fc pvalue        biotype
1  TP53    2.5  1e-03 protein_coding
2 BRCA1   -1.8  3e-03 protein_coding
3  EGFR    3.2  1e-04 protein_coding
4   MYC   -2.1  2e-03 protein_coding
5  PTEN    1.5  1e-02 protein_coding

You can combine multiple conditions by separating them with commas. Commas work like AND:

degs |> filter(pvalue < 0.05, log2fc > 1)

  gene log2fc pvalue        biotype
1 TP53    2.5  1e-03 protein_coding
2 EGFR    3.2  1e-04 protein_coding
3 PTEN    1.5  1e-02 protein_coding

This returns only genes that are both significant and upregulated.

select

select() picks columns by name. Use it to keep only the columns you need.

degs |> select(gene, log2fc, pvalue)

   gene log2fc pvalue
1  TP53    2.5 0.0010
2 BRCA1   -1.8 0.0030
3  EGFR    3.2 0.0001
4  KRAS    0.3 0.4500
5   MYC   -2.1 0.0020
6  PTEN    1.5 0.0100
7   RB1   -0.2 0.6700
8   APC    0.8 0.1200

You can also drop columns by prefixing the name with a minus sign:

degs |> select(-biotype)

This produces the same output as above. Both approaches are useful. Choose whichever is shorter for your situation.

mutate

mutate() creates new columns or modifies existing ones. The original columns are preserved.

Here we compute the negative log10 p-value and classify each gene as up or down:

degs |> mutate(
  neg_log10p = -log10(pvalue),
  direction = ifelse(log2fc > 0, "up", "down")
)

   gene log2fc pvalue        biotype neg_log10p direction
1  TP53    2.5 0.0010 protein_coding  3.0000000        up
2 BRCA1   -1.8 0.0030 protein_coding  2.5228787      down
3  EGFR    3.2 0.0001 protein_coding  4.0000000        up
4  KRAS    0.3 0.4500 protein_coding  0.3467875        up
5   MYC   -2.1 0.0020 protein_coding  2.6989700      down
6  PTEN    1.5 0.0100 protein_coding  2.0000000        up
7   RB1   -0.2 0.6700 protein_coding  0.1739252      down
8   APC    0.8 0.1200 protein_coding  0.9208188        up

The neg_log10p column is commonly used in volcano plots. The direction column labels genes based on the sign of their fold change.

arrange

arrange() sorts rows. By default it sorts in ascending order.

Sort genes by p-value, most significant first:

degs |> arrange(pvalue)

   gene log2fc pvalue        biotype
1  EGFR    3.2 0.0001 protein_coding
2  TP53    2.5 0.0010 protein_coding
3   MYC   -2.1 0.0020 protein_coding
4 BRCA1   -1.8 0.0030 protein_coding
5  PTEN    1.5 0.0100 protein_coding
6   APC    0.8 0.1200 protein_coding
7  KRAS    0.3 0.4500 protein_coding
8   RB1   -0.2 0.6700 protein_coding

Use desc() for descending order. Wrapping a column in abs() lets you sort by magnitude regardless of sign:

degs |> arrange(desc(abs(log2fc)))

   gene log2fc pvalue        biotype
1  EGFR    3.2 0.0001 protein_coding
2  TP53    2.5 0.0010 protein_coding
3   MYC   -2.1 0.0020 protein_coding
4 BRCA1   -1.8 0.0030 protein_coding
5  PTEN    1.5 0.0100 protein_coding
6   APC    0.8 0.1200 protein_coding
7  KRAS    0.3 0.4500 protein_coding
8   RB1   -0.2 0.6700 protein_coding

This is useful when you want to find the genes with the largest expression changes in either direction.

group_by and summarise

group_by() splits your data into groups. summarise() then computes summary statistics for each group. This combination is essential for comparing conditions in an experiment.

sample_counts <- data.frame(
  sample = c("S1", "S1", "S2", "S2", "S3", "S3"),
  gene = c("TP53", "BRCA1", "TP53", "BRCA1", "TP53", "BRCA1"),
  condition = c("control", "control", "treated", "treated", "treated", "treated"),
  count = c(100, 250, 450, 120, 380, 95)
)

sample_counts |>
  group_by(condition) |>
  summarise(
    mean_count = mean(count),
    n_samples = n()
  )

# A tibble: 2 × 3
  condition mean_count n_samples
  <chr>          <dbl>     <int>
1 control         175          2
2 treated         261.         4

The n() function counts rows in each group. You can use any summary function inside summarise(), such as mean(), median(), sd(), min(), or max().

Chaining operations

The real power of the pipe is chaining multiple operations into a readable pipeline. Each step transforms the data and passes it to the next.

Here we filter for significant genes, add a direction column, sort by p-value, and keep only the columns we need:

degs |>
  filter(pvalue < 0.05) |>
  mutate(direction = ifelse(log2fc > 0, "up", "down")) |>
  arrange(pvalue) |>
  select(gene, log2fc, pvalue, direction)

   gene log2fc pvalue direction
1  EGFR    3.2  1e-04        up
2  TP53    2.5  1e-03        up
3   MYC   -2.1  2e-03      down
4 BRCA1   -1.8  3e-03      down
5  PTEN    1.5  1e-02        up

Read this pipeline top to bottom: start with all genes, keep only significant ones, label their direction, sort by significance, then select the columns to display. This pattern is the standard way to build analysis steps in the tidyverse.

Reshaping data

Bioinformatics data often needs to be reshaped. Gene expression matrices are typically wide, with one column per sample. Many plotting and analysis functions require long format instead.

pivot_longer

pivot_longer() converts wide data to long format. Each sample column becomes a row.

Start with a wide expression matrix:

expression_wide <- data.frame(
  gene = c("TP53", "BRCA1", "EGFR"),
  Sample1 = c(100, 250, 50),
  Sample2 = c(120, 300, 75),
  Sample3 = c(90, 280, 60)
)
expression_wide

   gene Sample1 Sample2 Sample3
1  TP53     100     120      90
2 BRCA1     250     300     280
3  EGFR      50      75      60

Convert it to long format:

expression_long <- expression_wide |>
  pivot_longer(
    cols = starts_with("Sample"),
    names_to = "sample",
    values_to = "count"
  )
expression_long

# A tibble: 9 × 3
  gene  sample  count
  <chr> <chr>   <dbl>
1 TP53  Sample1   100
2 TP53  Sample2   120
3 TP53  Sample3    90
4 BRCA1 Sample1   250
5 BRCA1 Sample2   300
6 BRCA1 Sample3   280
7 EGFR  Sample1    50
8 EGFR  Sample2    75
9 EGFR  Sample3    60

The cols argument selects which columns to pivot. The names_to argument names the new column that will hold the old column names. The values_to argument names the new column that will hold the values.

pivot_wider

pivot_wider() does the reverse. It spreads long data back into wide format.

expression_long |>
  pivot_wider(
    names_from = sample,
    values_from = count
  )

# A tibble: 3 × 4
  gene  Sample1 Sample2 Sample3
  <chr>   <dbl>   <dbl>   <dbl>
1 TP53      100     120      90
2 BRCA1     250     300     280
3 EGFR       50      75      60

You will use pivot_longer() more often than pivot_wider() in practice. Most plotting libraries and statistical functions expect long format data.

Reading files

The readr package provides read_csv() and read_tsv() for reading delimited files. These functions are faster than base R’s read.csv() and read.table(). They also return tibbles instead of plain data frames.

Write and then read back a TSV file:

write_tsv(degs, "/tmp/degs.tsv")
read_degs <- read_tsv("/tmp/degs.tsv", show_col_types = FALSE)
read_degs

# A tibble: 8 × 4
  gene  log2fc pvalue biotype
  <chr>  <dbl>  <dbl> <chr>
1 TP53     2.5 0.001  protein_coding
2 BRCA1   -1.8 0.003  protein_coding
3 EGFR     3.2 0.0001 protein_coding
4 KRAS     0.3 0.45   protein_coding
5 MYC     -2.1 0.002  protein_coding
6 PTEN     1.5 0.01   protein_coding
7 RB1     -0.2 0.67   protein_coding
8 APC      0.8 0.12   protein_coding

Use read_csv() for comma separated files and read_tsv() for tab separated files. The show_col_types = FALSE argument suppresses the column type message. Tibbles print more neatly than data frames and never convert strings to factors automatically.

Quick reference

Function	Purpose	Example
filter()	Keep rows matching a condition	degs |> filter(pvalue < 0.05)
select()	Pick or drop columns	degs |> select(gene, log2fc)
mutate()	Create or modify columns	degs |> mutate(sig = pvalue < 0.05)
arrange()	Sort rows	degs |> arrange(pvalue)
group_by()	Group rows for summarization	df |> group_by(condition)
summarise()	Compute summary statistics per group	df |> summarise(mean = mean(x))
pivot_longer()	Wide to long format	df |> pivot_longer(cols = ...)
pivot_wider()	Long to wide format	df |> pivot_wider(names_from = ...)
read_csv()	Read a CSV file	read_csv("data.csv")
read_tsv()	Read a TSV file	read_tsv("data.tsv")

Next steps

You now have the core tidyverse skills for manipulating gene expression data, filtering results, and reshaping matrices. Next, learn how Bioconductor organizes genomic data using formal class systems in OOP with S4 Classes.