Why Reproducibility Matters

The problem

A colleague asks you to rerun an analysis from six months ago. You open your project folder. The scripts are there. The data is there. You run the pipeline and get different results.

This is not a rare scenario. A 2016 survey in Nature found that over 70% of researchers had failed to reproduce another scientist’s experiments. Computational work should be easier to reproduce than wet lab experiments. There are no pipetting errors or batch effects from reagent lots. Yet computational results are just as fragile.

Why bioinformatics analyses break

Several factors make bioinformatics results hard to reproduce.

Software versions change behavior

STAR 2.7.9a and STAR 2.7.11b can produce different alignment counts for the same FASTQ file. DESeq2 1.38 and DESeq2 1.42 may report different p-values for the same count matrix. These are not bugs. Algorithms improve, defaults change, and edge cases get handled differently between versions.

If you do not record which version you used, you cannot reproduce your results. If a collaborator installs a newer version, they will get different numbers.

Reference genomes and annotations evolve

GENCODE releases a new human gene annotation roughly every six months. Each release adds genes, changes transcript boundaries, and reclassifies biotypes. An RNA-seq analysis run against GENCODE v38 and GENCODE v44 will produce different count matrices, even with identical software and FASTQ files.

The same applies to genome assemblies. GRCh38 patch releases add alternate loci and fix sequence errors. Variant calling results depend on which patch you used.

Undocumented manual steps

Many analyses include steps that never make it into a script. Filtering a sample list in Excel. Renaming files by hand. Adjusting a threshold after looking at a plot. These decisions are invisible to anyone trying to reproduce the work, including your future self.

Environment drift

Your laptop today is not your laptop six months from now. Operating system updates, library upgrades, and Python version changes can all alter results. A script that worked in January may fail in July because a dependency updated its API.

The three pillars of reproducibility

Three tools address these failure modes. Each solves a different part of the problem.

1. Version control tracks your decisions

Git records every change you make to your code. It captures what changed, when it changed, and why. If you modified a filtering threshold, the commit history shows the old value and the new value. If a collaborator asks why you removed a sample, the answer is in the log.

Version control does not just protect against mistakes. It documents the reasoning behind your analysis. This site has a full section on Git if you are new to it.

2. Containers freeze your environment

A container packages your software, libraries, and dependencies into a single portable image. When you run an analysis inside a container, you get the same versions of every tool regardless of the host machine.

If your container has STAR 2.7.11b and DESeq2 1.42, every person who runs that container gets STAR 2.7.11b and DESeq2 1.42. On any machine. On any operating system. A year from now or five years from now.

This site covers containers in the Containers with Podman section. The key idea is that containers eliminate “works on my machine” as a category of failure.

3. Workflow managers automate your pipeline

A workflow manager connects your analysis steps into a directed graph. Each step declares its inputs, outputs, and the command to run. The workflow manager handles execution order, parallelism, error recovery, and logging.

Without a workflow manager, your pipeline is a collection of scripts that you run in a specific order. That order lives in your head or in a README that may be out of date. With a workflow manager, the pipeline is a single executable definition. Anyone can run it with one command and get the same results.

Workflow managers for bioinformatics

Several workflow managers are used in bioinformatics. The two most popular are Nextflow and Snakemake.

Nextflow

Nextflow uses a Groovy-based DSL to define workflows. It has native support for Docker, Singularity, and cloud platforms like AWS Batch. The nf-core community maintains over 100 production-ready pipelines for common bioinformatics tasks.

Nextflow is the dominant choice in genomics core facilities and large sequencing centers. If you work with nf-core pipelines, you are using Nextflow.

This site covers Nextflow in detail in the Nextflow & nf-core subsection.

Snakemake

Snakemake uses Python syntax to define workflows. It determines what to run based on file targets, similar to GNU Make. It integrates tightly with Conda for environment management and supports Singularity containers.

Snakemake is popular in academic research labs, especially those with existing Python expertise. It has a lower barrier to entry if you already know Python.

This site covers Snakemake on the Snakemake page.

Other workflow managers

CWL and WDL are also used in bioinformatics. CWL is a standard backed by the Global Alliance for Genomics and Health. WDL is developed by the Broad Institute and powers the Terra/Cromwell platform. Both are less common in day-to-day academic research than Nextflow or Snakemake.

Comparison

Feature	Nextflow	Snakemake	CWL	WDL
Language	Groovy DSL	Python-based	YAML/JSON	Custom DSL
Container support	Docker, Singularity, Podman	Singularity, Docker	Docker, Singularity	Docker
Cloud support	AWS, Google, Azure	Google, AWS, Azure	Platform-dependent	Terra/Cromwell
Community pipelines	nf-core (100+)	Snakemake Catalog	Few	WARP (Broad)
Conda integration	Via profiles	Native	Limited	No
Resume/caching	Built-in	Built-in	Implementation-dependent	Via Cromwell
Learning curve	Moderate (Groovy)	Low (Python)	High (verbose YAML)	Moderate

How containers and workflow managers work together

Containers and workflow managers solve different problems. Containers freeze the software environment. Workflow managers automate the execution. Together, they make an analysis fully reproducible.

In a well-designed pipeline, each step runs inside its own container. The workflow manager pulls the right container, passes in the data, runs the command, and collects the output. No tools are installed on the host machine. No versions drift over time.

Both Nextflow and Snakemake support this pattern. When you run an nf-core pipeline, every process has a pinned container image. When you define a Snakemake rule with a container: directive, that rule runs in an isolated environment.

A reproducible project looks like this

A well-structured bioinformatics project combines all three pillars:

my-rnaseq-project/
├── workflow/           # Nextflow or Snakemake pipeline
│   ├── main.nf        # Pipeline definition
│   └── nextflow.config
├── containers/         # Dockerfiles for custom tools
│   └── Dockerfile
├── data/               # Raw data or download scripts
│   └── samplesheet.csv
├── results/            # Pipeline output (gitignored)
├── docs/               # Analysis notes and decisions
└── .git/               # Version control history

The pipeline definition is in version control. The software versions are locked in containers. The execution is automated by the workflow manager. Anyone with access to the repo and the raw data can reproduce the results.

Next steps

The following pages cover each workflow manager in detail. Start with What is Nextflow if you work with nf-core pipelines, or jump to Snakemake if you prefer a Python-based approach.