What is Nextflow

Nextflow is a workflow manager built for computational biology. It runs the same pipeline on a laptop, an HPC cluster, or AWS Batch with no code changes. It powers nf-core, the community-curated set of production pipelines for RNA-seq, variant calling, single-cell, metagenomics, and more.

This page covers the mental model you need to read or write a Nextflow pipeline, what a pipeline actually looks like on disk, how to run one, and how to debug it when things go wrong.

The problem with bash scripts

A typical RNA-seq analysis runs FastQC, trims adapters, aligns reads, quantifies transcripts, and merges counts. Thirty samples mean hundreds of commands. You can chain them in a bash script. Many people do. They break in predictable ways:

• A step fails silently and the next step runs on corrupted input.
• You cannot easily restart from a failed step without rerunning everything.
• Scaling from 4 samples to 40 means rewriting your parallelisation logic.
• Moving from your laptop to a cluster or cloud means rewriting how jobs are submitted.

Workflow managers solve these. They track dependencies, parallelise automatically, cache completed work, and separate analysis logic from execution environment.

The mental model: dataflow

Nextflow is not a procedural language. It is a dataflow language. The unit of computation is not a function call, it is a stream of items.

You build a pipeline by declaring processes (the steps) and connecting them with channels (the streams). When you submit the workflow, Nextflow looks at how the channels connect and figures out what to run, in what order, and in parallel.

You never write a for loop over samples. You put samples on a channel and let Nextflow handle parallelism. You never explicitly schedule jobs. You declare the work and the executor (local, SLURM, AWS Batch) decides where each task runs.

This shift takes a few hours to internalise. Once it clicks, the rest of Nextflow makes sense.

Vocabulary

Term	What it is
Process	One step of the pipeline. Has inputs, outputs, and a script block.
Channel	A queue or stream of items flowing into or out of a process.
Workflow	The block that wires processes and channels together.
Task	One execution of a process on one item. A process producing 30 outputs runs 30 tasks.
Work directory	Where each task runs. One subdirectory per task.
Profile	A named configuration bundle for a target environment (laptop, cluster, AWS).
Executor	The backend that runs tasks. Local processes, SLURM, AWS Batch, etc.

A minimal pipeline

This is a two-step pipeline that runs FastQC on a folder of FASTQ files and then trims them.

process FASTQC {
    input:
    path reads

    output:
    path "*.html"

    script:
    """
    fastqc $reads
    """
}

process TRIM {
    input:
    path reads

    output:
    path "*.trimmed.fastq.gz"

    script:
    """
    fastp -i $reads -o ${reads.simpleName}.trimmed.fastq.gz
    """
}

workflow {
    reads_ch = Channel.fromPath("data/*.fastq.gz")
    FASTQC(reads_ch)
    TRIM(reads_ch)
}

A few things to notice:

• Each process declares only its inputs, outputs, and a script that does the work.
• The script is plain shell (or Python, R, etc.) and uses the input variables directly.
• The workflow block creates a channel from a glob, then passes it to two processes.
• Both processes consume the same channel. Nextflow handles fan-out automatically.
• If the channel has 30 files, FastQC runs 30 times in parallel. So does TRIM.

You do not write any parallelisation logic.

DSL2 modules

Real pipelines have many processes. DSL2 lets you put each process in its own file and import it.

include { FASTQC } from './modules/fastqc'
include { TRIM   } from './modules/trim'
include { ALIGN  } from './modules/align'

workflow {
    reads_ch = Channel.fromPath(params.input)
    FASTQC(reads_ch)
    TRIM(reads_ch)
    ALIGN(TRIM.out)
}

This modular design is the foundation of every nf-core pipeline. Each tool sits in its own module file. The nf-core community goes one step further: it maintains a shared module library at nf-core/modules that any pipeline can pull from. When nf-core/rnaseq and nf-core/sarek both need to call samtools sort, they share the same module file. Bug fixes flow to every pipeline at once.

Resource requests

Each process declares the resources it needs. Nextflow forwards these to the executor.

process ALIGN {
    cpus 8
    memory '32 GB'
    time '4h'

    input:
    path reads

    output:
    path "*.bam"

    script:
    """
    STAR --runThreadN $task.cpus --readFilesIn $reads ...
    """
}

On a laptop these are advisory. On AWS Batch or SLURM they become real resource reservations. The $task.cpus variable is interpolated into the script so the tool uses the requested thread count.

How Nextflow runs your pipeline

When you submit a workflow, Nextflow does this for every task:

1. Hash the task inputs, the script, and the container image into a unique key.
2. Create a directory at work/<first-2-chars>/<rest-of-hash>/.
3. Stage the input files into that directory as symlinks.
4. Write a .command.sh script and a .command.run wrapper.
5. Hand .command.run to the executor.
6. When the task finishes, the outputs sit in that same directory.

This is why -resume works. If you re-run with the same inputs, the hash is the same, the work directory exists, and Nextflow skips the task.

nextflow run my_pipeline.nf -resume

You can run with -resume after a failure, after a parameter tweak that does not affect upstream tasks, or after editing only one process. Nextflow recomputes only what changed.

The cost: the work/ directory grows. Clean it up with nextflow clean -f once a project is done.

Execution reports

Add three flags and Nextflow produces an HTML report, a TSV trace, and a timeline.

nextflow run my_pipeline.nf \
    -with-report report.html \
    -with-trace trace.txt \
    -with-timeline timeline.html

The report shows CPU and memory usage per task. The trace is a flat table you can grep. The timeline is a Gantt chart of when each task ran. All three are essential when a pipeline is too slow or hits OOM.

Profiles and portability

Nextflow separates analysis logic from execution. The same pipeline runs on a laptop, an HPC cluster, or AWS Batch. You switch environments by selecting a profile.

# Local with Docker
nextflow run my_pipeline.nf -profile docker

# HPC with Singularity
nextflow run my_pipeline.nf -profile singularity

# AWS Batch
nextflow run my_pipeline.nf -profile docker -c aws.config

A profile is a named block in nextflow.config that defines the executor, container engine, and resource limits for that environment. Your pipeline code stays the same.

Configuration layers

Three places set parameters and configuration. They override in this order, lowest to highest:

1. The pipeline’s own nextflow.config.
2. A user-supplied -c custom.config or -params-file params.yml.
3. Command-line --param value flags.

So --input data/ on the command line beats both the pipeline default and any custom config. This layering lets you ship sensible defaults in the pipeline, override them per project, and tweak one parameter on the fly.

When something fails

Pipelines fail. Your task is to find which task failed and why. The Nextflow output tells you both.

When a task errors, Nextflow prints the failing process name and the path to its work directory:

ERROR ~ Error executing process > 'FASTQC (sample_3)'

Caused by:
  Process `FASTQC (sample_3)` terminated with an error exit status (127)

Command executed:
  fastqc sample_3.fastq.gz

Work dir:
  /scratch/work/8a/3f72c1...

cd into that work directory and look at the diagnostic files Nextflow stages there:

File	Contains
.command.sh	The exact script that ran. Run this to reproduce.
.command.err	What the tool wrote to stderr. The actual error message lives here.
.command.log	Combined stdout and stderr.
.command.run	The wrapper Nextflow used. Useful for debugging the executor.
.exitcode	The exit code.

Most failures are obvious from .command.err: a missing file, an out-of-memory exit, a tool installation issue. Fix the cause, re-run with -resume, and only the failed task and its descendants rerun.

What is nf-core

nf-core is a community of bioinformaticians who maintain production-ready Nextflow pipelines. Every nf-core pipeline meets the same standards:

• Peer-reviewed code. Each pipeline is reviewed before release.
• Containerised tools. Every process pins a container image, so the result is reproducible across machines and over time.
• Standardised inputs. A CSV samplesheet with a documented column format.
• Comprehensive testing. A built-in test profile and continuous integration.
• Detailed documentation. Every pipeline has a website with parameters, outputs, and worked examples.

Pipeline layout on disk

Clone any nf-core pipeline and you see the same structure:

nf-core-rnaseq/
├── main.nf                # workflow entry point
├── nextflow.config        # default parameters and profiles
├── nextflow_schema.json   # parameter schema for the launcher UI
├── conf/
│   ├── base.config        # default resource requests
│   ├── modules.config     # per-module overrides
│   └── test.config        # tiny dataset for CI
├── modules/               # per-tool process definitions
├── workflows/             # composition of modules into the pipeline
├── subworkflows/          # smaller reusable composites
├── assets/
│   └── samplesheet_schema.json
├── docs/
└── README.md

When something is unclear, the answer is usually in conf/, modules.config, or the docs.

Modules: write once, reuse everywhere

The modules/ directory holds one process per tool. nf-core pipelines do not write these from scratch. They install them from the shared nf-core/modules repo using the nf-core CLI:

nf-core modules install samtools/sort
nf-core modules install fastqc
nf-core modules update samtools/sort

Each module is a small folder with a main.nf (the process), a meta.yml (input/output schema), and tests. Versioning is per-module, so you can update FastQC without touching anything else.

Why this matters as a user: when you read an nf-core pipeline and want to know what FastQC version it runs, you look at modules/nf-core/fastqc/main.nf in the pipeline, then cross-reference the same file in nf-core/modules to see if it has been updated since.

Config files: where you actually customise

nf-core pipelines are deeply configurable without editing the pipeline itself. Three layers of config files do the work:

nextflow.config (in the pipeline root) holds the default parameters and the profile definitions. Read it to discover every parameter the pipeline accepts. Do not edit it.

conf/base.config sets default resources per label. Processes are tagged with labels like process_low, process_medium, process_high, process_high_memory. The base config maps each label to CPU and memory budgets. Override here when your cluster has unusual resource limits.

conf/modules.config is where per-process behaviour lives. This is the file you will read most often. Two key directives:

• publishDir decides which output files end up in your --outdir, where, and how they are named.
• ext.args injects extra command-line flags into the tool. So if nf-core/rnaseq calls STAR with default arguments and you want to add --alignIntronMax 100000, you do it here without touching the module.

process {
    withName: 'STAR_ALIGN' {
        ext.args = '--alignIntronMax 100000 --twopassMode Basic'
        publishDir = [
            path: "${params.outdir}/star",
            mode: 'copy',
            pattern: '*.bam'
        ]
    }
}

You drop a snippet like this into your own custom.config and pass it on the command line: nextflow run nf-core/rnaseq ... -c custom.config. Your changes layer on top of the pipeline defaults without forking anything.

Institutional configs

The nf-core/configs repo holds ready-made config files for major HPC sites and cloud accounts (UPPMAX, EBI, Sanger, Crick, Broad, etc.). If your institution is in there you can chain its profile in with a comma:

nextflow run nf-core/rnaseq -profile crick,docker --input samplesheet.csv --outdir results

The institution’s config sets the executor (SLURM, LSF), queue names, scratch paths, container cache locations, and resource limits that match its hardware. You stop reinventing those settings.

If your institution is not in nf-core/configs you can write a minimal local profile and submit a PR to share it with everyone else who works there.

Samplesheets

Every nf-core pipeline takes a CSV samplesheet as its primary input. For nf-core/rnaseq it looks like:

sample,fastq_1,fastq_2,strandedness
TREATED_REP1,/data/treated_rep1_R1.fastq.gz,/data/treated_rep1_R2.fastq.gz,auto
TREATED_REP2,/data/treated_rep2_R1.fastq.gz,/data/treated_rep2_R2.fastq.gz,auto
CONTROL_REP1,/data/control_rep1_R1.fastq.gz,/data/control_rep1_R2.fastq.gz,auto
CONTROL_REP2,/data/control_rep2_R1.fastq.gz,/data/control_rep2_R2.fastq.gz,auto

The exact columns differ by pipeline. Each pipeline ships a samplesheet_schema.json and an example file. You build yours, point the pipeline at it, and let Nextflow handle the rest.

nf-core pipelines for bioinformatics

nf-core maintains over 100 pipelines covering most areas of genomics:

Pipeline	Purpose
nf-core/rnaseq	Bulk RNA-seq quantification
nf-core/differentialabundance	Differential expression analysis
nf-core/scrnaseq	Single-cell RNA-seq processing
nf-core/sarek	Variant calling from DNA-seq
nf-core/taxprofiler	Metagenomic taxonomic profiling
nf-core/chipseq	ChIP-seq peak calling
nf-core/fetchngs	Download from SRA/ENA
nf-core/airrflow	BCR/TCR repertoire analysis

Browse the full list at nf-co.re/pipelines.

Installing Nextflow

Nextflow needs Java 11 or later and a container engine.

# Java (Ubuntu/Debian)
sudo apt update && sudo apt install -y default-jre

# Nextflow itself
curl -s https://get.nextflow.io | bash
mv nextflow ~/bin/

Verify:

nextflow -version

      N E X T F L O W
      version 25.10.2 build 5935
      created 20-01-2025 14:52 UTC
      cite doi:10.1038/nbt.3820
      http://nextflow.io

For containers: Docker or Podman on a laptop, Singularity or Apptainer on HPC, AWS Batch handles its own on the cloud.

Running your first nf-core pipeline

Every nf-core pipeline ships with a tiny test dataset. Use it to verify your setup.

nextflow run nf-core/rnaseq -r 3.14.0 -profile test,docker --outdir test_results

This downloads the pipeline, pulls the containers, runs a minimal dataset, and exits in a few minutes. If it completes without errors, your Nextflow plus container setup is working.

Monitoring with Seqera Platform

Seqera Platform (formerly Nextflow Tower) gives you a web UI for monitoring runs, sharing pipelines with a team, and launching jobs on AWS Batch from a browser. It is optional but useful when several people share a compute environment. The free tier handles individual use.

Nextflow vs Snakemake

Nextflow and Snakemake solve the same problem with different philosophies. Nextflow is dataflow-first and built around containers. Snakemake is rule-based and Python-native, popular on HPC clusters. The Why Reproducibility Matters page has a side-by-side comparison and guidance on when to pick which.

Next steps

Nextflow runs on a laptop. Real datasets need more compute. The next page covers setting up AWS so you can run nf-core pipelines on AWS Batch.