Snakemake

What is Snakemake

Snakemake is a workflow manager that uses Python syntax to define analysis pipelines. It was created in 2012 by Johannes Köster and has become one of the most popular tools for automating bioinformatics workflows.

Snakemake follows the same philosophy as GNU Make. You define the output files you want. Snakemake figures out which steps to run and in what order to produce them. If an intermediate file already exists and its inputs have not changed, Snakemake skips that step.

Core concepts

Rules

A rule is a single step in your pipeline. It declares input files, output files, and a shell command. Here is a rule that runs FastQC on a FASTQ file:

rule fastqc:
    input:
        "data/{sample}.fastq.gz"
    output:
        "results/fastqc/{sample}_fastqc.html"
    shell:
        "fastqc {input} --outdir results/fastqc/"

Wildcards

The {sample} in the rule above is a wildcard. Snakemake replaces it with actual sample names at runtime. If you request output for three samples, Snakemake creates three jobs from the same rule.

rule all:
    input:
        expand("results/fastqc/{sample}_fastqc.html",
               sample=["sample1", "sample2", "sample3"])

The rule all is a convention. It lists all final output files you want. Snakemake works backward from these targets to determine which rules to run.

The DAG

Snakemake builds a Directed Acyclic Graph from your rules. Each node is a job. Edges represent dependencies. Snakemake analyzes which jobs can run in parallel and which must wait for upstream results.

You can visualize the DAG for any workflow:

snakemake --dag | dot -Tpng > dag.png

This produces a graph showing every job and its dependencies. It is useful for debugging and for explaining your pipeline to collaborators.

A simple RNA-seq workflow

Here is a minimal Snakemake workflow that trims reads, aligns them, and counts features. This is not production code. It shows how rules chain together.

# Snakefile

SAMPLES = ["sample1", "sample2", "sample3"]

rule all:
    input:
        "results/counts/all_counts.tsv"

rule trim:
    input:
        r1="data/{sample}_R1.fastq.gz",
        r2="data/{sample}_R2.fastq.gz"
    output:
        r1="results/trimmed/{sample}_R1.fastq.gz",
        r2="results/trimmed/{sample}_R2.fastq.gz"
    shell:
        """
        trim_galore --paired {input.r1} {input.r2} \
            --output_dir results/trimmed/
        """

rule align:
    input:
        r1="results/trimmed/{sample}_R1.fastq.gz",
        r2="results/trimmed/{sample}_R2.fastq.gz",
        index="references/genome_index"
    output:
        "results/aligned/{sample}.bam"
    threads: 8
    shell:
        """
        STAR --runThreadN {threads} \
            --genomeDir {input.index} \
            --readFilesIn {input.r1} {input.r2} \
            --readFilesCommand zcat \
            --outSAMtype BAM SortedByCoordinate \
            --outFileNamePrefix results/aligned/{wildcards.sample}
        mv results/aligned/{wildcards.sample}Aligned.sortedByCoord.out.bam {output}
        """

rule count:
    input:
        bam="results/aligned/{sample}.bam",
        gtf="references/genes.gtf"
    output:
        "results/counts/{sample}.txt"
    shell:
        "featureCounts -a {input.gtf} -o {output} {input.bam}"

rule merge_counts:
    input:
        expand("results/counts/{sample}.txt", sample=SAMPLES)
    output:
        "results/counts/all_counts.tsv"
    shell:
        "paste {input} | cut -f1,7,14,21 > {output}"

Snakemake reads the rule all target, traces the dependency chain, and runs each rule in the correct order. If you add a fourth sample to the SAMPLES list, Snakemake runs only the new jobs.

Running Snakemake

Basic execution

# Dry run: show what would be executed without running anything
snakemake -n

# Run with 4 parallel jobs
snakemake --cores 4

# Run a specific target
snakemake results/counts/all_counts.tsv --cores 4

The --cores flag controls how many jobs run simultaneously. Snakemake respects the threads directive in each rule when allocating cores.

Resume after failure

Snakemake tracks which output files exist and which are up to date. If a run fails at the alignment step, fix the problem and rerun. Snakemake skips all completed steps automatically. There is no special flag needed.

# Just rerun the same command. Completed steps are skipped.
snakemake --cores 4

If you suspect a partial output file from a failed run, use the --rerun-incomplete flag to force recomputation of unfinished jobs.

Container support

Snakemake can run each rule inside a container. This ensures every step uses exactly the software versions you specify.

Singularity and Apptainer

Most HPC clusters do not allow Docker for security reasons. Snakemake supports Singularity and its successor Apptainer natively.

rule align:
    input:
        r1="results/trimmed/{sample}_R1.fastq.gz",
        r2="results/trimmed/{sample}_R2.fastq.gz"
    output:
        "results/aligned/{sample}.bam"
    container:
        "docker://quay.io/biocontainers/star:2.7.11b--h43eeafb_0"
    threads: 8
    shell:
        """
        STAR --runThreadN {threads} \
            --genomeDir references/genome_index \
            --readFilesIn {input.r1} {input.r2} \
            --readFilesCommand zcat \
            --outSAMtype BAM SortedByCoordinate \
            --outFileNamePrefix results/aligned/{wildcards.sample}
        """

Run the workflow with container support enabled:

snakemake --cores 4 --use-singularity

Snakemake pulls the container image and runs the rule inside it. The container directive pins the exact tool version.

Conda integration

Snakemake also supports Conda environments as an alternative to containers. Each rule can specify a Conda environment file.

rule fastqc:
    input:
        "data/{sample}.fastq.gz"
    output:
        "results/fastqc/{sample}_fastqc.html"
    conda:
        "envs/fastqc.yaml"
    shell:
        "fastqc {input} --outdir results/fastqc/"

The environment file specifies the packages and versions:

envs/fastqc.yaml

channels:
  - bioconda
  - conda-forge
dependencies:
  - fastqc=0.12.1

Run with Conda enabled:

snakemake --cores 4 --use-conda

Snakemake creates the Conda environment on the first run and reuses it for subsequent runs. This approach is simpler than containers but less portable. Conda environments can behave differently across operating systems.

Configuration files

Hard-coding sample names and paths in the Snakefile is fine for small projects. For larger analyses, use a configuration file.

config.yaml

samples:
  - sample1
  - sample2
  - sample3

genome_index: "references/genome_index"
gtf: "references/genes.gtf"

Access the configuration in your Snakefile:

configfile: "config.yaml"

SAMPLES = config["samples"]

rule align:
    input:
        r1="results/trimmed/{sample}_R1.fastq.gz",
        r2="results/trimmed/{sample}_R2.fastq.gz",
        index=config["genome_index"]
    ...

Cluster and cloud execution

Snakemake supports execution on HPC clusters and cloud platforms.

SLURM clusters

snakemake --cores 100 --executor slurm \
    --default-resources slurm_partition=main mem_mb=8000

Snakemake submits each job to SLURM and monitors its status. You can set per-rule resources to request more memory or CPUs for expensive steps.

Cloud execution

Snakemake supports Google Cloud, AWS, and Azure through executor plugins. Cloud execution uploads your data, runs each job on a cloud instance, and downloads the results.

The Snakemake Workflow Catalog

The Snakemake Workflow Catalog is a curated collection of community workflows. It serves a similar purpose to nf-core for Nextflow. Workflows in the catalog follow standardized structure and testing guidelines.

The catalog is smaller than nf-core. It covers common use cases like RNA-seq, variant calling, and ATAC-seq. If you need a pipeline for a standard analysis, check the catalog before writing your own.

When to choose Snakemake vs Nextflow

Both tools solve the same fundamental problem. The choice often comes down to your environment and team.

Choose Snakemake if:

• Your team already writes Python.
• You work on an HPC cluster with SLURM.
• You want tight Conda integration for managing environments.
• You prefer file-based dependency resolution.
• Your workflows are custom and not covered by nf-core.

Choose Nextflow if:

• You want to use nf-core pipelines out of the box.
• You run analyses on AWS Batch or other cloud platforms.
• Your institution already uses Nextflow.
• You need production-grade pipelines maintained by a large community.

Both tools support containers. Both support resuming failed runs. Both produce reproducible results when configured correctly. The best workflow manager is the one your team will actually use.

Installing Snakemake

The recommended installation uses Conda or Mamba:

# Install with Mamba (faster than Conda)
mamba create -n snakemake -c conda-forge -c bioconda snakemake

# Activate the environment
mamba activate snakemake

# Verify the installation
snakemake --version

You can also install with pip, but the Conda installation is preferred because it includes all optional dependencies for cluster and cloud execution.

Next steps

If you want to run production-ready pipelines with minimal setup, continue to the Nextflow & nf-core pages. These cover running community-maintained pipelines on AWS Batch with real datasets.