WDL & Cromwell

WDL, the Workflow Description Language, is the Broad Institute’s language for describing analysis pipelines. It powers the GATK best practices workflows, the WARP production pipelines, and the Terra cloud platform. If you run clinical or population scale genomics, you will meet it.

WDL takes a different bet from Nextflow and Snakemake. Where those two optimise for flexibility, WDL optimises for strictness. Every input is typed. The engine validates the whole workflow before it launches a single job, so a mistyped path or a wrong data type fails in seconds instead of three hours into a run. That rigidity is the point. It is what makes WDL the standard in regulated, large scale genomics.

The Why Reproducibility Matters page has the side by side comparison of all four workflow managers. This page covers WDL itself: the syntax, how to run it on your own machine, and how it scales to the cloud.

Tasks

A task is the unit of work in WDL. It wraps one command and declares everything around it: the inputs it needs, the command to run, the outputs it produces, and the runtime it wants. A task is self contained, so the same task file can be imported into many workflows.

version 1.0

task fastqc {
    input {
        File fastq
        String sample_name
    }

    command <<<
        fastqc ~{fastq} -o .
    >>>

    output {
        File html = "~{sample_name}_fastqc.html"
        File zip = "~{sample_name}_fastqc.zip"
    }

    runtime {
        docker: "biocontainers/fastqc:v0.11.9"
        memory: "4 GB"
        cpu: 1
    }
}

The ~{...} syntax interpolates a WDL value into the command or an output filename. The runtime block pins the container image and the resources. On a laptop those numbers are advisory. On the cloud they become real reservations.

Workflows

A workflow wires tasks together. It moves data from the output of one task into the input of the next, and it decides what runs in parallel. Here a workflow runs QC on every FASTQ, then aligns every FASTQ, using two imported task files.

version 1.0

import "tasks/fastqc.wdl" as qc
import "tasks/align.wdl" as align

workflow RNAseqPipeline {
    input {
        Array[File] fastqs
        File reference
    }

    scatter (fastq in fastqs) {
        call qc.fastqc {
            input:
                fastq = fastq
        }
    }

    scatter (fastq in fastqs) {
        call align.star {
            input:
                fastq = fastq,
                reference = reference
        }
    }

    output {
        Array[File] qc_reports = fastqc.html
        Array[File] bams = star.bam
    }
}

Scatter and gather

The scatter block is how WDL parallelises. It runs a task once per item in an array, and the engine schedules those runs concurrently. After a scatter, the task’s outputs become an array you can pass to a later task that gathers them.

# Scatter: one run per sample, in parallel
scatter (sample in samples) {
    call process_sample {
        input: sample = sample
    }
}

# Gather: collect every result into one call
call merge_results {
    input:
        files = process_sample.output_file
}

You never write a loop or a job array. You declare the scatter and the engine works out the fan out.

Types and structs

WDL is statically typed. A value is a File, a String, an Int, a Float, a Boolean, or an array or struct of those. A struct groups related fields, which keeps a samplesheet of paired reads and metadata tidy.

version 1.0

struct SampleInfo {
    String sample_id
    File fastq_r1
    File fastq_r2
    String condition
}

workflow AnalyzeSamples {
    input {
        Array[SampleInfo] samples
    }

    scatter (sample in samples) {
        call align {
            input:
                sample_id = sample.sample_id,
                r1 = sample.fastq_r1,
                r2 = sample.fastq_r2
        }
    }
}

The types are not bureaucracy. They are what lets the engine reject a broken workflow before it starts, which is the whole reason to reach for WDL.

Running WDL on your own machine

Two engines run WDL locally. miniwdl is a Python tool with clear error messages, best for development. Cromwell is the Broad’s production engine, a Java jar that runs the same workflow unchanged on a laptop, an HPC cluster, or the cloud.

The example below is small enough to run end to end with no genome and no cloud. It takes a list of sample names, splits each into a condition and a replicate in a scattered task, and gathers the results into one manifest. Every task leaves its runtime block empty, with no docker attribute, so Cromwell uses its Local backend and runs each command as a direct subprocess. That is the trick that lets the whole thing run inside a single container without launching nested containers.

version 1.0

workflow SampleManifest {
    input {
        Array[String] samples
    }

    # Scatter: parse one sample name per task, in parallel.
    scatter (sample in samples) {
        call describe_sample { input: sample = sample }
    }

    # Gather: collect the parsed rows into a single typed manifest.
    call build_manifest { input: rows = describe_sample.row }

    output {
        Int n_samples = build_manifest.n_samples
        File manifest = build_manifest.manifest
    }
}

task describe_sample {
    input {
        String sample
    }

    command <<<
        name="~{sample}"
        condition="${name%%_*}"   # text before the first underscore
        replicate="${name#*_}"    # text after the first underscore
        printf '%s\t%s\t%s\n' "$name" "$condition" "$replicate"
    >>>

    output {
        String row = read_string(stdout())
    }

    runtime {
        # No docker attribute: Cromwell's Local backend runs this directly.
    }
}

task build_manifest {
    input {
        Array[String] rows
    }

    command <<<
        printf 'sample\tcondition\treplicate\n' > manifest.tsv
        cat ~{write_lines(rows)} >> manifest.tsv
        echo $(( $(wc -l < manifest.tsv) - 1 ))
    >>>

    output {
        File manifest = "manifest.tsv"
        Int n_samples = read_int(stdout())
    }

    runtime {}
}

The inputs are a plain JSON file. Keys are Workflow.input_name.

{
    "SampleManifest.samples": [
        "control_rep1",
        "control_rep2",
        "treated_rep1",
        "treated_rep2"
    ]
}

Validate before you run

WDL’s headline feature is that a workflow is checked before anything executes. Both engines do this. womtool validate returns one word when the types line up:

Success!

miniwdl check goes further and prints the structure it parsed, which is a quick way to confirm the scatter and the two tasks are wired the way you meant:

sample_manifest.wdl
    workflow SampleManifest
        scatter sample
            call describe_sample
        call build_manifest
    task build_manifest
    task describe_sample

If a type were wrong, say an Int output declared where the command prints text, both tools would name the task and the line and stop. No compute is spent finding out.

Run it with Cromwell

java -jar cromwell.jar run sample_manifest.wdl -i inputs.json

Cromwell scatters the four describe_sample tasks, gathers them in build_manifest, and prints the workflow outputs as JSON when it finishes:

{
  "outputs": {
    "SampleManifest.n_samples": 4,
    "SampleManifest.manifest": ".../call-build_manifest/execution/manifest.tsv"
  }
}

The gathered manifest is the real file that run produced:

sample  condition  replicate
control_rep1  control  rep1
control_rep2  control  rep2
treated_rep1  treated  rep1
treated_rep2  treated  rep2

Everything above, the validation output, the parsed structure, the outputs JSON, and the manifest, comes from a real run of this workflow in the companion wdl-tools container. Cromwell wrote the manifest to a hashed path under cromwell-executions/; that absolute path is abbreviated above.

Installing the engines

# miniwdl, for local development
pip install miniwdl
miniwdl check workflow.wdl
miniwdl run workflow.wdl -i inputs.json

# Cromwell, the production engine
wget https://github.com/broadinstitute/cromwell/releases/download/87/cromwell-87.jar
java -jar cromwell-87.jar run workflow.wdl --inputs inputs.json

One difference matters in practice. miniwdl run launches a container for every task, so it needs Docker or Podman available. Cromwell’s Local backend runs a task’s command directly when the task declares no docker runtime, which is why the manifest example above runs with nothing but a JVM.

Running WDL on the cloud

Reference

The cloud sections below need an AWS or Google account, IAM roles, and real reference data. Their code is faithful but was not run here, the same way the Nextflow AWS pages are reference. The manifest example above is the part of this page that actually runs.

Cromwell’s portability is the reason the Broad built on it. The same WDL runs on a laptop, an HPC scheduler, or a cloud batch service. You change the backend, not the workflow.

AWS HealthOmics

HealthOmics is AWS’s managed genomics service. It runs WDL natively without you standing up an engine.

aws omics create-workflow \
    --name "rnaseq-pipeline" \
    --definition-zip fileb://workflow.zip \
    --engine WDL

aws omics start-run \
    --workflow-id wf-1234567890 \
    --role-arn arn:aws:iam::123456789:role/OmicsRole \
    --output-uri s3://my-bucket/outputs/ \
    --parameters file://inputs.json

Cromwell on AWS Batch

For more control, run your own Cromwell against an AWS Batch backend. The backend lives in a HOCON config file.

aws.conf

backend {
  default = "AWSBATCH"
  providers {
    AWSBATCH {
      actor-factory = "cromwell.backend.impl.aws.AwsBatchBackendLifecycleActorFactory"
      config {
        root = "s3://my-cromwell-bucket/cromwell-execution"
        auth = "default"
        default-runtime-attributes {
          queueArn = "arn:aws:batch:us-east-1:123456789:job-queue/cromwell-queue"
        }
        filesystems {
          s3 { auth = "default" }
        }
      }
    }
  }
}

java -Dconfig.file=aws.conf -jar cromwell.jar run workflow.wdl -i inputs.json

Inputs then point at S3 instead of local paths, and the engine stages data in and out for you.

{
    "RNAseqPipeline.fastqs": [
        "s3://my-bucket/reads/sample1.fastq.gz",
        "s3://my-bucket/reads/sample2.fastq.gz"
    ],
    "RNAseqPipeline.reference": "s3://my-bucket/ref/genome.fa"
}

To hold cloud costs down, request Spot instances for tasks that tolerate a restart, right size each task with runtime attributes, and let call caching skip work that already succeeded.

runtime {
    docker: "biocontainers/bwa:v0.7.17"
    memory: "16 GB"
    cpu: 4
    preemptible: 3   # retry on Spot up to three times
}

WARP pipelines

WARP, the WDL Analysis Research Pipelines, are the Broad’s production ready workflows. If your task is a standard one, start here rather than writing your own.

Pipeline	Use case
WholeGenomeGermlineSingleSample	WGS variant calling
JointGenotyping	Multi sample variant calling
ExomeGermlineSingleSample	WES variant calling
RNAWithUMIsPipeline	RNA-seq with UMIs

git clone https://github.com/broadinstitute/warp.git
cd warp

miniwdl run pipelines/broad/dna_seq/germline/single_sample/wgs/WholeGenomeGermlineSingleSample.wdl \
    -i inputs.json

These pipelines expect real reference bundles. The germline inputs, for instance, point at the public hg38 references in Google Cloud:

{
    "WholeGenomeGermlineSingleSample.sample_name": "NA12878",
    "WholeGenomeGermlineSingleSample.references": {
        "ref_fasta": "gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.fasta",
        "ref_fasta_index": "gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.fasta.fai",
        "ref_dict": "gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.dict"
    }
}

Terra

Terra is the Broad’s web platform built on WDL and Cromwell. You import a workflow from Dockstore or GitHub, configure its inputs, and launch without touching an engine.

Its data model is the reason people use it. Rather than hardcoding file paths, you upload a data table, a spreadsheet where each row is a sample and columns hold metadata and bucket paths. The same workflow then runs on a new batch of five hundred samples by selecting a different set of rows. Data and logic stay separate, which is exactly the discipline the workflow manager is there to enforce.

When to reach for WDL

Choose WDL when you run GATK best practices, work on Terra, or operate in a clinical or production setting where strict validation and typing earn their keep. Its portability through Cromwell means one script runs unchanged from a laptop to a cluster to the cloud.

The tradeoffs are real. WDL is more verbose than Snakemake’s Python and less flexible than a general purpose language. Its community is smaller than Nextflow’s or Snakemake’s and centred on human genomics. If you want community pipelines out of the box for RNA-seq or variant calling, nf-core has more of them. If your team already writes Python and runs on SLURM, Snakemake will feel more natural.

WDL earns its place when the cost of a pipeline failing halfway through, on real patient data, is high enough that catching the error before launch is worth the extra ceremony.

Next steps

The Why Reproducibility Matters page compares WDL, Nextflow, and Snakemake side by side. For running community pipelines with minimal setup, the Nextflow & nf-core pages cover nf-core on AWS Batch with real datasets.