AWS Batch for Nextflow

AWS Batch is a managed compute service. You define the types of machines you want and Batch handles provisioning, scaling, and cleanup. Nextflow submits each pipeline process as a Batch job. Batch spins up EC2 instances to run the jobs, then shuts them down when the work is done.

This is the key advantage over running Nextflow on a single EC2 instance. Instead of being limited to one machine, Batch can launch dozens of instances in parallel. A 30-sample RNA-seq pipeline that takes 8 hours on a single machine can finish in under an hour on Batch.

Spot instances: save 60 to 70 percent

AWS Spot instances are spare EC2 capacity sold at a steep discount. Spot prices are typically 60 to 70% lower than on-demand prices. The tradeoff is that AWS can reclaim a Spot instance with 2 minutes notice if capacity runs low.

This sounds risky, but Nextflow handles interruptions gracefully. If a Spot instance is reclaimed, Nextflow detects the failure and retries the task on a new instance. Combined with Nextflow’s caching, you lose at most a few minutes of work on the interrupted task.

For bioinformatics pipelines, Spot is almost always the right choice. The savings are significant and the risk is minimal.

Instance	vCPUs	RAM	On-demand	Spot (typical)	Savings
c5.xlarge	4	8 GB	$0.170/hr	$0.060/hr	65%
m5.xlarge	4	16 GB	$0.192/hr	$0.070/hr	64%
r5.xlarge	4	32 GB	$0.252/hr	$0.080/hr	68%

Setting up AWS Batch

Setting up Batch requires three things: IAM roles, a compute environment, and a job queue. The following sections walk through each one using the AWS CLI.

Step 1: Create IAM roles

AWS Batch needs three IAM roles.

Batch service-linked role. This lets the Batch service manage EC2 instances on your behalf:

aws iam create-service-linked-role \
  --aws-service-name batch.amazonaws.com

ECS instance role. The EC2 instances launched by Batch need permission to pull container images and access S3:

# Create the role
aws iam create-role \
  --role-name ecsInstanceRole \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {"Service": "ec2.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }]
  }'

# Attach policies
aws iam attach-role-policy \
  --role-name ecsInstanceRole \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role

aws iam attach-role-policy \
  --role-name ecsInstanceRole \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess

# Create instance profile
aws iam create-instance-profile \
  --instance-profile-name ecsInstanceRole

aws iam add-role-to-instance-profile \
  --instance-profile-name ecsInstanceRole \
  --role-name ecsInstanceRole

Spot Fleet role. This lets Batch request Spot instances:

aws iam create-role \
  --role-name AmazonEC2SpotFleetRole \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {"Service": "spotfleet.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }]
  }'

aws iam attach-role-policy \
  --role-name AmazonEC2SpotFleetRole \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEC2SpotFleetTaggingRole

Step 2: Create a custom AMI

Batch launches EC2 instances from an Amazon Machine Image. The default ECS-optimized AMI has a small root volume that fills up when pulling Docker images. Create a custom AMI with more storage.

1. Launch an instance using the ECS-optimized Amazon Linux 2 AMI.
2. Set the root volume to 100 GB gp3.
3. Once the instance is running, stop it.
4. In the EC2 console, select the instance and choose Actions > Image and templates > Create image.
5. Name it (e.g., ecs-nextflow-100gb).
6. Note the AMI ID (e.g., ami-0abc1234def56789). You will use it in the compute environment.

The 100 GB root volume ensures there is enough space for the Docker images that nf-core pipelines pull during execution.

Step 3: Create a compute environment

The compute environment defines the pool of machines Batch can use. Create a Spot compute environment with a mix of instance families:

aws batch create-compute-environment \
  --compute-environment-name nextflow-ce \
  --type MANAGED \
  --compute-resources '{
    "type": "SPOT",
    "allocationStrategy": "SPOT_CAPACITY_OPTIMIZED",
    "bidPercentage": 40,
    "minvCpus": 0,
    "maxvCpus": 256,
    "instanceTypes": ["c5", "m5", "r5"],
    "imageId": "ami-0abc1234def56789",
    "subnets": ["subnet-xxxx"],
    "securityGroupIds": ["sg-xxxx"],
    "instanceRole": "arn:aws:iam::ACCOUNT_ID:instance-profile/ecsInstanceRole",
    "spotIamFleetRole": "arn:aws:iam::ACCOUNT_ID:role/AmazonEC2SpotFleetRole"
  }'

Key settings explained:

Setting	Value	Why
type	SPOT	60 to 70% cost savings
allocationStrategy	SPOT_CAPACITY_OPTIMIZED	Picks instance pools with the lowest interruption rate
bidPercentage	40	Caps Spot price at 40% of on-demand
minvCpus	0	Scales to zero when idle, no cost
maxvCpus	256	Upper limit on parallel capacity
instanceTypes	c5, m5, r5	Compute-optimized, general purpose, and memory-optimized families

Replace subnet-xxxx, sg-xxxx, and ACCOUNT_ID with your own values. You can find your default subnet and security group in the VPC console.

Step 4: Create a job queue

The job queue connects Nextflow to the compute environment:

aws batch create-job-queue \
  --job-queue-name nextflow-queue \
  --priority 1 \
  --state ENABLED \
  --compute-environment-order '[{
    "order": 1,
    "computeEnvironment": "nextflow-ce"
  }]'

Wait a few minutes for both the compute environment and job queue to reach VALID and ENABLED state:

aws batch describe-compute-environments \
  --compute-environments nextflow-ce \
  --query 'computeEnvironments[0].status'

aws batch describe-job-queues \
  --job-queues nextflow-queue \
  --query 'jobQueues[0].status'

Wave and Fusion

Wave and Fusion are two technologies from Seqera that simplify running Nextflow on AWS.

Wave adds a thin layer to Docker images at runtime. It injects the Fusion client into containers without modifying the original images. You do not need to build custom containers.

Fusion mounts S3 as a POSIX filesystem inside each container. Tools that expect local file paths work with S3 paths transparently. Without Fusion, you would need to copy files from S3 into the container and back. Fusion eliminates this overhead.

Together, Wave and Fusion mean you can use standard nf-core containers with S3 storage, with no extra configuration inside the containers.

The Nextflow AWS config file

Create a configuration file that tells Nextflow to use AWS Batch with Wave and Fusion:

aws.config

workDir = "s3://my-nfcore-bucket/work"

aws {
    region = 'us-east-1'
}

wave {
    enabled = true
}

fusion {
    enabled = true
    exportStorageCredentials = true
}

process {
    executor = 'awsbatch'
    queue    = 'nextflow-queue'
}

params {
    max_memory = '200.GB'
    max_cpus   = 32
    max_time   = '48.h'
}

The workDir is where Nextflow stores intermediate files. It must be an S3 path when using AWS Batch. The process block tells Nextflow to submit every task to the nextflow-queue job queue. The params block sets upper limits for resource requests.

Save this file on your EC2 head node. You will reference it with -c aws.config when running pipelines.

Verifying the setup

Run the nf-core/rnaseq test profile on AWS Batch to confirm everything works:

nextflow run nf-core/rnaseq \
  -r 3.14.0 \
  -profile docker \
  -c aws.config \
  --input 's3://my-nfcore-bucket/test/samplesheet.csv' \
  --outdir 's3://my-nfcore-bucket/test/results' \
  --genome GRCh38

Watch the Nextflow log for tasks transitioning from SUBMITTED to RUNNING to COMPLETED. You can also monitor jobs in the AWS Batch console.

Cost example: real pipeline run

Here is the actual cost breakdown from running both nf-core/rnaseq and nf-core/differentialabundance on 8 airway RNA-seq samples using this setup:

Item	Cost
rnaseq Spot compute (2.57 CPU-hours)	$0.041
differentialabundance Spot compute (1.21 CPU-hours)	$0.019
S3 storage (12.6 GB for 1 day)	$0.010
S3 requests (~10k PUT + GET)	$0.027
S3 egress (~120 MB download)	$0.011
ECR image pulls	$0.004
Total	$0.112

The entire two-pipeline analysis cost 11 cents. Spot instances saved about 63% compared to on-demand pricing.

Troubleshooting

Jobs stuck in RUNNABLE

If jobs stay in RUNNABLE and never start, check:

• The compute environment is in VALID state.
• Your subnets have internet access for pulling container images.
• The ecsInstanceRole has the required policies attached.
• The maxvCpus is not set to 0.

S3 permission denied

If tasks fail with S3 access errors:

• Verify AmazonS3FullAccess is attached to ecsInstanceRole.
• Confirm fusion.exportStorageCredentials = true is in your config.
• Check that the S3 bucket is in the same region as your Batch environment.

Fusion license warning

You may see a Fusion license warning in the Nextflow log. Fusion is free for open-source pipelines like nf-core. The warning can be ignored.

Cleanup

When you are done with your analysis, clean up to avoid ongoing charges:

# Disable and delete the job queue
aws batch update-job-queue --job-queue nextflow-queue --state DISABLED
# Wait for state change, then:
aws batch delete-job-queue --job-queue nextflow-queue

# Disable and delete the compute environment
aws batch update-compute-environment --compute-environment nextflow-ce --state DISABLED
# Wait for state change, then:
aws batch delete-compute-environment --compute-environment nextflow-ce

# Deregister your custom AMI
aws ec2 deregister-image --image-id ami-0abc1234def56789

# Delete S3 data if no longer needed
aws s3 rb s3://my-nfcore-bucket --force

Keep your EC2 head node if you plan to run more analyses. Just stop it when not in use.

Next steps

With AWS Batch configured, you are ready to run a real analysis. The next page walks through running nf-core/rnaseq on the airway dataset from end to end.