Setting Up AWS

Running nf-core pipelines on your laptop works for test datasets. Real datasets with dozens of samples need more compute power. AWS provides on-demand servers that you pay for by the minute, so you only spend money while your pipeline is running.

This page walks through three steps: creating an AWS account, launching an EC2 instance, and installing Nextflow on it. The EC2 instance acts as a head node that submits pipeline tasks to AWS Batch. The head node itself is small and cheap. The heavy computation happens on Batch workers that spin up automatically and shut down when finished.

Creating an AWS account

I followed this video step by step when I first set up my own AWS account. It is a good starting point for anyone new to AWS.

The video covers:

• Creating a root account
• Setting up billing alerts to avoid surprises
• Creating an IAM admin user so you do not use the root account for daily work
• Enabling multi-factor authentication

Free Tier

AWS offers a 12-month free tier that includes 750 hours per month of t2.micro or t3.micro EC2 instances. This is enough for a Nextflow head node. The actual compute happens on AWS Batch, which is billed separately based on the instances your pipeline uses.

Creating an S3 bucket

Nextflow on AWS Batch needs an S3 bucket to store input files, intermediate results, and final outputs. S3 is object storage. It works like a cloud hard drive that every AWS service can access.

Create a bucket in the AWS console or with the CLI:

aws s3 mb s3://my-nfcore-bucket --region us-east-1

Pick a region close to you. All AWS resources for your pipeline should be in the same region. This tutorial uses us-east-1.

Bucket names are globally unique

S3 bucket names must be unique across all AWS accounts. If my-nfcore-bucket is taken, pick a different name. A good pattern is nfcore-yourname or nfcore-yourlab.

Upload your data to S3

Your FASTQ files and reference genome need to be in S3 for AWS Batch to access them. Upload with the AWS CLI:

# Upload FASTQ files
aws s3 cp data/fastq/ s3://my-nfcore-bucket/data/fastq/ --recursive

# Upload reference genome
aws s3 cp data/human_genome.fa.gz s3://my-nfcore-bucket/data/
aws s3 cp data/human_annotation.gtf.gz s3://my-nfcore-bucket/data/

Verify the upload:

aws s3 ls s3://my-nfcore-bucket/data/ --recursive --human-readable

Launching an EC2 instance

The EC2 instance runs Nextflow itself. It does not run the pipeline tools. Nextflow on this instance submits jobs to AWS Batch, monitors their progress, and collects results. A small instance is sufficient.

Choose your instance

Setting	Value
AMI	Amazon Linux 2023
Instance type	t3.medium (2 vCPU, 4 GB RAM)
Storage	30 GB gp3
Region	us-east-1

A t3.medium costs about $0.04 per hour. A typical RNA-seq pipeline run takes 1 to 2 hours, so the head node cost is under $0.10.

Launch from the console

1. Open the EC2 console.
2. Click Launch Instance.
3. Name your instance (e.g., nextflow-head).
4. Select Amazon Linux 2023 as the AMI.
5. Select t3.medium as the instance type.
6. Create a new key pair or select an existing one. Download the .pem file and keep it safe.
7. Under Network settings, allow SSH from your IP address.
8. Set storage to 30 GB gp3.
9. Click Launch Instance.

Connect via SSH

Once the instance is running, connect with SSH using the key pair you downloaded:

ssh -i ~/Downloads/my-key.pem ec2-user@<public-ip>

Replace <public-ip> with the public IPv4 address shown in the EC2 console.

Protect your key file

If SSH gives a “permissions too open” error, restrict the key file permissions:

chmod 400 ~/Downloads/my-key.pem

Assign an IAM role to the instance

The EC2 instance needs permission to interact with S3 and AWS Batch. Create an IAM role with the required policies and attach it to your instance.

1. Open the IAM console.
2. Create a new role with EC2 as the trusted entity.
3. Attach these managed policies:
   • AmazonS3FullAccess for reading and writing data
   • AWSBatchFullAccess for submitting and monitoring jobs
4. Name the role (e.g., nextflow-head-role).
5. In the EC2 console, select your instance, choose Actions > Security > Modify IAM role, and attach the role.

With the role attached, the AWS CLI on your instance can access S3 and Batch without storing credentials on disk.

Installing Nextflow on the EC2 instance

SSH into your instance and install Java and Nextflow:

# Install Java
sudo yum install -y java-21-amazon-corretto-headless

# Install Nextflow
curl -s https://get.nextflow.io | bash
sudo mv nextflow /usr/local/bin/

# Verify
nextflow -version

Install the AWS CLI

Amazon Linux 2023 comes with the AWS CLI pre-installed. Verify it:

aws --version

If it is missing, install it:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Test Nextflow locally on the instance

Run the nf-core/rnaseq test profile to confirm Nextflow is working:

nextflow run nf-core/rnaseq -r 3.14.0 -profile test,docker --outdir test_results

Note

Docker is pre-installed on Amazon Linux 2023. If you get a permission error, add your user to the docker group:

sudo usermod -aG docker ec2-user
# Log out and back in for the change to take effect

Keeping costs under control

EC2 instances are billed per second while they are running. When you are not using the head node, stop it from the EC2 console. A stopped instance costs nothing for compute. You only pay a small fee for the attached storage.

Do not terminate the instance unless you are done with it permanently. Terminating deletes the instance and its storage.

Set a billing alarm

In the AWS Billing console, create a billing alarm that notifies you if your monthly spend exceeds a threshold. A $10 alarm is a good starting point for pipeline testing.

Next steps

Your EC2 instance can run Nextflow locally with Docker, but this is limited to the 2 CPUs and 4 GB of RAM on a t3.medium. The next page covers setting up AWS Batch so Nextflow can distribute work across many machines automatically.