Pipes & Redirection

Every Linux command communicates through three channels called standard streams. Learning how to redirect these streams is one of the most powerful skills in the command line. It lets you chain simple commands into complex pipelines without writing a single script.

Standard streams

Every command you run automatically gets three streams:

Stream	Name	Number	Default destination
stdin	Standard input	0	Keyboard
stdout	Standard output	1	Terminal screen
stderr	Standard error	2	Terminal screen

stdin is where a command reads its input. By default, that is your keyboard.

stdout is where a command sends its normal output. By default, it prints to your terminal.

stderr is where a command sends error messages. It also prints to your terminal by default, but it is a separate channel from stdout.

This separation matters. It means you can capture normal output in one place and error messages in another.

Output redirection with > and >>

The > operator sends stdout to a file instead of the terminal. If the file exists, it gets overwritten.

$ ls *.fastq.gz > fastq_files.txt
$ cat fastq_files.txt
sample_01.fastq.gz
sample_02.fastq.gz
sample_03.fastq.gz

This is useful when you want to save a list of files for later processing.

Caution

The > operator overwrites the file without warning. Double check the file name before running the command.

The >> operator appends to the end of a file instead of overwriting it.

$ echo "Run started: $(date)" >> pipeline.log
$ echo "Run started: $(date)" >> pipeline.log
$ cat pipeline.log
Run started: Fri Mar 28 10:00:00 EDT 2026
Run started: Fri Mar 28 10:05:00 EDT 2026

Use >> when you want to add to a log file without losing previous entries.

Input redirection with <

The < operator feeds a file into a command as stdin.

$ wc -l < gene_list.txt
1542

Here, wc -l reads from gene_list.txt instead of waiting for keyboard input. The result is the same as wc -l gene_list.txt, but the mechanism is different. With <, the shell opens the file and passes it to the command. The command itself never sees a file name.

This distinction matters when a command does not accept file names as arguments.

Pipes

The pipe operator | connects the stdout of one command to the stdin of the next. This is the core of the Linux philosophy: small tools chained together.

A simple example

$ ls /data/fastq/ | head -5
sample_01_R1.fastq.gz
sample_01_R2.fastq.gz
sample_02_R1.fastq.gz
sample_02_R2.fastq.gz
sample_03_R1.fastq.gz

The ls command lists all files. The head -5 command takes only the first five lines. The pipe connects them.

Counting unique genes

Suppose you have a file called genes.txt with one gene name per line. Some genes appear more than once. You want to count the unique entries.

$ cat genes.txt | sort | uniq | wc -l
347

Here is what each step does:

1. cat genes.txt reads the file and sends it to stdout.
2. sort alphabetizes the lines. This is required because uniq only removes adjacent duplicates.
3. uniq removes consecutive duplicate lines.
4. wc -l counts the remaining lines.

The result is the number of unique genes in your file.

Building longer pipelines

You can chain as many commands as you need. Each | passes output to the next command.

$ grep ">" reference.fasta | cut -d " " -f1 | sed 's/>//' | sort > chromosome_names.txt

This pipeline extracts chromosome names from a FASTA header. It grabs header lines, isolates the first field, removes the > character, sorts the names, and saves them to a file.

Redirecting stderr

Sometimes a command produces both output and errors. You may want to handle them separately.

Send errors to a file

The 2> operator redirects stderr to a file.

$ samtools view input.bam 2> alignment_errors.log

Normal output still goes to your terminal. Errors go to alignment_errors.log.

Discard errors

If you want to ignore errors entirely, send them to /dev/null. This is a special file that discards everything written to it.

$ find /data/sequencing/ -name "*.fastq.gz" 2>/dev/null

This suppresses “Permission denied” errors from directories you cannot access.

Merge stderr into stdout

The 2>&1 operator sends stderr to the same place as stdout. This is useful when you want to capture everything in one file.

$ bash run_pipeline.sh > pipeline.log 2>&1

Both normal output and error messages end up in pipeline.log. The order matters here. The > redirect must come before 2>&1.

Note

A shorter way to write this is &>. It redirects both stdout and stderr to a file in one step.

$ bash run_pipeline.sh &> pipeline.log

Practical examples

Count reads in a FASTQ file

A FASTQ file uses exactly four lines per read. Count the lines and divide by four.

$ cat sample.fastq | wc -l
2000000

That means the file contains 500,000 reads. For compressed files, use zcat instead of cat.

$ zcat sample.fastq.gz | wc -l
2000000

Extract a column and count unique values

Suppose results.tsv has gene names in the second column. You want to see which genes appear most often.

$ cut -f2 results.tsv | sort | uniq -c | sort -rn | head
    152 TP53
     98 BRCA1
     76 EGFR
     54 KRAS
     41 MYC

Here is the breakdown:

1. cut -f2 extracts the second column.
2. sort groups identical names together.
3. uniq -c counts how many times each name appears.
4. sort -rn sorts numerically in reverse order, putting the highest counts first.
5. head shows the top 10 results.

Chain samtools commands

You can pipe BAM data directly between samtools subcommands. This avoids writing intermediate files.

$ samtools view -b input.bam chr1 | samtools sort -o chr1_sorted.bam

This extracts reads from chromosome 1 and sorts them in one step. No temporary file is needed between the two commands.

Filter and count aligned reads

$ samtools view -F 4 sample.bam | cut -f3 | sort | uniq -c | sort -rn | head
 125000 chr1
 118000 chr2
 112000 chr3

The -F 4 flag excludes unmapped reads. The rest of the pipeline counts how many reads aligned to each chromosome.

The tee command

Sometimes you want to save output to a file and see it on screen at the same time. The tee command does exactly this.

$ cat gene_counts.txt | sort -rn | tee sorted_counts.txt | head

The output flows through tee, which writes it to sorted_counts.txt. At the same time, it passes the data along the pipe to head. You get both a saved file and terminal output.

Use tee -a to append instead of overwrite.

$ echo "Step 3 complete" | tee -a pipeline.log
Step 3 complete

The message prints to your terminal and gets appended to the log file.

tee is especially useful in long pipelines when you want to inspect intermediate results.

$ cut -f1 annotations.gff | sort | uniq -c | sort -rn | tee feature_summary.txt

This saves the full summary to a file while displaying it on screen.

Chaining commands

Pipes connect the output of one command to the input of another. But sometimes you want to run commands one after another without piping data between them. Linux offers several ways to do this.

Run sequentially with &&

The && operator runs the next command only if the previous one succeeded.

# Update package list, then install samtools
$ sudo apt update && sudo apt install -y samtools

# Download data, then run quality control
$ wget https://example.com/reads.fastq.gz && fastqc reads.fastq.gz

If the first command fails, the second one does not run. This prevents wasting time on steps that depend on earlier ones.

Run regardless with ;

The ; operator runs the next command no matter what. Even if the first command fails, the second one still runs.

# Run both commands even if the first fails
$ samtools index input.bam ; echo "Done"

Use ; when the commands are independent. Use && when the second command depends on the first.

Run on failure with ||

The || operator runs the next command only if the previous one failed.

# Try to create a directory, print a message if it already exists
$ mkdir results || echo "results directory already exists"

Combining operators

You can combine these in a single line.

# Download file, process if successful, print error if not
$ wget https://example.com/data.csv && Rscript analyze.R || echo "Something failed"

Running in the background with &

Add & at the end of a command to run it in the background. Your terminal is free immediately.

# Run FastQC in the background
$ fastqc *.fastq.gz &

# Check background jobs
$ jobs

Quick reference

Syntax	Description
command > file	Redirect stdout to file, overwriting it
command >> file	Redirect stdout to file, appending to it
command < file	Redirect file to stdin
command1 | command2	Pipe stdout of command1 into stdin of command2
command 2> file	Redirect stderr to file
command 2>> file	Append stderr to file
command > file 2>&1	Redirect both stdout and stderr to file
command &> file	Shorthand for redirecting both stdout and stderr
command 2>/dev/null	Discard stderr
command | tee file	Write to file and stdout at the same time
command | tee -a file	Append to file and stdout at the same time
cmd1 && cmd2	Run cmd2 only if cmd1 succeeds
cmd1 ; cmd2	Run cmd2 regardless of cmd1 result
cmd1 || cmd2	Run cmd2 only if cmd1 fails
command &	Run command in the background

Next steps

Now that you can chain commands and redirect output, you are ready to learn the text processing tools that make pipelines truly powerful. Continue to Text Processing.