Text Processing

Bioinformatics means working with text files. Genome annotations, gene lists, alignment results, and variant calls all come as plain text. Linux provides a powerful set of tools for searching, filtering, and transforming these files right from the command line.

This page covers the essential text processing commands. You will use these daily when working with sequencing data.

grep: Search for patterns

grep searches for text patterns inside files. It prints every line that matches your pattern.

Find a gene in a GTF file

GTF files contain genome annotations. Each line describes a feature like a gene, transcript, or exon. To find all lines mentioning the gene TP53:

$ grep "TP53" genes.gtf
chr17   HAVANA  gene    7668402 7687550 .   -   .   gene_name "TP53";
chr17   HAVANA  transcript  7668402 7687550 .   -   .   gene_name "TP53";
chr17   HAVANA  exon    7687377 7687550 .   -   .   gene_name "TP53";

Find header lines in a FASTA file

FASTA header lines start with >. You can search for that character:

$ grep ">" sequences.fasta
>NM_000546.6 Homo sapiens tumor protein p53 (TP53)
>NM_007294.4 Homo sapiens BRCA1 DNA repair associated (BRCA1)
>NM_000059.4 Homo sapiens BRCA2 DNA repair associated (BRCA2)

Useful grep flags

Case insensitive search with -i:

$ grep -i "brca1" genes.gtf
chr17   HAVANA  gene    43044295    43170327    .   -   .   gene_name "BRCA1";

Count matches with -c:

$ grep -c "exon" genes.gtf
48293

Invert the match with -v:

This prints lines that do not match. It is useful for removing comment lines:

$ grep -v "^#" annotations.gff > annotations_no_comments.gff

Show line numbers with -n:

$ grep -n "KRAS" genes.gtf
1042:chr12  HAVANA  gene    25205246    25250929    .   -   .   gene_name "KRAS";

Search all files in a directory with -r:

$ grep -r "ERROR" logs/
logs/alignment.log:ERROR: index file not found
logs/variant_call.log:ERROR: invalid read group

cut: Extract columns

Many bioinformatics files are tabular. Columns are separated by tabs or other delimiters. cut extracts specific columns from these files.

Extract gene names from a TSV

Suppose you have a file called results.tsv with columns: gene, chromosome, start, end, p-value. To extract only the gene name column:

$ cut -f 1 results.tsv
gene
TP53
BRCA1
EGFR
KRAS

The -f flag selects fields by number. Fields are numbered starting from 1.

Extract multiple columns

You can select several columns at once:

$ cut -f 1,5 results.tsv
gene    pvalue
TP53    0.001
BRCA1   0.023
EGFR    0.870
KRAS    0.004

Change the delimiter

By default, cut splits on tabs. Use -d to specify a different delimiter. For a CSV file:

$ cut -d ',' -f 2 samples.csv
sample_name
tumor_01
normal_01
tumor_02

sort: Sort lines

sort arranges lines in order. By default it sorts alphabetically.

Sort a gene list

$ sort gene_list.txt
BRCA1
BRCA2
EGFR
KRAS
TP53

Sort numerically

Alphabetical sorting treats numbers as text. The number 9 would come after 80 because “9” comes after “8”. Use -n to sort by numeric value:

$ sort -n counts.txt
3
12
45
108
1200

Sort in reverse order

$ sort -nr counts.txt
1200
108
45
12
3

Sort by a specific column

Use -k to pick the column and -t to set the delimiter. To sort a TSV by the p-value in column 5:

$ sort -t $'\t' -k 5 -n results.tsv
KRAS    chr12   25205246    25250929    0.001
TP53    chr17   7668402 7687550 0.003
BRCA1   chr17   43044295    43170327    0.023
EGFR    chr7    55019017    55211628    0.870

The -t $'\t' tells sort that the delimiter is a tab character.

uniq: Remove or count duplicates

uniq removes consecutive duplicate lines. This means you must sort the data first. Otherwise uniq will only remove duplicates that happen to be next to each other.

Count chromosomes in a BED file

BED files list genomic regions. The first column is the chromosome. To count how many regions are on each chromosome:

$ cut -f 1 regions.bed | sort | uniq -c
     12 chr1
      8 chr2
      5 chr3
     15 chr7
      3 chrX

The -c flag adds a count before each unique line.

Get a unique gene list

If your file contains repeated gene names, you can deduplicate it:

$ sort gene_list.txt | uniq
BRCA1
BRCA2
EGFR
KRAS
TP53

awk: Field-based processing

awk is a small programming language built for processing tabular data. It splits each line into fields automatically. Fields are accessed as $1, $2, $3, and so on. The entire line is $0.

Print specific columns

Print the first and third columns of a TSV:

$ awk '{print $1, $3}' results.tsv
gene start
TP53 7668402
BRCA1 43044295
EGFR 55019017

Filter rows by value

Print only rows where the p-value in column 5 is less than 0.05:

$ awk '$5 < 0.05' results.tsv
TP53    chr17   7668402 7687550 0.003
BRCA1   chr17   43044295    43170327    0.023
KRAS    chr12   25205246    25250929    0.001

Filter rows by text

Print only lines where column 2 equals “chr17”:

$ awk '$2 == "chr17"' results.tsv
TP53    chr17   7668402 7687550 0.003
BRCA1   chr17   43044295    43170327    0.023

Simple arithmetic

Calculate the mean of a column of numbers. This example computes the average p-value from column 5, skipping the header line:

$ awk 'NR > 1 {sum += $5; count++} END {print sum/count}' results.tsv
0.22425

NR is the current line number. NR > 1 skips the first line. The END block runs after all lines have been processed.

Set the field separator

By default, awk splits on whitespace. Use -F to set a different separator. For a CSV:

$ awk -F ',' '{print $1}' samples.csv
sample_id
1
2
3

sed: Stream editor

sed edits text as it flows through a pipeline. The most common use is find and replace.

Basic substitution

The pattern is s/old/new/g. The g at the end means replace all occurrences on each line, not just the first.

Rename chromosome labels from “chr1” format to just the number:

$ sed 's/^chr//' regions.bed
1   1000    2000
2   3000    4000
17  5000    6000
X   7000    8000

The ^ matches the start of the line. This ensures we only remove “chr” at the beginning.

Replace tabs with commas

Convert a TSV to CSV:

$ sed 's/\t/,/g' results.tsv > results.csv

Remove header lines

Delete the first line of a file:

$ sed '1d' results.tsv
TP53    chr17   7668402 7687550 0.003
BRCA1   chr17   43044295    43170327    0.023
EGFR    chr7    55019017    55211628    0.870

Delete all lines that start with #:

$ sed '/^#/d' annotations.gff

Edit a file in place

Use the -i flag to modify the file directly instead of printing to the terminal:

$ sed -i 's/^chr//' regions.bed

Be careful with -i. There is no undo. Make a backup first or use -i.bak to create one automatically:

$ sed -i.bak 's/^chr//' regions.bed

Combining tools with pipes

The real power comes from chaining these commands together. Each command does one small job. Pipes connect them into a workflow.

Count genes per chromosome from a GTF file

This pipeline extracts all gene names and counts how many appear on each chromosome:

$ grep "gene_name" genes.gtf | awk '$3 == "gene"' | cut -f 1 | sort | uniq -c | sort -rn
   2345 chr1
   1523 chr2
   1210 chr19
   1198 chr11
   1145 chr17
    356 chrX
     72 chrY

Here is what each step does:

1. grep "gene_name" keeps only lines that contain a gene name attribute.
2. awk '$3 == "gene"' filters for lines where the feature type is “gene”, removing exons and transcripts.
3. cut -f 1 extracts the chromosome column.
4. sort groups identical chromosome names together.
5. uniq -c counts how many times each chromosome appears.
6. sort -rn sorts the counts from highest to lowest.

Filter significant results and extract gene names

Find genes with a p-value below 0.01 and save them to a file:

$ awk 'NR > 1 && $5 < 0.01' results.tsv | cut -f 1 | sort > significant_genes.txt

Count unique genes in a GFF file

$ grep -v "^#" annotations.gff | awk -F '\t' '{print $9}' | grep -o 'gene_id "[^"]*"' | sort -u | wc -l
19438

Quick reference

Command	Purpose	Common flags
grep	Search for patterns	-i case insensitive, -c count, -v invert, -r recursive, -n line numbers
cut	Extract columns	-f fields, -d delimiter
sort	Sort lines	-n numeric, -r reverse, -k column, -t delimiter
uniq	Remove/count duplicates	-c count
awk	Process fields and filter rows	-F delimiter, $1 first field, NR line number
sed	Find and replace text	s/old/new/g substitute, -i in place, d delete
wc	Count lines, words, bytes	-l lines only

Next steps

These commands become even more useful inside scripts. Head to Shell Scripts to learn how to save and reuse your pipelines.