Skip to main content

Introduction

From Reads to Variants moved data through three file formats, FASTQ for reads, BAM for reads placed on the reference and VCF for variant calls, and explained just enough of each to follow along. This chapter covers them in full, along with the formats that list regions and describe genes, and the compression that most of them share.

Genomic file formats are simpler than they first look. Most are plain text. SAM, VCF, BED and GTF/GFF3 put one record on each line, with its fields separated by tabs, and FASTQ spends four lines on each read. BAM, CRAM and BCF store the same records in a compressed, binary form, and tools such as samtools and bcftools turn them back into text. So you can look inside nearly any genomic file, and do a lot of work with it, using ordinary command-line tools such as grep, cut, sort and awk.

Sandbox.bio has some great tutorials to teach you these things. Terminal Basics covers grep and pipes, Terminal Exercises practices cut and sort, and Data Exploration with awk covers awk.

Try It Yourself​

Each page ends with a Try It section that runs real commands on real or simulated data. Like From Reads to Variants, the exercises need only a terminal on macOS or Linux.

In This Chapter​

  • Compression and BGZF: how genomic files are compressed, and how an index lets tools jump straight to one region.
  • FASTQ Files: reads and their quality scores, as they come off the sequencer.
  • SAM/BAM/CRAM Files: reads placed on the reference, stored as text, as compressed binary, or as differences from the reference.
  • VCF/BCF Files: variant calls, and each sample's genotype at every one.
  • BED Files: lists of regions, and the two ways of counting positions.
  • GTF/GFF3 Files: where the genes are, and how each one is built.
  • Conclusion: how the formats fit together, and tutorials for more practice.