The Linux storage stack¶

When a program writes to /srv/data/report.csv, the bytes may pass through half a dozen pieces of software before they reach a disk: a filesystem, a logical volume, an encryption layer, a RAID array, a partition and the disk driver. Each piece presents something that looks like a disk to the piece above it and uses something that looks like a disk below it.

That one idea, block devices stacked on block devices, explains most of Linux storage administration. It's why you can grow a filesystem without touching the hardware, encrypt a RAID array as easily as a single disk, and replace a failed drive while the system keeps running. It's also why a storage problem can be hard to locate: there are many layers, and each one can fail, fill up or be configured wrongly.

This page describes the layers from the bottom up, then discusses why the design is layered and what that costs.

The layers at a glance¶

flowchart TB
    app["Applications<br/>open(), read(), write()"] --> vfs["VFS<br/>one directory tree, mount points"]
    vfs --> fs["Filesystem<br/>ext4, XFS, Btrfs, vfat"]
    fs --> lv["LVM logical volume<br/>(device-mapper)"]
    lv --> crypt["LUKS / dm-crypt<br/>(device-mapper)"]
    crypt --> md["Software RAID<br/>(md, managed with mdadm)"]
    md --> part1["GPT partition<br/>on disk 1"]
    md --> part2["GPT partition<br/>on disk 2"]
    part1 --> disk1["Block device<br/>/dev/nvme0n1"]
    part2 --> disk2["Block device<br/>/dev/nvme1n1"]

Not every system uses every layer, and the order in the middle can change: LUKS above or below LVM, RAID done by md or by LVM. A cloud virtual machine often has only a disk, one partition and a filesystem. What stays fixed is that the filesystem sits at the top of the block layers and the VFS sits above all filesystems.

lsblk shows the stack on a running system as a tree, with each device's children indented beneath it.

Block devices¶

A block device is the kernel's abstraction of storage that can be read and written in fixed-size blocks at any position. Physical disks appear under names that reflect their driver: /dev/sda for SCSI, SATA and USB disks, /dev/nvme0n1 for NVMe namespaces, /dev/vda for virtio disks in virtual machines. These names are assigned in the order devices are found, so they can change between boots when hardware is added, removed or detected in a different order. That is the reason for persistent names: udev creates symbolic links under /dev/disk/by-id/, by-uuid/, by-label/, by-partuuid/ and by-path/ from properties that don't depend on detection order.

Every layer described below creates a new block device. To the kernel's block layer, a partition, an encrypted mapping or a logical volume is just another device to which it can send read and write requests. The block layer queues those requests, merges adjacent ones, and passes them to the driver through an I/O scheduler (often none on fast NVMe devices, where the device does its own scheduling, and mq-deadline or bfq on slower ones).

Devices also report their sector sizes. Many modern disks have 4096-byte physical sectors but accept 512-byte requests for compatibility; each layer above should be aligned to the physical size, or a single logical write can turn into a read-modify-write on the device. Current partitioning tools and LVM align to 1 MiB by default, which satisfies every common sector and RAID chunk size.

Partitions and GPT¶

A partition table divides a disk into regions, each of which becomes its own block device (/dev/sda1, /dev/nvme0n1p1). Partitioning is the oldest layer, and also the least flexible one: partitions are contiguous ranges of the disk, so growing one usually requires free space directly after it.

Two formats are in use:

MBR (the DOS partition table) dates from the early PC. It allows four primary partitions (more through an "extended" partition) and, with 512-byte sectors, can't address beyond 2 TiB.
GPT (the GUID Partition Table), defined in the UEFI specification, is the modern default. It stores 128 partition entries by default, uses 64-bit addresses, gives each partition a type GUID, a unique GUID (the PARTUUID) and an optional name (the PARTLABEL), and keeps a backup copy of its header and table at the end of the disk. A "protective MBR" in the first sector stops older tools from treating the disk as empty.

UEFI firmware reads GPT directly and loads the boot loader from the EFI system partition, a small FAT filesystem. That makes partitioning the one layer most systems still need even when the rest of the disk is managed by LVM or used whole. For disks that hold only data, many administrators skip partitioning and hand the whole disk to LVM or md. This works, but other operating systems and tools may then see the disk as unused.

Device-mapper¶

Device-mapper is the kernel framework that most of the remaining layers are built on. It creates virtual block devices (/dev/dm-0, /dev/dm-1, … with friendlier names under /dev/mapper/) whose contents are defined by a mapping table: a list of ranges, each handled by a target that sends I/O for that range somewhere else.

Some of the targets are:

Target	What it does
`linear`	Maps a range onto a range of another device. The building block of ordinary logical volumes.
`striped`	Spreads consecutive chunks across several devices (RAID 0).
`crypt`	Encrypts on write and decrypts on read (dm-crypt).
`thin-pool`, `thin`	Allocates space on demand, allowing over-provisioning and cheap snapshots.
`snapshot`, `snapshot-origin`	Classic copy-on-write snapshots.
`raid`	RAID 0, 1, 4, 5, 6 and 10, reusing the kernel's md RAID code.
`cache`, `writecache`	Places a fast device in front of a slow one.

The kernel knows nothing about "volume groups" or "LUKS headers". It only executes tables. Userspace tools such as LVM and cryptsetup read their own on-disk metadata, compute tables from it, and load them into the kernel. That split keeps the kernel side small and general, and means a single mechanism provides logical volumes, encryption, thin provisioning, caching and multipath. dmsetup table shows the raw tables, which is occasionally the clearest way to see what a higher-level tool actually built.

LVM¶

The Logical Volume Manager (LVM2) is the most common user of device-mapper. It replaces fixed partitions with pools of storage that can be divided and resized freely, using three concepts:

A physical volume (PV) is a block device (a whole disk, a partition, a RAID array or an unlocked LUKS device) that has been initialised with LVM metadata.
A volume group (VG) pools one or more PVs and divides their space into physical extents, 4 MiB each by default.
A logical volume (LV) is a block device made from extents allocated from the VG. They don't need to be contiguous or even on the same PV, because device-mapper's linear target can stitch any set of ranges together.

LVM stores its metadata as text, in a small area near the start of every PV. Each PV carries a description of the whole VG, so the group can be reassembled from its disks alone. Logical volumes appear as /dev/<VG>/<LV> and /dev/mapper/<VG>-<LV>, names that come from the metadata and so stay stable across reboots.

Because an LV is only a list of extents, LVM can do things that are hard with partitions:

Grow a volume by appending extents from anywhere in the group, and grow the group by adding another PV. How to extend an LVM logical volume online shows this in practice.
Move data between PVs while volumes stay in use (pvmove), which is how you retire an old disk without downtime.
Snapshot a volume, either with classic copy-on-write snapshots or, more efficiently, with thin provisioning.
Mirror, stripe or use RAID per volume, and add a cache device in front of a slow one.

LVM operates only on block devices. Growing an LV doesn't grow the filesystem inside it, which is why the filesystem has to be resized as a separate step (or by lvextend --resizefs, which runs the resize tool of the filesystem for you).

Software RAID with md¶

Linux has two RAID implementations. The older and more widely used one is the md (multiple devices) driver, managed with mdadm. It is separate from device-mapper and creates devices named /dev/md0 or /dev/md/<name>. md supports RAID 0, 1, 4, 5, 6 and 10, records its own metadata (a superblock on each member device), and shows the state of every array, including rebuild progress, in /proc/mdstat.

LVM can also create RAID logical volumes through device-mapper's raid target, which uses the md code internally. Performance is similar, because the RAID code is shared; the approaches differ in how arrays are described and managed. With md, one array (built from whole disks or partitions) gives the same RAID level to everything stored on it, and its tools are familiar to most administrators and recovery software; with LVM RAID, you choose the RAID level for each logical volume within one group.

Whichever you use, RAID protects against device failure, not against deletion, corruption written by software, or the loss of the whole machine. It is not a substitute for backups. RAID 5 and 6 also have a well-known weakness: after a crash during a write, data and parity can disagree for the stripe being written (the "write hole"), which md addresses with an optional journal or partial-parity log.

Encryption with LUKS¶

dm-crypt is the device-mapper crypt target: it encrypts every block written to the device below and decrypts every block read. LUKS (Linux Unified Key Setup) is the on-disk format that cryptsetup uses to manage it. A LUKS header at the start of the device stores the cipher settings and several key slots. Each slot holds a copy of the single volume key, encrypted with a different passphrase or key file. That is why you can add or revoke a passphrase without re-encrypting the data: only a key slot changes. LUKS2, the default format in current cryptsetup releases, adds a redundant JSON-based header and support for unlocking with tokens such as TPM2 chips and FIDO2 keys.

Once unlocked, the plaintext view appears as /dev/mapper/<name> and is used like any other block device. Placement in the stack matters:

LUKS below LVM (one encrypted PV, many LVs inside) needs only one passphrase at boot and hides even the layout of the volumes.
LUKS above LVM (an encrypted LV) allows different keys per volume, and unencrypted volumes alongside encrypted ones.
LUKS above md RAID encrypts once, so the CPU cost isn't repeated for every mirror copy.

Encryption at this level protects data at rest, on a powered-off or stolen disk. It does nothing once the device is unlocked: to every process on the running system, the data is plaintext.

Filesystems¶

A filesystem turns a flat range of blocks into files and directories, with names, permissions, timestamps and free-space management. It is the topmost block layer, and the first one that knows about files. The common choices on the distributions covered here are:

ext4, the long-standing default on Debian and Ubuntu: mature, with well-understood repair tools. It can be grown online and shrunk offline.
XFS, the default on the RHEL family: designed for large files, large filesystems and parallel I/O. It can be grown online but, in practice, not shrunk: since Linux 5.12 only unused space at the very end of the filesystem can be removed.
Btrfs, a copy-on-write filesystem with checksums, snapshots, compression and its own multi-device and RAID support.
vfat, needed for the EFI system partition and for exchanging data with other operating systems.

Btrfs (and ZFS, which is maintained outside the kernel) deliberately collapse several layers into one. Because the filesystem itself manages multiple devices, redundancy and snapshots, it knows which blocks are in use and can checksum data end to end. It can therefore rebuild only live data after a disk failure, and repair a bad block from a good copy. A filesystem on top of md and LVM can't do either, because the lower layers know nothing about files. The price is that one complex component now does everything, with its own tools and failure modes, instead of several simple ones you can reason about separately.

The VFS and mounting¶

Above all filesystems sits the kernel's virtual filesystem switch (VFS). It gives every filesystem the same interface (open, read, write, stat and the rest) and keeps the shared structures: inodes for files, dentries for directory entries, and the page cache that keeps recently used file data in memory. Applications never talk to ext4 or XFS directly; they talk to the VFS, which calls the implementation in the filesystem.

Unlike systems that give each volume a drive letter, Linux presents one directory tree. Mounting attaches the root directory of a filesystem to a directory in that tree, the mount point, and from then on the VFS sends lookups below that point to the mounted filesystem. The same mechanism handles filesystems that have no block device at all: proc, sysfs, tmpfs, and network filesystems such as NFS.

Which filesystems are mounted where at boot is described in /etc/fstab; the /etc/fstab reference covers its format. On systemd distributions, each entry becomes a mount unit with dependencies on its device. That is how the boot process waits for an LVM volume or an unlocked LUKS device before mounting what's on it. On Gentoo with OpenRC, the swap, localmount and netmount services enable and mount the entries instead, with swapon -a and mount -a, in file order.

Why layering works¶

The stack's strength is that each layer has one job and a common interface (a block device) above and below it. That has several consequences:

Composability. Any layer can be placed on any other. Encrypted RAID, RAID of encrypted disks, LVM on either, a filesystem on any of them: the combinations come from the interface, not from features someone had to add for each pairing.
Independent change. You can replace a disk under md, move a PV to new hardware with pvmove, or change a LUKS passphrase without the layers above noticing.
Separation of concerns. Each tool is narrow, so each is easier to understand and test. mdadm knows nothing about encryption; cryptsetup knows nothing about volume groups.
Late decisions. With LVM, you don't need to decide at installation time how big /var should be. Free extents in the group are capacity you assign when you know what needs it.

What each layer costs¶

Layering isn't free. Each layer adds something to understand, to monitor and to recover:

Layer	Main cost
Partitions	Rigid: growing a partition usually means moving the ones after it.
Device-mapper (generally)	Another level of names (`dm-3`) to map back to something meaningful; small CPU overhead for simple targets.
LVM	Metadata that must be intact to find your data; snapshots that slow writes and can fill up and become invalid; one more layer to activate at boot.
md RAID	Rebuilds that take hours on large disks and stress the remaining members; the write penalty and write hole of parity RAID.
LUKS	CPU cost for every block (small on processors with AES instructions); a passphrase or key needed at boot; data lost for good if the header and all keys are lost.
Filesystem	Each has its own limits (XFS can't usefully shrink), repair tools and tuning.

Some costs come from the layers interacting:

Boot complexity. Every layer must be assembled, in the right order, before the root filesystem can be mounted. That is the job of the initramfs, and why changing the stack under / sometimes means regenerating it.
Information doesn't flow upward. A filesystem on LVM on md doesn't know which blocks are mirrored, and md doesn't know which blocks hold live files, so it resynchronises unused space too. This is exactly the gap Btrfs and ZFS close.
Discard and flush propagation. TRIM requests and write barriers have to pass through every layer to reach the device. Most layers pass them through; dm-crypt doesn't pass discards by default, because they reveal which blocks are unused, and has to be told to.
Harder troubleshooting. "The disk is full" can mean the filesystem is full, the volume group has no free extents, or a thin pool has run out of space underneath an apparently half-empty filesystem. Errors at the bottom of the stack can surface as strange behaviour at the top.

The usual result is a compromise: use the layers that solve a problem you actually have. A small VM with one virtual disk gets little from LVM or RAID, because its hypervisor already provides resizing, snapshots and redundancy. A physical server with several disks and changing space requirements gains a lot from md or LVM RAID, LUKS where data at rest needs protecting, and LVM for flexible volumes.