The Linux boot process¶

Booting is a relay race. Each stage is a small program whose only job is to find, load and start the next one, and then get out of the way. The firmware knows nothing about Linux; the bootloader knows nothing about your services; the kernel doesn't know what a web server is. Each stage knows just enough to hand over to the next one.

That design is why boot problems look so different from each other. A machine that stops at a firmware message, one that drops to a grub> prompt, one that panics with VFS: Unable to mount root fs, and one that reaches "You are in emergency mode" have failed at four different handovers. Recognising which handover failed tells you which part of the system to look at, and which tools you still have available to fix it.

The stages at a glance¶

flowchart TD
    fw["Firmware<br/>BIOS or UEFI"] -->|loads from disk or ESP| bl["Bootloader<br/>GRUB or systemd-boot"]
    bl -->|loads into memory, with a command line| k["Linux kernel"]
    bl -->|loads alongside| ird["initramfs"]
    k -->|unpacks and runs /init| ird
    ird -->|finds, checks and mounts the real root,<br/>then switch_root| init["PID 1 on the real root<br/>systemd, or sysvinit + OpenRC"]
    init -->|starts services in dependency order| tgt["Target or runlevel reached<br/>multi-user.target / default"]
    tgt --> login["Login prompt<br/>getty, sshd, display manager"]

Firmware: BIOS and UEFI¶

When the power comes on, the CPU runs code from the firmware. After testing and initialising the hardware, the firmware's remaining job is to find something to boot.

Legacy BIOS firmware reads the first 512-byte sector of a disk (the master boot record), checks for a boot signature, and jumps to the code it contains. That's only 446 bytes of code: far too small for a real bootloader, so it loads a larger second stage from somewhere else. On GPT-partitioned disks booted by BIOS, GRUB keeps that second stage in a small, unformatted "BIOS boot partition".

UEFI firmware is much more capable. It understands partition tables and the FAT filesystem, and it keeps a list of boot entries in non-volatile memory (NVRAM). Each entry points to an executable .efi file on the EFI system partition (ESP), a small FAT partition usually mounted at /boot/efi, /efi or, on systems using systemd-boot, often /boot itself. If no entry works, UEFI falls back to a default path on the ESP, \EFI\BOOT\BOOTX64.EFI on x86-64. You can list and change the NVRAM entries from a running Linux system with efibootmgr, which is why a UEFI system can lose its bootloader "entry" without losing the bootloader file itself.

Secure Boot, a UEFI feature, only runs executables signed with a trusted key. Distributions get around the fact that firmware usually only trusts Microsoft's key with shim, a small loader signed by Microsoft that in turn trusts the distribution's key and checks GRUB and the kernel. A Secure Boot failure happens before any Linux code runs, so the firmware's own error message is all you get.

Many current disk images carry both a BIOS boot partition and an ESP, so the same disk can start either way. You can tell which one booted a running system by checking whether /sys/firmware/efi exists.

The bootloader: GRUB and systemd-boot¶

The job of the bootloader is to load the kernel and the initramfs into memory and start the kernel with a command line: a string of parameters such as root=UUID=…, ro, quiet or console=ttyS0. The command line is the main way you can influence a boot before any of the installed system runs, which is why rescue procedures so often start at the bootloader menu.

GRUB 2 is the most common bootloader, and the one used by the reference images of every distribution family covered here (Arch and Gentoo leave the choice to the administrator; Arch's archinstall also offers systemd-boot on UEFI systems). It has its own drivers for filesystems, LVM, RAID and even LUKS encryption, so it can read /boot from almost anywhere. Its menu is generated: you edit /etc/default/grub and the scripts in /etc/grub.d/, then a generator (grub-mkconfig, wrapped as update-grub on Debian and Ubuntu, or grub2-mkconfig on the RHEL family) writes grub.cfg. On RHEL, AlmaLinux and Rocky Linux, GRUB also reads Boot Loader Specification entries, one small file per kernel in /boot/loader/entries/, managed with grubby. A grub> prompt means GRUB started but couldn't find or parse its configuration; grub rescue> means it couldn't even load its own modules, usually because the partition holding them moved or was damaged.

systemd-boot is a much simpler UEFI-only boot manager. It has no filesystem drivers of its own beyond what the firmware provides, so the kernels and initramfs images it starts must sit on the ESP (or an extended boot partition). Each menu entry is a Boot Loader Specification file in loader/entries/; there is no generated configuration. It's common on Arch Linux and an option on most other distributions, installed and managed with bootctl. Its simplicity means fewer things to break, at the price of fewer things it can do (no BIOS support, no reading kernels from LVM or encrypted volumes).

A third approach is the unified kernel image (UKI), which bundles kernel, initramfs and command line into one signed .efi file. The firmware can start it directly, without a bootloader, but in practice it's usually started from systemd-boot's menu. That fits well with Secure Boot, because the command line can no longer be changed without breaking the signature, and it changes how rescue works, since editing the command line at boot may no longer be possible.

The kernel and the initramfs¶

Once started, the kernel decompresses itself, sets up memory management and the scheduler, initialises the hardware it has built-in drivers for, and parses its command line. Then it has a problem: to start PID 1 it needs the root filesystem, but the root filesystem might be on a disk whose driver is a module, inside an LVM volume, on a RAID array, encrypted with LUKS, or on the network. All the tools and modules to deal with that live on the root filesystem.

The initramfs solves this chicken-and-egg problem. It's a compressed archive (cpio) loaded into memory by the bootloader alongside the kernel. The kernel unpacks it into a temporary in-memory root filesystem and runs its /init. That small early userspace loads the right storage drivers, assembles RAID and LVM, asks for the LUKS passphrase, checks and mounts the real root filesystem (usually read-only at first), and finally replaces itself with the real system using switch_root, which starts the real /sbin/init as PID 1.

Distributions build initramfs images with different tools, and that affects troubleshooting:

Because the initramfs is built from the installed system at a point in time, it can be out of step with the system: a new storage driver, a changed root UUID or a new LUKS layout often needs the image to be regenerated. A system that boots an old kernel but not a new one frequently has a broken or missing initramfs for the new kernel rather than a broken kernel.

Parameters starting with rd. are meant for the initramfs rather than the installed system. Some belong to systemd and work in any initramfs that runs systemd, including dracut images and mkinitcpio's systemd hook: rd.systemd.unit=, rd.luks.* and rd.udev.*, for example. Others are specific to dracut, such as rd.break and rd.shell, which ask the initramfs to stop and give you a shell at a particular point, typically just before switching to the real root. Debian's initramfs-tools has its own break= parameter for the same purpose.

PID 1 and the init system¶

After switch_root, the kernel's involvement in starting things is over. /sbin/init runs as process ID 1, and from now on everything that happens is userspace policy. What "init" means and how systemd and OpenRC divide its jobs is the subject of OpenRC and other init systems; here only its place in the boot matters.

The kernel lets you choose which program becomes PID 1 with init= on the command line. init=/bin/bash replaces the whole init system with a single shell: no services, no further mounts, often a read-only root. It's the bluntest rescue tool there is, useful precisely because it depends on almost nothing.

systemd: from default.target to a login¶

systemd starts by working out a goal: the unit named by systemd.unit= on the command line if there is one, otherwise default.target, which is a symlink to graphical.target or multi-user.target. It then computes everything that goal requires and wants, orders it by After= and Before=, and starts as much as it can in parallel.

The path to the goal passes through a few synchronisation points:

• sysinit.target: early setup such as mounting the local filesystems from /etc/fstab (via generated .mount units), activating swap, applying sysctl settings and starting udev.
• basic.target: sockets, timers and paths are ready.
• multi-user.target: a non-graphical system with all normal services, including getty login prompts and sshd.
• graphical.target: multi-user.target plus a display manager.

Two special targets exist for when things go wrong. rescue.target gets the basic system up (local filesystems mounted, but no network or normal services) and starts a root shell on the console. emergency.target does almost nothing: it starts a root shell before even mounting the local filesystems, with the root filesystem often still read-only. systemd also switches to emergency.target by itself when a unit the boot can't do without fails, most commonly a filesystem in /etc/fstab that can't be mounted or checked. Both shells need the root password, which matters on cloud images and systems where root has no password.

OpenRC: runlevels¶

On Gentoo's default OpenRC setup, PID 1 is sysvinit. It reads /etc/inittab, which tells it to run OpenRC for the sysinit runlevel, then boot, then the default runlevel (named default), before starting agetty login prompts on the consoles. Each runlevel is a directory in /etc/runlevels/; OpenRC starts the services in it in dependency order.

OpenRC's recovery options follow the same model. Passing single or S on the kernel command line makes sysvinit go to its single-user runlevel, S. Gentoo's /etc/inittab maps that runlevel to openrc single, which stops all but a minimal set of services, followed by sulogin, which provides the root shell. OpenRC can also be pointed at a different named runlevel at boot, and can run interactively, asking about each service before it starts.

Why the stage matters when troubleshooting¶

Each handover passes only a little information to the next stage, so each stage has its own tools, its own logs (or none), and its own way of failing. The practical questions are always the same: which stage was the last to succeed, and what does that stage give you to work with?

• Before the bootloader, nothing on the disk has run. You're limited to firmware settings and boot media; the problem is hardware, boot order, a missing NVRAM entry or Secure Boot.
• At the bootloader, you have the shell built into the bootloader and the ability to edit the kernel command line for one boot. That one-time edit is the gateway to every other rescue option, which is why a hidden or zero-second GRUB menu turns a small problem into a big one.
• In the kernel or initramfs, the installed system hasn't started yet. Its logs aren't being written to disk, and journalctl from a previous boot won't show what happened. You work from the messages on the console and the emergency shell in the initramfs, and fixes are typically to the root device, the kernel command line, or the initramfs itself.
• Under PID 1, the real system is running, at least partly. You have its tools, its logs (journalctl -b, or /var/log on OpenRC) and its service manager. Most problems here are configuration: a bad /etc/fstab line, a service that blocks boot, a broken unit or init script. The rescue and emergency targets, or single-user mode on OpenRC, exist to give you this stage with as little as possible running.
• After boot, if the machine is up but unreachable, the boot process itself is fine. Look at networking, sshd, firewalls and authentication instead.

The rescue modes form a ladder, each rung relying on less of the system than the one above:

How to use them is in How to boot into rescue mode.

Design decisions and trade-offs¶

The staged design is a consequence of history and of separation of concerns. Firmware has to work for every operating system, so it can't know about Linux filesystems. The kernel can't contain every driver for every way of storing a root filesystem without growing enormous, so the policy of how to find root moved out into the initramfs, where distributions can change it without rebuilding kernels. And the init system is userspace so that how services start is something administrators can change.

The price is that there are several independent places configuration can go wrong, each with its own regeneration step: the NVRAM entries, grub.cfg or the loader entries, the initramfs, /etc/fstab and the service configuration. Many "it won't boot after an update" problems are one of these generated artefacts being out of step with the rest.

The newer trend runs in the opposite direction: unified kernel images, measured boot with a TPM, and systemd running inside the initramfs make the early stages more uniform and more secure. They also make them less editable at boot time, which shifts recovery from "edit the command line" towards keeping a known-good boot entry or using external rescue media.

Further reading¶

• systemd.special(7): the special units used during boot, including rescue.target and emergency.target
• kernel-command-line(7) and the kernel's parameter documentation
• dracut.cmdline(7): the rd.* parameters understood by dracut
• GNU GRUB manual
• systemd-boot and the Boot Loader Specification
• UEFI specifications

Related¶

• Explanation: OpenRC and other init systems
• How-to: How to boot into rescue mode
• How-to: How to run a script as a service
• Reference: systemd unit directives