Lab environments¶

Every command in a tutorial or how-to guide is run in a disposable virtual machine (a lab) before the page is published. This page describes the parts that make that work, how to build and control them, and what to do when something goes wrong. For writing verifiable pages, see Command verification; the design is recorded in ADR-009, ADR-010, ADR-016, ADR-017 and ADR-018 (docs/plan/adr/).

Requirements¶

• A Linux host with KVM: your user can open /dev/kvm (usually through the kvm group).
• QEMU (qemu-system-x86_64 and qemu-img), xorriso, and OpenSSH (ssh, ssh-keygen).
• UEFI firmware (OVMF) for the Gentoo lab: /usr/share/OVMF/OVMF_CODE_4M.fd and OVMF_VARS_4M.fd, or set MAGELLANIC_OVMF_CODE and MAGELLANIC_OVMF_VARS.
• Optional: Docker with the Compose plugin, for the package cache.
• Disk space: about 5 GB for the official images, 4 GB per baked Gentoo image and its cached first stage, and up to 40 GB for the package cache.
• Memory: 2 GiB per lab, 6 GiB for a Gentoo build.

make install sets up everything else (Python toolchain, hooks).

Components¶

How a lab starts¶

1. Image. lab.py uses the frozen image if images are frozen. Otherwise it looks up the current official image, downloads it once and verifies it against the publisher's checksum. For Gentoo it builds the lab image from the official one (see Building the Gentoo image).
2. Acceptance. Before an image is first used, and whenever its checks change, it is booted once and its acceptance checks run. An image that fails isn't used.
3. Slot. The lab waits for one of max_vms slots, shared by every process on the host.
4. Boot. A copy-on-write overlay on the image boots under QEMU/KVM with a cloud-init seed: a user, labuser, with passwordless sudo and a one-time SSH key. The first NIC reaches the internet through QEMU's user-mode network, IPv4 only (its IPv6 has no route out and only slows guests down). Extra NICs sit on an isolated segment and stay unconfigured: on NetworkManager images (network_manager = true), /etc/NetworkManager/conf.d/90-magellanic-lab.conf turns off its automatic DHCP profiles, written early in boot from a systemd credential. Extra disks are blank.
5. Wait. lab.py waits for SSH and cloud-init, watching the serial console for fatal errors. The limit is three times the image's measured boot time (at least two minutes).
6. Package cache. If the cache is running and the page allows it, the lab's package sources are pointed at it.
7. Run. Commands run over SSH, as labuser or as root. Each block's script and output are logged.
8. Clean-up. The lab and its files are deleted, unless you asked to keep them or it failed to boot.

Build¶

Official images¶

scripts/uv run python scripts/lab.py fetch                # download and verify every image
scripts/uv run python scripts/lab.py fetch debian-13      # one image
make lab-accept                                           # run the acceptance checks on every image

New official images are picked up automatically whenever a lab starts (unless images are frozen). Arch and Gentoo are rolling: their identifiers carry a snapshot date, such as arch@2026-11-01.

Building the Gentoo image¶

Gentoo publishes no OpenRC cloud image, so the project builds one (ADR-016) in two stages (ADR-017):

1. Stage 1 (bake-1-world.sh, the slow part: about 10 minutes) runs as root in a builder VM booted from the official image. It syncs the Portage tree (its date becomes the image's snapshot date), switches to the OpenRC profile, rebuilds @world and removes systemd. The builder is powered off through SysRq (the old init system is gone), and the filesystem is repaired offline in a Debian helper lab. The result is cached.
2. Stage 2 (bake-2-configure.sh, a few minutes) runs in a chroot of the cached stage 1 image from a Debian helper lab: runlevels, networking, the initramfs and GRUB, and per-instance state.
3. Settle boot. The result boots once under OpenRC and shuts down cleanly, so every lab starts from a clean filesystem.
4. Acceptance. labs/gentoo-openrc/accept.sh proves PID 1 is OpenRC's init, no systemd remains in the initramfs or cgroups, DNS works, the filesystem is clean, and the kernel tools are installed. A failing image is renamed ….rejected.qcow2 and not used.

A build starts automatically when a Gentoo lab is needed and no current image exists. To build ahead of time:

scripts/uv run python scripts/lab.py fetch gentoo-openrc

Caching is by content: stage 1 is keyed by the official image, the stage 1 recipe and BAKE_VERSION (in scripts/lab.py); the final image also by stage 2. So:

• changing bake-2-configure.sh rebuilds only stage 2;
• changing bake-1-world.sh, a new official image, or a new BAKE_VERSION rebuilds both;
• bump BAKE_VERSION when you change how lab.py builds images, not when you change a recipe.

Only one build or acceptance run per distribution happens at a time; other processes wait for it.

Writing acceptance checks¶

Write the check before the change it proves. Every check must be able to fail:

• assert that the tool exists and that its output isn't empty before interpreting it;
• capture output in a variable, then match it (never command | grep -q under pipefail);
• print PASS … or FAIL … for each check and exit non-zero on any failure (WARN … is reported but doesn't fail);
• try a new check once against a broken image or input, and confirm it fails.

labs/common/accept.sh applies to every image; add labs/<distro>/accept.sh for checks one image needs. Changing either re-runs acceptance on the next use.

Control¶

Everyday commands¶

Freezing images for a wave¶

make lab-freeze       # pin the current, accepted images; record the output in the wave manifest
make lab-unfreeze     # release the freeze between waves

While images are frozen, labs never download or build a newer image, so results stay comparable across a wave. lab.py list marks frozen images. The freeze is stored on your machine (freeze.json); record what you froze in docs/plan/phase-3-manifest.md.

Capacity¶

The number of labs that may run at once on a host is [lab] max_vms in labs/images.toml. Measure it rather than guessing:

make lab-loadtest     # boots 1, 2, 4, 6… labs with a fixed workload; writes the result to labs/images.toml

It recommends the largest count whose boot and workload times stay within 25 % of a single lab's, capped so a fifth of available memory stays free. Stop other lab work first. MAGELLANIC_LAB_MAX_VMS overrides the setting for one shell.

The package cache¶

make lab-cache-up        # start it (127.0.0.1:3142 on the host)
make lab-cache-status    # health, size, and cache hits versus misses since it started
make lab-cache-down      # stop it, keeping its contents
make lab-cache-purge     # stop it and delete its contents

While it runs, each lab's package sources (APT, dnf, pacman and Portage) point at it (labs reach the host at 10.0.2.2), so each package is downloaded once. Index files are always fetched fresh; packages are kept for 60 days, up to 40 GB. Package managers still verify every signature. Inside a lab, /etc/magellanic-lab-cache lists the files that were changed, and the originals are under /var/lib/magellanic-lab-cache/orig. Image builds, settle boots and acceptance runs never use the cache.

Pages that show or change repository configuration opt out with <!-- verify: no-cache -->. To add a mirror, add a location to labs/cache/nginx.conf and a matching rule to labs/cache/guest.sh.

Environment variables¶

Files on your machine¶

All under ~/.cache/magellanic/labs/ (or MAGELLANIC_LAB_CACHE):

Run directories are deleted when a lab stops, unless you passed --keep or the lab failed to boot.

Troubleshoot¶

Start with the message lab.py prints: it names the failure and the log to read.

Stray labs and leftovers¶

A lab stopped abruptly (a killed terminal, a crashed process) can leave a QEMU process and a run directory behind. Find lab VMs by their run directory, and stop only those, never every QEMU process on the host:

pgrep -af 'magellanic/labs/run'             # lab VMs still running
pkill -f 'magellanic/labs/run/<distro>-'    # stop one lab VM
ls ~/.cache/magellanic/labs/run/            # leftover run directories

Delete a leftover run directory once no process uses it. Old images, rejected images and logs can be deleted at any time; lab.py downloads or rebuilds what it needs.

Keeping a lab for inspection¶

scripts/uv run python scripts/verify_commands.py PAGE --distro rocky-10 --keep

The run directory is kept and its path is printed. A lab that failed to boot is always kept.