Load average and memory¶

Two numbers dominate first impressions of a Linux server's health: the load average in uptime and top, and the free memory in free. Both are routinely misread. A load of 12 can be harmless or a crisis depending on the machine and on what the tasks are waiting for, and a server showing almost no "free" memory is usually perfectly healthy.

This page explains what the kernel actually measures in each case, so that the numbers lead to the right questions. Finding the processes behind them is covered in How to find what is using CPU and memory.

Load average¶

What it counts¶

The load average is three numbers, for example load average: 3.12, 1.87, 0.95. They are averages, over roughly the last 1, 5 and 15 minutes, of the number of tasks that were either:

runnable: running on a CPU or waiting in a run queue for one (state R), or
in uninterruptible sleep: waiting inside the kernel in a section that signals can't interrupt, most often for disk or network-filesystem I/O (state D).

The kernel samples this count about every 5 seconds and folds it into three exponentially damped moving averages. "1-minute" is a time constant, not a window: a sudden burst raises the 1-minute figure within seconds and then decays smoothly, and the 15-minute figure moves slowly in both directions. Comparing the three gives the trend: 8.0, 4.0, 1.0 is a load that is rising now; 1.0, 4.0, 8.0 is one that is subsiding.

The same numbers are in /proc/loadavg, followed by two more fields:

0.53 0.12 0.04 1/120 1147

1/120 is the number of tasks runnable at this instant over the total number of tasks (threads included), and 1147 is the most recently allocated PID.

The counts include threads, not just processes, and they are system-wide, not per CPU. A load of 4.0 means something very different on a 2-CPU virtual machine (twice as much demand as capacity) and on a 64-CPU server (mostly idle). The rough reading is to compare the load with the number of CPUs (nproc), remembering that the D share in it isn't demand for CPU at all.

Why Linux counts uninterruptible sleep¶

Most Unix systems compute the load average from runnable tasks only, so on them it measures CPU demand. Linux has included uninterruptible tasks since 1993, on the reasoning that a task waiting for a disk is also waiting for the system to do work for it: the load average was meant to describe demand on the whole system, not only on the CPUs.

The consequence is that a high load on Linux has two possible causes, with very different remedies:

What you see	Likely meaning
High load, CPUs busy (`us`, `sy` high in `top`), many `R` tasks	CPU saturation: more work than CPUs
High load, CPUs mostly idle, many `D` tasks, high `wa` (I/O wait)	Tasks blocked on storage or a network filesystem
High load, CPUs idle, a few `D` tasks that never change	Tasks stuck on a hung device or an unreachable NFS server; the load stays high but nothing is slow except what touches that device

The last case is common enough to recognise on sight: a machine with a load of 20 that feels completely responsive, until you run ls on the wrong mount point. Listing tasks in D state, and the kernel function they're waiting in (the WCHAN column of ps), points to the device.

Some kernel threads sleep in this state without being blocked on anything that matters to users; the kernel puts its idle worker threads in a separate I state so they don't inflate the load.

Pressure stall information: a sharper tool¶

Because the load average mixes CPU and I/O and isn't normalised to the machine's size, newer kernels (4.20 and later) also report pressure stall information (PSI) in /proc/pressure/cpu, /proc/pressure/memory and /proc/pressure/io:

some avg10=0.00 avg60=0.00 avg300=0.00 total=366
full avg10=0.00 avg60=0.00 avg300=0.00 total=171

Each line gives the percentage of time, averaged over 10, 60 and 300 seconds, during which some task (or, for full, all non-idle tasks at once) was stalled waiting for that resource; total is the cumulative stall time in microseconds. Unlike load, PSI answers directly "how much is this resource holding work up?", separately for each resource, as a percentage independent of the number of CPUs. The same files exist per cgroup, as described in Control groups.

Memory¶

Where memory goes¶

The kernel uses physical memory for a few broad purposes:

Anonymous memory: the heap, stacks and private data of processes. It has no file behind it, so it can only be freed by the process or by writing it to swap.
Page cache: the contents of files that have been read or written, kept in memory so the next access is fast. This includes executables and shared libraries that are running. Clean cache pages can be dropped at any moment, because they are copies of what is on disk; dirty pages, written by programs but not yet written to disk, must be written back first.
Kernel memory: the kernel's own structures, much of it in the slab allocator. Part of it (SReclaimable, such as the cache of directory entries and inodes) can be freed under pressure; the rest (SUnreclaim) can't.
Shared memory and tmpfs (Shmem): looks like cache, but can't simply be dropped, because there is no disk copy.

The kernel's philosophy is that unused memory is wasted memory. After a while, almost all memory not used by processes fills with page cache, because keeping a file in memory costs nothing until the memory is needed for something else.

Reading `free`¶

               total        used        free      shared  buff/cache   available
Mem:           1.9Gi       217Mi       1.7Gi       516Ki       119Mi       1.7Gi
Swap:             0B          0B          0B

free is memory holding nothing at all. On a machine that has been up for a while it is usually small, and that is normal.
buff/cache is page cache, buffers and reclaimable slab.
available is the kernel's estimate (MemAvailable in /proc/meminfo, added in Linux 3.14) of how much memory could be given to new work without swapping: free memory plus the part of the cache and reclaimable slab that can actually be reclaimed. This is the number to watch.
used, in the procps-ng 4 versions on the distributions covered here, is total minus available. Older versions computed it differently, which is one reason figures from older documentation don't always add up.
shared is mostly tmpfs and shared memory segments; it is counted inside buff/cache as well.

A server with 2% free and 60% available is healthy. A server where available is falling steadily towards zero, or where it stays low while swap use and memory pressure grow, is running out of memory.

Swap¶

Swap is disk space where the kernel can put anonymous pages that haven't been used recently, freeing physical memory for something more useful, such as page cache for a busy database. A little swap in use is not a sign of trouble; it often means the kernel moved out pages that were touched once at start-up and never again. What matters is swap activity: pages being swapped in and out continuously (the si and so columns of vmstat), which is when performance collapses.

vm.swappiness (0 to 200, default 60) tells the kernel how to weigh reclaiming anonymous memory (swapping it) against reclaiming page cache, when it has to reclaim something. Lower values protect process memory at the expense of cache; higher values do the opposite. It isn't a threshold ("start swapping when memory is 60% full") despite being widely described as one. Values for each distribution are listed in Common sysctl parameters.

Swap can also be in memory: zram creates a compressed block device in RAM to swap to, and zswap compresses pages on their way to a real swap device. Both trade CPU time for effective memory. The cloud images used to verify this knowledge base are configured differently from installed systems here: the Debian, Ubuntu, AlmaLinux, Rocky Linux and Gentoo images have no swap at all, and the Arch image has a 512 MiB swap file. swapon --show lists what a system has.

Overcommit¶

When a program asks for memory (with malloc(), which calls mmap() or brk()), the kernel normally only reserves address space. Physical pages are allocated later, page by page, when the program first touches them. Many programs ask for far more than they ever use, so the kernel overcommits: it promises more memory than it has, betting that not all of it will be touched at once.

vm.overcommit_memory sets the policy:

Value	Policy
`0` (default)	Heuristic: refuse only requests that obviously can't be satisfied
`1`	Always allow; never refuse an allocation
`2`	Strict accounting: refuse any allocation that would take the total committed memory beyond `CommitLimit`, which is swap plus `vm.overcommit_ratio` percent (default 50) of RAM, or plus `vm.overcommit_kbytes` if set

Committed_AS and CommitLimit in /proc/meminfo show the promises made and, under policy 2, the ceiling. Strict accounting makes allocations fail cleanly with an error instead of the process being killed later, which some databases prefer; but with the default ratio a machine without swap can then use only about half of its RAM, so the ratio has to be raised with it.

The OOM killer¶

Overcommitment means the bet can be lost: processes touch more memory than exists, reclaim (dropping cache, swapping) can't free enough, and an allocation can't be satisfied. The kernel then invokes the out-of-memory (OOM) killer, which picks a process and kills it with SIGKILL to free its memory.

The victim is chosen by a badness score, visible as /proc/<pid>/oom_score: mostly the share of memory the process uses (resident memory, swap and page tables), adjusted by /proc/<pid>/oom_score_adj. That adjustment runs from -1000 (never kill this process) to +1000 (kill this first). Service managers can set it per service; systemd's OOMScoreAdjust= does this.

An OOM kill leaves a clear trace in the kernel log, readable with dmesg or journalctl -k: a line saying Out of memory: Killed process <PID> (<NAME>) preceded by a table of candidate processes. A process killed this way exits with signal 9, which a shell or container runtime reports as exit status 137.

The OOM killer also works inside cgroups. When a cgroup reaches its memory.max limit and its memory can't be reclaimed, the kernel kills a process in that cgroup only, logging Memory cgroup out of memory instead. The rest of the machine is unaffected, and memory.events in the cgroup counts the kill. This is often the more common OOM in practice: a service with a memory limit that is too tight is killed while the host has plenty to spare.

A system that is short of memory can spend a long time thrashing (reclaiming and re-reading pages constantly, nearly unresponsive) before the kernel's OOM killer acts, because the kernel kills only when reclaim makes no progress at all. User-space OOM daemons such as systemd-oomd and earlyoom exist to act earlier, based on memory pressure or swap use, and to kill a whole cgroup rather than one process.

Tuning knobs that matter¶

A handful of kernel parameters shape this behaviour. vm.swappiness, the overcommit settings, vm.dirty_ratio and vm.dirty_background_ratio (how much dirty page cache may accumulate before writers are made to wait, and before background writeback starts) and vm.min_free_kbytes (the reserve the kernel keeps free for itself) are the ones most often changed. Changing them is covered in How to change kernel parameters persistently.

On the RHEL family, the tuned service, where it is installed, applies a profile of such settings chosen for the hardware. On AlmaLinux 10's cloud image it runs the virtual-guest profile, which sets vm.swappiness to 30 and vm.dirty_ratio to 30; Rocky Linux 10's cloud image doesn't include tuned, so the kernel defaults apply there. This is why the same parameter can have different values on two otherwise identical RHEL-family machines.

Design decisions and trade-offs¶

Load average as a single number. The load average survives because it is cheap, universal and has decades of history behind it, not because it is precise. Including D tasks makes it a better measure of "is the system keeping up?" and a worse measure of "are the CPUs busy?". PSI, per-CPU utilisation and run-queue length are better for the second question.

Overcommit and the OOM killer. Linux chose to make allocation cheap and optimistic, and to deal with the rare failure by killing something. The alternative, strict accounting, makes failures predictable but wastes memory on promises that are never used, and most software doesn't handle allocation failure gracefully anyway. Cgroup memory limits and user-space OOM daemons are ways of making the failure smaller and earlier, rather than changing the underlying bet.

Cache by default. Filling memory with cache makes the system fast, at the price of making memory figures harder to read. The available estimate exists precisely because "free" stopped being meaningful.