Control groups¶

A Linux system has always had per-process limits: ulimit caps how much CPU time, stack or open files a single process may use. What it lacked for a long time was a way to treat a group of processes as one thing: to say that a web server and every worker, script and helper it starts may together use 2 GiB of memory, or that a backup job, however many processes it forks, gets no more than half a CPU. It also lacked a reliable way to answer "which processes belong to this service?" when daemons fork, double-fork and leave their PID files behind.

Control groups (cgroups) are the kernel's answer to both. A cgroup is a set of processes that the kernel tracks as a unit, accounts resources for, and can apply limits to. Every modern service manager on Linux builds on them, and so do containers: a container's resource limits are cgroup limits.

This page explains how cgroups are organised, what the controllers do, and how systemd and OpenRC use them. Applying limits to a service is covered in How to limit a service's resources.

The hierarchy¶

cgroups are exposed as a filesystem, cgroup2, mounted at /sys/fs/cgroup on every distribution covered here. Each directory in it is a cgroup; creating a directory creates a cgroup, and removing an empty one removes it. Each file in a directory is an interface to that cgroup: cgroup.procs lists its member processes, memory.current reports its memory use, memory.max sets its memory limit, and so on.

/sys/fs/cgroup/                     root cgroup
├── init.scope/                     PID 1
├── system.slice/                   system services
│   ├── ssh.service/
│   ├── cron.service/
│   └── nginx.service/
└── user.slice/                     login sessions
    └── user-1000.slice/
        ├── user@1000.service/      the user's own systemd manager
        └── session-4.scope/        an SSH login and its shell

The rules that make this work are few:

Every process is in exactly one cgroup. A new process starts in its parent's cgroup. Moving a process means writing its PID into the cgroup.procs file of another cgroup, which requires write access to that file (normally root, or a user to whom a subtree has been delegated).
Processes can't leave by themselves. Forking, double-forking, changing user or calling setsid() doesn't change a process's cgroup. That is what makes cgroups reliable for tracking services where PID files and parent-child relationships are not.
Limits are hierarchical. A limit on a cgroup applies to everything below it. A child can set a tighter limit of its own, but it can never use more than its ancestors allow.
Processes live only in leaves, as far as resource control is concerned: a cgroup that distributes a resource to child cgroups can't have processes of its own (the "no internal processes" rule; the root cgroup is the exception).

You can see which cgroup any process is in from /proc/<pid>/cgroup. On a cgroup v2 system it has a single line starting 0::, followed by the path below /sys/fs/cgroup:

$ cat /proc/self/cgroup
0::/user.slice/user-1000.slice/session-4.scope

On Gentoo with OpenRC the same file shows a path such as 0::/openrc.sshd, as explained below.

Controllers¶

A cgroup on its own only groups processes and counts them. Resource control comes from controllers, kernel subsystems that each manage one kind of resource:

Controller	Manages	Typical interface files
`cpu`	CPU time: relative weight and absolute bandwidth caps	`cpu.weight`, `cpu.max`, `cpu.stat`
`memory`	Process memory, page cache and kernel memory charged to the group	`memory.current`, `memory.high`, `memory.max`, `memory.min`, `memory.low`, `memory.events`, `memory.swap.max`
`io`	Block device I/O: weights, bandwidth and IOPS limits per device	`io.weight`, `io.max`, `io.stat`
`pids`	The number of tasks (processes and threads)	`pids.current`, `pids.max`
`cpuset`	Which CPUs and memory nodes the group may run on	`cpuset.cpus`, `cpuset.mems`
`hugetlb`, `rdma`, `misc`	Huge pages, RDMA resources, other countable resources

The controllers a kernel offers are listed in cgroup.controllers in the root cgroup. A controller is available in a cgroup only if its parent has enabled it for its children, by listing it in the parent's cgroup.subtree_control. This is a deliberate economy: a controller costs something to run, so it is switched on only for the parts of the tree that need it.

$ cat /sys/fs/cgroup/cgroup.controllers
cpuset cpu io memory hugetlb pids rdma misc
$ cat /sys/fs/cgroup/cgroup.subtree_control
cpu memory pids

The first file is what the kernel supports; the second is what the service manager has switched on below the root. The output above is from Debian 13, where systemd enables CPU, memory and task accounting by default; some kernels list one or two more controllers, such as dmem.

Pressure stall information¶

Alongside usage counters, each cgroup (and the system as a whole, in /proc/pressure/) reports pressure stall information (PSI): the share of recent time in which some or all tasks in the group were stalled waiting for CPU, memory or I/O. memory.pressure showing some avg10=25.00 means that, over the last 10 seconds, at least one task in the group was waiting for memory a quarter of the time. Pressure answers a question usage counters can't: not "how much is it using?" but "is it being slowed down by not getting enough?" Tools such as systemd-oomd act on it.

Two kinds of limit: weights and caps¶

Controllers offer two styles of control, and the difference matters when choosing settings.

Weights share a resource in proportion when there is contention, and do nothing when there isn't. cpu.weight (1 to 10000, default 100) and io.weight work this way. A service with weight 200 gets twice the CPU of one with weight 100 when both want more than is available; on an idle machine either can use every CPU. Weights never waste capacity, which makes them the gentle choice for "this matters more than that".

Caps are absolute and apply even when the machine is otherwise idle. cpu.max takes a quota and a period in microseconds: 50000 100000 lets the group run for 50 ms in every 100 ms, which is half of one CPU, however many CPUs are idle. memory.max is a hard ceiling on memory use, pids.max on the number of tasks, io.max on bytes or operations per second for one device. Caps are predictable, but they leave capacity unused and can turn a busy service into a slow one.

Memory has a richer set of knobs because it can't be shared out after the fact like CPU time:

memory.max is the hard limit. When the group reaches it, the kernel first tries to reclaim memory from the group itself (dropping its page cache, swapping its anonymous memory if swap is available). If that fails, the out-of-memory (OOM) killer runs within the group and kills one of its processes. The rest of the system is unaffected, which is the point.
memory.high is a throttle. Above it, the kernel reclaims aggressively from the group and slows its allocations, but never kills anything. It is the better tool when the goal is "don't let this grow without limit" rather than "kill it if it does".
memory.min and memory.low protect a group's memory from being reclaimed when other groups are under pressure: a hard and a best-effort guarantee.
memory.events counts how often each of these boundaries was hit, including oom_kill, which is often the quickest way to learn that a service was killed for exceeding its limit.

Memory accounting in a cgroup includes the page cache the group's processes caused, not just their private memory. A service that reads large files will appear to "use" a lot of memory; most of it is reclaimable cache, and it is reclaimed first when the group approaches its limit. This is the same distinction as between "used" and "available" memory system-wide, discussed in Load average and memory.

How systemd uses cgroups¶

systemd is the only process that writes to the cgroup tree on a systemd system (apart from subtrees it delegates). It maps its units onto the hierarchy:

Every service gets its own cgroup, named after the unit (ssh.service), containing every process it started.
Scopes are cgroups for processes systemd didn't start itself but has been asked to track, such as a login session (session-4.scope, registered by systemd-logind) or a container started by a runtime.
Slices are inner nodes that group services and scopes for resource management: system.slice for system services, user.slice for users, machine.slice for virtual machines and containers. A slice's name encodes its path: user-1000.slice lives inside user.slice.

Because every process of a service is in the service's cgroup, systemd can do things that are otherwise hard: stop a service by signalling every process in its cgroup (with KillMode=control-group, the default), know when a service has really exited (its cgroup is empty), and show accurate per-service resource use with systemctl status and systemd-cgtop.

Resource limits are unit settings, described in systemd.resource-control(5). systemd translates each into the corresponding cgroup file: MemoryMax= into memory.max, MemoryHigh= into memory.high, CPUWeight= into cpu.weight, CPUQuota= into cpu.max, TasksMax= into pids.max, IOWeight= and IOReadBandwidthMax= into the io.* files. A setting on a slice limits everything in it together, so user-1000.slice can carry a limit for all of one user's sessions.

CPUQuota= is expressed as a percentage of one CPU: CPUQuota=50% writes 50000 100000 to cpu.max, half of one CPU; CPUQuota=200% allows two full CPUs' worth.

Accounting is on by default: on the distributions covered here systemd enables CPU, memory and task accounting for every unit, and enables the cpu, memory and pids controllers below the root accordingly. I/O accounting is off by default (DefaultIOAccounting=no); it is switched on for a unit when it gets an I/O setting. These defaults are set in /etc/systemd/system.conf, and systemctl show --property=DefaultMemoryAccounting displays the current value. systemd also applies a default TasksMax= to services and user slices, derived from the kernel's limit on the number of tasks, so a fork bomb in one service or session can't exhaust the PID space for everyone.

systemd can also act on cgroup pressure through systemd-oomd, a user-space OOM killer that watches PSI and swap use per slice and kills a whole cgroup before the kernel's OOM killer would have to act. It is enabled by default on some desktop distributions; none of the server cloud images used for this knowledge base ship it enabled.

How OpenRC uses cgroups¶

OpenRC, the service manager on Gentoo's default profile, also uses cgroups on Linux, but more modestly and as an option rather than a foundation. With rc_cgroup_mode="unified" (OpenRC's default, used when /etc/rc.conf doesn't set it, as Gentoo's doesn't), OpenRC mounts the cgroup v2 hierarchy, enables the available controllers for the whole tree, and starts each service in a cgroup of its own named openrc.<service>, directly below the root:

/sys/fs/cgroup/                 root cgroup: PID 1, the gettys, kernel threads
├── openrc.sshd/                sshd and the logins it spawned
├── openrc.sysklogd/
├── openrc.udev/
├── openrc.net.enp0s3/
└── openrc.user.alice/          alice's OpenRC user-service manager

There is no init.scope: PID 1 and the processes OpenRC doesn't start as services, such as the agetty processes on the consoles, stay in the root cgroup. There are no slices either, and login sessions stay in the cgroup of the service that accepted them, which is why a shell opened over SSH reports 0::/openrc.sshd. The openrc.user.<user> cgroup, created through pam_openrc when the user logs in, holds only the supervise-daemon that runs that user's OpenRC user services, not the login session itself.

OpenRC uses the cgroup in two ways:

Cleanup. With rc_cgroup_cleanup="YES", stopping a service kills every process left in its cgroup, using the kernel's cgroup.kill where available. This closes the same "the daemon forked and escaped" gap that pushed systemd to cgroups, though OpenRC leaves it off by default.
Limits. rc_cgroup_settings holds lines of the form <file> <value>, such as memory.max 256M, which OpenRC writes into the service's cgroup when it starts it. Set in /etc/rc.conf it applies to all services; set in /etc/conf.d/<service> it applies to one.

The difference in approach is the one described in OpenRC and other init systems: systemd treats cgroups as the definition of a service and wraps every controller in a named directive; OpenRC uses them as a tool and lets you write the kernel's own interface files directly. The kernel side, and therefore the meaning of memory.max or cpu.max, is identical.

cgroup v1 and v2¶

The original cgroup implementation (now called v1, merged in Linux 2.6.24 in 2008) let each controller have its own, independent hierarchy, mounted separately under /sys/fs/cgroup/cpu, /sys/fs/cgroup/memory and so on. A process could be in one group for CPU and an unrelated group for memory. That flexibility turned out to make controllers hard to coordinate (memory writeback, for example, is both a memory and an I/O problem) and the interfaces were inconsistent between controllers.

cgroup v2, declared stable in Linux 4.5 (2016), has one unified hierarchy that every controller shares, the subtree_control model, the "no internal processes" rule, PSI, and consistent interface files. For some years distributions ran in a hybrid mode, with v2 used for process tracking and v1 for controllers. All the distributions covered here now boot in pure v2 ("unified") mode by default: systemd has dropped support for v1 in recent releases, and OpenRC's default mode is unified. Old instructions that refer to files such as memory.limit_in_bytes or cpu.cfs_quota_us are about v1; the v2 equivalents are memory.max and cpu.max. stat -fc %T /sys/fs/cgroup prints cgroup2fs on a v2 system.

Design decisions and trade-offs¶

One writer. cgroup v2 expects a single manager to own the tree, and delegation to hand subtrees to others (a user's systemd instance, a container runtime) explicitly. Writing to /sys/fs/cgroup behind systemd's back works, but systemd may undo it at the next daemon reload or unit restart. On systemd systems, the supported way to change limits is through unit settings or systemctl set-property; on OpenRC, through rc_cgroup_settings.

Limits versus overcommitment. Hard caps make behaviour predictable but leave resources idle; weights use the whole machine but give weaker guarantees. Most well-run systems use weights and memory.high broadly, and hard caps such as memory.max and CPUQuota= sparingly, for services that must never be allowed to affect their neighbours.

Group-level OOM. A memory limit turns a system-wide out-of-memory event into a local one: the kernel kills a process in the offending group instead of choosing a victim from the whole machine. That is usually what you want, but it means a service with a too-tight memory.max will be killed repeatedly while the machine has plenty of free memory. memory.events and the kernel log make that visible.