How to limit a service's resources¶

A service is using more than its share of the machine: it grows until other services are short of memory, or keeps CPUs busy that other work needs. This guide caps the service's memory, CPU time and number of tasks with control groups, applies the change, and checks that the limits are in the kernel's cgroup files and actually enforced. On systemd distributions the limits go in a drop-in file for the unit; on Gentoo with OpenRC they go in the service's /etc/conf.d/ file.

The limits apply to the service as a whole: every process it starts counts against them together. How cgroups work is explained in Control groups.

Before you begin¶

You need root access through sudo.
The system uses cgroup v2, as every distribution covered here does by default. stat -fc %T /sys/fs/cgroup prints cgroup2fs.
Know the service's name, and roughly how much memory and CPU it needs when it's working normally: a limit below that makes it slow or gets it killed. systemctl status <SERVICE> shows its current memory and CPU use on systemd; How to find what is using CPU and memory covers both init systems.

Throughout this guide, replace this placeholder:

Placeholder	Meaning	Example
`<SERVICE>`	The service's name, without `.service`	`report-builder`

The example limits are:

Resource	Limit	Effect
Memory, soft	200 MiB	Above this, the kernel reclaims the service's memory aggressively and slows its allocations
Memory, hard	256 MiB	The service can't use more; if reclaim fails, the kernel's OOM killer kills one of its processes
CPU	50%	Half of one CPU's time, even when other CPUs are idle
Tasks	64	At most 64 processes and threads; further `fork()` calls fail

Warning

A memory limit below what the service needs gets it killed, possibly repeatedly, and a low CPU limit can make it time out. Limiting sshd, your login session or a database without knowing its working size can lock you out or lose transactions. Start with generous values, watch the service under real load, and tighten them gradually.

Steps¶

Set the limits.
Debian/UbuntuRHEL/Alma/RockyArchGentoo
Create a drop-in directory for the unit and a file in it. Settings in a drop-in add to or override the unit file without changing it, so package updates don't undo them:
sudo mkdir -p /etc/systemd/system/<SERVICE>.service.d
Create /etc/systemd/system/<SERVICE>.service.d/50-limits.conf:
[Service] MemoryHigh=200M MemoryMax=256M CPUQuota=50% TasksMax=64
Make systemd read it, and restart the service so it starts under the new limits:
sudo systemctl daemon-reload sudo systemctl restart <SERVICE>.service
sudo systemctl edit <SERVICE>.service creates the same kind of drop-in (named override.conf) in an editor and reloads systemd when you save.
Create a drop-in directory for the unit and a file in it. Settings in a drop-in add to or override the unit file without changing it, so package updates don't undo them:
sudo mkdir -p /etc/systemd/system/<SERVICE>.service.d
Create /etc/systemd/system/<SERVICE>.service.d/50-limits.conf:
[Service] MemoryHigh=200M MemoryMax=256M CPUQuota=50% TasksMax=64
Make systemd read it, and restart the service so it starts under the new limits:
sudo systemctl daemon-reload sudo systemctl restart <SERVICE>.service
sudo systemctl edit <SERVICE>.service creates the same kind of drop-in (named override.conf) in an editor and reloads systemd when you save.
Create a drop-in directory for the unit and a file in it. Settings in a drop-in add to or override the unit file without changing it, so package updates don't undo them:
sudo mkdir -p /etc/systemd/system/<SERVICE>.service.d
Create /etc/systemd/system/<SERVICE>.service.d/50-limits.conf:
[Service] MemoryHigh=200M MemoryMax=256M CPUQuota=50% TasksMax=64
Make systemd read it, and restart the service so it starts under the new limits:
sudo systemctl daemon-reload sudo systemctl restart <SERVICE>.service
sudo systemctl edit <SERVICE>.service creates the same kind of drop-in (named override.conf) in an editor and reloads systemd when you save.
With OpenRC, each service runs in its own cgroup, /sys/fs/cgroup/openrc.<SERVICE>, as long as rc_cgroup_mode in /etc/rc.conf is unified (the default, also when the line is commented out). OpenRC writes the lines of rc_cgroup_settings into the files of that cgroup when it starts the service. Each line is the name of a cgroup v2 interface file and the value to write to it.

Add the setting to the service's configuration file, /etc/conf.d/<SERVICE> (create it if it doesn't exist):
# /etc/conf.d/<SERVICE> rc_cgroup_settings=" memory.high 200M memory.max 256M cpu.max 50000 100000 pids.max 64 "
cpu.max 50000 100000 allows 50,000 microseconds of CPU time in every 100,000: half of one CPU. OpenRC skips a line whose file doesn't exist in the cgroup without an error, so check the spelling of each name against the files in /sys/fs/cgroup/openrc.<SERVICE>/.

Restart the service so it starts under the new limits:
sudo rc-service <SERVICE> restart
With a systemd profile, use a drop-in as in the other tabs (systemd profile, not verified).
The settings work like this:
- MemoryHigh= (memory.high) is a throttle: the service slows down rather than being killed. MemoryMax= (memory.max) is the hard limit. Set both, with the soft limit somewhat below the hard one, so the service gets a chance to shed memory before anything is killed.
- Memory sizes accept the suffixes K, M, G and T (powers of 1024). In a systemd unit, MemoryMax= also accepts a percentage of physical memory, such as 25%.
- CPUQuota= is a percentage of one CPU: 200% allows two CPUs' worth of time. It is written to cpu.max in the service's cgroup with the default period of 100 ms.
- TasksMax= (pids.max) counts threads as well as processes. A multi-threaded service such as a Java application server can need hundreds.
<!-- verify-check

The limits from step 1 are in place after the restart, before step 2 touches anything.¶

if command -v systemctl >/dev/null; then cg=/sys/fs/cgroup$(systemctl show report-builder.service -p ControlGroup --value) else cg=/sys/fs/cgroup/openrc.report-builder fi test "$(cat "$cg/memory.max")" = 268435456 test "$(cat "$cg/memory.high")" = 209715200 test "$(cat "$cg/cpu.max")" = "50000 100000" test "$(cat "$cg/pids.max")" = 64 →
If the service must not be restarted now, apply the limits to the running service instead.
Debian/UbuntuRHEL/Alma/RockyArchGentoo
systemctl set-property changes the running service's cgroup at once and saves the setting under /etc/systemd/system.control/, so it also applies after a restart or reboot:
sudo systemctl set-property <SERVICE>.service MemoryHigh=200M MemoryMax=256M CPUQuota=50% TasksMax=64
Add --runtime to try values only until the next reboot. Settings made with set-property take precedence over drop-ins in /etc/systemd/system/; remove the file in /etc/systemd/system.control/<SERVICE>.service.d/ and run sudo systemctl daemon-reload to drop them.
systemctl set-property changes the running service's cgroup at once and saves the setting under /etc/systemd/system.control/, so it also applies after a restart or reboot:
sudo systemctl set-property <SERVICE>.service MemoryHigh=200M MemoryMax=256M CPUQuota=50% TasksMax=64
Add --runtime to try values only until the next reboot. Settings made with set-property take precedence over drop-ins in /etc/systemd/system/; remove the file in /etc/systemd/system.control/<SERVICE>.service.d/ and run sudo systemctl daemon-reload to drop them.
systemctl set-property changes the running service's cgroup at once and saves the setting under /etc/systemd/system.control/, so it also applies after a restart or reboot:
sudo systemctl set-property <SERVICE>.service MemoryHigh=200M MemoryMax=256M CPUQuota=50% TasksMax=64
Add --runtime to try values only until the next reboot. Settings made with set-property take precedence over drop-ins in /etc/systemd/system/; remove the file in /etc/systemd/system.control/<SERVICE>.service.d/ and run sudo systemctl daemon-reload to drop them.
OpenRC only writes rc_cgroup_settings when it starts a service. To limit a running service without restarting it, write the same values into its cgroup directly; keep them in /etc/conf.d/<SERVICE> as well, so they apply at the next start:
echo 200M | sudo tee /sys/fs/cgroup/openrc.<SERVICE>/memory.high echo 256M | sudo tee /sys/fs/cgroup/openrc.<SERVICE>/memory.max echo "50000 100000" | sudo tee /sys/fs/cgroup/openrc.<SERVICE>/cpu.max echo 64 | sudo tee /sys/fs/cgroup/openrc.<SERVICE>/pids.max

Verify¶

Check that the kernel has the limits, in the service's cgroup files.

Debian/UbuntuRHEL/Alma/RockyArchGentoo

$ systemctl show <SERVICE>.service -p MemoryHigh -p MemoryMax -p CPUQuotaPerSecUSec -p TasksMax
MemoryHigh=209715200
MemoryMax=268435456
CPUQuotaPerSecUSec=500ms
TasksMax=64
$ cg=/sys/fs/cgroup$(systemctl show <SERVICE>.service -p ControlGroup --value)
$ cat "$cg/memory.high" "$cg/memory.max" "$cg/cpu.max" "$cg/pids.max"
209715200
268435456
50000 100000
64

systemctl show reports what systemd intends; the files under /sys/fs/cgroup are what the kernel enforces.

$ systemctl show <SERVICE>.service -p MemoryHigh -p MemoryMax -p CPUQuotaPerSecUSec -p TasksMax
MemoryHigh=209715200
MemoryMax=268435456
CPUQuotaPerSecUSec=500ms
TasksMax=64
$ cg=/sys/fs/cgroup$(systemctl show <SERVICE>.service -p ControlGroup --value)
$ cat "$cg/memory.high" "$cg/memory.max" "$cg/cpu.max" "$cg/pids.max"
209715200
268435456
50000 100000
64

systemctl show reports what systemd intends; the files under /sys/fs/cgroup are what the kernel enforces.

$ systemctl show <SERVICE>.service -p MemoryHigh -p MemoryMax -p CPUQuotaPerSecUSec -p TasksMax
MemoryHigh=209715200
MemoryMax=268435456
CPUQuotaPerSecUSec=500ms
TasksMax=64
$ cg=/sys/fs/cgroup$(systemctl show <SERVICE>.service -p ControlGroup --value)
$ cat "$cg/memory.high" "$cg/memory.max" "$cg/cpu.max" "$cg/pids.max"
209715200
268435456
50000 100000
64

systemctl show reports what systemd intends; the files under /sys/fs/cgroup are what the kernel enforces.

$ cat /proc/$(pgrep -o -x <SERVICE>)/cgroup
0::/openrc.report-builder
$ cg=/sys/fs/cgroup/openrc.<SERVICE>
$ cat "$cg/memory.high" "$cg/memory.max" "$cg/cpu.max" "$cg/pids.max"
209715200
268435456
50000 100000
64

The first command confirms that the service's process is in the cgroup that carries the limits.

Check that the CPU limit is enforced. A service that wants a whole CPU should now get about half of one. cpu.stat in the service's cgroup counts the CPU time used, in microseconds, and how often the group was throttled:

Debian/UbuntuRHEL/Alma/RockyArchGentoo

$ cg=/sys/fs/cgroup$(systemctl show <SERVICE>.service -p ControlGroup --value)
$ grep -e usage_usec -e nr_throttled "$cg/cpu.stat"
usage_usec 5012345
nr_throttled 98

$ cg=/sys/fs/cgroup$(systemctl show <SERVICE>.service -p ControlGroup --value)
$ grep -e usage_usec -e nr_throttled "$cg/cpu.stat"
usage_usec 5012345
nr_throttled 98

$ cg=/sys/fs/cgroup$(systemctl show <SERVICE>.service -p ControlGroup --value)
$ grep -e usage_usec -e nr_throttled "$cg/cpu.stat"
usage_usec 5012345
nr_throttled 98

$ cg=/sys/fs/cgroup/openrc.<SERVICE>
$ grep -e usage_usec -e nr_throttled "$cg/cpu.stat"
usage_usec 5012345
nr_throttled 98

A growing nr_throttled count shows that the quota is cutting the service off. Read usage_usec twice, 10 seconds apart: with CPUQuota=50% it grows by no more than about 5,000,000 (5 seconds of CPU time). top shows the same thing as a %CPU of about 50 for the service's process.

Check after the service has run for a while under real load whether it hit the memory limits. memory.events in the same directory counts each event:

Debian/UbuntuRHEL/Alma/RockyArchGentoo

$ cg=/sys/fs/cgroup$(systemctl show <SERVICE>.service -p ControlGroup --value)
$ cat "$cg/memory.events"
low 0
high 0
max 0
oom 0
oom_kill 0
oom_group_kill 0

$ cg=/sys/fs/cgroup$(systemctl show <SERVICE>.service -p ControlGroup --value)
$ cat "$cg/memory.events"
low 0
high 0
max 0
oom 0
oom_kill 0
oom_group_kill 0

$ cg=/sys/fs/cgroup$(systemctl show <SERVICE>.service -p ControlGroup --value)
$ cat "$cg/memory.events"
low 0
high 0
max 0
oom 0
oom_kill 0
oom_group_kill 0

$ cg=/sys/fs/cgroup/openrc.<SERVICE>
$ cat "$cg/memory.events"
low 0
high 0
max 0
oom 0
oom_kill 0
oom_group_kill 0

high counts how often the service was throttled for exceeding the soft limit, max how often it reached the hard limit, and oom_kill how many of its processes the kernel killed. Steadily growing high or max counts mean the limits are tighter than the service's working size.

Troubleshooting¶

systemctl show lists the new values, but the cgroup files still say max. The service hasn't been restarted since you ran daemon-reload, or you're reading the cgroup of a different unit. Restart it, or use systemctl set-property to apply the limits to the running service, and check ControlGroup again.

The drop-in has no effect at all. Check that the directory is named after the unit exactly (<SERVICE>.service.d), that the file name ends in .conf, that the settings are under a [Service] heading, and that you ran sudo systemctl daemon-reload. systemctl cat <SERVICE>.service shows the unit with every drop-in that systemd has loaded, and systemctl status lists the drop-ins under Drop-In:.

The service is killed, or restarts repeatedly, after the change. It reached MemoryMax= and the kernel's OOM killer ended one of its processes. memory.events shows a growing oom_kill count, and the kernel log has a Memory cgroup out of memory line naming the process: check with sudo dmesg | grep -i 'memory cgroup' or, on systemd, journalctl -k -g 'memory cgroup'. Raise the limit, or lower MemoryHigh= relative to it so the service is throttled earlier.

New processes or threads fail with "Resource temporarily unavailable". The service reached its task limit. pids.events in its cgroup counts the refused forks; raise TasksMax= or pids.max.

On Gentoo, the cgroup files keep their default values. OpenRC skips any line of rc_cgroup_settings whose interface file doesn't exist, without an error. Check the names against ls /sys/fs/cgroup/openrc.<SERVICE>/, make sure rc_cgroup_mode in /etc/rc.conf isn't set to legacy or hybrid, and restart the service. The settings apply only to services started through OpenRC, not to programs started by hand.

Explanation: Control groups
Explanation: Load average and memory
Explanation: OpenRC and other init systems
Reference: Process information fields
How-to: How to find what is using CPU and memory
How-to: How to change kernel parameters persistently

How to limit a service's resources¶

Before you begin¶

Steps¶

The limits from step 1 are in place after the restart, before step 2 touches anything.¶

Verify¶

Troubleshooting¶

Related¶