Skip to content

How to find what is using CPU and memory

A server is slow, its load average is high, or it's running short of memory, and you need to know which processes are responsible. This guide goes from a quick overview to the individual processes and services using CPU and memory, with tools that are on every distribution (top, ps, /proc) and pidstat from the sysstat package for measurements over an interval.

What the load average and the memory figures mean is explained in Load average and memory; the columns used here are listed in Process information fields.

Before you begin

  • You need a shell on the machine. Any user can see every process's CPU and memory use; sudo is needed to read the kernel log and some details of other users' processes.
  • Installing sysstat needs sudo and access to the distribution's package repositories.

Steps

  1. Get an overview: how long the machine has been up, the load average, the number of CPUs, and memory:

    $ uptime
 13:40:12 up 2 days,  3:12,  1 user,  load average: 1.08, 0.97, 0.62
$ nproc
2
$ free -h
               total        used        free      shared  buff/cache   available
Mem:           1.9Gi       643Mi       1.1Gi       5.1Mi       375Mi       1.3Gi
Swap:             0B          0B          0B

    Compare the load average with the number of CPUs: a load close to or above nproc means work is queueing. For memory, read the available column, not free: page cache in buff/cache is given back when programs need it.

  2. Take a snapshot of the busiest processes with top in batch mode, sorted by CPU:

    $ top -b -n 1 -o %CPU | sed -n '1,12p'
top - 13:40:15 up 2 days,  3:12,  1 user,  load average: 1.08, 0.97, 0.62
Tasks: 121 total,   2 running, 119 sleeping,   0 stopped,   0 zombie
%Cpu(s): 50.0 us,  0.0 sy,  0.0 ni, 50.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   1974.2 total,   1144.6 free,    643.0 used,    375.3 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   1331.2 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   1153 root      20   0    2584   1536   1408 R 100.0   0.1   2:01.03 report-+
   1155 root      20   0  398528 392448   3200 S   0.0  19.4   0:01.12 cache-l+
      1 root      20   0   24092  14500  10740 S   0.0   0.7   0:01.60 systemd

    sed -n '1,12p' keeps the first 12 lines of output. In the header, %Cpu(s) splits CPU time into user (us), system (sy), nice (ni), idle (id), waiting for I/O (wa), interrupts (hi, si) and time taken by the hypervisor (st, "steal"). In the process list, %CPU is the share of one CPU over the last interval, so a single process can show up to 100 per CPU it uses, and RES is the resident memory in KiB.

    Run plain top to watch the same view refresh every few seconds. Inside it, press P to sort by CPU, M to sort by memory, 1 to show each CPU separately, c to show full command lines, H to show threads, and q to quit.

  3. List processes sorted by resident memory, with ps:

    $ ps -eo pid,ppid,user,rss,vsz,%mem,comm --sort=-rss | sed -n '1,6p'
    PID    PPID USER       RSS    VSZ %MEM COMMAND
   1155       1 root    392448 398528 19.4 cache-loader
    328       1 root     14500  42728  0.7 systemd-journal
      1       0 root     14500  24092  0.7 systemd
    827       1 labuser  11956  21796  0.5 systemd
    872     870 labuser   5376   8232  0.2 bash

    RSS (resident set size, in KiB) is the physical memory a process is using now, including shared libraries it maps; VSZ is the address space it has reserved, which is usually much larger and rarely matters. To sort by CPU instead, use --sort=-%cpu; but note that the %CPU column of ps is the average over the process's whole lifetime, not its current use. For current use, use top (step 2) or pidstat (step 5).

  4. Look at one process in detail. Replace <PID> with a process ID from the previous steps:

    $ ps -o pid,ppid,user,stat,etime,time,args -p <PID>
    PID    PPID USER     STAT     ELAPSED     TIME COMMAND
   1155       1 root     S          02:03 00:00:01 /bin/bash /usr/local/bin/cache-loader
$ grep -e '^State' -e '^VmRSS' -e '^RssAnon' -e '^RssFile' -e '^VmSwap' -e '^Threads' /proc/<PID>/status
State:  S (sleeping)
VmRSS:    392448 kB
RssAnon:    390784 kB
RssFile:      1664 kB
VmSwap:         0 kB
Threads:  1
$ cat /proc/<PID>/cgroup
0::/user.slice/user-1000.slice/session-4.scope
    • args shows the full command line, which tells apart processes with the same name.
    • RssAnon is memory only the process itself can free (heap and stack); RssFile is mapped files, which the kernel can drop and re-read.
    • /proc/<PID>/cgroup shows which service or session the process belongs to: a path ending in .service on systemd, or openrc.<SERVICE> on Gentoo with OpenRC.

  5. Measure CPU and memory use over an interval with pidstat. Install the sysstat package if it isn't installed already:

    sudo apt-get update
sudo apt-get install -y sysstat

    Ubuntu's cloud images have sysstat installed already.

    sudo dnf install -y sysstat

    sudo pacman -Syu --needed --noconfirm sysstat

    -Syu updates the whole system along with installing the package, because Arch doesn't support installing new packages against an outdated system.

    sudo emerge --ask=n --noreplace --getbinpkg app-admin/sysstat

    --getbinpkg uses a prebuilt binary package when one matches your configuration, and compiles from source otherwise.

    Then sample every process's CPU use once a second, five times:

    $ pidstat -u 1 5
Linux 6.12.115+deb13.1-cloud-amd64 (server1)        11/07/26        _x86_64_        (2 CPU)

13:41:02      UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
13:41:03        0      1153  100.00    0.00    0.00    0.00  100.00     0  report-builder
…
Average:      UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
Average:        0      1153  100.00    0.00    0.00    0.00  100.00     -  report-builder

    pidstat lists only processes that used CPU during each interval, which makes busy processes easy to spot. %usr and %system split the time between the program's own code and work the kernel did for it; a high %wait means the process was ready to run but waiting for a CPU. Other useful views:

    • pidstat -r 1 5: memory, including page faults per second. A high majflt/s means the process is waiting for pages to be read from disk or swap.
    • pidstat -d 1 5: disk reads and writes per process (as root, to see every process).
    • pidstat -t -p <PID> 1 5: each thread of one process.

  6. Find out which service is using the resources.

    systemd-cgtop shows CPU and memory per service, slice and session, refreshed like top; in batch mode it prints one snapshot:

    $ systemd-cgtop -b -n 1 --order=memory | sed -n '1,6p'
user.slice                                     10      -   497.6M        -        -
user.slice/user-1000.slice                     10      -   497.6M        -        -
/                                             118      -   430.8M        -        -
user.slice/user-1000.slice/session-4.scope      3      -   193.3M        -        -
system.slice                                   12      -   109.8M        -        -
system.slice/systemd-journald.service           1      -    12.2M        -        -

    The columns are the cgroup, its number of tasks, CPU use, memory, and I/O rates. In batch mode the first snapshot has no CPU figures (-), because CPU use is measured between two refreshes; run systemd-cgtop without options to watch it refresh, and press c, m or t to sort by CPU, memory or number of tasks.

    systemctl status <SERVICE> shows the same memory and CPU figures for one service, followed by its processes.

    systemd-cgtop shows CPU and memory per service, slice and session, refreshed like top; in batch mode it prints one snapshot:

    $ systemd-cgtop -b -n 1 --order=memory | sed -n '1,6p'
user.slice                                     10      -   497.6M        -        -
user.slice/user-1000.slice                     10      -   497.6M        -        -
/                                             118      -   430.8M        -        -
user.slice/user-1000.slice/session-4.scope      3      -   193.3M        -        -
system.slice                                   12      -   109.8M        -        -
system.slice/systemd-journald.service           1      -    12.2M        -        -

    The columns are the cgroup, its number of tasks, CPU use, memory, and I/O rates. In batch mode the first snapshot has no CPU figures (-), because CPU use is measured between two refreshes; run systemd-cgtop without options to watch it refresh, and press c, m or t to sort by CPU, memory or number of tasks.

    systemctl status <SERVICE> shows the same memory and CPU figures for one service, followed by its processes.

    systemd-cgtop shows CPU and memory per service, slice and session, refreshed like top; in batch mode it prints one snapshot:

    $ systemd-cgtop -b -n 1 --order=memory | sed -n '1,6p'
user.slice                                     10      -   497.6M        -        -
user.slice/user-1000.slice                     10      -   497.6M        -        -
/                                             118      -   430.8M        -        -
user.slice/user-1000.slice/session-4.scope      3      -   193.3M        -        -
system.slice                                   12      -   109.8M        -        -
system.slice/systemd-journald.service           1      -    12.2M        -        -

    The columns are the cgroup, its number of tasks, CPU use, memory, and I/O rates. In batch mode the first snapshot has no CPU figures (-), because CPU use is measured between two refreshes; run systemd-cgtop without options to watch it refresh, and press c, m or t to sort by CPU, memory or number of tasks.

    systemctl status <SERVICE> shows the same memory and CPU figures for one service, followed by its processes.

    With OpenRC, each service has a cgroup named openrc.<SERVICE>, and the kernel keeps its totals in memory.current (bytes) and cpu.stat (microseconds of CPU time since the service started). List the memory use of every service, largest first:

    $ grep -H . /sys/fs/cgroup/openrc.*/memory.current | sort -t: -k2 -rn
/sys/fs/cgroup/openrc.sshd/memory.current:421105664
/sys/fs/cgroup/openrc.udev/memory.current:9113600
/sys/fs/cgroup/openrc.sysklogd/memory.current:1093632

    Programs started from an SSH login stay in the cgroup of sshd, so on a server they are counted under openrc.sshd.

    In systemd-cgtop, user.slice holds everything started from login sessions; a service that is busy appears under system.slice with its unit name. Memory per cgroup includes the page cache its processes caused, so a service that reads large files can show more memory than the sum of its processes' RSS.

  7. If memory ran out, check whether the kernel's OOM killer ended a process:

    sudo dmesg -T | grep -i -e 'out of memory' -e 'oom-kill' || echo "no OOM kills since boot"

    Each OOM kill logs a line such as Out of memory: Killed process 1155 (cache-loader), or Memory cgroup out of memory when a service reached its own memory limit. The kernel log in memory covers the current boot only; on systemd systems with a persistent journal, journalctl -k -b -1 shows the previous boot's kernel messages, and on Gentoo with OpenRC, sysklogd writes kernel messages to /var/log/kern.log and /var/log/syslog (OOM reports are logged at error level, so they don't appear in /var/log/messages).

  8. If the load is high but the CPUs are mostly idle, look for processes in uninterruptible sleep (state D), which are counted in the load average while they wait for I/O:

    ps -eo pid,stat,wchan:32,comm | awk 'NR == 1 || $2 ~ /^D/'

    The WCHAN column names the kernel function each process is waiting in. The same few processes staying in D state for a long time usually point to a slow or hung disk, or an unreachable network filesystem.

Verify

You've found the cause when the processes you identified account for the load or memory use you saw in step 1: their %CPU values in top or pidstat add up to roughly the busy CPU time in the %Cpu(s) line, or their memory accounts for the drop in available. After you stop or limit them, top and free -h should show the change within a few seconds, and the 1-minute load average within a minute or two.

Troubleshooting

pidstat prints only its header lines. No process used CPU during the interval, so there was nothing to list; on an idle machine that's expected. Use a longer interval, or pidstat -p ALL to list every process whether it was active or not.

top shows high st (steal). The hypervisor is giving the virtual machine's CPU time to other guests. No process inside the machine is responsible; the fix is on the host or with the provider.

%CPU in top adds up to more than 100. top reports each process as a share of one CPU, so a multi-threaded process on a machine with 8 CPUs can show up to 800. Press I (Irix mode off) to show shares of the whole machine instead.

Memory use grows but no process has a large RSS. Look at the rest of /proc/meminfo: large Shmem values point to tmpfs filesystems (check with df -h -t tmpfs), and a large SUnreclaim points to kernel memory, often from a driver or a very large number of network connections or files.