Processes and signals¶

Almost everything an administrator does touches a process: starting a service, stopping a runaway job, working out why a machine is slow, or why a daemon ignored a request to reload its configuration. The tools (ps, top, kill, the shell's & and Ctrl+Z) are thin layers over a small number of kernel ideas. Once those ideas are clear, much of the behaviour that looks strange, such as a "zombie" that can't be killed, a process stuck in D state that ignores kill -9, or a background job that stops as soon as it prints something, follows directly from them.

This page explains the process model, process states, signals, and job control. The individual signals are listed in Signals, and the fields ps and /proc show in Process information fields.

The process model: fork and exec¶

A process is a running program together with the resources the kernel tracks for it: an address space, open file descriptors, a current directory, credentials (user and group IDs, capabilities), resource limits, signal dispositions, and a place in the process tree. Each process has a numeric process ID (PID).

Unix splits "start a new program" into two separate operations, and Linux keeps that design:

fork() creates a new process that is an almost exact copy of the caller. The copy, the child, gets a new PID; the caller is its parent, and the child records the parent's PID as its PPID. Memory isn't copied immediately: parent and child share the pages and the kernel copies a page only when one of them writes to it (copy-on-write), so forking even a large process is cheap.
execve() replaces the program running in a process with a new one. The PID stays the same, and so do the open file descriptors (except those marked close-on-exec), the current directory, the credentials (unless the new file is set-user-ID or has file capabilities) and the resource limits. The code, data and stack are thrown away and replaced.

A shell running ls therefore forks a copy of itself, the copy adjusts its own environment (redirecting standard output to a file, for example), and then execs ls. The separation is what makes that adjustment simple: everything between fork() and execve() is ordinary code running in the child, affecting only the child. On Linux both operations are built on a more general system call, clone(), which lets the caller choose what the new task shares with its creator; vfork() is a variant of fork() that shares the parent's memory and suspends the parent until the child calls execve() or exits, and posix_spawn() combines creating a process and running a program in one call.

sequenceDiagram
    participant P as Shell (PID 1200)
    participant C as Child (PID 1201)
    P->>C: fork(): copy of the shell
    Note over C: set up redirections,<br/>reset signals
    C->>C: execve("/usr/bin/ls"): same PID, new program
    P->>P: wait() for the child
    C-->>P: exit(0): status kept until collected
    Note over P: wait() returns status 0;<br/>PID 1201 is freed

Threads are tasks too¶

Inside the kernel, the unit the scheduler runs is a task. A multi-threaded program is a group of tasks that share one address space and one set of file descriptors. The ID of the first task is the thread group ID (TGID), and that is what user space calls the PID; each thread also has its own thread ID (TID). That's why /proc/<pid>/task/ lists one directory per thread, why ps -L shows threads as separate lines with a shared PID, and why top can be switched between showing processes and threads.

The process tree¶

Every process except the first has a parent, so processes form a tree. The kernel starts two processes itself: PID 1, the init system (systemd on most of the distributions covered here, sysvinit's init with OpenRC on Gentoo), and PID 2, kthreadd, the parent of all kernel threads. Kernel threads have no user-space program and no command line, which is why ps shows their names in square brackets, such as [kworker/0:1].

Running ps -ef --forest or pstree -p on any server shows the shape: sshd has a child for each login, each of which has a shell, which has the commands you run. Most daemons appear as children of PID 1, whatever the init system: either the service manager started them directly, or they forked into the background and were adopted by PID 1 as orphans. Where init systems differ is in whether they keep watching those processes afterwards, which is a matter of supervision and cgroups rather than of the tree's shape.

PID numbers are allocated in increasing order and wrap around at the limit in kernel.pid_max; a PID is reused only after its previous owner has completely gone. Inside a PID namespace, used by containers, processes see their own numbering starting from 1, while the host sees the same processes under different, host-wide PIDs.

Exit, wait, zombies and orphans¶

When a process finishes, by calling exit() or by being killed by a signal, the kernel releases almost everything it held: memory, file descriptors, locks. It keeps one small record: the PID and the exit status (a number from 0 to 255, or the signal that killed it). That record exists so the parent can find out how its child ended, which it does by calling one of the wait() family of system calls. The kernel also sends the parent a SIGCHLD signal to tell it a child has changed state.

Between the child's exit and the parent's wait(), the child is a zombie: state Z in ps, shown as <defunct>. A zombie isn't running and holds no memory, so it can't be killed: there is nothing left to kill, and signals sent to it are ignored. A few zombies that come and go are normal. Zombies that accumulate point to a bug in the parent, which isn't collecting its children. The fix is to make the parent call wait() (often by restarting it), or to end the parent, at which point the zombies are adopted as orphans and cleaned up. A large number of zombies is a problem mainly because each one still occupies a PID.

An orphan is the opposite case: a process whose parent exited first. The kernel immediately gives it a new parent so that someone will eventually collect its exit status. By default the new parent is PID 1, which is why one of the init system's jobs is to reap orphans in a loop. A process can also declare itself a child subreaper (with prctl(PR_SET_CHILD_SUBREAPER)); orphans below it then get it as their new parent rather than PID 1. systemd's per-user manager (systemd --user) and container runtimes use this so that the processes they start stay inside their part of the tree.

Daemons traditionally relied on orphaning on purpose: the classic way to start a daemon was to fork, let the parent exit so the child was adopted by init, and start a new session so the daemon had no controlling terminal. Service managers that supervise processes, such as systemd and OpenRC's supervise-daemon, prefer programs that stay in the foreground instead, because then the manager is the parent and is told directly when the program exits.

Process states¶

At any moment a task is in one state, shown in the STAT (or S) column of ps and top:

Code	State	What it means
`R`	Running or runnable	On a CPU, or waiting in a run queue for one
`S`	Interruptible sleep	Waiting for an event (input, a timer, a lock, a child); a signal wakes it
`D`	Uninterruptible sleep	Waiting inside the kernel, usually for I/O, in a section that can't be interrupted by signals
`T`	Stopped	Stopped by a job-control signal such as `SIGSTOP` or `SIGTSTP`; resumes on `SIGCONT`
`t`	Tracing stop	Stopped by a debugger, such as `gdb` or `strace`
`Z`	Zombie	Exited, waiting for its parent to collect its status
`I`	Idle	An idle kernel thread; it isn't waiting for anything in particular and doesn't count towards load

Most processes on a healthy server spend almost all their time in S. R means the task wants a CPU; a long queue of R tasks is what CPU saturation looks like.

D deserves special attention. A task enters uninterruptible sleep when it is in the middle of an operation the kernel can't safely abandon half-way, classically waiting for a disk read or an NFS server. Signals are queued but not acted on until the operation finishes, so even SIGKILL has no visible effect on a task stuck in D. Brief D states are normal and constant on a busy file server. Tasks stuck in D for minutes point to the thing they are waiting for: a failing disk, a hung storage path, an unreachable NFS server. Many kernel paths now use a variant of this sleep that does respond to fatal signals such as SIGKILL, but not all do. Because Linux counts D tasks in the load average, a stuck storage device can drive the load up on an otherwise idle machine; see Load average and memory.

ps adds modifier letters after the state: s for a session leader, + for a member of the foreground process group of its terminal, l for a multi-threaded process, < and N for high and low priority (negative or positive nice value), and L for pages locked in memory.

Signals¶

A signal is a small asynchronous notification sent to a process: a number with no payload (apart from optional extra data for real-time signals). Signals come from three kinds of source:

Other processes, through kill(), tgkill() or sigqueue(), which is what the kill, pkill and killall commands and systemctl kill use. An unprivileged process can signal another only if its real or effective user ID matches the target's real or saved set-user-ID; root, or a process with CAP_KILL, can signal any.
The kernel, because of something the process did: SIGSEGV for an invalid memory access, SIGFPE for an arithmetic error such as integer division by zero, SIGPIPE for writing to a pipe nobody reads, SIGBUS, SIGILL.
The kernel, on behalf of the terminal or other events: SIGINT when you press Ctrl+C, SIGTSTP for Ctrl+Z, SIGHUP when a terminal hangs up, SIGCHLD when a child changes state, SIGWINCH when a terminal window is resized, SIGXCPU when a CPU-time limit is exceeded.

Disposition: what happens when a signal arrives¶

For each signal, a process has a disposition:

Default action, which depends on the signal: terminate the process, terminate and write a core dump, ignore the signal, stop the process, or continue it if stopped.
Ignore it.
Catch it: run a handler function in the process, after which the process carries on.

A process can also block a signal temporarily; it then stays pending until it is unblocked.

Two signals can never be caught, blocked or ignored: SIGKILL, which always terminates, and SIGSTOP, which always stops. They exist so that there is always a way to stop a process that doesn't cooperate. PID 1 is protected even from those: the kernel doesn't deliver signals to init unless init has installed a handler for them, so kill -9 1 does nothing.

Dispositions interact with fork() and exec() in a way that explains some old idioms. A child inherits its parent's dispositions. On exec, caught signals are reset to their default (the handler code no longer exists), but ignored signals stay ignored. nohup works exactly this way: it sets SIGHUP to "ignore" and then execs the command, which therefore survives a terminal hang-up unless it changes the disposition back itself.

Terminating politely and impolitely¶

The choice between signals for stopping a program is a choice about who does the clean-up:

SIGTERM (15) is the conventional request to terminate, and the default for kill. A program can catch it and finish cleanly: complete the request in progress, flush buffers, remove its PID file and temporary files. If it doesn't catch it, the default action terminates it anyway.
SIGINT (2) is the interactive version, sent by Ctrl+C. Programs often treat it like SIGTERM; interactive programs such as shells and editors may use it to cancel the current operation instead.
SIGKILL (9) removes the process without giving it any chance to react. Nothing the program would have done on exit happens: temporary files remain, child processes may be orphaned, data in its own buffers is lost, and a database may need recovery at its next start. The kernel still closes its file descriptors and releases its memory and locks.

Service managers follow the same escalation. When systemd stops a service, it sends SIGTERM (configurable with KillSignal=) to the main process, or to every process in the service's cgroup depending on KillMode=, waits up to TimeoutStopSec= (90 seconds by default), and then sends SIGKILL to whatever is left. OpenRC's start-stop-daemon and supervise-daemon follow a retry schedule instead: by default they send SIGTERM and wait 5 seconds, and a service can set a longer schedule that ends in SIGKILL, such as SIGTERM/30/SIGKILL/5.

Signals as commands¶

Because the default action of SIGHUP is to terminate, and daemons have no terminal to hang up, many daemons reuse it to mean "re-read your configuration" or "reopen your log files". SIGUSR1 and SIGUSR2 have no meaning at all until a program gives them one, and programs use them for things like dumping statistics or toggling debug logging. These conventions belong to each program, so kill -HUP on a program that doesn't handle it ends the program. Check the program's documentation before using a signal as a command, or see what its service manager does to reload it: the ExecReload= line of a systemd unit, or the reload function of an OpenRC init script (listed in its extra_started_commands).

Exit statuses after a signal¶

When a process is killed by a signal, its parent learns the signal number rather than an exit code. Shells report that as an exit status of 128 plus the signal number, so $? is 130 after Ctrl+C (signal 2), 137 after SIGKILL (9) and 143 after SIGTERM (15). The same arithmetic explains exit codes reported by container runtimes and job schedulers: 137 usually means the out-of-memory killer, or another SIGKILL, ended the process.

Real-time signals¶

Signals 1 to 31 are the classic standard signals. If the same standard signal is sent several times while it is pending, the process sees it once: standard signals aren't queued. The kernel provides signals 32 to 64 as real-time signals: they are queued, delivered in order, and can carry a small value. The GNU C library keeps 32 and 33 for its threads implementation, so programs and kill -l see SIGRTMIN as 34 and refer to the others as SIGRTMIN+n. systemd uses some of them as commands to PID 1; most other software never sees them.

Job control: sessions, process groups and the terminal¶

Interactive shells let you run several commands at once, suspend one, and move it between the foreground and background. This job control is built from three kernel concepts.

A process group is a set of processes that receive terminal signals together. A shell puts each job (a single command or a pipeline such as sort big.txt | uniq -c) into its own process group, so Ctrl+C interrupts every process in the pipeline at once.
A session is a set of process groups, normally everything started from one login. The process that creates it is the session leader, usually the login shell.
A session can have a controlling terminal. At any time exactly one of the session's process groups is the terminal's foreground process group; the others are in the background.

flowchart TB
    subgraph S["Session (leader: bash, PID 1200)"]
        direction LR
        G1["Process group 1200<br/>bash"]
        G2["Process group 1310 (foreground)<br/>sort | uniq -c"]
        G3["Process group 1320 (background)<br/>tar czf backup.tgz /srv"]
    end
    T["Terminal /dev/pts/0"] -- "Ctrl+C, Ctrl+Z, input" --> G2

The terminal driver in the kernel, not the shell, turns special keys into signals for the foreground group: Ctrl+C sends SIGINT, Ctrl+\ sends SIGQUIT, and Ctrl+Z sends SIGTSTP, whose default action stops the job. The shell notices (through SIGCHLD and waitpid()) that its foreground job stopped and takes the terminal back. The fg, bg and jobs built-ins then change which group is in the foreground and send SIGCONT to resume a stopped one.

Background jobs are protected from mixing their input with yours. A background process that tries to read from the terminal is sent SIGTTIN, which stops it until you bring it to the foreground. One that writes is sent SIGTTOU only if the terminal has the tostop setting (stty tostop); by default, background output is simply mixed into your screen.

When the terminal goes away, for example because an SSH connection drops, the kernel sends SIGHUP to the session leader, and shells pass SIGHUP on to their jobs, so by default everything started from that login ends. There are several ways to keep work running past a disconnection, and they work at different levels:

nohup makes the command ignore SIGHUP and redirects its output away from the terminal.
disown (in Bash) removes a job from the shell's job table, so the shell doesn't forward SIGHUP to it.
setsid starts the command in a new session with no controlling terminal, so it never receives a hang-up.
A terminal multiplexer such as tmux or screen keeps the session's terminal alive in a server process, so nothing hangs up at all.
A transient service, for example with systemd-run, hands the process to the service manager, outside your login session entirely.

On systemd systems one more mechanism applies: systemd-logind can kill all processes of a login session when it ends, if KillUserProcesses=yes is set in logind.conf. The distributions covered here ship with it set to no, so the signal-based behaviour above is what you normally see.

Design decisions and trade-offs¶

The fork-and-exec split is old (it dates from early Unix in the 1970s) and has been criticised for decades. Forking a process with a large address space or many threads only to replace it immediately is wasted work, and the copied state, such as file descriptors that weren't marked close-on-exec, can leak into programs that shouldn't have it. Copy-on-write made forking cheap enough, and posix_spawn() and vfork() cover the performance-critical case, but the model has survived mostly because it is so easy to compose: every feature that changes how a process runs (redirections, namespaces, resource limits, credentials) can be applied in the child between the two steps without any new system call to combine them.

Signals are one of the least loved parts of Unix. A handler can interrupt the program at almost any instruction, so very little can be done safely inside it, and standard signals can be lost if they arrive twice. Modern programs often avoid handlers altogether and receive signals as ordinary readable events through signalfd() or a self-pipe, and long-running services increasingly receive commands through sockets or D-Bus instead. Signals remain because they are universal: every process can receive SIGTERM, and every tool can send it.

Tracking processes by parentage also has a limit: a process that double-forks escapes its parent, and PID files that record a daemon's PID can go stale. That weakness is a large part of why Linux service managers moved to tracking services with control groups, explained in Control groups.