Linux logging architecture¶

When something goes wrong on a Linux machine, the first question is usually "what do the logs say?". The answer depends on where you look, because a Linux system doesn't have one log. The kernel keeps its own buffer of messages, the systemd journal collects messages from almost everything, a syslog daemon may write some of them to text files, and many applications write files of their own. Which of these exist, how they're connected and what survives a reboot differs between distributions, and even between installations of the same distribution.

This page explains the parts, how messages move between them, and why the distribution families covered here make different choices.

The producers¶

Log messages start in one of a few places.

The kernel writes messages with printk() into the kernel ring buffer, a fixed-size area of memory. When the buffer is full, the oldest messages are overwritten, which is why the earliest boot messages can be gone on a machine that has been up for a while. dmesg reads the buffer, and /dev/kmsg gives programs access to it: that is how journald and syslog daemons collect kernel messages. Each kernel message has a severity, and the kernel's console log level decides which ones also appear on the console.

Programs using the syslog interface call syslog(3) in the C library, which sends each message as a datagram to the Unix socket /dev/log. The message carries a facility (such as daemon, auth or mail) and a severity (from emerg to debug), the program's name and its process ID, and free text. Whatever process listens on /dev/log receives it; if nothing listens, the message is lost. The logger command does the same from the shell.

Services managed by systemd often don't log at all in the syslog sense: they write to standard output and standard error, and systemd connects those streams to the journal. The journal records each line with the unit it came from, so a service doesn't need to know anything about logging.

Applications with their own log files, such as web servers and databases, open files under /var/log themselves. Nothing in the system logging chain sees those messages; they're managed entirely by the application and by tools like logrotate.

The collectors¶

systemd-journald¶

On a systemd system, systemd-journald is the central collector. It owns /dev/log (as a link to its own socket, /run/systemd/journal/dev-log), reads kernel messages from /dev/kmsg, captures the output of every service, receives messages through its native API, and can also record the kernel's audit events.

The journal is structured. Each entry is a set of fields: the message, its priority and facility, and many fields that journald adds itself and that the sender can't forge, such as the process ID, user ID, executable path, systemd unit and boot ID. The entries are stored in indexed binary files, which is why journalctl can select one unit's messages from the previous boot at a given priority, instantly, without searching through text.

The price is that the files can only be read with journalctl or the systemd libraries. You can't grep them, and a damaged file is harder to salvage than a damaged text file, although journald is designed to write its files so that a crash loses as little as possible.

Syslog daemons¶

Syslog daemons predate systemd. A syslog daemon reads messages, from /dev/log or from the journal, and routes them by facility and severity to text files, to logged-in users, to programs or to other machines. The rule language is the classic selector syntax: authpriv.* /var/log/secure sends every authentication message to one file.

Three syslog daemons appear on the systems covered here:

• rsyslog is the most common. It reads the traditional configuration format and adds its own scripting language, many input and output modules, and reliable queues for forwarding.
• syslog-ng has a different, block-structured configuration language of sources, filters, destinations and log paths; Arch packages it, but doesn't install any syslog daemon by default.
• sysklogd is the descendant of the original BSD and Linux syslogd, kept small. Version 2 adds RFC 5424 support and built-in rotation; remote logging is over UDP only. It is one of several syslog daemons Gentoo offers, and the one the Gentoo lab for these pages runs.

How they connect¶

On a systemd system with a syslog daemon, the journal is always first, and the syslog daemon gets a copy. There are two ways to pass it on:

• Forwarding through a socket. With ForwardToSyslog=yes, journald passes every message it receives to the socket /run/systemd/journal/syslog, where the syslog daemon reads it as if from /dev/log. Debian and Ubuntu set this option in a drop-in file.
• Reading the journal. The syslog daemon reads the journal files itself, through rsyslog's imjournal module or syslog-ng's systemd-journal() source. The RHEL family configures rsyslog this way, and turns off its own socket input. Arch's syslog-ng package does the same.

flowchart LR
    k["Kernel<br/>printk → ring buffer"] -->|/dev/kmsg| j
    p["Programs<br/>syslog(3), logger"] -->|/dev/log| j
    s["systemd services<br/>stdout, stderr"] --> j
    j["systemd-journald<br/>structured, indexed"] --> jv["/run/log/journal<br/>volatile"]
    j --> jp["/var/log/journal<br/>persistent"]
    j -->|ForwardToSyslog socket<br/>or reading the journal| r["rsyslog / syslog-ng"]
    r --> f["Text files<br/>/var/log/syslog, messages, secure"]
    r -->|TCP or UDP 514| remote["Central log server"]

On Gentoo with OpenRC there is no journal. Gentoo leaves the choice of syslog daemon to the administrator (the Handbook suggests sysklogd, syslog-ng and metalog, and others, such as rsyslog, are packaged too); this site's Gentoo pages use sysklogd, as the lab does. The syslog daemon is the only collector: it listens on /dev/log and reads the kernel's messages itself. Services started by OpenRC log through syslog, or to their own files; their standard output goes nowhere in particular unless the service script redirects it.

flowchart LR
    k["Kernel"] -->|/dev/kmsg| d["Syslog daemon<br/>(sysklogd in the lab)"]
    p["Programs and services<br/>syslog(3), logger"] -->|/dev/log| d
    d --> f["/var/log/syslog, messages,<br/>auth.log, kern.log"]

Structured and text logs¶

The two kinds of log answer different needs.

A text log is universal. Every tool can read it, it can be copied, compressed, searched with grep and read on a machine without systemd. But the structure is gone: the facility and severity used to route a message are usually not written into the file, so you can't ask a text file for "errors only" afterwards, unless the daemon was configured to write errors to a file of their own. Time stamps in the traditional file format, which sysklogd and the RHEL family's rsyslog configuration still use, have no year and no time zone.

A structured log keeps every field and makes queries cheap: by unit, by priority, by boot, by user, by time window, or by any field a program attached. It can also be exported as JSON for other tools. But it needs its own reader, and its storage format is specific to one implementation.

Distributions that keep both, such as Ubuntu and the RHEL family, get text files for habit and for tools that expect them, and the journal for queries; the cost is that every message is stored twice.

Persistence¶

journald can keep the journal in memory, in /run/log/journal, which is lost at every reboot, or on disk, in /var/log/journal. The Storage= setting chooses: volatile, persistent, or auto, which means "persistent if /var/log/journal exists". Even with persistent storage, journald writes to /run during early boot, before /var is available, and moves those entries to disk once it is (systemd-journal-flush.service does this at boot).

The upstream default was auto for a long time, so whether a system kept its journal depended on whether the distribution or the installer created /var/log/journal. The labs show how much that varies: Debian 13, Ubuntu 26.04, Arch and Rocky Linux 10 have the directory. On AlmaLinux and Rocky Linux it comes from the systemd package itself, so standard installations have it; but the AlmaLinux 10 cloud image's build removes it, so that image loses its journal at every reboot, while rsyslog keeps text copies in /var/log/messages. Newer systemd releases (259 on Ubuntu 26.04, 262 on Arch) default to persistent, which removes the dependency on the directory.

Text logs written by a syslog daemon are on disk from the start, and survive reboots by default.

Size and rotation¶

Logs grow without end unless something limits them, and a full /var stops services and logins. The two families of collectors solve this differently.

journald limits itself. It splits the journal into files, archives a file when it reaches a size or an age, and deletes the oldest archived files when the journal exceeds its limits: by default 10% of the filesystem, at most 4 GiB, while leaving 15% of the filesystem free. There's nothing to schedule.

Text files need an outside tool. logrotate, run daily by a systemd timer or by cron, renames each file, starts a new one, compresses and eventually deletes the old copies, and tells the daemon to reopen its files. Applications that write their own files rely on the same mechanism, through a rule each package drops into /etc/logrotate.d/. sysklogd can also rotate its files itself, by size, which is useful on small systems without cron; Gentoo's sysklogd package turns this on in /etc/conf.d/sysklogd.

Logs on other machines¶

A log on the machine it describes has two weaknesses: it is lost with the machine's disk, and an intruder with root access can edit it. Sending a copy of every message to a central server addresses both, and lets you search many machines at once.

The traditional transport is the syslog protocol itself: plain messages to UDP port 514, as in the original BSD design and RFC 3164. UDP is simple but unreliable; messages sent while the server is unreachable are lost without notice. TCP (also commonly port 514, with framing described in RFC 6587) lets the client notice failures, and rsyslog adds disk-backed queues to hold messages until the server is back. RFC 5424 defines a newer message format, with full time stamps and structured data, and RFC 5425 defines syslog over TLS, usually on port 6514, which adds encryption and authentication.

systemd offers a separate path, systemd-journal-upload and systemd-journal-remote, which copy journal entries over HTTPS and keep all their fields; it's less widely supported by central log systems than syslog.

Why the families differ¶

The families start from the same parts and make different choices.

Debian stopped installing rsyslog by default and relies on a persistent journal, while still configuring journald to forward to syslog so that installing rsyslog later just works. Ubuntu keeps rsyslog for continuity with tools and habits that expect /var/log/syslog. The RHEL family keeps rsyslog as the route its documentation describes for text files and central logging, and has it read the journal rather than receive a second copy of every message through a socket. Arch installs as little as possible, and the journal covers most needs; a syslog daemon is an addition you choose. Gentoo with OpenRC has no journal to build on, so a traditional syslog daemon is the collector; Gentoo has no preferred one and leaves the choice to you.

Because these choices belong to the installation and not only to the distribution, check the machine in front of you: whether /var/log/journal exists, which syslog daemon (if any) runs, and what its configuration routes where.

Further reading¶

• systemd-journald.service(8), journald.conf(5): systemd documentation
• rsyslog documentation, syslog-ng documentation, sysklogd
• logrotate
• RFC 3164, RFC 5424, RFC 5425, RFC 6587

Related¶

• Reference: journalctl, Syslog facilities and priorities, logrotate directives
• How-to: How to search the logs for a problem, How to keep logs across reboots, How to rotate an application's logs, How to forward logs to a remote syslog server
• Explanation: OpenRC and other init systems, The Linux boot process