How to keep logs across reboots¶
After a crash or an unexpected reboot, the messages you need are the ones from before it.
On some systems the systemd journal is kept only in memory, under /run/log/journal, and is lost at every reboot.
This guide makes the journal persistent with a limit on the disk space it may use, or, on Gentoo with OpenRC, checks that the syslog files persist and sets their size, and then proves it with a reboot.
Before you begin¶
- You need a shell with
sudoon the machine, and you must be able to reboot it. - Make sure the filesystem holding
/var/loghas room for the limit you choose: check withdf -h /var/log.
Whether the journal is persistent already depends on the systemd version and on the installation, not only on the distribution:
- With
Storage=auto, the default up to systemd 257 (Debian 13, AlmaLinux 10, Rocky Linux 10), the journal is persistent only if the directory/var/log/journalexists. Most installations create it; the AlmaLinux 10 cloud image, for example, doesn't. - Newer systemd releases, such as 259 on Ubuntu 26.04 and 262 on Arch, default to
Storage=persistent.
Setting Storage=persistent explicitly, as below, gives the same result everywhere and adds a size limit of your choosing.
Steps¶
-
Check where the logs are kept now.
$ sudo journalctl --list-boots IDX BOOT ID FIRST ENTRY LAST ENTRY 0 946456f3acdd4507bc8165ba12e04ed7 Sat 2026-11-07 13:23:06 UTC Sat 2026-11-07 13:41:12 UTC $ sudo journalctl --header | grep 'File path' File path: /var/log/journal/ad8948ea882649bb8419d974d1bcf343/system.journal File path: /var/log/journal/ad8948ea882649bb8419d974d1bcf343/user-1000.journal$ sudo journalctl --list-boots IDX BOOT ID FIRST ENTRY LAST ENTRY 0 0ef9a283824143e4adab38ea7e7b100f Sat 2026-11-07 13:23:05 UTC Sat 2026-11-07 13:41:13 UTC $ sudo journalctl --header | grep 'File path' File path: /run/log/journal/0ce30a64d8fb49b392bfe8c573aa15fc/system.journalThis output is from the AlmaLinux 10 cloud image, whose build removes
/var/log/journal. Other installations, including Rocky Linux 10's cloud image, normally show/var/log/journal, which thesystemdpackage creates. On a machine like the AlmaLinux image,rsyslogstill copies messages into text files such as/var/log/messagesand/var/log/secure, which persist; but it doesn't keep the journal's fields, and the journal itself is lost.With OpenRC there is no journal. Gentoo leaves the choice of syslog daemon to the administrator; this guide uses
sysklogd, as the lab does. It writes text files in/var/log, which is on disk, so they already persist across reboots. Check that the daemon is running and which files it writes:A path under
/run/log/journalmeans the journal is volatile and is lost at the next reboot. A path under/var/log/journalmeans it's already persistent; continue anyway to set a size limit. -
Configure persistent storage and its limits.
Create a drop-in directory for journald's configuration:
Create
/etc/systemd/journald.conf.d/50-persistent.conf:Create a drop-in directory for journald's configuration:
Create
/etc/systemd/journald.conf.d/50-persistent.conf:Create a drop-in directory for journald's configuration:
Create
/etc/systemd/journald.conf.d/50-persistent.conf:sysklogdrotates its own files by size. Gentoo's/etc/conf.d/sysklogdstarts it with-r 10M:10: each file is rotated when it reaches 10 MiB; the newest old copy stays uncompressed, and older ones are compressed with gzip. The number after the colon limits the compressed copies, sosysklogd2.7 keeps one more old copy than that number: with:10,syslog.0andsyslog.1.gztosyslog.10.gz. Change the size, here to 50 MiB per file:$ grep SYSLOGD= /etc/conf.d/sysklogd SYSLOGD="-m 0 -s -s -r 10M:10" $ sudo sed -i 's/-r 10M:10/-r 50M:10/' /etc/conf.d/sysklogd $ grep SYSLOGD= /etc/conf.d/sysklogd SYSLOGD="-m 0 -s -s -r 50M:10"Each log file then takes at most 50 MiB for the current file and 50 MiB for the newest rotated copy, plus the compressed older copies. If you rotate the files with
logrotateinstead, remove the-roption, so that the two don't both rotate the same files.The settings in the drop-in file mean:
Setting Effect Storage=persistentStore the journal in /var/log/journal, creating the directory if neededSystemMaxUse=1GThe journal may use at most 1 GiB of disk space SystemKeepFree=2GAlways leave at least 2 GiB free on the filesystem, even if that means using less than SystemMaxUseMaxRetentionSec=3monthDelete journal files whose entries are all older than three months When a limit is reached, journald deletes the oldest archived journal files first. Without
SystemMaxUseandSystemKeepFree, journald limits itself to a share of the size of the filesystem (see journalctl). -
Apply the configuration.
Restarting
systemd-journaldmakes it read the new configuration and create/var/log/journal.journalctl --flushmoves the entries still held in/run/log/journalinto the persistent journal, so that the current boot's earlier messages are kept too. -
Write a test message, and reboot the machine:
Verify¶
After the machine is back, log in again and look for the test message from the previous boot.
$ sudo journalctl --list-boots
IDX BOOT ID FIRST ENTRY LAST ENTRY
-1 946456f3acdd4507bc8165ba12e04ed7 Sat 2026-11-07 13:23:06 UTC Sat 2026-11-07 13:44:10 UTC
0 5b1f0c39d5a44e44a0b5e8f4f5f8a1c7 Sat 2026-11-07 13:44:21 UTC Sat 2026-11-07 13:44:40 UTC
$ sudo journalctl -b -1 -t reboot-test --no-pager
Nov 07 13:44:05 lab reboot-test[1540]: written before the reboot
$ sudo journalctl --list-boots
IDX BOOT ID FIRST ENTRY LAST ENTRY
-1 0ef9a283824143e4adab38ea7e7b100f Sat 2026-11-07 13:23:05 UTC Sat 2026-11-07 13:44:10 UTC
0 3c6d1f2b8e9a4c0d9f7e6a5b4c3d2e1f Sat 2026-11-07 13:44:21 UTC Sat 2026-11-07 13:44:40 UTC
$ sudo journalctl -b -1 -t reboot-test --no-pager
Nov 07 13:44:05 lab reboot-test[1540]: written before the reboot
$ sudo journalctl --list-boots
IDX BOOT ID FIRST ENTRY LAST ENTRY
-1 d657631e56b14174bca5504f99625a1e Sat 2026-11-07 13:23:05 UTC Sat 2026-11-07 13:44:10 UTC
0 8a7b6c5d4e3f4a2b9c1d0e9f8a7b6c5d Sat 2026-11-07 13:44:21 UTC Sat 2026-11-07 13:44:40 UTC
$ sudo journalctl -b -1 -t reboot-test --no-pager
Nov 07 13:44:05 lab reboot-test[1540]: written before the reboot
journalctl --list-boots lists more than one boot, and the message written before the reboot is shown from boot -1, the previous one.
Check how much space the journal uses with journalctl --disk-usage.
Troubleshooting¶
journalctl -b -1reportsNo journal boot entry found for the specified boot (-1).orSpecifying boot ID or boot offset has no effect, no persistent journal was found.- The previous boot wasn't recorded in a persistent journal. The second message, shown when the journal is still volatile, comes with the current boot's entries instead of an error. Check the drop-in file for typing errors with
systemd-analyze cat-config systemd/journald.conf, and that/var/log/journalexists. Messages from boots before the change are lost; only the boots after it are kept. - The journal uses less space than
SystemMaxUse SystemKeepFreetakes priority: if the filesystem has less free space than that, journald keeps the journal smaller. Check withdf -h /var/logand lowerSystemKeepFreeif needed.- You need to free space now
- Remove the oldest archived journal files down to a size or an age:
sudo journalctl --vacuum-size=500Morsudo journalctl --vacuum-time=2weeks. Vacuuming only removes archived files; runsudo journalctl --rotatefirst to archive the active ones too. - Old messages are missing from
/var/log/syslogon Gentoo sysklogdrotated the file. Search the rotated files as well, withsudo zgrep -h '<TEXT>' /var/log/syslog*, where<TEXT>is the text you're looking for.
Related¶
- Reference: journalctl, logrotate directives
- Explanation: Linux logging architecture
- How-to: How to search the logs for a problem, How to rotate an application's logs, How to forward logs to a remote syslog server