Skip to content

How to search the logs for a problem

A service or scheduled job failed, and you need to find what it logged. This guide narrows the system log down to the messages that matter: by time, by the program that wrote them, by priority and by text. The example is a job called nightly-backup that failed; replace the name and the search text with your own.

Before you begin

  • You need a shell with sudo on the machine. Unprivileged users see only part of the log.
  • Know roughly when the problem happened, and which service or program was involved, if you can.
  • Debian, Ubuntu, the RHEL family and Arch keep their logs in the systemd journal, which you read with journalctl. Gentoo with OpenRC has no journal: a syslog daemon writes text files under /var/log, which you search with grep. Gentoo leaves the choice of daemon to the administrator; this guide uses sysklogd, as the lab does. With another daemon, the file names differ: check its configuration.

Throughout this guide, replace these placeholders with your own values:

Placeholder Meaning Example
<UNIT> The systemd unit or program name to search for nightly-backup
<TEXT> A word or phrase from the error No space left

Steps

  1. Look at everything the service logged recently.

    $ sudo journalctl -u <UNIT> --since "2 hours ago" --no-pager
Nov 07 13:40:02 lab systemd[1]: Started nightly-backup.service - Nightly backup.
Nov 07 13:40:02 lab nightly-backup[1712]: starting backup of /srv/data
Nov 07 13:40:02 lab nightly-backup[1712]: cannot write /srv/backup/data.tar: No space left on device
Nov 07 13:40:02 lab systemd[1]: nightly-backup.service: Main process exited, code=exited, status=1/FAILURE
Nov 07 13:40:02 lab systemd[1]: nightly-backup.service: Failed with result 'exit-code'.

    $ sudo journalctl -u <UNIT> --since "2 hours ago" --no-pager
Nov 07 13:40:02 lab systemd[1]: Started nightly-backup.service - Nightly backup.
Nov 07 13:40:02 lab nightly-backup[1712]: starting backup of /srv/data
Nov 07 13:40:02 lab nightly-backup[1712]: cannot write /srv/backup/data.tar: No space left on device
Nov 07 13:40:02 lab systemd[1]: nightly-backup.service: Main process exited, code=exited, status=1/FAILURE
Nov 07 13:40:02 lab systemd[1]: nightly-backup.service: Failed with result 'exit-code'.

    $ sudo journalctl -u <UNIT> --since "2 hours ago" --no-pager
Nov 07 13:40:02 lab systemd[1]: Started nightly-backup.service - Nightly backup.
Nov 07 13:40:02 lab nightly-backup[1712]: starting backup of /srv/data
Nov 07 13:40:02 lab nightly-backup[1712]: cannot write /srv/backup/data.tar: No space left on device
Nov 07 13:40:02 lab systemd[1]: nightly-backup.service: Main process exited, code=exited, status=1/FAILURE
Nov 07 13:40:02 lab systemd[1]: nightly-backup.service: Failed with result 'exit-code'.

    With sysklogd's default /etc/syslog.conf, /var/log/syslog receives every message except authentication messages. Search it for the program's name:

    $ sudo grep '<UNIT>' /var/log/syslog
Nov  7 13:40:02 lab nightly-backup: starting backup of /srv/data
Nov  7 13:40:02 lab nightly-backup: cannot write /srv/backup/data.tar: No space left on device

    OpenRC doesn't log a service's exit status the way systemd does; check the service's state with rc-service <UNIT> status or rc-status --crashed.

    -u selects the messages of one unit, including systemd's own messages about starting and stopping it. If the program isn't a systemd unit, for example a script run from cron, select it by the name it logs under with -t (the syslog identifier) instead: sudo journalctl -t <UNIT>.

    Note

    If journalctl prints -- No entries -- for a time before the last reboot, the journal on that machine may be kept only in memory. Check with sudo journalctl --list-boots; if only one boot is listed, see How to keep logs across reboots.

  2. If you don't know which service is involved, list only the errors of the current boot.

    sudo journalctl -b -p err --no-pager

    sudo journalctl -b -p err --no-pager

    sudo journalctl -b -p err --no-pager

    The default sysklogd log format doesn't record a message's priority, so you can't filter the files by priority after the event. Search for typical error words instead, ignoring case:

    sudo grep -iE 'error|fail|cannot|denied' /var/log/syslog

    -p err shows messages of priority err and more severe (crit, alert, emerg). Use -p warning to include warnings, and -b -1 instead of -b to look at the previous boot.

  3. Search for the error text across the whole log, to see when it first appeared and whether other programs report it too.

    sudo journalctl -g '<TEXT>' --no-pager

    sudo journalctl -g '<TEXT>' --no-pager

    sudo journalctl -g '<TEXT>' --no-pager

    With Gentoo's /etc/conf.d/sysklogd, sysklogd rotates /var/log/syslog itself when it reaches 10 MiB, keeping the older files as syslog.0 and compressed syslog.1.gz, syslog.2.gz and so on. zgrep searches compressed and uncompressed files alike:

    sudo zgrep -h '<TEXT>' /var/log/syslog*

    -g takes a regular expression, and matches without regard to case unless the pattern contains an upper-case letter.

  4. Read what else happened on the system around that time. Errors in one program often follow a problem elsewhere, such as a full disk, a network outage or the kernel killing a process. Take the time of the error from the previous steps and read the messages from every source around it:

    sudo journalctl --since "2 hours ago" --until now -o short-iso --no-pager

    sudo journalctl --since "2 hours ago" --until now -o short-iso --no-pager

    sudo journalctl --since "2 hours ago" --until now -o short-iso --no-pager

    Show the lines before and after the error, and check the kernel's messages separately:

    sudo grep -B 10 -A 10 '<TEXT>' /var/log/syslog
sudo grep -iE 'error|fail|oom|killed' /var/log/kern.log

    The example shows the last two hours; for a real investigation, give the exact window around the error, such as --since "2026-11-07 13:35" --until "2026-11-07 13:45". -o short-iso prints full ISO 8601 timestamps, which are easier to compare than the default format. To see only kernel messages, add -k on the systemd families.

Verify

Check that you're searching the right place, by writing a test message of your own and finding it again. logger sends a message to the system log with the tag and priority you give it:

logger -t log-search-test -p user.err "test message from logger"

$ sudo journalctl -t log-search-test --no-pager
Nov 07 13:52:40 lab log-search-test[1890]: test message from logger

$ sudo journalctl -t log-search-test --no-pager
Nov 07 13:52:40 lab log-search-test[1890]: test message from logger

$ sudo journalctl -t log-search-test --no-pager
Nov 07 13:52:40 lab log-search-test[1890]: test message from logger

$ sudo grep log-search-test /var/log/syslog
Nov  7 13:52:40 lab log-search-test: test message from logger

If the test message appears but the messages you're looking for don't, the program logs somewhere else: look for its own files under /var/log/<UNIT>/ or in its configuration.

Troubleshooting

Hint: You are currently not seeing messages from other users and the system.
You ran journalctl without sudo, as a user who isn't in the systemd-journal or adm group, so only your own messages are shown. Run it with sudo, or add the user to one of those groups.
-- No entries -- for a unit you know exists
Check the unit's exact name with systemctl list-units --all '<UNIT>*': -u needs the full name, although the .service suffix can be left out. Also check the time window: --since without a date means today.
grep: /var/log/syslog: No such file or directory
No syslog daemon writes that file. On Debian 13 and Arch no syslog daemon is installed by default, so use journalctl. On the RHEL family, rsyslog writes /var/log/messages instead. On Gentoo, check that the syslog daemon is running with rc-service sysklogd status.
Messages from before the last reboot are missing
The journal is kept only in memory (in /run/log/journal) on this machine. See How to keep logs across reboots.