Skip to content

How to forward logs to a remote syslog server

A central log server keeps a copy of each machine's logs where an intruder or a failed disk on that machine can't remove them, and lets you search many machines in one place. This guide configures a machine to send all its log messages to such a server over TCP, with a queue that holds messages while the server is unreachable. It uses rsyslog on Debian, Ubuntu, the RHEL family and Gentoo, and syslog-ng on Arch.

Before you begin

  • You need a shell with sudo on the machine that sends the logs (the client).
  • You need a syslog server that accepts messages over TCP, usually on port 514, and its address. Setting up the server is a separate task; on an rsyslog server it means loading the imtcp input module and opening the port in its firewall.
  • Plain TCP syslog isn't encrypted or authenticated. Use it only on a network you trust, such as a management network; across untrusted networks, use syslog over TLS (port 6514) instead. This guide doesn't cover TLS: it needs certificates on both sides, and is configured with rsyslog's TLS network stream drivers (ossl or gtls) or syslog-ng's transport("tls"); see their documentation.

Throughout this guide, replace this placeholder with your own value:

Placeholder Meaning Example
<SERVER> The IP address or host name of the central syslog server 192.0.2.10

Steps

  1. Check that the client can reach the server on TCP port 514:

    $ timeout 3 bash -c 'exec 3<>/dev/tcp/<SERVER>/514' && echo "port 514 is open"
port 514 is open

    If the command prints nothing, or Connection refused, fix the network path or the server's firewall first: nothing you do on the client can work until this succeeds.

  2. Install the syslog daemon that will do the forwarding.

    Ubuntu installs rsyslog by default; Debian 13 doesn't, and keeps its logs only in the systemd journal. Install it where it's missing; apt-get keeps an existing installation as it is:

    sudo apt-get update
sudo apt-get install -y rsyslog

    The journal passes every message on to rsyslog (Debian and Ubuntu set ForwardToSyslog=yes for journald). On Debian, installing rsyslog also starts writing local text files such as /var/log/syslog and /var/log/auth.log, which duplicate the journal on disk; the package's rule in /etc/logrotate.d/rsyslog rotates them if logrotate is installed and scheduled (see How to rotate an application's logs).

    rsyslog is installed and running by default, and reads messages from the journal. Check:

    $ systemctl is-active rsyslog
active

    Arch keeps its logs only in the systemd journal, and rsyslog isn't in its official repositories. Install syslog-ng, which reads the journal and can forward over TCP:

    sudo pacman -S --needed --noconfirm syslog-ng

    Gentoo leaves the choice of syslog daemon to the administrator; this guide starts from sysklogd, as the lab does. sysklogd 2.7 forwards messages only over UDP, which loses messages without notice when the server or the network is down. For TCP with a queue, replace it with rsyslog (if rsyslog is already your syslog daemon, skip to step 3):

    sudo emerge --getbinpkg --noreplace app-admin/rsyslog

    This needs a synchronised ebuild repository (emerge --sync).

  3. Configure forwarding.

    Create /etc/rsyslog.d/90-forward.conf:

    # Forward every message to the central log server over TCP.
*.* action(
    type="omfwd"
    target="<SERVER>"
    port="514"
    protocol="tcp"
    action.resumeRetryCount="-1"
    queue.type="LinkedList"
    queue.filename="forward_central"
    queue.maxDiskSpace="100m"
    queue.saveOnShutdown="on"
)

    Create /etc/rsyslog.d/90-forward.conf:

    # Forward every message to the central log server over TCP.
*.* action(
    type="omfwd"
    target="<SERVER>"
    port="514"
    protocol="tcp"
    action.resumeRetryCount="-1"
    queue.type="LinkedList"
    queue.filename="forward_central"
    queue.maxDiskSpace="100m"
    queue.saveOnShutdown="on"
)

    Arch's /etc/syslog-ng/syslog-ng.conf defines the source s_local, which reads the journal. Add a destination for the server and a log path to it at the end of the file:

    sudo tee -a /etc/syslog-ng/syslog-ng.conf >/dev/null <<'EOF'

# Forward every message to the central log server over TCP.
destination d_central {
    network("<SERVER>" port(514) transport("tcp"));
};
log {
    source(s_local);
    destination(d_central);
};
EOF

    Create /etc/rsyslog.d/90-forward.conf:

    # Forward every message to the central log server over TCP.
*.* action(
    type="omfwd"
    target="<SERVER>"
    port="514"
    protocol="tcp"
    action.resumeRetryCount="-1"
    queue.type="LinkedList"
    queue.filename="forward_central"
    queue.maxDiskSpace="100m"
    queue.saveOnShutdown="on"
)

    In the rsyslog rule, the selector *.* matches every facility and priority, and the omfwd action sends the messages to <SERVER> over TCP. The queue settings give the action its own queue in memory, so a slow or unreachable server doesn't hold up local logging. Because the queue has a file name, it is disk-assisted: when the queue in memory fills up, and when rsyslog stops (queue.saveOnShutdown="on"), messages are written to files in rsyslog's work directory, up to 100 MB. action.resumeRetryCount="-1" makes rsyslog retry indefinitely instead of dropping messages. If the server accepts only UDP, use protocol="udp" instead (or @<SERVER> in the legacy syntax, transport("udp") for syslog-ng); messages sent while it is unreachable are then lost without notice. To forward less, use a narrower selector, such as *.warning or auth,authpriv.* (see Syslog facilities and priorities).

    syslog-ng keeps messages for a TCP destination in memory while it reconnects; it retries every 10 seconds, as set by time_reopen(10) in Arch's configuration.

  4. Check the configuration, then restart the daemon to apply it.

    $ sudo rsyslogd -N1
rsyslogd: version 8.2504.0, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.
$ sudo systemctl restart rsyslog

    $ sudo rsyslogd -N1
rsyslogd: version 8.2510.0-5.el10_2.2, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.
$ sudo systemctl restart rsyslog

    syslog-ng --syntax-only prints nothing when the configuration is valid:

    sudo syslog-ng --syntax-only
sudo systemctl enable --now syslog-ng@default.service

    Check the configuration, then stop sysklogd and start rsyslog in its place, at boot as well. Both daemons read the /dev/log socket, so only one of them may run:

    sudo rsyslogd -N1
sudo rc-service sysklogd stop
sudo rc-update del sysklogd default
sudo rc-update add rsyslog default
sudo rc-service rsyslog start

    Gentoo's rsyslog configuration, in /etc/rsyslog.d/50-default.conf, writes the local files differently from sysklogd: for example, there is no /var/log/syslog, and daemon messages go to /var/log/daemon.log. Its rotation rules in /etc/logrotate.d/rsyslog take effect only once app-admin/logrotate is installed and run regularly (see How to rotate an application's logs).

Verify

Send a test message, and check that the client has a TCP connection to the server:

$ logger -t forward-test "test message for the central log server"
$ ss -tn state established '( dport = :514 )'
Recv-Q Send-Q Local Address:Port   Peer Address:Port
0      0         192.0.2.10:43822    192.0.2.10:514

The output lists one connection to port 514 of <SERVER>. (In this example the client and the server are the same host, so the local and peer addresses are the same.)

Then, on the server, look for the message in the files where it stores remote messages; their location depends on the server's configuration. For example, on an rsyslog server that writes each client's messages to its own file under /var/log/remote/:

$ sudo grep -r forward-test /var/log/remote/
/var/log/remote/client.example.com.log:Nov  7 14:02:11 client forward-test[2291]: test message for the central log server

Troubleshooting

No connection in ss, and the message never arrives
The daemon can't reach the server, or isn't running. Check the client's own log for its error: sudo journalctl -u rsyslog -n 20 (-u syslog-ng@default on Arch), or sudo tail /var/log/messages on Gentoo. Make sure step 1 succeeds.
rsyslogd -N1 reports an error with a line number
A quote or bracket is missing in /etc/rsyslog.d/90-forward.conf. Compare it with the example; each parameter is written name="value".
Messages arrive late, all at once
That is the queue at work: while the server was unreachable, messages were kept and are sent once the connection is back.
Messages arrive twice
The client has two forwarding rules, for example from an earlier @@<SERVER> line in /etc/rsyslog.conf or another file in /etc/rsyslog.d/. Search with grep -r '<SERVER>' /etc/rsyslog.conf /etc/rsyslog.d/ and keep one.