11

On Ubuntu 18.04 both journald and rsyslog are installed. Both serve the same purpose of collecting log messages and storing them. So we have two programs doing the same work here.

What I could find out is that journald only saves log messages in its own journal files which can be queried with the journalctl command. No readable files are written. Also, journal will always collect log output from all the services that were started by systemd, which should always be all of them because ultimately systemd starts all processes even from older schemes like init scripts.

Rsyslog also receives log messages, through some socket that works like syslog has worked the past 50 years or so. Applications have to send their messages to that socket and rsyslog will receive them. Not sure how journald actually receives anything. Then rsyslog forwards these received messages into different readable files as per its configuration.

In Ubuntu, both are installed but do not seem to be connected. At least rsyslog does not load the "imjournal" module with which it could "read" log messages from journald. I don't understand how rsyslog (actively) reads anything, I thought it only (passively) receives messages. But it's not using that anyway.

The file /var/log/syslog contains largely the same content as shown by journalctl, just the output from sudo is missing in the file. But I haven't compared it all.

So how does this work? Who sends log messages to where and who receives them and where are they forwarded to? If both destinations are filled, does every application have to send messages to both syslog and journald separately? If they are interconnected, why are no messages duplicated? Are they filtered out again? Which location is more complete and where should I look if I want the complete picture?

If any of this has changed in newer versions of Ubuntu, I'd also be interested in that, as I'm going to switch an older server to 20.04 next year.

Edit: Here are some more resources on the topic:

All of this information leaves me a bit confused. It seems like journal receives all syslog data from the system call in the first place (my /dev/log points to /run/systemd/journal/dev-log), and rsyslog fetches it from journal through a socket that looks like the socket that is had used before there was journal (possibly /run/systemd/journal/syslog, implicitly used if it sees systemd around). Then rsyslog can do all its advanced log processing (including writing the legacy /var/log files). If I don't need any of that magic, I might just disable rsyslog. But I'm not sure about that at all!

edmz
  • 123
ygoe
  • 988
  • 1
    Same problem. I don't understand why do logs of services running with systemd appear in /var/log/syslog (e.g. docker, cron) Systemd by default sends logs to journal. Journald is not configured to push logs to rsyslog. Rsyslog doesn't use imjournal module. – Dmitry Pugachev Aug 15 '20 at 08:09
  • 1
    This question doesn't seem to be answerable, or the folks who coded that stuff are not around here and nobody else knows what's actually happening on modern Linux systems anymore. That's a bit frightening… Meanwhile I decided to change the rsyslog config to remain completely silent and not write anything except emergency messages to all users' console. I don't need logs twice! I may even go further and uninstall rsyslog completely. I don't need unused software! – ygoe Aug 15 '20 at 12:50

1 Answers1

1

Message storage locations

journald message storage

When it's time to clean up older messages in journald (I do it monthly) you can see the files where messages are stored:

Deleted archived journal /var/log/journal/1ff17e6df1874fb3b2a75e669fa978f1/system@00059368465fc63c-5ca08f36fa6e6f04.journal~ (32.0M).
Deleted archived journal /var/log/journal/1ff17e6df1874fb3b2a75e669fa978f1/user-1000@0005936848a6d1ec-316ff74be646031e.journal~ (8.0M).

As you can see the file names are very long and cryptic.

syslog message storage

syslog messages are kept in a more traditional filename format:

$ ll /var/log/syslog*
-rw-r----- 1 syslog adm   2161 Nov 29 04:47 /var/log/syslog
-rw-r----- 1 syslog adm 159700 Nov 29 04:32 /var/log/syslog.1
-rw-r----- 1 syslog adm  24466 Nov 28 04:27 /var/log/syslog.2.gz
-rw-r----- 1 syslog adm  23489 Nov 27 04:30 /var/log/syslog.3.gz
-rw-r----- 1 syslog adm  28087 Nov 26 04:28 /var/log/syslog.4.gz
-rw-r----- 1 syslog adm  28300 Nov 25 04:30 /var/log/syslog.5.gz
-rw-r----- 1 syslog adm  27445 Nov 24 05:36 /var/log/syslog.6.gz
-rw-r----- 1 syslog adm  27460 Nov 23 08:25 /var/log/syslog.7.gz

The filenames ending in .gz have compressed data to save space on disk.

Messages recorded are not the same

A quick test reveals the two message logging systems aren't exact duplicates. We'll put a message into the system with logger command and then search for it and display the five lines before it.

$ logger $0 "ygoe to two logs?"

$ cat /var/log/syslog | grep ygoe -a5 Nov 27 20:53:35 alien upowerd[2032]: message repeated 3 times: [ (upowerd:2032): UPower-Linux-WARNING **: treating change event as add on /sys/devices/pci0000:00/0000:00:14.0/usb1/1-9/1-9:1.2/0003:046D:C52B.017E/0003:046D:101A.017F/power_supply/hidpp_battery_63] Nov 27 20:54:23 alien upowerd[2032]: (upowerd:2032): UPower-Linux-WARNING **: treating change event as add on /sys/devices/pci0000:00/0000:00:14.0/usb1/1-9/1-9:1.2/0003:046D:C52B.017E/0003:046D:2010.0180/power_supply/hidpp_battery_64 Nov 27 20:59:25 alien wpa_supplicant[1591]: wlp60s0: WPA: Group rekeying completed with ae:20:2e:cc:94:50 [GTK=CCMP] Nov 27 21:00:02 alien CRON[24890]: (root) CMD (/usr/bin/updatedb) Nov 27 21:07:48 alien upowerd[2032]: (upowerd:2032): UPower-Linux-WARNING **: treating change event as add on /sys/devices/pci0000:00/0000:00:14.0/usb1/1-9/1-9:1.2/0003:046D:C52B.017E/0003:046D:2010.0180/power_supply/hidpp_battery_64 Nov 27 21:08:22 alien rick: bash ygoe to two logs?

$ journalctl -xe | grep ygoe -a5 Nov 27 20:59:25 alien wpa_supplicant[1591]: wlp60s0: WPA: Group rekeying completed with ae:20:2e:cc:94:50 [GTK=CCMP] Nov 27 21:00:01 alien CRON[24881]: pam_unix(cron:session): session opened for user root by (uid=0) Nov 27 21:00:01 alien CRON[24890]: (root) CMD (/usr/bin/updatedb) Nov 27 21:00:04 alien CRON[24881]: pam_unix(cron:session): session closed for user root Nov 27 21:07:48 alien upowerd[2032]: (upowerd:2032): UPower-Linux-WARNING **: treating change event as add on /sys/devices/pci0000:00/0000:00:14.0/usb1/1-9/1-9:1.2/0003:046D:C52B.017E/0003:046D:2010.0180/power_supply/hidpp_battery_64 Nov 27 21:08:22 alien rick[8000]: bash ygoe to two logs?

As you can see journalctl provides more details than syslog. Additionally journalctl:

There was debate among developers about duplicating journald and syslog a few years ago however, I couldn't find the link just now.

WinEunuuchs2Unix
  • 102,282
  • After looking at journalctl -o json I immediately believe that journal has more metadata (but does it have more lines/items, i.e. are none missing?), and journalctl is way more comfortable than a plain log file. But I still don't know for sure what route log messages actually go. – ygoe Nov 28 '19 at 22:36
  • 2
    Personally I like journalctl a lot more than syslig. I think the more time you experiment with the various parameters the more you will like it. – WinEunuuchs2Unix Nov 28 '19 at 22:53
  • That's your optinion, and I can understand it, but it doesn't answer my question. – ygoe Nov 29 '19 at 08:36
  • @ygoe Fair enough. I've added the message storage locations to the answer. – WinEunuuchs2Unix Nov 29 '19 at 11:55
  • This does not answer the question. As others note journalctl does store more fields in its database but what determines which messages end up where? The only real difference between your examples is the pam_unix entries which only appear in the journal. How are the written only there and not in the syslog file? – ghostly_s Mar 17 '23 at 06:32