.png?table=block&id=a121a8eb-e744-4819-af3f-4e7022618a37&cache=v2)
History of logging
If each program had to manage its own logging, it would have led to significant challenges with different systems handling logs in various ways. To address this, the Unix world introduced a centralised logging service.
Initially, logs were managed using a utility called
syslog, which was later succeeded by syslog-ng (the new generation) and subsequently rsyslog.rsyslog remains widely used within the GNU/Linux ecosystem. It can receive logs from various programs and networks, storing them in multiple locations (such as files, network locations, or even triggering actions based on log entries). These logs are saved in a text format, making them easily manageable with standard Linux text processing tools like grep and less.However, with the growing dominance of
systemd in the Linux world, logging has also evolved. The logging service in systemd is called journald, and logs are accessed using the journalctl command.Unfortunately, the logs are no longer stored as plain text files.
Anyone who wants to log something will typically send it to the/dev/logor/dev/kmsgdevices, often with the help of additional tools. The logging utility, such asrsyslog, will then read from these devices and process the logs according to its configuration settings.
dmesg
As mentioned earlier, Linux is very log-friendly and will eagerly log everything. But what happens during the boot process? What if the kernel needs to log something before the disks are ready? In such cases, the kernel stores these logs in the Kernel Ring Buffer, which can be accessed using the
dmesg command.log rotation
The
logrotate utility manages log cleanup, ensuring that logs don't fill up the disk space. Its configuration file can be found at /etc/logrotate.conf.~:cat /etc/logrotate.conf # see "man logrotate" for details # global options do not affect preceding include directives # rotate log files weekly weekly # use the adm group by default, since this is the owning group # of /var/log/. su root adm # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # use date as a suffix of the rotated file #dateext # uncomment this if you want your log files compressed #compress # packages drop log rotation information into this directory include /etc/logrotate.d # system-specific logs may also be configured here.
Sometimes there might be services that do create their own logs. You can find them usually in
/etc/logrotate.d~:cat /etc/logrotate.d/apt /var/log/apt/term.log { rotate 12 monthly compress missingok notifempty } /var/log/apt/history.log { rotate 12 monthly compress missingok notifempty }
The
logrotate runs using crons and does its job on a daily basis based on a configuration file located on /etc/cron.daily/logrotate~:cat /etc/cron.daily/logrotate #!/bin/sh # skip in favour of systemd timer if [ -d /run/systemd/system ]; then exit 0 fi # this cronjob persists removals (but not purges) if [ ! -x /usr/sbin/logrotate ]; then exit 0 fi /usr/sbin/logrotate /etc/logrotate.conf EXITVALUE=$? if [ $EXITVALUE != 0 ]; then /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]" fi exit $EXITVALUE
Well known log files
Generally logs are saved at
/var/log. In case of any issue and if you do not know what to do, it is a common practice to run a ls -ltrh /var/log/ to see if any program generated a new log or not.But lets have a look at some of the famous logs:
/var/log/auth.log(in Debian Based)- Authentication processes will log here. Things like
cronjobs, failed logins,sudoinformations, etc
/var/log/syslog- A centralized place most of the logs received by
rsyslogdif a specific logfile is not provided in/etc/rsyslog.conf
/var/log/debug- Debug information from programs
/var/log/kern.log- Kernel messages.
/var/log/messages- Informative messages from services. This is also the default place for logs for remote clients.
/var/run/utmp&/var/log/wtmp- Successful logins.
/var/log/btmp- Failed logins. You can check this to see if anyone is trying to guess your passwords!
/var/log/faillog- Failed authentication attempts.
/var/log/lastlog- Date and time of recent user logins.
Service Logs
Service logs do create files or directories at
/var/log and update their logs there. For example there might be a /var/log/apache2/ (or /var/log/httpd) for the Apache HTTP server or a /var/log/mysql for the MySQL DB.rsyslog
This newer generation of syslog,
rsyslog, is widely used in many environments currently. Its main configuration file is located at /etc/rsyslog.conf, and it also reads any configurations from the /etc/rsyslog.d/ directory. The configuration is divided into three main sections:- MODULES: Specifies the modules to be used, such as enabling
rsyslogto use UDP connections.
- GLOBAL DIRECTIVES: General settings, including directory access and other global configurations.
- RULES: Defines how
rsysloghandles each log, using a combination of facilities, priorities, and actions.
~:cat /etc/rsyslog.conf # /etc/rsyslog.conf configuration file for rsyslog # # For more information install rsyslog-doc and see # /usr/share/doc/rsyslog-doc/html/configuration/index.html # # Default logging rules can be found in /etc/rsyslog.d/50-default.conf ################# #### MODULES #### ################# module(load="imuxsock") # provides support for local system logging #module(load="immark") # provides --MARK-- message capability # provides UDP syslog reception #module(load="imudp") #input(type="imudp" port="514") # provides TCP syslog reception #module(load="imtcp") #input(type="imtcp" port="514") # provides kernel logging support and enable non-kernel klog messages module(load="imklog" permitnonkernelfacility="on") ########################### #### GLOBAL DIRECTIVES #### ########################### # Filter duplicated messages $RepeatedMsgReduction on # # Set the default permissions for all log files. # $FileOwner syslog $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 $Umask 0022 $PrivDropToUser syslog $PrivDropToGroup syslog # # Where to place spool and state files # $WorkDirectory /var/spool/rsyslog # # Include all config files in /etc/rsyslog.d/ # $IncludeConfig /etc/rsyslog.d/*.conf
Facilities could be things like:
kern,user,daemon,cron,auth,ntp,security,console,syslog, ...
Priority could be one of the below:
emerg/panic,alert,crit,err/error,warn/warning,notice,infoordebug.
On the action part we can have things like these:
action | sample | meaning |
filename | /usr/log/logings.log | will write the log to this file |
username | jadi | will notify that person on the screen |
@ip | @192.168.1.100 | will send this log to this log server and that log server will decide what to do with it based on its configs |
So a line like this will show the kernel panics to a remote log server and also will log everything on every level to a log file:
kern.panic @192.168.1.100 *.* /var/log/messages
If you log some specific priority, all the more important things will be logged too! So if you write
cron.notice /var/log/cron/cron.log, you are logging the emerg/panic, alert, critical, error, warning and notice logs of the cron category too.It's important to note that the binary responsible for logging the kern category is a standalone daemon called
klogd. It uses the same configuration files as the other logging services. Why is this significant? Because even if everything else crashes, klogd can still log kernel crashes.logger
If you need to send something toward the
rsyslog, you can use the logger tool.~:logger Testing my lovely tool ~:logger local1.emerg Nothing emergent for sure ~:tail -3 /var/log/syslog 2024-08-25T09:46:30.257550+00:00 boredtoolbox-do-ubuntu systemd[1]: Finished motd-news.service - Message of the Day. 2024-08-25T09:49:56.922651+00:00 boredtoolbox-do-ubuntu root: Testing my lovely tool 2024-08-25T09:50:01.029872+00:00 boredtoolbox-do-ubuntu root: local1.emerg Nothing emergent for sure
Using Python To Log to /var/log/syslog
Sample Python Script to call
Logger :import logging import logging.handlers # Configure the logger logger = logging.getLogger('MyLogger') logger.setLevel(logging.DEBUG) # Set the logging level to DEBUG # Create a SysLogHandler to log to /var/log/syslog syslog_handler = logging.handlers.SysLogHandler(address='/dev/log') syslog_handler.setLevel(logging.DEBUG) # Set the level for this handler # Create a formatter and set it for the handler formatter = logging.Formatter('%(asctime)s %(name)s %(levelname)s: %(message)s') syslog_handler.setFormatter(formatter) # Add the handler to the logger logger.addHandler(syslog_handler) # Example function that performs some action and logs it def perform_action(): try: logger.info('Starting the action.') # Simulate doing something result = 10 / 2 # Example computation logger.info(f'Result of computation: {result}') # Simulate an error # Uncomment the next line to test error logging # raise ValueError('An example error occurred!') logger.info('Action completed successfully.') except Exception as e: logger.error(f'An error occurred: {e}') if __name__ == '__main__': perform_action()
~/temp-scripts:vi log_example.py ~/temp-scripts:chmod +x log_example.py ~/temp-scripts:python3 log_example.py ~/temp-scripts:tail /var/log/syslog 2024-08-26T03:27:43.705176+00:00 boredtoolbox-do-ubuntu systemd[18877]: Listening on dbus.socket - D-Bus User Message Bus Socket. 2024-08-26T03:27:43.705952+00:00 boredtoolbox-do-ubuntu systemd[18877]: Reached target sockets.target - Sockets. 2024-08-26T03:27:43.706156+00:00 boredtoolbox-do-ubuntu systemd[18877]: Reached target basic.target - Basic System. 2024-08-26T03:27:43.707406+00:00 boredtoolbox-do-ubuntu systemd[1]: Started user@0.service - User Manager for UID 0. 2024-08-26T03:27:43.709196+00:00 boredtoolbox-do-ubuntu systemd[18877]: Reached target default.target - Main User Target. 2024-08-26T03:27:43.709427+00:00 boredtoolbox-do-ubuntu systemd[18877]: Startup finished in 168ms. 2024-08-26T03:27:43.713520+00:00 boredtoolbox-do-ubuntu systemd[1]: Started session-51.scope - Session 51 of User root. 2024-08-26T03:46:30.489352+00:00 boredtoolbox-do-ubuntu 2024-08-26 03:46:30,488 MyLogger INFO: Starting the action... 2024-08-26T03:46:30.489772+00:00 boredtoolbox-do-ubuntu 2024-08-26 03:46:30,489 MyLogger INFO: Result: 5.0 2024-08-26T03:46:30.489975+00:00 boredtoolbox-do-ubuntu 2024-08-26 03:46:30,489 MyLogger INFO: Action completed successfully.
Error Example:
~/temp-scripts:python3 log_example.py ~/temp-scripts:tail /var/log/syslog 2024-08-26T03:49:49.560535+00:00 boredtoolbox-do-ubuntu systemd[1]: Finished fwupd-refresh.service - Refresh fwupd metadata and update motd. 2024-08-26T03:50:52.240511+00:00 boredtoolbox-do-ubuntu 2024-08-26 03:50:52,239 MyLogger INFO: Starting the action... 2024-08-26T03:50:52.241037+00:00 boredtoolbox-do-ubuntu 2024-08-26 03:50:52,240 MyLogger INFO: Result: 5.0 2024-08-26T03:50:52.241286+00:00 boredtoolbox-do-ubuntu 2024-08-26 03:50:52,241 MyLogger ERROR: An error occured: An example error occurred! 2024-08-26T03:51:03.437172+00:00 boredtoolbox-do-ubuntu 2024-08-26 03:51:03,436 MyLogger INFO: Starting the action... 2024-08-26T03:51:03.437656+00:00 boredtoolbox-do-ubuntu 2024-08-26 03:51:03,437 MyLogger INFO: Result: 5.0 2024-08-26T03:51:03.437936+00:00 boredtoolbox-do-ubuntu 2024-08-26 03:51:03,437 MyLogger ERROR: An error occured: An example error occurred! 2024-08-26T03:51:12.284312+00:00 boredtoolbox-do-ubuntu 2024-08-26 03:51:12,283 MyLogger INFO: Starting the action... 2024-08-26T03:51:12.284827+00:00 boredtoolbox-do-ubuntu 2024-08-26 03:51:12,284 MyLogger INFO: Result: 5.0 2024-08-26T03:51:12.285094+00:00 boredtoolbox-do-ubuntu 2024-08-26 03:51:12,285 MyLogger ERROR: An error occured: An example error occurred!
Explanation:
- Logger Configuration:
logger.setLevel(logging.DEBUG): Sets the minimum level of severity for the logger.DEBUGlevel ensures that all levels of logs are captured.SysLogHandler(address='/dev/log'): Directs logs to the syslog service. On most systems,/dev/logis the default location where syslog listens for log messages.
- Formatter:
formatter = logging.Formatter('%(asctime)s %(name)s %(levelname)s: %(message)s'): Defines the format of the log messages.
- Logging Actions:
logger.info(),logger.error(): Logs informational messages and errors, respectively.
- Handling Exceptions:
- The script includes basic error handling that logs any exceptions that occur during the execution of
perform_action.
- Logger Configuration:
logger.setLevel(logging.DEBUG): Sets the minimum level of severity for the logger.DEBUGlevel ensures that all levels of logs are captured.SysLogHandler(address='/dev/log'): Directs logs to the syslog service. On most systems,/dev/logis the default location where syslog listens for log messages.
- Formatter:
formatter = logging.Formatter('%(asctime)s %(name)s %(levelname)s: %(message)s'): Defines the format of the log messages.
- Logging Actions:
logger.info(),logger.error(): Logs informational messages and errors, respectively.
- Handling Exceptions:
- The script includes basic error handling that logs any exceptions that occur during the execution of
perform_action.
journald
~:systemctl status systemd-journald ● systemd-journald.service - Journal Service Loaded: loaded (/lib/systemd/system/systemd-journald.service; static) Active: active (running) since Sat 2024-08-24 10:48:06 UTC; 23h ago TriggeredBy: ○ systemd-journald-audit.socket ● systemd-journald.socket ● systemd-journald-dev-log.socket Docs: man:systemd-journald.service(8) man:journald.conf(5) Main PID: 263 (systemd-journal) Status: "Processing requests..." Tasks: 1 (limit: 496) Memory: 21.2M CPU: 9.695s CGroup: /system.slice/systemd-journald.service └─263 /lib/systemd/systemd-journald Aug 24 10:48:06 boredtoolbox-do-ubuntu systemd-journald[263]: Collecting audit messages is disabled. Aug 24 10:48:06 boredtoolbox-do-ubuntu systemd-journald[263]: Journal started Aug 24 10:48:06 boredtoolbox-do-ubuntu systemd-journald[263]: Runtime Journal (/run/log/journal/8f4fb8ec3b4be7883236da53662de4ff) is 576.0K, max 4.4M> Aug 24 10:48:06 boredtoolbox-do-ubuntu systemd-journald[263]: Time spent on flushing to /var/log/journal/8f4fb8ec3b4be7883236da53662de4ff is 151.869m> Aug 24 10:48:06 boredtoolbox-do-ubuntu systemd-journald[263]: System Journal (/var/log/journal/8f4fb8ec3b4be7883236da53662de4ff) is 877.7M, max 885.3> Aug 24 10:48:08 boredtoolbox-do-ubuntu systemd-journald[263]: Received client request to flush runtime journal. Aug 24 10:48:08 boredtoolbox-do-ubuntu systemd-journald[263]: /var/log/journal/8f4fb8ec3b4be7883236da53662de4ff/system.journal: Monotonic clock jumpe> Aug 24 10:48:08 boredtoolbox-do-ubuntu systemd-journald[263]: Rotating system journal. Notice: journal has been rotated since unit was started, output may be incomplete.
~:cat /etc/systemd/journald.conf # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it under the # terms of the GNU Lesser General Public License as published by the Free # Software Foundation; either version 2.1 of the License, or (at your option) # any later version. # # Entries in this file show the compile time defaults. Local configuration # should be created by either modifying this file, or by creating "drop-ins" in # the journald.conf.d/ subdirectory. The latter is generally recommended. # Defaults can be restored by simply deleting this file and all drop-ins. # # Use 'systemd-analyze cat-config systemd/journald.conf' to display the full config. # # See journald.conf(5) for details. [Journal] #Storage=auto #Compress=yes #Seal=yes #SplitMode=uid #SyncIntervalSec=5m #RateLimitIntervalSec=30s #RateLimitBurst=10000 #SystemMaxUse= #SystemKeepFree= #SystemMaxFileSize= #SystemMaxFiles=100 #RuntimeMaxUse= #RuntimeKeepFree= #RuntimeMaxFileSize= #RuntimeMaxFiles=100 #MaxRetentionSec= #MaxFileSec=1month #ForwardToSyslog=no #ForwardToKMsg=no #ForwardToConsole=no #ForwardToWall=yes #TTYPath=/dev/console #MaxLevelStore=debug #MaxLevelSyslog=debug #MaxLevelKMsg=notice #MaxLevelConsole=info #MaxLevelWall=emerg #LineMax=48K #ReadKMsg=yes #Audit=yes
~:journalctl Jul 11 10:48:04 boredtoolbox-do-ubuntu sshd[1419127]: Failed password for root from 80.229.18.62 port 40440 ssh2 Jul 11 10:48:05 boredtoolbox-do-ubuntu systemd-journald[261]: Data hash table of /var/log/journal/8f4fb8ec3b4be7883236da53662de4ff/system.journal has> Jul 11 10:48:05 boredtoolbox-do-ubuntu systemd-journald[261]: /var/log/journal/8f4fb8ec3b4be7883236da53662de4ff/system.journal: Journal header limits> Jul 11 10:48:06 boredtoolbox-do-ubuntu sshd[1419127]: Received disconnect from 80.229.18.62 port 40440:11: Bye Bye [preauth] Jul 11 10:48:06 boredtoolbox-do-ubuntu sshd[1419127]: Disconnected from authenticating user root 80.229.18.62 port 40440 [preauth] Jul 11 10:48:10 boredtoolbox-do-ubuntu sshd[1419130]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=113.200> Jul 11 10:48:12 boredtoolbox-do-ubuntu sshd[1419130]: Failed password for root from 113.200.60.74 port 42655 ssh2 Jul 11 10:48:14 boredtoolbox-do-ubuntu sshd[1419130]: Received disconnect from 113.200.60.74 port 42655:11: Bye Bye [preauth] Jul 11 10:48:14 boredtoolbox-do-ubuntu sshd[1419130]: Disconnected from authenticating user root 113.200.60.74 port 42655 [preauth] Jul 11 10:48:30 boredtoolbox-do-ubuntu sshd[1419132]: Connection closed by 125.124.200.52 port 39482 [preauth] Jul 11 10:48:40 boredtoolbox-do-ubuntu sshd[1419134]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=150.109> Jul 11 10:48:42 boredtoolbox-do-ubuntu sshd[1419134]: Failed password for root from 150.109.18.87 port 48988 ssh2 Jul 11 10:48:43 boredtoolbox-do-ubuntu sshd[1419134]: Received disconnect from 150.109.18.87 port 48988:11: Bye Bye [preauth] Jul 11 10:48:43 boredtoolbox-do-ubuntu sshd[1419134]: Disconnected from authenticating user root 150.109.18.87 port 48988 [preauth] Jul 11 10:48:47 boredtoolbox-do-ubuntu sshd[1419136]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=80.229.> Jul 11 10:48:49 boredtoolbox-do-ubuntu sshd[1419136]: Failed password for root from 80.229.18.62 port 53850 ssh2 Jul 11 10:48:51 boredtoolbox-do-ubuntu sshd[1419136]: Received disconnect from 80.229.18.62 port 53850:11: Bye Bye [preauth] Jul 11 10:48:51 boredtoolbox-do-ubuntu sshd[1419136]: Disconnected from authenticating user root 80.229.18.62 port 53850 [preauth] Jul 11 10:49:11 boredtoolbox-do-ubuntu sshd[1419138]: Invalid user jenkins from 113.200.60.74 port 49077 Jul 11 10:49:11 boredtoolbox-do-ubuntu sshd[1419138]: pam_unix(sshd:auth): check pass; user unknown Jul 11 10:49:11 boredtoolbox-do-ubuntu sshd[1419138]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=113.200> Jul 11 10:49:13 boredtoolbox-do-ubuntu sshd[1419138]: Failed password for invalid user jenkins from 113.200.60.74 port 49077 ssh2 Jul 11 10:49:14 boredtoolbox-do-ubuntu sshd[1419138]: Received disconnect from 113.200.60.74 port 49077:11: Bye Bye [preauth] Jul 11 10:49:14 boredtoolbox-do-ubuntu sshd[1419138]: Disconnected from invalid user jenkins 113.200.60.74 port 49077 [preauth] Jul 11 10:49:27 boredtoolbox-do-ubuntu sshd[1419140]: Connection closed by 125.124.200.52 port 53912 [preauth] Jul 11 10:49:30 boredtoolbox-do-ubuntu sshd[1419142]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=150.109> Jul 11 10:49:31 boredtoolbox-do-ubuntu sshd[1419142]: Failed password for root from 150.109.18.87 port 35458 ssh2 Jul 11 10:49:33 boredtoolbox-do-ubuntu sshd[1419142]: Received disconnect from 150.109.18.87 port 35458:11: Bye Bye [preauth] Jul 11 10:49:33 boredtoolbox-do-ubuntu sshd[1419142]: Disconnected from authenticating user root 150.109.18.87 port 35458 [preauth] Jul 11 10:49:37 boredtoolbox-do-ubuntu sshd[1419144]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=80.229.> Jul 11 10:49:39 boredtoolbox-do-ubuntu sshd[1419144]: Failed password for root from 80.229.18.62 port 48196 ssh2 Jul 11 10:49:40 boredtoolbox-do-ubuntu sshd[1419144]: Received disconnect from 80.229.18.62 port 48196:11: Bye Bye [preauth] Jul 11 10:49:40 boredtoolbox-do-ubuntu sshd[1419144]: Disconnected from authenticating user root 80.229.18.62 port 48196 [preauth] Jul 11 10:50:11 boredtoolbox-do-ubuntu sshd[1419148]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=113.200> Jul 11 10:50:11 boredtoolbox-do-ubuntu sshd[1419146]: error: kex_exchange_identification: read: Connection reset by peer Jul 11 10:50:11 boredtoolbox-do-ubuntu sshd[1419146]: Connection reset by 60.191.20.210 port 23456 Jul 11 10:50:13 boredtoolbox-do-ubuntu sshd[1419148]: Failed password for root from 113.200.60.74 port 55496 ssh2 Jul 11 10:50:14 boredtoolbox-do-ubuntu sshd[1419148]: Received disconnect from 113.200.60.74 port 55496:11: Bye Bye [preauth] Jul 11 10:50:14 boredtoolbox-do-ubuntu sshd[1419148]: Disconnected from authenticating user root 113.200.60.74 port 55496 [preauth] Jul 11 10:50:21 boredtoolbox-do-ubuntu sshd[1419152]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=150.109> Jul 11 10:50:23 boredtoolbox-do-ubuntu sshd[1419152]: Failed password for root from 150.109.18.87 port 50160 ssh2
Querying journal contents
If you run the
journalctl you will get all the logs... too much. So there are some useful switches:Switch | Meaning |
-r | show in reverse; newer on top |
-f | keep showing the tail |
-e | show and go to the end |
-n | show this many lines from the end (newest ones) |
-k | show kernel message (equal to dmesg) |
-b | show from a specific boot, -1 is the previous one, 0 is the current one. You can check the list using --list-boots |
-p | from specific priority, say -p err |
-u | from a specific systemd unit |
You can also use the
--since and --until to show a specific time range, It is possible to provide time as YYYY-MM-DD HH:MM:SS or use things like yesterday, today, tomorrow, now or even --since "10 minutes ago" which is equals to --since "-2 minutes".You can also add the name of the program to the command to see the logs from that specific program; say
journalctl /usr/bin/xrdp or use some fields like PRIORITY=, _PID= and provide values.systemd-cat
This tool is used when you want to manually send logs to the journaling system. It sends its input to the journal or if provided, runs the command and sends the result to the journal.
~:echo "This is my first test" | systemd-cat ~:systemd-cat -p info uptime #sending priority too ~:journalctl -n 3 Aug 25 10:01:40 boredtoolbox-do-ubuntu uptime[12413]: 10:01:40 up 23:13, 1 user, load average: 0.00, 0.00, 0.00 Aug 25 10:01:41 boredtoolbox-do-ubuntu sshd[12411]: Failed password for root from 170.64.169.187 port 54990 ssh2 Aug 25 10:01:42 boredtoolbox-do-ubuntu sshd[12411]: Connection closed by authenticating user root 170.64.169.187 port 54990 [preauth] ~:
Managing Storage
The systemd journals can keep their logs in memeory or write them to the disk or drop all the logs. This is determined by the configurations in
/etc/systemd/journald.conf. But the default behaviour is as follow:- The system checks the
/var/log/journal. If this directory, exists logs will be saved in a directory inside this. The name of the directory will be decided by looking at/etc/machine-id
- If the
/var/log/journalis not there, the logs will be saved in memory at/run/log/jouranland in the directory decided by/etc/machine-id.
But you can overcome these configs by the following sections:
Storage=volatile # keep the logs in memory Storage=persistent # keep on disk, if needed create the directories Storage=auto # will write to disk only if the directory exists Storage=none # do not keep logs
If the logs are being written to disk, these variable will manage the disk usage:
Variable Name | Usage |
SystemMaxUse | The Maximum disk usage, say 500M. The default is on 10% |
SystemKeepFree | Keep at least this much free, say 1G. The default is 15% |
SystemMaxFileSize | Maximum size of each individual file. The default is 1/8 of SystemMaxUse |
SystemMaxFiles | Max number of non-currently-active files. The default is 100 |
If you are keeping the logs in memory, the equivalent variables are
RuntimeMaxUse, RuntimeKeepFree, RuntimeMaxFileSize, & RuntimeMaxFiles.In case you need to do the clean-up manually (with the help of systemd tools for sure), you can run
journalctl with these switches:Switch | Usage |
--vacuum-time | clean everything older than this. for example --vacuum-time=3months clean anything older than 3 months. You can use s, m, h for seconds, minutes and days and d/days, months, weeks/w and years/y. |
--vacuum-size | eliminate till logs occupy a specific size; say 1G |
--vacuum-files | Only keep this much archive files |
If services or applications are configured to use a verbose or debug logging level, journald will record a lot more information. This can lead to large log files if not managed properly.
journald can automatically throttle excessive logging through rate limiting. This is done using two configuration options: RateLimitInterval and RateLimitBurst. These settings are defined in the journald.conf configuration file.Here's how they work:
- RateLimitInterval: This specifies the time interval (in seconds) during which a specific number of messages can be logged. The default is typically 30 seconds.
- RateLimitBurst: This defines the maximum number of log messages that can be logged within the
RateLimitInterval. If the number of messages exceeds this limit, any additional messages will be temporarily dropped until the interval resets.
For example, if you set:
RateLimitInterval=30s
RateLimitBurst=200
journald will allow 200 log messages every 30 seconds from a single service. If more than 200 messages are logged within those 30 seconds, the additional messages will be suppressed until the next interval.These settings help prevent a flood of logs from overwhelming the system, ensuring that logging remains efficient and manageable.
checking logs from a recovered system
If you have a crashed or non-booting system, you can still examine its logs, provided the files are intact. These logs are stored in
/var/log/journal/{machine-id}, where the machine ID is found in /etc/machine-id on the crashed system.You can access these files by booting the crashed machine with a live Linux environment, mounting its hard drive on another machine, or even examining them in place. The
-D (or --directory) switch in journalctl allows you to specify the location of the journal files.For instance, if your crashed machine's ID is
54c29e2891b2aaf910850e79b1ac7b4854c29e2891b2aaf910850e79b1ac7b48 and you've mounted its /var/ directory under /mnt/var/, you can use the following command to read its logs and see what happened:journalctl -D=/mnt/var/log/journal/54c29e2891b2aaf910850e79b1ac7b48/
You can also use the
--merge switch to combine these logs with those on your machine, or use --file to inspect a specific journal file. If you're unsure of the exact location of the journal files, you can use the --root /mnt option to instruct journalctl to search for journal files in that directory.frequent troubleshooting for journald:
1. High Disk Usage
Issue:
journald can consume a lot of disk space, especially if logging levels are set high or if logs are not properly rotated.Troubleshooting Techniques:
- Check Disk Usage:
sudo journalctl --disk-usage
This command shows the total disk space used by journal logs.
- Configure Log Retention:
Edit the
journaldconfiguration file (/etc/systemd/journald.conf) to set limits on disk usage and log retention. For example:
[Journal] SystemMaxUse=1G SystemKeepFree=100M
SystemMaxUse limits the maximum disk space used by journal logs.SystemKeepFree ensures some disk space is kept free.- Manual Cleanup: You can manually delete old logs to free up space:
sudo journalctl --vacuum-size=500M
This command removes the oldest logs until the total disk usage is below 500 MB.
- Automatic Cleanup:
Set up automatic cleanup by configuring
journaldsettings:
[Journal] MaxRetentionSec=1month
This keeps logs for up to one month before deleting them.
2. Missing Logs
Issue: Sometimes logs might be missing or not appearing in
journald.Troubleshooting Techniques:
- Check Configuration:
Ensure
journaldis properly configured to capture logs. Review the configuration file (/etc/systemd/journald.conf) for any settings that might be filtering out logs.
- Verify Service Status:
Make sure
systemd-journaldis running:
sudo systemctl status systemd-journald
Restart the service if needed:
sudo systemctl restart systemd-journald
- Check Permissions:
Verify that the log files in
/var/log/journal/have the correct permissions. Thesystemd-journaldservice should have the necessary permissions to read/write to these files.
- Inspect Log Levels: Some logs might be filtered out due to log level settings. Ensure that the log level is set appropriately to capture all necessary logs:
sudo journalctl -p err..info
This command filters logs based on priority.
- Check for Log Forwarding Issues: If logs are forwarded to another service or destination, ensure that forwarding is configured correctly and that the destination is functioning as expected.
3. Log Corruption
Issue: Logs might become corrupted, leading to errors or missing entries.
Troubleshooting Techniques:
- Check Log Integrity:
Look for any error messages in the system logs related to
journald:
sudo journalctl -u systemd-journald
- Repair Logs:
If you suspect log corruption, you might need to remove the corrupted journal files. This can be done by stopping the
journaldservice, deleting the corrupted files, and then restarting the service:
sudo systemctl stop systemd-journald sudo rm -rf /var/log/journal/* sudo systemctl start systemd-journald
Note: This will delete all existing journal logs. Use this as a last resort.
4. Performance Issues
Issue:
journald might cause performance issues due to high logging volume or excessive querying.Troubleshooting Techniques:
- Optimize Queries: When querying logs, use specific filters to avoid excessive performance overhead:
sudo journalctl -u <service> --since "2024-08-01" --until "2024-08-20"
- Adjust Log Retention: Limit the number of logs retained and the amount of disk space used to minimize performance impacts.
- Monitor Resource Usage:
Keep an eye on the resource usage of
systemd-journald:
sudo systemctl status systemd-journald
