Last modified: May 03, 2026
This article is written in: πΊπΈ
Understanding how logging works in Linux is like learning the language your system uses to communicate. Logs are the detailed records that your system keeps about its activities, and they are invaluable for troubleshooting, monitoring performance, and ensuring security. Let's embark on a journey to demystify log files, journals, and the various logging systems used in Linux.
A log is a record of events produced by an operating system, application, service, script, device, or security component.
Logs answer questions like:
A typical log entry contains:
+--------------------------------------------------------------------------------+
| TIMESTAMP | SEVERITY | SOURCE / SERVICE | MESSAGE |
|---------------------|----------|------------------|----------------------------|
| 2026-05-03 09:00:01 | INFO | nginx | Server started |
| 2026-05-03 09:01:10 | WARNING | kernel | CPU temperature high |
| 2026-05-03 09:02:44 | ERROR | myapp | Database connection failed |
+--------------------------------------------------------------------------------+
May 03 09:02:44 web01 myapp[1234]: ERROR Database connection failed
β β β β β
β β β β βββ Message
β β β βββββββββββ Process ID
β β βββββββββββββββββ Service / program name
β βββββββββββββββββββββββ Hostname
ββββββββββββββββββββββββββββββββββββββ Timestamp
Logs are used for:
+--------------------+--------------------------------------------------+
| Purpose | Example |
|--------------------|--------------------------------------------------|
| Troubleshooting | Why did nginx fail to start? |
| Security | Who tried to SSH into the server? |
| Auditing | Who used sudo? |
| Monitoring | Is disk space filling up? |
| Performance | Why is the app slow? |
| Compliance | Can we prove access attempts were recorded? |
+--------------------+--------------------------------------------------+
Without logs, debugging becomes guessing.
With logs, debugging becomes investigation.
Linux systems usually have more than one logging system working together.
+----------------------+
| Applications |
| Python, nginx, SSHD |
+----------+-----------+
|
v
+-------------+ +---------+----------+ +----------------+
| Kernel |---------> | systemd-journald | -------> | journalctl |
| hardware | | binary journal | | query journal |
+-------------+ +---------+----------+ +----------------+
|
v
+----------+-----------+
| rsyslog / syslog |
| text log routing |
+----------+-----------+
|
v
+----------+-----------+
| /var/log/*.log |
| plain text logs |
+----------------------+
Modern Linux systems commonly use:
| System | Purpose |
| journald | Collects structured logs from systemd, kernel, apps |
| journalctl | Reads and filters journald logs |
| rsyslog | Routes logs into text files or remote log servers |
| /var/log | Directory where many plain-text logs live |
| logrotate | Rotates, compresses, and deletes old text logs |
| logger | Sends custom log messages from shell scripts |
| dmesg | Reads kernel ring buffer messages |
Most traditional logs live under:
/var/log
Example:
/var/log/
βββ syslog
βββ auth.log
βββ kern.log
βββ dmesg
βββ boot.log
βββ dpkg.log
βββ apt/
β βββ history.log
β βββ term.log
βββ nginx/
β βββ access.log
β βββ error.log
βββ apache2/
β βββ access.log
β βββ error.log
βββ mysql/
βββ postgresql/
βββ journal/
βββ rotated logs:
βββ syslog.1
βββ syslog.2.gz
βββ auth.log.1.gz
Different distributions use different files.
For example:
Debian / Ubuntu:
/var/log/syslog
/var/log/auth.log
RHEL / CentOS / Fedora:
/var/log/messages
/var/log/secure
Plain text logs are ordinary files you can read with tools like:
catlesstailgrepawksedExample:
sudo tail -n 50 /var/log/syslog
Example log lines:
May 03 10:15:42 web01 NetworkManager[1234]: device eth0 state changed
May 03 10:15:45 web01 kernel: eth0: Link is Down
May 03 10:16:01 web01 CRON[2222]: (root) CMD (/usr/local/bin/backup.sh)
May 03 10:16:01 web01 CRON[2222]: (root) CMD (/usr/local/bin/backup.sh)
β β β β β
β β β β βββ Message
β β β βββββββββββ PID
β β ββββββββββββββββ Program
β βββββββββββββββββββββ Hostname
ββββββββββββββββββββββββββββββββββββ Timestamp
journalctlMany modern Linux distributions use systemd-journald.
journald collects logs from:
+-------------------------+
| systemd-journald |
+-----------+-------------+
^
|
+-----------+------------+----------------+----------------+----------------+
| | | | |
| systemd services | kernel | applications | stdout/stderr |
| nginx.service | hardware | custom apps | service logs |
+------------------------+----------------+----------------+----------------+
Unlike plain text logs, the systemd journal is structured and usually stored in binary format.
You do not normally read the journal file directly.
You use:
journalctl
journalctl CommandsView all logs:
journalctl
This command displays all logs stored in the systemd journal. It shows logs from system services, kernel messages, user sessions, boot activity, and other system events. By default, the output is shown in a pager, so you can scroll through it using keyboard controls.
Example output:
May 03 09:15:01 server systemd[1]: Started Session 12 of User root.
May 03 09:15:05 server sshd[1245]: Accepted publickey for admin from 192.168.1.20
May 03 09:15:10 server sudo[1302]: admin : TTY=pts/0 ; COMMAND=/usr/bin/systemctl status nginx
Explanation:
q to exit the log viewer.View newest logs:
journalctl -n 50
This command displays the newest 50 log entries. It is useful when you only want to inspect the most recent system activity instead of scrolling through the entire journal.
Example output:
May 03 10:01:22 server nginx[2210]: Configuration file /etc/nginx/nginx.conf test is successful
May 03 10:01:23 server systemd[1]: Reloaded nginx.service - A high performance web server
May 03 10:02:01 server CRON[2250]: pam_unix(cron:session): session opened for user root
Explanation:
-n 50 limits the output to the last 50 log lines.50 to any number, such as 100 or 200.Follow logs live:
journalctl -f
This command follows logs in real time. New log entries appear automatically as they are written to the journal. It works similarly to tail -f.
Example output:
May 03 10:05:12 server sshd[2401]: Failed password for invalid user test from 203.0.113.10
May 03 10:05:15 server sshd[2401]: Connection closed by invalid user test 203.0.113.10
May 03 10:05:20 server nginx[2410]: 192.168.1.25 - - "GET / HTTP/1.1" 200
Explanation:
Ctrl + C to stop following logs.View logs since boot:
journalctl -b
This command displays logs from the current system boot only. It filters out logs from previous boots and shows what has happened since the machine last started.
Example output:
May 03 08:00:01 server kernel: Linux version 6.8.0
May 03 08:00:04 server systemd[1]: Starting system initialization...
May 03 08:00:15 server systemd[1]: Started ssh.service - OpenSSH server daemon
Explanation:
-b means current boot.View previous boot:
journalctl -b -1
This command shows logs from the previous boot. It is useful when the system crashed, rebooted unexpectedly, or had an issue before the current startup.
Example output:
May 02 22:41:03 server kernel: Out of memory: Killed process 1884
May 02 22:41:10 server systemd[1]: nginx.service: Failed with result 'exit-code'
May 02 22:42:01 server systemd[1]: Reached target Reboot
Explanation:
-b -1 means one boot before the current boot.journalctl --list-boots can be used to see available boot records.View logs from the last hour:
journalctl --since "1 hour ago"
This command displays logs generated during the last hour. It is helpful when an issue happened recently and you do not want to search through older logs.
Example output:
May 03 09:12:44 server docker[1805]: Container web_app started
May 03 09:25:10 server nginx[1921]: connect() failed while connecting to upstream
May 03 09:58:31 server sshd[2150]: Accepted password for deploy from 192.168.1.30
Explanation:
--since filters logs by start time."1 hour ago" is a relative time expression."10 minutes ago" or "yesterday".View logs from a time range:
journalctl --since "2026-05-03 09:00" --until "2026-05-03 10:00"
This command displays logs between a specific start time and end time. It is useful when you know exactly when a problem occurred and want to inspect only that period.
Example output:
May 03 09:05:14 server nginx[1602]: 502 Bad Gateway while reading response header from upstream
May 03 09:20:44 server postgresql[1710]: checkpoint complete
May 03 09:45:02 server systemd[1]: Started cleanup temporary files
Explanation:
--since defines the beginning of the time range.--until defines the end of the time range.YYYY-MM-DD HH:MM.journalctl -u ssh.service
journalctl -u nginx.service
journalctl -u docker.service
journalctl -u postgresql.service
These commands display logs for specific systemd services. Filtering by service is one of the most common ways to use journalctl because it removes unrelated system messages and focuses only on the service you are troubleshooting.
Example output:
May 03 10:12:01 server sshd[2501]: Server listening on 0.0.0.0 port 22
May 03 10:12:08 server sshd[2510]: Accepted publickey for admin from 192.168.1.20
May 03 10:12:10 server sshd[2510]: pam_unix(sshd:session): session opened for user admin
Explanation:
-u filters logs by systemd unit name..service.ssh.service, nginx.service, docker.service, and postgresql.service.Follow one service live:
journalctl -u nginx.service -f
This command follows only the logs from nginx.service in real time. It is useful when testing configuration changes, watching requests, or troubleshooting live service behavior.
Example output:
May 03 10:20:01 server nginx[2701]: 192.168.1.50 - - "GET /api/status HTTP/1.1" 200
May 03 10:20:08 server nginx[2701]: 192.168.1.51 - - "POST /login HTTP/1.1" 302
May 03 10:20:12 server nginx[2701]: connect() failed while connecting to upstream
Explanation:
-u nginx.service filters logs to Nginx only.-f follows new log entries live.Ctrl + C to stop.Show only recent logs for a service:
journalctl -u nginx.service -n 100
This command displays the most recent 100 log entries for nginx.service. It is useful when you want recent service logs without following them live.
Example output:
May 03 10:30:05 server nginx[2801]: signal process started
May 03 10:30:06 server systemd[1]: Reloaded nginx.service - A high performance web server
May 03 10:31:10 server nginx[2801]: 192.168.1.70 - - "GET /health HTTP/1.1" 200
Explanation:
-u nginx.service limits logs to the Nginx service.-n 100 shows the last 100 entries.journalctl -p err
Shows errors and anything more severe.
This command filters logs by priority level. The err level shows error messages and more severe messages such as critical, alert, and emergency logs. It is useful when you want to quickly find serious problems without reading informational logs.
Example output:
May 03 10:40:11 server nginx[3001]: connect() failed while connecting to upstream
May 03 10:41:03 server kernel: EXT4-fs error on device sda1
May 03 10:42:18 server systemd[1]: docker.service: Failed with result 'exit-code'
Explanation:
-p err shows priority level err and anything more severe.-u to filter errors for a specific service.Priority levels:
| Name | Code | Meaning |
| emerg | 0 | System unusable |
| alert | 1 | Immediate action required |
| crit | 2 | Critical condition |
| err | 3 | Error |
| warning | 4 | Warning |
| notice | 5 | Normal but important |
| info | 6 | Informational |
| debug | 7 | Debug messages |
Examples:
journalctl -p warning
This command shows warning messages and anything more severe. It is useful when you want to see potential problems before they become errors.
Example output:
May 03 10:50:01 server nginx[3100]: conflicting server name ignored
May 03 10:51:22 server kernel: CPU temperature above threshold
May 03 10:52:05 server systemd[1]: service restart operation timed out
Explanation:
warning, err, crit, alert, and emerg messages.-p err.journalctl -p err -u ssh.service
This command shows only error-level and more severe logs for the SSH service. It is useful when troubleshooting failed SSH logins, SSH service failures, or authentication problems.
Example output:
May 03 11:00:12 server sshd[3301]: error: kex_exchange_identification: client sent invalid protocol identifier
May 03 11:01:44 server sshd[3310]: fatal: Timeout before authentication
May 03 11:03:01 server sshd[3322]: error: PAM: Authentication failure for illegal user test
Explanation:
-p err filters by error severity.-u ssh.service limits logs to SSH.journalctl -p debug -u myapp.service
This command shows debug-level logs for myapp.service. Since debug is the lowest priority level, this may include very detailed messages depending on how the service logs its output.
Example output:
May 03 11:10:01 server myapp[3500]: DEBUG Loading configuration from /etc/myapp/config.yml
May 03 11:10:02 server myapp[3500]: DEBUG Database connection pool initialized
May 03 11:10:03 server myapp[3500]: INFO Application started successfully
Explanation:
-p debug includes debug logs and all higher-priority messages.-u to avoid too much unrelated output.Normal output:
journalctl -u nginx
This command shows logs for the Nginx service using the default journal output format. The default format is readable and suitable for most manual troubleshooting tasks.
Example output:
May 03 11:20:01 server nginx[3700]: 192.168.1.80 - - "GET / HTTP/1.1" 200
May 03 11:20:05 server nginx[3700]: 192.168.1.81 - - "GET /favicon.ico HTTP/1.1" 404
May 03 11:20:10 server systemd[1]: Reloaded nginx.service - A high performance web server
Explanation:
Short ISO timestamps:
journalctl -u nginx -o short-iso
This command shows Nginx logs with ISO-style timestamps. This format is useful when you need clearer timestamps, especially for comparing logs across systems or matching logs with external monitoring tools.
Example output:
2026-05-03T11:25:01+0200 server nginx[3800]: 192.168.1.90 - - "GET / HTTP/1.1" 200
2026-05-03T11:25:03+0200 server nginx[3800]: 192.168.1.91 - - "POST /api/login HTTP/1.1" 401
2026-05-03T11:25:08+0200 server nginx[3800]: upstream timed out while reading response header
Explanation:
-o short-iso changes the output format.Verbose metadata:
journalctl -u nginx -o verbose
This command shows detailed metadata for each journal entry. It includes fields such as systemd unit name, process ID, executable path, hostname, priority, and other internal journal fields.
Example output:
MESSAGE=192.168.1.90 - - "GET / HTTP/1.1" 200
_PID=3800
_UID=33
_GID=33
_SYSTEMD_UNIT=nginx.service
_COMM=nginx
_HOSTNAME=server
PRIORITY=6
Explanation:
-o verbose shows detailed journal fields.JSON output:
journalctl -u nginx -o json
This command outputs each journal entry as a single JSON object. It is useful when logs need to be processed by scripts, command-line tools, or log aggregation systems.
Example output:
{"MESSAGE":"192.168.1.90 - - \"GET / HTTP/1.1\" 200","_PID":"3800","_SYSTEMD_UNIT":"nginx.service","PRIORITY":"6","_HOSTNAME":"server"}
{"MESSAGE":"upstream timed out while reading response header","_PID":"3800","_SYSTEMD_UNIT":"nginx.service","PRIORITY":"3","_HOSTNAME":"server"}
Explanation:
-o json prints logs as JSON.jq.Pretty JSON:
journalctl -u nginx -o json-pretty
This command outputs journal entries as formatted JSON. It is easier for humans to read than normal JSON output because fields are split across multiple lines with indentation.
Example output:
{
"MESSAGE" : "192.168.1.90 - - \"GET / HTTP/1.1\" 200",
"_PID" : "3800",
"_SYSTEMD_UNIT" : "nginx.service",
"PRIORITY" : "6",
"_HOSTNAME" : "server"
}
Explanation:
-o json-pretty formats JSON across multiple lines.-o json.This is useful when feeding logs into scripts or log aggregation systems. JSON formats allow tools to read fields such as service name, priority, process ID, hostname, and message without relying on text parsing. For quick manual troubleshooting, normal or short output is usually easier to read. For automation and structured logging workflows, JSON output is usually better.
A useful way to understand logs is by layers.
| Layer | Components |
| Application Layer | Python apps, nginx, Apache, PostgreSQL, Docker apps |
| Service Layer | systemd services, cron, SSH, NetworkManager |
| OS Layer | package manager, sudo, auth, syslog |
| Kernel Layer | drivers, hardware, memory, disk, CPU, networking |
| Boot Layer | bootloader, initramfs, systemd startup |
Kernel logs are useful for debugging:
Kernel logs come from the Linux kernel rather than from normal user-space applications. They are especially useful when troubleshooting low-level system problems, such as a disk failing, a network card disconnecting, a USB device not being detected, a driver crashing, or the system reporting CPU or memory warnings.
Commands:
dmesg
This command displays messages from the kernel ring buffer. These messages usually include boot messages, device detection, driver messages, hardware warnings, and kernel-level errors.
Example output:
[ 0.000000] Linux version 6.8.0-31-generic
[ 1.245011] usb 1-1: new high-speed USB device number 2 using xhci_hcd
[ 2.884310] eth0: renamed from enp0s3
[ 15.902144] EXT4-fs (sda1): mounted filesystem
Explanation:
dmesg -T
This command displays kernel messages with human-readable timestamps. Instead of showing only seconds since boot, it converts timestamps into normal date and time format.
Example output:
[Sun May 3 09:01:10 2026] Linux version 6.8.0-31-generic
[Sun May 3 09:01:12 2026] usb 1-1: new high-speed USB device number 2 using xhci_hcd
[Sun May 3 09:01:15 2026] eth0: renamed from enp0s3
[Sun May 3 09:02:01 2026] EXT4-fs (sda1): mounted filesystem
Explanation:
-T shows readable timestamps.journalctl -k
This command displays kernel messages from the systemd journal. It is similar to dmesg, but it reads kernel logs from the journal instead of only the kernel ring buffer.
Example output:
May 03 09:01:10 server kernel: Linux version 6.8.0-31-generic
May 03 09:01:12 server kernel: usb 1-1: new high-speed USB device number 2 using xhci_hcd
May 03 09:01:15 server kernel: eth0: renamed from enp0s3
May 03 09:02:01 server kernel: EXT4-fs (sda1): mounted filesystem
Explanation:
-k shows kernel messages from the journal.journalctl filtering options.journalctl -k -b
This command displays kernel messages from the current boot only. It is useful when troubleshooting hardware, driver, or boot-related issues from the current running session.
Example output:
May 03 09:01:10 server kernel: Linux version 6.8.0-31-generic
May 03 09:01:13 server kernel: ACPI: bus type USB registered
May 03 09:01:18 server kernel: e1000e 0000:00:19.0 eth0: NIC Link is Up
May 03 09:04:44 server kernel: EXT4-fs (sda1): re-mounted filesystem
Explanation:
-k filters journal entries to kernel messages.-b limits output to the current boot.Examples:
dmesg | grep -i error
This command searches kernel messages for the word error, ignoring letter case. It is useful for quickly finding kernel-level failures.
Example output:
[ 1234.442100] EXT4-fs error (device sda1): ext4_find_entry: inode read error
[ 1240.112901] blk_update_request: I/O error, dev sda, sector 884120
[ 1244.650331] usb 2-1: device descriptor read/64, error -71
Explanation:
grep -i error searches for error, Error, or ERROR.failed, timeout, or reset.fail, warn, and timeout.dmesg | grep -i usb
This command filters kernel messages related to USB devices. It is useful when checking whether a USB drive, keyboard, network adapter, or other USB device was detected correctly.
Example output:
[ 1.245011] usb 1-1: new high-speed USB device number 2 using xhci_hcd
[ 1.601233] usb 1-1: New USB device found, idVendor=0781, idProduct=5567
[ 1.604811] usb-storage 1-1:1.0: USB Mass Storage device detected
Explanation:
dmesg | grep -i eth
This command searches kernel messages for Ethernet-related entries. It is useful when checking network interface detection, link state changes, and network driver messages.
Example output:
[ 2.884310] eth0: renamed from enp0s3
[ 45.812001] e1000e 0000:00:19.0 eth0: NIC Link is Up 1000 Mbps Full Duplex
[ 300.441221] eth0: Link is Down
Explanation:
eth.enp0s3, ens160, or eno1 instead of eth0.journalctl -k -p warning
This command shows kernel warning messages and anything more severe. It is useful when you want to focus on kernel problems without reading normal informational messages.
Example output:
May 03 10:12:44 server kernel: CPU0: Core temperature above threshold
May 03 10:13:02 server kernel: blk_update_request: I/O error, dev sda, sector 884120
May 03 10:13:10 server kernel: EXT4-fs warning (device sda1): mounting fs with errors
Explanation:
-k filters logs to kernel messages.-p warning shows warnings, errors, critical, alert, and emergency messages.-b to show warnings from the current boot only.Sample kernel log:
[12345.678901] eth0: Link is Down
[12346.123456] EXT4-fs error: I/O error while writing superblock
[12347.222222] CPU0: Core temperature above threshold
This sample shows three different kernel-level problems. The first line indicates a network interface link problem, the second line indicates a filesystem or disk write problem, and the third line indicates a CPU temperature warning.
Interpretation:
eth0 Link is Down β network interface disconnected
EXT4-fs error β filesystem or disk issue
temperature threshold β cooling or hardware problem
Explanation:
eth0 Link is Down usually means the network cable was unplugged, the virtual interface disconnected, or the switch port went down.EXT4-fs error usually points to filesystem corruption, disk I/O errors, storage problems, or an unsafe shutdown.I/O error while writing superblock is serious because the superblock contains important filesystem metadata.CPU temperature above threshold means the CPU is getting too hot and may throttle performance or shut down to protect hardware.Authentication logs record:
Ubuntu/Debian:
sudo less /var/log/auth.log
RHEL-based systems:
sudo less /var/log/secure
Using journald:
journalctl -u ssh.service
journalctl _COMM=sshd
Common searches:
sudo grep "Failed password" /var/log/auth.log
sudo grep "Accepted password" /var/log/auth.log
sudo grep "sudo" /var/log/auth.log
sudo grep "session opened" /var/log/auth.log
Example:
May 03 11:00:01 server sshd[23456]: Failed password for invalid user admin from 203.0.113.10 port 54323 ssh2
Breakdown:
Failed password β login failed
invalid user admin β account does not exist
203.0.113.10 β source IP
sshd β SSH daemon
Most services managed by systemd can be debugged with:
systemctl status service-name
journalctl -u service-name
Example:
systemctl status nginx
journalctl -u nginx
journalctl -u nginx -f
Useful pattern:
sudo systemctl restart nginx
journalctl -u nginx -n 50 --no-pager
This lets you restart a service and immediately inspect the most recent logs.
Cron logs scheduled jobs.
Depending on distro, cron logs may be in:
/var/log/syslog
/var/log/cron
journalctl -u cron
journalctl -u crond
Examples:
grep CRON /var/log/syslog
journalctl -u cron
Sample:
May 03 12:00:01 server CRON[4567]: (root) CMD (/usr/local/bin/backup.sh)
This means cron started the script.
It does not guarantee the script succeeded.
For proper script debugging, the script itself should log success and failure.
Useful when asking:
What changed recently?
Was a package upgraded?
Did an update break something?
Debian/Ubuntu:
less /var/log/dpkg.log
less /var/log/apt/history.log
less /var/log/apt/term.log
Examples:
grep "install " /var/log/dpkg.log
grep "upgrade " /var/log/dpkg.log
grep nginx /var/log/apt/history.log
RHEL/Fedora:
less /var/log/dnf.log
less /var/log/yum.log
rpm -qa --last
Common files:
/var/log/nginx/access.log
/var/log/nginx/error.log
Access log example:
192.168.1.50 - - [03/May/2026:13:00:01 +0200] "GET /index.html HTTP/1.1" 200 612
Meaning:
192.168.1.50 β client IP
GET /index.html β requested path
200 β HTTP status code
612 β bytes sent
Error log example:
2026/05/03 13:01:22 [error] 1234#1234: *55 connect() failed while connecting to upstream
This often means nginx cannot reach the backend app.
Useful commands:
sudo tail -f /var/log/nginx/access.log
sudo tail -f /var/log/nginx/error.log
sudo grep " 500 " /var/log/nginx/access.log
sudo grep "connect() failed" /var/log/nginx/error.log
Common files:
/var/log/apache2/access.log
/var/log/apache2/error.log
or:
/var/log/httpd/access_log
/var/log/httpd/error_log
PostgreSQL logs may be in:
/var/log/postgresql/
journalctl -u postgresql
MySQL/MariaDB logs may be in:
/var/log/mysql/
journalctl -u mysql
journalctl -u mariadb
Common things to search:
grep -i error /var/log/postgresql/*.log
grep -i "connection refused" /var/log/mysql/error.log
journalctl -u postgresql -p err
Docker has its own logging path.
View container logs:
docker logs container_name
docker logs -f container_name
docker logs --tail 100 container_name
docker logs --since 1h container_name
Docker service logs:
journalctl -u docker
Docker Compose logs:
docker compose logs
docker compose logs -f
docker compose logs api
docker compose logs --tail 100
Common Docker debugging flow:
docker ps
docker ps -a
docker logs container_name
docker inspect container_name
journalctl -u docker
Python applications should not rely only on print().
Use the built-in logging module.
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s: %(message)s"
)
logger = logging.getLogger("myapp")
logger.info("Application started")
logger.warning("Disk space is getting low")
logger.error("Database connection failed")
Example output:
2026-05-03 14:00:01 INFO myapp: Application started
2026-05-03 14:00:02 WARNING myapp: Disk space is getting low
2026-05-03 14:00:03 ERROR myapp: Database connection failed
import logging
logging.basicConfig(
filename="/var/log/myapp.log",
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s [%(process)d]: %(message)s"
)
logger = logging.getLogger("myapp")
logger.info("Server started")
logger.error("Could not connect to database")
Important:
The application user must have permission to write to the log file.
Example:
sudo touch /var/log/myapp.log
sudo chown myappuser:myappuser /var/log/myapp.log
If your Python app runs as a systemd service, logging to stdout/stderr is often best.
Python app:
import logging
import sys
logger = logging.getLogger("myapp")
logger.setLevel(logging.INFO)
handler = logging.StreamHandler(sys.stdout)
formatter = logging.Formatter(
"%(asctime)s %(levelname)s %(name)s: %(message)s"
)
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.info("Application started")
logger.error("Something failed")
Systemd service:
[Unit]
Description=My Python App
After=network.target
[Service]
User=myappuser
WorkingDirectory=/opt/myapp
ExecStart=/usr/bin/python3 /opt/myapp/app.py
Restart=always
[Install]
WantedBy=multi-user.target
Then logs can be viewed with:
journalctl -u myapp.service
journalctl -u myapp.service -f
Flow:
Python stdout/stderr
|
v
systemd service manager
|
v
systemd-journald
|
v
journalctl -u myapp.service
For standalone apps, you can rotate logs from inside Python.
import logging
from logging.handlers import RotatingFileHandler
logger = logging.getLogger("myapp")
logger.setLevel(logging.INFO)
handler = RotatingFileHandler(
"/var/log/myapp.log",
maxBytes=5_000_000,
backupCount=5
)
formatter = logging.Formatter(
"%(asctime)s %(levelname)s %(name)s: %(message)s"
)
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.info("App started")
logger.warning("Something looks suspicious")
logger.error("Something failed")
This creates files like:
/var/log/myapp.log
/var/log/myapp.log.1
/var/log/myapp.log.2
/var/log/myapp.log.3
Use this when the app owns its logs.
Use logrotate when Linux should manage the log files externally.
Rotate every day:
import logging
from logging.handlers import TimedRotatingFileHandler
logger = logging.getLogger("myapp")
logger.setLevel(logging.INFO)
handler = TimedRotatingFileHandler(
"/var/log/myapp.log",
when="midnight",
interval=1,
backupCount=14
)
formatter = logging.Formatter(
"%(asctime)s %(levelname)s %(name)s: %(message)s"
)
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.info("Daily rotating logger started")
Result:
myapp.log
myapp.log.2026-05-01
myapp.log.2026-05-02
myapp.log.2026-05-03
JSON logs are easier for machines to parse.
import logging
import json
import sys
from datetime import datetime, timezone
class JsonFormatter(logging.Formatter):
def format(self, record):
log_record = {
"timestamp": datetime.now(timezone.utc).isoformat(),
"level": record.levelname,
"logger": record.name,
"message": record.getMessage(),
"module": record.module,
"line": record.lineno,
"process": record.process,
}
if record.exc_info:
log_record["exception"] = self.formatException(record.exc_info)
return json.dumps(log_record)
logger = logging.getLogger("myapp")
logger.setLevel(logging.INFO)
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JsonFormatter())
logger.addHandler(handler)
logger.info("Application started")
Example output:
{
"timestamp": "2026-05-03T12:00:01+00:00",
"level": "INFO",
"logger": "myapp",
"message": "Application started",
"module": "app",
"line": 31,
"process": 1234
}
JSON logs are good for:
Loki
Elasticsearch
OpenSearch
Splunk
Fluent Bit
Vector
Logstash
custom Python parsers
Bad:
try:
1 / 0
except Exception as e:
logger.error(f"Error: {e}")
Better:
try:
1 / 0
except Exception:
logger.exception("Unexpected calculation error")
logger.exception() includes the traceback.
Example:
ERROR myapp: Unexpected calculation error
Traceback (most recent call last):
File "app.py", line 10, in <module>
1 / 0
ZeroDivisionError: division by zero
Tracebacks are extremely important for debugging.
You can send Python logs to syslog.
import logging
from logging.handlers import SysLogHandler
logger = logging.getLogger("myapp")
logger.setLevel(logging.INFO)
handler = SysLogHandler(address="/dev/log")
formatter = logging.Formatter("myapp: %(levelname)s %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.info("Application started")
logger.error("Database connection failed")
Then check:
journalctl | grep myapp
grep myapp /var/log/syslog
loggerThe logger command sends messages to syslog/journald.
Simple example:
logger "Backup completed successfully"
With tag:
logger -t backup_script "Backup completed successfully"
With severity:
logger -t backup_script -p local0.info "Backup started"
logger -t backup_script -p local0.err "Backup failed"
Example script:
#!/bin/bash
SOURCE="/data"
DEST="/backup"
logger -t backup_script -p local0.info "Backup started"
if rsync -a "$SOURCE" "$DEST"; then
logger -t backup_script -p local0.info "Backup completed successfully"
else
logger -t backup_script -p local0.err "Backup failed"
exit 1
fi
View logs:
journalctl -t backup_script
grep backup_script /var/log/syslog
rsyslog is a powerful syslog daemon used to:
Basic flow:
Application / Kernel / Service
|
v
journald
|
v
rsyslog
|
+------+------+
| |
v v
/var/log/syslog Remote log server
Classic format:
facility.priority action
Example:
authpriv.* /var/log/auth.log
kern.* /var/log/kern.log
mail.info /var/log/mail.info
*.err /var/log/errors.log
Facilities:
| Facility | Meaning |
| auth | Authentication |
| authpriv | Private auth messages |
| cron | Cron jobs |
| daemon | System daemons |
| kern | Kernel messages |
| Mail system | |
| syslog | Syslog internal messages |
| user | User-level messages |
| local0-7 | Custom use |
Priorities:
debug < info < notice < warning < err < crit < alert < emerg
Create:
sudo nano /etc/rsyslog.d/30-myapp.conf
Example:
if $programname == 'myapp' then /var/log/myapp.log
& stop
Restart rsyslog:
sudo systemctl restart rsyslog
Test it:
logger -t myapp "Hello from myapp"
cat /var/log/myapp.log
Enable TCP receiver:
module(load="imtcp")
input(type="imtcp" port="514")
Store logs by hostname:
template(name="RemoteLogs" type="string" string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log")
*.* ?RemoteLogs
Restart:
sudo systemctl restart rsyslog
Send logs to server:
*.* @@logserver.example.com:514
@ means UDP.
@@ means TCP.
Restart:
sudo systemctl restart rsyslog
Architecture:
+-----------+ TCP 514 +----------------+
| server01 | ------------------> | logserver |
| server02 | ------------------> | /var/log/remote|
| server03 | ------------------> | |
+-----------+ +----------------+
Logs grow forever unless managed.
Housekeeping means:
There are two major housekeeping systems:
| System | Manages |
| logrotate | Plain text logs in /var/log |
| journald config | systemd journal size and retention |
logrotate manages traditional log files.
Config locations:
/etc/logrotate.conf
/etc/logrotate.d/
View configs:
cat /etc/logrotate.conf
ls /etc/logrotate.d/
cat /etc/logrotate.d/nginx
cat /etc/logrotate.d/rsyslog
Example config:
/var/log/myapp.log {
daily
rotate 14
compress
delaycompress
missingok
notifempty
create 0640 myappuser adm
postrotate
systemctl reload myapp.service > /dev/null 2>&1 || true
endscript
}
Meaning:
daily rotate every day
rotate 14 keep 14 old logs
compress gzip old logs
delaycompress wait one cycle before compression
missingok do not error if file is missing
notifempty do not rotate empty logs
create create a new file with permissions/owner/group
postrotate run command after rotation
Logrotate state file:
cat /var/lib/logrotate/status
Debug logrotate without changing files:
sudo logrotate -d /etc/logrotate.conf
Force rotation:
sudo logrotate -f /etc/logrotate.conf
Force one config:
sudo logrotate -f /etc/logrotate.d/nginx
Check rotated files:
ls -lh /var/log/syslog*
ls -lh /var/log/nginx/*
Example:
/var/log/syslog
/var/log/syslog.1
/var/log/syslog.2.gz
/var/log/syslog.3.gz
On many systems, logrotate is run by a systemd timer:
systemctl status logrotate.timer
systemctl list-timers | grep logrotate
Or by cron:
ls /etc/cron.daily/
cat /etc/cron.daily/logrotate
Useful check:
systemctl status logrotate.service
journalctl -u logrotate.service
Check journal disk usage:
journalctl --disk-usage
Vacuum old journal logs by size:
sudo journalctl --vacuum-size=1G
Vacuum by time:
sudo journalctl --vacuum-time=14d
Vacuum by number of files:
sudo journalctl --vacuum-files=10
Config file:
/etc/systemd/journald.conf
Common settings:
[Journal]
Storage=persistent
SystemMaxUse=1G
SystemKeepFree=2G
MaxRetentionSec=1month
Compress=yes
Restart journald after changes:
sudo systemctl restart systemd-journald
Volatile journal:
/run/log/journal
Lost after reboot.
Persistent journal:
/var/log/journal
Survives reboot.
Enable persistent journal:
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
Check:
ls -ld /var/log/journal
journalctl --list-boots
This is where practical debugging happens.
| Tool | Use |
| less | Read large files interactively |
| tail | Show last lines / follow live logs |
| grep | Search text |
| awk | Extract columns / summarize |
| sed | Transform/filter text |
| cut | Extract fields |
| sort | Sort results |
| uniq | Count repeated lines |
| wc | Count lines |
| jq | Parse JSON logs |
| journalctl | Query systemd journal |
| zgrep | Search compressed .gz logs |
| lnav | Interactive log viewer |
Follow a file live:
tail -f /var/log/syslog
Follow multiple files:
tail -f /var/log/syslog /var/log/auth.log
Search for errors:
grep -i error /var/log/syslog
Search compressed rotated logs:
zgrep -i error /var/log/syslog.*.gz
Search current and rotated logs:
grep -i error /var/log/syslog /var/log/syslog.1
zgrep -i error /var/log/syslog.*.gz
Count failed SSH attempts by IP:
grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr
Find top requested URLs in nginx access log:
awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head
Find top HTTP status codes:
awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -nr
Find 500 errors:
awk '$9 >= 500 {print}' /var/log/nginx/access.log
journalctl for Fast FilteringErrors from current boot:
journalctl -b -p err
Warnings and errors from nginx:
journalctl -u nginx -p warning
Logs since 10 minutes ago:
journalctl --since "10 minutes ago"
Logs for one executable:
journalctl _COMM=sshd
Logs for one PID:
journalctl _PID=1234
Kernel logs from current boot:
journalctl -k -b
Show logs without pager:
journalctl -u nginx --no-pager
jqExample JSON log:
{"timestamp":"2026-05-03T12:00:00Z","level":"ERROR","service":"api","message":"database timeout"}
Show only errors:
jq 'select(.level == "ERROR")' app.log
Print timestamp and message:
jq -r 'select(.level == "ERROR") | "\(.timestamp) \(.message)"' app.log
Count by level:
jq -r '.level' app.log | sort | uniq -c
lnavlnav is an interactive log viewer that can understand many log formats.
Install:
sudo apt install lnav
Open logs:
sudo lnav /var/log/syslog /var/log/auth.log
Open all nginx logs:
sudo lnav /var/log/nginx/*.log
Benefits:
For troubleshooting, it is often useful to collect logs into one bundle.
Example:
mkdir -p debug-logs
journalctl -b > debug-logs/journal-current-boot.log
journalctl -p err > debug-logs/journal-errors.log
dmesg -T > debug-logs/dmesg.log
systemctl status nginx > debug-logs/nginx-status.txt
journalctl -u nginx > debug-logs/nginx-journal.log
cp /var/log/nginx/error.log debug-logs/
tar -czf debug-logs.tar.gz debug-logs
Result:
debug-logs.tar.gz
You can send this to another admin or attach it to a bug report.
For one server, local logs may be enough.
For many servers, centralized logging is better.
+-----------+ +-----------+
| app01 | | app02 |
+-----+-----+ +-----+-----+
| |
v v
+-------------------------------+
| Log Collector |
| rsyslog / Fluent Bit / Vector |
+---------------+---------------+
|
v
+-------------------------------+
| Storage / Search |
| Loki / Elasticsearch / Splunk |
+-------------------------------+
|
v
+-------------------------------+
| Dashboard / Alerts |
| Grafana / Kibana / SIEM |
+-------------------------------+
Common tools:
| Tool | Purpose |
| rsyslog | Classic syslog forwarding and routing |
| syslog-ng | Alternative syslog daemon |
| Fluent Bit | Lightweight log collector/forwarder |
| Fluentd | Heavier log collector/processor |
| Vector | Fast log/event pipeline |
| Logstash | Processing pipeline for Elastic/OpenSearch |
| Filebeat | Ships log files to Elastic/OpenSearch |
| Promtail | Ships logs to Loki |
| Loki | Log storage/query system by Grafana ecosystem |
| Elasticsearch | Search/index log storage |
| OpenSearch | Open-source Elasticsearch alternative |
| Splunk | Commercial log analytics platform |
| Grafana | Dashboards for logs and metrics |
| Kibana | Elasticsearch visualization UI |
Example: nginx fails.
Step 1:
systemctl status nginx
Step 2:
journalctl -u nginx -n 100 --no-pager
Step 3:
sudo nginx -t
Step 4:
sudo tail -n 100 /var/log/nginx/error.log
Flow:
Service failed
|
v
systemctl status
|
v
journalctl -u service
|
v
application-specific config test
|
v
application-specific error log
Check SSH service:
systemctl status ssh
journalctl -u ssh
Check auth logs:
sudo tail -f /var/log/auth.log
Search failures:
sudo grep "Failed password" /var/log/auth.log
Search accepted logins:
sudo grep "Accepted" /var/log/auth.log
Common causes:
Check disk:
df -h
Find largest log directories:
sudo du -sh /var/log/* | sort -h
Find huge log files:
sudo find /var/log -type f -size +100M -exec ls -lh {} \;
Check journal size:
journalctl --disk-usage
Clean journal safely:
sudo journalctl --vacuum-size=1G
Force logrotate:
sudo logrotate -f /etc/logrotate.conf
Do not blindly delete active logs.
Safer truncate if needed:
sudo truncate -s 0 /var/log/huge.log
Check nginx:
sudo tail -f /var/log/nginx/error.log
Check backend service:
systemctl status myapp
journalctl -u myapp -n 100
Check port listening:
ss -tulpn
Check app logs:
journalctl -u myapp -f
Common meaning:
502 Bad Gateway nginx cannot talk to backend
503 Service Unavailable backend unavailable or overloaded
504 Gateway Timeout backend too slow or unreachable
List boots:
journalctl --list-boots
View previous boot errors:
journalctl -b -1 -p err
View previous boot kernel messages:
journalctl -k -b -1
Search shutdown/reboot messages:
journalctl -b -1 | grep -i "shutdown\|reboot\|panic\|oom\|killed"
Check for OOM killer:
journalctl -k | grep -i "out of memory\|oom"
sudo grep "Failed password" /var/log/auth.log
Count by source IP:
sudo grep "Failed password" /var/log/auth.log \
| awk '{print $(NF-3)}' \
| sort \
| uniq -c \
| sort -nr
sudo grep "sudo" /var/log/auth.log
Example:
May 03 14:00:01 server sudo: alice : TTY=pts/0 ; PWD=/home/alice ; USER=root ; COMMAND=/usr/bin/apt update
Meaning:
alice used sudo
from terminal pts/0
while in /home/alice
to run apt update as root
Common locations:
/var/log/fail2ban.log
journalctl -u fail2ban
Commands:
sudo fail2ban-client status
sudo fail2ban-client status sshd
Avoid logging:
Bad:
User login failed with password hunter2
Better:
User login failed for username alice from 203.0.113.10
tail -f /var/log/syslog
less /var/log/syslog
grep -i error /var/log/syslog
zgrep -i error /var/log/syslog.*.gz
journalctl
journalctl -n 100
journalctl -f
journalctl -b
journalctl -b -1
journalctl -p err
journalctl -u nginx
journalctl -u nginx -f
journalctl --since "1 hour ago"
journalctl --disk-usage
dmesg
dmesg -T
journalctl -k
journalctl -k -b
systemctl status nginx
journalctl -u nginx -n 100
systemctl restart nginx
grep "Failed password" /var/log/auth.log
grep "sudo" /var/log/auth.log
journalctl _COMM=sshd
cat /etc/logrotate.conf
ls /etc/logrotate.d/
cat /var/lib/logrotate/status
sudo logrotate -d /etc/logrotate.conf
sudo logrotate -f /etc/logrotate.conf
df -h
du -sh /var/log/*
find /var/log -type f -size +100M -exec ls -lh {} \;
journalctl --disk-usage
Logwatch or Splunk to enhance efficiency and reduce manual oversight.Nagios or Prometheus to ensure rapid response to incidents.logger command to create custom messages in the system logs. Experiment with different flags, such as specifying the facility or severity level, and explain how logger can be used to add entries manually or from within scripts for testing or informational purposes.logrotate to automate log file management. Set up a basic configuration to rotate, compress, and delete log files on a schedule, and discuss how logrotate helps prevent logs from consuming excessive disk space. Explain the importance of log rotation in production systems.grep, journalctl, or awk to extract meaningful information from log files. Perform tasks such as searching for specific events, identifying patterns, and generating summary reports. Explain how log analysis helps administrators identify issues and monitor system health.