Last modified: June 11, 2024

This article is written in: 🇺🇸

Logging is an essential part of system administration. It provides crucial insights into the system's operation by keeping a record of significant events. System logs are valuable resources for troubleshooting issues, auditing security, and optimizing system performance. Linux utilizes various logging methods, including plain text files, journald, and rsyslog, each with its unique advantages and limitations.

+-----------------------------------------------------------+
| LOG FILE                                                  |
|-----------------------------------------------------------|
| TIMESTAMP        | SEVERITY  | SERVICE   | MESSAGE        |
|-----------------------------------------------------------|
| 2023-08-01 09:00 | INFO      | myapp     | Server started |
| 2023-08-01 09:01 | WARNING   | myapp     | High CPU usage |
| 2023-08-01 09:02 | ERROR     | myapp     | Server crashed |
+-----------------------------------------------------------+

Overview of Logging Methods

Here are the most common logging methods used in Linux systems:

• Text files: The simplest form of logging involves storing log data in plain text files. These log files can contain any data that system administrators choose to record, providing them full control over what is logged. This method is simple and universal, allowing for easy reading and processing with common command-line tools.

• Journald: journald is a logging system utilized by Linux distributions that use systemd as their init system. It stores logs related to the boot process, services, and the kernel in binary files. Unlike traditional text files, journald logs include metadata that can facilitate more sophisticated log queries. These binary logs are not meant for long-term storage as they are designed to be rotated and deleted when they are no longer needed or when the log storage limit is reached.

• Rsyslog: rsyslog is a powerful, open-source log processor and forwarder which provides features such as TCP syslog transport, content-based filtering, and high-precision timestamps. It is capable of handling logs from different sources and forwarding them to different destinations. By default, rsyslog writes log files to the /var/log directory, where they can be kept for longer periods, providing a more persistent log storage solution.

Common System Log Files

Log files are typically stored in the /var/log directory. Below are some of the common log files you'll find:

• syslog: This is a catch-all log file containing messages from the kernel and various applications and services. It's a centralized place to look for system activity.

• kern.log: This log file is dedicated to kernel-related messages. It includes logs related to hardware events, driver issues, and system errors. It's especially useful when you're troubleshooting hardware or driver problems.

• auth.log: This log file keeps track of authentication and authorization events. This includes user logins, password changes, privilege escalations, and more, making it an important resource for security audits.

• dmesg: This file contains messages related to the kernel ring buffer. These are particularly useful for diagnosing hardware and driver issues during the system startup process.

The specific logs available and their locations can vary between distributions.

Journal Files in systemd

systemd is the default system and service manager for many popular distributions. It comes with a robust and centralized logging system called the journal. Unlike traditional text-based logs, the journal stores log data in binary format, providing several advantages such as metadata support, data integrity through hashing, compression for efficient storage, and indexing for quick searching.

Viewing Journal Entries with journalctl

The journalctl command is used to query and display entries from the systemd journal. By default, it lists all the journal entries in chronological order, starting from the oldest:

journalctl

This command displays a comprehensive list of system logs, including those from the kernel, systemd services, and other system components.

Filtering and Manipulating Journal Entries

journalctl supports various options for filtering and displaying journal entries to make it easier to find relevant logs. Here are some commonly used options:

• -u, --unit: This option allows you to show log entries for a specific systemd unit, such as a service. For example, to view logs related to the ssh.service unit, you would use:

journalctl -u ssh.service

• -b, --boot: This option is used to show log entries from the current boot session or a specific boot session. By default, it displays logs from the current boot. However, you can specify a particular boot session like this:

journalctl -b -1  # Logs from the previous boot
journalctl -b 0  # Logs from the current boot

• --since, --until: These options are used to display log entries between specific dates and times. The format for the date is 'YYYY-MM-DD HH:MM:SS'. For example:

journalctl --since "2023-01-01 00:00:00" --until "2023-01-02 00:00:00"

• -f, --follow: Similar to the tail -f command, this option allows you to follow the journal in real time. It continuously displays new log entries as they are added to the journal:

journalctl -f

Rsyslog for Log Processing

Rsyslog is an enhanced syslogd supporting a range of input and output methods, and numerous advanced configurations. It is an efficient and robust system for log processing, offering high performance, modularity, and extensive security features.

Initially developed as an extension of syslogd, rsyslog has evolved into a sophisticated logging system capable of receiving input from a multitude of sources, processing logs, and outputting the information to a wide array of destinations.

For instance, rsyslog can collect logs from multiple machines across a network, funnel them to a centralized location, and store them for later analysis. This feature is particularly beneficial in large-scale deployments, where managing logs from individual machines can be daunting.

The main configuration file for rsyslog is located at /etc/rsyslog.conf. Editing this file requires a good understanding of its scripting language syntax. It allows the definition of rulesets for different log sources, configuration of input and output modules, and application of filters based on severity levels or various other criteria.

Understanding Severity Levels in Rsyslog

Severity levels in rsyslog provide a mechanism to categorize log messages according to their importance. These severity levels range from 0 (emergency) to 7 (debug). Here is a quick overview:

These severity levels facilitate filtering log messages to display only the relevant information or to route certain messages to specific destinations. For example, critical messages (levels 0 to 2) might be sent to an email alert system, while informational messages could be directed to a general log server.

Managing Rsyslog Services

You can start, stop, restart, and check the status of the rsyslog service using systemctl:

sudo systemctl start rsyslog      # Starts the service
sudo systemctl stop rsyslog       # Stops the service
sudo systemctl restart rsyslog    # Restarts the service
sudo systemctl status rsyslog     # Checks the status of the service

Scenario: Centralized Log Management with Rsyslog

Suppose you have a network with multiple servers, and you want to consolidate all logs in a central location for streamlined monitoring and analysis. This can be achieved using rsyslog on both the centralized log server (also known as the log collector) and the client machines that generate the logs.

Setting up the Rsyslog Server

Firstly, ensure rsyslog is installed on the server machine. If not, use the following command to install it:

sudo apt-get install rsyslog

Next, edit the rsyslog configuration file located at /etc/rsyslog.conf. Uncomment or add the following lines to enable the server to receive incoming logs on UDP port 514:

module(load="imudp")
input(type="imudp" port="514")

Apply the changes by restarting rsyslog:

sudo systemctl restart rsyslog

Optionally, if you have a firewall configured, allow incoming connections on UDP port 514:

sudo ufw allow 514/udp

Setting up the Rsyslog Client

On the client machines that generate logs, install rsyslog if it's not already installed:

sudo apt-get install rsyslog

Next, edit the rsyslog configuration file located at /etc/rsyslog.conf. Add the following line at the end of the file, replacing <SERVER_IP> with the IP address of the rsyslog server:

*.* @<server_ip>:514

This configuration instructs the client to forward all log messages (.) to the rsyslog server via UDP.

Apply the changes by restarting rsyslog:

sudo systemctl restart rsyslog

Now, the client machines will forward their log messages to the centralized rsyslog server. The server stores the received logs in the /var/log directory. These logs can be analyzed using various log analysis tools or further processed using rsyslog's scripting language for better organization and easy retrieval of logs.

Logger Utility

Logger is a command-line utility in Linux used for generating log messages from the terminal. These messages are added to the local /var/log/syslog file or can be directed to a remote syslog server. Logger provides several options for specifying the priority of messages, defining the syslog port, or indicating a remote system, making it an adaptable tool for various logging needs.

The usage and options of logger can be understood in more detail through the manual pages:

man logger

Basic Usage of Logger

A simple example of using logger to generate a log message:

logger "This is a sample log message"

This command will append the text "This is a sample log message" to the /var/log/syslog file.

Sending Log Message to Remote Server with Logger

Logger can also be used to send log messages to a remote syslog server. Here is an example:

logger -n 192.168.10.27 -P 514 "This is a sample log message for remote server"

In this command:

• -n flag is used to specify the IP address of the remote syslog server.
• -P flag is used to specify the port number on which the remote syslog server is listening. The default syslog port is 514.

Adjusting Log Message Severity

Logger allows us to adjust the severity of a log message using the -p option followed by the desired facility and priority level. For example:

logger -p auth.info "User John logged in successfully"

In this command:

• auth is the facility (representing the source of the message).
• info is the priority (representing the severity of the message).

The auth.info message would thus indicate an informational message from the authentication and authorization system (e.g., a successful user login).

Logger, being a flexible tool, fits perfectly into scripts or automated tasks where logging is required for monitoring or troubleshooting purposes.

Logrotate

Logrotate is an essential utility for managing log files. It is designed to automate the process of rotating, compressing, and deleting log files to prevent them from consuming all available disk space on a system. The unchecked growth of log files can lead to performance issues and can even make a system unusable.

Logrotate maintains system health and efficient disk space usage by rotating log files based on configurations specified by the user. It compresses old logs, deletes logs older than a certain threshold, and facilitates the creation of new logs for ongoing tracking.

To check the installed version of logrotate, use:

logrotate --version

Configuration of Logrotate

Logrotate uses configuration files to manage log files. The main configuration file is located at /etc/logrotate.conf, and it allows for additional, application-specific configuration files in the /etc/logrotate.d/ directory.

These configuration files dictate how and when logrotate performs actions on specific log files or sets of log files. For instance, you may have a web server generating access and error logs daily. These logs can rapidly increase in size, causing them to be challenging to manage. With logrotate, you can automate the rotation of these logs daily, compress logs older than a week, and delete logs older than a month. This approach ensures efficient disk usage and readily accessible recent logs.

Here's an example of a logrotate configuration for a web server:

/var/log/httpd/*log {
    daily
    compress
    missingok
    notifempty
    rotate 30
    create 0640 root adm
    postrotate
    /sbin/service httpd reload > /dev/null 2>/dev/null || true
    endscript
}

In this configuration:

• The /var/log/httpd/*log line specifies the logs to be managed by this configuration.
• daily means the logs will be rotated every day.
• compress tells logrotate to compress the log files when they are rotated.
• missingok instructs logrotate to continue without error if a log file is missing.
• notifempty ensures that log files are not rotated if they are empty.
• rotate 30 retains 30 days' worth of logs.
• create 0640 root adm sets the owner and group of the new log file and sets the permissions to 0640.
• The postrotate/endscript section specifies commands to be executed after the log file is rotated. In this case, it reloads the web server.

Challenges

1. Detail the reasons why logging is crucial in system administration. Discuss its role in maintaining system health, identifying issues, and aiding in security auditing.
2. Provide an in-depth description of Journald, its functions, advantages, and how it is different from traditional text-file-based logging systems. Discuss its relationship with systemd.
3. Explain how Rsyslog operates. Discuss its configuration, its use in centralized logging, and the use of severity levels in filtering and categorizing log messages.
4. Use the logger command to create and send messages to system logs. Explain the role and usage of different flags that can be used with this command.
5. Explain how to configure and use logrotate to manage log files' sizes. Discuss how it can help automate the process of rotating, compressing, and deleting log files.
6. Discuss common log file formats and their differences. Consider elements like structure, readability, and compatibility with different log analysis tools.
7. Show how to use log filters to selectively include or exclude log messages based on certain criteria. Use Rsyslog or Journald as an example.
8. Explain how to use log analysis tools to extract useful information from log files. This could include searching for specific events, identifying trends, or generating reports.
9. Describe best practices for managing and maintaining logs in a production environment. Discuss strategies for retaining logs, securing log data, and ensuring the reliability and availability of log files.
10. Discuss common logging-related problems in a Linux environment, such as missing logs, log corruption, or full disk space due to log files. Explain how to diagnose and resolve these issues.