Loading...

VONDS ©

Lab 25: Diagnose Log Issues and Manage Rotation

Lab Index

Diagnose performance issues by interrogating system logs, then reclaim disk space safely by truncating oversized log files without deleting them. Validate and adjust log rotation policy by inspecting logrotate configuration and forcing a controlled rotation run.

troubleshooting services core

Scenario

A user reports the system is running slowly and the root filesystem is close to full. You suspect a log storm or a runaway service is filling /var/log. Your job is to watch logs live, identify a large log file consuming space, reclaim storage safely, and confirm that logrotate policy is in place and functioning.

Operator context

You are fixing the immediate disk pressure without destroying evidence. The goal is “stabilize first, preserve data, then tune rotation.”

Objective

  • Follow system logs in real time to observe active errors and noisy services.
  • Identify an oversized log file and reclaim disk space safely by truncating the file.
  • Locate the master logrotate configuration and the directory containing per-package rotation rules.
  • Force a manual logrotate run to validate configuration and immediate effect.

What You’ll Practice

  • Live log inspection using journalctl -f.
  • Disk usage triage using du.
  • Safe log reclamation using truncate (preserving inode and permissions).
  • Log rotation policy discovery with /etc/logrotate.conf and /etc/logrotate.d.
  • Controlled rotation testing using logrotate -f.

Walkthrough

Step 1 : Watch system logs in real time.
Command 
journalctl -f

Following the journal gives you immediate evidence of what is spamming errors or creating sustained load. If disk fills due to logs, the journal often shows the root cause at the same time. 

-- Logs begin at Tue 2025-07-01 09:00:00, end at Tue 2025-07-18 14:30:00 --
Jul 18 14:25:00 lpic-lab25 kernel: CPU soft lockup detected...
Step 2 : Check size and truncate an oversized log safely.
Safety note

Prefer truncation over deletion. Truncation preserves file ownership, SELinux labels, and open file handles while reclaiming disk space.

Command 
du -sh /var/log/syslog && truncate -s 0 /var/log/syslog

du -sh gives a quick “how bad is it” snapshot. truncate -s 0 drops file contents to zero bytes without removing the file itself. 

5.0G    /var/log/syslog
Log file truncated safely.
Step 3 : Identify the master logrotate configuration.
Command 
/etc/logrotate.conf

/etc/logrotate.conf is the entry point for the rotation policy. It usually defines global defaults and includes the per-package rules directory.

Step 4 : Find per-package rotation rules.
Command 
/etc/logrotate.d

Most services drop their own rotation configuration into /etc/logrotate.d. This is where you verify that your noisy service actually has a rotation rule.

Step 5 : Force logrotate to run immediately.
Command 
logrotate -f /etc/logrotate.conf

Forcing a run is how you validate that your configuration is syntactically correct and that rotations happen as intended without waiting for the daily timer/cron. 

logrotate executed manually using config.

Reference

  • journalctl -f : Follows the systemd journal in real time.
  • du -sh <path> : Summarizes disk usage for a file or directory.
  • truncate -s 0 <file> : Safely clears a file’s contents without deleting the file.
  • /etc/logrotate.conf : Master logrotate configuration.
  • /etc/logrotate.d : Per-package/service log rotation rules.
  • logrotate -f /etc/logrotate.conf : Forces an immediate logrotate run using the master config.