Loading...

VONDS ©

Lab 25: Diagnose Log Issues and Manage Rotation

Back to Lab Index

Diagnose performance issues by interrogating system logs, then reclaim disk space safely by truncating oversized log files without deleting them. Validate and adjust log rotation policy by inspecting logrotate configuration and forcing a controlled rotation run.

troubleshooting services core

Scenario

A user reports the system is running slowly and the root filesystem is close to full. You suspect a log storm or a runaway service is filling /var/log. Your job is to watch logs live, identify a large log file consuming space, reclaim storage safely, and confirm that logrotate policy is in place and functioning.

Operator context

You are fixing the immediate disk pressure without destroying evidence. The goal is stabilize first, preserve data, then tune rotation.

Objective

  • Follow system logs in real time to observe active errors and noisy services.
  • Identify oversized logs under /var/log and confirm disk pressure.
  • Reclaim space safely by truncating a large log file without deleting it.
  • Locate the master logrotate configuration and the directory containing per-package rules.
  • Force a manual logrotate run to validate configuration and immediate effect.

Concepts

  • Log storms are symptoms. Your first job is to identify the noisy unit, not just delete output.
  • Truncation preserves the inode, ownership, labels, and open file handles, while reclaiming space.
  • logrotate policy comes from global defaults plus per-service rules; rotations happen on schedule but can be validated on demand.
  • Forcing a rotation is a controlled test: confirm the rule matches the file, the state file updates, and the rotated artifacts appear where expected.

Walkthrough

Step 1 : Watch system logs in real time.
Command 
journalctl -f

Following the journal gives you immediate evidence of what is spamming errors or creating sustained load. If disk fills due to logs, the journal often shows the root cause at the same time. 

-- Logs begin at Tue 2025-07-01 09:00:00, end at Tue 2025-07-18 14:30:00 --
Jul 18 14:25:00 lab25 kernel: CPU soft lockup detected...
Step 2 : Identify large logs under /var/log and confirm disk pressure.
Commands 
df -h /
sudo du -sh /var/log/* 2>/dev/null | sort -h | tail

df -h confirms whether the filesystem is actually under pressure. du gives you evidence of which files or directories are consuming space. Sorting and showing the largest entries gets you to the culprit fast. 

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda2        40G   39G  1.0G  98% /

5.0G    /var/log/syslog
Step 3 : Truncate an oversized log safely.
Safety note

Prefer truncation over deletion. Truncation preserves file ownership, permissions, SELinux labels, and open file handles while reclaiming disk space.

Command 
sudo truncate -s 0 /var/log/syslog

truncate -s 0 drops file contents to zero bytes without removing the file itself. This is the “stabilize without destroying evidence structure” move. 

ls -l /var/log/syslog
sudo du -sh /var/log/syslog
Step 4 : Identify the master logrotate configuration and included rules.
Commands 
ls -l /etc/logrotate.conf
ls -l /etc/logrotate.d

/etc/logrotate.conf is the entry point for rotation policy. It typically sets global defaults and includes per-package rules from /etc/logrotate.d. This is where you confirm that the relevant service has a rotation rule.

Step 5 : Force logrotate to run immediately.
Command 
sudo logrotate -f /etc/logrotate.conf

Forcing a run validates that your configuration is syntactically correct and rotations happen as intended without waiting for the schedule. After the run, verify that rotated artifacts exist and that the file does not immediately regrow due to an ongoing log storm. 

sudo ls -lh /var/log | head
sudo du -sh /var/log/* 2>/dev/null | sort -h | tail

Breakpoints

Disk still full after truncation

The space may be consumed elsewhere, or the process is still writing aggressively. Re-run df -h and re-check the largest paths under /var. If a deleted log is still held open by a process, truncation usually helps, but confirm growth rate by watching the file size for a minute.

Log file keeps growing instantly

You stabilized momentarily, but the root cause is still producing output. Use journalctl -f to identify the service and then investigate that unit. Rotation is not a fix for a runaway error loop.

logrotate -f runs but nothing rotates

The rule may not match the file path, the config may be in a different per-service file, or the state file prevents rotation conditions from being met without -f. Confirm the target log is covered by a rule in /etc/logrotate.d.

Permission or SELinux denial during rotation

If rotation creates new files, permissions and contexts must be correct. Investigate denials in logs and confirm the rotation rule sets ownership and modes as needed.

Cleanup checklist

This lab makes no permanent changes besides truncating a log and manually forcing rotation. Your cleanup is verifying the filesystem is stable and rotation policy is discoverable and consistent.

Commands 
df -h /
sudo du -sh /var/log/* 2>/dev/null | sort -h | tail
ls -l /etc/logrotate.conf
ls -l /etc/logrotate.d
Success signal

Root filesystem usage drops, the oversized log is no longer consuming gigabytes, and logrotate policy is clearly defined under /etc/logrotate.conf and /etc/logrotate.d.

Reference

  • journalctl: systemd journal viewer
    • -f : Follow new log entries in real time.
  • df: filesystem space reporting
    • -h : Human-readable sizes (GiB, MiB).
  • du: disk usage reporting
    • -s : Summarize total for each argument.
    • -h : Human-readable sizes (GiB, MiB).
  • truncate: resize a file to a specified size
    • -s 0 : Truncate file to zero bytes (preserve inode and permissions).
  • logrotate rotate, compress, and manage log files
    • -f : Force rotation regardless of normal conditions.
  • /etc/logrotate.conf: global logrotate configuration
  • /etc/logrotate.d: per-service logrotate rules directory