Loading...

VONDS ©

Lab 13: Log Management and Troubleshooting with journalctl

Lab Index

Investigate cron execution issues by querying systemd journal logs with targeted unit filters and priority views. Validate evidence across journalctl output and classic log files, then confirm how log rotation is triggered and controlled.

troubleshooting services core

Scenario

A user reports that cron jobs are not being executed properly. You must inspect cron service logs, pull the most recent high-signal messages with context, filter for failures, and confirm where persistent logs live on disk. To close the loop, you will identify how log rotation is invoked and where per-service rotation policies are stored.

Operator context

When scheduled jobs “don’t run,” the fastest path is proving: (1) the cron daemon is active, (2) the job fired, and (3) the output/errors exist somewhere you can actually read.

Objective

  • Show service logs for the cron unit using journalctl -u .
  • Pull the most recent unit messages with context using -x and -e .
  • Filter log output for failures using either grep or priority filters.
  • View classic persistent logs under /var/log when present.
  • Display the last 20 journal entries.
  • Identify how log rotation is invoked and where rotation policies live.

What You’ll Practice

  • Unit-scoped journal queries using journalctl -u .
  • High-signal troubleshooting views using journalctl -xeu .
  • Failure-focused log filtering using journalctl -p and simple grep pipelines.
  • Reading persistent log files in /var/log using safe pagers like less .
  • Quick recent history checks using journalctl -n .
  • Understanding log rotation tooling and where policy is defined.

Walkthrough

Step 1 : Show logs for the cron service unit.
Command 
journalctl -u cron
# OR (common on some distros)
journalctl -u crond

The -u filter limits output to a single unit. This is your baseline view to confirm the service is starting, receiving jobs, and emitting relevant messages for execution. 

# Example patterns:
systemd[1]: Started Regular background program processing daemon.
cron[891]: (root) CMD (/usr/local/bin/daily-report.sh)
Step 2 : Show the most recent unit messages with extra context.
Command 
journalctl -xeu cron
# OR
journalctl -xeu crond

-e jumps to the end (most recent), -x adds explanatory text when available, and -u keeps output scoped to the unit. This is a high-signal view when you need to focus on “what just happened.” 

# Example patterns:
CRON[1023]: (lab) CMD (/usr/local/bin/backup.sh)
pam_unix(cron:session): session opened for user lab by (uid=0)
pam_unix(cron:session): session closed for user lab
Step 3 : Filter logs for failures.
Command 
journalctl -p err -b
# OR (simple keyword sweep)
journalctl | grep -i fail

Priority filtering is cleaner than raw grep when you want true “errors and above.” Adding -b limits output to the current boot, which reduces noise when troubleshooting something that “stopped working today.” 

# Example pattern:
Failed password for invalid user guest from 10.0.0.23 port 53218 ssh2
Step 4 : View classic persistent logs when present.
Command 
less /var/log/syslog
# OR (quick tail view)
tail /var/log/syslog

Some environments still write service messages into classic log files (often via rsyslog). Checking /var/log/syslog is a quick “is anything writing to disk” validation, and it helps when journal retention is short or logs are not persistent. 

# Example patterns:
CRON[1201]: (lab) CMD (/usr/local/bin/backup.sh)
systemd[1]: Finished Cleanup of Temporary Directories.
sshd[1350]: Failed password for invalid user admin from 192.168.1.50 port 45822 ssh2
Step 5 : Show the last 20 journal entries.
Command 
journalctl -n 20

This is a fast “what just happened” snapshot when you do not yet know which unit is involved. It is also useful after you reproduce an issue and want immediate evidence. 

# Example header and patterns:
-- Logs begin Mon 2025-07-01 08:00:00 --
CRON[1127]: (root) CMD (/usr/local/bin/daily-report.sh)
Step 6 : Trigger log rotation and confirm where policies live.
Command 
sudo logrotate /etc/logrotate.conf
# OR (force a run)
sudo logrotate -f /etc/logrotate.conf

logrotate applies rotation rules to log files on disk (typically under /var/log ). The main configuration is /etc/logrotate.conf , and per-service rules commonly live in /etc/logrotate.d/ . 

# Inspect policy locations:
ls -l /etc/logrotate.d/
cat /etc/logrotate.conf

Reference

  • journalctl -u <unit> : Shows journal entries for a specific systemd unit (for example cron or crond ).
  • journalctl -xeu <unit> : Jumps to the most recent unit messages and includes additional context.
    • -x : Adds explanatory text when available.
    • -e : Jumps to the end (most recent messages).
    • -u <unit> : Filters output to a single unit.
  • journalctl -p err -b : Shows error-priority journal entries from the current boot.
    • -p err : Limits output to error severity and above.
    • -b : Limits output to the current boot.
  • /var/log/syslog : Common persistent log file on Debian/Ubuntu systems (often written via rsyslog).
  • journalctl -n 20 : Displays the last 20 journal entries.
  • logrotate : Rotates, compresses, and manages retention for log files on disk.
    • /etc/logrotate.conf : Main logrotate configuration.
    • /etc/logrotate.d/ : Per-service rotation policies.