Loading...

VONDS ©

Lab 13: Log Management and Troubleshooting with journalctl

Back to Lab Index

Investigate cron execution issues by querying the systemd journal with unit filters and priority views. Correlate evidence across journal output and classic log files, then confirm how log rotation is invoked and where rotation policy is defined.

troubleshooting services core

Scenario

A user reports that cron jobs are not being executed properly. You must inspect cron service logs, pull the most recent high-signal messages with context, filter for failures, and confirm where persistent logs live on disk. To close the loop, identify how log rotation is invoked and where per-service rotation policies are stored.

Operator context

When scheduled jobs do not run, the fastest path is proving: (1) the cron daemon is active, (2) the job fired, and (3) output or errors exist somewhere you can read.

Objective

  • Show service logs for the cron unit using journalctl -u.
  • Pull the most recent unit messages with context using -x and -e.
  • Filter log output for failures using either grep or priority filters.
  • View classic persistent logs under /var/log when present.
  • Display the last 20 journal entries.
  • Identify how log rotation is invoked and where rotation policies live.

Concepts

  • The journal is queryable by unit, time range, and priority. That is usually faster than scanning a full log file.
  • Cron may run as cron or crond depending on distro and packaging.
  • Not all environments persist journal logs to disk. If persistence is disabled, older entries may disappear after reboot.
  • Classic log files under /var/log still matter when rsyslog is in the pipeline or when you need file-based evidence.
  • logrotate manages file-based logs. Policy is typically centralized in /etc/logrotate.conf and /etc/logrotate.d/.

Walkthrough

Step 1: Show logs for the cron service unit.
Command 
journalctl -u cron
# OR (common on some distros)
journalctl -u crond

The -u filter limits output to a single unit. This is your baseline view to confirm the service is starting, receiving jobs, and emitting messages that indicate execution. 

# Example patterns:
systemd[1]: Started Regular background program processing daemon.
CRON[891]: (root) CMD (/usr/local/bin/daily-report.sh)
Step 2: Show the most recent unit messages with extra context.
Command 
journalctl -xeu cron
# OR
journalctl -xeu crond

-e jumps to the end (most recent), -x adds explanatory text when available, and -u keeps output scoped to the unit. This is the quickest way to see what just happened. 

# Example patterns:
CRON[1023]: (lab) CMD (/usr/local/bin/backup.sh)
pam_unix(cron:session): session opened for user lab by (uid=0)
pam_unix(cron:session): session closed for user lab
Step 3: Filter logs for failures.
Command 
journalctl -p err -b
# OR (quick keyword sweep when you do not know the unit yet)
journalctl -b | grep -i -E 'fail|error|denied'

Priority filtering is cleaner than raw grep when you want true errors and above. Adding -b limits output to the current boot, which reduces noise when the report is time-bound. 

# Example patterns:
CRON[1555]: (lab) ERROR (No MTA installed, discarding output)
sudo[1601]: pam_unix(sudo:auth): authentication failure
Step 4: Validate persistent logs under /var/log when present.
Command 
# Debian/Ubuntu commonly:
less /var/log/syslog

# RHEL-family commonly:
less /var/log/messages

Some environments still write service messages into classic log files (often via rsyslog). This is a quick “is anything being written to disk” check, and it helps when journal retention is short or persistence is disabled. 

# Example patterns:
CRON[1201]: (lab) CMD (/usr/local/bin/backup.sh)
CRON[1201]: (lab) ERROR (No MTA installed, discarding output)
Step 5: Show the last 20 journal entries.
Command 
journalctl -n 20

This is a fast snapshot when you do not yet know which unit is involved. It is also useful immediately after reproducing an issue. 

# Example header and patterns:
-- Logs begin Mon 2025-07-01 08:00:00 --
CRON[1127]: (root) CMD (/usr/local/bin/daily-report.sh)
Step 6: Trigger log rotation and confirm where policies live.
Command 
# Policy locations:
cat /etc/logrotate.conf
ls -l /etc/logrotate.d/

# Run logrotate (dry run first):
sudo logrotate -d /etc/logrotate.conf

# Force a rotation run (use carefully on real systems):
sudo logrotate -f /etc/logrotate.conf

logrotate manages file-based logs (typically under /var/log). A dry run shows what would happen without changing anything. Policies are usually split into a global config plus per-service files. 

# Example output clue (dry run):
reading config file /etc/logrotate.conf
including /etc/logrotate.d
considering log /var/log/syslog

Common breakpoints

No output for the unit

The unit name may differ. Try cron and crond, or list candidates with systemctl list-units | grep -i cron.

Logs disappear after reboot

Journal persistence may be disabled. Confirm whether persistent storage is configured, and rely on file-based logs if required.

Cron shows CMD, but nothing else

Cron may discard output if mail is not configured. Redirect stdout and stderr in the job or script so failures are captured.

logrotate changes too much

Start with logrotate -d to preview behavior. Forcing rotation is useful for validation, but avoid it on production systems without approval.

Cleanup checklist

This lab reads logs and configuration. If you created any test cron jobs in earlier labs, remove them separately and verify the crontab is clean.

Commands 
crontab -l
crontab -e
Success signal

You can show (1) unit-scoped journal evidence, (2) filtered error views, and (3) where file-based rotation policy lives for your platform.

Reference

  • journalctl -u <unit>: Shows journal entries for a specific systemd unit (for example cron or crond).
  • journalctl -xeu <unit>: High-signal view of recent unit messages.
    • -x: Adds explanatory text when available.
    • -e: Jumps to the end (most recent).
    • -u: Filters output to a single unit.
  • journalctl -p err -b: Shows error-priority entries from the current boot.
    • -p err: Error severity and above.
    • -b: Current boot only.
  • journalctl -n 20: Displays the last 20 journal entries.
  • /var/log/syslog: Common on Debian/Ubuntu systems (often written via rsyslog).
  • /var/log/messages: Common on RHEL-family systems.
  • logrotate: Rotates, compresses, and manages retention for file-based logs.
    • /etc/logrotate.conf: Main configuration.
    • /etc/logrotate.d/: Per-service policies.
    • logrotate -d: Dry run (preview).
    • logrotate -f: Force a run.