Loading...

VONDS ©

Lab 136: Fix Missing Journal Logs and Restore Time Sync

Back to Lab Index

Investigate missing recent journal logs after an outage and restore log visibility for a service unit. Re-enable time synchronization using chronyd and prove the host is syncing again with chronyc.

services logging time-sync incident-response core

Scenario

This host is failing health checks after an outage. The app team cannot find recent logs in the system journal, and TLS checks fail because the system clock is drifting. Your job is to restore journald visibility for the affected unit and re-enable time synchronization.

Operator context

Health checks do not care why the outage happened. You need logs for root cause work and accurate time for TLS, authentication, and correlation across systems.

Objective

  • Confirm systemd-journald is running.
  • Inspect recent unit logs for webapp.service.
  • Restart journald to restore logging flow.
  • Confirm logs from the current boot are present.
  • Check time sync state and confirm NTP is not active.
  • Enable and start chronyd and turn NTP on.
  • Verify the host is syncing time using chronyc.

Concepts

  • “Missing logs” can be service state, journal state, scope (wrong boot/unit), or disk pressure.
  • In incidents, scope to the current boot with journalctl -b to avoid chasing old data.
  • Accurate time is a dependency for TLS, Kerberos, API signatures, and cross-host correlation.
  • “Service is running” is not proof of sync; use chronyc tracking for evidence.

Walkthrough

Step 1 : Confirm journald is running.
Command 
systemctl status systemd-journald --no-pager
# OR
sudo systemctl status systemd-journald --no-pager

Start with service health. If journald is down, you will not get reliable unit logs. 

● systemd-journald.service - Journal Service
     Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static)
     Active: active (running) since Thu 2026-01-15 11:03:12 UTC; 6min ago
   Main PID: 610 (systemd-journald)
Step 2 : Pull recent logs for the failing unit.
Command 
journalctl -u webapp.service -n 20 --no-pager
# OR
sudo journalctl -u webapp.service -n 20 --no-pager

Unit-scoped logs keep output focused. If you see errors that hint at disk pressure or journal write failures, treat that as a strong lead. 

Jan 15 11:06:41 host webapp[2142]: ERROR: failed to write to journal (No space left on device)
Jan 15 11:06:41 host webapp[2142]: WARN: continuing without structured logs
Step 3 : Restart journald to restore logging.
Command 
sudo systemctl restart systemd-journald
# OR
systemctl restart systemd-journald

A restart is a common first remediation when journaling is wedged after an outage. Validate that logs are flowing again next.

Step 4 : Confirm logs exist for the current boot.
Command 
journalctl -b -u webapp.service -n 10 --no-pager
# OR
sudo journalctl -b -u webapp.service -n 10 --no-pager

Scoping to the current boot (-b) proves you are seeing live data, not old history. 

Jan 15 11:09:02 host systemd[1]: Started webapp.service - Internal Web App.
Jan 15 11:09:02 host webapp[2310]: INFO: listening on 0.0.0.0:8080
Step 5 : Check time synchronization state.
Command 
timedatectl status
# OR
sudo timedatectl status

Confirm whether the system clock is synchronized and whether an NTP service is active. 

Local time: Thu 2026-01-15 11:09:10 UTC
Universal time: Thu 2026-01-15 11:09:10 UTC
RTC time: Thu 2026-01-15 10:37:51
Time zone: UTC (UTC, +0000)
System clock synchronized: no
NTP service: inactive
Step 6 : Enable time sync with chronyd and turn NTP on.
Commands 
sudo systemctl enable --now chronyd
sudo timedatectl set-ntp true

Enable the chrony client and ensure NTP is turned on so the host can converge with time sources.

Step 7 : Prove the system is syncing with chrony.
Command 
chronyc tracking
# OR
sudo chronyc tracking

Look for a valid reference, a reasonable stratum, and a normal leap status. 

Reference ID    : 203.0.113.10 (ntp1.example.net)
Stratum         : 3
System time     : 0.000002341 seconds slow of NTP time
Last offset     : -0.000001102 seconds
Leap status     : Normal

Breakpoints

No logs after restart

Confirm you are scoping correctly with -b and -u. If the unit is not emitting logs, verify it is actually starting and check systemctl status webapp.service.

Journal writes fail with disk pressure

If you see “No space left on device”, journald can drop messages even if the service is running. Confirm free space on the filesystem that backs the journal and restore headroom.

Chrony is running but sync is not happening

If chronyc tracking shows no reference or an abnormal leap status, check network reachability to NTP sources and confirm any egress firewall policy allows NTP.

Cleanup checklist

If you need to disable NTP again for a controlled test, turn it off and stop chrony.

Commands 
sudo timedatectl set-ntp false
sudo systemctl disable --now chronyd

Reference

  • systemctl status <unit> : Check whether a unit is active and view recent status info.
    • --no-pager : Print output without paging.
  • journalctl -u <unit> : View logs for a specific service unit.
    • -n <N> : Show the most recent N lines.
    • --no-pager : Print output without paging.
  • journalctl -b : Scope logs to a boot.
    • -b : Current boot (use -b -1 for previous boot).
  • systemctl restart systemd-journald : Restart journald.
  • timedatectl status : Show time, timezone, and synchronization state.
  • systemctl enable --now chronyd : Enable chrony at boot and start it immediately.
  • timedatectl set-ntp true|false : Enable or disable system time synchronization.
  • chronyc tracking : Show chrony synchronization status and offset details.