Loading...

VONDS ©

Lab 144: Journald Persistence & Rsyslog Remote Forward

Back to Lab Index

Enable persistent journald storage using a drop-in config and confirm /var/log/journal exists. Configure rsyslog to forward all messages to a central collector over TCP/514 using a safe drop-in, validate config syntax, restart cleanly, then generate a test message and confirm it lands locally. Finish by confirming NTP synchronization state using chronyc tracking.

journald rsyslog chrony

Scenario

Ops needs persistent journald logs for auditing, and rsyslog must forward logs to a central collector over TCP. You will implement and verify both changes, then confirm NTP sync state.

Operational note

In production, log forwarding is only half the story: you also validate delivery on the collector. This lab keeps validation local and focuses on safe configuration and verification habits.

Objective

  • Check current journal disk usage with journalctl --disk-usage.
  • Enable persistent journald storage via a drop-in and restart cleanly.
  • Confirm persistent storage directory /var/log/journal exists.
  • Configure rsyslog forwarding over TCP/514 using a drop-in file.
  • Validate rsyslog configuration syntax and restart the service.
  • Generate a test log event and confirm local ingestion.
  • Verify NTP synchronization state using chronyc tracking.

Concepts

  • journald persistence is controlled by Storage= and becomes visible on disk under /var/log/journal.
  • drop-ins keep config changes isolated and auditable, and reduce drift during package updates.
  • rsyslog forwarding uses @ for UDP and @@ for TCP; forwarding rules can live in /etc/rsyslog.d/.
  • rsyslogd -N1 is your “syntax gate” before restarts to avoid self-inflicted outages.
  • time sync is a logging dependency; chronyc tracking confirms whether your timestamps are trustworthy.

Walkthrough

Step 1: Check journald disk usage.
Command 
journalctl --disk-usage

This provides a fast baseline for how much space the journal is consuming on disk.

Archived and active journals take up 144.0M in the file system.
Step 2: Create a journald drop-in to enable persistent storage.
Command 
sudo vim /etc/systemd/journald.conf.d/10-persistent.conf

Add the following content:

File content 
[Journal]
Storage=persistent
Why a drop-in

Drop-ins keep changes isolated and auditable. They also reduce merge conflicts during updates, compared to editing the main config directly.

Step 3: Restart journald to apply the change.
Command 
sudo systemctl restart systemd-journald

Restarting applies the new storage mode and triggers directory creation when appropriate.

Step 4: Verify /var/log/journal exists.
Command 
ls -ld /var/log/journal

Persistent journals live under /var/log/journal. Directory ownership and permissions matter for correct writes. 

drwxr-sr-x. 3 root systemd-journal 4096 Jan 25 07:34 /var/log/journal
Step 5: Configure rsyslog to forward all logs to a collector over TCP.
Command 
sudo vim /etc/rsyslog.d/90-forward.conf

Add the following line:

File content 
*.* @@192.0.2.10:514
TCP vs UDP in rsyslog forwarding

@ is UDP, @@ is TCP. This lab forwards via TCP/514 for more reliable delivery semantics.

Step 6: Validate rsyslog configuration syntax.
Command 
rsyslogd -N1

Always validate syntax before restarting. It prevents self-inflicted outages.

rsyslogd: version 8.2310.0, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.
Step 7: Restart rsyslog to apply forwarding.
Command 
sudo systemctl restart rsyslog

A clean restart after validation ensures the forward rule is active.

Step 8: Generate a test log message.
Command 
logger -p local0.notice 'test: forwarding enabled'

This produces a deterministic event you can trace through the local pipeline.

Step 9: Confirm the message appears locally.
Command 
tail -n 5 /var/log/messages

Local confirmation validates ingestion. Forwarding confirmation requires checking the collector side.

Jan 25 07:36:18 lab144 rsyslogd[1043]: [origin software="rsyslogd" swVersion="8.2310.0" x-pid="1043" x-info="https://www.rsyslog.com"] start
Jan 25 07:36:25 lab144 lab[pts/0]: test: forwarding enabled
Step 10: Verify NTP sync status with chrony.
Command 
chronyc tracking

chronyc tracking summarizes whether the system clock is synchronized and how stable it is.

Reference ID    : C0A80101 (ntp1.example.com)
Stratum         : 3
Ref time (UTC)  : Sun Jan 25 12:36:11 2026
System time     : 0.000012345 seconds fast of NTP time
Last offset     : -0.000004321 seconds
RMS offset      : 0.000015678 seconds
Frequency       : 15.123 ppm fast
Residual freq   : -0.002 ppm
Skew            : 0.045 ppm
Root delay      : 0.012345 seconds
Root dispersion : 0.001234 seconds
Update interval : 64.0 seconds
Leap status     : Normal

Breakpoints

/etc/systemd/journald.conf.d does not exist

Create it first: sudo mkdir -p /etc/systemd/journald.conf.d. Then add the drop-in file.

/var/log/journal is missing after restart

Ensure the drop-in is under /etc/systemd/journald.conf.d, and that it contains a valid [Journal] section with Storage=persistent. Then restart journald again.

rsyslog restart fails

Run rsyslogd -N1 and fix the first reported error before restarting. If SELinux is enforcing, check the journal for denial messages.

No message in /var/log/messages

On some systems, messages may be routed differently or handled by journald only. Verify rsyslog is running and confirm log destination config. Use journalctl -t lab or search for the message in the journal.

chronyc tracking shows unsynchronized

Confirm the service is running, check sources with chronyc sources -v, and verify network reachability to the configured NTP servers.

Cleanup checklist

This lab makes durable changes (journald persistence and rsyslog forwarding). In a real environment, you would leave them in place. For a disposable lab host, you may want a quick rollback plan.

Optional rollback 
sudo rm -f /etc/systemd/journald.conf.d/10-persistent.conf
sudo rm -f /etc/rsyslog.d/90-forward.conf
sudo systemctl restart systemd-journald
sudo systemctl restart rsyslog
Final verification signal

journalctl --disk-usage reports expected behavior, /var/log/journal exists, rsyslog validates cleanly, and chronyc tracking shows normal sync state.

Reference

  • journalctl --disk-usage: show current journal space usage.
  • vim /etc/systemd/journald.conf.d/10-persistent.conf: create a journald drop-in for persistence.
  • Storage=persistent: store journals on disk under /var/log/journal.
  • systemctl restart systemd-journald: apply journald configuration changes.
  • ls -ld /var/log/journal: verify persistent journal directory exists and check permissions.
    • -l: long listing.
    • -d: list the directory itself.
  • vim /etc/rsyslog.d/90-forward.conf: create an rsyslog forwarding drop-in.
  • *.* @@host:514: forward all syslog messages to a remote collector over TCP.
    • @: UDP forwarding.
    • @@: TCP forwarding.
  • rsyslogd -N1: validate rsyslog configuration syntax.
  • systemctl restart rsyslog: apply rsyslog configuration changes.
  • logger -p local0.notice 'message': generate a controlled syslog event.
    • -p facility.level: set syslog priority (facility + severity).
  • tail -n 5 /var/log/messages: confirm recent rsyslog-ingested messages locally.
    • -n 5: show the last 5 lines.
  • chronyc tracking: show chrony synchronization and clock stability summary.