Loading...

VONDS ©

Lab 2: Real-Time Log Filter Setup

Back to Lab Index

Set up a real-time log monitoring pipeline to surface critical system failures as they occur using standard CLI tooling. Validate log locations, stream updates safely, and confirm kernel-managed inspection paths under /sys.

troubleshooting core services

Scenario

Your team needs to monitor critical system events in real time. You are tasked with identifying where logs live on disk, building a live filter that highlights failures (panic, denied, segfault, crash, and login failures), and validating key kernel inspection paths under /sys.

Operator context

Establish this level of baseline log visibility before declaring an incident, escalating the issue, or making changes to the system state.

Objective

  • Identify common system log file locations under /var/log.
  • Stream log updates live and filter for high-signal error patterns using tail -f and grep -E.
  • Confirm kernel-managed inspection paths under /sys for module and kernel feature visibility.

Concepts

  • Log locations vary by distro and logging stack; confirm what exists under /var/log before building tooling around it.
  • Live streaming versus historical review: tail -f is for “what is happening now,” not “what happened last night.”
  • High-signal filtering: use a small regex set that catches failures without flooding you with noise.
  • sysfs is a read-only view into live kernel state; it is useful for validating modules and kernel subsystems while you correlate events.

Walkthrough

Step 1: Explore available system log files.
Command 
ls /var/log

This confirms where your distribution keeps system and service logs. Common files you might see include syslog, messages, and auth.log, depending on distro and logging configuration. 

alternatives.log  syslog  dmesg  messages  auth.log
Step 2: Show only critical errors in real time.
Command 
sudo tail -f /var/log/syslog | grep -Ei 'error|fail|panic|denied|segfault|crash'

Use tail -f to stream live updates as they are written, then filter down to high-signal lines with grep -Ei. On some systems, you may stream /var/log/messages instead of /var/log/syslog. 

Jul 16 08:11:17 kernel: [12345.678901] segfault at 00000000 ip 00007f3cd7e1d92c
sp 00007fff5ccfca60 error 4 in libc-2.27.so
Jul 16 08:11:20 sshd[1938]: Failed password for root from 192.168.1.42 port 55874 ssh2
Jul 16 08:11:22 kernel: panic: attempting to kill init!
Jul 16 08:11:23 audit[2001]: denied attempt to access /etc/shadow by uid 1000
Jul 16 08:11:25 systemd[1]: Failed to start Network Manager.
Jul 16 08:11:27 systemd[1]: user@1000.service: Main process exited, code=exited, status=1/FAILURE
Jul 16 08:11:30 crash-handler[2042]: Process 1892 (firefox) crashed with signal 11 (SIGSEGV)
Step 3: Inspect kernel-managed paths.
Command 
ls /sys/module

The /sys filesystem reflects live kernel state. Listing /sys/module shows modules visible to the running kernel, which is useful when correlating hardware, drivers, and kernel events. 

8250             drm_kms_helper      intel_powerclamp  soundcore
acpi_cpufreq     e1000e              i915              serio_raw
ahci             crc32c_intel        input_core        rfkill
Alternate 
ls /sys/kernel

Listing /sys/kernel exposes kernel-level features and subsystems. This can be useful when you need to confirm runtime knobs for logging, tracing, security, or debugging. 

debug            notes               printk            random
security         system              tracing           uevent_helper

Common breakpoints

/var/log/syslog does not exist

Some distros use /var/log/messages or only journald. List /var/log and switch to a file that exists. On systemd-only setups, use journalctl -f instead.

No output from the grep filter

That can be normal if the system is quiet. Trigger a safe event (for example, a failed login) or broaden the pattern temporarily.

Permissions errors when reading logs

Many logs are root-readable. Use sudo or ensure your account has appropriate group membership for log access.

sysfs paths are missing or look different

sysfs layout can vary slightly by kernel and configuration. Confirm /sys is mounted and browse the relevant subtree.

Cleanup checklist

This lab is read-only. Cleanup is simply stopping your live streams cleanly and leaving the terminal in a known state.

Checklist 
# Stop the live stream (Ctrl+C) in the terminal where tail is running.
# Optional: clear the screen and return to your working directory.
clear
pwd
Success signal

You can confidently reproduce the pipeline, adjust log targets per distro, and confirm kernel-visible state under /sys.

Reference

  • ls: Lists files and directories.
  • ls /var/log: Lists common system log files on disk.
    • /var/log: Standard log directory for many services and system components.
  • tail: Prints the last part of files.
  • tail -f <file>: Streams appended log lines as they are written.
    • -f: Follow the file and print new lines in real time.
    • <file>: Path to the log file to follow.
  • grep: Searches input text for matching patterns.
  • grep -E '<pattern>': Filters text output using extended regular expressions.
    • -E: Use extended regular expressions (supports | alternation).
    • <pattern>: Regex pattern to match.
  • grep -i '<pattern>': Filters text output using case-insensitive matching.
    • -i: Case-insensitive matching.
    • <pattern>: Regex pattern to match.
  • grep -Ei '<pattern>': Filters text output using extended regex and case-insensitive matching.
    • -E: Use extended regular expressions (supports | alternation).
    • -i: Case-insensitive matching.
    • <pattern>: Regex pattern to match.
  • /sys: sysfs virtual filesystem exposing live kernel object state.
  • ls /sys/module: Lists modules visible to the running kernel.
    • /sys/module: Directory containing loaded module objects.
  • ls /sys/kernel: Lists kernel feature and subsystem entries exposed via sysfs.
    • /sys/kernel: Directory containing kernel subsystem entries.