Loading...

VONDS ©

Lab 2: Real-Time Log Filter Setup

Lab Index

Set up a real-time log monitoring pipeline to surface critical system failures as they occur using standard CLI tooling. Validate log locations, stream updates safely, and confirm kernel-managed inspection paths under /sys.

troubleshooting core services

Scenario

Your team needs to monitor critical system events in real time. You are tasked with identifying where logs live on disk, building a live filter that highlights failures (panic, denied, segfault, crash, and login failures), and validating key kernel inspection paths under /sys.

Operator context

Establish this level of baseline log visibility before declaring an incident, escalating the issue, or making changes to the system state.

Objective

  • Identify common system log file locations under /var/log.
  • Stream log updates live and filter for high-signal error patterns using tail -f and grep -E.
  • Confirm kernel-managed inspection paths under /sys for module and kernel feature visibility.

What You’ll Practice

  • Discovering available log files using ls /var/log.
  • Building a live monitoring pipeline using tail -f piped into grep -Ei.
  • Writing practical regex patterns for high-impact events (error, fail, panic, denied, segfault, crash).
  • Inspecting kernel-managed paths using /sys/module and /sys/kernel.

Walkthrough

Step 1 : Explore available system log files.
Command 
ls /var/log

This confirms where your distribution keeps system and service logs. Common files you might see include syslog, messages, and auth.log, depending on distro and logging configuration. 

alternatives.log  syslog  dmesg  messages  auth.log
Step 2 : Show only critical errors in real time.
Command 
sudo tail -f /var/log/syslog | grep -Ei 'error|fail|panic|denied|segfault|crash'

Use tail -f to stream live updates as they are written, then filter down to high-signal lines with grep -Ei. On some systems, you may stream /var/log/messages instead of /var/log/syslog. 

Jul 16 08:11:17 kernel: [12345.678901] segfault at 00000000 ip 00007f3cd7e1d92c
sp 00007fff5ccfca60 error 4 in libc-2.27.so
Jul 16 08:11:20 sshd[1938]: Failed password for root from 192.168.1.42 port 55874 ssh2
Jul 16 08:11:22 kernel: panic: attempting to kill init!
Jul 16 08:11:23 audit[2001]: denied attempt to access /etc/shadow by uid 1000
Jul 16 08:11:25 systemd[1]: Failed to start Network Manager.
Jul 16 08:11:26 gdm-password[1982]: gkr-pam: unable to locate daemon control file
Jul 16 08:11:27 systemd[1]: user@1000.service: Main process exited, code=exited,
status=1/FAILURE
Jul 16 08:11:28 kernel: device-mapper: thin: 253:1: reached low water mark
Jul 16 08:11:30 crash-handler[2042]: Process 1892 (firefox) crashed with signal 11 (SIGSEGV)
Step 3 : Inspect kernel-managed paths.
Command 
ls /sys/module

The /sys filesystem reflects live kernel state. Listing /sys/module shows modules visible to the running kernel, which is useful when correlating hardware, drivers, and kernel events. 

8250             drm_kms_helper      intel_powerclamp  soundcore
acpi_cpufreq     e1000e              i915              serio_raw
ahci             crc32c_intel        input_core        rfkill
Alternate 
ls /sys/kernel

Listing /sys/kernel exposes kernel-level features and subsystems. This can be useful when you need to confirm runtime knobs for logging, tracing, security, or debugging. 

debug            notes               printk            random
security         system              tracing           uevent_helper

Reference

  • ls /var/log : Lists common system log files on disk.
    • /var/log : Standard log directory for many services and system components.
  • tail -f <file> : Streams appended log lines as they are written.
    • -f : Follow the file and print new lines in real time.
  • grep -Ei '<pattern>' : Filters text output using extended regex and case-insensitive matching.
    • -E : Use extended regular expressions (supports | alternation).
    • -i : Case-insensitive matching.
  • /sys : sysfs virtual filesystem exposing live kernel object state.
  • ls /sys/module : Lists modules visible to the running kernel.
  • ls /sys/kernel : Lists kernel feature/subsystem entries exposed via sysfs.