System logs are the black box of every Linux server. When an intrusion occurs, logs are often the first — and sometimes only — evidence trail available. Knowing where to look, what to search for, and how to automate that process is a fundamental skill for any security professional or sysadmin serious about defending their infrastructure.

Critical Log Files You Must Know

Linux stores most of its log data under /var/log/. Not every file matters equally from a security standpoint. Here are the ones you should prioritize:

  • /var/log/auth.log (Debian/Ubuntu) or /var/log/secure (RHEL/CentOS) — Authentication events: SSH logins, sudo usage, PAM activity. This is ground zero for brute-force detection.
  • /var/log/syslog or /var/log/messages — General system activity including kernel messages, service start/stop events, and hardware errors.
  • /var/log/kern.log — Kernel-level messages. Useful for spotting iptables drops, module loading, and hardware anomalies.
  • /var/log/faillog — Records failed login attempts. Read it with the faillog command.
  • /var/log/lastlog — Last login records for every user. Read with lastlog.
  • /var/log/cron — Cron job execution logs. Attackers frequently abuse cron for persistence.
  • /var/log/apache2/access.log or /var/log/nginx/access.log — Web server access logs, essential for detecting scanning, exploitation attempts, and abnormal traffic patterns.

Quick Health Check

A fast way to gauge recent suspicious activity on any server:

# Last 20 authentication failures
grep "Failed password" /var/log/auth.log | tail -20

# Who logged in recently
last -20

# Users that have never logged in
lastlog | grep "Never logged in"

Mastering journalctl

On systems running systemd, journalctl is the primary interface to the journal — a structured, indexed binary log that replaces or complements traditional text logs.

Essential Security Queries

# All SSH-related messages since yesterday
journalctl -u sshd --since "yesterday"

# Authentication failures in the last hour
journalctl -p err -t sshd --since "1 hour ago"

# Kernel messages (firewall drops, module loads)
journalctl -k --since today

# Follow logs in real time for a specific service
journalctl -u nginx.service -f

# Show logs from a specific boot (useful after a crash)
journalctl -b -1 -p warning

The -p flag filters by priority (emerg, alert, crit, err, warning, notice, info, debug), which is invaluable for narrowing down noise during incident response.

Persistent Journal Storage

By default, some distributions store the journal only in memory. To ensure logs survive reboots:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Grep and Awk Patterns for Threat Detection

Raw log files are text. Grep and awk are your scalpels.

Detecting Brute-Force SSH Attacks

# Count failed SSH attempts per IP address
grep "Failed password" /var/log/auth.log \
  | awk '{print $(NF-3)}' \
  | sort | uniq -c | sort -rn | head -20

This one-liner extracts the source IP from each failed password line, counts occurrences, and sorts them. Any IP with hundreds or thousands of attempts is almost certainly a brute-force bot.

Spotting Suspicious Sudo Activity

# All sudo commands executed today
grep "COMMAND=" /var/log/auth.log | grep "$(date '+%b %e')"

# Failed sudo attempts (wrong password or unauthorized user)
grep "NOT in sudoers\|authentication failure" /var/log/auth.log

A user not in the sudoers file attempting sudo is a red flag — it could indicate lateral movement after an initial compromise.

Hunting for Unusual Login Times

# Logins outside business hours (before 7 AM or after 9 PM)
awk '/session opened/ && /sshd/ {print $1, $2, $3}' /var/log/auth.log \
  | awk -F: '{if ($1 < 7 || $1 > 21) print}'

Web Server Attack Patterns

# SQL injection attempts in Apache logs
grep -iE "(union.*select|drop.*table|<script>|\.\.\/)" /var/log/apache2/access.log

# Top 10 IPs by request count (DDoS indicator)
awk '{print $1}' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn | head -10

Log Rotation: Don’t Let Logs Eat Your Disk

The logrotate utility handles automatic compression and deletion of old logs. Its configuration lives at /etc/logrotate.conf with per-service overrides in /etc/logrotate.d/.

A security-hardened rotation policy for auth logs:

# /etc/logrotate.d/auth-security
/var/log/auth.log {
    weekly
    rotate 52
    compress
    delaycompress
    missingok
    notifempty
    create 0640 root adm
}

Keeping 52 weeks of authentication logs gives you a full year of forensic history. Adjust retention based on your compliance requirements and disk budget.

Centralized Logging with rsyslog

On multi-server environments, shipping logs to a central collector is non-negotiable. Rsyslog is installed by default on most distributions and supports TCP/UDP forwarding with TLS encryption.

Client Configuration

# /etc/rsyslog.d/50-remote.conf
*.* @@logserver.internal.net:514   # TCP forwarding (double @)

Server Configuration

# /etc/rsyslog.conf — enable TCP listener
module(load="imtcp")
input(type="imtcp" port="514")

# Organize logs by source hostname
template(name="RemoteLogs" type="string"
  string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log")
*.* ?RemoteLogs

Centralized logging ensures that even if an attacker wipes local logs after gaining root, your evidence is already stored elsewhere.

Tools That Amplify Your Analysis

Fail2Ban

Fail2Ban monitors log files for patterns (like repeated SSH failures) and automatically bans offending IPs via iptables or nftables. Its default SSH jail is effective out of the box:

sudo apt install fail2ban
sudo systemctl enable fail2ban

# Check banned IPs
sudo fail2ban-client status sshd

# Manually unban an IP
sudo fail2ban-client set sshd unbanip 203.0.113.50

Customize /etc/fail2ban/jail.local to tune maxretry, findtime, and bantime thresholds. For aggressive attackers, setting bantime = -1 imposes a permanent ban.

GoAccess

GoAccess is a real-time web log analyzer that generates interactive HTML reports or runs in the terminal:

# Terminal dashboard
goaccess /var/log/nginx/access.log --log-format=COMBINED

# Generate an HTML report
goaccess /var/log/nginx/access.log -o /var/www/html/report.html \
  --log-format=COMBINED --real-time-html

It visualizes top visitors, requested paths, HTTP status codes, and geographic data — perfect for spotting scanning bots and unusual traffic spikes at a glance.

Building a Daily Monitoring Habit

Automated tools catch the obvious threats. Skilled analysis catches the subtle ones. A practical daily routine looks like this:

  1. Check Fail2Ban status — review newly banned IPs and any recidivists.
  2. Scan auth.log — look for new user creation, sudo anomalies, and off-hours logins.
  3. Review web access logs — hunt for 4xx/5xx spikes and suspicious user agents.
  4. Verify log integrity — confirm rsyslog is forwarding, disk space is healthy, and rotation is running.

Logs don’t lie, but they also don’t explain themselves. The difference between a compromised server and a defended one is often just someone reading the logs before it was too late.