Logs & journald¶

When something breaks, the logs tell you why. Modern systems collect logs in two places: the systemd journal (queried with journalctl) and traditional text files under /var/log. This page covers both, plus log rotation.

Tested on

AlmaLinux 9.4 and Ubuntu 22.04 LTS. journalctl is identical on both; the traditional log filenames differ (noted inline).

The systemd journal¶

systemd-journald collects logs from the kernel, services, and applications into a structured, indexed binary store. You read it with journalctl. It captures far more than plain text files: per-message metadata, the originating unit, priority, and more.

Everyday journalctl¶

# View the whole journal in a pager (oldest first)
journalctl

# Jump to the END (newest entries) immediately
journalctl -e

# Show only the last N lines
journalctl -n 50

# Follow the log live, like 'tail -f'
journalctl -f

Filtering by unit (service)¶

# Logs for a single service
journalctl -u sshd

# Follow one service live
journalctl -u nginx -f

# Combine: last 100 lines of a service, then follow
journalctl -u nginx -n 100 -f

This is the single most useful pattern when debugging a service — see systemd Service Management for managing the services themselves.

Filtering by priority¶

Priorities range from 0 (emerg) to 7 (debug). -p shows the level you name and more severe.

# Show errors and worse (err, crit, alert, emerg)
journalctl -p err

# Errors from one service since today
journalctl -u nginx -p err --since today

Level	Number	Keyword
Emergency	0	`emerg`
Alert	1	`alert`
Critical	2	`crit`
Error	3	`err`
Warning	4	`warning`
Notice	5	`notice`
Info	6	`info`
Debug	7	`debug`

Filtering by time¶

# Since/until accept absolute and relative times
journalctl --since "2026-06-07 09:00:00"
journalctl --since "1 hour ago"
journalctl --since yesterday --until "09:00"
journalctl -u sshd --since "10 min ago"

Boots and the kernel¶

# List all recorded boots (only if persistent storage is on — see below)
journalctl --list-boots

# Logs from the current boot only
journalctl -b

# Logs from the PREVIOUS boot (great for diagnosing a crash/reboot)
journalctl -b -1

# Kernel messages only (equivalent to dmesg, but timestamped & persistent)
journalctl -k

Combine freely

Filters stack. journalctl -u nginx -p warning -b --since "30 min ago" reads as: nginx, warnings-or-worse, this boot, last 30 minutes.

Journal disk usage and persistence¶

How much space is it using?¶

journalctl --disk-usage

Archived and active journals take up 312.0M in the file system.

Trimming the journal¶

# Keep at most 200 MiB of journal data
sudo journalctl --vacuum-size=200M

# Or drop anything older than 2 weeks
sudo journalctl --vacuum-time=2weeks

Make logs survive reboots¶

By default on many systems the journal is stored in /run/log/journal (volatile — wiped on reboot). To keep history across reboots, enable persistent storage:

# Create the persistent directory
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal

Then set the storage mode in /etc/systemd/journald.conf:

# /etc/systemd/journald.conf
[Journal]
Storage=persistent
SystemMaxUse=500M          # cap total journal size

# Apply
sudo systemctl restart systemd-journald
journalctl --list-boots    # should now list more than one boot

Why this matters

Without persistent storage, journalctl -b -1 is empty — you lose the logs that explain why the system rebooted or crashed. On servers, enabling persistence is strongly recommended.

Traditional log files in /var/log¶

Even with journald, many programs (and rsyslog) still write plain-text logs. These are easy to grep, tail, and ship to other tools.

File (RHEL/AlmaLinux)	Debian/Ubuntu equivalent	Contents
`/var/log/messages`	`/var/log/syslog`	General system / service messages
`/var/log/secure`	`/var/log/auth.log`	Authentication, sudo, SSH logins
`/var/log/cron`	`/var/log/syslog` (cron lines)	Cron job execution
`/var/log/boot.log`	`/var/log/boot.log`	Boot-time service output
`/var/log/dnf.log`	`/var/log/apt/`	Package manager activity
`/var/log/httpd/`	`/var/log/apache2/` or `/var/log/nginx/`	Web server access/error logs

# Follow auth attempts live (RHEL)
sudo tail -f /var/log/secure
# Debian/Ubuntu:
sudo tail -f /var/log/auth.log

# Search for failed SSH logins
sudo grep "Failed password" /var/log/secure

logrotate¶

Without rotation, log files grow until they fill the disk. logrotate runs (via a systemd timer or cron) to compress, archive, and prune old logs.

Main config: /etc/logrotate.conf
Per-application rules: /etc/logrotate.d/*

# Example: /etc/logrotate.d/myapp
cat /etc/logrotate.d/myapp

/var/log/myapp/*.log {
    daily                 # rotate once a day
    rotate 14             # keep 14 old copies
    compress              # gzip rotated logs
    delaycompress         # don't compress the most recent rotation yet
    missingok             # don't error if the log is missing
    notifempty            # skip rotation if the log is empty
    create 0640 myapp myapp   # recreate with these perms/owner
    postrotate
        systemctl reload myapp >/dev/null 2>&1 || true
    endscript
}

# Test a config without rotating (dry run, verbose)
sudo logrotate -d /etc/logrotate.d/myapp

# Force a rotation now (for testing)
sudo logrotate -f /etc/logrotate.d/myapp

postrotate

After rotating, a service may still hold the old (now-renamed) file open. The postrotate block reloads or signals the service so it reopens the fresh file. For journald-managed logs you don't need logrotate — journald handles its own retention via the limits above.

A note on rsyslog¶

rsyslog is the classic syslog daemon. On RHEL/AlmaLinux it's what writes /var/log/messages, /var/log/secure, etc. by reading from the journal and applying rules in /etc/rsyslog.conf and /etc/rsyslog.d/.

# Is rsyslog running?
systemctl status rsyslog

Its main job today is routing: filtering messages by facility/severity into specific files, and forwarding logs to a central server (e.g. a log host or SIEM):

# /etc/rsyslog.d/50-remote.conf — forward everything to a central collector over TCP
*.*  @@logserver.example.com:514

(@@ = TCP, a single @ = UDP.) Restart with sudo systemctl restart rsyslog after changes.

Troubleshooting tips¶

Start at the service: journalctl -u <service> -e then scroll up from the newest entries.
Watch live while reproducing: run journalctl -u <service> -f in one terminal and trigger the problem in another.
Narrow by time: if you know roughly when it broke, --since "10 min ago" cuts the noise dramatically.
Filter to errors: journalctl -p err -b shows everything that went wrong this boot.
After a crash/reboot: journalctl -b -1 -p warning (requires persistent storage) to see what the last boot logged before going down.
Auth issues (SSH, sudo, login failures): check /var/log/secure (RHEL) or /var/log/auth.log (Debian).
Disk full from logs: journalctl --disk-usage and du -sh /var/log/* to find the culprit, then vacuum or fix the noisy service.

Verify your work¶

# Confirm journald is collecting logs and how much space it uses
systemctl is-active systemd-journald
journalctl --disk-usage

# Confirm persistent storage is on (more than one boot listed)
journalctl --list-boots

# Confirm you can read a service's logs
journalctl -u sshd -n 20 --no-pager

# Confirm a logrotate rule parses cleanly
sudo logrotate -d /etc/logrotate.conf >/dev/null && echo "logrotate config OK"

If --list-boots shows multiple boots, persistence is working; a clean logrotate -d means your rotation rules are valid.

Summary¶

The systemd journal is the primary log store; query it with journalctl — -u (unit), -f (follow), -e/-n (end/last N), -p (priority), --since/--until, -b (boot), -k (kernel).
Manage journal size with journalctl --disk-usage and --vacuum-size/--vacuum-time; enable history across reboots with /var/log/journal + Storage=persistent.
Traditional text logs live in /var/log — messages/secure/cron on RHEL, syslog/auth.log on Debian/Ubuntu.
logrotate (/etc/logrotate.d/*) compresses and prunes log files; rsyslog routes and can forward logs to a central server.
When troubleshooting: start at the service unit, follow live while reproducing, filter by time and priority, and check the previous boot after a crash.