Disk Full: No space left on device¶

A full filesystem is one of the most common — and most disruptive — failures on a Linux server. This playbook walks you from the error to the cause to the fix, including the two traps that catch everyone: deleted-but-still-open files and inode exhaustion.

Tested on

AlmaLinux 9 / RHEL 9 with the default XFS root filesystem. Most commands are distro-agnostic. On Debian/Ubuntu the journal lives under /var/log/journal the same way, but the default root filesystem is usually ext4 — grow it with resize2fs instead of xfs_growfs (noted inline below).

Symptom¶

Commands and applications fail with No space left on device (errno ENOSPC).
Services crash or refuse to start; databases go read-only; cron jobs fail silently.
You sometimes can't even log in, because the session can't write its files (e.g. ~/.bashrc history, temp files), or sudo can't write a lock.
Writes fail even though df -h looks like there's free space (classic inode exhaustion).

Likely causes¶

Runaway logs — an application or journald writing gigabytes to /var/log.
One or more large files — a forgotten core dump, a giant SQL/backup dump, an unrotated app log.
Deleted-but-still-open files — a file was rm'd but a running process still holds it open, so the space is not released.
Inode exhaustion — millions of tiny files (mail queues, session files, caches) use up all inodes while bytes remain free.
Old package and journal data — dnf caches and accumulated journal history.

Diagnose¶

1. Confirm space and inodes¶

# Bytes free per filesystem
df -h

# Inodes free per filesystem — DO NOT skip this
df -i

Check df -i early

If df -h shows free space but you still get No space left on device, look at df -i. A IUse% of 100% means you're out of inodes, and freeing bytes won't help — you must delete files (count, not size).

2. Find where the space went¶

# Top-level usage of the root filesystem only (-x stays on one filesystem)
sudo du -xhd1 / | sort -rh | head

# Then drill into the biggest directory it reports, e.g.:
sudo du -xhd1 /var | sort -rh | head

Repeat the drill-down until you reach the offending directory.

3. Hunt for individual large files¶

# Files over 500 MB, staying on the same filesystem
sudo find / -xdev -type f -size +500M -exec ls -lh {} \; 2>/dev/null

4. Look for deleted-but-still-open files¶

This is the trap that makes df and du disagree: du can't see the file (it's unlinked), but df still counts the space because a process holds it open.

# Files that are open but have been unlinked (deleted)
sudo lsof -nP +L1

# Or the classic grep form
sudo lsof -nP | grep -i deleted

The output shows the PID holding the file and how big it is. Note the process — you'll need to restart it (or truncate via its /proc fd) to reclaim the space.

5. Check journald's footprint¶

journalctl --disk-usage

Fix¶

Work from the safest, most reversible action toward the more involved ones. Always re-run df -h (and df -i) after each step to confirm progress.

Free space from a growing log — truncate, don't rm¶

If a log file is being actively written by a running service, truncating it frees space immediately while keeping the file handle valid:

sudo truncate -s 0 /var/log/some-app/huge.log

Why rm of an open file does NOT free space

If you rm a file that a process still has open, the directory entry disappears but the data blocks stay allocated until that process closes the file (or restarts). du will show the space as gone while df still shows it used — exactly the deleted-but-open case from step 4. Use truncate -s 0 on the live file instead, or restart the process holding the deleted file. As a last resort you can zero it through its file descriptor: sudo truncate -s 0 /proc/<pid>/fd/<fd>.

Trim the systemd journal¶

# Keep at most 200 MB of journal data
sudo journalctl --vacuum-size=200M

# Or keep only the last 7 days
sudo journalctl --vacuum-time=7d

Clear package manager caches¶

sudo dnf clean all          # RHEL/AlmaLinux
# Debian/Ubuntu equivalent:
# sudo apt-get clean

Clear old / rotated logs¶

# See what logrotate has accumulated
ls -lhS /var/log/*.gz /var/log/*/*.gz 2>/dev/null | head

# Remove already-rotated, compressed logs (safe — these are archives)
sudo find /var/log -type f -name "*.gz" -delete

# Force a rotation cycle if logs aren't being rotated
sudo logrotate -f /etc/logrotate.conf

Extend an LVM volume (when you genuinely need more space)¶

If the filesystem is on LVM and the volume group has free extents, you can grow it online with no downtime. First confirm there's room:

sudo vgs        # look at VFree on the volume group
sudo lvs        # current logical volume sizes

Then extend the logical volume and grow the filesystem onto it:

# Add 10 GB to the LV
sudo lvextend -L +10G /dev/mapper/vg_root-lv_root

# Grow the filesystem to fill the LV — XFS (RHEL/AlmaLinux default):
sudo xfs_growfs /            # XFS grows by mountpoint and must be mounted

# Debian/Ubuntu (ext4) equivalent:
# sudo resize2fs /dev/mapper/vg_root-lv_root

One-step grow

lvextend -r -L +10G /dev/mapper/vg_root-lv_root resizes the filesystem automatically (-r) for both XFS and ext4. For the full picture of physical volumes, volume groups, and filesystem growth, see Storage & Filesystems.

XFS can only grow, never shrink

xfs_growfs can enlarge an XFS filesystem but cannot reduce it. Size new volumes with that in mind.

Prevent¶

Rotate logs with logrotate so no single log can grow unbounded. Add a drop-in in /etc/logrotate.d/ for any app that writes its own logs. See Logs & journald.
Cap the journal persistently in /etc/systemd/journald.conf:
```
[Journal]
SystemMaxUse=500M
```
Then sudo systemctl restart systemd-journald.
Monitor disk and inodes with alerts (Zabbix, node_exporter, or a simple cron) so you hear about 80% before you hit 100%. Alert on both bytes and inodes.
Put /var on its own partition/volume so a runaway log or full /var/log can't take down the root filesystem and lock you out.
Restart long-running services after large deletions so they release any deleted-but-open files instead of silently holding the space.

For the general approach behind this playbook, see How to Troubleshoot Any Linux Problem.