GrepBoard/Manuals
51 manuals ·

Find what is using the space when a disk is full

·Ubuntu 24.04.5 LTS · coreutils 9.4 · lsof 4.95.0 · ext4 loop file system·6 steps

Find the full file system with df, narrow down big files with du and find, then handle space that does not come back after a delete and inode exhaustion.

When you see No space left on device, first tell whether bytes or inodes ran out, then narrow down where the space went. The output below was run on an ext4 loop file system mounted at /srv/data; only the path was changed to a generic one.

[1] Find the full file system. df -h shows bytes, df -i shows inodes (the limit on the number of files).

console
# df -h /srv/dataFilesystem      Size  Used Avail Use% Mounted on/dev/loop0      574M  474M   59M  89% /srv/data# df -i /srv/dataFilesystem     Inodes IUsed IFree IUse% Mounted on/dev/loop0      38400    17 38383    1% /srv/data

[2] Descend from the mount point and narrow down the big directories. -x stays on one file system, and sort -h sorts M and G units correctly. To scan the whole server, run the same command on /.

console
# du -xh --max-depth=1 /srv/data | sort -h16K	/srv/data/lost+found3.1M	/srv/data/cache121M	/srv/data/log201M	/srv/data/backup324M	/srv/data

[3] Find big files inside the largest directory. -xdev skips other mounts here too.

console
# find /srv/data -xdev -size +100M -exec ls -lh {} +-rw-r--r-- 1 root root 200M Oct 19 12:17 /srv/data/backup/db-2026-09.dump-rw-r--r-- 1 root root 120M Oct 19 12:17 /srv/data/log/app.log

[4] If df does not drop after you delete a file, a process still holds the deleted file open. In this example df reports 474M used while the du total is 324M; the 150M difference is a deleted log file.

console
# rm /srv/data/log/big.log# df -h /srv/dataFilesystem      Size  Used Avail Use% Mounted on/dev/loop0      574M  474M   59M  89% /srv/data

lsof +L1 lists open files with a link count below 1 (zero). Add -a and a mount point to limit it to that file system.

console
# lsof -a +L1 /srv/dataCOMMAND   PID USER   FD   TYPE DEVICE  SIZE/OFF NLINK NODE NAMEsleep   48534 root    1w   REG    7,0 157286400     0   16 /srv/data/log/big.log (deleted)

[5] Get the space back. Preferably restart the process under PID (or reload it so it reopens its log). If you cannot restart it, truncate the open file through the number in the FD column (1w means 1).

console
# ls -l /proc/48534/fdlr-x------ 1 root root 64 Oct 19 12:17 0 -> /dev/nulll-wx------ 1 root root 64 Oct 19 12:17 1 -> /srv/data/log/big.log (deleted)l-wx------ 1 root root 64 Oct 19 12:17 2 -> /dev/null# : > /proc/48534/fd/1# df -h /srv/dataFilesystem      Size  Used Avail Use% Mounted on/dev/loop0      574M  324M  209M  61% /srv/data

[6] If bytes are free but you get No space left on device, check inodes. This is common in directories that collect countless small files (mail queues, sessions, caches). The run below keeps creating files on a small file system built with 256 inodes.

console
# touch /srv/data/spool/m244touch: cannot touch '/srv/data/spool/m244': No space left on device# df -h /srv/dataFilesystem      Size  Used Avail Use% Mounted on/dev/loop0       60M   28K   56M   1% /srv/data# df -i /srv/dataFilesystem     Inodes IUsed IFree IUse% Mounted on/dev/loop0        256   256     0  100% /srv/data

Used is only 28K, yet IUse% is 100%. Find the directory holding the most files with du --inodes.

console
# du -x --inodes --max-depth=1 /srv/data | sort -n1	/srv/data/lost+found245	/srv/data/spool247	/srv/data

#df#du#lsof#disk