No space left on device – check inodes

Is my drive really full? Recently I came across the same problem again, and once again I was looking through my notes to recall my scripts. One of the servers claimed “no space left on device” when trying to do the apt-get update. Surprisingly, the partition was not even nearly full.

Looking around

When I was running df -h I obtained the following results:

$ df -h

Filesystem Size Used Avail Use% Mounted on
/dev/simfs 45G 5.3G 40G 12% /
devtmpfs 1.0G 0 1.0G 0% /dev
tmpfs 1.0G 0 1.0G 0% /dev/shm
tmpfs 1.0G 99M 926M 10% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.0G 0 1.0G 0% /sys/fs/cgroup
none 1.0G 0 1.0G 0% /run/shm

As you can see, my drive is almost not used. The problem is that there is not only the drive space itself which is essential but also inodes – data structures which are keeping the information about files. Let’s check them using df -i – here is what I found:

$ df -i

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/simfs 23592960 23440033 152927 99% /
devtmpfs 262144 58 262086 1% /dev
tmpfs 262144 1 262143 1% /dev/shm
tmpfs 262144 325 261819 1% /run
tmpfs 262144 12 262132 1% /run/lock
tmpfs 262144 10 262134 1% /sys/fs/cgroup
none 262144 1 262143 1% /run/shm

It looks like there is a problem with the root partition – 99% of inodes are taken, and when apt-get is trying to unpack contents of the software archives, the remaining amount is not sufficient. In such case, you should look for the place where there is a bunch of very small or empty files.

Let’s pinpoint them

I want to review directories and count files in them. I can do this in one step by using such set of commands:

for i in /*; do echo $i; find $i | wc -l; done

This way I’m obtaining the list of directories in the root partition and the number of files in each directory. Sometimes it stops in the particular directory (I just don’t want to wait so long) and this is also the sign that there is a significant amount of files in there. Once I know what the top-level directory that causes the issue is, I can adjust my script to dig deeper:

for i in /my-suspect-dir/*; do echo $i; find $i | wc -l; done

I’m repeating the same operation till I find the particular directory which filled with thousands (sometimes even millions) of files. In most cases the only thing left to do is to remove them:

$ rm -rf /my-suspect-dir/lots-of-files-dir/

That’s it. In most cases I came at such a set of empty files in temporary directories or cache directories of old websites, sometimes there is a user who generates them for some reason… One way or another, in such a few steps you should be able to deal with them.