Last night, I recieved a alert SMS, one server goes down... Server on an Aliyun ECS Ubuntu 14.04, when I SSH into the server and when trying to excute some commands for example I get the following:
-bash: cannot create temp file for here-document: No space left on device
The filesystem will usually go into read-only while the system is running if there is a filesystem consistency issue. I checked disk usage:
$ df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda1 20G 8.2G 11G 45% / none 4.0K 0 4.0K 0% /sys/fs/cgroup udev 492M 4.0K 492M 1% /dev tmpfs 101M 408K 100M 1% /run none 5.0M 0 5.0M 0% /run/lock none 501M 0 501M 0% /run/shm none 100M 0 100M 0% /run/user
In this case, I want to check of all filesystem, but not work, it look likes should mount ISO to run repair command.
$ sudo mount -o remount,rw / $ sudo umount /dev/xvda1 $ sudo fsck -y /dev/xvda1 fsck from util-linux 2.20.1 e2fsck 1.42.9 (4-Feb-2014) /dev/xvda1 is mounted. e2fsck: Cannot continue, aborting. 2015-06-23 03:14:03
But finally I check Inodes info if disk:
# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/xvda1 1310720 1310720 0 100% /
Inodes is full, because stores a amount of fragmented files on server. Disk blocks and inodes one full, will lead to not create the file, suggesting that the disk is full, server's file system went into Read-only.