Serious problem with rootfs full and no space left on device

[Kevin who]

Hi everyone!

First of all, it was so wired when I wanted to download some files via transmission torrent docker. No doubt I still have some TB spaces on my disks, but transmission says I have no space on my device.

So I check my btrfs file system, It said that there were no error, and I found out that my rootfs was full.

Now, I reboot the system and I can't even open my web UI. 

These are my diagnostics files.

mr-black-diagnostics-20211001-1024.zip

[Squid]

Sep 19 03:48:02 Mr-Black shfs: copy_file: /mnt/disk1/domains/Windows 10/vdisk1.img /mnt/cache/domains/Windows 10/vdisk1.img (28) No space left on device
Sep 19 04:03:51 Mr-Black shfs: copy_file: /mnt/disk2/domains/Ubuntu/vdisk1.img /mnt/cache/domains/Ubuntu/vdisk1.img (28) No space left on device
Sep 19 04:03:57 Mr-Black shfs: copy_file: /mnt/disk2/domains/ycxp.vmdk /mnt/cache/domains/ycxp.vmdk (28) No space left on device

 

What *appears* to have happened is that either you or an improperly configured docker app has created /mnt/cache/ and because of that mover is attempting to move stuff into RAM (since you don't have a cache drive)

 

Check all of the host paths you've set on the various containers on the docker tab and make sure that none directly references /mnt/cache/....  

 

You're ultimately going to have to reboot to fix this up.  (But, you may have data loss unless you manually move via command line (mc?) from /mnt/cache to be in /mnt/disk1 (2)

[Kevin who]

4 hours ago, Squid said:

Sep 19 03:48:02 Mr-Black shfs: copy_file: /mnt/disk1/domains/Windows 10/vdisk1.img /mnt/cache/domains/Windows 10/vdisk1.img (28) No space left on device
Sep 19 04:03:51 Mr-Black shfs: copy_file: /mnt/disk2/domains/Ubuntu/vdisk1.img /mnt/cache/domains/Ubuntu/vdisk1.img (28) No space left on device
Sep 19 04:03:57 Mr-Black shfs: copy_file: /mnt/disk2/domains/ycxp.vmdk /mnt/cache/domains/ycxp.vmdk (28) No space left on device

 

What *appears* to have happened is that either you or an improperly configured docker app has created /mnt/cache/ and because of that mover is attempting to move stuff into RAM (since you don't have a cache drive)

 

Check all of the host paths you've set on the various containers on the docker tab and make sure that none directly references /mnt/cache/....  

 

You're ultimately going to have to reboot to fix this up.  (But, you may have data loss unless you manually move via command line (mc?) from /mnt/cache to be in /mnt/disk1 (2)

Thank you! Seems problem solved.

 

Strange thing is, I don't find any container which make the directory or makes by myself.

So, is there a way to monitor the whole system about which process create which directory? I don't find the `mkdir` log in my machine right now😂

I don't know if this will happen again.