March 12, 201610 yr I am working on a family members system and actually had this problem myself caused by the file integrity plugin which is not present. I've removed 90% of all plugins and all but 3 essential dockers and and I can't get this issue to stop. I even have docker disabled for now. By the time it gets full, many of my diagnostic commands error out with an I/O warning file system full message. To catch this while it was happening and before the file system got full, I had to put in a cron job. Running the below command in the cron job fails to stay within the root file system like it should and I see all sorts of results in like /mnt/cache. That made the results largely useless. find / -type d -xdev -exec du -a {} + | sort -n -r Anyone have any ideas on what I can do to isolate this? I manually did du's on all my root folders that I know are used by the memory based root file system and not a disk or flash drive but nothing shows up as taking space here. root@BernadettesSvr:/# ls bin/ boot/ dev/ etc/ home/ init@ lib/ lib64/ mnt/ proc/ root/ run/ sbin/ sys/ tmp/ usr/ var/ root@BernadettesSvr:/# du -sm bin 10 bin root@BernadettesSvr:/# du -sm dev 0 dev root@BernadettesSvr:/# du -sm etc 5 etc root@BernadettesSvr:/# du -sm home 0 home root@BernadettesSvr:/# du -sm lib 15 lib root@BernadettesSvr:/# du -sm lib64 20 lib64 root@BernadettesSvr:/# du -sm root 1 root root@BernadettesSvr:/# du -sm proc du: cannot access ‘proc/7972/task/7972/fd/3’: No such file or directory du: cannot access ‘proc/7972/task/7972/fdinfo/3’: No such file or directory du: cannot access ‘proc/7972/fd/4’: No such file or directory du: cannot access ‘proc/7972/fdinfo/4’: No such file or directory 0 proc root@BernadettesSvr:/# du -sm run 1 run root@BernadettesSvr:/# du -sm sbin 15 sbin root@BernadettesSvr:/# du -sm sys du: cannot access ‘sys/kernel/slab/unraid/md’: No such file or directory 0 sys root@BernadettesSvr:/# du -sm tmp 1 tmp root@BernadettesSvr:/# du -sm usr 190 usr root@BernadettesSvr:/# du -sm var 6 var root@BernadettesSvr:/# df -h / Filesystem Size Used Avail Use% Mounted on - 2.9G 2.8G 164M 95% / I can't see where the space is at, like it's invisible or something. I can't use lsof to see what processes are using the file system since to run it against the root file system produces far too big of a list. These are the only plugins left attached showing there versions. Docker remains disabled for problem isolation. I am working on the system over VPN which is why the openVPN plugin remains. Any bright ideas? The fact I disabled so many components and the problem remains is a little concerning.
March 12, 201610 yr Author son of a gun I think I found it. I assumed everything in mnt was a disk and I was wrong. Looks like an unassigned devices update changed the name of my disks that are mounted outside of the RAID array ever so slightly. For this setup I have a rsync job that does a disk to disk backup sync to disks mounted with unassigned devices. Looks like I am now syncing to the memory file system and the job is creating the disk directory rather then sync to a disk., That'll run the root file system out of space in a hurry.
March 12, 201610 yr Community Expert son of a gun I think I found it. I assumed everything in mnt was a disk and I was wrong. Looks like an unassigned devices update changed the name of my disks that are mounted outside of the RAID array ever so slightly. For this setup I have a rsync job that does a disk to disk backup sync to disks mounted with unassigned devices. Looks like I am now syncing to the memory file system and the job is creating the disk directory rather then sync to a disk., That'll run the root file system out of space in a hurry. I know at some point the Unassigned Devices plugin was changed so that the disks it mounts always appear under /mnt/disks/. Was this perhaps your issue because you assumed a different mount point?
March 12, 201610 yr Author /mnt/disks/ST3000DM001-1CH166_Z1F64H2W <== before /mnt/disks/ST3000DM001_1CH166_Z1F64H2W <=== after, note the underscore is used instead of a hyphen. Interestingly the WebGUI still shows the hyphen. If you click and expand the device in the GUI, it shows you the mount which uses a underscore instead in the device name.
Archived
This topic is now archived and is closed to further replies.