September 23, 20241 yr Normally when this happens it's simply because I left an SSH session open somewhere in /mnt/. This time, that doesn't seem to be the case. Output of: lsof | grep /mnt shfs 32561 root 20r REG 0,41 725135360 66485 /mnt/cache/isos/template/iso/virtio-win-0.1.262.iso shfs 32561 1756 shfs root 20r REG 0,41 725135360 66485 /mnt/cache/isos/template/iso/virtio-win-0.1.262.iso shfs 32561 1762 shfs root 20r REG 0,41 725135360 66485 /mnt/cache/isos/template/iso/virtio-win-0.1.262.iso shfs 32561 11339 shfs root 20r REG 0,41 725135360 66485 /mnt/cache/isos/template/iso/virtio-win-0.1.262.iso shfs 32561 31176 shfs root 20r REG 0,41 725135360 66485 /mnt/cache/isos/template/iso/virtio-win-0.1.262.iso shfs 32561 32562 shfs root 20r REG 0,41 725135360 66485 /mnt/cache/isos/template/iso/virtio-win-0.1.262.iso shfs 32561 32564 shfs root 20r REG 0,41 725135360 66485 /mnt/cache/isos/template/iso/virtio-win-0.1.262.iso shfs 32561 32589 shfs root 20r REG 0,41 725135360 66485 /mnt/cache/isos/template/iso/virtio-win-0.1.262.iso
September 23, 20241 yr Author Yes, though all of them were paused and it completed shutting down services, so I wouldn't expect them to be holding things open now. Edited September 23, 20241 yr by brainbone_
September 23, 20241 yr Author Still stuck. Not exactly sure what I should do, but need to get this unraid server back up in a hurry.
September 23, 20241 yr Try umount -f -l /mnt/cache That's a lowercase L for lazy in the command switches.
September 23, 20241 yr Author After doing that, shfs / /mnt/cache/isos/template/iso/virtio-win-0.1.262.iso lines no longer show in lsof | grep /mnt, but still stuck. Tail of syslog: Sep 23 10:33:55 backup01 emhttpd: shcmd (5230): /usr/sbin/zfs unmount -a Sep 23 10:33:55 backup01 emhttpd: shcmd (5231): umount /mnt/user Sep 23 10:33:55 backup01 root: umount: /mnt/user: target is busy. Sep 23 10:33:55 backup01 emhttpd: shcmd (5231): exit status: 32 Sep 23 10:33:55 backup01 emhttpd: shcmd (5232): rmdir /mnt/user Sep 23 10:33:55 backup01 root: rmdir: failed to remove '/mnt/user': Device or resource busy Sep 23 10:33:55 backup01 emhttpd: shcmd (5232): exit status: 1 Sep 23 10:33:55 backup01 emhttpd: shcmd (5234): rm -f /boot/config/plugins/dynamix/mover.cron Sep 23 10:33:55 backup01 emhttpd: shcmd (5235): /usr/local/sbin/update_cron Sep 23 10:33:55 backup01 emhttpd: Retry unmounting user share(s)... Edited September 23, 20241 yr by brainbone_
September 23, 20241 yr Author 9 minutes ago, JonathanM said: What does mount show? rootfs on / type rootfs (rw,size=99017528k,nr_inodes=24754382,inode64) proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs (rw,relatime) tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=131072k,mode=755,inode64) /dev/sda1 on /boot type vfat (rw,noatime,nodiratime,fmask=0177,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,flush,errors=remount-ro) /boot/bzmodules on /lib type squashfs (ro,relatime,errors=continue) overlay on /lib type overlay (rw,relatime,lowerdir=/lib,upperdir=/var/local/overlay/lib,workdir=/var/local/overlay-work/lib) /boot/bzfirmware on /usr type squashfs (ro,relatime,errors=continue) overlay on /usr type overlay (rw,relatime,lowerdir=/usr,upperdir=/var/local/overlay/usr,workdir=/var/local/overlay-work/usr) devtmpfs on /dev type devtmpfs (rw,relatime,size=8192k,nr_inodes=24754382,mode=755,inode64) devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /dev/shm type tmpfs (rw,relatime,inode64) fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime) hugetlbfs on /hugetlbfs type hugetlbfs (rw,relatime,pagesize=2M) cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) tmpfs on /var/log type tmpfs (rw,relatime,size=131072k,mode=755,inode64) rootfs on /mnt type rootfs (rw,size=99017528k,nr_inodes=24754382,inode64) tmpfs on /mnt/disks type tmpfs (rw,relatime,size=1024k,inode64) tmpfs on /mnt/remotes type tmpfs (rw,relatime,size=1024k,inode64) tmpfs on /mnt/addons type tmpfs (rw,relatime,size=1024k,inode64) tmpfs on /mnt/rootshare type tmpfs (rw,relatime,size=1024k,inode64) nfsd on /proc/fs/nfs type nfsd (rw,relatime) nfsd on /proc/fs/nfsd type nfsd (rw,relatime) /dev/md1p1 on /mnt/disk1 type xfs (rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/md2p1 on /mnt/disk2 type xfs (rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/md3p1 on /mnt/disk3 type xfs (rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/md4p1 on /mnt/disk4 type xfs (rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/md5p1 on /mnt/disk5 type xfs (rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/md6p1 on /mnt/disk6 type xfs (rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota) shfs on /mnt/user type fuse.shfs (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)
September 23, 20241 yr You can try using the umount command on all the /mnt/diskX, check if they are gone with the mount command, finally umount /mnt/user. This is obviously not the "correct" way of doing things, but it's the least harm I can think of, since if all the disks are unmounted parity should stay valid.
September 23, 20241 yr Author 8 minutes ago, JonathanM said: You can try using the umount command on all the /mnt/diskX, check if they are gone with the mount command, finally umount /mnt/user. This is obviously not the "correct" way of doing things, but it's the least harm I can think of, since if all the disks are unmounted parity should stay valid. Looks like maybe I should have force unmounted /mnt/user first. Syslog now showing: ep 23 12:00:55 FTbackup1 emhttpd: shcmd (11455): umount /mnt/disk5 Sep 23 12:00:55 FTbackup1 root: umount: /mnt/disk5: not mounted. Sep 23 12:00:55 FTbackup1 emhttpd: shcmd (11455): exit status: 32 Sep 23 12:00:55 FTbackup1 emhttpd: shcmd (11456): umount /mnt/disk6 Sep 23 12:00:55 FTbackup1 root: umount: /mnt/disk6: not mounted. Sep 23 12:00:55 FTbackup1 emhttpd: shcmd (11456): exit status: 32 Sep 23 12:00:55 FTbackup1 emhttpd: shcmd (11457): umount /mnt/cache Sep 23 12:00:55 FTbackup1 root: umount: /mnt/cache: not mounted. Sep 23 12:00:55 FTbackup1 emhttpd: shcmd (11457): exit status: 32 Sep 23 12:00:55 FTbackup1 emhttpd: Retry unmounting disk share(s)... Edit: Seems kinda like a failure of the stop array process to refuse to continue when it can't unmount an already unmounted disk share. Edited September 23, 20241 yr by brainbone_
September 23, 20241 yr Author Forced a restart, system came back up and is running a parity check. Sure would like to know what happened to cause this. Seems like maybe the virtio iso was stuck in use on NFS by a remote VM (proxmox) and maybe because I have support Hard Links off in order to keep the NFS mounts from dropping on proxmox every mover runs? Still, would suck if unraid can't handle this kind of thing.
September 23, 20241 yr Solution 2 minutes ago, brainbone_ said: Seems like maybe the virtio iso was stuck in use on NFS by a remote VM That would probably do it. I thought there is logic in place to eventually force the shutdown, but maybe that only applies to UPS power failure shutdowns.
September 23, 20241 yr Author 8 minutes ago, JonathanM said: That would probably do it. I thought there is logic in place to eventually force the shutdown, but maybe that only applies to UPS power failure shutdowns. Yeah, I'll shut cache off for isos since I really don't need it there anyway and see how it goes. I would have expected the NFS server shutting down would have made it a non issues, but maybe something with the mover and files in use? Who knows. Anyway, thanks for your help!
September 23, 20241 yr Do you have QEMU guest agent running for all your VMs? Are you able to completely shut down every VM from the VM menus? Do you have USB manager installed? A VM not shutting down or restarting after it shut down, preventing the system from going down (most of the time) is what's caused this for me. The last time was one I couldn't figure out because I'd addressed the guest agent issue which caused me not to be able to shut down some VMs in the past. SimonF figured it was a USB Manager option and sure enough, he was right. I had an option set that rested the VM when a specific USB device was inserted. And that was getting triggered every time I tried to shut that system down, leaving the whole thing in a loop and from my perspective just stuck.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.