UPDATE: in this version change inotifywait becomes a crucial part of the shutdown process. Previously I had some rare issues with that process preventing shutdown from a different plugin and had it killed on a shutdown script, which caused the shutdown process to stop and never recover.

I recently upgraded to 6.12.15 from 6.12.10 to try to fix some zfs issues (migrating from my own zfs management to unRAID's) and now it refuses to stop the array causing unclean shutdown/reboots.

I had run into this message plenty of times before in the past due to other issues, but to my surprise, all the disks and zfs are actually already unmounted. From the timestamps it looks like from stop array request to unmounting the last disk is around 2 minutes. Then unRAID just sits there with this message in the GUI.

Attached diagnostics and some of my own attempts and investigating. How can I figure out why it is unhappy?

EDIT: ps/top/lsof outputs here

$ findmnt
TARGET                        SOURCE                 FSTYPE    OPTIONS
/                             rootfs                 rootfs    rw,size=16391148k,nr_inodes=4097787,inode64
├─/proc                       proc                   proc      rw,relatime
├─/sys                        sysfs                  sysfs     rw,relatime
│ ├─/sys/fs/fuse/connections  fusectl                fusectl   rw,relatime
│ └─/sys/fs/cgroup            cgroup2                cgroup2   rw,nosuid,nodev,noexec,relatime
├─/run                        tmpfs                  tmpfs     rw,nosuid,nodev,noexec,relatime,size=131072k,mode=755,inode64
│ └─/run/docker/netns/default nsfs[net:[4026531840]] nsfs      rw
├─/boot                       /dev/sda1              vfat      rw,noatime,nodiratime,fmask=0177,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,flush,errors=remount-ro
├─/lib                        /dev/loop0             squashfs  ro,relatime,errors=continue
│ └─/lib                      overlay                overlay   rw,relatime,lowerdir=/lib,upperdir=/var/local/overlay/lib,workdir=/var/local/overlay-work/lib
├─/usr                        /dev/loop1             squashfs  ro,relatime,errors=continue
│ └─/usr                      overlay                overlay   rw,relatime,lowerdir=/usr,upperdir=/var/local/overlay/usr,workdir=/var/local/overlay-work/usr
├─/dev                        devtmpfs               devtmpfs  rw,relatime,size=8192k,nr_inodes=4097787,mode=755,inode64
│ ├─/dev/pts                  devpts                 devpts    rw,relatime,gid=5,mode=620,ptmxmode=000
│ └─/dev/shm                  tmpfs                  tmpfs     rw,relatime,inode64
├─/hugetlbfs                  hugetlbfs              hugetlbfs rw,relatime,pagesize=2M
├─/var/log                    tmpfs                  tmpfs     rw,relatime,size=131072k,mode=755,inode64
└─/mnt                        rootfs[/mnt]           rootfs    rw,size=16391148k,nr_inodes=4097787,inode64
  ├─/mnt/disks                tmpfs                  tmpfs     rw,relatime,size=1024k,inode64
  ├─/mnt/remotes              tmpfs                  tmpfs     rw,relatime,size=1024k,inode64
  ├─/mnt/addons               tmpfs                  tmpfs     rw,relatime,size=1024k,inode64
  └─/mnt/rootshare            tmpfs                  tmpfs     rw,relatime,size=1024k,inode64

$ tail /var/log/syslog
Jul 10 13:04:09 t1001 root: cannot unmount '/mnt/cache_zfs': pool or dataset is busy
Jul 10 13:04:25 t1001 emhttpd: shcmd (160): umount /mnt/user
Jul 10 13:04:25 t1001 root: umount: /mnt/user: target is busy.
Jul 10 13:04:25 t1001 emhttpd: shcmd (160): exit status: 32
Jul 10 13:04:25 t1001 emhttpd: shcmd (161): rmdir /mnt/user
Jul 10 13:04:25 t1001 root: rmdir: failed to remove '/mnt/user': Device or resource busy
Jul 10 13:04:25 t1001 emhttpd: shcmd (161): exit status: 1
Jul 10 13:04:25 t1001 emhttpd: shcmd (163): rm -f /boot/config/plugins/dynamix/mover.cron
Jul 10 13:04:25 t1001 emhttpd: shcmd (164): /usr/local/sbin/update_cron
Jul 10 13:04:25 t1001 emhttpd: Retry unmounting user share(s)...
Jul 10 13:04:27 t1001 kernel: script (5153): drop_caches: 3

$ ## the error messages above actually successfully completed at some later time, see dmesg tail below
$ umount /mnt/user
umount: /mnt/user: no mount point specified.
$ rmdir /mnt/user
rmdir: failed to remove '/mnt/user': No such file or directory
$ zpool list
no pools available
$ date
Thu Jul 10 13:25:23 EDT 2025

$ dmesg -T | tail
[Thu Jul 10 13:04:02 2025] veth8f0ab61: renamed from eth0
[Thu Jul 10 13:04:02 2025] br-e769d98838f0: port 1(veth4e25d88) entered disabled state
[Thu Jul 10 13:04:02 2025] device veth4e25d88 left promiscuous mode
[Thu Jul 10 13:04:02 2025] br-e769d98838f0: port 1(veth4e25d88) entered disabled state
[Thu Jul 10 13:04:27 2025] script (5153): drop_caches: 3
[Thu Jul 10 13:04:40 2025] BTRFS info (device dm-0): last unmount of filesystem 3b2ddebb-0f7c-470e-b80b-746480624f45
[Thu Jul 10 13:04:40 2025] BTRFS info (device dm-1): last unmount of filesystem 781ec874-cba1-4e0b-9de6-6724ea81035c
[Thu Jul 10 13:04:41 2025] BTRFS info (device dm-2): last unmount of filesystem ab612068-00dd-4727-9c67-44b97ca3b95a
[Thu Jul 10 13:04:41 2025] BTRFS info (device dm-3): last unmount of filesystem 71712954-1765-4aa6-84dc-1ee9efeb66e8
[Thu Jul 10 13:04:42 2025] XFS (dm-4): Unmounting Filesystem

$ lsof /mnt/user* /mnt/disk* /mnt/cache*
lsof: status error on /mnt/user*: No such file or directory

$ lsof /mnt/disk* /mnt/cache*

$ find /mnt/
/mnt/
/mnt/cache_zfs
/mnt/rootshare
/mnt/addons
/mnt/remotes
/mnt/disks

Thought it might be open luks devices, but after this it is still on that message

$ dmsetup ls --target crypt
md1p1   (254, 0)
md2p1   (254, 1)
md3p1   (254, 2)
md4p1   (254, 3)
nvme0n1p1       (254, 4)
sdb1    (254, 5)
sdc1    (254, 6)
sdd1    (254, 7)

$ for d in /dev/mapper/*; do [[ "$(basename "$d")" != "control" ]] && cryptsetup luksClose "$(basename "$d")"; done

$ dmsetup ls --target crypt
No devices found

t1001-diagnostics-20250710-1320.zip

Edited August 5, 2025Aug 5 by robobub

13 hours ago, robobub said:

/mnt/cache_zfs

This appears to be the problem, since it's the first error:

Jul 10 13:04:09 t1001 root: cannot unmount '/mnt/cache_zfs': pool or dataset is busy

Disable Docker and VM service and reboot in safe mode, start the array and see if it stops, if yes, retest again after booting in normal mode to rule out a plugin issue, leave the services still disabled, if still OK enable one of the services and retest, then the other, note that if the issue happens with the docker service enabled only, you may need to try enabling one or a few containers at a time to find the culprit.

4 hours ago, JorgeB said:

This appears to be the problem, since it's the first error:

Jul 10 13:04:09 t1001 root: cannot unmount '/mnt/cache_zfs': pool or dataset is busy

Disable Docker and VM service and reboot in safe mode, start the array and see if it stops, if yes, retest again after booting in normal mode to rule out a plugin issue, leave the services still disabled, if still OK enable one of the services and retest, th.en the other, note that if the issue happens with the docker service enabled only, you may need to try enabling one or a few containers at a time to find the culprit.

But as with the /mnt/user error, /mnt/cache_zfs is already unmounted, as you can see from the outputs from findmnt ,lsof /mnt/cache*, and find /mnt/. zpool status shows no pools are available anymore.

Docker and VM services were successfully shut down at this point of array stopping, and nothing was showing up in htop/top, but I did try disabling them and starting/stopping the array and the same issue occurs.

I just left for a weekend trip so I'll be holding off on trying safe mode. Though in theory I could try it with Wireguard/Tailscale being part of the core unRAID and you can change the default grub boot option.

I'm sure it's some weird plugin/configuration, but I mean everything except tmpfs is unmounted, and I can halt with no data loss, it's just for some reason unRAID won't shut down and marks it as an unclean shutdown. htop/top/ps doesn't show anything obvious in terms of plugins or processes (is that output in diagnostics?).

2 hours ago, robobub said:

htop/top/ps doesn't show anything obvious in terms of plugins or processes (is that output in diagnostics?).

Not that I can see, hence my recomendation.

23 hours ago, JorgeB said:

Not that I can see, hence my recomendation.

Thanks for your help, I will definitely try that out when I get home. I was hoping to leave the AC off during the day and only boot up the server at night for backups on my trip, but this unclean shutdown definitely makes it difficult.

Here's ps aux/top/pstree/lsof/zpool status outputs.

$ zpool list
no pools available

ps_aux.txt pstree.txt top.txt lsof2.txt