unRAID OS Version: 6.0 beta 12

 

Description: If something keeps cache drive busy, array cannot be stopped from webgui. Keeps trying to unmount disks.

 

How to reproduce: I had a situation where the docker volume failed to unmount when I stopped the docker system. I would think any kind of mounted image on the cache drive would have the same effect.

 

Expected results: Array should stop even if cache drive is busy. In fact, the array can be stopped manually with '/root/mdcmd stop', and webgui should try to make that effort!

 

Actual results: Keeps trying to unmount disks in a loop. Syslog entries say cachedisk is busy.

 

Jan 25 21:57:00 FileServer emhttp: Unmounting disks...
Jan 25 21:57:00 FileServer emhttp: shcmd (11013): umount /mnt/disk1 |& logger
Jan 25 21:57:00 FileServer logger: umount: /mnt/disk1: not found
Jan 25 21:57:00 FileServer emhttp: shcmd (11014): rmdir /mnt/disk1 |& logger
Jan 25 21:57:00 FileServer logger: rmdir: failed to remove '/mnt/disk1': No such file or directory
Jan 25 21:57:00 FileServer emhttp: shcmd (11015): umount /mnt/disk2 |& logger
Jan 25 21:57:00 FileServer logger: umount: /mnt/disk2: not found
Jan 25 21:57:00 FileServer emhttp: shcmd (11016): rmdir /mnt/disk2 |& logger
Jan 25 21:57:00 FileServer logger: rmdir: failed to remove '/mnt/disk2': No such file or directory
Jan 25 21:57:00 FileServer emhttp: shcmd (11017): umount /mnt/disk3 |& logger
Jan 25 21:57:00 FileServer logger: umount: /mnt/disk3: not found
Jan 25 21:57:00 FileServer emhttp: shcmd (11018): rmdir /mnt/disk3 |& logger
Jan 25 21:57:00 FileServer logger: rmdir: failed to remove '/mnt/disk3': No such file or directory
Jan 25 21:57:00 FileServer emhttp: shcmd (11019): umount /mnt/disk4 |& logger
Jan 25 21:57:00 FileServer logger: umount: /mnt/disk4: not found
Jan 25 21:57:00 FileServer emhttp: shcmd (11020): rmdir /mnt/disk4 |& logger
Jan 25 21:57:00 FileServer logger: rmdir: failed to remove '/mnt/disk4': No such file or directory
Jan 25 21:57:00 FileServer emhttp: shcmd (11021): umount /mnt/disk5 |& logger
Jan 25 21:57:00 FileServer logger: umount: /mnt/disk5: not found
Jan 25 21:57:00 FileServer emhttp: shcmd (11022): rmdir /mnt/disk5 |& logger
Jan 25 21:57:00 FileServer logger: rmdir: failed to remove '/mnt/disk5': No such file or directory
Jan 25 21:57:00 FileServer emhttp: shcmd (11023): umount /mnt/disk6 |& logger
Jan 25 21:57:00 FileServer logger: umount: /mnt/disk6: not found
Jan 25 21:57:00 FileServer emhttp: shcmd (11024): rmdir /mnt/disk6 |& logger
Jan 25 21:57:00 FileServer logger: rmdir: failed to remove '/mnt/disk6': No such file or directory
Jan 25 21:57:00 FileServer emhttp: shcmd (11025): umount /mnt/disk7 |& logger
Jan 25 21:57:00 FileServer logger: umount: /mnt/disk7: not found
Jan 25 21:57:00 FileServer emhttp: shcmd (11026): rmdir /mnt/disk7 |& logger
Jan 25 21:57:00 FileServer logger: rmdir: failed to remove '/mnt/disk7': No such file or directory
Jan 25 21:57:00 FileServer emhttp: shcmd (11027): umount /mnt/disk8 |& logger
Jan 25 21:57:00 FileServer logger: umount: /mnt/disk8: not found
Jan 25 21:57:00 FileServer emhttp: shcmd (11028): rmdir /mnt/disk8 |& logger
Jan 25 21:57:00 FileServer logger: rmdir: failed to remove '/mnt/disk8': No such file or directory
Jan 25 21:57:00 FileServer emhttp: shcmd (11029): umount /mnt/disk9 |& logger
Jan 25 21:57:00 FileServer logger: umount: /mnt/disk9: not found
Jan 25 21:57:00 FileServer emhttp: shcmd (11030): rmdir /mnt/disk9 |& logger
Jan 25 21:57:00 FileServer logger: rmdir: failed to remove '/mnt/disk9': No such file or directory
Jan 25 21:57:00 FileServer emhttp: shcmd (11031): umount /mnt/disk10 |& logger
Jan 25 21:57:00 FileServer logger: umount: /mnt/disk10: not found
Jan 25 21:57:00 FileServer emhttp: shcmd (11032): rmdir /mnt/disk10 |& logger
Jan 25 21:57:00 FileServer logger: rmdir: failed to remove '/mnt/disk10': No such file or directory
Jan 25 21:57:00 FileServer emhttp: shcmd (11033): umount /mnt/disk11 |& logger
Jan 25 21:57:00 FileServer logger: umount: /mnt/disk11: not found
Jan 25 21:57:00 FileServer emhttp: shcmd (11034): rmdir /mnt/disk11 |& logger
Jan 25 21:57:00 FileServer logger: rmdir: failed to remove '/mnt/disk11': No such file or directory
Jan 25 21:57:00 FileServer emhttp: shcmd (11035): umount /mnt/disk12 |& logger
Jan 25 21:57:00 FileServer logger: umount: /mnt/disk12: not found
Jan 25 21:57:00 FileServer emhttp: shcmd (11036): rmdir /mnt/disk12 |& logger
Jan 25 21:57:00 FileServer logger: rmdir: failed to remove '/mnt/disk12': No such file or directory
Jan 25 21:57:00 FileServer emhttp: shcmd (11037): umount /mnt/disk13 |& logger
Jan 25 21:57:00 FileServer logger: umount: /mnt/disk13: not found
Jan 25 21:57:00 FileServer emhttp: shcmd (11038): rmdir /mnt/disk13 |& logger
Jan 25 21:57:00 FileServer logger: rmdir: failed to remove '/mnt/disk13': No such file or directory
Jan 25 21:57:00 FileServer emhttp: shcmd (11039): umount /mnt/disk14 |& logger
Jan 25 21:57:00 FileServer logger: umount: /mnt/disk14: not found
Jan 25 21:57:00 FileServer emhttp: shcmd (11040): rmdir /mnt/disk14 |& logger
Jan 25 21:57:00 FileServer logger: rmdir: failed to remove '/mnt/disk14': No such file or directory
Jan 25 21:57:00 FileServer emhttp: shcmd (11041): umount /mnt/disk15 |& logger
Jan 25 21:57:00 FileServer logger: umount: /mnt/disk15: not found
Jan 25 21:57:00 FileServer emhttp: shcmd (11042): rmdir /mnt/disk15 |& logger
Jan 25 21:57:00 FileServer logger: rmdir: failed to remove '/mnt/disk15': No such file or directory
Jan 25 21:57:00 FileServer emhttp: shcmd (11043): umount /mnt/cache |& logger
Jan 25 21:57:00 FileServer logger: umount: /mnt/cache: device is busy.
Jan 25 21:57:00 FileServer logger:         (In some cases useful info about processes that use
Jan 25 21:57:00 FileServer logger:          the device is found by lsof( or fuser(1))
Jan 25 21:57:00 FileServer emhttp: Retry unmounting disk share(s)...

 

Other information:

Do you have powerdown plugin installed? Don't know if it matters or not but I do and don't have any problems shutting down with dockers or VMs running.

 

Do you have powerdown plugin installed? Don't know if it matters or not but I do and don't have any problems shutting down with dockers or VMs running.

 

I don't think it matters, but I do have dlandon's powerdown plugin installed. My problem is some hard crashes have resulted in some kind of docker image or xml file corruption. The image scrubbed fine, but all of my files under /boot/config/plugins/dockerMan disappeared (I had only 3 empty subfolders). The dockers still autostarted and worked fine, so I didn't notice for a while. Whatever the reason was, the result was the docker image not unmounting when unraid tries to stop array. The docker image was still accessible under /var/lib/docker. (I didn't try the powerdown script, but would be surprised if it was able to unmount the docker btrfs image).

 

Now, that may be difficult to reproduce, but try this: Open a telnet session and 'cd /mnt/cache/whatever'. Or mount some other kind of image on the cache drive (a nice and fast BTRFS disk for the plex media library? Use your imagination). Now try to stop the array. Won't do it. Now type /root/mdcmd stop. See how easy it was to do a clean stop of the array in this dire situation?

 

The better design would be if unraid tried to unmount cache drive, but did not wait forever for it to unmount. At least I think that might be the better design. Could have some implications for unassigning and changing the cache drive that would need to be handled - I would understand if unraid refused to unassign the cache drive in this situation.  Overall unraid should prioritize to stop the array clean.

I have not tried the experiment of being in telnet or console with cache or any other drive as my current directory. I know that has been a problem and should probably be fixed.

 

As for mounting something on cache, I have a btrfs cache pool of 2 SSDs, and it contains my docker.img which I understand is mounted by unRAID in /var/lib/docker. I also have a mythbuntu.qcow2 with a running VM on cache. For me, these get stopped when I stop the array.

 

Just trying to provide another data point to this issue.

For me, these get stopped when I stop the array.

 

Yes, my setup worked great when I set it up as well. Right until it didn't. I've started over with a fresh docker brtfs volume and it works fine now. I'm just pointing out a problem with the basic array stopping procedure that can be improved and reduce the impact of this type of problem.

I don't have this problem because when the array is stopped, all containers are requested to stop (nicely) before they will be forcibly stopped.  Might be something else on your system causing this.  Remove the powerdown plugin you installed and see if the issue persists.

 

Separating the starting / stopping of cache from array has been something we've discussed, but haven't committed to implementing just yet.