ljm42 Posted July 31, 2023 Share Posted July 31, 2023 On 7/30/2023 at 11:21 AM, miicar said: So i don't have the docker hang issue (although its randomly come up in the past BEFORE i upgraded to 6.12.2), my "bzfirmware" and "bzmodules" are the ones hanging themselves! and unmount script listed (and my old one) doesn't work for these loops....i could unclean shutdown...but that's dum too! Ahh shucks, i was hoping to update my production server to 6.12.x to solve some weird SMB/NGIX bug that randomly chokes transfer speeds... This sounds like something else is going on, I'd recommend starting a new thread with details of your issue. 1 Link to comment
miicar Posted August 4, 2023 Share Posted August 4, 2023 On 7/31/2023 at 5:46 PM, ljm42 said: This sounds like something else is going on, I'd recommend starting a new thread with details of your issue. as the update to 6.12.3 seemed to have stopped this issue...i would say its a related issue Link to comment
ljm42 Posted August 4, 2023 Share Posted August 4, 2023 1 hour ago, miicar said: as the update to 6.12.3 seemed to have stopped this issue...i would say its a related issue I am glad your issue appears to be resolved! This thread was for a very specific issue, so if your issue comes back please start a new thread with all the details, as that would mean it is not similar enough to have the same solution being discussed in this thread. 1 Link to comment
K1LLA_KING_KING Posted August 15, 2023 Share Posted August 15, 2023 (edited) Array wont stop, the solutions in this thread dont work either. How do i avoid an unclean shutdown? Im on 6.12.3 Edited August 15, 2023 by K1LLA_KING_KING Link to comment
JorgeB Posted August 15, 2023 Share Posted August 15, 2023 1 hour ago, K1LLA_KING_KING said: Array wont stop, the solutions in this thread dont work either. If there was an unclean shutdown Unraid should have saved the current diags in the flash drive /logs folder, post those. Link to comment
falkfyrebeard Posted August 23, 2023 Share Posted August 23, 2023 On 7/31/2023 at 3:46 PM, ljm42 said: This sounds like something else is going on, I'd recommend starting a new thread with details of your issue. I know this seems to be marked as fixed in 6.12.3 but I am still running in to it on both of my servers on 6.12.3. Happened to both of them yesterday multiple times. The solution "umount /dev/loop2" fixes it every time. Link to comment
ljm42 Posted August 23, 2023 Share Posted August 23, 2023 1 hour ago, falkfyrebeard said: I know this seems to be marked as fixed in 6.12.3 but I am still running in to it on both of my servers on 6.12.3. Happened to both of them yesterday multiple times. The solution "umount /dev/loop2" fixes it every time. Hmm, let's see what is going on: stop the array to reproduce the problem (don't do a full shutdown because you won't be able to grab diagnostics) show the output of `losetup` so we can see what /dev/loop2 points to on your system. run `umount /dev/loop2` to resolve the issue and let the array stop generate and upload your diagnostics.zip (from Tools -> Diagnostics) so we can see logs of the problem happening feel free to start the array again Link to comment
kn0wt Posted August 29, 2023 Share Posted August 29, 2023 On 7/6/2023 at 8:01 PM, ljm42 said: UPDATE! please see my comment further down in this thread https://forums.unraid.net/topic/141479-6122-array-stop-stuck-on-retry-unmounting-disk-shares/#comment-1283203 ------------------------------- Original message: ------------------------------- I hit this today when stopping my array. Here is what worked for me, would appreciate if someone hitting this would confirm it works for them too. To get into this state, stop the array. If you are having this issue you will see "retry unmounting shares" in the lower left corner. Note: There are other reasons this message could happen (like if you left an SSH terminal open while cd'd into the array). This discussion assumes none of the usual suspects apply. In a web terminal or SSH type 'losetup'. In my case it showed: root@Tower:/etc/rc.d# losetup NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 1 1 /boot/bzfirmware 0 512 /dev/loop2 0 0 1 0 /mnt/cache/system/docker/docker.img 0 512 /dev/loop0 0 0 1 1 /boot/bzmodules 0 512 The problem is that docker.img is still mounted. Note that in my case is it on /dev/loop2 Then run `/etc/rc.d/rc.docker status` to confirm that docker has stopped: # /etc/rc.d/rc.docker status status of dockerd: stopped (It should be stopped, since you were in the process of stopping the array. But if Docker is still running, you can type `/etc/rc.d/rc.docker stop` and wait a bit, then run status again until it has stopped.) Then to fix the problem, type: umount /dev/loop2 (use whatever /dev/loopX docker.img is on, as noted above) Once that is unmounted, the array will automatically finish stopping. We are looking into a fix for this, but it would help if we could reliably reproduce the problem (it has only happened to me once). If anyone is able to identify what it takes to make this happen I'd appreciate it. sitting here tearing out my hair trying to figure out why i was stuck with the unmounting drives.... low and behold this solved the issue on 6.12.3... Link to comment
kn0wt Posted August 29, 2023 Share Posted August 29, 2023 On 8/23/2023 at 10:57 AM, ljm42 said: Hmm, let's see what is going on: stop the array to reproduce the problem (don't do a full shutdown because you won't be able to grab diagnostics) show the output of `losetup` so we can see what /dev/loop2 points to on your system. run `umount /dev/loop2` to resolve the issue and let the array stop generate and upload your diagnostics.zip (from Tools -> Diagnostics) so we can see logs of the problem happening feel free to start the array again having the same issue as @falkfyrebeard on 6.12.3, I attached my diagnostics and the output of `losetup`.. hopefully it'll help as well? ive had it happen to me the last 2 times ive stopped the array.. 1st time i waitied 20-30 min and then just hard rebooted the server. then just did it again and had the same issue.. root@Excalibur:~# losetup NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 1 1 /boot/bzfirmware 0 512 /dev/loop2 0 0 1 0 /mnt/disk1/system/docker/docker.img 0 512 /dev/loop0 0 0 1 1 /boot/bzmodules 0 512 root@Excalibur:~# /etc/rc.d/rc.docker status status of dockerd: stopped root@Excalibur:~# umount /dev/loop2 root@Excalibur:~# excalibur-diagnostics-20230829-1006.zip Link to comment
ljm42 Posted August 29, 2023 Share Posted August 29, 2023 34 minutes ago, kn0wt said: having the same issue as @falkfyrebeard on 6.12.3, I attached my diagnostics and the output of `losetup`.. hopefully it'll help as well? ive had it happen to me the last 2 times ive stopped the array.. 1st time i waitied 20-30 min and then just hard rebooted the server. then just did it again and had the same issue.. root@Excalibur:~# losetup NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 1 1 /boot/bzfirmware 0 512 /dev/loop2 0 0 1 0 /mnt/disk1/system/docker/docker.img 0 512 /dev/loop0 0 0 1 1 /boot/bzmodules 0 512 root@Excalibur:~# /etc/rc.d/rc.docker status status of dockerd: stopped root@Excalibur:~# umount /dev/loop2 root@Excalibur:~# Your logs show that disk1 is in use when the array tries to stop: Aug 28 09:56:09 Excalibur emhttpd: Retry unmounting disk share(s)... Aug 28 09:56:14 Excalibur emhttpd: Unmounting disks... Aug 28 09:56:14 Excalibur emhttpd: shcmd (161854): umount /mnt/disk1 Aug 28 09:56:14 Excalibur root: umount: /mnt/disk1: target is busy. Aug 28 09:56:14 Excalibur emhttpd: shcmd (161854): exit status: 32 The most common reason for this is having a terminal window open and sitting in the /mnt/disk1/ directory structure. When stopping the array you need to be sure that no bash terminals are cd'd to /mnt Another option is to install the Dynamix Stop Shell plugin, which will automatically kill any bash shells that are open to /mnt when stopping the array. Link to comment
JorgeB Posted August 29, 2023 Share Posted August 29, 2023 Docker image was in use before Unraid tried to mount it: Aug 28 09:54:13 Excalibur emhttpd: shcmd (161580): /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 50 Aug 28 09:54:13 Excalibur root: '/mnt/user/system/docker/docker.img' is in-use, cannot mount Reboot, start array, stop array, and if it doesn't stop post new diags. Link to comment
kn0wt Posted August 29, 2023 Share Posted August 29, 2023 5 minutes ago, ljm42 said: Your logs show that disk1 is in use when the array tries to stop: Aug 28 09:56:09 Excalibur emhttpd: Retry unmounting disk share(s)... Aug 28 09:56:14 Excalibur emhttpd: Unmounting disks... Aug 28 09:56:14 Excalibur emhttpd: shcmd (161854): umount /mnt/disk1 Aug 28 09:56:14 Excalibur root: umount: /mnt/disk1: target is busy. Aug 28 09:56:14 Excalibur emhttpd: shcmd (161854): exit status: 32 The most common reason for this is having a terminal window open and sitting in the /mnt/disk1/ directory structure. When stopping the array you need to be sure that no bash terminals are cd'd to /mnt Another option is to install the Dynamix Stop Shell plugin, which will automatically kill any bash shells that are open to /mnt when stopping the array. i had a terminal window open, but it was on the default location ~: i'll try out the plugin and see if that helps.. Link to comment
Timski Posted September 2, 2023 Share Posted September 2, 2023 On 8/30/2023 at 1:49 AM, ljm42 said: The most common reason for this is having a terminal window open ... Brother, you are a genius .. many thanks .. I was having this same issue (docker stopped and fully unmounted). Any attempt to umount the offending mount point resulted in "invalid parameters" error response. Finally I realised I had a MobaXTerm window open and SSH'd into Unraid. Killed that and array stopped beautifully. Thanks for all your inputs to this and other threads .. what an amazing community 🙏 1 Link to comment
CaptainMorganCrunch Posted September 16, 2023 Share Posted September 16, 2023 This is still a problem in 6.12.4. I had replaced a failed disk on a 2 parity disk array and it failed in the first 20 minutes or so and just before I went to shut it down to swap it out, another disk reported failed. I shut it down and replaced both disks with larger ones. When I restarted unraid, the array was started and both disks just showing as "missing". I went to stop the array and had the problem with docker preventing it from unmounting. I ran the first command, umount /dev/loop2, but it was umount /var/lib/docker that stopped the array. I was then able to select the two replacements and start the array. Link to comment
Cyborg Posted September 16, 2023 Share Posted September 16, 2023 10 hours ago, CaptainMorganCrunch said: This is still a problem in 6.12.4. I had replaced a failed disk on a 2 parity disk array and it failed in the first 20 minutes or so and just before I went to shut it down to swap it out, another disk reported failed. I shut it down and replaced both disks with larger ones. When I restarted unraid, the array was started and both disks just showing as "missing". I went to stop the array and had the problem with docker preventing it from unmounting. I ran the first command, umount /dev/loop2, but it was umount /var/lib/docker that stopped the array. I was then able to select the two replacements and start the array. I have similar problems. I swap red the disk slots and rebuild it. Now it once again says it can unmount I think it is related to zfs formatted disks that can’t unmount and edger force reboot it gets in some way destroyed. Link to comment
JorgeB Posted September 17, 2023 Share Posted September 17, 2023 13 hours ago, Cyborg said: Now it once again says it can unmount Please post the diagnostics. Link to comment
takkischitt Posted September 27, 2023 Share Posted September 27, 2023 On 7/12/2023 at 10:08 PM, ljm42 said: Update - if the array can't stop due to "Retry unmounting shares" in 6.12.0 - 6.12.2, the quick fix is to open a web terminal and type: umount /var/lib/docker The array should then stop and prevent an unclean shutdown. (It is possible the array won't stop for other reasons, such as having a web terminal open to a folder on the array. Make sure to exit any web terminals or SSH sessions in this case) We have a fix in the 6.12.3-rc3 prerelease, available here: The fix is in the 6.12.3 release, available here: It would be helpful if some of the folks who have been having an issue stopping the array could upgrade and confirm the issue is resolved in this version. No need, this fix is confirmed. Thanks to everyone who helped track this down! I've got this problem (running 6.12.4) and the terminal command doesn't work. I get the response "bash: unmount: command not found". 1 Link to comment
JorgeB Posted September 27, 2023 Share Posted September 27, 2023 49 minutes ago, takkischitt said: "bash: unmount It's umount 1 Link to comment
takkischitt Posted September 28, 2023 Share Posted September 28, 2023 On 9/27/2023 at 5:34 PM, JorgeB said: It's umount Doh! I'm an idiot! Link to comment
Waves00 Posted October 1, 2023 Share Posted October 1, 2023 (edited) I'm pretty sure I have this same issue on 6.12.4. I do not have any ssh sessions open when it happens. Disabling docker in settings reliably fixes the issue. Is there some kind of fix for 6.12.4 other than having to manually unmount docker.img? Having to manually unmount docker is inconvenient. Disk1 always refuses to unmount. tower-diagnostics-20230930-1519.zip Edited October 1, 2023 by Waves00 Link to comment
bombz Posted October 9, 2023 Share Posted October 9, 2023 On 7/13/2023 at 12:42 AM, jflad17 said: I can try doing this now. I had this issue on 6.11.5. I tried all suggestions and nothing seemed to work. I think the issue might be related to libvirt. I saw a log in that section of settings I think, where it kept throwing libvirt errors. All I was trying to do was get smb working properly for Synology after noticing files not having certain permissions. I'll reply back if this did indeed fix my issue. Hello, THANK YOU for this post! umount /var/lib/docker Worked perfectly to unmount disk shares on 6.12.2 Was rebooting to update to 6.12.4 and this unmount looping issue occurred. Many thanks! Link to comment
RealActorRob Posted October 13, 2023 Share Posted October 13, 2023 (edited) Edit: Finally unounted and was able to stop the array. Restarted to 'fix' acpi_pad multiple processes consuming all of the CPU resources. Waiting on other disk share unmounts but it did unmount the docker image finally. The acpi_pad sucking up all CPUs is perhaps why THAT is taking so long. Still mounted: NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 1 1 /boot/bzfirmware 0 512 /dev/loop0 0 0 1 1 /boot/bzmodules 0 512 Edited October 13, 2023 by RealActorRob Link to comment
RealActorRob Posted November 17, 2023 Share Posted November 17, 2023 Still having this issue. One Clue? Web GUI said it was still unmounting but it shut down clean fine via GUI when I pressed that button in the GUI after about 10 mins. Shut down so I could add an NVME. root@BlackTower:~# losetup NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 1 1 /boot/bzfirmware 0 512 /dev/loop0 0 0 1 1 /boot/bzmodules 0 512 root@BlackTower:~# umount /var/lib/docker umount: /var/lib/docker: not mounted. root@BlackTower:~# Link to comment
andyd Posted November 19, 2023 Share Posted November 19, 2023 (edited) Running into the same issue RealActorRob. Not sure if it’s safe to shutdown? In my case, it was because I had a window open to the web terminal Edited November 19, 2023 by andyd Link to comment
rbroberts Posted December 20, 2023 Share Posted December 20, 2023 Just happened to me today. I had removed two of my four cache SSDs (they're Samsung 870 EVOs that are going out for warranty repair/replacement). I did them one at a time, and was stopping the array just to reduce the cache size to 2 disks when it hung for over an hour. I had walked away and came back to find it in that state. Opening a web terminal and unmounting /var/lib/docker did the trick, so this has the same symptoms as the original, but I'm on Unraid 6.12.6. Link to comment
Recommended Posts