[6.12.2] Array stop stuck on "Retry unmounting disk share(s)"

[ljm42]

On 7/30/2023 at 11:21 AM, miicar said:

So i don't have the docker hang issue (although its randomly come up in the past BEFORE i upgraded to 6.12.2), my "bzfirmware" and "bzmodules" are the ones hanging themselves!  and unmount script listed (and my old one) doesn't work for these loops....i could unclean shutdown...but that's dum too!  

Ahh shucks, i was hoping to update my production server to 6.12.x to solve some weird SMB/NGIX bug that randomly chokes transfer speeds...

 

This sounds like something else is going on, I'd recommend starting a new thread with details of your issue. 

[miicar]

On 7/31/2023 at 5:46 PM, ljm42 said:

 

This sounds like something else is going on, I'd recommend starting a new thread with details of your issue. 

as the update to 6.12.3 seemed to have stopped this issue...i would say its a related issue

[ljm42]

1 hour ago, miicar said:

as the update to 6.12.3 seemed to have stopped this issue...i would say its a related issue

 

I am glad your issue appears to be resolved! This thread was for a very specific issue, so if your issue comes back please start a new thread with all the details, as that would mean it is not similar enough to have the same solution being discussed in this thread.

[K1LLA_KING_KING]

Array wont stop, the solutions in this thread dont work either. How do i avoid an unclean shutdown?
Im on 6.12.3

Edited August 15, 2023 by K1LLA_KING_KING

[JorgeB]

1 hour ago, K1LLA_KING_KING said:

Array wont stop, the solutions in this thread dont work either.

If there was an unclean shutdown Unraid should have saved the current diags in the flash drive /logs folder, post those.

[falkfyrebeard]

On 7/31/2023 at 3:46 PM, ljm42 said:

 

This sounds like something else is going on, I'd recommend starting a new thread with details of your issue. 

I know this seems to be marked as fixed in 6.12.3 but I am still running in to it on both of my servers on 6.12.3. Happened to both of them yesterday multiple times. The solution "umount /dev/loop2" fixes it every time.

[ljm42]

1 hour ago, falkfyrebeard said:

I know this seems to be marked as fixed in 6.12.3 but I am still running in to it on both of my servers on 6.12.3. Happened to both of them yesterday multiple times. The solution "umount /dev/loop2" fixes it every time.

 

Hmm, let's see what is going on:

• stop the array to reproduce the problem (don't do a full shutdown because you won't be able to grab diagnostics)
• show the output of `losetup` so we can see what /dev/loop2 points to on your system.
• run `umount /dev/loop2` to resolve the issue and let the array stop
• generate and upload your diagnostics.zip (from Tools -> Diagnostics) so we can see logs of the problem happening
• feel free to start the array again

[kn0wt]

On 7/6/2023 at 8:01 PM, ljm42 said:

UPDATE! please see my comment further down in this thread https://forums.unraid.net/topic/141479-6122-array-stop-stuck-on-retry-unmounting-disk-shares/#comment-1283203

 

 

-------------------------------

Original message:

-------------------------------

 

I hit this today when stopping my array.

 

Here is what worked for me, would appreciate if someone hitting this would confirm it works for them too.

 

To get into this state, stop the array. If you are having this issue you will see "retry unmounting shares" in the lower left corner.  Note: There are other reasons this message could happen (like if you left an SSH terminal open while cd'd into the array). This discussion assumes none of the usual suspects apply.

 

In a web terminal or SSH type 'losetup'. In my case it showed:

root@Tower:/etc/rc.d# losetup
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                           DIO LOG-SEC
/dev/loop1         0      0         1  1 /boot/bzfirmware                      0     512
/dev/loop2         0      0         1  0 /mnt/cache/system/docker/docker.img   0     512
/dev/loop0         0      0         1  1 /boot/bzmodules                       0     512

 

The problem is that docker.img is still mounted. Note that in my case is it on /dev/loop2

 

Then run `/etc/rc.d/rc.docker status` to confirm that docker has stopped:

# /etc/rc.d/rc.docker status
status of dockerd: stopped

(It should be stopped, since you were in the process of stopping the array. But if Docker is still running, you can type `/etc/rc.d/rc.docker stop` and wait a bit, then run status again until it has stopped.)

 

Then to fix the problem, type:

umount /dev/loop2

(use whatever /dev/loopX docker.img is on, as noted above) 

 

Once that is unmounted, the array will automatically finish stopping.

 

We are looking into a fix for this, but it would help if we could reliably reproduce the problem (it has only happened to me once). If anyone is able to identify what it takes to make this happen I'd appreciate it.

sitting here tearing out my hair trying to figure out why i was stuck with the unmounting drives.... low and behold this solved the issue on 6.12.3...

[kn0wt]

On 8/23/2023 at 10:57 AM, ljm42 said:

 

Hmm, let's see what is going on:

• stop the array to reproduce the problem (don't do a full shutdown because you won't be able to grab diagnostics)
• show the output of `losetup` so we can see what /dev/loop2 points to on your system.
• run `umount /dev/loop2` to resolve the issue and let the array stop
• generate and upload your diagnostics.zip (from Tools -> Diagnostics) so we can see logs of the problem happening
• feel free to start the array again

having the same issue as @falkfyrebeard on 6.12.3, I attached my diagnostics and the output of `losetup`.. hopefully it'll help as well? ive had it happen to me the last 2 times ive stopped the array.. 1st time i waitied 20-30 min and then just hard rebooted the server. then just did it again and had the same issue..

root@Excalibur:~# losetup
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                           DIO LOG-SEC
/dev/loop1         0      0         1  1 /boot/bzfirmware                      0     512
/dev/loop2         0      0         1  0 /mnt/disk1/system/docker/docker.img   0     512
/dev/loop0         0      0         1  1 /boot/bzmodules                       0     512
root@Excalibur:~# /etc/rc.d/rc.docker status
status of dockerd: stopped
root@Excalibur:~# umount /dev/loop2
root@Excalibur:~# 

excalibur-diagnostics-20230829-1006.zip

[ljm42]

34 minutes ago, kn0wt said:

having the same issue as @falkfyrebeard on 6.12.3, I attached my diagnostics and the output of `losetup`.. hopefully it'll help as well? ive had it happen to me the last 2 times ive stopped the array.. 1st time i waitied 20-30 min and then just hard rebooted the server. then just did it again and had the same issue..

root@Excalibur:~# losetup
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                           DIO LOG-SEC
/dev/loop1         0      0         1  1 /boot/bzfirmware                      0     512
/dev/loop2         0      0         1  0 /mnt/disk1/system/docker/docker.img   0     512
/dev/loop0         0      0         1  1 /boot/bzmodules                       0     512
root@Excalibur:~# /etc/rc.d/rc.docker status
status of dockerd: stopped
root@Excalibur:~# umount /dev/loop2
root@Excalibur:~# 

 

Your logs show that disk1 is in use when the array tries to stop:

Aug 28 09:56:09 Excalibur emhttpd: Retry unmounting disk share(s)...
Aug 28 09:56:14 Excalibur emhttpd: Unmounting disks...
Aug 28 09:56:14 Excalibur emhttpd: shcmd (161854): umount /mnt/disk1
Aug 28 09:56:14 Excalibur root: umount: /mnt/disk1: target is busy.
Aug 28 09:56:14 Excalibur emhttpd: shcmd (161854): exit status: 32

The most common reason for this is having a terminal window open and sitting in the /mnt/disk1/ directory structure.  When stopping the array you need to be sure that no bash terminals are cd'd to /mnt

 

Another option is to install the Dynamix Stop Shell plugin, which will automatically kill any bash shells that are open to /mnt when stopping the array.

 

 

 

[JorgeB]

Docker image was in use before Unraid tried to mount it:

Aug 28 09:54:13 Excalibur emhttpd: shcmd (161580): /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 50
Aug 28 09:54:13 Excalibur root: '/mnt/user/system/docker/docker.img' is in-use, cannot mount

 

Reboot, start array, stop array, and if it doesn't stop post new diags.

[kn0wt]

5 minutes ago, ljm42 said:

 

Your logs show that disk1 is in use when the array tries to stop:

Aug 28 09:56:09 Excalibur emhttpd: Retry unmounting disk share(s)...
Aug 28 09:56:14 Excalibur emhttpd: Unmounting disks...
Aug 28 09:56:14 Excalibur emhttpd: shcmd (161854): umount /mnt/disk1
Aug 28 09:56:14 Excalibur root: umount: /mnt/disk1: target is busy.
Aug 28 09:56:14 Excalibur emhttpd: shcmd (161854): exit status: 32

The most common reason for this is having a terminal window open and sitting in the /mnt/disk1/ directory structure.  When stopping the array you need to be sure that no bash terminals are cd'd to /mnt

 

Another option is to install the Dynamix Stop Shell plugin, which will automatically kill any bash shells that are open to /mnt when stopping the array.

 

 

 

i had a terminal window open, but it was on the default location ~: i'll try out the plugin and see if that helps..

 

[Timski]

On 8/30/2023 at 1:49 AM, ljm42 said:

The most common reason for this is having a terminal window open ...

 

Brother, you are a genius .. many thanks .. I was having this same issue (docker stopped and fully unmounted). Any attempt to umount the offending mount point resulted in "invalid parameters" error response. Finally I realised I had a MobaXTerm window open and SSH'd into Unraid. Killed that and array stopped beautifully.

 

Thanks for all your inputs to this and other threads .. what an amazing community 🙏

[CaptainMorganCrunch]

This is still a problem in 6.12.4.

 

I had replaced a failed disk on a 2 parity disk array and it failed in the first 20 minutes or so and just before I went to shut it down to swap it out, another disk reported failed. I shut it down and replaced both disks with larger ones. When I restarted unraid, the array was started and both disks just showing as "missing".

I went to stop the array and had the problem with docker preventing it from unmounting. I ran the first command, umount /dev/loop2, but it was umount /var/lib/docker that stopped the array.

 

I was then able to select the two replacements and start the array.

[Cyborg]

10 hours ago, CaptainMorganCrunch said:

This is still a problem in 6.12.4.

 

I had replaced a failed disk on a 2 parity disk array and it failed in the first 20 minutes or so and just before I went to shut it down to swap it out, another disk reported failed. I shut it down and replaced both disks with larger ones. When I restarted unraid, the array was started and both disks just showing as "missing".

I went to stop the array and had the problem with docker preventing it from unmounting. I ran the first command, umount /dev/loop2, but it was umount /var/lib/docker that stopped the array.

 

I was then able to select the two replacements and start the array.

I have similar problems. I swap red the disk slots and rebuild it. Now it once again says it can unmount 

 

I think it is related to zfs formatted disks that can’t unmount and edger force reboot it gets in some way destroyed. 

[JorgeB]

13 hours ago, Cyborg said:

Now it once again says it can unmount 

Please post the diagnostics.

[takkischitt]

On 7/12/2023 at 10:08 PM, ljm42 said:

Update - if the array can't stop due to "Retry unmounting shares" in 6.12.0 - 6.12.2, the quick fix is to open a web terminal and type:

umount /var/lib/docker

The array should then stop and prevent an unclean shutdown.

 

(It is possible the array won't stop for other reasons, such as having a web terminal open to a folder on the array. Make sure to exit any web terminals or SSH sessions in this case)

 

We have a fix in the 6.12.3-rc3 prerelease, available here:

The fix is in the 6.12.3 release, available here:

It would be helpful if some of the folks who have been having an issue stopping the array could upgrade and confirm the issue is resolved in this version. No need, this fix is confirmed.  Thanks to everyone who helped track this down!

 

I've got this problem (running 6.12.4) and the terminal command doesn't work. I get the response "bash: unmount: command not found".

[JorgeB]

49 minutes ago, takkischitt said:

"bash: unmount

It's umount

[takkischitt]

On 9/27/2023 at 5:34 PM, JorgeB said:

It's umount

Doh! I'm an idiot!

[Waves00]

I'm pretty sure I have this same issue on 6.12.4. I do not have any ssh sessions open when it happens. Disabling docker in settings reliably fixes the issue. Is there some kind of fix for 6.12.4 other than having to manually unmount docker.img? Having to manually unmount docker is inconvenient. Disk1 always refuses to unmount.

tower-diagnostics-20230930-1519.zip

Edited October 1, 2023 by Waves00

[bombz]

On 7/13/2023 at 12:42 AM, jflad17 said:

I can try doing this now. I had this issue on 6.11.5. I tried all suggestions and nothing seemed to work. I think the issue might be related to libvirt. I saw a log in that section of settings I think, where it kept throwing libvirt errors. All I was trying to do was get smb working properly for Synology after noticing files not having certain permissions. I'll reply back if this did indeed fix my issue. 

 

Hello,

 

THANK YOU for this post!
 

umount /var/lib/docker

 

Worked perfectly to unmount disk shares on 6.12.2

Was rebooting to update to 6.12.4 and this unmount looping issue occurred. 
Many thanks! 

[RealActorRob]

Edit: Finally unounted and was able to stop the array. Restarted to 'fix' acpi_pad multiple processes consuming all of the CPU resources.

 

Waiting on other disk share unmounts but it did unmount the docker image finally. 

 

The acpi_pad sucking up all CPUs is perhaps why THAT is taking so long.

 

Still mounted: 

NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE        DIO LOG-SEC

/dev/loop1         0      0         1  1 /boot/bzfirmware   0     512

/dev/loop0         0      0         1  1 /boot/bzmodules    0     512

 

Edited October 13, 2023 by RealActorRob

[RealActorRob]

Still having this issue. 

 

One Clue?  Web GUI said it was still unmounting but it shut down clean fine via GUI when I pressed that button in the GUI after about 10 mins. 

 

Shut down so I could add an NVME.

 

root@BlackTower:~# losetup
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE        DIO LOG-SEC
/dev/loop1         0      0         1  1 /boot/bzfirmware   0     512
/dev/loop0         0      0         1  1 /boot/bzmodules    0     512
root@BlackTower:~# umount /var/lib/docker
umount: /var/lib/docker: not mounted.
root@BlackTower:~# 

[andyd]

Running into the same issue RealActorRob. Not sure if it’s safe to shutdown?

 

In my case, it was because I had a window open to the web terminal

Edited November 19, 2023 by andyd

[rbroberts]

Just happened to me today. I had removed two of my four cache SSDs (they're Samsung 870 EVOs that are going out for warranty repair/replacement). I did them one at a time, and was stopping the array just to reduce the cache size to 2 disks when it hung for over an hour. I had walked away and came back to find it in that state.

 

Opening a web terminal and unmounting /var/lib/docker did the trick, so this has the same symptoms as the original, but I'm on Unraid 6.12.6.