[6.12.2] Array stop stuck on "Retry unmounting disk share(s)"

[RealActorRob]

If anyone is running a storj node, it likes to leave open files even with docker stopped.

 

 

Use the "Open Files" plugin to see what it's not closing and kill that process after making sure the Docker is stopped. This happens fairly often.

 

In this case, CLI from web works if this plugin doesn't....window popped up blank so :

 

kill 41524

 

 

Edited December 22, 2023 by RealActorRob
add kill command

[oliver]

i just got this on 6.12.6. 

 

  

On 12/20/2023 at 12:44 PM, rbroberts said:

Just happened to me today. I had removed two of my four cache SSDs (they're Samsung 870 EVOs that are going out for warranty repair/replacement). I did them one at a time, and was stopping the array just to reduce the cache size to 2 disks when it hung for over an hour. I had walked away and came back to find it in that state.

 

Opening a web terminal and unmounting /var/lib/docker did the trick, so this has the same symptoms as the original, but I'm on Unraid 6.12.6.

 

thanks, this did the trick.  

 

it's kind of sad how after all this time, unraid still cannot reliably shut an array down.  

Edited January 6 by oliver

[Surgikill]

On 8/23/2023 at 11:57 AM, ljm42 said:

 

Hmm, let's see what is going on:

• stop the array to reproduce the problem (don't do a full shutdown because you won't be able to grab diagnostics)
• show the output of `losetup` so we can see what /dev/loop2 points to on your system.
• run `umount /dev/loop2` to resolve the issue and let the array stop
• generate and upload your diagnostics.zip (from Tools -> Diagnostics) so we can see logs of the problem happening
• feel free to start the array again

I'm having this issue on 6.12.3.

 

losetup returns

NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                                    DIO LOG-SEC
/dev/loop1         0      0         1  1 /boot/bzfirmware                               0     512
/dev/loop2         0      0         1  0 /mnt/appdata-disk/system/docker/docker.img     0     512
/dev/loop0         0      0         1  1 /boot/bzmodules                                0     512
/dev/loop3         0      0         1  0 /mnt/appdata-disk/system/libvirt/libvirt.img   0     512

 

I ran 'umount /dev/loop2 twice, on the second time it reported that /dev/loop2: not mounted.

 

The array is still trying to unmount shares

 

[ljm42]

9 hours ago, Surgikill said:

I ran 'umount /dev/loop2 twice, on the second time it reported that /dev/loop2: not mounted.

 

Glad you got past it. 6.12.6 has additional protections for this issue with the Docker image.

 

9 hours ago, Surgikill said:

The array is still trying to unmount shares

 

This means something is holding the share open. Be sure to close any open web terminals or SSH shells, if they are open to a share that will prevent the array from stopping.

 

[ljm42]

So Docker has still been holding the /var/lib/docker mount open for some folks. If you have been affected, please test Unraid 6.12.7-rc2 which does a "lazy unmount" of /var/lib/docker. This means instead of trying to unmount and giving up if it can't, it will wait until it is not in use and then automatically unmount. This should eliminate the need to ever have to manually run `umount /var/lib/docker` again.

 

 

[guruleenyc]

Update - if the array can't stop due to "Retry unmounting shares" in 6.12.0 - 6.12.2, the quick fix is to open a web terminal and type:

umount /var/lib/docker

The array should then stop and prevent an unclean shutdown.
 
(It is possible the array won't stop for other reasons, such as having a web terminal open to a folder on the array. Make sure to exit any web terminals or SSH sessions in this case)
 
We have a fix in the 6.12.3-rc3 prerelease, available here:
The fix is in the 6.12.3 release, available here:
It would be helpful if some of the folks who have been having an issue stopping the array could upgrade and confirm the issue is resolved in this version. No need, this fix is confirmed.  Thanks to everyone who helped track this down!

This issue hit me yesterday after two months of uptime while on 6.12.6.

Thank you for the quick fix to get the array stopped and restarted without needing to reboot.

Sent from my SM-N986U using Tapatalk

[ljm42]

6 hours ago, guruleenyc said:

This issue hit me yesterday after two months of uptime while on 6.12.6.

 

Unraid 6.12.8 has further improvements in this area. Oh I see in your other comment that you updated to 6.12.8, so in theory this particular cause of Docker preventing the array from stopping should not be an issue for you going forward.

[Zerginator]

Same issue happens when using the "clear_array_for_shrink" userscript or removing a disc from the array manually without partity rebuild as to the official guide, as you have to unmount a drive to zero it out.

Afterwards the array cannot be stopped, as it gets stuck at trying to unmount the already manually unmounted drive.