RealActorRob Posted December 22, 2023 Share Posted December 22, 2023 (edited) If anyone is running a storj node, it likes to leave open files even with docker stopped. Use the "Open Files" plugin to see what it's not closing and kill that process after making sure the Docker is stopped. This happens fairly often. In this case, CLI from web works if this plugin doesn't....window popped up blank so : kill 41524 Edited December 22, 2023 by RealActorRob add kill command Link to comment
oliver Posted January 6 Share Posted January 6 (edited) i just got this on 6.12.6. On 12/20/2023 at 12:44 PM, rbroberts said: Just happened to me today. I had removed two of my four cache SSDs (they're Samsung 870 EVOs that are going out for warranty repair/replacement). I did them one at a time, and was stopping the array just to reduce the cache size to 2 disks when it hung for over an hour. I had walked away and came back to find it in that state. Opening a web terminal and unmounting /var/lib/docker did the trick, so this has the same symptoms as the original, but I'm on Unraid 6.12.6. thanks, this did the trick. it's kind of sad how after all this time, unraid still cannot reliably shut an array down. Edited January 6 by oliver Link to comment
Surgikill Posted January 15 Share Posted January 15 On 8/23/2023 at 11:57 AM, ljm42 said: Hmm, let's see what is going on: stop the array to reproduce the problem (don't do a full shutdown because you won't be able to grab diagnostics) show the output of `losetup` so we can see what /dev/loop2 points to on your system. run `umount /dev/loop2` to resolve the issue and let the array stop generate and upload your diagnostics.zip (from Tools -> Diagnostics) so we can see logs of the problem happening feel free to start the array again I'm having this issue on 6.12.3. losetup returns NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 1 1 /boot/bzfirmware 0 512 /dev/loop2 0 0 1 0 /mnt/appdata-disk/system/docker/docker.img 0 512 /dev/loop0 0 0 1 1 /boot/bzmodules 0 512 /dev/loop3 0 0 1 0 /mnt/appdata-disk/system/libvirt/libvirt.img 0 512 I ran 'umount /dev/loop2 twice, on the second time it reported that /dev/loop2: not mounted. The array is still trying to unmount shares Link to comment
ljm42 Posted January 16 Share Posted January 16 9 hours ago, Surgikill said: I ran 'umount /dev/loop2 twice, on the second time it reported that /dev/loop2: not mounted. Glad you got past it. 6.12.6 has additional protections for this issue with the Docker image. 9 hours ago, Surgikill said: The array is still trying to unmount shares This means something is holding the share open. Be sure to close any open web terminals or SSH shells, if they are open to a share that will prevent the array from stopping. Link to comment
ljm42 Posted February 13 Share Posted February 13 So Docker has still been holding the /var/lib/docker mount open for some folks. If you have been affected, please test Unraid 6.12.7-rc2 which does a "lazy unmount" of /var/lib/docker. This means instead of trying to unmount and giving up if it can't, it will wait until it is not in use and then automatically unmount. This should eliminate the need to ever have to manually run `umount /var/lib/docker` again. Link to comment
guruleenyc Posted March 18 Share Posted March 18 Update - if the array can't stop due to "Retry unmounting shares" in 6.12.0 - 6.12.2, the quick fix is to open a web terminal and type:umount /var/lib/docker The array should then stop and prevent an unclean shutdown. (It is possible the array won't stop for other reasons, such as having a web terminal open to a folder on the array. Make sure to exit any web terminals or SSH sessions in this case) We have a fix in the 6.12.3-rc3 prerelease, available here: The fix is in the 6.12.3 release, available here: It would be helpful if some of the folks who have been having an issue stopping the array could upgrade and confirm the issue is resolved in this version. No need, this fix is confirmed. Thanks to everyone who helped track this down!This issue hit me yesterday after two months of uptime while on 6.12.6.Thank you for the quick fix to get the array stopped and restarted without needing to reboot. Sent from my SM-N986U using Tapatalk Link to comment
ljm42 Posted March 18 Share Posted March 18 6 hours ago, guruleenyc said: This issue hit me yesterday after two months of uptime while on 6.12.6. Unraid 6.12.8 has further improvements in this area. Oh I see in your other comment that you updated to 6.12.8, so in theory this particular cause of Docker preventing the array from stopping should not be an issue for you going forward. Link to comment
Zerginator Posted March 31 Share Posted March 31 Same issue happens when using the "clear_array_for_shrink" userscript or removing a disc from the array manually without partity rebuild as to the official guide, as you have to unmount a drive to zero it out. Afterwards the array cannot be stopped, as it gets stuck at trying to unmount the already manually unmounted drive. Link to comment
Espressomatic Posted May 13 Share Posted May 13 (edited) On 3/31/2024 at 4:16 AM, Zerginator said: Same issue happens when using the "clear_array_for_shrink" userscript or removing a disc from the array manually without partity rebuild as to the official guide, as you have to unmount a drive to zero it out. Afterwards the array cannot be stopped, as it gets stuck at trying to unmount the already manually unmounted drive. Same issue today. On top of that I accidentally removed the wrong drive while trying to take out the unassigned one - the Disk Location plugin was flashing the wrong drive and I made the mistake of not shutting down tghe aerray before physically removing it). So now it looks like it's going to be 22+ hours to rebuild the 8TB drive I accidentally removed. Ugh. Edited May 13 by Espressomatic Link to comment
dragondiary Posted May 13 Share Posted May 13 (edited) Same issue today.After using the "clear_array_for_shrink" userscript array not stopped so ; -I manually shutdown the server, -removed the disk which one is cleared by the script -restart server -array not started automatically -in main menu opsions Parity is valid didn't appeared -then went to tools -new config -Preserve current assignments click to all -checked I want to do this -then in main menu Parity is valid appeared -then checked Parity is valid and start array -everything is normal now -I bought 20 tb hdd and replace 3 3tb hdd one by one with this method and everything is correctly working now. -Thanks to spaceinvader Edited May 13 by dragondiary Link to comment
Espressomatic Posted May 13 Share Posted May 13 (edited) All this musical chairs with drives and extended downtime caused by rebuilding has put the final nail in my decision to split apart my Unraid usage. A single server to host my NAS disks (the array) and VM/Docker (all running on NVMe), isn't what I want/need anymore. Later this summer I hope to set up a smaller system 100% solid state for only VM/Docker onm Unraid, while keeping an array on the current 16-bay system, hopefully using a solution more performant and safer than Unraid. It sucks to have docker and VMs down while the array is in maintenace mode doing a rebuild. Edited May 20 by Espressomatic Link to comment
jjross Posted Thursday at 03:59 PM Share Posted Thursday at 03:59 PM (edited) On 5/13/2024 at 9:39 AM, dragondiary said: Same issue today.After using the "clear_array_for_shrink" userscript array not stopped so ; -I manually shutdown the server, -removed the disk which one is cleared by the script -restart server -array not started automatically -in main menu opsions Parity is valid didn't appeared -then went to tools -new config -Preserve current assignments click to all -checked I want to do this -then in main menu Parity is valid appeared -then checked Parity is valid and start array -everything is normal now -I bought 20 tb hdd and replace 3 3tb hdd one by one with this method and everything is correctly working now. -Thanks to spaceinvader Thank you so much for this post. I had this exact same issue on UNRAID 6.12.10 For me, the script wouldn't run, seemingly because the names in /dev/ had changes so I manually ran dd and cleared things out. Then I couldn't stop the array but I followed your steps and everything seems to be working perfectly now. **EDIT** Whoops, I wasn't using "clear_array_for_shrink", I was trying to use "clear_an_array_drive" but had the same symptoms Edited Thursday at 04:01 PM by jjross Link to comment
ljm42 Posted Thursday at 06:03 PM Share Posted Thursday at 06:03 PM Hi folks, This thread is very specifically for people having trouble stopping the array while shutting down, and that issue should be resolved as of 6.12.8 However, there are other causes of "Retry unmounting disk share(s)", for instance, if you leave an SSH terminal open and cd'd to an array drive then the array will not be able to stop. You need to cd out of that directory or simply exit the terminal in order for the array to stop. Anyway, if you are having issues with a particular user script that is going to have a completely different root cause than the issue being discussed here. If you would like help with that issue, please start a new thread with full and complete steps on how to reproduce the issue, along with diagnostics showing the problem. Keep in mind that user scripts are not officially supported, so you should tag the author of the user script as well, since they may know more about what is going on. I'm going to lock this thread as the issue being discussed here appears to be solved. Link to comment
Recommended Posts