DivideBy0 Posted September 4, 2023 Share Posted September 4, 2023 I have a pretty simple system. Upgrade went OK but when back it started a parity check? That's not cool and I am simply annoyed I have a second UnRaid that I am debating if I should upgrade given this experience Quote Link to comment
itimpi Posted September 4, 2023 Share Posted September 4, 2023 7 minutes ago, johnwhicker said: Upgrade went OK but when back it started a parity check? That's not cool and I am simply annoyed That implies the array shutdown before the reboot did not complete successfully. If you continue to get this when you reboot then you probably need to look into troubleshooting as described here. Quote Link to comment
DivideBy0 Posted September 4, 2023 Author Share Posted September 4, 2023 13 minutes ago, itimpi said: That implies the array shutdown before the reboot did not complete successfully. If you continue to get this when you reboot then you probably need to look into troubleshooting as described here. Well is doing parity check and can't touch it. Looking at my IPMI log file it was a forced shutdown? Say is collecting a diagnostic file Where can I find the diagnostic file so I can upload it here? Quote Link to comment
Niklas Posted September 4, 2023 Share Posted September 4, 2023 1 hour ago, johnwhicker said: Well is doing parity check and can't touch it. Looking at my IPMI log file it was a forced shutdown? Say is collecting a diagnostic file Where can I find the diagnostic file so I can upload it here? /flash/logs Quote Link to comment
DivideBy0 Posted September 4, 2023 Author Share Posted September 4, 2023 (edited) Again is absolutely SILLY that a system can't actually unmount the darn mounts and just reboot normally 🙂 I had nothing going on the background and this is a very basic, primitive system. Sep 4 08:54:23 NAS-UNRAID-2 emhttpd: Unmounting disks... Sep 4 08:54:23 NAS-UNRAID-2 emhttpd: shcmd (299477): umount /mnt/cache Sep 4 08:54:23 NAS-UNRAID-2 root: umount: /mnt/cache: target is busy. Sep 4 08:54:23 NAS-UNRAID-2 emhttpd: shcmd (299477): exit status: 32 Sep 4 08:54:23 NAS-UNRAID-2 emhttpd: Retry unmounting disk share(s)... Sep 4 08:54:28 NAS-UNRAID-2 emhttpd: Unmounting disks... Sep 4 08:54:28 NAS-UNRAID-2 emhttpd: shcmd (299478): umount /mnt/cache Sep 4 08:54:28 NAS-UNRAID-2 root: umount: /mnt/cache: target is busy. Sep 4 08:54:28 NAS-UNRAID-2 emhttpd: shcmd (299478): exit status: 32 Sep 4 08:54:28 NAS-UNRAID-2 emhttpd: Retry unmounting disk share(s)... Sep 4 08:54:29 NAS-UNRAID-2 root: Status of all loop devices Sep 4 08:54:29 NAS-UNRAID-2 root: /dev/loop1: [2049]:12 (/boot/previous/bzfirmware) Sep 4 08:54:29 NAS-UNRAID-2 root: /dev/loop2: [0043]:260 (/mnt/cache/system/docker/docker.img) Sep 4 08:54:29 NAS-UNRAID-2 root: /dev/loop0: [2049]:10 (/boot/previous/bzmodules) Sep 4 08:54:29 NAS-UNRAID-2 root: Active pids left on /mnt/* Sep 4 08:54:29 NAS-UNRAID-2 root: USER PID ACCESS COMMAND Sep 4 08:54:29 NAS-UNRAID-2 root: /mnt/addons: root kernel mount /mnt/addons Sep 4 08:54:29 NAS-UNRAID-2 root: /mnt/cache: root kernel mount /mnt/cache Sep 4 08:54:29 NAS-UNRAID-2 root: /mnt/disks: root kernel mount /mnt/disks Sep 4 08:54:29 NAS-UNRAID-2 root: /mnt/remotes: root kernel mount /mnt/remotes Sep 4 08:54:29 NAS-UNRAID-2 root: /mnt/rootshare: root kernel mount /mnt/rootshare Sep 4 08:54:29 NAS-UNRAID-2 root: Active pids left on /dev/md* Sep 4 08:54:29 NAS-UNRAID-2 root: Generating diagnostics... Edited September 4, 2023 by johnwhicker Quote Link to comment
ljm42 Posted September 4, 2023 Share Posted September 4, 2023 1 hour ago, johnwhicker said: Again is absolutely SILLY that a system can't actually unmount the darn mounts and just reboot normally 🙂 I had nothing going on the background and this is a very basic, primitive system. Sep 4 08:54:23 NAS-UNRAID-2 emhttpd: Unmounting disks... Sep 4 08:54:23 NAS-UNRAID-2 emhttpd: shcmd (299477): umount /mnt/cache Sep 4 08:54:23 NAS-UNRAID-2 root: umount: /mnt/cache: target is busy. Would you please post the full diagnostics? We need to see what happened before this for a clue as to what is keeping the drive busy Quote Link to comment
DivideBy0 Posted September 4, 2023 Author Share Posted September 4, 2023 9 minutes ago, ljm42 said: Would you please post the full diagnostics? We need to see what happened before this for a clue as to what is keeping the drive busy I sure can Sir. Please see attached nas-unraid-2-diagnostics-20230904-0854.zip Quote Link to comment
JorgeB Posted September 4, 2023 Share Posted September 4, 2023 Problem is with docker not stopping, but don't see why: Sep 4 08:53:30 NAS-UNRAID-2 root: stopping dockerd ... Sep 4 08:53:31 NAS-UNRAID-2 emhttpd: shcmd (299442): umount /var/lib/docker Sep 4 08:53:31 NAS-UNRAID-2 root: umount: /var/lib/docker: target is busy. Quote Link to comment
ljm42 Posted September 4, 2023 Share Posted September 4, 2023 FWIW, this was an issue that happened when shutting down 6.12.3, it was not related to the upgrade to 6.12.4. However, if there is a bug here it was probably not solved by 6.12.4. The logs show that we issued the `rc.docker stop` command: Sep 4 08:53:21 NAS-UNRAID-2 emhttpd: shcmd (299441): /etc/rc.d/rc.docker stop Sep 4 08:53:30 NAS-UNRAID-2 root: stopping dockerd ... but when we try to unmount the volume, it fails: Sep 4 08:53:31 NAS-UNRAID-2 emhttpd: shcmd (299442): umount /var/lib/docker Sep 4 08:53:31 NAS-UNRAID-2 root: umount: /var/lib/docker: target is busy. Sep 4 08:53:31 NAS-UNRAID-2 emhttpd: shcmd (299442): exit status: 32 We may need to add more logging to figure out what the issue is. For now, I would suggest that you follow the instructions in this post when shutting down: https://forums.unraid.net/topic/144429-unraid-os-version-6124-available/#comment-1300996 i.e. 1) stop the array. 2) `umount /var/lib/docker` if needed, then 3) shutdown/restart. This will avoid any unclean shutdowns by ensuring the array is stopped before shutting down. Quote Link to comment
DivideBy0 Posted September 4, 2023 Author Share Posted September 4, 2023 Not sure why this was moved to 6.12.3 Unclean Shutdown as I believe will be misleading for the folks here. This is a symptom of my upgrade from 6.12.3 to 6.12.4 and have nothing to do with an intentional shutdown or reboot. All this happened after the upgrade took place and is asking for a reboot and that is exactly what I did. Not sure where the confusion. I do have another UnRaid system due for 6.12.4 which I will not upgrade as I am concerned with the same behavior. 1 Quote Link to comment
ljm42 Posted September 4, 2023 Share Posted September 4, 2023 31 minutes ago, ljm42 said: FWIW, this was an issue that happened when shutting down 6.12.3, it was not related to the upgrade to 6.12.4. The upgrade changes the files on the flash drive, it does not change the running system. So applying an upgrade will have no effect on whether the running system is able to shutdown properly. Quote Link to comment
DivideBy0 Posted September 4, 2023 Author Share Posted September 4, 2023 (edited) 10 minutes ago, ljm42 said: The upgrade changes the files on the flash drive, it does not change the running system. So applying an upgrade will have no effect on whether the running system is able to shutdown properly. OK, so then how do we explain this unwanted behavior? Obviously when an upgrade is done and a reboot is required I would expect some checks and balances in the upgrade / reboot process? Can't just blindly upgrade and then blindly reboot, hoping that everything is gonna turn ok? I am extremely confused here. Do we need to worry every time we do a simple upgrade now? Very few dockers running in this system: Edited September 4, 2023 by johnwhicker Quote Link to comment
ljm42 Posted September 4, 2023 Share Posted September 4, 2023 What I am saying is that the system likely would have had the same unclean shutdown if you had simply restarted it without doing an upgrade to 6.12.4, because the logs show that 6.12.3 issued the `rc.docker stop` command: Sep 4 08:53:21 NAS-UNRAID-2 emhttpd: shcmd (299441): /etc/rc.d/rc.docker stop Sep 4 08:53:30 NAS-UNRAID-2 root: stopping dockerd ... but when it later tried to unmount the volume, it failed: Sep 4 08:53:31 NAS-UNRAID-2 emhttpd: shcmd (299442): umount /var/lib/docker Sep 4 08:53:31 NAS-UNRAID-2 root: umount: /var/lib/docker: target is busy. Sep 4 08:53:31 NAS-UNRAID-2 emhttpd: shcmd (299442): exit status: 32 For now, I would suggest that you follow the instructions in this post when shutting down: https://forums.unraid.net/topic/144429-unraid-os-version-6124-available/#comment-1300996 i.e. 1) stop the array. 2) `umount /var/lib/docker` if needed, then 3) shutdown/restart. This will avoid any unclean shutdowns by ensuring the array is stopped before shutting down. 1 Quote Link to comment
DivideBy0 Posted September 4, 2023 Author Share Posted September 4, 2023 That is an easy fix and I wish I knew. My assumption was that is fixed in 6.12.3 but I guess is still not fixed. If the system is currently running 6.12.0 - 6.12.2, we're going to suggest that you stop the array at this point. If it gets stuck on "Retry unmounting shares", open a web terminal and type: umount /var/lib/docker The array should now stop successfully (This issue was resolved with 6.12.3) Quote Link to comment
jonathanljs Posted September 5, 2023 Share Posted September 5, 2023 (edited) I have this issue with 6.12.4, and a little research has shown it's an acknowledged issue with the "parity check tuning" plugin and it's being worked on right now. Edited September 5, 2023 by jonathanljs Quote Link to comment
DivideBy0 Posted September 5, 2023 Author Share Posted September 5, 2023 I do not have "Parity Check Tuning" installed. Quote Link to comment
itimpi Posted September 5, 2023 Share Posted September 5, 2023 27 minutes ago, johnwhicker said: I do not have "Parity Check Tuning" installed. In which case you will not get a spurious notification from the Parity Check Tuning plugin 😊 Quote Link to comment
faxxe71 Posted September 5, 2023 Share Posted September 5, 2023 Today i did some changes in the Samba settings and therefore i stopped the array. When i start the array i got the unclean shutdown message again. I did no reboot! 6.12.4 & PCT plugin is installed. -faxxe Quote Link to comment
ljm42 Posted November 21, 2023 Share Posted November 21, 2023 Hi folks, Unraid 6.12.5-rc1 makes more changes to Docker so *knocks on wood* it should not be possible for Docker containers to cause an unclean shutdown: Quote Link to comment
ljm42 Posted February 13 Share Posted February 13 So Docker has still been holding the /var/lib/docker mount open for some folks. If you have been affected, please test Unraid 6.12.7-rc2 which does a "lazy unmount" of /var/lib/docker. This means instead of trying to unmount and giving up if it can't, it will wait until it is not in use and then automatically unmount. This should eliminate the need to ever have to manually run `umount /var/lib/docker` again. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.