6.12.3 Unclean shutdown

[DivideBy0]

I have a pretty simple system. Upgrade went OK but when back it started a parity check? That's not cool and I am simply annoyed I have a second UnRaid that I am debating if I should upgrade given this experience

[itimpi]

7 minutes ago, johnwhicker said:

Upgrade went OK but when back it started a parity check? That's not cool and I am simply annoyed

That implies the array shutdown before the reboot did not complete successfully.    If you continue to get this when you reboot then you probably need to look into troubleshooting as described here.

[DivideBy0]

13 minutes ago, itimpi said:

That implies the array shutdown before the reboot did not complete successfully.    If you continue to get this when you reboot then you probably need to look into troubleshooting as described here.

 

Well is doing parity check and can't touch it.  Looking at my IPMI log file it was a forced shutdown? Say is collecting a diagnostic file Where can I find the diagnostic file so I can upload it here?

[Niklas]

1 hour ago, johnwhicker said:

 

Well is doing parity check and can't touch it.  Looking at my IPMI log file it was a forced shutdown? Say is collecting a diagnostic file Where can I find the diagnostic file so I can upload it here?

 

/flash/logs

[DivideBy0]

Again is absolutely SILLY that a system can't actually unmount the darn mounts and just reboot normally 🙂   I had nothing going on the background and this is a very basic, primitive system. 

 

Sep  4 08:54:23 NAS-UNRAID-2 emhttpd: Unmounting disks...
Sep  4 08:54:23 NAS-UNRAID-2 emhttpd: shcmd (299477): umount /mnt/cache
Sep  4 08:54:23 NAS-UNRAID-2 root: umount: /mnt/cache: target is busy.
Sep  4 08:54:23 NAS-UNRAID-2 emhttpd: shcmd (299477): exit status: 32
Sep  4 08:54:23 NAS-UNRAID-2 emhttpd: Retry unmounting disk share(s)...
Sep  4 08:54:28 NAS-UNRAID-2 emhttpd: Unmounting disks...
Sep  4 08:54:28 NAS-UNRAID-2 emhttpd: shcmd (299478): umount /mnt/cache
Sep  4 08:54:28 NAS-UNRAID-2 root: umount: /mnt/cache: target is busy.
Sep  4 08:54:28 NAS-UNRAID-2 emhttpd: shcmd (299478): exit status: 32
Sep  4 08:54:28 NAS-UNRAID-2 emhttpd: Retry unmounting disk share(s)...
Sep  4 08:54:29 NAS-UNRAID-2 root: Status of all loop devices
Sep  4 08:54:29 NAS-UNRAID-2 root: /dev/loop1: [2049]:12 (/boot/previous/bzfirmware)
Sep  4 08:54:29 NAS-UNRAID-2 root: /dev/loop2: [0043]:260 (/mnt/cache/system/docker/docker.img)
Sep  4 08:54:29 NAS-UNRAID-2 root: /dev/loop0: [2049]:10 (/boot/previous/bzmodules)
Sep  4 08:54:29 NAS-UNRAID-2 root: Active pids left on /mnt/*
Sep  4 08:54:29 NAS-UNRAID-2 root:                      USER        PID ACCESS COMMAND
Sep  4 08:54:29 NAS-UNRAID-2 root: /mnt/addons:         root     kernel mount /mnt/addons
Sep  4 08:54:29 NAS-UNRAID-2 root: /mnt/cache:          root     kernel mount /mnt/cache
Sep  4 08:54:29 NAS-UNRAID-2 root: /mnt/disks:          root     kernel mount /mnt/disks
Sep  4 08:54:29 NAS-UNRAID-2 root: /mnt/remotes:        root     kernel mount /mnt/remotes
Sep  4 08:54:29 NAS-UNRAID-2 root: /mnt/rootshare:      root     kernel mount /mnt/rootshare
Sep  4 08:54:29 NAS-UNRAID-2 root: Active pids left on /dev/md*
Sep  4 08:54:29 NAS-UNRAID-2 root: Generating diagnostics...

 

 

Edited September 4, 2023 by johnwhicker

[ljm42]

1 hour ago, johnwhicker said:

Again is absolutely SILLY that a system can't actually unmount the darn mounts and just reboot normally 🙂   I had nothing going on the background and this is a very basic, primitive system. 

 

Sep  4 08:54:23 NAS-UNRAID-2 emhttpd: Unmounting disks...
Sep  4 08:54:23 NAS-UNRAID-2 emhttpd: shcmd (299477): umount /mnt/cache
Sep  4 08:54:23 NAS-UNRAID-2 root: umount: /mnt/cache: target is busy.

 

Would you please post the full diagnostics? We need to see what happened before this for a clue as to what is keeping the drive busy

[DivideBy0]

9 minutes ago, ljm42 said:

 

Would you please post the full diagnostics? We need to see what happened before this for a clue as to what is keeping the drive busy

 I sure can Sir.  Please see attached

 

 

nas-unraid-2-diagnostics-20230904-0854.zip

[JorgeB]

Problem is with docker not stopping, but don't see why:

 

Sep  4 08:53:30 NAS-UNRAID-2 root: stopping dockerd ...
Sep  4 08:53:31 NAS-UNRAID-2 emhttpd: shcmd (299442): umount /var/lib/docker
Sep  4 08:53:31 NAS-UNRAID-2 root: umount: /var/lib/docker: target is busy.

 

[ljm42]

FWIW, this was an issue that happened when shutting down 6.12.3, it was not related to the upgrade to 6.12.4.  However, if there is a bug here it was probably not solved by 6.12.4.

 

The logs show that we issued the `rc.docker stop` command:

Sep  4 08:53:21 NAS-UNRAID-2 emhttpd: shcmd (299441): /etc/rc.d/rc.docker stop
Sep  4 08:53:30 NAS-UNRAID-2 root: stopping dockerd ...

but when we try to unmount the volume, it fails:

Sep  4 08:53:31 NAS-UNRAID-2 emhttpd: shcmd (299442): umount /var/lib/docker
Sep  4 08:53:31 NAS-UNRAID-2 root: umount: /var/lib/docker: target is busy.
Sep  4 08:53:31 NAS-UNRAID-2 emhttpd: shcmd (299442): exit status: 32

We may need to add more logging to figure out what the issue is.

 

For now, I would suggest that you follow the instructions in this post when shutting down:
  https://forums.unraid.net/topic/144429-unraid-os-version-6124-available/#comment-1300996

 

i.e. 1) stop the array. 2) `umount /var/lib/docker` if needed, then 3) shutdown/restart. This will avoid any unclean shutdowns by ensuring the array is stopped before shutting down.

 

[DivideBy0]

Not sure why this was moved to 6.12.3 Unclean Shutdown as I believe will be misleading for the folks here.

 

This is a symptom of my upgrade from 6.12.3 to 6.12.4 and have nothing to do with an intentional shutdown or reboot.  All this happened after the upgrade took place and is asking for a reboot and that is exactly what I did.  Not sure where the confusion.

 

I do have another UnRaid system due for 6.12.4  which I will not upgrade as I am concerned with the same behavior. 

[ljm42]

31 minutes ago, ljm42 said:

FWIW, this was an issue that happened when shutting down 6.12.3, it was not related to the upgrade to 6.12.4.

 

The upgrade changes the files on the flash drive, it does not change the running system. So applying an upgrade will have no effect on whether the running system is able to shutdown properly.

[DivideBy0]

10 minutes ago, ljm42 said:

 

The upgrade changes the files on the flash drive, it does not change the running system. So applying an upgrade will have no effect on whether the running system is able to shutdown properly.

 

OK, so then how do we explain this unwanted behavior? Obviously when an upgrade is done and a reboot is required I would expect some checks and balances in the upgrade / reboot process? Can't just blindly upgrade and then blindly reboot, hoping that everything is gonna turn ok?  I am extremely confused here. Do we need to worry every time we do a simple upgrade now?

 

Very few dockers running in this system:

Edited September 4, 2023 by johnwhicker

[ljm42]

What I am saying is that the system likely would have had the same unclean shutdown if you had simply restarted it without doing an upgrade to 6.12.4, because the logs show that 6.12.3 issued the `rc.docker stop` command:

Sep  4 08:53:21 NAS-UNRAID-2 emhttpd: shcmd (299441): /etc/rc.d/rc.docker stop
Sep  4 08:53:30 NAS-UNRAID-2 root: stopping dockerd ...

but when it later tried to unmount the volume, it failed:

Sep  4 08:53:31 NAS-UNRAID-2 emhttpd: shcmd (299442): umount /var/lib/docker
Sep  4 08:53:31 NAS-UNRAID-2 root: umount: /var/lib/docker: target is busy. 
Sep  4 08:53:31 NAS-UNRAID-2 emhttpd: shcmd (299442): exit status: 32

 

For now, I would suggest that you follow the instructions in this post when shutting down:
https://forums.unraid.net/topic/144429-unraid-os-version-6124-available/#comment-1300996

 

i.e. 1) stop the array. 2) `umount /var/lib/docker` if needed, then 3) shutdown/restart. This will avoid any unclean shutdowns by ensuring the array is stopped before shutting down.

 

[DivideBy0]

That is an easy fix and I wish I knew. My assumption was that is fixed in 6.12.3 but I guess is still not fixed. 

 

If the system is currently running 6.12.0 - 6.12.2, we're going to suggest that you stop the array at this point. If it gets stuck on "Retry unmounting shares", open a web terminal and type:

 

umount /var/lib/docker

 

The array should now stop successfully (This issue was resolved with 6.12.3)

 

 

[jonathanljs]

I have this issue with 6.12.4, and a little research has shown it's an acknowledged issue with the "parity check tuning" plugin and it's being worked on right now.

 

Edited September 5, 2023 by jonathanljs

[DivideBy0]

I do not have "Parity Check Tuning" installed.

[itimpi]

27 minutes ago, johnwhicker said:

I do not have "Parity Check Tuning" installed.

In which case you will not get a spurious notification from the Parity Check Tuning plugin 😊

[faxxe71]

Today i did some changes in the Samba settings and therefore i stopped the array.

When i start the array i got the unclean shutdown message again. I did no reboot!

6.12.4 & PCT plugin is installed.

-faxxe

 

[ljm42]

Hi folks, Unraid 6.12.5-rc1 makes more changes to Docker so *knocks on wood* it should not be possible for Docker containers to cause an unclean shutdown:

 

[ljm42]

So Docker has still been holding the /var/lib/docker mount open for some folks. If you have been affected, please test Unraid 6.12.7-rc2 which does a "lazy unmount" of /var/lib/docker. This means instead of trying to unmount and giving up if it can't, it will wait until it is not in use and then automatically unmount. This should eliminate the need to ever have to manually run `umount /var/lib/docker` again.