Reboot after update from 6.12.5 to .6, "unclean shutdown detected"

[ericswpark]

Hi,

 

After updating from 6.12.5 to .6, I issued a reboot, and when the server came back up it said it detected an unclean shutdown. I'm guessing something took a while to unmount and UnRAID had to forcefully reboot.

 

I don't remember where the logs for the hard shutdown are stored – is there a way to recover that? I've attached a diagnostics zip taken after the reboot, but I'm not sure if that includes the logs before the shutdown. Any advice on pinpointing this would be appreciated!

dipper-diagnostics-20231202-1120.zip

[itimpi]

18 minutes ago, ericswpark said:

I don't remember where the logs for the hard shutdown are stored – is there a way to recover that?

If there is a saved log it will be in the ‘logs’ folder on the flash drive.

[ericswpark]

@itimpi thanks! Found the diagnostics that UnRAID saved before rebooting (attached below). Note that while the timestamp is newer, that's because the diagnostics in the OP was generated and downloaded from my browser which has an earlier time zone. This is the diagnostics captured before the one in OP.

 

I did skim through the logs but I don't understand what it means:

 

Dec  3 01:15:55 dipper root: rmdir: failed to remove '/mnt/user': Device or resource busy
Dec  3 01:15:55 dipper emhttpd: shcmd (1343364): exit status: 1
Dec  3 01:15:55 dipper emhttpd: shcmd (1343366): rm -f /boot/config/plugins/dynamix/mover.cron
Dec  3 01:15:55 dipper emhttpd: shcmd (1343367): /usr/local/sbin/update_cron
Dec  3 01:15:55 dipper emhttpd: Retry unmounting user share(s)...
Dec  3 01:15:58 dipper root: Status of all loop devices
Dec  3 01:15:58 dipper root: /dev/loop1: [2049]:12 (/boot/previous/bzfirmware)
Dec  3 01:15:58 dipper root: /dev/loop0: [2049]:10 (/boot/previous/bzmodules)
Dec  3 01:15:58 dipper root: Active pids left on /mnt/*
Dec  3 01:15:58 dipper root:                      USER        PID ACCESS COMMAND
Dec  3 01:15:58 dipper root: /mnt/addons:         root     kernel mount /mnt/addons
Dec  3 01:15:58 dipper root: /mnt/cache:          root     kernel mount /mnt/cache
Dec  3 01:15:58 dipper root: /mnt/disk1:          root     kernel mount /mnt/disk1
Dec  3 01:15:58 dipper root: /mnt/disk2:          root     kernel mount /mnt/disk2
Dec  3 01:15:58 dipper root: /mnt/disk3:          root     kernel mount /mnt/disk3
Dec  3 01:15:58 dipper root: /mnt/disks:          root     kernel mount /mnt/disks
Dec  3 01:15:58 dipper root: /mnt/remotes:        root     kernel mount /mnt/remotes
Dec  3 01:15:58 dipper root: /mnt/rootshare:      root     kernel mount /mnt/rootshare
Dec  3 01:15:58 dipper root: /mnt/user:           root     kernel mount /mnt/user
Dec  3 01:15:58 dipper root:                      root       1142 ..c.. rpcd_mdssvc
Dec  3 01:15:58 dipper root:                      root       2998 ..c.. rpcd_mdssvc
Dec  3 01:15:58 dipper root: Active pids left on /dev/md*
Dec  3 01:15:58 dipper root: Generating diagnostics...
Dec  3 01:16:00 dipper emhttpd: shcmd (1343368): /usr/sbin/zfs unmount -a
Dec  3 01:16:00 dipper emhttpd: shcmd (1343369): umount /mnt/user
Dec  3 01:16:00 dipper root: umount: /mnt/user: target is busy.
Dec  3 01:16:00 dipper emhttpd: shcmd (1343369): exit status: 32
Dec  3 01:16:00 dipper emhttpd: shcmd (1343370): rmdir /mnt/user
Dec  3 01:16:00 dipper root: rmdir: failed to remove '/mnt/user': Device or resource busy
Dec  3 01:16:00 dipper emhttpd: shcmd (1343370): exit status: 1
Dec  3 01:16:00 dipper emhttpd: shcmd (1343372): rm -f /boot/config/plugins/dynamix/mover.cron
Dec  3 01:16:00 dipper emhttpd: shcmd (1343373): /usr/local/sbin/update_cron
Dec  3 01:16:00 dipper emhttpd: Retry unmounting user share(s)...

 

I'm guessing PIDs 1142 and 2998 of the process `rpcd_mdssvc` is responsible for not allowing the unmount, but I don't know what that process is. Google says it's related to Samba, but I thought UnRAID was supposed to terminate Samba before trying unmount. Any ideas why this didn't happen here?

dipper-diagnostics-20231203-0115.zip

Edited December 2, 2023 by ericswpark
typo