My Unraid server locked up today. The web UI stopped responding, and I couldn't SSH in. Ping still worked though.

 

When I tried to shut it down via IPMI, it got stuck at "Starting diagnostics collection":

 

Diagnostics after rebooting:

danserver-diagnostics-20230901-1104.zip

 

I've enabled "Mirror syslog to flash" to collect syslogs in case it happens again.

 

I'm using an Asus Pro WS W680M-ACE SE motherboard, Intel Core i5-13500, and 64 GB of ECC memory.

 

Any ideas?

Edited September 1, 20232 yr by Daniel15

Post the persistent log if it happens again.

Did you ever find a reason for this? I just shutdown to upgrade from 6.12.11 to 6.12.13 (not because of a hang) and mine has stuck at the "Starting Diagnostics collection". It's been there for about 15 minutes now and it hasn't ever done this before.

8 hours ago, Shonky said:

It's been there for about 15 minutes now

You will need to force a reboot.

My server was in the same state (hanging at the "Starting diagnostics collection" after reboot command) and letting it sit there for around 30 minutes resulted in a proper restart and a diagnostic zip file as well as the syslog files in the /boot/log directory.

 

I think the process of sanitizing the diags leads to long gathering times when mover logging is enabled and a lot of moving actions were performed. (I did resize the /var/log because the syslogs filled up the initial 128MB with mover log entries).

17 hours ago, Plak said:

I think the process of sanitizing the diags leads to long gathering times when mover logging is enabled and a lot of moving actions were performed.

That can take a long time, but note that mover logging should typically only be enabled for troubleshooting purposes.

why is it collecting a diagnostic on stop? Often, my server refuse to run a diagnostic and hang because of that....

10 hours ago, Nodiaque said:

why is it collecting a diagnostic on stop?

If you mean array stop, it shouldn't; it will on server reboot/shutdown if it exceeds the set timeout.

Well I called a shutdown so normally it stopped the array. I looked at my log and everytime I call a shutdown, a diagnostic is done

That suggests the array is not stopping cleanly. Disable Docker and VM service and reboot in safe mode, start the array and see if it stops, if yes, retest again after booting in normal mode to rule out a plugin issue, leave the services still disabled, if still OK enable one of the services and retest, then the other, note that if the issue happens with the docker service enabled only, you may need to try enabling one or a few containers at a time to find the culprit.

I do have some docker that doesn't shutdown properly from the UI, AKA I receive an error message when I try to restart or stop them (but they do shutdown). Weirdly, using docker stop <container> shut them properly in a secondes, it's only through the GUI this happen.

But also, diagnostic itself not working isn't new, I had that many times in the past.

So right now, I can easily run diagnostic and restart. But I know that randomly, sooner or later, this is not true anymore. Weird thing is yesterday, I restarted 3 times before it decided to hang on the diagnostic. All of them where shutdown from the gui

Like mentioned, I would recommend first trying to find out what is preventing Unraid from cleanly shutting down after some time; that is likely what is also causing the diags issues.

So I just did a manual reboot from GUI. Right before, I ran a diagnostic just to be sure that diagnostic would finish. Well guess what, it's stuck.

I can SSH in it no problem.

syslog show it did a diagnostic, failed and tried to start nginx back (Why?) and then it's stuck

syslog.txt

edit: I tried calling a shutdown -r now, I lost my ssh session and cannot open a new one. 10 minutes have passed, I can still ping but that's it. I had to force shutdown. It's not normal. All docker where shutdown and vm also (I've checked with ssh and syslog shows it). I can't realy on something that cannot be shutdown when UPS is running, it's absurd

servraid-diagnostics-20250924-2145.zip

Edited September 25, 2025Sep 25 by Nodiaque

You'll need to force a reboot.

1 hour ago, JorgeB said:

You'll need to force a reboot.

Whilst that seems to be the only solution which you repeat each time, it's not ideal. The problem is that if you reboot it, the next time it shuts down fine. It really should have some form of timeout on these processes rather than hanging there forever. There's not really anything logged to the screen to determined exactly what it's waiting on or anything. Perhaps it's because the logs are large but it shouldn't take 15 minutes or more.

Edit: I can't even find where the "Starting diagnostics collection..." string even comes from. Seems a bit more than just a simple shutdown script but even grepping through the binaries I'm not seeing it.

Edited September 25, 2025Sep 25 by Shonky

4 hours ago, JorgeB said:

You'll need to force a reboot.

That's what I did. The question is why do I need it. This time if you look at log, it doesn't even to have stuck at the unneeded diagnostic. That server was restarted like 4 days before without any problem. I was restarting this time because of nvidia driver update. And that happen. This make the solution very unreliable in case of emergency shutdown because of UPS (happened more then once that the server died because UPS battery was dead waiting for the server to close more then 30 minutes). Something is needed to prevent this issue. I never saw an issue on shutdown on any unix system. I've uploaded the logs I could give, if something else is needed I'll give it.

edit: unsure if you check the syslog. But everything stopped promptly. All service stopped, all docker, vm, network connection, all disk where unmounted. But then

Sep 24 21:14:31 #### root: Capture diagnostics to /boot/logs
Sep 24 21:14:31 #### publish: curl to diagnostics failed
Sep 24 21:14:31 #### publish: curl to diagnostics failed
Sep 24 21:14:31 #### publish: curl to diagnostics failed (repeating infinitly)
Sep 24 21:14:32 #### kernel: XFS (nvme2n1p1): Unmounting Filesystem 0dc5511c-9b2f-46cc-bfa6-de412b4769e0
Sep 24 21:14:31 #### publish: curl to diagnostics failed (repeating infinitly)

Why a diagnostic, everything shutdown properly.

And then, it try to kill nginx and crash

Sep 24 21:14:36 #### rc.nginx: Stopping Nginx server daemon forcibly...
Sep 24 21:14:36 #### rc.nginx: Nginx server daemon...  Already stopped.
Sep 24 21:14:39 #### kernel: BUG: unable to handle page fault for address: ffffffffa7466458
Sep 24 21:14:39 #### kernel: #PF: supervisor read access in kernel mode
Sep 24 21:14:39 #### kernel: #PF: error_code(0x0000) - not-present page
Sep 24 21:14:39 #### kernel: PGD 541a067 P4D 541a067 PUD 541b063 PMD 10a506067 PTE 0
Sep 24 21:14:39 #### kernel: Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI
Sep 24 21:14:39 #### kernel: CPU: 0 UID: 0 PID: 1445014 Comm: btrfs Tainted: P           O       6.12.24-Unraid #1
Sep 24 21:14:39 #### kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
Sep 24 21:14:39 #### kernel: Hardware name: Dell Inc. Precision 5820 Tower/06JWJY, BIOS 2.11.0 09/30/2021
Sep 24 21:14:39 #### kernel: RIP: 0010:bdev_open+0x8e/0x300
Sep 24 21:14:39 #### kernel: Code: 00 48 89 c7 48 89 44 24 08 e8 ae d3 59 00 49 8b 46 40 48 83 b8 80 04 00 00 00 75 0a b8 fa ff ff ff e9 35 02 00 00 49 8b 46 48 <48> 8b 78 78 e8 09 bd b7 ff 84 c0 41 89 c7 74 e2 80 3d 23 35 fb 00
Sep 24 21:14:39 #### kernel: RSP: 0018:ffffc90001f23c38 EFLAGS: 00010282
Sep 24 21:14:39 #### kernel: RAX: ffffffffa74663e0 RBX: ffff888107518c80 RCX: 0000000000000000
Sep 24 21:14:39 #### kernel: RDX: ffff88844828dd00 RSI: 0000000000000009 RDI: ffff88810a6929f8
Sep 24 21:14:39 #### kernel: RBP: 0000000000000009 R08: ffff88826b30ca80 R09: 0000000000000009
Sep 24 21:14:39 #### kernel: R10: 0000000000000f01 R11: fefefefefefefe01 R12: ffff88826b30ca80
Sep 24 21:14:39 #### kernel: R13: 0000000000000000 R14: ffff88810a692800 R15: 0000000000000000
Sep 24 21:14:39 #### kernel: FS:  0000152b0e43f900(0000) GS:ffff88a01fa00000(0000) knlGS:0000000000000000
Sep 24 21:14:39 #### kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 24 21:14:39 #### kernel: CR2: ffffffffa7466458 CR3: 0000000139e32002 CR4: 00000000003706f0
Sep 24 21:14:39 #### kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Sep 24 21:14:39 #### kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Sep 24 21:14:39 #### kernel: Call Trace:
Sep 24 21:14:39 #### kernel: <TASK>
Sep 24 21:14:39 #### kernel: ? iput+0x64/0x1a0
Sep 24 21:14:39 #### kernel: ? __pfx_blkdev_open+0x10/0x10
Sep 24 21:14:39 #### kernel: blkdev_open+0x96/0xc0
Sep 24 21:14:39 #### kernel: do_dentry_open+0x1ae/0x330
Sep 24 21:14:39 #### kernel: vfs_open+0x27/0x50
Sep 24 21:14:39 #### kernel: path_openat+0x8e8/0x9f0
Sep 24 21:14:39 #### kernel: ? terminate_walk+0x1d/0x70
Sep 24 21:14:39 #### kernel: do_filp_open+0xa4/0x110
Sep 24 21:14:39 #### kernel: ? __pfx_kfree_link+0x10/0x10
Sep 24 21:14:39 #### kernel: ? kmem_cache_debug_flags+0xc/0x20
Sep 24 21:14:39 #### kernel: ? rawspin_unlock+0x14/0x30
Sep 24 21:14:39 #### kernel: do_sys_openat2+0x6d/0xc0
Sep 24 21:14:39 #### kernel: do_sys_open+0x3a/0x60
Sep 24 21:14:39 #### kernel: do_syscall_64+0x68/0xe0
Sep 24 21:14:39 #### kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Sep 24 21:14:39 #### kernel: RIP: 0033:0x152b0e4d95c7
Sep 24 21:14:39 #### kernel: Code: 46 b8 04 00 00 00 48 8b 15 1e 78 16 00 64 89 02 48 c7 c2 ff ff ff ff 5b 48 89 d0 c3 0f 1f 80 00 00 00 00 48 8b 44 24 10 0f 05 <48> 63 d0 3d 00 f0 ff ff 77 0f 48 89 d0 5b c3 66 2e 0f 1f 84 00 00
Sep 24 21:14:39 #### kernel: RSP: 002b:00007ffefba98510 EFLAGS: 00000202 ORIG_RAX: 0000000000000101
Sep 24 21:14:39 #### kernel: RAX: ffffffffffffffda RBX: 0000152b0e43f900 RCX: 0000152b0e4d95c7
Sep 24 21:14:39 #### kernel: RDX: 0000000000080800 RSI: 0000000000521ec0 RDI: ffffffffffffff9c
Sep 24 21:14:39 #### kernel: RBP: 0000000000520430 R08: 0000000000000000 R09: 0000000000000000
Sep 24 21:14:39 #### kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000068d4977f
Sep 24 21:14:39 #### kernel: R13: 0000000068d4976c R14: 0000152b0e82aa9c R15: 0000000000000013
Sep 24 21:14:39 #### kernel: </TASK>
Sep 24 21:14:39 #### kernel: Modules linked in: nft_compat tls cmac cifs cifs_arc4 nls_ucs2_utils cifs_md4 dns_resolver af_packet bluetooth rfkill xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle iptable_mangle vhost_iotlb nft_m
asq nft_chain_nat nvidia_uvm(PO) veth nf_conntrack_netlink xt_nat xt_conntrack bridge xfrm_user xt_addrtype xt_MASQUERADE xt_tcpudp xt_mark tun nf_tables nfnetlink ip6table_nat iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv
4 ntfs3 tcp_diag inet_diag zfs(PO) spl(O) ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs macvtap macvlan tap 8021q garp mrp stp llc ixgbe xfrm_algo mdio e1000e nvidia_drm(PO) nvidia_modeset(PO) intel_rapl_common io
sf_mbi nvidia(PO) x86_pkg_temp_thermal intel_powerclamp coretemp drm_ttm_helper ttm video drm_kms_helper drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel crypto_simd cryp
td rapl i2c_i801 backlight intel_cstate input_leds intel_wmi_thunderbolt wmi_bmof dell_smm_hwmon
Sep 24 21:14:39 #### kernel: intel_uncore led_class tpm_crb mpt3sas nvme agpgart i2c_smbus mei_me raid_class nvme_core scsi_transport_sas i2c_core ahci mei vmd tpm_tis libahci tpm_tis_core wmi tpm libaescfb ecdh_generic ecc acpi_tad
button [last unloaded: md_mod]
Sep 24 21:14:39 #### kernel: CR2: ffffffffa7466458
Sep 24 21:14:39 #### kernel: ---[ end trace 0000000000000000 ]---
Sep 24 21:14:39 #### kernel: pstore: backend (erst) writing error (-28)
Sep 24 21:14:39 #### kernel: RIP: 0010:bdev_open+0x8e/0x300
Sep 24 21:14:39 #### kernel: Code: 00 48 89 c7 48 89 44 24 08 e8 ae d3 59 00 49 8b 46 40 48 83 b8 80 04 00 00 00 75 0a b8 fa ff ff ff e9 35 02 00 00 49 8b 46 48 <48> 8b 78 78 e8 09 bd b7 ff 84 c0 41 89 c7 74 e2 80 3d 23 35 fb 00
Sep 24 21:14:39 #### kernel: RSP: 0018:ffffc90001f23c38 EFLAGS: 00010282
Sep 24 21:14:39 #### kernel: RAX: ffffffffa74663e0 RBX: ffff888107518c80 RCX: 0000000000000000
Sep 24 21:14:39 #### kernel: RDX: ffff88844828dd00 RSI: 0000000000000009 RDI: ffff88810a6929f8
Sep 24 21:14:39 #### kernel: RBP: 0000000000000009 R08: ffff88826b30ca80 R09: 0000000000000009
Sep 24 21:14:39 #### kernel: R10: 0000000000000f01 R11: fefefefefefefe01 R12: ffff88826b30ca80
Sep 24 21:14:39 #### kernel: R13: 0000000000000000 R14: ffff88810a692800 R15: 0000000000000000
Sep 24 21:14:39 #### kernel: FS:  0000152b0e43f900(0000) GS:ffff88a01fa00000(0000) knlGS:0000000000000000
Sep 24 21:14:39 #### kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 24 21:14:39 #### kernel: CR2: ffffffffa7466458 CR3: 0000000139e32002 CR4: 00000000003706f0
Sep 24 21:14:39 #### kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Sep 24 21:14:39 #### kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Sep 24 21:14:39 #### kernel: note: btrfs[1445014] exited with irqs disabled

And then that curl error about diagnostics again and... it start nginx back?????

Sep 24 21:14:39 #### publish: curl to diagnostics failed
Sep 24 21:14:39 #### publish: curl to diagnostics failed
Sep 24 21:14:39 #### publish: curl to diagnostics failed
Sep 24 21:14:41 #### rc.nginx: Starting Nginx server daemon...
Sep 24 21:14:42 #### rc.nginx: Nginx server daemon...  Started.

It's not normal. Honestly, that diagnostic on shutdown is good for nothing, specially in case of emergency reboot. Someone needing to shutdown because of battery low could have a server crash just because it decided to ran a diagnostic nobody will check.

Edited September 25, 2025Sep 25 by Nodiaque

The diagnostics are only saved on shutdown if Unraid cannot do a clean shutdown. Does it normally shut down cleanly after a boot or is this every time?

It's random. I rebooted maybe 4 days before without any issue. But randomly, it happen and I never can find anything in log saying with suddently, it need it. Honestly, even on unclean shutdown, it need a failsafe to say if it hang trying to get it, stop and close the server. We are closing it, I need it close not hanging eating battery. Maybe an option to remove diagnostic on shutdown and if it's needed, we activate it for something.

But even there, we have the diagnostic and it seems it doesn't tell you why it's an unclean shutdown, meaning it useless

15 hours ago, JorgeB said:

The diagnostics are only saved on shutdown if Unraid cannot do a clean shutdown. Does it normally shut down cleanly after a boot or is this every time?

This is not the correct. Or at least mine does the diagnostics collection thing every time and always has as far as I know. This is on my current VM but running it bare metal was my original post where it did the same. It may be an option I've set but I can't seem to find it now.

This is an example of a successful shutdown where it doesn't hang, it successfully unmounts everything and reboots fine. But it still does the diagnostics collection. I note that a real "unclean" shutdown usually does a parity check. Even when I have to hard reset after the diagnostics hang, the array is considered clean.

I think my next step will be to find the reboot/shutdown script and remove diagnostic from it. I really don't care about a diagnostic on shutdown that prevent a shutdown. I will get a warning about dirty shutdown anyway on next boot. Right now, it's way worst to have a hard shutdown be Aude of battery failure or hard reset cause it's frozen.

8 minutes ago, Nodiaque said:

I think my next step will be to find the reboot/shutdown script and remove diagnostic from it. I really don't care about a diagnostic on shutdown that prevent a shutdown. I will get a warning about dirty shutdown anyway on next boot. Right now, it's way worst to have a hard shutdown be Aude of battery failure or hard reset cause it's frozen.

It's more than a script. I've grepped through the whole FS and can't seem to find even where that string comes from.

The docs indicate unclean is really a timeout on shutdown. It would seem perhaps mine is triggered by a docker shutdown timeout. That's not really an unclean shutdown IMO.

Setting the General shutdown timer on mine to 10 minutes still results in the diagnostics collection and it reboots and restarts all the way in less than 5 minutes so that's not triggering it.

https://docs.unraid.net/unraid-os/troubleshooting/common-issues/unclean-shutdowns/

Still doesn't really change the fact that the diagnostics collection should timeout. Particularly in the case of a bad system I want a best effort to shutdown.

Edit: It's not the docker one. I set it the timeout to 30 seconds (which is supposed to be per docker). It shutdown 20+ dockers and got to the "diagnostics collection" in about 47 seconds based on the web interface.

Edited September 26, 2025Sep 26 by Shonky

Wow a timeout.... Unraid does have a hard time closing some docker from the webgui. Something I've saw many post on the forum with no fixe. You send the docker stop in terminal and the docker stop in 2 seconds but through webgui, it timeout.

Still, nothing should prevent a shutdown except a kernel panic.

My server already take over 5 minutes to boot and about the same time to close without diagnostic.

I feel ill see where the diagnostic command is and render it none working. So it will launch it but do nothing.

3 hours ago, Shonky said:

This is not the correct.

This means the shutdown did not occur before the timeout, hence the generated diags:

8 hours ago, JorgeB said:

This means the shutdown did not occur before the timeout, hence the generated diags:

Which timeout? I changed it to 600 seconds (so this now said "Waiting up to 600 seconds...") and it shutdown in under 5 minutes including the doing the diagnostics collection when it worked. So no chance it reached the set timeout. The diagnostics should still timeout. Do you know where this diagnostics collection is implemented?

Edited September 26, 2025Sep 26 by Shonky