Frequent crashes problem.

[KOSSDUST]

I have unraid that is crashing almost every few hours. I have done a memory test and passed. I have run in safemode for a day and a half and no crash. I have tried to update and check versions of all installed dockers/plugins to verify they are the latest. 

 

I am at a loss and do not know what to check next. 

miniserver-diagnostics-20230311-1937.zip syslog

[itimpi]

Start by removing NerdPack plugin as that is incompatible with Unraid 6.11.5 and rebooting.

 

[KOSSDUST]

12 minutes ago, itimpi said:

Start by removing NerdPack plugin as that is incompatible with Unraid 6.11.5 and rebooting.

 

Thanks for the quick reply, but maybe I am missing something. I do not see the plugin installed and when I check apps for what is installed I am not seeing it. I have no idea how to uninstall it as I can't find it. Google has not helped me with uninstalling it either. 

 

When I view the plugin txt file I am not seeing it either:

ca.update.applications.plg - 2023.02.20  (Up to date)
community.applications.plg - 2023.03.11  (Up to date)
docker.patch.plg - 2023.01.28  (Up to date)
dynamix.system.info.plg - 2023.02.05  (Up to date)
dynamix.system.stats.plg - 2023.02.14  (Up to date)
dynamix.system.temp.plg - 2023.02.04b  (Up to date)
dynamix.unraid.net.plg - 2023.03.09.1140  (Up to date)
fix.common.problems.plg - 2023.03.04  (Up to date)
gpustat.plg - 2022.11.30a  (Up to date)
nvidia-driver.plg - 2023.03.02  (Up to date)
theme.engine.plg - 2023.01.17  (Up to date)
unassigned.devices.plg - 2023.03.03  (Up to date)
unRAIDServer.plg - 6.11.5    
user.scripts.plg - 2023.03.04  (Up to date)
 

[itimpi]

Just realized that the syslog file you provided is from the syslog server and the log entry referencing NerdPack is an old one - sorry about that.

 

If the server is stable when booted in Safe Mode it is going to be one of your plugins - but not sure which one it would be so it may be a case of eliminating them one at a time.  You can disable any plugin by renaming its .plg file to have a different extension and rebooting.   I would probably suggest starting with the gpustat one simply because it appears to be some time since it was last updated with the next most likely one being the NVida driver one as being the most instrusive.    However those are just guesses as I have no specific reason to assume which would be the guilty suspect.

[KOSSDUST]

Removed the plugins and also went back to safemode, disabled VMs and deepstack docker and server stayed up for 12ish hours. 

 

Got a new sys log with what seems to be some odd stuff repeated in the last couple of hours, of course I have no idea what I am looking at:

 

Mar 13 02:11:07 MiniServer kernel: <TASK>
Mar 13 02:11:07 MiniServer kernel: do_raw_spin_lock+0x14/0x1a
Mar 13 02:11:07 MiniServer kernel: release_stripe+0x20/0x37 [md_mod]
Mar 13 02:11:07 MiniServer kernel: unraidd+0x10ce/0x1140 [md_mod]
Mar 13 02:11:07 MiniServer kernel: md_thread+0x103/0x12e [md_mod]
Mar 13 02:11:07 MiniServer kernel: ? _raw_spin_rq_lock_irqsave+0x20/0x20
Mar 13 02:11:07 MiniServer kernel: ? md_seq_show+0x720/0x720 [md_mod]
Mar 13 02:11:07 MiniServer kernel: kthread+0xe7/0xef
Mar 13 02:11:07 MiniServer kernel: ? kthread_complete_and_exit+0x1b/0x1b
Mar 13 02:11:07 MiniServer kernel: ret_from_fork+0x22/0x30
Mar 13 02:11:07 MiniServer kernel: </TASK>
Mar 13 02:14:07 MiniServer kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Mar 13 02:14:07 MiniServer kernel: rcu:     3-...0: (3 ticks this GP) idle=f4f/1/0x4000000000000000 softirq=4472796/4472798 fqs=2287826 
Mar 13 02:14:07 MiniServer kernel:     (detected by 8, t=9420265 jiffies, g=7404321, q=1341088 ncpus=16)
Mar 13 02:14:07 MiniServer kernel: Sending NMI from CPU 8 to CPUs 3:
Mar 13 02:14:07 MiniServer kernel: NMI backtrace for cpu 3
Mar 13 02:14:07 MiniServer kernel: CPU: 3 PID: 3330 Comm: unraidd0 Tainted: P      D    O      5.19.17-Unraid #2
Mar 13 02:14:07 MiniServer kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470D4U, BIOS P4.20 04/14/2021
Mar 13 02:14:07 MiniServer kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x87/0x1d0
Mar 13 02:14:07 MiniServer kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 0f 9d 77 00 49 c7 c4 00 ce
Mar 13 02:14:07 MiniServer kernel: RSP: 0018:ffffc900028f7da0 EFLAGS: 00000002
Mar 13 02:14:07 MiniServer kernel: RAX: 00000000001c0101 RBX: ffff888105828570 RCX: 0000000000000000
Mar 13 02:14:07 MiniServer kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff888105828570
Mar 13 02:14:07 MiniServer kernel: RBP: ffff888105828570 R08: 0000000000000000 R09: ffffc900028f7d88
Mar 13 02:14:07 MiniServer kernel: R10: 00000000ffffffff R11: ffff8881b7add1f0 R12: ffff888105828000
Mar 13 02:14:07 MiniServer kernel: R13: ffff8881b7add850 R14: ffff8881b7add978 R15: ffff8881040fe718
Mar 13 02:14:07 MiniServer kernel: FS:  0000000000000000(0000) GS:ffff888ffe8c0000(0000) knlGS:0000000000000000
Mar 13 02:14:07 MiniServer kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 13 02:14:07 MiniServer kernel: CR2: 00007fff89c55678 CR3: 000000062bae0000 CR4: 0000000000350ee0
Mar 13 02:14:07 MiniServer kernel: DR0: 0000000002ef84f7 DR1: 0000000000000000 DR2: 0000000000000000
Mar 13 02:14:07 MiniServer kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 13 02:14:07 MiniServer kernel: Call Trace:

syslog

Edited March 13, 2023 by KOSSDUST

[JorgeB]

Last crashes are from the Unraid driver itself, these are usually caused by hardware issues.