Hello everyone. I have a Dell R510 with Unraid. Unfortunately, it goes down every 4-5 days and doesn't come back online. I saw something in the logs but can't make sense of it. Can someone help me? Many thanks to everyone.
Log:
Aug 2 07:07:55 Tower kernel: ? kthread_complete_and_exit+0x1b/0x1b
Aug 2 07:07:55 Tower kernel: ret_from_fork+0x22/0x30
Aug 2 07:07:55 Tower kernel: </TASK>
Aug 2 07:09:50 Tower webGUI: Successful login user root from 192.168.2.1
Aug 2 07:10:00 Tower rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="1147" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Aug 2 07:10:55 Tower kernel: rcu: INFO: rcu_preempt self-detected stall on CPU
Aug 2 07:10:55 Tower kernel: rcu: 10-....: (240002 ticks this GP) idle=b18c/1/0x4000000000000000 softirq=25392315/25392315 fqs=119752
Aug 2 07:10:55 Tower kernel: (t=240003 jiffies g=90254029 q=731854 ncpus=16)
Aug 2 07:10:55 Tower kernel: CPU: 10 PID: 13138 Comm: unraidd0 Tainted: P W IO 6.1.99-Unraid #1
Aug 2 07:10:55 Tower kernel: Hardware name: Dell Inc. PowerEdge R510/0DPRKF, BIOS 1.14.0 05/30/2018
Aug 2 07:10:55 Tower kernel: RIP: 0010:schedule_read+0x77/0x86 [md_mod]
Aug 2 07:10:55 Tower kernel: Code: 00 b8 11 ff ff 01 b9 00 04 00 00 48 c1 e2 29 48 03 93 a8 00 00 00 48 c1 e0 27 48 c1 fa 06 48 c1 e2 0c 48 01 c2 31 c0 48 89 d7 <f3> ab 48 81 0b 00 01 00 00 5b c3 cc cc cc cc 0f 1f 44 00 00 41 57
Aug 2 07:10:55 Tower kernel: RSP: 0018:ffffc9000c3a7de0 EFLAGS: 00010246
Aug 2 07:10:55 Tower kernel: RAX: 0000000000000000 RBX: ffff88837f6e84a8 RCX: 00000000000003f9
Aug 2 07:10:55 Tower kernel: RDX: ffff88837f7a2000 RSI: 0000000000000005 RDI: ffff88837f7a201c
Aug 2 07:10:55 Tower kernel: RBP: 0000000000000000 R08: 0000000000000002 R09: 0000000000000002
Aug 2 07:10:55 Tower kernel: R10: 0000000000000000 R11: 00000000000002c4 R12: 000000000000000b
Aug 2 07:10:55 Tower kernel: R13: 0000000000000005 R14: 000000000000000c R15: 0000000000000000
Aug 2 07:10:55 Tower kernel: FS: 0000000000000000(0000) GS:ffff888623d40000(0000) knlGS:0000000000000000
Aug 2 07:10:55 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 2 07:10:55 Tower kernel: CR2: 0000149bd5ebe010 CR3: 000000000220a006 CR4: 00000000000226e0
Aug 2 07:10:55 Tower kernel: Call Trace:
Aug 2 07:10:55 Tower kernel: <IRQ>
Aug 2 07:10:55 Tower kernel: ? rcu_dump_cpu_stacks+0x95/0xb9
Aug 2 07:10:55 Tower kernel: ? rcu_sched_clock_irq+0x345/0xa48
Aug 2 07:10:55 Tower kernel: ? tick_init_jiffy_update+0x7c/0x7c
Aug 2 07:10:55 Tower kernel: ? update_process_times+0x62/0x81
Aug 2 07:10:55 Tower kernel: ? tick_sched_timer+0x43/0x71
Aug 2 07:10:55 Tower kernel: ? __hrtimer_run_queues+0xeb/0x190
Aug 2 07:10:55 Tower kernel: ? hrtimer_interrupt+0x9c/0x16e
Aug 2 07:10:55 Tower kernel: ? __sysvec_apic_timer_interrupt+0xc5/0x12f
Aug 2 07:10:55 Tower kernel: ? sysvec_apic_timer_interrupt+0x80/0xa6
Aug 2 07:10:55 Tower kernel: </IRQ>
Aug 2 07:10:55 Tower kernel: <TASK>
Aug 2 07:10:55 Tower kernel: ? asm_sysvec_apic_timer_interrupt+0x16/0x20
Aug 2 07:10:55 Tower kernel: ? schedule_read+0x77/0x86 [md_mod]
Aug 2 07:10:55 Tower kernel: unraidd+0xd54/0x1140 [md_mod]
Aug 2 07:10:55 Tower kernel: md_thread+0xf7/0x122 [md_mod]
Aug 2 07:10:55 Tower kernel: ? _raw_spin_rq_lock_irqsave+0x20/0x20
Aug 2 07:10:55 Tower kernel: ? signal_pending+0x1d/0x1d [md_mod]
Aug 2 07:10:55 Tower kernel: kthread+0xe7/0xef
Aug 2 07:10:55 Tower kernel: ? kthread_complete_and_exit+0x1b/0x1b
Aug 2 07:10:55 Tower kernel: ret_from_fork+0x22/0x30
Aug 2 07:10:55 Tower kernel: </TASK>
Aug 2 07:11:00 Tower vnstatd[4105]: Traffic rate for "br0" higher than set maximum 10 Mbit (20s->27262976, r513879924 t2861048, 64bit:1), syncing.
Aug 2 07:11:04 Tower kernel: clocksource: Long readout interval, skipping watchdog check: cs_nsec: 244185826794 wd_nsec: 0
Aug 2 07:11:20 Tower vnstatd[4105]: Traffic rate for "br0" higher than set maximum 10 Mbit (20s->27262976, r2216818136 t5746769, 64bit:1), syncing.
Aug 2 07:11:40 Tower vnstatd[4105]: Traffic rate for "br0" higher than set maximum 10 Mbit (20s->27262976, r2169698597 t4950160, 64bit:1), syncing.
Aug 2 07:12:00 Tower vnstatd[4105]: Detected bandwidth limit for "br0" changed from 10 Mbit to 1000 Mbit.
Aug 2 07:12:04 Tower kernel: rcu: INFO: rcu_preempt self-detected stall on CPU
Aug 2 07:12:04 Tower kernel: rcu: 0-....: (60000 ticks this GP) idle=943c/1/0x4000000000000000 softirq=52581962/52581962 fqs=27526
Aug 2 07:12:04 Tower kernel: (t=60001 jiffies g=90254037 q=785985 ncpus=16)
Aug 2 07:12:04 Tower kernel: CPU: 0 PID: 13138 Comm: unraidd0 Tainted: P W IO 6.1.99-Unraid #1
Aug 2 07:12:04 Tower kernel: Hardware name: Dell Inc. PowerEdge R510/0DPRKF, BIOS 1.14.0 05/30/2018
Aug 2 07:12:04 Tower kernel: RIP: 0010:schedule_read+0x77/0x86 [md_mod]
Aug 2 07:12:04 Tower kernel: Code: 00 b8 11 ff ff 01 b9 00 04 00 00 48 c1 e2 29 48 03 93 a8 00 00 00 48 c1 e0 27 48 c1 fa 06 48 c1 e2 0c 48 01 c2 31 c0 48 89 d7 <f3> ab 48 81 0b 00 01 00 00 5b c3 cc cc cc cc 0f 1f 44 00 00 41 57
Aug 2 07:12:04 Tower kernel: RSP: 0018:ffffc9000c3a7de0 EFLAGS: 00010246
Aug 2 07:12:04 Tower kernel: RAX: 0000000000000000 RBX: ffff88837f6ec3a8 RCX: 00000000000002c3
Aug 2 07:12:04 Tower kernel: RDX: ffff88837f7f3000 RSI: 0000000000000008 RDI: ffff88837f7f34f4
Aug 2 07:12:04 Tower kernel: RBP: 0000000000000000 R08: 0000000000000002 R09: 0000000000000002
Aug 2 07:12:04 Tower kernel: R10: 0000000000000000 R11: 000000000002aeeb R12: 000000000000000b
Aug 2 07:12:04 Tower kernel: R13: 0000000000000008 R14: 000000000000000c R15: 0000000000000000
Aug 2 07:12:04 Tower kernel: FS: 0000000000000000(0000) GS:ffff888623c00000(0000) knlGS:0000000000000000
Aug 2 07:12:04 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 2 07:12:04 Tower kernel: CR2: 0000014b0aa47a9b CR3: 000000000220a001 CR4: 00000000000226f0
Aug 2 07:12:04 Tower kernel: Call Trace:
Aug 2 07:12:04 Tower kernel: <IRQ>
Aug 2 07:12:04 Tower kernel: ? rcu_dump_cpu_stacks+0x95/0xb9
Aug 2 07:12:04 Tower kernel: ? rcu_sched_clock_irq+0x345/0xa48
Aug 2 07:12:04 Tower kernel: ? tick_init_jiffy_update+0x7c/0x7c
Aug 2 07:12:04 Tower kernel: ? update_process_times+0x62/0x81
Aug 2 07:12:04 Tower kernel: ? tick_sched_timer+0x43/0x71
Aug 2 07:12:04 Tower kernel: ? __hrtimer_run_queues+0xeb/0x190
Aug 2 07:12:04 Tower kernel: ? hrtimer_interrupt+0x9c/0x16e
Aug 2 07:12:04 Tower kernel: ? __sysvec_apic_timer_interrupt+0xc5/0x12f
Aug 2 07:12:04 Tower kernel: ? sysvec_apic_timer_interrupt+0x80/0xa6
Aug 2 07:12:04 Tower kernel: </IRQ>
Aug 2 07:12:04 Tower kernel: <TASK>
Aug 2 07:12:04 Tower kernel: ? asm_sysvec_apic_timer_interrupt+0x16/0x20
Aug 2 07:12:04 Tower kernel: ? schedule_read+0x77/0x86 [md_mod]
Aug 2 07:12:04 Tower kernel: unraidd+0xd54/0x1140 [md_mod]
Aug 2 07:12:04 Tower kernel: md_thread+0xf7/0x122 [md_mod]
Aug 2 07:12:04 Tower kernel: ? _raw_spin_rq_lock_irqsave+0x20/0x20
Aug 2 07:12:04 Tower kernel: ? signal_pending+0x1d/0x1d [md_mod]
Aug 2 07:12:04 Tower kernel: kthread+0xe7/0xef
Aug 2 07:12:04 Tower kernel: ? kthread_complete_and_exit+0x1b/0x1b
Aug 2 07:12:04 Tower kernel: ret_from_fork+0x22/0x30
Aug 2 07:12:04 Tower kernel: </TASK>
Aug 2 07:13:09 Tower webGUI: Successful login user root from 192.168.188.42
Aug 2 07:13:13 Tower nginx: 2024/08/02 07:13:13 [crit] 10046#10046: *1671272 connect() to unix:/var/run/syslog.sock failed (2: No such file or directory) while connecting to upstream, client: 192.168.188.42, server: , request: "GET /webterminal/syslog/ HTTP/1.1", upstream: "http://unix:/var/run/syslog.sock:/", host: "192.168.188.173", referrer: "http://192.168.188.173/Main"
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: <--- start FTQ dump --->
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: RV2P_PFTQ_CTL 00010002
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: RV2P_TFTQ_CTL 00020000
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: RV2P_MFTQ_CTL 00004000
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: TBDR_FTQ_CTL 00004002
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: TDMA_FTQ_CTL 00010002
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: TXP_FTQ_CTL 00010002
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: TXP_FTQ_CTL 00010002
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: TPAT_FTQ_CTL 00010000
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: RXP_CFTQ_CTL 00008000
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: RXP_FTQ_CTL 00100000
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: COM_COMXQ_FTQ_CTL 00010000
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: COM_COMTQ_FTQ_CTL 00020000
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: COM_COMQ_FTQ_CTL 00010000
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: CP_CPQ_FTQ_CTL 00004000
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: CPU states:
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 045000 mode b84c state 80001000 evt_mask 500 pc 800128c pc 8001284 instr 38640001
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 085000 mode b84c state 80001000 evt_mask 500 pc 8000a5c pc 8000a4c instr 38420001
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 0c5000 mode b84c state 80005000 evt_mask 500 pc 8004c18 pc 8004c14 instr 32070001
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 105000 mode b8cc state 80000000 evt_mask 500 pc 8000b24 pc 8000b2c instr 32620001
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 145000 mode b880 state 80000000 evt_mask 500 pc 800d244 pc 800d1b0 instr 24130001
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 185000 mode b8cc state 80000000 evt_mask 500 pc 8000c58 pc 8000c58 instr 1092823
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: <--- end FTQ dump --->
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: <--- start TBDC dump --->
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: TBDC free cnt: 32
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: LINE CID BIDX CMD VALIDS
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 00 000800 ae30 00 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 01 000800 ae30 00 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 02 000800 ad68 00 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 03 001100 af20 00 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 04 001000 3988 00 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 05 001000 3990 00 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 06 001000 3998 00 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 07 001000 0038 00 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 08 001080 0080 00 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 09 001080 0088 00 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 0a 001080 0098 00 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 0b 001080 00a8 00 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 0c 001000 42f0 00 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 0d 000800 1dd0 00 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 0e 001100 f118 00 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 0f 001100 1538 00 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 10 000800 4288 00 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 11 001100 6048 00 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 12 001100 6050 00 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 13 19ed80 af48 be [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 14 0f5f80 fbe0 3f [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 15 0edf80 3ff8 ff [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 16 1ff780 b698 ef [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 17 1d7d80 f7e0 9b [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 18 1e5780 bff8 7b [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 19 1dbf80 fb38 f9 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 1a 1ddb80 ff68 fb [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 1b 1f7f80 7f38 64 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 1c 1fef80 f3f8 7b [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 1d 10a780 f6f8 6f [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 1e 1f7f80 ef68 e6 [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: 1f 1dfe00 f7d8 9d [0]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: <--- end TBDC dump --->
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: DEBUG: intr_sem[0] PCI_CMD[00100406]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: DEBUG: PCI_PM[19002008] PCI_MISC_CFG[92000088]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: DEBUG: EMAC_TX_STATUS[0000000e] EMAC_RX_STATUS[00000000]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: DEBUG: RPM_MGMT_PKT_CTRL[40000088]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: DEBUG: HC_STATS_INTERRUPT_STATUS[01fb0004]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: DEBUG: PBA[00000000]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: <--- start MCP states dump --->
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: DEBUG: MCP_STATE_P0[0003610e] MCP_STATE_P1[0003600e]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: DEBUG: MCP mode[0000b880] state[80000000] evt_mask[00000500]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: DEBUG: pc[0800b26c] pc[0800b114] instr[afbf0048]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: DEBUG: shmem states:
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: DEBUG: drv_mb[01030009] fw_mb[00000009] link_status[0000006f]
Aug 2 07:13:41 Tower kernel: drv_pulse_mb[00007ebe]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: DEBUG: dev_info_signature[44564907] reset_type[01005254]
Aug 2 07:13:41 Tower kernel: condition[0003610e]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: DEBUG: 000001c0: 01005254 42530085 0003610e 00000000
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: DEBUG: 000003cc: 00000000 00000000 00000000 00000000
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: DEBUG: 000003dc: 00000000 00000000 00000000 00000000
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: DEBUG: 000003ec: 00000000 00000000 00000000 00000000
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: DEBUG: 0x3fc[00000000]
Aug 2 07:13:41 Tower kernel: bnx2 0000:01:00.0 eth0: <--- end MCP states dump --->
Aug 2 07:13:47 Tower kernel: bnx2 0000:01:00.0 eth0: <--- start FTQ dump --->
Aug 2 07:13:47 Tower kernel: bnx2 0000:01:00.0 eth0: RV2P_PFTQ_CTL 00010002
Aug 2 07:13:47 Tower kernel: bnx2 0000:01:00.0 eth0: RV2P_TFTQ_CTL 00020000
Aug 2 07:13:47 Tower kernel: bnx2 0000:01:00.0 eth0: RV2P_MFTQ_CTL 00004000
Aug 2 07:13:47 Tower kernel: bnx2 0000:01:00.0 eth0: TBDR_FTQ_CTL 00004002
tower-diagnostics-20240614-1800.zip