Hi Team,

I've been trying to track down why my unRAID server randomly goes completely unresponsive. I unfortunately have to yank the power plug as I can't get to it via HTTP, SSH, or any docker/VMs. (haven't yet tried to plug a monitor in; JetKVM is on the way!). I have syslog server set up to output my log.

 

Running unRAID 7.0.0

Intel Core i3-13100

Gigabyte B760I AORUS PRO DDR4

BIOS Version F12 Dated 09/27/2024

32GB Corsair Vengeance DDR4 3600

5x Seagate EXOS 18TB array [one drive connected via JMB582 PCIe card]

2x Crucial T500 cache pool

64GB Samsung Fit Plus USB drive

 

This is what has been repeating in the log.

 

Jan 13 06:28:04 Unraid kernel: DMAR: [DMA Write NO_PASID] Request device [04:00.0] fault addr 0x60410daf000 [fault reason 0x04] Access beyond MGAW
Jan 13 06:28:04 Unraid kernel: DMAR: DRHD: handling fault status reg 3
Jan 13 06:28:08 Unraid kernel: dmar_fault: 137 callbacks suppressed

 

Looking at System Devices, 04:00.0 refers to Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)

It's plugged in to my Ubiquiti USW Enterprise 24 PoE and negotiates at 2.5gbe.

 

After the above error was repeating for hours (and by the looks of it, days), I came across this repeating in the log as well, referring to eth0.

Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: NETDEV WATCHDOG: CPU: 7: transmit queue 3 timed out 5217 ms
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: Register Dump
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: Register Name   Value
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: CTRL            081c0641
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: STATUS          40680683
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: CTRL_EXT        100000c0
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: MDIC            1805d181
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: ICR             000000c1
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: RCTL            0440803a
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: RDLEN[0-3]      00001000 00001000 00001000 00001000
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: RDH[0-3]        0000006d 000000bf 000000fc 00000028
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: RDT[0-3]        0000006d 000000be 000000fb 000000c3
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: RXDCTL[0-3]     02040808 02040808 02040808 02040808
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: RDBAL[0-3]      4a8dd000 46f38000 45623000 45662000
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: RDBAH[0-3]      00000001 00000001 00000001 00000001
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: TCTL            a503f0fa
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: TDBAL[0-3]      48c07000 46f37000 044af000 45644000
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: TDBAH[0-3]      00000001 00000001 00000001 00000001
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: TDLEN[0-3]      00001000 00001000 00001000 00001000
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: TDH[0-3]        0000005e 00000025 00000023 000000c7
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: TDT[0-3]        0000005e 00000025 00000023 000000c7
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: TXDCTL[0-3]     02100108 02100108 02100108 02100108
Jan 16 05:30:18 Unraid kernel: igc 0000:04:00.0 eth0: Reset adapter
Jan 16 05:30:20 Unraid kernel: igc 0000:04:00.0 eth0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX
Jan 16 05:30:25 Unraid kernel: igc 0000:04:00.0 eth0: NETDEV WATCHDOG: CPU: 7: transmit queue 0 timed out 7168 ms
Jan 16 05:30:32 Unraid kernel: igc 0000:04:00.0 eth0: NETDEV WATCHDOG: CPU: 7: transmit queue 0 timed out 13312 ms
Jan 16 05:30:37 Unraid kernel: igc 0000:04:00.0 eth0: NETDEV WATCHDOG: CPU: 7: transmit queue 0 timed out 18944 ms
Jan 16 05:30:42 Unraid kernel: igc 0000:04:00.0 eth0: NETDEV WATCHDOG: CPU: 7: transmit queue 0 timed out 24064 ms
Jan 16 05:30:46 Unraid kernel: rcu: INFO: rcu_preempt self-detected stall on CPU
Jan 16 05:30:46 Unraid kernel: rcu: 	3-....: (60461 ticks this GP) idle=d4c4/1/0x4000000000000004 softirq=25275163/25275165 fqs=27387
Jan 16 05:30:46 Unraid kernel: rcu: 	(t=60000 jiffies g=36840005 q=246830 ncpus=8)
Jan 16 05:30:46 Unraid kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: P           O       6.6.68-Unraid #1
Jan 16 05:30:46 Unraid kernel: Hardware name: Gigabyte Technology Co., Ltd. B760I AORUS PRO DDR4/B760I AORUS PRO DDR4, BIOS F12 09/27/2024
Jan 16 05:30:46 Unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x176/0x1cc
Jan 16 05:30:46 Unraid kernel: Code: 48 03 14 f5 40 b9 2e 82 48 89 0a 8b 51 08 85 d2 75 04 f3 90 eb f5 48 8b 11 48 85 d2 74 03 0f 0d 0a 8b 33 66 85 f6 74 04 f3 90 <eb> f5 39 c6 74 0a 48 85 d2 c6 03 01 75 1b eb 0d be 01 00 00 00 f0
Jan 16 05:30:46 Unraid kernel: RSP: 0018:ffffc900002ec950 EFLAGS: 00000206
Jan 16 05:30:46 Unraid kernel: RAX: 0000000000100000 RBX: ffff88845227fe04 RCX: ffff88889faee2c0
Jan 16 05:30:46 Unraid kernel: RDX: 0000000000000000 RSI: 0000000000040200 RDI: ffff88845227fe04
Jan 16 05:30:46 Unraid kernel: RBP: 0000000000100000 R08: ffffffffa0f96986 R09: 0000000000000014
Jan 16 05:30:46 Unraid kernel: R10: 1ae52181b743a495 R11: 000635c200000000 R12: ffff88889faee2c0
Jan 16 05:30:46 Unraid kernel: R13: 0000000000000000 R14: 0000000000000003 R15: 0000000000000000
Jan 16 05:30:46 Unraid kernel: FS:  0000000000000000(0000) GS:ffff88889fac0000(0000) knlGS:0000000000000000
Jan 16 05:30:46 Unraid kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 16 05:30:46 Unraid kernel: CR2: 00007f687a287e80 CR3: 0000000460a5a000 CR4: 0000000000752ee0
Jan 16 05:30:46 Unraid kernel: PKRU: 55555554
Jan 16 05:30:46 Unraid kernel: Call Trace:
Jan 16 05:30:46 Unraid kernel: <IRQ>
Jan 16 05:30:46 Unraid kernel: ? rcu_dump_cpu_stacks+0x94/0xb8
Jan 16 05:30:46 Unraid kernel: ? rcu_sched_clock_irq+0x34c/0xa25
Jan 16 05:30:46 Unraid kernel: ? notifier_call_chain+0x85/0xae
Jan 16 05:30:46 Unraid kernel: ? timekeeping_update+0xea/0x118
Jan 16 05:30:46 Unraid kernel: ? __pfx_tick_sched_timer+0x10/0x10
Jan 16 05:30:46 Unraid kernel: ? update_process_times+0x62/0x81
Jan 16 05:30:46 Unraid kernel: ? tick_sched_timer+0x49/0x79
Jan 16 05:30:46 Unraid kernel: ? __hrtimer_run_queues+0xed/0x195
Jan 16 05:30:46 Unraid kernel: ? hrtimer_interrupt+0x9c/0x16e
Jan 16 05:30:46 Unraid kernel: ? __sysvec_apic_timer_interrupt+0xc2/0x12f
Jan 16 05:30:46 Unraid kernel: ? sysvec_apic_timer_interrupt+0x36/0xb5
Jan 16 05:30:46 Unraid kernel: ? asm_sysvec_apic_timer_interrupt+0x16/0x20
Jan 16 05:30:46 Unraid kernel: ? native_queued_spin_lock_slowpath+0x176/0x1cc
Jan 16 05:30:46 Unraid kernel: do_raw_spin_lock+0x14/0x1a
Jan 16 05:30:46 Unraid kernel: nf_conntrack_tcp_packet+0x12f/0xf76 [nf_conntrack]
Jan 16 05:30:46 Unraid kernel: ? ip_rcv_core+0x31d/0x32e
Jan 16 05:30:46 Unraid kernel: ? ip_rcv+0x19/0x43
Jan 16 05:30:46 Unraid kernel: ? nf_ct_key_equal+0x50/0x6e [nf_conntrack]
Jan 16 05:30:46 Unraid kernel: ? __nf_conntrack_find_get+0xe5/0x11e [nf_conntrack]
Jan 16 05:30:46 Unraid kernel: nf_conntrack_in+0x315/0x4cf [nf_conntrack]
Jan 16 05:30:46 Unraid kernel: ? __pfx_br_nf_pre_routing_finish+0x10/0x10 [br_netfilter]
Jan 16 05:30:46 Unraid kernel: nf_hook_slow+0x3a/0x96
Jan 16 05:30:46 Unraid kernel: NF_HOOK.isra.0+0xc3/0x126 [br_netfilter]
Jan 16 05:30:46 Unraid kernel: ? __pfx_br_nf_pre_routing_finish+0x10/0x10 [br_netfilter]
Jan 16 05:30:46 Unraid kernel: br_nf_pre_routing+0x236/0x24a [br_netfilter]
Jan 16 05:30:46 Unraid kernel: ? __pfx_br_nf_pre_routing_finish+0x10/0x10 [br_netfilter]
Jan 16 05:30:46 Unraid kernel: br_handle_frame+0x280/0x2eb [bridge]
Jan 16 05:30:46 Unraid kernel: ? __pfx_br_handle_frame_finish+0x10/0x10 [bridge]
Jan 16 05:30:46 Unraid kernel: ? __pfx_br_handle_frame+0x10/0x10 [bridge]
Jan 16 05:30:46 Unraid kernel: __netif_receive_skb_core.constprop.0+0x590/0x75b
Jan 16 05:30:46 Unraid kernel: ? find_busiest_group+0x50/0x3cb
Jan 16 05:30:46 Unraid kernel: ? __update_load_avg_cfs_rq+0xf0/0x210
Jan 16 05:30:46 Unraid kernel: __netif_receive_skb_list_core+0x86/0x11c
Jan 16 05:30:46 Unraid kernel: netif_receive_skb_list_internal+0x1d0/0x209
Jan 16 05:30:46 Unraid kernel: ? inet_gro_receive+0x236/0x265
Jan 16 05:30:46 Unraid kernel: gro_normal_list+0x1d/0x3f
Jan 16 05:30:46 Unraid kernel: napi_complete_done+0x7b/0x110
Jan 16 05:30:46 Unraid kernel: igc_poll+0xd1b/0xf32 [igc]
Jan 16 05:30:46 Unraid kernel: ? wakeup_preempt+0x49/0x5a
Jan 16 05:30:46 Unraid kernel: __napi_poll.constprop.0+0x28/0x121
Jan 16 05:30:46 Unraid kernel: net_rx_action+0x109/0x200
Jan 16 05:30:46 Unraid kernel: handle_softirqs+0x124/0x262
Jan 16 05:30:46 Unraid kernel: __irq_exit_rcu+0x39/0x91
Jan 16 05:30:46 Unraid kernel: common_interrupt+0xaa/0xd0
Jan 16 05:30:46 Unraid kernel: </IRQ>

 

Diagnostics are attached.

unraid-diagnostics-20250116-0931.zip

Edited January 16, 20251 yr by spencers

Try using a different NIC if available.

I’m having the same issue. My server is becoming unresponsive about twice per day since upgrading to 7.0. Going to try to figure out how to roll mine back a version when I get home.

So far I've tried a different ethernet cord, thinking that the Monoprice SlimRun cat6a cable was to blame.

 

Overnight the server hung again. All I see in the logs prior to me yanking the power cable was two drives spinning down after a Macbook ran a Time Machine backup.  I also see these errors somewhat frequently, but I'm not sure if they're related.

 

Jan 16 22:20:27 Unraid kernel: x86/split lock detection: #AC: CPU 0/KVM/10979 took a split_lock trap at address: 0xfffff80608e0e1ed

 

My next course of action is to try disabling USB passthrough of a Sonoff zigbee dongle to a Home Assistant VM.

 

If unsuccessful, perhaps I'll try changing the switch port from auto-negotiate to force 1gbps FDX.

Disabling USB passthrough / moving my HomeAssistant VM off the unRAID server, over to a Raspberry Pi, has resolved the lockups, I believe.

 

Overnight last night I continued to get errors with the NIC. One occurred while I was watching something on Plex, which stopped playback.

Jan 23 01:43:19 Unraid kernel: igc 0000:04:00.0 eth0: NIC Link is Down
Jan 23 01:43:19 Unraid kernel: br0: port 1(eth0) entered disabled state
Jan 23 01:43:23 Unraid kernel: igc 0000:04:00.0 eth0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX
Jan 23 01:43:23 Unraid kernel: br0: port 1(eth0) entered blocking state
Jan 23 01:43:23 Unraid kernel: br0: port 1(eth0) entered forwarding state
Jan 23 02:47:27 Unraid kernel: igc 0000:04:00.0 eth0: NIC Link is Down
Jan 23 02:47:27 Unraid kernel: br0: port 1(eth0) entered disabled state
Jan 23 02:47:31 Unraid kernel: igc 0000:04:00.0 eth0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX
Jan 23 02:47:31 Unraid kernel: br0: port 1(eth0) entered blocking state
Jan 23 02:47:31 Unraid kernel: br0: port 1(eth0) entered forwarding state
Jan 23 03:50:21 Unraid kernel: igc 0000:04:00.0 eth0: NIC Link is Down
Jan 23 03:50:21 Unraid kernel: br0: port 1(eth0) entered disabled state
Jan 23 03:50:25 Unraid kernel: igc 0000:04:00.0 eth0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX
Jan 23 03:50:25 Unraid kernel: br0: port 1(eth0) entered blocking state
Jan 23 03:50:25 Unraid kernel: br0: port 1(eth0) entered forwarding state
Jan 23 04:54:43 Unraid kernel: igc 0000:04:00.0 eth0: NIC Link is Down
Jan 23 04:54:43 Unraid kernel: br0: port 1(eth0) entered disabled state
Jan 23 04:54:47 Unraid kernel: igc 0000:04:00.0 eth0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX
Jan 23 04:54:47 Unraid kernel: br0: port 1(eth0) entered blocking state
Jan 23 04:54:47 Unraid kernel: br0: port 1(eth0) entered forwarding state
Jan 23 05:58:23 Unraid kernel: igc 0000:04:00.0 eth0: NIC Link is Down
Jan 23 05:58:23 Unraid kernel: br0: port 1(eth0) entered disabled state
Jan 23 05:58:27 Unraid kernel: igc 0000:04:00.0 eth0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX
Jan 23 05:58:27 Unraid kernel: br0: port 1(eth0) entered blocking state
Jan 23 05:58:27 Unraid kernel: br0: port 1(eth0) entered forwarding state

 

So this morning I moved unRAID to a different port on my switch. And for the heck of it (likely unrelated), I moved docker over from macvlan to ipvlan.

 

Still getting network drops. Sadly I'm not sure what to try next.

Jan 23 12:21:35 Unraid kernel: igc 0000:04:00.0 eth0: NIC Link is Down
Jan 23 12:21:35 Unraid kernel: br0: port 1(eth0) entered disabled state
Jan 23 12:21:39 Unraid kernel: igc 0000:04:00.0 eth0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX
Jan 23 12:21:39 Unraid kernel: br0: port 1(eth0) entered blocking state
Jan 23 12:21:39 Unraid kernel: br0: port 1(eth0) entered forwarding state

 

This is typically a cable/switch/NIC issue.

Gave up and replaced motherboard with a different model containing Realtek 2.5gbe

On 1/26/2025 at 8:18 PM, spencers said:

Gave up and replaced motherboard with a different model containing Realtek 2.5gbe

Did this resolve the watchdog issue and becoming unresponsive? this has happened to me every day the last 4 or 5 days.  I dont have anything being passed through. Fortunately I have a KVM and can safe shutdown, but Im losing my mind doing this every day.

23 minutes ago, stayupthetree said:

Did this resolve the watchdog issue and becoming unresponsive? this has happened to me every day the last 4 or 5 days.  I dont have anything being passed through. Fortunately I have a KVM and can safe shutdown, but Im losing my mind doing this every day.

Funny you should ask! Just recently I had the issue resurface on the new motherboard.

 

Jan 30 12:08:10 Unraid kernel: r8169 0000:04:00.0 eth0: Link is Down
Jan 30 12:08:10 Unraid kernel: br0: port 1(eth0) entered disabled state
Jan 30 12:08:13 Unraid kernel: r8169 0000:04:00.0 eth0: Link is Up - 2.5Gbps/Full - flow control off
Jan 30 12:08:13 Unraid kernel: br0: port 1(eth0) entered blocking state
Jan 30 12:08:13 Unraid kernel: br0: port 1(eth0) entered forwarding state
Jan 30 12:08:33 Unraid kernel: r8169 0000:04:00.0 eth0: Link is Down
Jan 30 12:08:33 Unraid kernel: br0: port 1(eth0) entered disabled state
Jan 30 12:08:37 Unraid kernel: r8169 0000:04:00.0 eth0: Link is Up - 2.5Gbps/Full - flow control off
Jan 30 12:08:37 Unraid kernel: br0: port 1(eth0) entered blocking state
Jan 30 12:08:37 Unraid kernel: br0: port 1(eth0) entered forwarding state
Jan 30 12:08:54 Unraid kernel: r8169 0000:04:00.0 eth0: Link is Down
Jan 30 12:08:54 Unraid kernel: br0: port 1(eth0) entered disabled state
Jan 30 12:08:58 Unraid kernel: r8169 0000:04:00.0 eth0: Link is Up - 2.5Gbps/Full - flow control off
Jan 30 12:08:58 Unraid kernel: br0: port 1(eth0) entered blocking state
Jan 30 12:08:58 Unraid kernel: br0: port 1(eth0) entered forwarding state
Jan 30 12:36:22 Unraid kernel: r8169 0000:04:00.0 eth0: Link is Down
Jan 30 12:36:22 Unraid kernel: br0: port 1(eth0) entered disabled state
Jan 30 12:36:26 Unraid kernel: r8169 0000:04:00.0 eth0: Link is Up - 2.5Gbps/Full - flow control off
Jan 30 12:36:26 Unraid kernel: br0: port 1(eth0) entered blocking state
Jan 30 12:36:26 Unraid kernel: br0: port 1(eth0) entered forwarding state

 

Also started noticing these periodically.

Jan 31 11:39:59 Unraid kernel: usb 1-2-port4: disabled by hub (EMI?), re-enabling...
Jan 31 11:39:59 Unraid kernel: usb 1-2.4: USB disconnect, device number 9
Jan 31 11:40:00 Unraid kernel: usb 1-2.4: new full-speed USB device number 10 using xhci_hcd
Jan 31 11:40:00 Unraid kernel: hid-generic 0003:0764:0501.0008: hiddev97,hidraw1: USB HID v1.10 Device [CPS CP1000PFCLCD] on usb-0000:00:14.0-2.4/input0
Jan 31 11:40:05 Unraid apcupsd[3034]: Communications with UPS restored.

 

So as a result I've pulled the Cyberpower UPS and replaced it with an APC. That's the only variable that hasn't changed throughout this issue. Figured I might as well test the theory.