Jump to content

Unraid not reachable but server still running

Jupiler

By Jupiler
in General Support

Recommended Posts

Jupiler Noob

Jupiler

Posted
Posted (edited)

I've been battling with an odd issue for the past few days, every so often Unraid would become totally unresponsive, usually the exact moment I'm doing something on the GUI, either loading the docker page of Unraid, or accessing a docker container webui (none in particular).

 

Sometimes I would still be able to ping the Unraid box on the network, but not able to SSH into it (SSH would just hang).

 

This time around the Unraid box became unresponsive but I'm also not able to ping it. I plugged a monitor and keyboard on the box itself and I'm still able to login directly. I've tried to troubleshoot from the console while I'm able to access it, but I'm not too sure what I should be looking for. Thinking of it being a network issue possibly I started checking some of the network interfaces, however when I typed `ifconfig bond0` it became unresponsive as well. Then only way out was to do a CTRL ALT DEL, which triggered a shutdown, however the shutdown process has been stuck at "Starting diagnostic collection" for the past couple of hours. (see image) 


Server's uptime before the issues started was 7 months without problems, no hardware changes were made recently.
 

Any idea on what I should do next to troubleshoot this issue?

I've also been running a syslog server to get a more permanent record of the syslog, see below (server became unresponsive at 17:00:00 or so):
  

Aug 14 14:30:10 BorgCube emhttpd: spinning down /dev/sdp
Aug 14 14:46:09 BorgCube emhttpd: spinning down /dev/sdf
Aug 14 14:47:16 BorgCube emhttpd: spinning down /dev/sdo
Aug 14 14:47:16 BorgCube emhttpd: spinning down /dev/sdi
Aug 14 14:54:02 BorgCube emhttpd: read SMART /dev/sdi
Aug 14 14:54:13 BorgCube emhttpd: read SMART /dev/sdp
Aug 14 14:54:44 BorgCube emhttpd: read SMART /dev/sdo
Aug 14 15:09:55 BorgCube emhttpd: read SMART /dev/sda
Aug 14 15:26:55 BorgCube emhttpd: spinning down /dev/sdo
Aug 14 15:33:16 BorgCube emhttpd: read SMART /dev/sdo
Aug 14 15:45:15 BorgCube emhttpd: spinning down /dev/sda
Aug 14 15:47:38 BorgCube emhttpd: spinning down /dev/sdp
Aug 14 15:52:59 BorgCube emhttpd: read SMART /dev/sdf
Aug 14 16:00:06 BorgCube emhttpd: read SMART /dev/sdp
Aug 14 16:00:44 BorgCube Parity Check Tuning: Send notification: mover running:   (type=normal link=/Settings/Scheduler)
Aug 14 16:18:19 BorgCube Parity Check Tuning: Send notification: mover no longer running:   (type=normal link=/Settings/Scheduler)
Aug 14 16:43:36 BorgCube emhttpd: spinning down /dev/sdp
Aug 14 16:45:58 BorgCube emhttpd: spinning down /dev/sdf
Aug 14 16:47:10 BorgCube emhttpd: spinning down /dev/sdo
Aug 14 16:47:10 BorgCube emhttpd: spinning down /dev/sdi
Aug 14 16:59:07 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 16:59:14 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 16:59:19 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 16:59:23 BorgCube kernel: rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 3-.... } 21145 jiffies s: 74473 root: 0x8/.
Aug 14 16:59:23 BorgCube kernel: rcu: blocking rcu_node structures (internal RCU debug):
Aug 14 16:59:23 BorgCube kernel: Sending NMI from CPU 7 to CPUs 3:
Aug 14 16:59:23 BorgCube kernel: NMI backtrace for cpu 3
Aug 14 16:59:23 BorgCube kernel: CPU: 3 PID: 17455 Comm: kworker/3:2 Tainted: P        W  O       6.1.99-Unraid #1
Aug 14 16:59:23 BorgCube kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z390M Pro4, BIOS P4.30 12/03/2019
Aug 14 16:59:23 BorgCube kernel: Workqueue: wg-crypt-wg0 wg_packet_tx_worker [wireguard]
Aug 14 16:59:23 BorgCube kernel: RIP: 0010:preempt_count_sub+0x85/0x93
Aug 14 16:59:23 BorgCube kernel: Code: d5 39 00 85 c0 74 2c 83 3d f3 73 31 01 00 75 23 48 c7 c6 f2 5b 0c 82 48 c7 c7 f8 39 0b 82 e8 2e 76 fd ff 0f 0b c3 cc cc cc cc <f7> df 65 01 3d e6 38 f8 7e c3 cc cc cc cc 0f 1f 44 00 00 85 ff 74
Aug 14 16:59:23 BorgCube kernel: RSP: 0018:ffffc900001d8ea8 EFLAGS: 00000202
Aug 14 16:59:23 BorgCube kernel: RAX: 0000000000000102 RBX: 0000000000000246 RCX: ffff888103c98c40
Aug 14 16:59:23 BorgCube kernel: RDX: 0000000000000102 RSI: 0000000000000246 RDI: 0000000000000001
Aug 14 16:59:23 BorgCube kernel: RBP: ffffc900001d8f70 R08: 0000000000000000 R09: 0000000000000082
Aug 14 16:59:23 BorgCube kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff888103c98da0
Aug 14 16:59:23 BorgCube kernel: R13: ffff888101143c00 R14: ffff888103c98000 R15: 0000000000001296
Aug 14 16:59:23 BorgCube kernel: FS:  0000000000000000(0000) GS:ffff88885dcc0000(0000) knlGS:0000000000000000
Aug 14 16:59:23 BorgCube kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 14 16:59:23 BorgCube kernel: CR2: 0000153e416f7000 CR3: 000000000220a006 CR4: 00000000003706e0
Aug 14 16:59:23 BorgCube kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 14 16:59:23 BorgCube kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug 14 16:59:23 BorgCube kernel: Call Trace:
Aug 14 16:59:23 BorgCube kernel: <NMI>
Aug 14 16:59:23 BorgCube kernel: ? nmi_cpu_backtrace+0xd3/0x104
Aug 14 16:59:23 BorgCube kernel: ? nmi_cpu_backtrace_handler+0xd/0x15
Aug 14 16:59:23 BorgCube kernel: ? nmi_handle+0x54/0x131
Aug 14 16:59:23 BorgCube kernel: ? preempt_count_sub+0x85/0x93
Aug 14 16:59:23 BorgCube kernel: ? default_do_nmi+0x66/0x15b
Aug 14 16:59:23 BorgCube kernel: ? exc_nmi+0xbf/0x11c
Aug 14 16:59:23 BorgCube kernel: ? end_repeat_nmi+0x16/0x67
Aug 14 16:59:23 BorgCube kernel: ? preempt_count_sub+0x85/0x93
Aug 14 16:59:23 BorgCube kernel: ? preempt_count_sub+0x85/0x93
Aug 14 16:59:23 BorgCube kernel: ? preempt_count_sub+0x85/0x93
Aug 14 16:59:23 BorgCube kernel: </NMI>
Aug 14 16:59:23 BorgCube kernel: <IRQ>
Aug 14 16:59:23 BorgCube kernel: _raw_spin_unlock_irqrestore+0x24/0x3a
Aug 14 16:59:23 BorgCube kernel: bottom_half+0x473/0x8ef [r8152]
Aug 14 16:59:23 BorgCube kernel: ? _raw_spin_unlock_irq+0x1a/0x2f
Aug 14 16:59:23 BorgCube kernel: tasklet_action_common.constprop.0+0x60/0xa6
Aug 14 16:59:23 BorgCube kernel: handle_softirqs+0x129/0x271
Aug 14 16:59:23 BorgCube kernel: do_softirq+0x7f/0xab
Aug 14 16:59:23 BorgCube kernel: </IRQ>
Aug 14 16:59:23 BorgCube kernel: <TASK>
Aug 14 16:59:23 BorgCube kernel: __local_bh_enable_ip+0x4c/0x6b
Aug 14 16:59:23 BorgCube kernel: wg_socket_send_skb_to_peer+0x9c/0xab [wireguard]
Aug 14 16:59:23 BorgCube kernel: wg_packet_tx_worker+0xa6/0x18a [wireguard]
Aug 14 16:59:23 BorgCube kernel: process_one_work+0x1a8/0x295
Aug 14 16:59:23 BorgCube kernel: worker_thread+0x18b/0x244
Aug 14 16:59:23 BorgCube kernel: ? rescuer_thread+0x281/0x281
Aug 14 16:59:23 BorgCube kernel: kthread+0xe4/0xef
Aug 14 16:59:23 BorgCube kernel: ? kthread_complete_and_exit+0x1b/0x1b
Aug 14 16:59:23 BorgCube kernel: ret_from_fork+0x1f/0x30
Aug 14 16:59:23 BorgCube kernel: </TASK>
Aug 14 16:59:24 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 16:59:29 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 16:59:36 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 16:59:41 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 16:59:46 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 16:59:51 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 16:59:58 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 16:59:58 BorgCube kernel: rcu: INFO: rcu_preempt self-detected stall on CPU
Aug 14 16:59:58 BorgCube kernel: rcu: #0113-....: (1 GPs behind) idle=ae14/1/0x4000000000000000 softirq=20606672/20606673 fqs=26324
Aug 14 16:59:58 BorgCube kernel: #011(t=60000 jiffies g=37415333 q=1118808 ncpus=8)
Aug 14 16:59:58 BorgCube kernel: CPU: 3 PID: 17455 Comm: kworker/3:2 Tainted: P        W  O       6.1.99-Unraid #1
Aug 14 16:59:58 BorgCube kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z390M Pro4, BIOS P4.30 12/03/2019
Aug 14 16:59:58 BorgCube kernel: Workqueue: wg-crypt-wg0 wg_packet_tx_worker [wireguard]
Aug 14 16:59:58 BorgCube kernel: RIP: 0010:sg_init_table+0xb/0x2a
Aug 14 16:59:58 BorgCube kernel: Code: 04 25 28 00 00 00 74 05 e8 0b 38 44 00 48 83 c4 50 48 89 d8 5b 5d 41 5c 41 5d c3 cc cc cc cc 89 f1 31 c0 48 89 fa 48 c1 e1 05 <f3> aa 8d 46 ff 48 c1 e0 05 48 01 c2 48 8b 02 48 83 e0 fe 48 83 c8
Aug 14 16:59:58 BorgCube kernel: RSP: 0018:ffffc900001d8eb8 EFLAGS: 00010202
Aug 14 16:59:58 BorgCube kernel: RAX: 0000000000000000 RBX: ffff888103c98ab0 RCX: 0000000000000160
Aug 14 16:59:58 BorgCube kernel: RDX: ffff888104640000 RSI: 0000000000000011 RDI: ffff8881046400c0
Aug 14 16:59:58 BorgCube kernel: RBP: ffffc900001d8f70 R08: 0000000000000000 R09: 0000000000000082
Aug 14 16:59:58 BorgCube kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff888103c98da0
Aug 14 16:59:58 BorgCube kernel: R13: ffff888101143c00 R14: ffff888103c98000 R15: 0000000000001296
Aug 14 16:59:58 BorgCube kernel: FS:  0000000000000000(0000) GS:ffff88885dcc0000(0000) knlGS:0000000000000000
Aug 14 16:59:58 BorgCube kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 14 16:59:58 BorgCube kernel: CR2: 0000153e416f7000 CR3: 000000000220a006 CR4: 00000000003706e0
Aug 14 16:59:58 BorgCube kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 14 16:59:58 BorgCube kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug 14 16:59:58 BorgCube kernel: Call Trace:
Aug 14 16:59:58 BorgCube kernel: <IRQ>
Aug 14 16:59:58 BorgCube kernel: ? rcu_dump_cpu_stacks+0x95/0xb9
Aug 14 16:59:58 BorgCube kernel: ? rcu_sched_clock_irq+0x345/0xa48
Aug 14 16:59:58 BorgCube kernel: ? tick_init_jiffy_update+0x7c/0x7c
Aug 14 16:59:58 BorgCube kernel: ? update_process_times+0x62/0x81
Aug 14 16:59:58 BorgCube kernel: ? tick_sched_timer+0x43/0x71
Aug 14 16:59:58 BorgCube kernel: ? __hrtimer_run_queues+0xe8/0x190
Aug 14 16:59:58 BorgCube kernel: ? hrtimer_interrupt+0x9c/0x16e
Aug 14 16:59:58 BorgCube kernel: ? __sysvec_apic_timer_interrupt+0xc2/0x12f
Aug 14 16:59:58 BorgCube kernel: ? sysvec_apic_timer_interrupt+0x27/0xa6
Aug 14 16:59:58 BorgCube kernel: ? asm_sysvec_apic_timer_interrupt+0x16/0x20
Aug 14 16:59:58 BorgCube kernel: ? sg_init_table+0xb/0x2a
Aug 14 16:59:58 BorgCube kernel: ? _raw_spin_unlock+0x14/0x29
Aug 14 16:59:58 BorgCube kernel: bottom_half+0x18e/0x8ef [r8152]
Aug 14 16:59:58 BorgCube kernel: ? _raw_spin_unlock_irq+0x1a/0x2f
Aug 14 16:59:58 BorgCube kernel: tasklet_action_common.constprop.0+0x60/0xa6
Aug 14 16:59:58 BorgCube kernel: handle_softirqs+0x129/0x271
Aug 14 16:59:58 BorgCube kernel: do_softirq+0x7f/0xab
Aug 14 16:59:58 BorgCube kernel: </IRQ>
Aug 14 16:59:58 BorgCube kernel: <TASK>
Aug 14 16:59:58 BorgCube kernel: __local_bh_enable_ip+0x4c/0x6b
Aug 14 16:59:58 BorgCube kernel: wg_socket_send_skb_to_peer+0x9c/0xab [wireguard]
Aug 14 16:59:58 BorgCube kernel: wg_packet_tx_worker+0xa6/0x18a [wireguard]
Aug 14 16:59:58 BorgCube kernel: process_one_work+0x1a8/0x295
Aug 14 16:59:58 BorgCube kernel: worker_thread+0x18b/0x244
Aug 14 16:59:58 BorgCube kernel: ? rescuer_thread+0x281/0x281
Aug 14 16:59:58 BorgCube kernel: kthread+0xe4/0xef
Aug 14 16:59:58 BorgCube kernel: ? kthread_complete_and_exit+0x1b/0x1b
Aug 14 16:59:58 BorgCube kernel: ret_from_fork+0x1f/0x30
Aug 14 16:59:58 BorgCube kernel: </TASK>
Aug 14 17:00:03 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:00:08 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:00:13 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:00:20 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:00:25 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:00:29 BorgCube kernel: rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 3-.... } 87705 jiffies s: 74473 root: 0x8/.
Aug 14 17:00:29 BorgCube kernel: rcu: blocking rcu_node structures (internal RCU debug):
Aug 14 17:00:29 BorgCube kernel: Sending NMI from CPU 7 to CPUs 3:
Aug 14 17:00:29 BorgCube kernel: NMI backtrace for cpu 3
Aug 14 17:00:29 BorgCube kernel: CPU: 3 PID: 17455 Comm: kworker/3:2 Tainted: P        W  O       6.1.99-Unraid #1
Aug 14 17:00:29 BorgCube kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z390M Pro4, BIOS P4.30 12/03/2019
Aug 14 17:00:29 BorgCube kernel: Workqueue: wg-crypt-wg0 wg_packet_tx_worker [wireguard]
Aug 14 17:00:29 BorgCube kernel: RIP: 0010:bottom_half+0xc1/0x8ef [r8152]
Aug 14 17:00:29 BorgCube kernel: Code: b0 fe ff ff 0f 84 26 08 00 00 49 8d 84 24 a0 fe ff ff 48 89 44 24 10 49 8b 84 24 a0 fe ff ff 48 39 44 24 10 0f 84 06 08 00 00 <49> 8d 84 24 e4 fe ff ff 31 db 48 89 c7 48 89 44 24 18 e8 d0 18 4e
Aug 14 17:00:29 BorgCube kernel: RSP: 0018:ffffc900001d8ec0 EFLAGS: 00000212
Aug 14 17:00:29 BorgCube kernel: RAX: ffff888103c98b08 RBX: ffff888103c98ab0 RCX: ffff888103c98c40
Aug 14 17:00:29 BorgCube kernel: RDX: 0000000000000102 RSI: 0000000000000246 RDI: 00000000ffffffff
Aug 14 17:00:29 BorgCube kernel: RBP: ffffc900001d8f70 R08: 0000000000000000 R09: 0000000000000082
Aug 14 17:00:29 BorgCube kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff888103c98da0
Aug 14 17:00:29 BorgCube kernel: R13: ffff888101143c00 R14: ffff888103c98000 R15: 0000000000001296
Aug 14 17:00:29 BorgCube kernel: FS:  0000000000000000(0000) GS:ffff88885dcc0000(0000) knlGS:0000000000000000
Aug 14 17:00:29 BorgCube kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 14 17:00:29 BorgCube kernel: CR2: 0000153e416f7000 CR3: 000000000220a006 CR4: 00000000003706e0
Aug 14 17:00:29 BorgCube kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 14 17:00:29 BorgCube kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug 14 17:00:29 BorgCube kernel: Call Trace:
Aug 14 17:00:29 BorgCube kernel: <NMI>
Aug 14 17:00:29 BorgCube kernel: ? nmi_cpu_backtrace+0xd3/0x104
Aug 14 17:00:29 BorgCube kernel: ? nmi_cpu_backtrace_handler+0xd/0x15
Aug 14 17:00:29 BorgCube kernel: ? nmi_handle+0x54/0x131
Aug 14 17:00:29 BorgCube kernel: ? bottom_half+0xc1/0x8ef [r8152]
Aug 14 17:00:29 BorgCube kernel: ? default_do_nmi+0x66/0x15b
Aug 14 17:00:29 BorgCube kernel: ? exc_nmi+0xbf/0x11c
Aug 14 17:00:29 BorgCube kernel: ? end_repeat_nmi+0x16/0x67
Aug 14 17:00:29 BorgCube kernel: ? bottom_half+0xc1/0x8ef [r8152]
Aug 14 17:00:29 BorgCube kernel: ? bottom_half+0xc1/0x8ef [r8152]
Aug 14 17:00:29 BorgCube kernel: ? bottom_half+0xc1/0x8ef [r8152]
Aug 14 17:00:29 BorgCube kernel: </NMI>
Aug 14 17:00:29 BorgCube kernel: <IRQ>
Aug 14 17:00:29 BorgCube kernel: ? _raw_spin_unlock_irq+0x1a/0x2f
Aug 14 17:00:29 BorgCube kernel: tasklet_action_common.constprop.0+0x60/0xa6
Aug 14 17:00:29 BorgCube kernel: handle_softirqs+0x129/0x271
Aug 14 17:00:29 BorgCube kernel: do_softirq+0x7f/0xab
Aug 14 17:00:29 BorgCube kernel: </IRQ>
Aug 14 17:00:29 BorgCube kernel: <TASK>
Aug 14 17:00:29 BorgCube kernel: __local_bh_enable_ip+0x4c/0x6b
Aug 14 17:00:29 BorgCube kernel: wg_socket_send_skb_to_peer+0x9c/0xab [wireguard]
Aug 14 17:00:29 BorgCube kernel: wg_packet_tx_worker+0xa6/0x18a [wireguard]
Aug 14 17:00:29 BorgCube kernel: process_one_work+0x1a8/0x295
Aug 14 17:00:29 BorgCube kernel: worker_thread+0x18b/0x244
Aug 14 17:00:29 BorgCube kernel: ? rescuer_thread+0x281/0x281
Aug 14 17:00:29 BorgCube kernel: kthread+0xe4/0xef
Aug 14 17:00:29 BorgCube kernel: ? kthread_complete_and_exit+0x1b/0x1b
Aug 14 17:00:29 BorgCube kernel: ret_from_fork+0x1f/0x30
Aug 14 17:00:29 BorgCube kernel: </TASK>
Aug 14 17:00:30 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:00:35 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:00:42 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:00:47 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:00:52 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:00:57 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:01:04 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:01:09 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:01:14 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:01:20 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:01:26 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:01:31 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:01:35 BorgCube kernel: rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 3-.... } 153241 jiffies s: 74473 root: 0x8/.
Aug 14 17:01:35 BorgCube kernel: rcu: blocking rcu_node structures (internal RCU debug):
Aug 14 17:01:35 BorgCube kernel: Sending NMI from CPU 7 to CPUs 3:
Aug 14 17:01:35 BorgCube kernel: NMI backtrace for cpu 3
Aug 14 17:01:35 BorgCube kernel: CPU: 3 PID: 17455 Comm: kworker/3:2 Tainted: P        W  O       6.1.99-Unraid #1
Aug 14 17:01:35 BorgCube kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z390M Pro4, BIOS P4.30 12/03/2019
Aug 14 17:01:35 BorgCube kernel: Workqueue: wg-crypt-wg0 wg_packet_tx_worker [wireguard]
Aug 14 17:01:35 BorgCube kernel: RIP: 0010:netif_unfreeze_queues+0x24/0x3c
Aug 14 17:01:35 BorgCube kernel: Code: 01 c3 cc cc cc cc 0f 1f 44 00 00 55 31 ed 53 48 89 fb 3b ab 88 03 00 00 73 21 89 ef 48 69 ff 40 01 00 00 48 03 bb 80 03 00 00 <f0> 80 a7 90 00 00 00 fb e8 8e 75 fb ff ff c5 eb d7 5b 5d c3 cc cc
Aug 14 17:01:35 BorgCube kernel: RSP: 0018:ffffc900001d8e98 EFLAGS: 00000286
Aug 14 17:01:35 BorgCube kernel: RAX: 0000000000000000 RBX: ffff888103c98000 RCX: ffff888146d564e0
Aug 14 17:01:35 BorgCube kernel: RDX: 0000000000000103 RSI: ffffffff820d96e1 RDI: ffff888101143c00
Aug 14 17:01:35 BorgCube kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000082
Aug 14 17:01:35 BorgCube kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff888103c98da0
Aug 14 17:01:35 BorgCube kernel: R13: ffff888101143c00 R14: ffff888103c98000 R15: 0000000000001296
Aug 14 17:01:35 BorgCube kernel: FS:  0000000000000000(0000) GS:ffff88885dcc0000(0000) knlGS:0000000000000000
Aug 14 17:01:35 BorgCube kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 14 17:01:35 BorgCube kernel: CR2: 0000153e416f7000 CR3: 000000000220a006 CR4: 00000000003706e0
Aug 14 17:01:35 BorgCube kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 14 17:01:35 BorgCube kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug 14 17:01:35 BorgCube kernel: Call Trace:
Aug 14 17:01:35 BorgCube kernel: <NMI>
Aug 14 17:01:35 BorgCube kernel: ? nmi_cpu_backtrace+0xd3/0x104
Aug 14 17:01:35 BorgCube kernel: ? nmi_cpu_backtrace_handler+0xd/0x15
Aug 14 17:01:35 BorgCube kernel: ? nmi_handle+0x54/0x131
Aug 14 17:01:35 BorgCube kernel: ? netif_unfreeze_queues+0x24/0x3c
Aug 14 17:01:35 BorgCube kernel: ? default_do_nmi+0x66/0x15b
Aug 14 17:01:35 BorgCube kernel: ? exc_nmi+0xbf/0x11c
Aug 14 17:01:35 BorgCube kernel: ? end_repeat_nmi+0x16/0x67
Aug 14 17:01:35 BorgCube kernel: ? netif_unfreeze_queues+0x24/0x3c
Aug 14 17:01:35 BorgCube kernel: ? netif_unfreeze_queues+0x24/0x3c
Aug 14 17:01:35 BorgCube kernel: ? netif_unfreeze_queues+0x24/0x3c
Aug 14 17:01:35 BorgCube kernel: </NMI>
Aug 14 17:01:35 BorgCube kernel: <IRQ>
Aug 14 17:01:35 BorgCube kernel: netif_tx_unlock+0xe/0x1b
Aug 14 17:01:35 BorgCube kernel: bottom_half+0x436/0x8ef [r8152]
Aug 14 17:01:35 BorgCube kernel: ? _raw_spin_unlock_irq+0x1a/0x2f
Aug 14 17:01:35 BorgCube kernel: tasklet_action_common.constprop.0+0x60/0xa6
Aug 14 17:01:35 BorgCube kernel: handle_softirqs+0x129/0x271
Aug 14 17:01:35 BorgCube kernel: do_softirq+0x7f/0xab
Aug 14 17:01:35 BorgCube kernel: </IRQ>
Aug 14 17:01:35 BorgCube kernel: <TASK>
Aug 14 17:01:35 BorgCube kernel: __local_bh_enable_ip+0x4c/0x6b
Aug 14 17:01:35 BorgCube kernel: wg_socket_send_skb_to_peer+0x9c/0xab [wireguard]
Aug 14 17:01:35 BorgCube kernel: wg_packet_tx_worker+0xa6/0x18a [wireguard]
Aug 14 17:01:35 BorgCube kernel: process_one_work+0x1a8/0x295
Aug 14 17:01:35 BorgCube kernel: worker_thread+0x18b/0x244
Aug 14 17:01:35 BorgCube kernel: ? rescuer_thread+0x281/0x281
Aug 14 17:01:35 BorgCube kernel: kthread+0xe4/0xef
Aug 14 17:01:35 BorgCube kernel: ? kthread_complete_and_exit+0x1b/0x1b
Aug 14 17:01:35 BorgCube kernel: ret_from_fork+0x1f/0x30
Aug 14 17:01:35 BorgCube kernel: </TASK>
Aug 14 17:01:36 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:01:42 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:01:48 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:01:53 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:01:58 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:02:05 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:02:10 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:02:15 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:02:20 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:02:27 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:02:32 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:02:37 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:02:40 BorgCube kernel: rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 3-.... } 218777 jiffies s: 74473 root: 0x8/.
Aug 14 17:02:40 BorgCube kernel: rcu: blocking rcu_node structures (internal RCU debug):
Aug 14 17:02:40 BorgCube kernel: Sending NMI from CPU 7 to CPUs 3:
Aug 14 17:02:40 BorgCube kernel: NMI backtrace for cpu 3
Aug 14 17:02:40 BorgCube kernel: CPU: 3 PID: 17455 Comm: kworker/3:2 Tainted: P        W  O       6.1.99-Unraid #1
Aug 14 17:02:40 BorgCube kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z390M Pro4, BIOS P4.30 12/03/2019
Aug 14 17:02:40 BorgCube kernel: Workqueue: wg-crypt-wg0 wg_packet_tx_worker [wireguard]
Aug 14 17:02:40 BorgCube kernel: RIP: 0010:netif_freeze_queues+0x53/0x77
Aug 14 17:02:40 BorgCube kernel: Code: db 40 01 00 00 48 03 9d 80 03 00 00 4c 8d b3 80 00 00 00 4c 89 f7 e8 78 a9 14 00 44 89 ab 84 00 00 00 f0 80 8b 90 00 00 00 04 <c7> 83 84 00 00 00 ff ff ff ff 4c 89 f7 41 ff c4 e8 2d aa 14 00 eb
Aug 14 17:02:40 BorgCube kernel: RSP: 0018:ffffc900001d8e90 EFLAGS: 00000206
Aug 14 17:02:40 BorgCube kernel: RAX: 0000000000000000 RBX: ffff888101143c00 RCX: ffff888146d564e0
Aug 14 17:02:40 BorgCube kernel: RDX: 0000000000000001 RSI: ffffffff820d96e1 RDI: ffff888101143c80
Aug 14 17:02:40 BorgCube kernel: RBP: ffff888103c98000 R08: 0000000000000000 R09: 0000000000000082
Aug 14 17:02:40 BorgCube kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Aug 14 17:02:40 BorgCube kernel: R13: 0000000000000003 R14: ffff888101143c80 R15: 0000000000001296
Aug 14 17:02:40 BorgCube kernel: FS:  0000000000000000(0000) GS:ffff88885dcc0000(0000) knlGS:0000000000000000
Aug 14 17:02:40 BorgCube kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 14 17:02:40 BorgCube kernel: CR2: 0000153e416f7000 CR3: 000000000220a006 CR4: 00000000003706e0
Aug 14 17:02:40 BorgCube kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 14 17:02:40 BorgCube kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug 14 17:02:40 BorgCube kernel: Call Trace:
Aug 14 17:02:40 BorgCube kernel: <NMI>
Aug 14 17:02:40 BorgCube kernel: ? nmi_cpu_backtrace+0xd3/0x104
Aug 14 17:02:40 BorgCube kernel: ? nmi_cpu_backtrace_handler+0xd/0x15
Aug 14 17:02:40 BorgCube kernel: ? nmi_handle+0x54/0x131
Aug 14 17:02:40 BorgCube kernel: ? netif_freeze_queues+0x53/0x77
Aug 14 17:02:40 BorgCube kernel: ? default_do_nmi+0x66/0x15b
Aug 14 17:02:40 BorgCube kernel: ? exc_nmi+0xbf/0x11c
Aug 14 17:02:40 BorgCube kernel: ? end_repeat_nmi+0x16/0x67
Aug 14 17:02:40 BorgCube kernel: ? netif_freeze_queues+0x53/0x77
Aug 14 17:02:40 BorgCube kernel: ? netif_freeze_queues+0x53/0x77
Aug 14 17:02:40 BorgCube kernel: ? netif_freeze_queues+0x53/0x77
Aug 14 17:02:40 BorgCube kernel: </NMI>
Aug 14 17:02:40 BorgCube kernel: <IRQ>
Aug 14 17:02:40 BorgCube kernel: bottom_half+0x401/0x8ef [r8152]
Aug 14 17:02:40 BorgCube kernel: ? _raw_spin_unlock_irq+0x1a/0x2f
Aug 14 17:02:40 BorgCube kernel: tasklet_action_common.constprop.0+0x60/0xa6
Aug 14 17:02:40 BorgCube kernel: handle_softirqs+0x129/0x271
Aug 14 17:02:40 BorgCube kernel: do_softirq+0x7f/0xab
Aug 14 17:02:40 BorgCube kernel: </IRQ>
Aug 14 17:02:40 BorgCube kernel: <TASK>
Aug 14 17:02:40 BorgCube kernel: __local_bh_enable_ip+0x4c/0x6b
Aug 14 17:02:40 BorgCube kernel: wg_socket_send_skb_to_peer+0x9c/0xab [wireguard]
Aug 14 17:02:40 BorgCube kernel: wg_packet_tx_worker+0xa6/0x18a [wireguard]
Aug 14 17:02:40 BorgCube kernel: process_one_work+0x1a8/0x295
Aug 14 17:02:40 BorgCube kernel: worker_thread+0x18b/0x244
Aug 14 17:02:40 BorgCube kernel: ? rescuer_thread+0x281/0x281
Aug 14 17:02:40 BorgCube kernel: kthread+0xe4/0xef
Aug 14 17:02:40 BorgCube kernel: ? kthread_complete_and_exit+0x1b/0x1b
Aug 14 17:02:40 BorgCube kernel: ret_from_fork+0x1f/0x30
Aug 14 17:02:40 BorgCube kernel: </TASK>
Aug 14 17:02:42 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:02:49 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:02:54 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:02:58 BorgCube kernel: rcu: INFO: rcu_preempt self-detected stall on CPU
Aug 14 17:02:58 BorgCube kernel: rcu: #0113-....: (1 GPs behind) idle=ae14/1/0x4000000000000000 softirq=20606672/20606673 fqs=107458
Aug 14 17:02:58 BorgCube kernel: #011(t=240005 jiffies g=37415333 q=4196028 ncpus=8)
Aug 14 17:02:58 BorgCube kernel: CPU: 3 PID: 17455 Comm: kworker/3:2 Tainted: P        W  O       6.1.99-Unraid #1
Aug 14 17:02:58 BorgCube kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z390M Pro4, BIOS P4.30 12/03/2019
Aug 14 17:02:58 BorgCube kernel: Workqueue: wg-crypt-wg0 wg_packet_tx_worker [wireguard]
Aug 14 17:02:58 BorgCube kernel: RIP: 0010:_raw_spin_unlock_irqrestore+0x1a/0x3a
Aug 14 17:02:58 BorgCube kernel: Code: 79 7e 85 c0 75 05 0f 1f 44 00 00 c3 cc cc cc cc 0f 1f 44 00 00 53 48 89 f3 e8 7a 54 83 ff 0f ba e3 09 73 06 fb 0f 1f 44 00 00 <bf> 01 00 00 00 e8 e7 15 81 ff 65 8b 05 54 4f 79 7e 85 c0 75 05 0f
Aug 14 17:02:58 BorgCube kernel: RSP: 0018:ffffc900001d8eb0 EFLAGS: 00000247
Aug 14 17:02:58 BorgCube kernel: RAX: ffff888103c98bb8 RBX: 0000000000000246 RCX: ffff888103c98c40
Aug 14 17:02:58 BorgCube kernel: RDX: 0000000000000001 RSI: 0000000000000246 RDI: ffff888103c98c84
Aug 14 17:02:58 BorgCube kernel: RBP: ffffc900001d8f70 R08: 0000000000000000 R09: 0000000000000082
Aug 14 17:02:58 BorgCube kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff888103c98da0
Aug 14 17:02:58 BorgCube kernel: R13: ffff888101143c00 R14: ffff888103c98000 R15: 0000000000001296
Aug 14 17:02:58 BorgCube kernel: FS:  0000000000000000(0000) GS:ffff88885dcc0000(0000) knlGS:0000000000000000
Aug 14 17:02:58 BorgCube kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 14 17:02:58 BorgCube kernel: CR2: 0000153e416f7000 CR3: 000000000220a006 CR4: 00000000003706e0
Aug 14 17:02:58 BorgCube kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 14 17:02:58 BorgCube kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug 14 17:02:58 BorgCube kernel: Call Trace:
Aug 14 17:02:58 BorgCube kernel: <IRQ>
Aug 14 17:02:58 BorgCube kernel: ? rcu_dump_cpu_stacks+0x95/0xb9
Aug 14 17:02:58 BorgCube kernel: ? rcu_sched_clock_irq+0x345/0xa48
Aug 14 17:02:58 BorgCube kernel: ? sched_slice+0xaf/0x101
Aug 14 17:02:58 BorgCube kernel: ? tick_init_jiffy_update+0x7c/0x7c
Aug 14 17:02:58 BorgCube kernel: ? update_process_times+0x62/0x81
Aug 14 17:02:58 BorgCube kernel: ? tick_sched_timer+0x43/0x71
Aug 14 17:02:58 BorgCube kernel: ? __hrtimer_run_queues+0xe8/0x190
Aug 14 17:02:58 BorgCube kernel: ? hrtimer_interrupt+0x9c/0x16e
Aug 14 17:02:58 BorgCube kernel: ? __sysvec_apic_timer_interrupt+0xc2/0x12f
Aug 14 17:02:58 BorgCube kernel: ? sysvec_apic_timer_interrupt+0x27/0xa6
Aug 14 17:02:58 BorgCube kernel: ? asm_sysvec_apic_timer_interrupt+0x16/0x20
Aug 14 17:02:58 BorgCube kernel: ? _raw_spin_unlock_irqrestore+0x1a/0x3a
Aug 14 17:02:58 BorgCube kernel: bottom_half+0x473/0x8ef [r8152]
Aug 14 17:02:58 BorgCube kernel: ? _raw_spin_unlock_irq+0x1a/0x2f
Aug 14 17:02:58 BorgCube kernel: tasklet_action_common.constprop.0+0x60/0xa6
Aug 14 17:02:58 BorgCube kernel: handle_softirqs+0x129/0x271
Aug 14 17:02:58 BorgCube kernel: do_softirq+0x7f/0xab
Aug 14 17:02:58 BorgCube kernel: </IRQ>
Aug 14 17:02:58 BorgCube kernel: <TASK>
Aug 14 17:02:58 BorgCube kernel: __local_bh_enable_ip+0x4c/0x6b
Aug 14 17:02:58 BorgCube kernel: wg_socket_send_skb_to_peer+0x9c/0xab [wireguard]
Aug 14 17:02:58 BorgCube kernel: wg_packet_tx_worker+0xa6/0x18a [wireguard]
Aug 14 17:02:58 BorgCube kernel: process_one_work+0x1a8/0x295
Aug 14 17:02:58 BorgCube kernel: worker_thread+0x18b/0x244
Aug 14 17:02:58 BorgCube kernel: ? rescuer_thread+0x281/0x281
Aug 14 17:02:58 BorgCube kernel: kthread+0xe4/0xef
Aug 14 17:02:58 BorgCube kernel: ? kthread_complete_and_exit+0x1b/0x1b
Aug 14 17:02:58 BorgCube kernel: ret_from_fork+0x1f/0x30
Aug 14 17:02:58 BorgCube kernel: </TASK>
Aug 14 17:02:59 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:03:04 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:03:11 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:03:16 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:03:21 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:03:26 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:03:33 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:03:38 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:03:43 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:03:46 BorgCube kernel: rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 3-.... } 284313 jiffies s: 74473 root: 0x8/.
Aug 14 17:03:46 BorgCube kernel: rcu: blocking rcu_node structures (internal RCU debug):
Aug 14 17:03:46 BorgCube kernel: Sending NMI from CPU 7 to CPUs 3:
Aug 14 17:03:46 BorgCube kernel: NMI backtrace for cpu 3
Aug 14 17:03:46 BorgCube kernel: CPU: 3 PID: 17455 Comm: kworker/3:2 Tainted: P        W  O       6.1.99-Unraid #1
Aug 14 17:03:46 BorgCube kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z390M Pro4, BIOS P4.30 12/03/2019
Aug 14 17:03:46 BorgCube kernel: Workqueue: wg-crypt-wg0 wg_packet_tx_worker [wireguard]
Aug 14 17:03:46 BorgCube kernel: RIP: 0010:netif_freeze_queues+0x53/0x77
Aug 14 17:03:46 BorgCube kernel: Code: db 40 01 00 00 48 03 9d 80 03 00 00 4c 8d b3 80 00 00 00 4c 89 f7 e8 78 a9 14 00 44 89 ab 84 00 00 00 f0 80 8b 90 00 00 00 04 <c7> 83 84 00 00 00 ff ff ff ff 4c 89 f7 41 ff c4 e8 2d aa 14 00 eb
Aug 14 17:03:46 BorgCube kernel: RSP: 0018:ffffc900001d8e90 EFLAGS: 00000206
Aug 14 17:03:46 BorgCube kernel: RAX: 0000000000000000 RBX: ffff888101143c00 RCX: ffff888146d564e0
Aug 14 17:03:46 BorgCube kernel: RDX: 0000000000000001 RSI: ffffffff820d96e1 RDI: ffff888101143c80
Aug 14 17:03:46 BorgCube kernel: RBP: ffff888103c98000 R08: 0000000000000000 R09: 0000000000000082
Aug 14 17:03:46 BorgCube kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Aug 14 17:03:46 BorgCube kernel: R13: 0000000000000003 R14: ffff888101143c80 R15: 0000000000001296
Aug 14 17:03:46 BorgCube kernel: FS:  0000000000000000(0000) GS:ffff88885dcc0000(0000) knlGS:0000000000000000
Aug 14 17:03:46 BorgCube kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 14 17:03:46 BorgCube kernel: CR2: 0000153e416f7000 CR3: 000000000220a006 CR4: 00000000003706e0
Aug 14 17:03:46 BorgCube kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 14 17:03:46 BorgCube kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug 14 17:03:46 BorgCube kernel: Call Trace:
Aug 14 17:03:46 BorgCube kernel: <NMI>
Aug 14 17:03:46 BorgCube kernel: ? nmi_cpu_backtrace+0xd3/0x104
Aug 14 17:03:46 BorgCube kernel: ? nmi_cpu_backtrace_handler+0xd/0x15
Aug 14 17:03:46 BorgCube kernel: ? nmi_handle+0x54/0x131
Aug 14 17:03:46 BorgCube kernel: ? netif_freeze_queues+0x53/0x77
Aug 14 17:03:46 BorgCube kernel: ? default_do_nmi+0x66/0x15b
Aug 14 17:03:46 BorgCube kernel: ? exc_nmi+0xbf/0x11c
Aug 14 17:03:46 BorgCube kernel: ? end_repeat_nmi+0x16/0x67
Aug 14 17:03:46 BorgCube kernel: ? netif_freeze_queues+0x53/0x77
Aug 14 17:03:46 BorgCube kernel: ? netif_freeze_queues+0x53/0x77
Aug 14 17:03:46 BorgCube kernel: ? netif_freeze_queues+0x53/0x77
Aug 14 17:03:46 BorgCube kernel: </NMI>
Aug 14 17:03:46 BorgCube kernel: <IRQ>
Aug 14 17:03:46 BorgCube kernel: bottom_half+0x401/0x8ef [r8152]
Aug 14 17:03:46 BorgCube kernel: ? _raw_spin_unlock_irq+0x1a/0x2f
Aug 14 17:03:46 BorgCube kernel: tasklet_action_common.constprop.0+0x60/0xa6
Aug 14 17:03:46 BorgCube kernel: handle_softirqs+0x129/0x271
Aug 14 17:03:46 BorgCube kernel: do_softirq+0x7f/0xab
Aug 14 17:03:46 BorgCube kernel: </IRQ>
Aug 14 17:03:46 BorgCube kernel: <TASK>
Aug 14 17:03:46 BorgCube kernel: __local_bh_enable_ip+0x4c/0x6b
Aug 14 17:03:46 BorgCube kernel: wg_socket_send_skb_to_peer+0x9c/0xab [wireguard]
Aug 14 17:03:46 BorgCube kernel: wg_packet_tx_worker+0xa6/0x18a [wireguard]
Aug 14 17:03:46 BorgCube kernel: process_one_work+0x1a8/0x295
Aug 14 17:03:46 BorgCube kernel: worker_thread+0x18b/0x244
Aug 14 17:03:46 BorgCube kernel: ? rescuer_thread+0x281/0x281
Aug 14 17:03:46 BorgCube kernel: kthread+0xe4/0xef
Aug 14 17:03:46 BorgCube kernel: ? kthread_complete_and_exit+0x1b/0x1b
Aug 14 17:03:46 BorgCube kernel: ret_from_fork+0x1f/0x30
Aug 14 17:03:46 BorgCube kernel: </TASK>
Aug 14 17:03:48 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:03:55 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:04:00 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:04:05 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:04:11 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:04:17 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:04:22 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:04:27 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:04:34 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:04:39 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:04:44 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:04:49 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:04:51 BorgCube kernel: rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 3-.... } 349849 jiffies s: 74473 root: 0x8/.
Aug 14 17:04:51 BorgCube kernel: rcu: blocking rcu_node structures (internal RCU debug):
Aug 14 17:04:51 BorgCube kernel: Sending NMI from CPU 7 to CPUs 3:
Aug 14 17:04:51 BorgCube kernel: NMI backtrace for cpu 3
Aug 14 17:04:51 BorgCube kernel: CPU: 3 PID: 17455 Comm: kworker/3:2 Tainted: P        W  O       6.1.99-Unraid #1
Aug 14 17:04:51 BorgCube kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z390M Pro4, BIOS P4.30 12/03/2019
Aug 14 17:04:51 BorgCube kernel: Workqueue: wg-crypt-wg0 wg_packet_tx_worker [wireguard]
Aug 14 17:04:51 BorgCube kernel: RIP: 0010:_raw_spin_unlock_irqrestore+0x1a/0x3a
Aug 14 17:04:51 BorgCube kernel: Code: 79 7e 85 c0 75 05 0f 1f 44 00 00 c3 cc cc cc cc 0f 1f 44 00 00 53 48 89 f3 e8 7a 54 83 ff 0f ba e3 09 73 06 fb 0f 1f 44 00 00 <bf> 01 00 00 00 e8 e7 15 81 ff 65 8b 05 54 4f 79 7e 85 c0 75 05 0f
Aug 14 17:04:51 BorgCube kernel: RSP: 0018:ffffc900001d8eb0 EFLAGS: 00000247
Aug 14 17:04:51 BorgCube kernel: RAX: ffff888103c98b08 RBX: 0000000000000246 RCX: ffff888103c98c40
Aug 14 17:04:51 BorgCube kernel: RDX: 0000000000000001 RSI: 0000000000000246 RDI: ffff888103c98c84
Aug 14 17:04:51 BorgCube kernel: RBP: ffffc900001d8f70 R08: 0000000000000000 R09: 0000000000000082
Aug 14 17:04:51 BorgCube kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff888103c98da0
Aug 14 17:04:51 BorgCube kernel: R13: ffff888101143c00 R14: ffff888103c98000 R15: 0000000000001296
Aug 14 17:04:51 BorgCube kernel: FS:  0000000000000000(0000) GS:ffff88885dcc0000(0000) knlGS:0000000000000000
Aug 14 17:04:51 BorgCube kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 14 17:04:51 BorgCube kernel: CR2: 0000153e416f7000 CR3: 000000000220a006 CR4: 00000000003706e0
Aug 14 17:04:51 BorgCube kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 14 17:04:51 BorgCube kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug 14 17:04:51 BorgCube kernel: Call Trace:
Aug 14 17:04:51 BorgCube kernel: <NMI>
Aug 14 17:04:51 BorgCube kernel: ? nmi_cpu_backtrace+0xd3/0x104
Aug 14 17:04:51 BorgCube kernel: ? nmi_cpu_backtrace_handler+0xd/0x15
Aug 14 17:04:51 BorgCube kernel: ? nmi_handle+0x54/0x131
Aug 14 17:04:51 BorgCube kernel: ? _raw_spin_unlock_irqrestore+0x1a/0x3a
Aug 14 17:04:51 BorgCube kernel: ? default_do_nmi+0x66/0x15b
Aug 14 17:04:51 BorgCube kernel: ? exc_nmi+0xbf/0x11c
Aug 14 17:04:51 BorgCube kernel: ? end_repeat_nmi+0x16/0x67
Aug 14 17:04:51 BorgCube kernel: ? _raw_spin_unlock_irqrestore+0x1a/0x3a
Aug 14 17:04:51 BorgCube kernel: ? _raw_spin_unlock_irqrestore+0x1a/0x3a
Aug 14 17:04:51 BorgCube kernel: ? _raw_spin_unlock_irqrestore+0x1a/0x3a
Aug 14 17:04:51 BorgCube kernel: </NMI>
Aug 14 17:04:51 BorgCube kernel: <IRQ>
Aug 14 17:04:51 BorgCube kernel: bottom_half+0x473/0x8ef [r8152]
Aug 14 17:04:51 BorgCube kernel: ? _raw_spin_unlock_irq+0x1a/0x2f
Aug 14 17:04:51 BorgCube kernel: tasklet_action_common.constprop.0+0x60/0xa6
Aug 14 17:04:51 BorgCube kernel: handle_softirqs+0x129/0x271
Aug 14 17:04:51 BorgCube kernel: do_softirq+0x7f/0xab
Aug 14 17:04:51 BorgCube kernel: </IRQ>
Aug 14 17:04:51 BorgCube kernel: <TASK>
Aug 14 17:04:51 BorgCube kernel: __local_bh_enable_ip+0x4c/0x6b
Aug 14 17:04:51 BorgCube kernel: wg_socket_send_skb_to_peer+0x9c/0xab [wireguard]
Aug 14 17:04:51 BorgCube kernel: wg_packet_tx_worker+0xa6/0x18a [wireguard]
Aug 14 17:04:51 BorgCube kernel: process_one_work+0x1a8/0x295
Aug 14 17:04:51 BorgCube kernel: worker_thread+0x18b/0x244
Aug 14 17:04:51 BorgCube kernel: ? rescuer_thread+0x281/0x281
Aug 14 17:04:51 BorgCube kernel: kthread+0xe4/0xef
Aug 14 17:04:51 BorgCube kernel: ? kthread_complete_and_exit+0x1b/0x1b
Aug 14 17:04:51 BorgCube kernel: ret_from_fork+0x1f/0x30
Aug 14 17:04:51 BorgCube kernel: </TASK>
Aug 14 17:04:56 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:05:01 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:05:06 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:05:11 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:05:18 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:05:23 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:05:28 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:05:33 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:05:40 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:05:45 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:05:50 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:05:55 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:05:57 BorgCube kernel: rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 3-.... } 415385 jiffies s: 74473 root: 0x8/.
Aug 14 17:05:57 BorgCube kernel: rcu: blocking rcu_node structures (internal RCU debug):
Aug 14 17:05:57 BorgCube kernel: Sending NMI from CPU 7 to CPUs 3:
Aug 14 17:05:57 BorgCube kernel: NMI backtrace for cpu 3
Aug 14 17:05:57 BorgCube kernel: CPU: 3 PID: 17455 Comm: kworker/3:2 Tainted: P        W  O       6.1.99-Unraid #1
Aug 14 17:05:57 BorgCube kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z390M Pro4, BIOS P4.30 12/03/2019
Aug 14 17:05:57 BorgCube kernel: Workqueue: wg-crypt-wg0 wg_packet_tx_worker [wireguard]
Aug 14 17:05:57 BorgCube kernel: RIP: 0010:netif_freeze_queues+0x13/0x77
Aug 14 17:05:57 BorgCube kernel: Code: c3 cc cc cc cc 48 89 f2 48 89 fe 48 c7 c7 20 d1 3a 82 e9 c1 f9 cf ff 0f 1f 44 00 00 41 56 41 55 41 54 45 31 e4 55 48 89 fd 53 <e8> 17 f0 13 00 41 89 c5 44 3b a5 88 03 00 00 73 46 44 89 e3 48 69
Aug 14 17:05:57 BorgCube kernel: RSP: 0018:ffffc900001d8e90 EFLAGS: 00000246
Aug 14 17:05:57 BorgCube kernel: RAX: 0000000000000000 RBX: ffff888103c98ab0 RCX: ffff888146d564e0
Aug 14 17:05:57 BorgCube kernel: RDX: 0000000000000001 RSI: ffff888103c98c50 RDI: ffff888103c98000
Aug 14 17:05:57 BorgCube kernel: RBP: ffff888103c98000 R08: 0000000000000000 R09: 0000000000000082
Aug 14 17:05:57 BorgCube kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Aug 14 17:05:57 BorgCube kernel: R13: ffffc900001d8f20 R14: ffff888103c98000 R15: 0000000000001296
Aug 14 17:05:57 BorgCube kernel: FS:  0000000000000000(0000) GS:ffff88885dcc0000(0000) knlGS:0000000000000000
Aug 14 17:05:57 BorgCube kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 14 17:05:57 BorgCube kernel: CR2: 0000153e416f7000 CR3: 000000000220a006 CR4: 00000000003706e0
Aug 14 17:05:57 BorgCube kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 14 17:05:57 BorgCube kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug 14 17:05:57 BorgCube kernel: Call Trace:
Aug 14 17:05:57 BorgCube kernel: <NMI>
Aug 14 17:05:57 BorgCube kernel: ? nmi_cpu_backtrace+0xd3/0x104
Aug 14 17:05:57 BorgCube kernel: ? nmi_cpu_backtrace_handler+0xd/0x15
Aug 14 17:05:57 BorgCube kernel: ? nmi_handle+0x54/0x131
Aug 14 17:05:57 BorgCube kernel: ? netif_freeze_queues+0x13/0x77
Aug 14 17:05:57 BorgCube kernel: ? default_do_nmi+0x66/0x15b
Aug 14 17:05:57 BorgCube kernel: ? exc_nmi+0xbf/0x11c
Aug 14 17:05:57 BorgCube kernel: ? end_repeat_nmi+0x16/0x67
Aug 14 17:05:57 BorgCube kernel: ? netif_freeze_queues+0x13/0x77
Aug 14 17:05:57 BorgCube kernel: ? netif_freeze_queues+0x13/0x77
Aug 14 17:05:57 BorgCube kernel: ? netif_freeze_queues+0x13/0x77
Aug 14 17:05:57 BorgCube kernel: </NMI>
Aug 14 17:05:57 BorgCube kernel: <IRQ>
Aug 14 17:05:57 BorgCube kernel: bottom_half+0x401/0x8ef [r8152]
Aug 14 17:05:57 BorgCube kernel: ? _raw_spin_unlock_irq+0x1a/0x2f
Aug 14 17:05:57 BorgCube kernel: tasklet_action_common.constprop.0+0x60/0xa6
Aug 14 17:05:57 BorgCube kernel: handle_softirqs+0x129/0x271
Aug 14 17:05:57 BorgCube kernel: do_softirq+0x7f/0xab
Aug 14 17:05:57 BorgCube kernel: </IRQ>
Aug 14 17:05:57 BorgCube kernel: <TASK>
Aug 14 17:05:57 BorgCube kernel: __local_bh_enable_ip+0x4c/0x6b
Aug 14 17:05:57 BorgCube kernel: wg_socket_send_skb_to_peer+0x9c/0xab [wireguard]
Aug 14 17:05:57 BorgCube kernel: wg_packet_tx_worker+0xa6/0x18a [wireguard]
Aug 14 17:05:57 BorgCube kernel: process_one_work+0x1a8/0x295
Aug 14 17:05:57 BorgCube kernel: worker_thread+0x18b/0x244
Aug 14 17:05:57 BorgCube kernel: ? rescuer_thread+0x281/0x281
Aug 14 17:05:57 BorgCube kernel: kthread+0xe4/0xef
Aug 14 17:05:57 BorgCube kernel: ? kthread_complete_and_exit+0x1b/0x1b
Aug 14 17:05:57 BorgCube kernel: ret_from_fork+0x1f/0x30
Aug 14 17:05:57 BorgCube kernel: </TASK>
Aug 14 17:05:58 BorgCube kernel: rcu: INFO: rcu_preempt self-detected stall on CPU
Aug 14 17:05:58 BorgCube kernel: rcu: #0113-....: (1 GPs behind) idle=ae14/1/0x4000000000000000 softirq=20606672/20606673 fqs=186790
Aug 14 17:05:58 BorgCube kernel: #011(t=420008 jiffies g=37415333 q=5508715 ncpus=8)
Aug 14 17:05:58 BorgCube kernel: CPU: 3 PID: 17455 Comm: kworker/3:2 Tainted: P        W  O       6.1.99-Unraid #1
Aug 14 17:05:58 BorgCube kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z390M Pro4, BIOS P4.30 12/03/2019
Aug 14 17:05:58 BorgCube kernel: Workqueue: wg-crypt-wg0 wg_packet_tx_worker [wireguard]
Aug 14 17:05:58 BorgCube kernel: RIP: 0010:preempt_count_sub+0x15/0x93
Aug 14 17:05:58 BorgCube kernel: Code: 77 0e 11 00 0f 1f 44 00 00 48 c7 c7 30 09 95 82 e9 96 0d 11 00 0f 1f 44 00 00 83 3d 80 9d 8b 01 00 75 77 65 8b 05 5f 39 f8 7e <89> c2 81 e2 ff ff ff 7f 39 fa 7d 2c e8 c0 d5 39 00 85 c0 74 64 83
Aug 14 17:05:58 BorgCube kernel: RSP: 0018:ffffc900001d8eb0 EFLAGS: 00000246
Aug 14 17:05:58 BorgCube kernel: RAX: 0000000000000102 RBX: ffff888103c98bb8 RCX: ffff888146d564e0
Aug 14 17:05:58 BorgCube kernel: RDX: ffff8881028a8000 RSI: ffffffff820d96e1 RDI: 0000000000000001
Aug 14 17:05:58 BorgCube kernel: RBP: ffffc900001d8f70 R08: 0000000000000000 R09: 0000000000000082
Aug 14 17:05:58 BorgCube kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff888103c98da0
Aug 14 17:05:58 BorgCube kernel: R13: ffff888101143c00 R14: ffff888103c98000 R15: 0000000000001296
Aug 14 17:05:58 BorgCube kernel: FS:  0000000000000000(0000) GS:ffff88885dcc0000(0000) knlGS:0000000000000000
Aug 14 17:05:58 BorgCube kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 14 17:05:58 BorgCube kernel: CR2: 0000153e416f7000 CR3: 000000000220a006 CR4: 00000000003706e0
Aug 14 17:05:58 BorgCube kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 14 17:05:58 BorgCube kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug 14 17:05:58 BorgCube kernel: Call Trace:
Aug 14 17:05:58 BorgCube kernel: <IRQ>
Aug 14 17:05:58 BorgCube kernel: ? rcu_dump_cpu_stacks+0x95/0xb9
Aug 14 17:05:58 BorgCube kernel: ? rcu_sched_clock_irq+0x345/0xa48
Aug 14 17:05:58 BorgCube kernel: ? nohz_balance_exit_idle+0x16/0xab
Aug 14 17:05:58 BorgCube kernel: ? trigger_load_balance+0x72/0x307
Aug 14 17:05:58 BorgCube kernel: ? tick_init_jiffy_update+0x7c/0x7c
Aug 14 17:05:58 BorgCube kernel: ? update_process_times+0x62/0x81
Aug 14 17:05:58 BorgCube kernel: ? tick_sched_timer+0x43/0x71
Aug 14 17:05:58 BorgCube kernel: ? __hrtimer_run_queues+0xe8/0x190
Aug 14 17:05:58 BorgCube kernel: ? hrtimer_interrupt+0x9c/0x16e
Aug 14 17:05:58 BorgCube kernel: ? __sysvec_apic_timer_interrupt+0xc2/0x12f
Aug 14 17:05:58 BorgCube kernel: ? sysvec_apic_timer_interrupt+0x27/0xa6
Aug 14 17:05:58 BorgCube kernel: ? asm_sysvec_apic_timer_interrupt+0x16/0x20
Aug 14 17:05:58 BorgCube kernel: ? preempt_count_sub+0x15/0x93
Aug 14 17:05:58 BorgCube kernel: _raw_spin_unlock+0x14/0x29
Aug 14 17:05:58 BorgCube kernel: bottom_half+0x436/0x8ef [r8152]
Aug 14 17:05:58 BorgCube kernel: ? _raw_spin_unlock_irq+0x1a/0x2f
Aug 14 17:05:58 BorgCube kernel: tasklet_action_common.constprop.0+0x60/0xa6
Aug 14 17:05:58 BorgCube kernel: handle_softirqs+0x129/0x271
Aug 14 17:05:58 BorgCube kernel: do_softirq+0x7f/0xab
Aug 14 17:05:58 BorgCube kernel: </IRQ>
Aug 14 17:05:58 BorgCube kernel: <TASK>
Aug 14 17:05:58 BorgCube kernel: __local_bh_enable_ip+0x4c/0x6b
Aug 14 17:05:58 BorgCube kernel: wg_socket_send_skb_to_peer+0x9c/0xab [wireguard]
Aug 14 17:05:58 BorgCube kernel: wg_packet_tx_worker+0xa6/0x18a [wireguard]
Aug 14 17:05:58 BorgCube kernel: process_one_work+0x1a8/0x295
Aug 14 17:05:58 BorgCube kernel: worker_thread+0x18b/0x244
Aug 14 17:05:58 BorgCube kernel: ? rescuer_thread+0x281/0x281
Aug 14 17:05:58 BorgCube kernel: kthread+0xe4/0xef
Aug 14 17:05:58 BorgCube kernel: ? kthread_complete_and_exit+0x1b/0x1b
Aug 14 17:05:58 BorgCube kernel: ret_from_fork+0x1f/0x30
Aug 14 17:05:58 BorgCube kernel: </TASK>
Aug 14 17:06:02 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:06:07 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:06:12 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:06:17 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:06:24 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:06:29 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:06:34 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:06:40 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:06:46 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:06:51 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:06:56 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout
Aug 14 17:07:02 BorgCube kernel: r8152 2-4:1.0 eth0: Tx timeout



borgcube-diagnostics-20240814-1801.zip

photo_2024-08-14_17-43-30.jpg

Edited by Jupiler
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing
×
×
  • Create New...