Skip to content
View in the app

A better way to browse. Learn more.
Unraid

A full-screen app on your home screen with push notifications, badges and more.

To install this app on iOS and iPadOS
  1. Tap the Share icon in Safari
  2. Scroll the menu and tap Add to Home Screen.
  3. Tap Add in the top-right corner.
To install this app on Android
  1. Tap the 3-dot menu (⋮) in the top-right corner of the browser.
  2. Tap Add to Home screen or Install app.
  3. Confirm by tapping Install.
Unraid Summer Sale in live: 20% off Starter and Unleashed + buy one, get one 50% off

Unraid crashing every 8-12 hours

InsaneNutter
in General Support

Featured Replies

Currently I am on Unraid 6.12.10, over the last few months Unraid has been become increasingly unstable.

The server started crashing ever few days, now its got to the point where its every 10-12 hours, sometimes less.

 

Crash = no output to a connected monitor, server totally inaccessible via the web ui, network shares, Docker containers down.

 

Yesterday I switched to a different motherboard, ram and processor, using my existing configuration and exactly the same has just happened after around 10 hours, so I feel it must be related to my configuration in some way and not the hardware its self.

 

I've setup a syslog server on my PC, so I guess its a waiting game to see if anything is logged when it crashes again.

 

Any advice would be appreciated.

 

Edit - from looking at what was logged so my syslog server before it crashed it would appear to be an nvme related issue:

 

192.168.1.2    Jun 16 08:47:12    Server    user    info    emhttpd    read SMART /dev/sdd
192.168.1.2    Jun 16 08:47:12    Server    user    info    emhttpd    read SMART /dev/sdc
192.168.1.2    Jun 16 08:51:06    Server    user    info    emhttpd    read SMART /dev/sdb
192.168.1.2    Jun 16 09:06:55    Server    kern    warning    kernel    nvme nvme0: I/O 698 (I/O Cmd) QID 1 timeout, aborting
192.168.1.2    Jun 16 09:06:55    Server    kern    warning    kernel    nvme nvme0: I/O 450 (I/O Cmd) QID 2 timeout, aborting
192.168.1.2    Jun 16 09:06:56    Server    kern    warning    kernel    nvme nvme0: I/O 697 (I/O Cmd) QID 1 timeout, aborting
192.168.1.2    Jun 16 09:07:25    Server    kern    warning    kernel    nvme nvme0: I/O 698 QID 1 timeout, reset controller
192.168.1.2    Jun 16 09:07:31    Server    kern    warning    kernel    nvme nvme0: I/O 8 QID 0 timeout, reset controller
192.168.1.2    Jun 16 09:10:37    Server    kern    err    kernel    nvme nvme0: Device not ready; aborting reset, CSTS=0x1
 

Edited by InsaneNutter

  • Quote

    • Solved by InsaneNutter

    Go to solution
    • Community Expert

    Looks like the NVMe device dropped, but usually that usually not a reason to crash the entire server, post the complete syslog in case there's something there.

  • Quote
    • Author
    30 minutes ago, JorgeB said:

    Looks like the NVMe device dropped, but usually that usually not a reason to crash the entire server, post the complete syslog in case there's something there.


     

    The full log from before the crash is quite short as it happened about an hour after i'd setup remote logging, however I have pasted it below:

      

    192.168.1.2	Jun 16 07:54:33	Server	syslog	info	rsyslogd	[origin software="rsyslogd" swVersion="8.2102.0" x-pid="7178" x-info="https://www.rsyslog.com"] start
192.168.1.2	Jun 16 08:11:43	Server	user	info	emhttpd	spinning down /dev/sdd
192.168.1.2	Jun 16 08:11:43	Server	user	info	emhttpd	spinning down /dev/sde
192.168.1.2	Jun 16 08:11:43	Server	user	info	emhttpd	spinning down /dev/sdb
192.168.1.2	Jun 16 08:11:49	Server	user	info	emhttpd	spinning down /dev/sdc
192.168.1.2	Jun 16 08:17:48	Server	kern	info	kernel	vethf705692: renamed from eth0
192.168.1.2	Jun 16 08:17:48	Server	kern	info	kernel	docker0: port 3(veth506f2af) entered disabled state
192.168.1.2	Jun 16 08:17:48	Server	kern	info	kernel	docker0: port 3(veth506f2af) entered disabled state
192.168.1.2	Jun 16 08:17:48	Server	kern	info	kernel	device veth506f2af left promiscuous mode
192.168.1.2	Jun 16 08:17:48	Server	kern	info	kernel	docker0: port 3(veth506f2af) entered disabled state
192.168.1.2	Jun 16 08:18:07	Server	kern	info	kernel	docker0: port 3(vethd4d190f) entered blocking state
192.168.1.2	Jun 16 08:18:07	Server	kern	info	kernel	docker0: port 3(vethd4d190f) entered disabled state
192.168.1.2	Jun 16 08:18:07	Server	kern	info	kernel	device vethd4d190f entered promiscuous mode
192.168.1.2	Jun 16 08:18:07	Server	kern	info	kernel	eth0: renamed from vethd9163fc
192.168.1.2	Jun 16 08:18:07	Server	kern	info	kernel	IPv6: ADDRCONF(NETDEV_CHANGE): vethd4d190f: link becomes ready
192.168.1.2	Jun 16 08:18:07	Server	kern	info	kernel	docker0: port 3(vethd4d190f) entered blocking state
192.168.1.2	Jun 16 08:18:07	Server	kern	info	kernel	docker0: port 3(vethd4d190f) entered forwarding state
192.168.1.2	Jun 16 08:23:57	Server	kern	info	kernel	br0: port 3(vnet1) entered blocking state
192.168.1.2	Jun 16 08:23:57	Server	kern	info	kernel	br0: port 3(vnet1) entered disabled state
192.168.1.2	Jun 16 08:23:57	Server	kern	info	kernel	device vnet1 entered promiscuous mode
192.168.1.2	Jun 16 08:23:57	Server	kern	info	kernel	br0: port 3(vnet1) entered blocking state
192.168.1.2	Jun 16 08:23:57	Server	kern	info	kernel	br0: port 3(vnet1) entered forwarding state
192.168.1.2	Jun 16 08:28:20	Server	kern	warning	kernel	x86/split lock detection: #AC: CPU 0/KVM/24166 took a split_lock trap at address: 0xfffff80545e4b9dd
192.168.1.2	Jun 16 08:32:11	Server	user	info	emhttpd	read SMART /dev/sde
192.168.1.2	Jun 16 08:35:28	Server	kern	warning	kernel	x86/split lock detection: #AC: CPU 2/KVM/24168 took a split_lock trap at address: 0x27403401dde
192.168.1.2	Jun 16 08:47:12	Server	user	info	emhttpd	read SMART /dev/sdd
192.168.1.2	Jun 16 08:47:12	Server	user	info	emhttpd	read SMART /dev/sdc
192.168.1.2	Jun 16 08:51:06	Server	user	info	emhttpd	read SMART /dev/sdb
192.168.1.2	Jun 16 09:06:55	Server	kern	warning	kernel	nvme nvme0: I/O 698 (I/O Cmd) QID 1 timeout, aborting
192.168.1.2	Jun 16 09:06:55	Server	kern	warning	kernel	nvme nvme0: I/O 450 (I/O Cmd) QID 2 timeout, aborting
192.168.1.2	Jun 16 09:06:56	Server	kern	warning	kernel	nvme nvme0: I/O 697 (I/O Cmd) QID 1 timeout, aborting
192.168.1.2	Jun 16 09:07:25	Server	kern	warning	kernel	nvme nvme0: I/O 698 QID 1 timeout, reset controller
192.168.1.2	Jun 16 09:07:31	Server	kern	warning	kernel	nvme nvme0: I/O 8 QID 0 timeout, reset controller
192.168.1.2	Jun 16 09:10:37	Server	kern	err	kernel	nvme nvme0: Device not ready; aborting reset, CSTS=0x1

     

    I found a post by yourself as it happens on a simlar issue: 

     

    I've appended: initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off as advised there to see if that helps.

     

    I suspect not ideal from a power use point of view as my drive will never enter a low power mode, curious as to if it helps however.

    Edited by InsaneNutter

  • Quote
    • Community Expert

    Worth a try, and if it crashes again, check to see if it was also after the NVMde device dropped.

  • Quote
    • Author

    Unfortunately Unraid has crashed again just after midnight, seems to be with the NVME drive.

      

    192.168.1.2	Jun 16 17:21:31	Server	kern	info	kernel	eth0: renamed from vethda87c4c
192.168.1.2	Jun 16 17:24:00	Server	user	info	emhttpd	read SMART /dev/sde
192.168.1.2	Jun 16 17:28:44	Server	user	info	emhttpd	spinning down /dev/sdd
192.168.1.2	Jun 16 17:46:26	Server	kern	info	kernel	eth0: renamed from vethbebac28
192.168.1.2	Jun 16 17:46:26	Server	kern	info	kernel	vethbebac28: renamed from eth0
192.168.1.2	Jun 16 17:48:21	Server	kern	info	kernel	docker0: port 1(vethd2f1ae1) entered blocking state
192.168.1.2	Jun 16 17:48:21	Server	kern	info	kernel	docker0: port 1(vethd2f1ae1) entered disabled state
192.168.1.2	Jun 16 17:48:21	Server	kern	info	kernel	device vethd2f1ae1 entered promiscuous mode
192.168.1.2	Jun 16 17:48:21	Server	kern	info	kernel	eth0: renamed from vethd699878
192.168.1.2	Jun 16 17:48:21	Server	kern	info	kernel	IPv6: ADDRCONF(NETDEV_CHANGE): vethd2f1ae1: link becomes ready
192.168.1.2	Jun 16 17:48:21	Server	kern	info	kernel	docker0: port 1(vethd2f1ae1) entered blocking state
192.168.1.2	Jun 16 17:48:21	Server	kern	info	kernel	docker0: port 1(vethd2f1ae1) entered forwarding state
192.168.1.2	Jun 16 17:48:54	Server	kern	info	kernel	vethd699878: renamed from eth0
192.168.1.2	Jun 16 17:48:54	Server	kern	info	kernel	docker0: port 1(vethd2f1ae1) entered disabled state
192.168.1.2	Jun 16 17:48:54	Server	kern	info	kernel	docker0: port 1(vethd2f1ae1) entered disabled state
192.168.1.2	Jun 16 17:48:54	Server	kern	info	kernel	device vethd2f1ae1 left promiscuous mode
192.168.1.2	Jun 16 17:48:54	Server	kern	info	kernel	docker0: port 1(vethd2f1ae1) entered disabled state
192.168.1.2	Jun 16 17:48:54	Server	kern	info	kernel	eth0: renamed from veth6f64f3c
192.168.1.2	Jun 16 17:55:43	Server	user	info	emhttpd	spinning down /dev/sde
192.168.1.2	Jun 16 18:02:32	Server	kern	info	kernel	vethd21dfcb: renamed from eth0
192.168.1.2	Jun 16 18:03:12	Server	kern	info	kernel	br-548a3d5ec277: port 2(vethe265799) entered blocking state
192.168.1.2	Jun 16 18:03:12	Server	kern	info	kernel	br-548a3d5ec277: port 2(vethe265799) entered disabled state
192.168.1.2	Jun 16 18:03:12	Server	kern	info	kernel	device vethe265799 entered promiscuous mode
192.168.1.2	Jun 16 18:03:12	Server	kern	info	kernel	br-548a3d5ec277: port 2(vethe265799) entered disabled state
192.168.1.2	Jun 16 18:03:12	Server	kern	info	kernel	device vethe265799 left promiscuous mode
192.168.1.2	Jun 16 18:03:12	Server	kern	info	kernel	br-548a3d5ec277: port 2(vethe265799) entered disabled state
192.168.1.2	Jun 16 18:03:31	Server	kern	info	kernel	eth0: renamed from veth73b23f5
192.168.1.2	Jun 16 18:04:17	Server	kern	info	kernel	veth5bc5fc6: renamed from eth0
192.168.1.2	Jun 16 18:04:17	Server	kern	info	kernel	br-548a3d5ec277: port 1(veth15f5eb2) entered disabled state
192.168.1.2	Jun 16 18:04:17	Server	kern	info	kernel	br-548a3d5ec277: port 1(veth15f5eb2) entered disabled state
192.168.1.2	Jun 16 18:04:17	Server	kern	info	kernel	device veth15f5eb2 left promiscuous mode
192.168.1.2	Jun 16 18:04:17	Server	kern	info	kernel	br-548a3d5ec277: port 1(veth15f5eb2) entered disabled state
192.168.1.2	Jun 16 18:04:17	Server	kern	info	kernel	eth0: renamed from veth6504ee7
192.168.1.2	Jun 16 18:30:06	Server	user	info	emhttpd	read SMART /dev/sde
192.168.1.2	Jun 16 18:30:40	Server	user	info	emhttpd	read SMART /dev/sdb
192.168.1.2	Jun 16 18:38:41	Server	user	info	emhttpd	read SMART /dev/sdd
192.168.1.2	Jun 16 19:02:43	Server	user	info	emhttpd	spinning down /dev/sdb
192.168.1.2	Jun 16 20:52:34	Server	user	info	emhttpd	spinning down /dev/sdd
192.168.1.2	Jun 16 20:53:22	Server	user	info	emhttpd	spinning down /dev/sde
192.168.1.2	Jun 16 22:10:13	Server	user	info	emhttpd	read SMART /dev/sde
192.168.1.2	Jun 16 22:56:54	Server	user	info	emhttpd	read SMART /dev/sdd
192.168.1.2	Jun 16 23:02:52	Server	user	info	emhttpd	spinning down /dev/sde
192.168.1.2	Jun 16 23:12:51	Server	user	info	emhttpd	read SMART /dev/sde
192.168.1.2	Jun 16 23:13:04	Server	user	info	emhttpd	spinning down /dev/sdc
192.168.1.2	Jun 17 00:00:49	Server	kern	warning	kernel	nvme nvme0: I/O 584 (I/O Cmd) QID 3 timeout, aborting
192.168.1.2	Jun 17 00:00:49	Server	kern	warning	kernel	nvme nvme0: I/O 665 (I/O Cmd) QID 6 timeout, aborting
192.168.1.2	Jun 17 00:00:50	Server	kern	warning	kernel	nvme nvme0: I/O 596 (I/O Cmd) QID 5 timeout, aborting
192.168.1.2	Jun 17 00:01:19	Server	kern	warning	kernel	nvme nvme0: I/O 584 QID 3 timeout, reset controller
192.168.1.2	Jun 17 00:01:38	Server	kern	warning	kernel	nvme nvme0: I/O 24 QID 0 timeout, reset controller
192.168.1.2	Jun 17 00:04:31	Server	kern	err	kernel	nvme nvme0: Device not ready; aborting reset, CSTS=0x1
192.168.1.2	Jun 17 00:04:37	Server	kern	warning	kernel	nvme nvme0: Abort status: 0x371
192.168.1.2	Jun 17 00:04:37	Server	kern	warning	kernel	nvme nvme0: Abort status: 0x371
192.168.1.2	Jun 17 00:04:37	Server	kern	warning	kernel	nvme nvme0: Abort status: 0x371
192.168.1.2	Jun 17 00:04:40	Server	kern	err	kernel	nvme 0000:01:00.0: Unable to change power state from D3cold to D0, device inaccessible
192.168.1.2	Jun 17 00:05:51	Server	kern	err	kernel	rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
192.168.1.2	Jun 17 00:05:51	Server	kern	err	kernel	rcu:  8-....: (52 ticks this GP) idle=905c/1/0x4000000000000000 softirq=6264128/6264128 fqs=25390
192.168.1.2	Jun 17 00:05:51	Server	kern	err	kernel	rcu:  9-....: (55 ticks this GP) idle=f4dc/1/0x4000000000000000 softirq=2796004/2796004 fqs=25390
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	 (detected by 5, t=60002 jiffies, g=9828341, q=376561 ncpus=12)
192.168.1.2	Jun 17 00:05:51	Server	kern	info	kernel	Sending NMI from CPU 5 to CPUs 8:
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	NMI backtrace for cpu 8
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	CPU: 8 PID: 64 Comm: ksoftirqd/8 Tainted: P           O       6.1.79-Unraid #1
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	Hardware name: Default string Default string/Default string, BIOS 5.27 05/06/2024
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	RIP: 0010:__ip_append_data+0x319/0xa24
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	Code: 24 5e 88 44 24 5f 8b 84 24 c0 00 00 00 01 45 14 48 85 db 74 12 45 31 d2 44 89 54 24 14 44 89 54 24 3c e9 57 04 00 00 45 31 c9 <44> 89 4c 24 3c 44 89 4c 24 14 44 89 4c 24 18 0f b7 44 24 68 0f b7
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	RSP: 0000:ffffc900003834f8 EFLAGS: 00000246
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	RAX: 0000000000000014 RBX: 0000000000000000 RCX: 0000000000000014
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	RDX: 00000000000005c8 RSI: ffffc90000383688 RDI: 0000000000000014
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	RBP: ffff888101aaa340 R08: ffff888101aaa1d8 R09: 0000000000000000
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	R10: 0000000000000000 R11: 0000000000000000 R12: ffff888186f03000
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	R13: 0000000000000000 R14: 0000000000000000 R15: ffff888101aaa000
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	FS:  0000000000000000(0000) GS:ffff88885fa00000(0000) knlGS:0000000000000000
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	CR2: 0000153948601003 CR3: 00000002bcd76000 CR4: 0000000000752ee0
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	PKRU: 55555554
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	Call Trace:
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	<NMI>
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? nmi_cpu_backtrace+0xd3/0x104
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? nmi_cpu_backtrace_handler+0xd/0x15
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? nmi_handle+0x54/0x131
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? __ip_append_data+0x319/0xa24
192.168.1.2	Jun 17 00:05:51	Server	kern	info	kernel	Sending NMI from CPU 5 to CPUs 9:
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? default_do_nmi+0x66/0x15b
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? exc_nmi+0xbf/0x130
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? end_repeat_nmi+0x16/0x67
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? __ip_append_data+0x319/0xa24
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? __ip_append_data+0x319/0xa24
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? __ip_append_data+0x319/0xa24
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	</NMI>
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	<TASK>
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? pskb_trim_unique+0x25/0x25
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? pskb_trim_unique+0x25/0x25
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	ip_append_data.part.0+0xa7/0xbe
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	ip_send_unicast_reply+0x2b1/0x3b7
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? fib_table_lookup+0x358/0x3ba
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? tcp_v4_send_reset+0x538/0x5e9
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	tcp_v4_send_reset+0x538/0x5e9
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? tcp_v4_rcv+0x74f/0x92c
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	tcp_v4_rcv+0x74f/0x92c
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	ip_protocol_deliver_rcu+0x9c/0x164
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	ip_local_deliver_finish+0x7e/0x8c
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	__netif_receive_skb_one_core+0x77/0x9c
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	netif_receive_skb+0xbf/0x127
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	br_handle_frame_finish+0x482/0x4bc [bridge]
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? nf_conntrack_in+0x30d/0x4bb [nf_conntrack]
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? br_pass_frame_up+0xdd/0xdd [bridge]
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	br_nf_hook_thresh+0xe2/0x109 [br_netfilter]
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? br_pass_frame_up+0xdd/0xdd [bridge]
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	br_nf_pre_routing_finish+0x2cd/0x2f8 [br_netfilter]
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? br_pass_frame_up+0xdd/0xdd [bridge]
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? br_nf_hook_thresh+0x109/0x109 [br_netfilter]
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	br_nf_pre_routing+0x236/0x24a [br_netfilter]
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? br_nf_hook_thresh+0x109/0x109 [br_netfilter]
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	br_handle_frame+0x277/0x2e0 [bridge]
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? br_pass_frame_up+0xdd/0xdd [bridge]
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	__netif_receive_skb_core.constprop.0+0x4fa/0x6e9
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? asm_sysvec_apic_timer_interrupt+0x16/0x20
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	__netif_receive_skb_list_core+0x8a/0x11e
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	netif_receive_skb_list_internal+0x1d2/0x20b
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	gro_normal_one+0x46/0x63
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	napi_gro_receive+0x97/0x10a
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	igc_poll+0xc8c/0xf4e [igc]
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	__napi_poll.constprop.0+0x28/0x124
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	net_rx_action+0x159/0x24f
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? raw_spin_rq_unlock_irq+0x5/0x10
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	__do_softirq+0x126/0x288
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	run_ksoftirqd+0x29/0x38
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	smpboot_thread_fn+0x1b6/0x1d3
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? find_next_bit+0x5/0x5
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	kthread+0xe4/0xef
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? kthread_complete_and_exit+0x1b/0x1b
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	ret_from_fork+0x1f/0x30
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	</TASK>
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	NMI backtrace for cpu 9
192.168.1.2	Jun 17 00:05:51	Server	kern	info	kernel	INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 9567.688 msecs
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	CPU: 9 PID: 541 Comm: kworker/u24:2 Tainted: P           O       6.1.79-Unraid #1
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	Hardware name: Default string Default string/Default string, BIOS 5.27 05/06/2024
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	Workqueue: nvme-reset-wq nvme_reset_work [nvme]
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	RIP: 0010:pci_mmcfg_read+0xa4/0xbc
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	Code: 83 fe 02 74 15 41 83 fe 04 74 1a 41 ff ce 75 1d 48 01 d8 8a 00 0f b6 c0 eb 10 48 01 d8 66 8b 00 0f b7 c0 eb 05 48 01 d8 8b 00 <89> 45 00 e8 26 65 89 ff 31 c0 5a 5b 5d 41 5c 41 5d 41 5e c3 cc cc
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	RSP: 0018:ffffc90044cd3d18 EFLAGS: 00000286
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	RAX: 00000000ffffffff RBX: 0000000000000ffc RCX: 0000000000000ffc
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	RDX: 00000000000000ff RSI: 0000000000000001 RDI: 0000000000000000
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	RBP: ffffc90044cd3d54 R08: 0000000000000004 R09: ffffc90044cd3d54
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	R10: 0000000000000001 R11: ffff8881004002f0 R12: 0000000000100000
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	R13: 0000000000000000 R14: 0000000000000004 R15: ffff88810192e0d0
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	FS:  0000000000000000(0000) GS:ffff88885fa40000(0000) knlGS:0000000000000000
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	CR2: 000014fd2fc01aa8 CR3: 0000000282f10000 CR4: 0000000000752ee0
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	PKRU: 55555554
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	Call Trace:
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	<NMI>
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? nmi_cpu_backtrace+0xd3/0x104
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? nmi_cpu_backtrace_handler+0xd/0x15
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? nmi_handle+0x54/0x131
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? pci_mmcfg_read+0xa4/0xbc
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? default_do_nmi+0x66/0x15b
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? exc_nmi+0xbf/0x130
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? end_repeat_nmi+0x16/0x67
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? pci_mmcfg_read+0xa4/0xbc
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? pci_mmcfg_read+0xa4/0xbc
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? pci_mmcfg_read+0xa4/0xbc
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	</NMI>
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	<TASK>
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	pci_bus_read_config_dword+0x46/0x7a
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	pci_find_next_ext_capability+0x9d/0xcb
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	pci_save_state+0x15a/0x1ad
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	nvme_reset_work+0x376/0x1044 [nvme]
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? _raw_spin_unlock+0x14/0x29
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? raw_spin_rq_unlock_irq+0x5/0x10
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? finish_task_switch.isra.0+0x140/0x218
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	process_one_work+0x1a8/0x295
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	worker_thread+0x18b/0x244
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? rescuer_thread+0x281/0x281
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	kthread+0xe4/0xef
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? kthread_complete_and_exit+0x1b/0x1b
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	ret_from_fork+0x1f/0x30
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	</TASK>
192.168.1.2	Jun 17 00:05:51	Server	kern	err	kernel	rcu: rcu_preempt kthread starved for 10528 jiffies! g9828341 f0x0 RCU_GP_DOING_FQS(6) ->state=0x0 ->cpu=0
192.168.1.2	Jun 17 00:05:51	Server	kern	err	kernel	rcu:  Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
192.168.1.2	Jun 17 00:05:51	Server	kern	err	kernel	rcu: RCU grace-period kthread stack dump:
192.168.1.2	Jun 17 00:05:51	Server	kern	info	kernel	task:rcu_preempt     state:R  running task     stack:0     pid:15    ppid:2      flags:0x00004808
192.168.1.2	Jun 17 00:05:51	Server	kern	info	kernel	Call Trace:
192.168.1.2	Jun 17 00:05:51	Server	kern	info	kernel	<TASK>
192.168.1.2	Jun 17 00:05:51	Server	kern	info	kernel	? rcu_gp_kthread+0x16d/0x16d
192.168.1.2	Jun 17 00:05:51	Server	kern	info	kernel	? _raw_spin_unlock_irqrestore+0x1a/0x3a
192.168.1.2	Jun 17 00:05:51	Server	kern	info	kernel	? force_qs_rnp+0x1b1/0x1d0
192.168.1.2	Jun 17 00:05:51	Server	kern	info	kernel	? rcu_gp_init+0x494/0x494
192.168.1.2	Jun 17 00:05:51	Server	kern	info	kernel	? rcu_gp_fqs_loop+0x231/0x475
192.168.1.2	Jun 17 00:05:51	Server	kern	info	kernel	? rcu_gp_kthread+0x151/0x16d
192.168.1.2	Jun 17 00:05:51	Server	kern	info	kernel	? kthread+0xe4/0xef
192.168.1.2	Jun 17 00:05:51	Server	kern	info	kernel	? kthread_complete_and_exit+0x1b/0x1b
192.168.1.2	Jun 17 00:05:51	Server	kern	info	kernel	? ret_from_fork+0x1f/0x30
192.168.1.2	Jun 17 00:05:51	Server	kern	info	kernel	</TASK>
192.168.1.2	Jun 17 00:05:51	Server	kern	err	kernel	rcu: Stack dump where RCU GP kthread last ran:
192.168.1.2	Jun 17 00:05:51	Server	kern	info	kernel	Sending NMI from CPU 5 to CPUs 0:
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	NMI backtrace for cpu 0
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	CPU: 0 PID: 15 Comm: rcu_preempt Tainted: P           O       6.1.79-Unraid #1
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	Hardware name: Default string Default string/Default string, BIOS 5.27 05/06/2024
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	RIP: 0010:preempt_count_sub+0x8e/0x93
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	Code: ab a5 31 01 00 75 23 48 c7 c6 53 50 0c 82 48 c7 c7 88 30 0b 82 e8 c6 78 fd ff 0f 0b c3 cc cc cc cc f7 df 65 01 3d 5e 6b f8 7e <c3> cc cc cc cc 0f 1f 44 00 00 85 ff 74 05 e9 1a 40 ff ff 65 48 8b
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	RSP: 0018:ffffc90000003f40 EFLAGS: 00000013
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	RAX: 0000000000010002 RBX: 0000000000000002 RCX: ffff888106b36e70
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	RDX: 0000000000010002 RSI: 0000000000000002 RDI: 00000000ffffffff
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	RBP: ffff888106b36e70 R08: ffff88885f8200e0 R09: ffff88885f8200e0
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	R10: ffff888192627090 R11: ffffc90000003ff8 R12: ffff88885f820080
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	R13: 0000301ec8482ee3 R14: 0000000000000002 R15: ffffffffa067ae75
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	FS:  0000000000000000(0000) GS:ffff88885f800000(0000) knlGS:0000000000000000
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	CR2: 00002a3c00175000 CR3: 00000002bcd76000 CR4: 0000000000752ef0
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	PKRU: 55555554
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	Call Trace:
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	<NMI>
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? nmi_cpu_backtrace+0xd3/0x104
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? nmi_cpu_backtrace_handler+0xd/0x15
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? nmi_handle+0x54/0x131
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? preempt_count_sub+0x8e/0x93
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? default_do_nmi+0x66/0x15b
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? exc_nmi+0xbf/0x130
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? end_repeat_nmi+0x16/0x67
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? pit_latch_count+0x2b/0x2b [kvm]
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? preempt_count_sub+0x8e/0x93
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? preempt_count_sub+0x8e/0x93
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? preempt_count_sub+0x8e/0x93
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	</NMI>
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	<IRQ>
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	_raw_spin_unlock_irqrestore+0x24/0x3a
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	__hrtimer_run_queues+0x9c/0x190
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	hrtimer_interrupt+0x9c/0x16e
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	__sysvec_apic_timer_interrupt+0xc2/0x12f
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	sysvec_apic_timer_interrupt+0x80/0xa6
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	</IRQ>
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	<TASK>
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	asm_sysvec_apic_timer_interrupt+0x16/0x20
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	RIP: 0010:_raw_spin_unlock_irqrestore+0x1a/0x3a
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	Code: 79 7e 85 c0 75 05 0f 1f 44 00 00 c3 cc cc cc cc 0f 1f 44 00 00 53 48 89 f3 e8 32 83 83 ff 0f ba e3 09 73 06 fb 0f 1f 44 00 00 <bf> 01 00 00 00 e8 5f 43 81 ff 65 8b 05 44 af 79 7e 85 c0 75 05 0f
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	RSP: 0018:ffffc900000e7e50 EFLAGS: 00000247
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	RAX: 0000000000000040 RBX: 0000000000000286 RCX: 000000000000000a
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	RDX: 0000000000000040 RSI: 0000000000000286 RDI: ffffffff822bae80
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	R10: 0000000000000020 R11: 000000000000009c R12: ffffffff810d9de9
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	R13: ffff8881002de000 R14: 0000000000000000 R15: 000000000002ee00
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? rcu_gp_kthread+0x16d/0x16d
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	force_qs_rnp+0x1b1/0x1d0
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? rcu_gp_init+0x494/0x494
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	rcu_gp_fqs_loop+0x231/0x475
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	rcu_gp_kthread+0x151/0x16d
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	kthread+0xe4/0xef
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	? kthread_complete_and_exit+0x1b/0x1b
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	ret_from_fork+0x1f/0x30
192.168.1.2	Jun 17 00:05:51	Server	kern	warning	kernel	</TASK>

     

    I wonder if the drive is starting to fail? although its only about an 8 month old Patriot P300.

  • Quote
    • Community Expert

    I also see a macvlan related crash, and those are known to make the server crash, change the network type to ipvlan and reboot.

     

    Regarding the NVMe, it doesn't necessarily mean it's failing, sometimes it's just a hardware combination issue, i.e., a different model may work better there and that one still work with a different board.

  • Quote
    • Author
    • Solution

    Just to update on this, I believe the issue was the NVME drive.

     

    I replaced the drive mid week with a Samsung 990 PRO, moved over the data from the old drive and Unraid seems to have been perfectly happy for the last couple of days.

  • Quote

    • Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Reply to this topic...
    Go to topic listing

    Account

    Navigation

    Search

    Search

    Configure browser push notifications
    Chrome (Android)
    1. Tap the lock icon next to the address bar.
    2. Tap Permissions → Notifications.
    3. Adjust your preference.
    Chrome (Desktop)
    1. Click the padlock icon in the address bar.
    2. Select Site settings.
    3. Find Notifications and adjust your preference.