Kernel panicked out of the blue, and now I’m seeing odd errors I’ve never seen before after reboot.

[relink]

So I’m not sure what’s going on but completely out of the blue my unraid server locked up.

 

I wasn’t able to get a diagnostics as I couldn’t even SSH in, I connected a monitor and saw that it had a kernel panic. The last line read;

not syncing: fatal exception in interupt

The only other indication I saw was a reference to “iommu”, but I don’t currently have anything passed through to a VM, in fact I don’t even have any VMs running.

 

After unfortunately doing a hard reboot, a parity check started as expected, and everything seems to be ok except the following at the very bottom on the unraid GUI;

Parse error: syntax error, unexpected '{', expecting '(' in /usr/local/emhttp/plugins/parity.check.tuning/parity.check.tuning.php on line 1396

As well as a message from fix common problems that says that “write cache is disabled on parity 2”.

 

 

 

There have been no major hardware changes recently other than adding some new drives, and a new cache pool. I’m not seeing any errors coming from any disks, and everything on both of my cache pools is running fine. The new drives were added about 2 weeks ago and I have had no issues between then and now. 
 

The last major change was changing my HBA but this was nearly a year ago at this point. 
 

Unraid and all plugins are fully updated.

 

[trurl]

Post diagnostics now and setup Syslog Server so you can get syslog after crash.

https://wiki.unraid.net/Troubleshooting#Persistent_Logs_.28Syslog_server.29

[itimpi]

7 hours ago, relink said:

After unfortunately doing a hard reboot, a parity check started as expected, and everything seems to be ok except the following at the very bottom on the unraid GUI;

Parse error: syntax error, unexpected '{', expecting '(' in /usr/local/emhttp/plugins/parity.check.tuning/parity.check.tuning.php on line 1396

That suggests the you have the Parity Check Tuning plugin installed and that it is not up-to-date.   There was a version released about a week ago that had that syntax error, but it was updated the next day with a version that cleared the error.

[relink]

5 hours ago, itimpi said:

Parity Check Tuning plugin installed and that it is not up-to-date.

Well that solved that issue, apparently everything was up to date except that one. Updated it, and now that error is gone. 

 

But im still concerned about the KP, thats never happened to me before.

 

Im also not sure what the Parity 2 cacheing being disabled in all about.

[relink]

10 hours ago, trurl said:

Post diagnostics now and setup Syslog Server so you can get syslog after crash.

https://wiki.unraid.net/Troubleshooting#Persistent_Logs_.28Syslog_server.29

Actually I have a syslog server that records everything on my Synology. What should I be looking for in the logs and I can go pull it and post it here. I remember around what time this all happened. 

[relink]

I went back into the logs, this seems to be when the crash started, maybe a little before, all the way up to me having to hard reset. 

 

4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	CR2: 0000000000000010
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	Modules linked in: md4 sha512_ssse3 sha512_generic cmac cifs libarc4 nfsv3 nfs nfs_ssc nvidia_uvm(PO) xt_CHECKSUM ipt_REJECT ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap macvlan veth xt_nat xt_MASQUERADE iptable_nat nf_nat ext4 mbcache jbd2 xfs nfsd lockd grace sunrpc md_mod nvidia_drm(PO) nvidia_modeset(PO) drm_kms_helper drm backlight agpgart syscopyarea sysfillrect sysimgblt fb_sys_fops nvidia(PO) ip6table_filter ip6_tables iptable_filter ip_tables bonding e1000e igb i2c_algo_bit edac_mce_amd kvm_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd nvme glue_helper mxm_wmi wmi_bmof ahci nvme_core rapl aacraid i2c_piix4 k10temp input_leds wmi i2c_core led_class ccp libahci button acpi_cpufreq [last unloaded: e1000e]
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	secondary_startup_64_no_verify+0xb0/0xbb
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	cpu_startup_entry+0x18/0x1a
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	do_idle+0x1a6/0x214
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	cpuidle_enter+0x25/0x31
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	cpuidle_enter_state+0xba/0x1c4
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	acpi_idle_enter+0x9a/0xa9
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	acpi_idle_do_entry+0x25/0x37
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	arch_safe_halt+0x5/0x8
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	? native_safe_halt+0x5/0x8
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	R13: ffff888100b0cc64 R14: ffffffff820cada8 R15: 0000000000000000
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	R10: 00000000000003e4 R11: 071c71c71c71c71c R12: 0000000000000001
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	RBP: ffff888104dd1c00 R08: ffff888100b0cc00 R09: 00000000000003e4
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	RDX: ffff8887feac0000 RSI: ffffffff820cad40 RDI: ffff888100b0cc64
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	RAX: 0000000000004000 RBX: 0000000000000001 RCX: 000000000000001f
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	RSP: 0018:ffffc90000177e78 EFLAGS: 00000246
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	Code: 60 02 df f0 83 44 24 fc 00 48 8b 00 a8 08 74 0b 65 81 25 b1 26 92 7e ff ff ff 7f c3 e8 4e ca 94 ff f4 c3 e8 47 ca 94 ff fb f4 <c3> 53 e8 a6 6b 9a ff e8 ba f2 97 ff 65 48 8b 1c 25 c0 7b 01 00 48
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	RIP: 0010:native_safe_halt+0x7/0x8
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	asm_common_interrupt+0x1e/0x40
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	common_interrupt+0xa5/0x12e
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	</IRQ>
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	asm_call_irq_on_stack+0xf/0x20
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	handle_edge_irq+0xb0/0xd0
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	handle_irq_event+0x34/0x51
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	handle_irq_event_percpu+0x2c/0x6f
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	__handle_irq_event_percpu+0x36/0xcb
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	aac_src_intr_message+0x321/0x35d [aacraid]
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	aac_intr_normal+0x2dc/0x2ff [aacraid]
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	aac_srb_callback+0x67/0x30d [aacraid]
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	<IRQ>
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	Call Trace:
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	CR2: 0000000000000010 CR3: 000000022f1ac000 CR4: 00000000003506e0
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	FS:  0000000000000000(0000) GS:ffff8887feac0000(0000) knlGS:0000000000000000
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	R13: ffff8881016340b8 R14: ffff888103380a9c R15: 0000000000000000
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000001f
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	RBP: ffff88813703ca10 R08: 0000000000000000 R09: 0000000000000000
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	RDX: 0000000000000020 RSI: 0000000000000000 RDI: 0000000000000000
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	RAX: ffffffff81436445 RBX: 0000000000000000 RCX: 0000000000000002
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	RSP: 0018:ffffc90000400e88 EFLAGS: 00010012
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	Code: 00 48 83 c4 20 5b 5d 41 5c 41 5d 41 5e c3 41 57 45 31 ff 41 56 41 55 49 89 fd 48 89 f7 41 54 41 89 d4 55 41 ff cc 53 48 89 f3 <4c> 8b 76 10 e8 42 8e eb ff 48 89 c5 45 39 fc 7e 06 83 7d 18 00 75
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	RIP: 0010:iommu_dma_unmap_sg+0x1c/0x68
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	Hardware name: System manufacturer System Product Name/ROG STRIX B450-F GAMING, BIOS 2301 04/19/2019
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	CPU: 11 PID: 0 Comm: swapper/11 Tainted: P        W  O      5.10.21-Unraid #1
4/4/21	17:46:06	Warning	SERVERUS	kern	kernel	Oops: 0000 [#1] SMP NOPTI
4/4/21	17:46:06	Information	SERVERUS	kern	kernel	PGD 22b484067 P4D 22b484067 PUD 0 
4/4/21	17:46:06	Alert	SERVERUS	kern	kernel	#PF: error_code(0x0000) - not-present page
4/4/21	17:46:06	Alert	SERVERUS	kern	kernel	#PF: supervisor read access in kernel mode
4/4/21	17:46:06	Alert	SERVERUS	kern	kernel	BUG: kernel NULL pointer dereference, address: 0000000000000010
4/4/21	17:46:05	Error	SERVERUS	kern	kernel	aacraid: Outstanding commands on (2,1,13,0):
4/4/21	17:46:05	Error	SERVERUS	kern	kernel	aacraid: Host adapter abort request.
4/4/21	17:45:37	Error	SERVERUS	kern	kernel	aacraid: Outstanding commands on (2,1,13,0):
4/4/21	17:45:37	Error	SERVERUS	kern	kernel	aacraid: Host adapter abort request.

 

[trurl]

12 hours ago, trurl said:

Post diagnostics now

That will give us more information about your hardware and configuration.

[relink]

Here are my most recent diagnostics. 

serverus-diagnostics-20210405-1316.zip

[trurl]

 

[relink]

I should already have C-states disabled, I read the threads on ryzen before buying the board I have. Plus I have had this board and cpu for around 2 years now and have never had this issue before. 
 

however I can double check, but I won’t be able to get physical access to my rig until later this afternoon. 

[relink]

Man life can get hectic. I finally got around to checking and the only option for C-states in my bios is a single option called “Global C-states control” and it was set to auto, I changed it to disabled. I completely forgot about that when I reset my BIOS about a month ago. Hopefully that’s all it was.