I recently upgrade the hardware in my server from a 3700K to a 14700K. Upgrades were a new motherboard MSI z790-p Pro Wifi, Intel 13700K, G.Skill Ripjaws S5 32GB (2 x 16GB) DDR5-6000 PC5-48000 CL36 Dual Channel (the Microcenter $499 bundle), a Fractal Design 7xl case,  Corsair 750w PS, and a Samsung 990 Pro 2 TB NVME.

 

I moved over my original 8 drives then added in 2 drives. I added the drives to the array 1 at a time. I also moved over my LSI 9211-8i HBA. Additionally I added a SAS expander.

 

I did not move over my Nvidia GTX 1070 as I want to use the integrated graphics for transcoding.

 

My old system was up for months with no issues.

 

First issue I had was the btfs on the Samsung 990 pro went read only. I read a forum post that said that can happen and ZFS wouldn't have that issue.

 

I upgraded my system from 6.11.5 to 6.12.6 to get the support for ZFS.

 

Since then my system uptime has been a bit over a day before I get the below message in the logs and the CPU usage spikes up. I then lost access to the web interface and the directly connected kvm. Granted it wasn't up for over a day before the upgrade to 6.12.6

 

 

Dec 15 15:54:58 Tower kernel: stack segment: 0000 [#1] PREEMPT SMP NOPTI
Dec 15 15:54:58 Tower kernel: CPU: 0 PID: 1189 Comm: dbuf_evict Tainted: P           O       6.1.64-Unraid #1
Dec 15 15:54:58 Tower kernel: Hardware name: Micro-Star International Co., Ltd. MS-7E06/PRO Z790-P WIFI (MS-7E06), BIOS A.A0 10/27/2023
Dec 15 15:54:58 Tower kernel: RIP: 0010:__mod_memcg_state+0x2f/0x50
Dec 15 15:54:58 Tower kernel: Code: 41 54 41 89 d4 55 48 89 fd 53 89 f3 e8 ff c7 ff ff 84 c0 75 2d 48 c7 c7 9e 3a 0c 82 e8 a5 f4 63 00 48 63 f3 49 63 d4 48 89 ef <48> 8b 85 d0 06 00 00 65 48 01 14 f0 44 89 e6 5b 5d 41 5c e9 5e d4
Dec 15 15:54:58 Tower kernel: RSP: 0018:ffffc90000f1fd78 EFLAGS: 00010046
Dec 15 15:54:58 Tower kernel: RAX: 0000000000000000 RBX: 000000000000002e RCX: 0000000000000004
Dec 15 15:54:58 Tower kernel: RDX: ffffffffffffffff RSI: 000000000000002e RDI: 0100000001000000
Dec 15 15:54:58 Tower kernel: RBP: 0100000001000000 R08: 0000000000000000 R09: ffffffff8297b8b8
Dec 15 15:54:58 Tower kernel: R10: ffffc9004da15010 R11: ffffc9004e209010 R12: 00000000ffffffff
Dec 15 15:54:58 Tower kernel: R13: 0000000000000000 R14: ffff88812ed85380 R15: ffff88813c085000
Dec 15 15:54:58 Tower kernel: FS:  0000000000000000(0000) GS:ffff88885f000000(0000) knlGS:0000000000000000
Dec 15 15:54:58 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 15 15:54:58 Tower kernel: CR2: 0000151edafa8000 CR3: 000000022aa90000 CR4: 0000000000750ef0
Dec 15 15:54:58 Tower kernel: PKRU: 55555554
Dec 15 15:54:58 Tower kernel: Call Trace:
Dec 15 15:54:58 Tower kernel: <TASK>
Dec 15 15:54:58 Tower kernel: ? __die_body+0x1a/0x5c
Dec 15 15:54:58 Tower kernel: ? die+0x30/0x49
Dec 15 15:54:58 Tower kernel: ? do_trap+0x7b/0xfe
Dec 15 15:54:58 Tower kernel: ? do_error_trap+0x6e/0x98
Dec 15 15:54:58 Tower kernel: ? exc_stack_segment+0x35/0x45
Dec 15 15:54:58 Tower kernel: ? asm_exc_stack_segment+0x22/0x30
Dec 15 15:54:58 Tower kernel: ? __mod_memcg_state+0x2f/0x50
Dec 15 15:54:58 Tower kernel: ? __mod_memcg_state+0x26/0x50
Dec 15 15:54:58 Tower kernel: mod_memcg_page_state.constprop.0+0x6a/0x82
Dec 15 15:54:58 Tower kernel: __vunmap+0x146/0x1a3
Dec 15 15:54:58 Tower kernel: spl_slab_reclaim+0x163/0x194 [spl]
Dec 15 15:54:58 Tower kernel: arc_buf_destroy_impl+0x4e/0x1a9 [zfs]
Dec 15 15:54:58 Tower kernel: arc_buf_destroy+0xb2/0xe2 [zfs]
Dec 15 15:54:58 Tower kernel: ? __thread_exit+0x13/0x13 [spl]
Dec 15 15:54:58 Tower kernel: dbuf_destroy+0x30/0x3b8 [zfs]
Dec 15 15:54:58 Tower kernel: ? dbuf_evict_one+0x11c/0x11c [zfs]
Dec 15 15:54:58 Tower kernel: ? __thread_exit+0x13/0x13 [spl]
Dec 15 15:54:58 Tower kernel: dbuf_evict_one+0xff/0x11c [zfs]
Dec 15 15:54:58 Tower kernel: dbuf_evict_thread+0xbb/0x119 [zfs]
Dec 15 15:54:58 Tower kernel: thread_generic_wrapper+0x57/0x65 [spl]
Dec 15 15:54:58 Tower kernel: kthread+0xe4/0xef
Dec 15 15:54:58 Tower kernel: ? kthread_complete_and_exit+0x1b/0x1b
Dec 15 15:54:58 Tower kernel: ret_from_fork+0x1f/0x30
Dec 15 15:54:58 Tower kernel: </TASK>
Dec 15 15:54:58 Tower kernel: Modules linked in: xt_connmark xt_mark iptable_mangle xt_comment iptable_raw wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs md_mod tcp_diag inet_diag jc42 regmap_i2c ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge stp llc zfs(PO) intel_rapl_msr i915 intel_rapl_common x86_pkg_temp_thermal intel_powerclamp zunicode(PO) coretemp zzstd(O) kvm_intel zlua(O) zavl(PO) kvm icp(PO) iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper btusb zcommon(PO) btrtl crct10dif_pclmul btbcm crc32_pclmul btintel znvpair(PO) drm_kms_helper crc32c_intel ghash_clmulni_intel bluetooth sha512_ssse3 spl(O) sha256_ssse3 sha1_ssse3 drm aesni_intel crypto_simd cryptd ecdh_generic input_leds
Dec 15 15:54:58 Tower kernel: led_class joydev ecc i2c_i801 rapl mei_hdcp mei_pxp intel_cstate mpt3sas nvme intel_gtt agpgart mei_me i2c_smbus mxm_wmi wmi_bmof intel_uncore igc ahci nvme_core i2c_core raid_class mei scsi_transport_sas libahci syscopyarea sysfillrect sysimgblt fb_sys_fops thermal fan video tpm_crb tpm_tis tpm_tis_core wmi tpm backlight intel_pmc_core acpi_pad acpi_tad button unix
Dec 15 15:54:58 Tower kernel: ---[ end trace 0000000000000000 ]---
Dec 15 15:54:58 Tower kernel: RIP: 0010:__mod_memcg_state+0x2f/0x50
Dec 15 15:54:58 Tower kernel: Code: 41 54 41 89 d4 55 48 89 fd 53 89 f3 e8 ff c7 ff ff 84 c0 75 2d 48 c7 c7 9e 3a 0c 82 e8 a5 f4 63 00 48 63 f3 49 63 d4 48 89 ef <48> 8b 85 d0 06 00 00 65 48 01 14 f0 44 89 e6 5b 5d 41 5c e9 5e d4
Dec 15 15:54:58 Tower kernel: RSP: 0018:ffffc90000f1fd78 EFLAGS: 00010046
Dec 15 15:54:58 Tower kernel: RAX: 0000000000000000 RBX: 000000000000002e RCX: 0000000000000004
Dec 15 15:54:58 Tower kernel: RDX: ffffffffffffffff RSI: 000000000000002e RDI: 0100000001000000
Dec 15 15:54:58 Tower kernel: RBP: 0100000001000000 R08: 0000000000000000 R09: ffffffff8297b8b8
Dec 15 15:54:58 Tower kernel: R10: ffffc9004da15010 R11: ffffc9004e209010 R12: 00000000ffffffff
Dec 15 15:54:58 Tower kernel: R13: 0000000000000000 R14: ffff88812ed85380 R15: ffff88813c085000
Dec 15 15:54:58 Tower kernel: FS:  0000000000000000(0000) GS:ffff88885f000000(0000) knlGS:0000000000000000
Dec 15 15:54:58 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 15 15:54:58 Tower kernel: CR2: 0000151edafa8000 CR3: 000000022aa90000 CR4: 0000000000750ef0
Dec 15 15:54:58 Tower kernel: PKRU: 55555554
Dec 15 15:54:58 Tower kernel: note: dbuf_evict[1189] exited with irqs disabled
Dec 15 15:56:25 Tower emhttpd: read SMART /dev/sdj
Dec 15 16:25:29 Tower emhttpd: spinning down /dev/sdf
Dec 15 16:34:25 Tower kernel: fbcon: i915drmfb (fb0) is primary device
Dec 15 16:34:25 Tower kernel: Console: switching to colour frame buffer device 480x135
Dec 15 16:34:25 Tower kernel: i915 0000:00:02.0: [drm] fb0: i915drmfb frame buffer device

 

 

Things I have tried from other forum posts, granted not same exact GPF:

1) I ran 4 iterations of MemTest86 with 0 errors

2) I disabled C States in the bios.

2) I switch from macvlan in the docker setup to ipvlan.

 

 

The most recent iteration a few minutes ago. I was able to log in via ssh. I tried using "powerdown -r" but the system would not go down. I ran the cli command to create the diagnostics before trying a few other command to get it to shut down. I finally hit the reset button. 

 

The load average was really high using top.

top - 17:12:20 up 1 day,  5:54,  3 users,  load average: 52.29, 53.26, 50.71

 

 

tower-diagnostics-20231215-1714.zip syslog-previous syslog

The errors in the syslog are from ZFS, could be a hardware issue or a bad pool, try running with just one stick of RAM, if the same try a different one, that will basically rule out a RAM issue.

So far my server has been up 2 days without error and locking up. Prior to that the longest was about 24 hours. 
 

Only change was that I upgraded the firmware on my Samsung 990 pro using the info from this forum post  

 

 

 

 

Edited December 17, 20232 yr by hjgarron

Ungrading the SSD firmware seems to have fixed this issue as it has Benninger a week without the server having the error and becoming unresponsive. After a few days of stability, I reenabled C states to drop the electric usage of the server. 

Update. I posted this as solved too soon. The firmware did help a bit, but the crashing came back on day 3 or so. The general protection fault and then the system would become un responsive. I rebooted moved everything off the Samsung NVME, removed the Samsung, put in the Crucial SSD from before the upgrade, and then used the Crucial SSD for the cache drive. I am at day 4.5 with no issues. I am going to wait a few more days then swap in a 2 TB Western Digital Black NVME that I purchased. As of right now I am not sure if the core issue is the Samsung drive, an issue with my MSI z790-p pro wifi motherboard hardware, or maybe a bios setting. 
 

if the western digital drive works then it is probably just a faulty Samsung drive or firmware issue on the Samsung drive.