hjgarron

I checked the power cable connector was a little loose and able to be wiggled slightly on the drive. I swapped it for a different one that fit snug with no wiggle. The data rebuild just fished up. Thanks.

I bought 4 new 26 tb drives over Black Friday. I successfully put 1 into replace my parity drive then let it rebuild the parity info. I then swapped out a 10 Tb for another of the brand new WD red pro 26 tb drives and then let the parity rebuild with no issues. I swapped out my last 10 tb for another 26 tb and the drive went to disabled 4 hours into the rebuild. I swapped that drive for the last of the new 26 tb and started the rebuild again and this one failed 11 hours into the rebuild. Between these 2 I have already swapped the data cable as I accidentally used the free one I had open for sdn during the first attempt. The second one I used the original sdl that had been in use for the prior 10 tb drive. I have the array started and emulating the missing disk. I am running extended smart test on the sdl disk. 1) should I let that run and then try adding in back in? 2) should I try putting in the 18 or 10 tb drive that I just recently swapped out. tower-diagnostics-20251207-1959.zip

So I have had 15 days of uptime so far. I made a few changes, so I am not sure of the culprit. 1. I follow the below directions from: Crashes related to i915 driver We are aware that some 11th gen Intel Rocket Lake systems are experiencing crashes related to the i915 iGPU. If your Rocket Lake system crashes under Unraid 6.12.0, open a web terminal and type this, then reboot: echo "options i915 enable_dc=0" >> /boot/config/modprobe.d/i915.conf Setting this option may result in higher power use but it may resolve this issue for these GPUs. 2. I disabled c-states in the bios for a while, but after doing the above I reenabled it. So I don't think this was a fix as I was still getting the above error randomly. It was also causing my power draw to be about 50 watts higher. 3. The type of CPU cooler had defaulted to water cooler in the bios. I corrected it to be a fan. 4 I enabled Docker, but disabled all containers but Plex for a week. I did fix the pointer for the GPU rendering to be my igpu from auto as I had a nvidia GPU prior. 5. I started enabling dockers one at a time with a few days in between. I have my main ones started with no issues, but I still have a few disabled. Again my symptoms was the GPF messages then a few days later my system would go unresponsive and need to be reset. Prior uptime was about 5 days.

I am getting a new General Protection Fault on my upgraded server hardware. Previous GPF post was this one I removed the Samsung 990 NVME and moved back to a Crucial SSD. I am getting a new GPF that happens after about 2.5 days of uptime. My system continues to function unlike the last GPF that caused it to become unresponsive. Last time I left it running for about 4 days then I started having issue with dockers and noticed that /mnt/user disappeared. Rebooting restored the missing /mnt/user. I am not sure if they are related yet. I am hopping some can tell me what this GPF error is from so 8 can continue to troubleshoot. Jan 2 12:05:09 Tower kernel: general protection fault, probably for non-canonical address 0xfeff888801254184: 0000 [#1] PREEMPT SMP NOPTI Jan 2 12:05:09 Tower kernel: CPU: 8 PID: 29084 Comm: find Tainted: P O 6.1.64-Unraid #1 Jan 2 12:05:09 Tower kernel: Hardware name: Micro-Star International Co., Ltd. MS-7E06/PRO Z790-P WIFI (MS-7E06), BIOS A.A0 10/27/2023 Jan 2 12:05:09 Tower kernel: RIP: 0010:__d_lookup_rcu+0x56/0xad Jan 2 12:05:09 Tower kernel: Code: 00 02 48 8b 34 24 74 16 59 4c 89 e2 48 89 ef 5b 5d 41 5c 41 5d 41 5e 41 5f e9 ae d9 ff ff 48 8b 18 48 83 e3 fe 48 85 db 74 44 <44> 8b 7b fc 48 39 6b 10 75 35 48 83 7b 08 00 74 2e 4c 39 6b 18 75 Jan 2 12:05:09 Tower kernel: RSP: 0018:ffffc90024e8fc18 EFLAGS: 00010286 Jan 2 12:05:09 Tower kernel: RAX: ffff88885d6cd030 RBX: feff888801254188 RCX: 000000000000000a Jan 2 12:05:09 Tower kernel: RDX: ffffc90024e8fd34 RSI: ffffc90024e8fd00 RDI: 00000000000d9a06 Jan 2 12:05:09 Tower kernel: RBP: ffff888763821b00 R08: 20167ba46d21188c R09: 0000000000000000 Jan 2 12:05:09 Tower kernel: R10: 0033539aefab7e39 R11: 0000000000000fe0 R12: ffffc90024e8fd34 Jan 2 12:05:09 Tower kernel: R13: 0000002036681aef R14: ffff8886b2403020 R15: ffff8886b2403000 Jan 2 12:05:09 Tower kernel: FS: 0000152ca9dc5740(0000) GS:ffff88885f200000(0000) knlGS:0000000000000000 Jan 2 12:05:09 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 2 12:05:09 Tower kernel: CR2: 00000000004f1008 CR3: 000000016d3aa000 CR4: 0000000000750ee0 Jan 2 12:05:09 Tower kernel: PKRU: 55555554 Jan 2 12:05:09 Tower kernel: Call Trace: Jan 2 12:05:09 Tower kernel: <TASK> Jan 2 12:05:09 Tower kernel: ? __die_body+0x1a/0x5c Jan 2 12:05:09 Tower kernel: ? die_addr+0x38/0x51 Jan 2 12:05:09 Tower kernel: ? exc_general_protection+0x30f/0x345 Jan 2 12:05:09 Tower kernel: ? asm_exc_general_protection+0x22/0x30 Jan 2 12:05:09 Tower kernel: ? __d_lookup_rcu+0x56/0xad Jan 2 12:05:09 Tower kernel: ? __d_lookup_rcu+0x2a/0xad Jan 2 12:05:09 Tower kernel: lookup_fast+0x26/0xc0 Jan 2 12:05:09 Tower kernel: walk_component+0x42/0xca Jan 2 12:05:09 Tower kernel: path_lookupat+0x78/0xfe Jan 2 12:05:09 Tower kernel: filename_lookup+0x5f/0xbc Jan 2 12:05:09 Tower kernel: ? slab_free_freelist_hook.constprop.0+0x3b/0xaf Jan 2 12:05:09 Tower kernel: ? slab_post_alloc_hook+0x4d/0x15e Jan 2 12:05:09 Tower kernel: ? kmem_cache_free+0xc9/0x154 Jan 2 12:05:09 Tower kernel: vfs_statx+0x62/0x126 Jan 2 12:05:09 Tower kernel: vfs_fstatat+0x46/0x62 Jan 2 12:05:09 Tower kernel: __do_sys_newfstatat+0x26/0x5c Jan 2 12:05:09 Tower kernel: ? rcu_read_unlock_sched+0xa/0x1f Jan 2 12:05:09 Tower kernel: ? f_dupfd+0x4a/0x53 Jan 2 12:05:09 Tower kernel: ? do_fcntl+0x17c/0x569 Jan 2 12:05:09 Tower kernel: ? fsnotify_find_mark+0x11/0x65 Jan 2 12:05:09 Tower kernel: ? fpregs_assert_state_consistent+0x20/0x44 Jan 2 12:05:09 Tower kernel: ? exit_to_user_mode_prepare+0xd8/0x112 Jan 2 12:05:09 Tower kernel: do_syscall_64+0x68/0x81 Jan 2 12:05:09 Tower kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce Jan 2 12:05:09 Tower kernel: RIP: 0033:0x152ca9ecb1ca Jan 2 12:05:09 Tower kernel: Code: 48 89 f2 b9 00 01 00 00 48 89 fe bf 9c ff ff ff e9 0b 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 90 41 89 ca b8 06 01 00 00 0f 05 <3d> 00 f0 ff ff 77 07 31 c0 c3 0f 1f 40 00 48 8b 15 19 4c 0e 00 f7 Jan 2 12:05:09 Tower kernel: RSP: 002b:00007ffe8121cfd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000106 Jan 2 12:05:09 Tower kernel: RAX: ffffffffffffffda RBX: 000000000045c130 RCX: 0000152ca9ecb1ca Jan 2 12:05:09 Tower kernel: RDX: 000000000045c1a0 RSI: 000000000045c230 RDI: 000000000000000c Jan 2 12:05:09 Tower kernel: RBP: 000000000045c1a0 R08: 0000000000000007 R09: 000000000044d860 Jan 2 12:05:09 Tower kernel: R10: 0000000000000100 R11: 0000000000000246 R12: 000000000045c130 Jan 2 12:05:09 Tower kernel: R13: 000000000044d860 R14: 00007ffe8121ef14 R15: 000000000000000b Jan 2 12:05:09 Tower kernel: </TASK> Jan 2 12:05:09 Tower kernel: Modules linked in: nft_chain_nat nft_compat nf_tables xt_connmark xt_mark iptable_mangle xt_comment iptable_raw wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag jc42 regmap_i2c ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge stp llc i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm iosf_mbi drm_buddy crct10dif_pclmul crc32_pclmul i2c_algo_bit crc32c_intel ghash_clmulni_intel ttm sha512_ssse3 sha256_ssse3 sha1_ssse3 drm_display_helper aesni_intel crypto_simd cryptd drm_kms_helper btusb btrtl rapl btbcm btintel Jan 2 12:05:09 Tower kernel: intel_cstate bluetooth drm mei_hdcp mei_pxp wmi_bmof mxm_wmi intel_gtt input_leds ecdh_generic led_class mpt3sas joydev ecc i2c_i801 agpgart i2c_smbus mei_me intel_uncore igc raid_class i2c_core ahci mei scsi_transport_sas libahci syscopyarea sysfillrect video sysimgblt thermal fb_sys_fops fan tpm_crb tpm_tis tpm_tis_core tpm wmi intel_pmc_core backlight acpi_tad acpi_pad button unix Jan 2 12:05:09 Tower kernel: ---[ end trace 0000000000000000 ]--- Jan 2 12:05:09 Tower kernel: RIP: 0010:__d_lookup_rcu+0x56/0xad Jan 2 12:05:09 Tower kernel: Code: 00 02 48 8b 34 24 74 16 59 4c 89 e2 48 89 ef 5b 5d 41 5c 41 5d 41 5e 41 5f e9 ae d9 ff ff 48 8b 18 48 83 e3 fe 48 85 db 74 44 <44> 8b 7b fc 48 39 6b 10 75 35 48 83 7b 08 00 74 2e 4c 39 6b 18 75 Jan 2 12:05:09 Tower kernel: RSP: 0018:ffffc90024e8fc18 EFLAGS: 00010286 Jan 2 12:05:09 Tower kernel: RAX: ffff88885d6cd030 RBX: feff888801254188 RCX: 000000000000000a Jan 2 12:05:09 Tower kernel: RDX: ffffc90024e8fd34 RSI: ffffc90024e8fd00 RDI: 00000000000d9a06 Jan 2 12:05:09 Tower kernel: RBP: ffff888763821b00 R08: 20167ba46d21188c R09: 0000000000000000 Jan 2 12:05:09 Tower kernel: R10: 0033539aefab7e39 R11: 0000000000000fe0 R12: ffffc90024e8fd34 Jan 2 12:05:09 Tower kernel: R13: 0000002036681aef R14: ffff8886b2403020 R15: ffff8886b2403000 Jan 2 12:05:09 Tower kernel: FS: 0000152ca9dc5740(0000) GS:ffff88885f200000(0000) knlGS:0000000000000000 Jan 2 12:05:09 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 2 12:05:09 Tower kernel: CR2: 00000000004f1008 CR3: 000000016d3aa000 CR4: 0000000000750ee0 Jan 2 12:05:09 Tower kernel: PKRU: 55555554 tower-diagnostics-20240102-1934.zip

Update. I posted this as solved too soon. The firmware did help a bit, but the crashing came back on day 3 or so. The general protection fault and then the system would become un responsive. I rebooted moved everything off the Samsung NVME, removed the Samsung, put in the Crucial SSD from before the upgrade, and then used the Crucial SSD for the cache drive. I am at day 4.5 with no issues. I am going to wait a few more days then swap in a 2 TB Western Digital Black NVME that I purchased. As of right now I am not sure if the core issue is the Samsung drive, an issue with my MSI z790-p pro wifi motherboard hardware, or maybe a bios setting. if the western digital drive works then it is probably just a faulty Samsung drive or firmware issue on the Samsung drive.

Ungrading the SSD firmware seems to have fixed this issue as it has Benninger a week without the server having the error and becoming unresponsive. After a few days of stability, I reenabled C states to drop the electric usage of the server.

So far my server has been up 2 days without error and locking up. Prior to that the longest was about 24 hours. Only change was that I upgraded the firmware on my Samsung 990 pro using the info from this forum post

I recently upgrade the hardware in my server from a 3700K to a 14700K. Upgrades were a new motherboard MSI z790-p Pro Wifi, Intel 13700K, G.Skill Ripjaws S5 32GB (2 x 16GB) DDR5-6000 PC5-48000 CL36 Dual Channel (the Microcenter $499 bundle), a Fractal Design 7xl case, Corsair 750w PS, and a Samsung 990 Pro 2 TB NVME. I moved over my original 8 drives then added in 2 drives. I added the drives to the array 1 at a time. I also moved over my LSI 9211-8i HBA. Additionally I added a SAS expander. I did not move over my Nvidia GTX 1070 as I want to use the integrated graphics for transcoding. My old system was up for months with no issues. First issue I had was the btfs on the Samsung 990 pro went read only. I read a forum post that said that can happen and ZFS wouldn't have that issue. I upgraded my system from 6.11.5 to 6.12.6 to get the support for ZFS. Since then my system uptime has been a bit over a day before I get the below message in the logs and the CPU usage spikes up. I then lost access to the web interface and the directly connected kvm. Granted it wasn't up for over a day before the upgrade to 6.12.6 Dec 15 15:54:58 Tower kernel: stack segment: 0000 [#1] PREEMPT SMP NOPTI Dec 15 15:54:58 Tower kernel: CPU: 0 PID: 1189 Comm: dbuf_evict Tainted: P O 6.1.64-Unraid #1 Dec 15 15:54:58 Tower kernel: Hardware name: Micro-Star International Co., Ltd. MS-7E06/PRO Z790-P WIFI (MS-7E06), BIOS A.A0 10/27/2023 Dec 15 15:54:58 Tower kernel: RIP: 0010:__mod_memcg_state+0x2f/0x50 Dec 15 15:54:58 Tower kernel: Code: 41 54 41 89 d4 55 48 89 fd 53 89 f3 e8 ff c7 ff ff 84 c0 75 2d 48 c7 c7 9e 3a 0c 82 e8 a5 f4 63 00 48 63 f3 49 63 d4 48 89 ef <48> 8b 85 d0 06 00 00 65 48 01 14 f0 44 89 e6 5b 5d 41 5c e9 5e d4 Dec 15 15:54:58 Tower kernel: RSP: 0018:ffffc90000f1fd78 EFLAGS: 00010046 Dec 15 15:54:58 Tower kernel: RAX: 0000000000000000 RBX: 000000000000002e RCX: 0000000000000004 Dec 15 15:54:58 Tower kernel: RDX: ffffffffffffffff RSI: 000000000000002e RDI: 0100000001000000 Dec 15 15:54:58 Tower kernel: RBP: 0100000001000000 R08: 0000000000000000 R09: ffffffff8297b8b8 Dec 15 15:54:58 Tower kernel: R10: ffffc9004da15010 R11: ffffc9004e209010 R12: 00000000ffffffff Dec 15 15:54:58 Tower kernel: R13: 0000000000000000 R14: ffff88812ed85380 R15: ffff88813c085000 Dec 15 15:54:58 Tower kernel: FS: 0000000000000000(0000) GS:ffff88885f000000(0000) knlGS:0000000000000000 Dec 15 15:54:58 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Dec 15 15:54:58 Tower kernel: CR2: 0000151edafa8000 CR3: 000000022aa90000 CR4: 0000000000750ef0 Dec 15 15:54:58 Tower kernel: PKRU: 55555554 Dec 15 15:54:58 Tower kernel: Call Trace: Dec 15 15:54:58 Tower kernel: <TASK> Dec 15 15:54:58 Tower kernel: ? __die_body+0x1a/0x5c Dec 15 15:54:58 Tower kernel: ? die+0x30/0x49 Dec 15 15:54:58 Tower kernel: ? do_trap+0x7b/0xfe Dec 15 15:54:58 Tower kernel: ? do_error_trap+0x6e/0x98 Dec 15 15:54:58 Tower kernel: ? exc_stack_segment+0x35/0x45 Dec 15 15:54:58 Tower kernel: ? asm_exc_stack_segment+0x22/0x30 Dec 15 15:54:58 Tower kernel: ? __mod_memcg_state+0x2f/0x50 Dec 15 15:54:58 Tower kernel: ? __mod_memcg_state+0x26/0x50 Dec 15 15:54:58 Tower kernel: mod_memcg_page_state.constprop.0+0x6a/0x82 Dec 15 15:54:58 Tower kernel: __vunmap+0x146/0x1a3 Dec 15 15:54:58 Tower kernel: spl_slab_reclaim+0x163/0x194 [spl] Dec 15 15:54:58 Tower kernel: arc_buf_destroy_impl+0x4e/0x1a9 [zfs] Dec 15 15:54:58 Tower kernel: arc_buf_destroy+0xb2/0xe2 [zfs] Dec 15 15:54:58 Tower kernel: ? __thread_exit+0x13/0x13 [spl] Dec 15 15:54:58 Tower kernel: dbuf_destroy+0x30/0x3b8 [zfs] Dec 15 15:54:58 Tower kernel: ? dbuf_evict_one+0x11c/0x11c [zfs] Dec 15 15:54:58 Tower kernel: ? __thread_exit+0x13/0x13 [spl] Dec 15 15:54:58 Tower kernel: dbuf_evict_one+0xff/0x11c [zfs] Dec 15 15:54:58 Tower kernel: dbuf_evict_thread+0xbb/0x119 [zfs] Dec 15 15:54:58 Tower kernel: thread_generic_wrapper+0x57/0x65 [spl] Dec 15 15:54:58 Tower kernel: kthread+0xe4/0xef Dec 15 15:54:58 Tower kernel: ? kthread_complete_and_exit+0x1b/0x1b Dec 15 15:54:58 Tower kernel: ret_from_fork+0x1f/0x30 Dec 15 15:54:58 Tower kernel: </TASK> Dec 15 15:54:58 Tower kernel: Modules linked in: xt_connmark xt_mark iptable_mangle xt_comment iptable_raw wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs md_mod tcp_diag inet_diag jc42 regmap_i2c ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge stp llc zfs(PO) intel_rapl_msr i915 intel_rapl_common x86_pkg_temp_thermal intel_powerclamp zunicode(PO) coretemp zzstd(O) kvm_intel zlua(O) zavl(PO) kvm icp(PO) iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper btusb zcommon(PO) btrtl crct10dif_pclmul btbcm crc32_pclmul btintel znvpair(PO) drm_kms_helper crc32c_intel ghash_clmulni_intel bluetooth sha512_ssse3 spl(O) sha256_ssse3 sha1_ssse3 drm aesni_intel crypto_simd cryptd ecdh_generic input_leds Dec 15 15:54:58 Tower kernel: led_class joydev ecc i2c_i801 rapl mei_hdcp mei_pxp intel_cstate mpt3sas nvme intel_gtt agpgart mei_me i2c_smbus mxm_wmi wmi_bmof intel_uncore igc ahci nvme_core i2c_core raid_class mei scsi_transport_sas libahci syscopyarea sysfillrect sysimgblt fb_sys_fops thermal fan video tpm_crb tpm_tis tpm_tis_core wmi tpm backlight intel_pmc_core acpi_pad acpi_tad button unix Dec 15 15:54:58 Tower kernel: ---[ end trace 0000000000000000 ]--- Dec 15 15:54:58 Tower kernel: RIP: 0010:__mod_memcg_state+0x2f/0x50 Dec 15 15:54:58 Tower kernel: Code: 41 54 41 89 d4 55 48 89 fd 53 89 f3 e8 ff c7 ff ff 84 c0 75 2d 48 c7 c7 9e 3a 0c 82 e8 a5 f4 63 00 48 63 f3 49 63 d4 48 89 ef <48> 8b 85 d0 06 00 00 65 48 01 14 f0 44 89 e6 5b 5d 41 5c e9 5e d4 Dec 15 15:54:58 Tower kernel: RSP: 0018:ffffc90000f1fd78 EFLAGS: 00010046 Dec 15 15:54:58 Tower kernel: RAX: 0000000000000000 RBX: 000000000000002e RCX: 0000000000000004 Dec 15 15:54:58 Tower kernel: RDX: ffffffffffffffff RSI: 000000000000002e RDI: 0100000001000000 Dec 15 15:54:58 Tower kernel: RBP: 0100000001000000 R08: 0000000000000000 R09: ffffffff8297b8b8 Dec 15 15:54:58 Tower kernel: R10: ffffc9004da15010 R11: ffffc9004e209010 R12: 00000000ffffffff Dec 15 15:54:58 Tower kernel: R13: 0000000000000000 R14: ffff88812ed85380 R15: ffff88813c085000 Dec 15 15:54:58 Tower kernel: FS: 0000000000000000(0000) GS:ffff88885f000000(0000) knlGS:0000000000000000 Dec 15 15:54:58 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Dec 15 15:54:58 Tower kernel: CR2: 0000151edafa8000 CR3: 000000022aa90000 CR4: 0000000000750ef0 Dec 15 15:54:58 Tower kernel: PKRU: 55555554 Dec 15 15:54:58 Tower kernel: note: dbuf_evict[1189] exited with irqs disabled Dec 15 15:56:25 Tower emhttpd: read SMART /dev/sdj Dec 15 16:25:29 Tower emhttpd: spinning down /dev/sdf Dec 15 16:34:25 Tower kernel: fbcon: i915drmfb (fb0) is primary device Dec 15 16:34:25 Tower kernel: Console: switching to colour frame buffer device 480x135 Dec 15 16:34:25 Tower kernel: i915 0000:00:02.0: [drm] fb0: i915drmfb frame buffer device Things I have tried from other forum posts, granted not same exact GPF: 1) I ran 4 iterations of MemTest86 with 0 errors 2) I disabled C States in the bios. 2) I switch from macvlan in the docker setup to ipvlan. The most recent iteration a few minutes ago. I was able to log in via ssh. I tried using "powerdown -r" but the system would not go down. I ran the cli command to create the diagnostics before trying a few other command to get it to shut down. I finally hit the reset button. The load average was really high using top. top - 17:12:20 up 1 day, 5:54, 3 users, load average: 52.29, 53.26, 50.71 tower-diagnostics-20231215-1714.zip syslog-previous syslog