Ever since upgrading to 7.1.x I have been having issues with random server hangs where I cannot ssh to server to get to the GUI. Also the server is not responding to ping either.

I noticed the following call-trace in the rsyslog this tainted error happens while Parity sync is running:

Jun 27 06:31:43 Star-Command kernel: sd 12:0:3:0: Power-on or device reset occurred
Jun 27 06:35:33 Star-Command kernel: kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
Jun 27 06:35:33 Star-Command kernel: BUG: unable to handle page fault for address: ffff88815cb58400
Jun 27 06:35:33 Star-Command kernel: #PF: supervisor instruction fetch in kernel mode
Jun 27 06:35:33 Star-Command kernel: #PF: error_code(0x0011) - permissions violation
Jun 27 06:35:33 Star-Command kernel: PGD 6001067 P4D 6001067 PUD 14ab26063 PMD 113fee063 PTE 800000015cb58163
Jun 27 06:35:33 Star-Command kernel: Oops: Oops: 0011 [#1] PREEMPT SMP NOPTI
Jun 27 06:35:33 Star-Command kernel: CPU: 9 UID: 0 PID: 7969 Comm: unraidd0 Tainted: P           O       6.12.24-Unraid #1
Jun 27 06:35:33 Star-Command kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
Jun 27 06:35:33 Star-Command kernel: Hardware name: Micro-Star International Co., Ltd. MS-7E25/MAG Z790 TOMAHAWK MAX WIFI (MS-7E25), BIOS A.80 09/27/2024
Jun 27 06:35:33 Star-Command kernel: RIP: 0010:0xffff88815cb58400
Jun 27 06:35:33 Star-Command kernel: Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 1e 2a 13 81 88 ff ff 00 67 84 ff ff e8 ff ff 00 d4 b7 56 81 88
Jun 27 06:35:33 Star-Command kernel: RSP: 0018:ffffc90004733c20 EFLAGS: 00010246
Jun 27 06:35:33 Star-Command kernel: RAX: 0000000000000001 RBX: ffff88815cb58400 RCX: ffff888114fc4200
Jun 27 06:35:33 Star-Command kernel: RDX: 0000000000000000 RSI: ffff88817ab8bf80 RDI: ffff88815cb58400
Jun 27 06:35:33 Star-Command kernel: RBP: ffff88817ab8bf80 R08: 0000000000000000 R09: ffff88817ab8bf80
Jun 27 06:35:33 Star-Command kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8881132a1e00
Jun 27 06:35:33 Star-Command kernel: R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000001
Jun 27 06:35:33 Star-Command kernel: FS:  0000000000000000(0000) GS:ffff889fff240000(0000) knlGS:0000000000000000
Jun 27 06:35:33 Star-Command kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 27 06:35:33 Star-Command kernel: CR2: ffff88815cb58400 CR3: 0000000803ef8006 CR4: 0000000000772ef0
Jun 27 06:35:33 Star-Command kernel: PKRU: 55555554
Jun 27 06:35:33 Star-Command kernel: Call Trace:
Jun 27 06:35:33 Star-Command kernel: <TASK>
Jun 27 06:35:33 Star-Command kernel: ? blk_attempt_bio_merge+0x3a/0xe0
Jun 27 06:35:33 Star-Command kernel: ? blk_attempt_plug_merge+0x38/0x60
Jun 27 06:35:33 Star-Command kernel: ? blk_mq_submit_bio+0x25e/0x530
Jun 27 06:35:33 Star-Command kernel: ? __submit_bio+0x3e/0xe0
Jun 27 06:35:33 Star-Command kernel: ? submit_bio_noacct_nocheck+0x152/0x2e0
Jun 27 06:35:33 Star-Command kernel: ? unraidd+0x11eb/0x1280 [md_mod]
Jun 27 06:35:33 Star-Command kernel: ? do_raw_spin_lock+0x14/0x20
Jun 27 06:35:33 Star-Command kernel: ? rawspin_lock_irqsave+0x1f/0x30
Jun 27 06:35:33 Star-Command kernel: ? md_thread+0xf6/0x130 [md_mod]
Jun 27 06:35:33 Star-Command kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Jun 27 06:35:33 Star-Command kernel: ? __pfx_md_thread+0x10/0x10 [md_mod]
Jun 27 06:35:33 Star-Command kernel: ? kthread+0xec/0x100
Jun 27 06:35:33 Star-Command kernel: ? __pfx_kthread+0x10/0x10
Jun 27 06:35:33 Star-Command kernel: ? ret_from_fork+0x21/0x40
Jun 27 06:35:33 Star-Command kernel: ? __pfx_kthread+0x10/0x10
Jun 27 06:35:33 Star-Command kernel: ? ret_from_fork_asm+0x1a/0x30
Jun 27 06:35:33 Star-Command kernel: </TASK>
Jun 27 06:35:33 Star-Command kernel: Modules linked in: xt_connmark xt_mark iptable_mangle xt_comment iptable_raw wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha veth ipvlan nf_conntrack_netlink xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nfnetlink xfrm_user xfrm_algo ip6table_nat iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype md_mod zfs(PO) spl(O) ntfs3 tcp_diag inet_diag nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge stp llc qrtr_mhi xe drm_gpuvm drm_exec gpu_sched drm_ttm_helper drm_suballoc_helper intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp i915 kvm_intel qrtr kvm ath12k iosf_mbi drm_buddy mhi crct10dif_pclmul ttm crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 qmi_helpers i2c_algo_bit drm_display_helper sha256_ssse3 sha1_ssse3 mac80211 btusb aesni_intel drm_kms_helper btrtl crypto_simd btbcm cryptd btintel bluetooth drm rapl mei_hdcp mei_pxp
Jun 27 06:35:33 Star-Command kernel: joydev input_leds nvme cfg80211 intel_cstate mpt3sas mei_me wmi_bmof intel_gtt intel_uncore mxm_wmi igc i2c_i801 nvme_core agpgart mei i2c_smbus raid_class ahci tpm_crb rfkill scsi_transport_sas i2c_core tpm_tis libahci libarc4 tpm_tis_core led_class tpm thermal fan video libaescfb ecdh_generic wmi ecc backlight acpi_tad acpi_pad button
Jun 27 06:35:33 Star-Command kernel: CR2: ffff88815cb58400

I also noticed this nginx entry in the logs:

Jun 27 02:04:15 Star-Command nginx: 2025/06/27 02:04:15 [crit] 6311#6311: ngx_slab_alloc() failed: no memory
Jun 27 02:04:15 Star-Command nginx: 2025/06/27 02:04:15 [error] 6311#6311: shpool alloc failed
Jun 27 02:04:15 Star-Command nginx: 2025/06/27 02:04:15 [error] 6311#6311: nchan: Out of shared memory while allocating message of size 13022. Increase nchan_max_reserved_memory.
Jun 27 02:04:15 Star-Command nginx: 2025/06/27 02:04:15 [error] 6311#6311: *88661 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
Jun 27 02:04:15 Star-Command nginx: 2025/06/27 02:04:15 [error] 6311#6311: MEMSTORE:00: can't create shared message for channel /devices
Jun 27 02:04:15 Star-Command webgui: Nchan out of shared memory.  Reloading nginx
Jun 27 02:04:15 Star-Command rc.nginx: Restarting Nginx server daemon...
Jun 27 02:04:15 Star-Command rc.nginx: Checking configuration for correct syntax and then trying to open files referenced in configuration...
Jun 27 02:04:15 Star-Command rc.nginx: /usr/sbin/nginx -t -c /etc/nginx/nginx.conf
Jun 27 02:04:15 Star-Command rc.nginx: Stopping Nginx server daemon gracefully...
Jun 27 02:04:27 Star-Command nginx: 2025/06/27 02:04:27 [error] 6311#6311: MEMSTORE:00: force-reaping msg with refcount 1
### [PREVIOUS LINE REPEATED 11520 TIMES] ###
Jun 27 02:04:27 Star-Command rc.nginx: Nginx server daemon...  Stopped.
Jun 27 02:04:27 Star-Command rc.nginx: Starting Nginx server daemon...
Jun 27 02:04:27 Star-Command rc.nginx: Nginx server daemon...  Started.
Jun 27 02:04:59 Star-Command monitor_nchan: Stop running nchan processes
Jun 27 03:00:05 Star-Command shfs: copy_file: //..y/...

I'm not sure if these 2 issues are part of a bigger issue or if they are unrelated. It takes an unclean shutdown to get the server back to a state where it is usable. However, the server only stays usable for a short time before it becomes inaccessible again. I just replaced the CPU and the Motherboard in the system so I'm not confident that either of those is the issue? Is this a bug in unraid 7? would a recommended fix here be to try to downgrade to unraid 6.12.15?

Diagnostic zip attached

star-command-diagnostics-20250627-0905.zip

Edited June 27, 20251 yr by dabbler455
added some more information

I downgraded from 7.2.4 to 6.12.15. I'll report tomorrow whether my issue still persists.

Unraid driver was crashing, that is almost always a hardware issue, most often bad RAM, or the CPU, when a 13700/13900/14700 or 14900 is being used, due to the Intel 13/14 gen issue.

I pulled 2 ram sticks out. I'll let it sit for a couple of days and see if the issue persists

After swapping memory everything was fine for a few days then the issues slowly crept back. In doing further investigation it looks like Intel has released new microcode for 13th and 14th gen CPUs as of 5/16/2025

CPU Microcode 0x12C

Update Intel uCode to 0x12F

I am updating the bios with new microcode and will report back in a few days if this fixes the 13th/14th gen issue. For anyone looking at this thread I recommend not going with newer generation Intel unless you are a glutton for punishment.

After a few days of running with the new microcode the system kernel panicked again. Time to replace the CPU for a second time.