Hello,
I have been running unraid for a few years now.
Last year I upgraded my hardware to a 13900K intel CPU running on a Gigabyte Z790 AORUS Elite AX motherboard and 64GB ddr5 memory.
All was running great until recently I started getting locked out of my server with nothing I can do but hard reset.
I tried reinstalling unraid's flash drive and copied over my config folder.
I rebuilt all my dockers, formatted the cache drive to ZFS and basically tried every suggestion in this forum before writing this post.
My server keeps freezing randomly and doesn't last more than few hours.
I gave TrueNas a shot on the same hardware (different disks though) and it seems to be running fine.
I am used to Unraid and I wish I can stay with it instead of rebuilding everything in TrueNas.
Can anybody give me a new direction to investigate before I give up?
Attached are my logs:

notice the line saying:
x86/split lock detection: #AC: crashing the kernel on kernel split_locks and warning on user-space split_locks
syslogsyslog-previousunbalance.log

Dec  6 05:30:55 HV-Unraid kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Dec  6 05:30:55 HV-Unraid kernel: ? _raw_spin_unlock+0x14/0x29
Dec  6 05:30:55 HV-Unraid kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

Macvlan call traces will usually end up crashing the server, switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)), then reboot.

I Have Used Macvlan for a long time, I read that it can cause issues and when I rebuilt my dockers I set it to ipvlan.
for some reason It doesn't persist.
Maybe that is the issue?

It's still set to ipVlan.
so why does it keep trying MacVlan?

Did you reboot after changing to ipvlan?

Yes, Multiple times unfortunatelly.
this is the current syslog (with ipvlan, from the screenshot above)
still getting "x86/split lock detection: #AC: crashing the kernel on kernel split_locks and warning on user-space split_locks"
no macvlan messages in this log.
 

syslog

Post the output of:

docker network ls

 

NETWORK ID     NAME      DRIVER    SCOPE
dd4cab41206e   br0       ipvlan    local
18215c076db7   bridge    bridge    local
57f7a15f2250   host      host      local
4ef795b60b3f   none      null      local

Thank you BTW 💙

 

The current log is spammed with messages that were'nt there before:

Dec  6 11:55:29 HV-Unraid kernel: [drm:nv_drm_gem_prime_fence_attach_ioctl [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000100] Padding fields must be zeroed
Dec  6 11:55:29 HV-Unraid kernel: [drm:nv_drm_gem_prime_fence_attach_ioctl [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000100] Padding fields must be zeroed
Dec  6 11:55:29 HV-Unraid kernel: [drm:nv_drm_gem_prime_fence_attach_ioctl [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000100] Padding fields must be zeroed
Dec  6 11:55:29 HV-Unraid kernel: [drm:nv_drm_gem_prime_fence_attach_ioctl [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000100] Padding fields must be zeroed
Dec  6 11:55:31 HV-Unraid kernel: usb 2-7: reset SuperSpeed USB device number 2 using xhci_hcd
Dec  6 11:55:31 HV-Unraid kernel: usb 2-7: LPM exit latency is zeroed, disabling LPM.
Dec  6 12:07:50 HV-Unraid kernel: BUG: unable to handle page fault for address: ffffffffffffffc7
Dec  6 12:07:50 HV-Unraid kernel: #PF: supervisor write access in kernel mode
Dec  6 12:07:50 HV-Unraid kernel: #PF: error_code(0x0002) - not-present page
Dec  6 12:07:50 HV-Unraid kernel: PGD 420e067 P4D 420e067 PUD 4210067 PMD 0 
Dec  6 12:07:50 HV-Unraid kernel: Oops: 0002 [#1] PREEMPT SMP NOPTI
Dec  6 12:07:50 HV-Unraid kernel: CPU: 4 PID: 27741 Comm: frigate.capture Tainted: P           O       6.1.64-Unraid #1
Dec  6 12:07:50 HV-Unraid kernel: Hardware name: Gigabyte Technology Co., Ltd. Z790 AORUS ELITE AX/Z790 AORUS ELITE AX, BIOS F10b 11/09/2023
Dec  6 12:07:50 HV-Unraid kernel: RIP: 0010:preempt_latency_start+0x1e/0x46
Dec  6 12:07:50 HV-Unraid kernel: Code: 00 00 00 74 02 eb e2 c3 cc cc cc cc 65 8b 05 0e 69 f8 7e 25 ff ff ff 7f 39 c7 75 31 53 48 8b 5c 24 08 48 89 df e8 cf 39 02 00 <85> c0 74 09 31 ff 31 db e8 c2 39 02 00 65 48 8b 04 25 c0 cb 01 00
Dec  6 12:07:50 HV-Unraid kernel: RSP: 0000:ffffc90021107d80 EFLAGS: 00010293
Dec  6 12:07:50 HV-Unraid kernel: RAX: 0000000000000000 RBX: ffffffff810b9041 RCX: 0000000004ed7e81
Dec  6 12:07:50 HV-Unraid kernel: RDX: 0000000000000004 RSI: 0000000000000000 RDI: ffffffff810b9041
Dec  6 12:07:50 HV-Unraid kernel: RBP: ffff88875d9b3478 R08: 0000160000000000 R09: ffff888000000000
Dec  6 12:07:50 HV-Unraid kernel: R10: 0000000000000402 R11: ffffc90021107c38 R12: ffff888150e03400
Dec  6 12:07:50 HV-Unraid kernel: R13: ffffea0020f3f5c0 R14: ffff8887c8439360 R15: 0000000000000000
Dec  6 12:07:50 HV-Unraid kernel: FS:  000014745cdf76c0(0000) GS:ffff88907f500000(0000) knlGS:0000000000000000
Dec  6 12:07:50 HV-Unraid kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  6 12:07:50 HV-Unraid kernel: CR2: ffffffffffffffc7 CR3: 0000000521e4e000 CR4: 0000000000750ee0
Dec  6 12:07:50 HV-Unraid kernel: PKRU: 55555554
Dec  6 12:07:50 HV-Unraid kernel: Call Trace:
Dec  6 12:07:50 HV-Unraid kernel: <TASK>
Dec  6 12:07:50 HV-Unraid kernel: ? __die_body+0x1a/0x5c
Dec  6 12:07:50 HV-Unraid kernel: ? page_fault_oops+0x329/0x376
Dec  6 12:07:50 HV-Unraid kernel: ? fixup_exception+0x22/0x24b
Dec  6 12:07:50 HV-Unraid kernel: ? exc_page_fault+0xf4/0x11d
Dec  6 12:07:50 HV-Unraid kernel: ? asm_exc_page_fault+0x22/0x30
Dec  6 12:07:50 HV-Unraid kernel: ? up_read+0x13/0x5d
Dec  6 12:07:50 HV-Unraid kernel: ? up_read+0x13/0x5d
Dec  6 12:07:50 HV-Unraid kernel: ? preempt_latency_start+0x1e/0x46
Dec  6 12:07:50 HV-Unraid kernel: up_read+0x13/0x5d
Dec  6 12:07:50 HV-Unraid kernel: fault_dirty_shared_page+0xa1/0xca
Dec  6 12:07:50 HV-Unraid kernel: __handle_mm_fault+0xc9c/0xcf9
Dec  6 12:07:50 HV-Unraid kernel: handle_mm_fault+0x13d/0x20f
Dec  6 12:07:50 HV-Unraid kernel: do_user_addr_fault+0x2c3/0x48d
Dec  6 12:07:50 HV-Unraid kernel: exc_page_fault+0xfb/0x11d
Dec  6 12:07:50 HV-Unraid kernel: asm_exc_page_fault+0x22/0x30
Dec  6 12:07:50 HV-Unraid kernel: RIP: 0033:0x147573c159a7
Dec  6 12:07:50 HV-Unraid kernel: Code: 10 00 00 48 83 ee 80 c5 fd e7 07 c5 fd e7 4f 20 c5 fd e7 57 40 c5 fd e7 5f 60 c5 fd e7 a7 00 10 00 00 c5 fd e7 af 20 10 00 00 <c5> fd e7 b7 40 10 00 00 c5 fd e7 bf 60 10 00 00 48 83 ef 80 ff c9
Dec  6 12:07:50 HV-Unraid kernel: RSP: 002b:000014745cdf6698 EFLAGS: 00010203
Dec  6 12:07:50 HV-Unraid kernel: RAX: 000014743b1ef000 RBX: 0000000000000000 RCX: 0000000000000001
Dec  6 12:07:50 HV-Unraid kernel: RDX: 0000000000001fc0 RSI: 0000147444380480 RDI: 000014743b55ffc0
Dec  6 12:07:50 HV-Unraid kernel: RBP: 000014745cdf66d0 R08: ffffffffffffffc0 R09: 000014744400f440
Dec  6 12:07:50 HV-Unraid kernel: R10: 00000000000001cb R11: 0000000001980000 R12: 000014745cdf6740
Dec  6 12:07:50 HV-Unraid kernel: R13: 000014752c273798 R14: 0000000000000001 R15: 0000000000000000
Dec  6 12:07:50 HV-Unraid kernel: </TASK>
Dec  6 12:07:50 HV-Unraid kernel: Modules linked in: udp_diag veth xt_nat nvidia_uvm(PO) xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap ipvlan xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs md_mod tcp_diag inet_diag nct6775_core hwmon_vid apex(O) gasket(O) ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc bonding tls nvidia_drm(PO) nvidia_modeset(PO) zfs(PO) zunicode(PO) zzstd(O) zlua(O) intel_rapl_msr intel_rapl_common zavl(PO) icp(PO) x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel zcommon(PO) znvpair(PO) i915 kvm btusb btrtl nvidia(PO) btbcm btintel crct10dif_pclmul spl(O) bluetooth crc32_pclmul crc32c_intel iosf_mbi ghash_clmulni_intel drm_buddy sha512_ssse3 i2c_algo_bit sha256_ssse3 ttm sha1_ssse3 aesni_intel drm_display_helper
Dec  6 12:07:50 HV-Unraid kernel: input_leds joydev crypto_simd led_class ecdh_generic cryptd cdc_acm drm_kms_helper ecc i2c_i801 intel_gtt rapl mei_hdcp mei_pxp intel_cstate nvme drm gigabyte_wmi wmi_bmof agpgart i2c_smbus intel_uncore mei_me r8169 nvme_core ahci i2c_core mei libahci realtek syscopyarea sysfillrect sysimgblt fb_sys_fops thermal fan tpm_crb video tpm_tis tpm_tis_core wmi tpm backlight intel_pmc_core acpi_tad acpi_pad button unix
Dec  6 12:07:50 HV-Unraid kernel: CR2: ffffffffffffffc7
Dec  6 12:07:50 HV-Unraid kernel: ---[ end trace 0000000000000000 ]---
Dec  6 12:07:50 HV-Unraid kernel: RIP: 0010:preempt_latency_start+0x1e/0x46
Dec  6 12:07:50 HV-Unraid kernel: Code: 00 00 00 74 02 eb e2 c3 cc cc cc cc 65 8b 05 0e 69 f8 7e 25 ff ff ff 7f 39 c7 75 31 53 48 8b 5c 24 08 48 89 df e8 cf 39 02 00 <85> c0 74 09 31 ff 31 db e8 c2 39 02 00 65 48 8b 04 25 c0 cb 01 00
Dec  6 12:07:50 HV-Unraid kernel: RSP: 0000:ffffc90021107d80 EFLAGS: 00010293
Dec  6 12:07:50 HV-Unraid kernel: RAX: 0000000000000000 RBX: ffffffff810b9041 RCX: 0000000004ed7e81
Dec  6 12:07:50 HV-Unraid kernel: RDX: 0000000000000004 RSI: 0000000000000000 RDI: ffffffff810b9041
Dec  6 12:07:50 HV-Unraid kernel: RBP: ffff88875d9b3478 R08: 0000160000000000 R09: ffff888000000000
Dec  6 12:07:50 HV-Unraid kernel: R10: 0000000000000402 R11: ffffc90021107c38 R12: ffff888150e03400
Dec  6 12:07:50 HV-Unraid kernel: R13: ffffea0020f3f5c0 R14: ffff8887c8439360 R15: 0000000000000000
Dec  6 12:07:50 HV-Unraid kernel: FS:  000014745cdf76c0(0000) GS:ffff88907f500000(0000) knlGS:0000000000000000
Dec  6 12:07:50 HV-Unraid kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  6 12:07:50 HV-Unraid kernel: CR2: ffffffffffffffc7 CR3: 0000000521e4e000 CR4: 0000000000750ee0
Dec  6 12:07:50 HV-Unraid kernel: PKRU: 55555554
Dec  6 12:07:50 HV-Unraid kernel: note: frigate.capture[27741] exited with irqs disabled
Dec  6 12:07:50 HV-Unraid kernel: note: frigate.capture[27741] exited with preempt_count 1
Dec  6 12:17:01 HV-Unraid kernel: veth62e90ad: renamed from eth0
Dec  6 12:17:01 HV-Unraid kernel: eth0: renamed from veth01a575d
Dec  6 12:23:44 HV-Unraid kernel: usb 1-11.2: USB disconnect, device number 4
Dec  6 12:23:44 HV-Unraid kernel: usb 1-11.2: new full-speed USB device number 9 using xhci_hcd
Dec  6 12:23:44 HV-Unraid kernel: cdc_acm 1-11.2:1.0: ttyACM0: USB ACM device
Dec  6 12:23:48 HV-Unraid kernel: usb 1-11.2: USB disconnect, device number 9
Dec  6 12:23:48 HV-Unraid kernel: usb 1-11.2: new full-speed USB device number 10 using xhci_hcd
Dec  6 12:23:48 HV-Unraid kernel: cdc_acm 1-11.2:1.0: ttyACM0: USB ACM device

another possible option is that in the release notes for Version 6.12.6 there is a known issue:
 

Quote

If you are using a Realtek 8125 2.5GbE NIC there is an issue with the 'stock' r8169 Linux driver that causes system hang if you enable jumbo frames. Either do not use jumbo frames or consider installing the Realtek vendor-supplied r8125 driver plugin. (FYI: the stock 'r8169' driver also handles the Realtek 8125 chip sets - there is no separate 'stock' r8125 driver, yes this is confusing).

This is my ethernet controller:
 

IOMMU group 15:[10ec:8125] 04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)

Any idea how I can replace the driver mentioned in the release notes?

56 minutes ago, Hananv said:

NETWORK ID     NAME      DRIVER    SCOPE

macvlan is no longer in use, so I would expect no more call traces about that.

 

54 minutes ago, Hananv said:

Dec  6 12:07:50 HV-Unraid kernel: BUG: unable to handle page fault for address: ffffffffffffffc7

No idea what this is about, it's not macvlan.

 

41 minutes ago, Hananv said:

Any idea how I can replace the driver mentioned in the release notes?

Note that this is only a problem if you are using jumbo frames with the stock driver, the driver plugin can still use it.