HI,

 

I see the below in my log: 

 

Nov 19 09:10:34 Svalbard kernel: kernel BUG at drivers/md/unraid.c:1617!
Nov 19 09:10:34 Svalbard kernel: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
Nov 19 09:10:34 Svalbard kernel: CPU: 19 PID: 12370 Comm: unraidd0 Tainted: P           O       6.1.106-Unraid #1
Nov 19 09:10:34 Svalbard kernel: Hardware name: Gigabyte Technology Co., Ltd. Z790M AORUS ELITE AX/Z790M AORUS ELITE AX, BIOS F10 09/27/2024
Nov 19 09:10:34 Svalbard kernel: RIP: 0010:unraidd+0x1051/0x1140 [md_mod]
Nov 19 09:10:34 Svalbard kernel: Code: 00 83 3d 83 50 00 00 03 7e 16 41 8b 56 98 89 e9 48 c7 c7 21 23 47 a0 48 8b 73 20 e8 82 7e 3f e1 41 f6 86 69 ff ff ff 02 75 02 <0f> 0b 48 8b 43 20 49 03 47 10 41 c7 46 b0 00 10 00 00 49 8b 56 10
Nov 19 09:10:34 Svalbard kernel: RSP: 0018:ffffc9000c36fdf0 EFLAGS: 00010246
Nov 19 09:10:34 Svalbard kernel: RAX: 0000000000000000 RBX: ffff88815a213658 RCX: 0000000000000000
Nov 19 09:10:34 Svalbard kernel: RDX: 0000000000000000 RSI: ffffffff829f0720 RDI: ffff888146ea5e38
Nov 19 09:10:34 Svalbard kernel: RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000
Nov 19 09:10:34 Svalbard kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8881479e4110
Nov 19 09:10:34 Svalbard kernel: R13: ffff88815a213850 R14: ffff88815a2138c8 R15: ffff8881395a12d8
Nov 19 09:10:34 Svalbard kernel: FS:  0000000000000000(0000) GS:ffff88907f8c0000(0000) knlGS:0000000000000000
Nov 19 09:10:34 Svalbard kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 19 09:10:34 Svalbard kernel: CR2: 0000148fe25b3010 CR3: 000000000420a005 CR4: 0000000000770ee0
Nov 19 09:10:34 Svalbard kernel: PKRU: 55555554
Nov 19 09:10:34 Svalbard kernel: Call Trace:
Nov 19 09:10:34 Svalbard kernel: <TASK>
Nov 19 09:10:34 Svalbard kernel: ? __die_body+0x1a/0x5c
Nov 19 09:10:34 Svalbard kernel: ? die+0x30/0x49
Nov 19 09:10:34 Svalbard kernel: ? do_trap+0x7b/0xfe
Nov 19 09:10:34 Svalbard kernel: ? unraidd+0x1051/0x1140 [md_mod]
Nov 19 09:10:34 Svalbard kernel: ? unraidd+0x1051/0x1140 [md_mod]
Nov 19 09:10:34 Svalbard kernel: ? do_error_trap+0x6e/0x98
Nov 19 09:10:34 Svalbard kernel: ? unraidd+0x1051/0x1140 [md_mod]
Nov 19 09:10:34 Svalbard kernel: ? exc_invalid_op+0x4c/0x60
Nov 19 09:10:34 Svalbard kernel: ? unraidd+0x1051/0x1140 [md_mod]
Nov 19 09:10:34 Svalbard kernel: ? asm_exc_invalid_op+0x16/0x20
Nov 19 09:10:34 Svalbard kernel: ? unraidd+0x1051/0x1140 [md_mod]
Nov 19 09:10:34 Svalbard kernel: md_thread+0xf4/0x122 [md_mod]
Nov 19 09:10:34 Svalbard kernel: ? _raw_spin_rq_lock_irqsave+0x20/0x20
Nov 19 09:10:34 Svalbard kernel: ? signal_pending+0x1d/0x1d [md_mod]
Nov 19 09:10:34 Svalbard kernel: kthread+0xe4/0xef
Nov 19 09:10:34 Svalbard kernel: ? kthread_complete_and_exit+0x1b/0x1b
Nov 19 09:10:34 Svalbard kernel: ret_from_fork+0x1f/0x30
Nov 19 09:10:34 Svalbard kernel: </TASK>
Nov 19 09:10:34 Svalbard kernel: Modules linked in: ipvlan wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha tun nft_compat nf_tables xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs md_mod tcp_diag inet_diag it87(O) hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs 8021q garp mrp bridge stp llc bonding tls r8125(O) zfs(PO) i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp zunicode(PO) zzstd(O) kvm_intel iosf_mbi drm_buddy i2c_algo_bit ttm btusb btrtl btbcm zlua(O) drm_display_helper btintel zavl(PO) kvm icp(PO) drm_kms_helper bluetooth drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 ecdh_generic sha1_ssse3 ecc ch341 aesni_intel usbserial zcommon(PO) crypto_simd intel_gtt
Nov 19 09:10:34 Svalbard kernel: cryptd znvpair(PO) rapl spl(O) mei_pxp mei_hdcp intel_cstate gigabyte_wmi wmi_bmof mpt3sas intel_uncore nvme i2c_i801 agpgart mei_me i2c_smbus i2c_core ahci nvme_core mei libahci raid_class scsi_transport_sas syscopyarea sysfillrect sysimgblt fb_sys_fops thermal fan tpm_crb tpm_tis video tpm_tis_core tpm wmi backlight intel_pmc_core acpi_tad acpi_pad button unix [last unloaded: r8125(O)]
Nov 19 09:10:34 Svalbard kernel: ---[ end trace 0000000000000000 ]---
Nov 19 09:10:34 Svalbard kernel: RIP: 0010:unraidd+0x1051/0x1140 [md_mod]
Nov 19 09:10:34 Svalbard kernel: Code: 00 83 3d 83 50 00 00 03 7e 16 41 8b 56 98 89 e9 48 c7 c7 21 23 47 a0 48 8b 73 20 e8 82 7e 3f e1 41 f6 86 69 ff ff ff 02 75 02 <0f> 0b 48 8b 43 20 49 03 47 10 41 c7 46 b0 00 10 00 00 49 8b 56 10
Nov 19 09:10:34 Svalbard kernel: RSP: 0018:ffffc9000c36fdf0 EFLAGS: 00010246
Nov 19 09:10:34 Svalbard kernel: RAX: 0000000000000000 RBX: ffff88815a213658 RCX: 0000000000000000
Nov 19 09:10:34 Svalbard kernel: RDX: 0000000000000000 RSI: ffffffff829f0720 RDI: ffff888146ea5e38
Nov 19 09:10:34 Svalbard kernel: RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000
Nov 19 09:10:34 Svalbard kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8881479e4110
Nov 19 09:10:34 Svalbard kernel: R13: ffff88815a213850 R14: ffff88815a2138c8 R15: ffff8881395a12d8
Nov 19 09:10:34 Svalbard kernel: FS:  0000000000000000(0000) GS:ffff88907f8c0000(0000) knlGS:0000000000000000
Nov 19 09:10:34 Svalbard kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 19 09:10:34 Svalbard kernel: CR2: 0000148fe25b3010 CR3: 00000008cc37c006 CR4: 0000000000770ee0
Nov 19 09:10:34 Svalbard kernel: PKRU: 55555554
Nov 19 09:10:34 Svalbard kernel: ------------[ cut here ]------------
Nov 19 09:10:34 Svalbard kernel: WARNING: CPU: 19 PID: 12370 at kernel/exit.c:816 do_exit+0x87/0x923
Nov 19 09:10:34 Svalbard kernel: Modules linked in: ipvlan wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha tun nft_compat nf_tables xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs md_mod tcp_diag inet_diag it87(O) hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs 8021q garp mrp bridge stp llc bonding tls r8125(O) zfs(PO) i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp zunicode(PO) zzstd(O) kvm_intel iosf_mbi drm_buddy i2c_algo_bit ttm btusb btrtl btbcm zlua(O) drm_display_helper btintel zavl(PO) kvm icp(PO) drm_kms_helper bluetooth drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 ecdh_generic sha1_ssse3 ecc ch341 aesni_intel usbserial zcommon(PO) crypto_simd intel_gtt
Nov 19 09:10:34 Svalbard kernel: cryptd znvpair(PO) rapl spl(O) mei_pxp mei_hdcp intel_cstate gigabyte_wmi wmi_bmof mpt3sas intel_uncore nvme i2c_i801 agpgart mei_me i2c_smbus i2c_core ahci nvme_core mei libahci raid_class scsi_transport_sas syscopyarea sysfillrect sysimgblt fb_sys_fops thermal fan tpm_crb tpm_tis video tpm_tis_core tpm wmi backlight intel_pmc_core acpi_tad acpi_pad button unix [last unloaded: r8125(O)]
Nov 19 09:10:34 Svalbard kernel: CPU: 19 PID: 12370 Comm: unraidd0 Tainted: P      D    O       6.1.106-Unraid #1
Nov 19 09:10:34 Svalbard kernel: Hardware name: Gigabyte Technology Co., Ltd. Z790M AORUS ELITE AX/Z790M AORUS ELITE AX, BIOS F10 09/27/2024
Nov 19 09:10:34 Svalbard kernel: RIP: 0010:do_exit+0x87/0x923
Nov 19 09:10:34 Svalbard kernel: Code: 24 74 04 75 13 b8 01 00 00 00 41 89 6c 24 60 48 c1 e0 22 49 89 44 24 70 4c 89 ef e8 41 30 81 00 48 83 bb b0 07 00 00 00 74 02 <0f> 0b 48 8b bb d8 06 00 00 e8 43 2f 81 00 48 8b 83 d0 06 00 00 83
Nov 19 09:10:34 Svalbard kernel: RSP: 0018:ffffc9000c36fee0 EFLAGS: 00010286
Nov 19 09:10:34 Svalbard kernel: RAX: 0000000000000000 RBX: ffff8881533b0000 RCX: 0000000000000000
Nov 19 09:10:34 Svalbard kernel: RDX: 0000000000000001 RSI: 0000000000002710 RDI: 00000000ffffffff
Nov 19 09:10:34 Svalbard kernel: RBP: 000000000000000b R08: 0000000000000000 R09: ffffffff829573f0
Nov 19 09:10:34 Svalbard kernel: R10: 00003fffffffffff R11: ffff8890bfbc0ce6 R12: ffff888137fbf000
Nov 19 09:10:34 Svalbard kernel: R13: ffff8881533a6300 R14: 0000000000000002 R15: ffffffff820b3185
Nov 19 09:10:34 Svalbard kernel: FS:  0000000000000000(0000) GS:ffff88907f8c0000(0000) knlGS:0000000000000000
Nov 19 09:10:34 Svalbard kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 19 09:10:34 Svalbard kernel: CR2: 0000148fe25b3010 CR3: 00000008cc37c006 CR4: 0000000000770ee0
Nov 19 09:10:34 Svalbard kernel: PKRU: 55555554
Nov 19 09:10:34 Svalbard kernel: Call Trace:
Nov 19 09:10:34 Svalbard kernel: <TASK>
Nov 19 09:10:34 Svalbard kernel: ? __warn+0xab/0x122
Nov 19 09:10:34 Svalbard kernel: ? report_bug+0x109/0x17e
Nov 19 09:10:34 Svalbard kernel: ? do_exit+0x87/0x923
Nov 19 09:10:34 Svalbard kernel: ? handle_bug+0x41/0x6f
Nov 19 09:10:34 Svalbard kernel: ? exc_invalid_op+0x13/0x60
Nov 19 09:10:34 Svalbard kernel: ? asm_exc_invalid_op+0x16/0x20
Nov 19 09:10:34 Svalbard kernel: ? do_exit+0x87/0x923
Nov 19 09:10:34 Svalbard kernel: make_task_dead+0x11c/0x11c
Nov 19 09:10:34 Svalbard kernel: rewind_stack_and_make_dead+0x17/0x17
Nov 19 09:10:34 Svalbard kernel: RIP: 0000:0x0
Nov 19 09:10:34 Svalbard kernel: Code: Unable to access opcode bytes at 0xffffffffffffffd6.
Nov 19 09:10:34 Svalbard kernel: RSP: 0000:0000000000000000 EFLAGS: 00000000 ORIG_RAX: 0000000000000000
Nov 19 09:10:34 Svalbard kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Nov 19 09:10:34 Svalbard kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Nov 19 09:10:34 Svalbard kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Nov 19 09:10:34 Svalbard kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Nov 19 09:10:34 Svalbard kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Nov 19 09:10:34 Svalbard kernel: </TASK>
Nov 19 09:10:34 Svalbard kernel: ---[ end trace 0000000000000000 ]---

 

What does it mean?

 

System is running a parity check following a crash yesterday. 

Finish your parity check first... please post a diag file...
 

this looks to me a networking bug between ipvlan and WireGuard at boot following a disruptive shutdown:

The kernel bug report you provided seems to indicate a critical error in the unraid.c driver related to the MD module (md_mod), which handles array management in unRAID. The error occurred during or after a disruptive power outage, potentially causing data corruption or inconsistency in the array.

 

Understanding the Error

Key Points in the Log:

File and Line: drivers/md/unraid.c:1617 – The issue is within the unRAID-specific MD driver.

Tainted Kernel: Tainted: P O – The kernel was tainted, meaning third-party modules or drivers (possibly proprietary ones) were loaded. This can complicate debugging.

Invalid Opcode: invalid opcode: 0000 – A fatal bug occurred due to invalid execution, possibly caused by memory corruption or an inconsistent state in the driver.

Triggered During Parity Check: Since the parity check was running, the bug was likely triggered by attempting to read from or write to an inconsistent or corrupted device.

Likely Causes:

Power Outage: A sudden power loss can leave the array in an inconsistent state, especially if writes were ongoing.

Driver Bug: There could be a bug in the MD driver that surfaced under the specific conditions of your system or disk states.

Hardware Issues: The power outage might have caused hardware faults, such as filesystem corruption, RAID metadata issues, or faulty sectors.

 

Steps to Resolve

 

Stop the Array:

If the system is operational, stop the array immediately to prevent further damage.

Avoid initiating any further writes or parity checks until the issue is resolved.

 

Check Disk Health:

Use SMART tests to verify the health of all drives in the array

smartctl -a /dev/sdX

Check for reallocated sectors, pending sectors, or other signs of drive failure.

 

Review System Logs:

Look for additional context in the system log (/var/log/syslog) around the time of the crash.

Use dmesg to capture recent kernel messages for clue

 

dmesg | less
 

Boot into Maintenance Mode:

Reboot unRAID in maintenance mode (no array start) to prevent writes during troubleshooting.

From maintenance mode, manually check the filesystem on each disk:

xfs_repair /dev/sdX

 

Disable Third-Party Plugins or Modules:

Remove or disable third-party plugins/modules, especially if they interact with storage devices or the array.

Manually Rebuild Parity:

Once the filesystem on all drives is confirmed healthy, you can attempt a full parity rebuild:

Stop the array.

Unassign the parity drive(s).

Start the array without the parity drive(s) (to clear any pending operations).

Reassign the parity drive(s) and initiate a parity rebuild.

Review and Report the Bug:

If the issue persists, collect relevant logs and submit them to the unRAID support forum or your support provider:

Include the kernel trace, unRAID version, hardware details, and reproduction steps.

 

 

Just got home. Should have provided more details, but I was at work when I saw this and figured I'd post it and see what comes back.

 

The time that this error occurred the system had already been running for 14 hours....so this isn't a boot issue....I'm also not running Wireguard (well....my GlueTun docker might be, but "unraid" isn't.

 

This error is spontaneous - here is the preceding hours of log:

Nov 19 02:19:03 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 02:49:35 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 02:49:37 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 03:20:09 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 03:20:10 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 03:50:44 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 03:50:46 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 04:21:18 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 04:21:21 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 04:22:19 Svalbard emhttpd: spinning down /dev/sdg
Nov 19 04:53:01 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 04:53:03 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 05:23:36 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 05:23:37 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 05:45:43 Svalbard shfs: /usr/sbin/zfs create 'm2-cache/archive_backups_pembroke' 2>&1
Nov 19 05:46:21 Svalbard kernel: br-1d649d9c9840: port 1(veth34ae42a) entered disabled state
Nov 19 05:46:21 Svalbard kernel: veth0ed63db: renamed from eth0
Nov 19 05:46:21 Svalbard kernel: br-1d649d9c9840: port 1(veth34ae42a) entered disabled state
Nov 19 05:46:21 Svalbard kernel: device veth34ae42a left promiscuous mode
Nov 19 05:46:21 Svalbard kernel: br-1d649d9c9840: port 1(veth34ae42a) entered disabled state
Nov 19 05:46:21 Svalbard kernel: br-1d649d9c9840: port 1(vetha97eefb) entered blocking state
Nov 19 05:46:21 Svalbard kernel: br-1d649d9c9840: port 1(vetha97eefb) entered disabled state
Nov 19 05:46:21 Svalbard kernel: device vetha97eefb entered promiscuous mode
Nov 19 05:46:21 Svalbard kernel: br-1d649d9c9840: port 1(vetha97eefb) entered blocking state
Nov 19 05:46:21 Svalbard kernel: br-1d649d9c9840: port 1(vetha97eefb) entered forwarding state
Nov 19 05:46:22 Svalbard kernel: eth0: renamed from veth39342eb
Nov 19 05:46:22 Svalbard kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vetha97eefb: link becomes ready
Nov 19 05:47:06 Svalbard kernel: br-1d649d9c9840: port 1(vetha97eefb) entered disabled state
Nov 19 05:47:06 Svalbard kernel: veth39342eb: renamed from eth0
Nov 19 05:47:06 Svalbard kernel: br-1d649d9c9840: port 1(vetha97eefb) entered disabled state
Nov 19 05:47:06 Svalbard kernel: device vetha97eefb left promiscuous mode
Nov 19 05:47:06 Svalbard kernel: br-1d649d9c9840: port 1(vetha97eefb) entered disabled state
Nov 19 05:47:06 Svalbard kernel: br-1d649d9c9840: port 1(vethbaf97ae) entered blocking state
Nov 19 05:47:06 Svalbard kernel: br-1d649d9c9840: port 1(vethbaf97ae) entered disabled state
Nov 19 05:47:06 Svalbard kernel: device vethbaf97ae entered promiscuous mode
Nov 19 05:47:06 Svalbard kernel: br-1d649d9c9840: port 1(vethbaf97ae) entered blocking state
Nov 19 05:47:06 Svalbard kernel: br-1d649d9c9840: port 1(vethbaf97ae) entered forwarding state
Nov 19 05:47:07 Svalbard kernel: eth0: renamed from vethd0bd9b2
Nov 19 05:47:07 Svalbard kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethbaf97ae: link becomes ready
Nov 19 06:00:01 Svalbard Docker Auto Update: Community Applications Docker Autoupdate running
Nov 19 06:00:01 Svalbard Docker Auto Update: Checking for available updates
Nov 19 06:00:43 Svalbard Docker Auto Update: No updates will be installed
Nov 19 06:31:26 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 06:31:28 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 07:00:01 Svalbard Plugin Auto Update: Checking for available plugin updates
Nov 19 07:00:08 Svalbard Plugin Auto Update: tips.and.tweaks.plg version 2024.11.17a does not meet age requirements to update - 2 days old
Nov 19 07:00:08 Svalbard Plugin Auto Update: Community Applications Plugin Auto Update finished
Nov 19 07:00:30 Svalbard emhttpd: spinning down /dev/sde
Nov 19 07:30:36 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 07:30:38 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 08:01:10 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 08:01:12 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 08:31:44 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 08:31:46 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 09:02:19 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 09:02:21 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 09:10:34 Svalbard kernel: ------------[ cut here ]------------
Nov 19 09:10:34 Svalbard kernel: kernel BUG at drivers/md/unraid.c:1617!
Nov 19 09:10:34 Svalbard kernel: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
Nov 19 09:10:34 Svalbard kernel: CPU: 19 PID: 12370 Comm: unraidd0 Tainted: P           O       6.1.106-Unraid #1
Nov 19 09:10:34 Svalbard kernel: Hardware name: Gigabyte Technology Co., Ltd. Z790M AORUS ELITE AX/Z790M AORUS ELITE AX, BIOS F10 09/27/2024

 

After that nothing again....everything is just normal:

 

Nov 19 09:10:34 Svalbard kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Nov 19 09:10:34 Svalbard kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Nov 19 09:10:34 Svalbard kernel: </TASK>
Nov 19 09:10:34 Svalbard kernel: ---[ end trace 0000000000000000 ]---
Nov 19 09:40:36 Svalbard emhttpd: spinning down /dev/sdj
Nov 19 09:40:36 Svalbard emhttpd: spinning down /dev/sdh
Nov 19 09:40:36 Svalbard emhttpd: spinning down /dev/sdf
Nov 19 09:40:36 Svalbard emhttpd: spinning down /dev/sdi
Nov 19 10:11:11 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 10:11:12 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 10:41:45 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 10:41:47 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 11:12:20 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 11:12:22 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 11:42:54 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 11:42:55 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 12:13:29 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 12:13:31 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 12:21:07 Svalbard emhttpd: read SMART /dev/sdh
Nov 19 12:21:18 Svalbard emhttpd: read SMART /dev/sdg
Nov 19 12:21:18 Svalbard emhttpd: read SMART /dev/sdf
Nov 19 12:21:31 Svalbard emhttpd: read SMART /dev/sdj
Nov 19 12:51:50 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 12:51:52 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 12:52:26 Svalbard emhttpd: spinning down /dev/sdh
Nov 19 12:52:26 Svalbard emhttpd: spinning down /dev/sdf
Nov 19 12:52:31 Svalbard emhttpd: spinning down /dev/sdj
Nov 19 12:52:31 Svalbard emhttpd: spinning down /dev/sdg
Nov 19 13:23:09 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 13:23:11 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 13:23:51 Svalbard emhttpd: read SMART /dev/sdh
Nov 19 13:24:06 Svalbard emhttpd: read SMART /dev/sdj
Nov 19 13:24:19 Svalbard emhttpd: read SMART /dev/sdf
Nov 19 13:24:30 Svalbard emhttpd: read SMART /dev/sdi
Nov 19 13:55:13 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 13:55:15 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 13:58:13 Svalbard emhttpd: spinning down /dev/sdi
Nov 19 13:59:28 Svalbard webGUI: Successful login user root from 192.168.3.5
Nov 19 14:00:49 Svalbard emhttpd: spinning down /dev/sdf
Nov 19 14:01:15 Svalbard emhttpd: spinning down /dev/sdh
Nov 19 14:01:21 Svalbard emhttpd: spinning down /dev/sdj
Nov 19 14:31:31 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 14:31:33 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 15:02:05 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 15:02:07 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 15:32:39 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 15:32:41 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 15:33:48 Svalbard emhttpd: read SMART /dev/sdh
Nov 19 15:34:00 Svalbard emhttpd: read SMART /dev/sdj
Nov 19 16:04:24 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 16:04:26 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 16:04:30 Svalbard emhttpd: spinning down /dev/sdj
Nov 19 16:04:32 Svalbard emhttpd: spinning down /dev/sdh
Nov 19 16:34:58 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 16:35:00 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 17:05:33 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 17:05:35 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 17:36:07 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 17:36:09 Svalbard emhttpd: read SMART /dev/sdd
Nov 19 18:06:41 Svalbard emhttpd: spinning down /dev/sdd
Nov 19 18:06:43 Svalbard emhttpd: read SMART /dev/sdd

 

It's a bit weird.  Parity check is on 45.5%, it will resume overnight.....other than happy.

 

That brings me to the crash yesterday. The system went offline, the syslog shows thousands of these:

Nov 18 18:43:18 Svalbard kernel: pcieport 0000:00:01.0: AER: Corrected error message received from 0000:01:00.0
Nov 18 18:43:18 Svalbard kernel: pcieport 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Nov 18 18:43:18 Svalbard kernel: pcieport 0000:01:00.0:   device [10b5:8724] error status/mask=00000001/0000a000
Nov 18 18:43:18 Svalbard kernel: pcieport 0000:01:00.0:    [ 0] RxErr                 
Nov 18 18:43:19 Svalbard kernel: pcieport 0000:00:01.0: AER: Corrected error message received from 0000:01:00.0
Nov 18 18:43:19 Svalbard kernel: pcieport 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Nov 18 18:43:19 Svalbard kernel: pcieport 0000:01:00.0:   device [10b5:8724] error status/mask=00000001/0000a000
Nov 18 18:43:19 Svalbard kernel: pcieport 0000:01:00.0:    [ 0] RxErr                 
Nov 18 18:43:20 Svalbard kernel: pcieport 0000:00:01.0: AER: Corrected error message received from 0000:01:00.0
Nov 18 18:43:20 Svalbard kernel: pcieport 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Nov 18 18:43:20 Svalbard kernel: pcieport 0000:01:00.0:   device [10b5:8724] error status/mask=00000001/0000a000
Nov 18 18:43:20 Svalbard kernel: pcieport 0000:01:00.0:    [ 0] RxErr                 
Nov 18 18:43:22 Svalbard kernel: pcieport 0000:00:01.0: AER: Corrected error message received from 0000:01:00.0
Nov 18 18:43:22 Svalbard kernel: pcieport 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Nov 18 18:43:22 Svalbard kernel: pcieport 0000:01:00.0:   device [10b5:8724] error status/mask=00000001/0000a000
Nov 18 18:43:22 Svalbard kernel: pcieport 0000:01:00.0:    [ 0] RxErr                 

 

Following a reboot, no more issues.

 

I am pusszledified...

 

I will upload the diagnostic once the parity check is finished.

 

🙂

 

 

Unraid driver is crashing, this is usually a hardware issue.

It could be the network card(s), SAS Controller, motherboard, CPU.....is there any way to figure out which hardware item is causing the problem?

More crashes.....diagnostics attached. 

 

In the log it did show that WireGuard was loading...but Wireguard is disabled.....?

svalbard-diagnostics-20241120-1418.zip

12 hours ago, ChirpyTurnip said:

is there any way to figure out which hardware item is causing the problem?

 

Typically RAM, CPU or board, if you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.

I've narrowed it down to an extra Ethernet controller (it has a RTL8125 chip - similar to this one). With the card removed Unraid runs in safe mode, completes parity check with no errors. Now it back running normal mode, but without the extra Ethernet controller - so far no errors.

 

The problem is that a bunch of my containers (some are important - like Frigate) are now offline as they rely on the vLANs that were routed through that interface....so I need to do something. If I put the card back I'm bound to get problems again.

 

I'm running on a Gigabyte Z790M Aorus Elite board....with all three M2 slots occupied, and both PCI slots (when the NIC is in). I'm pretty sure the onboard NIC is also a RTL8125 chip, so I'm not convinced it is the driver.

 

That leaves the card itself....or the motherboard....I could swap the card, but if I do, with what? And Intel-based one might be better but they are not cheap (here at least). I can also get from Amazon or ebay but delivery time is two weeks....and the shipping is more than the card. I've seen references to Unraid "supported" ethernet cards but all the links to the documentation are 404.

 

So, for example, is there support for Intel 226 chipset, or should I look for an older Pro/1000 type card?

Are you running Unraid with the stock drivers from Unraid itself? Or are you using the drivers from the CA AppStore? Maybe it's worth a try to use "the other ones"?

 

Are you using any power-saving features (ASPM, powertop and stuff like that)? Normally, they are running fine with Realtek cards, but in rare circumstances they are producing weird errors. So maybe you'll try NOT to use this stuff...

 

Unraid is supporting I226 based cards.

I tried the RTL8125 drivers from the CA, but that didn't make any noticeable difference. 

 

My boot options are:

 

append initrd=/bzroot pcie_aspm=off, nvme_core.default_ps_max_latency_us=5500

 

So power saving is disabled, and I had a weird thing with the NVME drives so found another post to set the max latency thing....without the NIC in there it has been perfectly stable...

 

I have ordered a new NIC based on the I210 chipset (this one) which will hopefully also work, and as a backup (and to use with a laptop if I need to connect to ethernet) this one which should also work with Unraid as it uses the ASIX AX88179A chipset. That means I can test a new card in the slot, or test without using the slot at all. I'm hoping this will narrow down the problem to either a fault slot, or a card/driver fault.....

 

This should be here tomorrow or the day after. If it doesn't work then I might retire the Unraid machine and make it my desktop machine (and if it fails under Windows get a MB replacement under warranty).  A new machine would then be needed....which then leads to the question: Ryzen 9, Core Ultra 7 (both new, both will need Intel Arc for quick sync and Unraid 7) or build another Core i7 machine (noting that here at least MBs are hard to get unless you buy MSI or something crazy like an Asus ProArt). Either way I will have an answer before Black Friday. 

 

remove the , that can cause some boot problems... that linee is seperated by space

append initrd=/bzroot pcie_aspm=off nvme_core.default_ps_max_latency_us=5500

 

Edited November 24, 20241 yr by bmartino1
typo

Quote

remove the , that can cause some boot problems... that line is separated by space

 

Not the problem, but it is always good to avoid a future problem! 🙂

 

Fixed!

21 hours ago, ChirpyTurnip said:

I've narrowed it down to an extra Ethernet controller (it has a RTL8125 chip - similar to this one). With the card removed Unraid runs in safe mode, completes parity check with no errors. Now it back running normal mode, but without the extra Ethernet controller - so far no errors.

 

The problem is that a bunch of my containers (some are important - like Frigate) are now offline as they rely on the vLANs that were routed through that interface....so I need to do something. If I put the card back I'm bound to get problems again.

 

I'm running on a Gigabyte Z790M Aorus Elite board....with all three M2 slots occupied, and both PCI slots (when the NIC is in). I'm pretty sure the onboard NIC is also a RTL8125 chip, so I'm not convinced it is the driver.

 

That leaves the card itself....or the motherboard....I could swap the card, but if I do, with what? And Intel-based one might be better but they are not cheap (here at least). I can also get from Amazon or ebay but delivery time is two weeks....and the shipping is more than the card. I've seen references to Unraid "supported" ethernet cards but all the links to the documentation are 404.

 

So, for example, is there support for Intel 226 chipset, or should I look for an older Pro/1000 type card?

are you on unraid beta or stble?

RTL8125 is a known driver problem lately with the constant unraid updates and kernel changes.
There is a plugin that is supposed to help

since this is a pcie card be sure to grab the pcie:

See more on the fourm...

 

You may be affected by the linux kernel shanagins... One of the reason why I have kernel 6... 

I am running 6.12.13. I did have the RTL8125 drivers added.....but it didn't seem to make anything better.....so I uninstalled them again. I find this curious as my motherboard has a RTl8125 chip also and it seems fine (although it runs at 2.5Gb/s on a simple connection, while the add-on card is limited to 1Gb/s and uses vLANs). 

 

How likely is it that there is a clash as I have two of the same? Unlikely I would think.....if it works for one it should work for the other....but who knows. It also used to be quite happy but then started crashing randomly. The only thing I can think of that changed is that I loaded a new BIOS version with the new 'safe' Intel microcode. Other than that nothing else has really changed (HW / driver wise at least). 

 

I can re-install the driver, and re-install the card.....but what's the bet the RX and crashing problems come back?

Assisting another user with similar but best to review the plugin forum, Jorge gave use a command to fix 2.5
 

as said earlier, there are known issues with that driver in the Linux kernel and the plugin helps but is not the one all be all install and done.
per the diag file with this user, other use it appears that the eth# keeps changing... which is causing problems... I used to have a udev rule to go back to debain/ubuntu for when os or pcie device change to no change the dam interface names...

Interface name changeing is a new one i haven't seen in unraid for a while. But is currently being seen for that driver, and you may be affected too. if using that device.

Edited November 24, 20241 yr by bmartino1

It's a tricky one.....a lot of the interface renaming seems to be coming form docker interfaces. 

 

The main puzzle I have is that my network cards do show up in Tools > System Devices. For example here is the onboard NIC:

[10ec:8125]	0b:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)

 

The weird thing is that the errors in the message log like these:

Nov 18 18:43:18 Svalbard kernel: pcieport 0000:00:01.0: AER: Corrected error message received from 0000:01:00.0
Nov 18 18:43:18 Svalbard kernel: pcieport 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Nov 18 18:43:18 Svalbard kernel: pcieport 0000:01:00.0:   device [10b5:8724] error status/mask=00000001/0000a000
Nov 18 18:43:18 Svalbard kernel: pcieport 0000:01:00.0:    [ 0] RxErr

 

Reference the PCI bus (based on these IDs):

[10b5:8724]	01:00.0 PCI bridge: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA (rev ca)
[10b5:8724]	02:00.0 PCI bridge: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA (rev ca)
[10b5:8724]	02:08.0 PCI bridge: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA (rev ca)
[10b5:8724]	02:09.0 PCI bridge: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA (rev ca)

 

Not the actual card, which *seemed* to be working fine as it was passing traffic as expected.

 

Now to be fair, the NIC and the HBA are both on the PCI bus, as are the NVMEs.....all I know now is that the memory is (probably) fine, and that without the extra Ethernet card everything is stable. 

 

So it could just the physical card itself, or a problem with the MB....I certainly don't have the other problems such as mac address not registering or not being able to use the card. All of that seems fine....and for periods of time post boot (10 minutes through to 10 hours) everything is fine before crashes start again.....with no card it has been four days now without a single error message or crash. 

its normal for docker to do it as it is creating virtual bridge interface to make network connection over the vhost@eth# tap

Sadly this is more toward the nic and reusing that this doesn't answer your orgianl form post, nor why the md kernel issue happened nor a solution to that. 

I will gladly help you where i can. This scan be tricky to diagnose due to pcie bifurcation and how the lanes are actually split.
?did you move pcie device around?

Some pcie port are not 100% wired as such some bottom x16 are actual wired x4. hba i have found need a x8 pcie port and using nvme can pull form those 8 pcie lane and make it 4

I think you have a MB pcie bifurcation issue.
 

 

 

Edited November 25, 20241 yr by bmartino1
typo - data

I haven't moved anything....I only have two slots so there's not much to shuffle in any case 🙂

 

Anyway......for the first time ever I'm looking at the MB block diagram:

 

 

On the CPU side I have a HBA in the PCIx16 slot and a M.2 in M2A_CPU. This all seems to be working fine. So no issues there that I know of. The manual does say however:

 

CPU:
- 1 x PCI Express x16 slot, supporting PCIe 5.0 and running at x16 (PCIEX16)
    * The PCIEX16 slot can only support a graphics card or an NVMe SSD. If only one
      graphics card is to be installed, be sure to install it in the PCIEX16 slot.

 

I'm picking that's a red herring....I don't have an NVME in that slot, nor graphics card. 

 

Looking at the Z790 chipset then - a single PCIe slot (x4) and two x4 M.2 slots (both occupied). There are no other slots....so I see no reason why a PCIe x1 NIC can't be put in the slot. 

 

There nothing that says for example that I must choose between the third M.2 slot or the PCIe card....and if there is bifurcation of the lanes then it looks like this is already done out of the box between the PCIe and M.2 slots as the default set up...

 

New Intel210 card should be here this time tomorrow....so that will be interesting. 

 

 

 

 

 

your right it could be a read hearing but worth a mentioned, lets review the diag syslog and grab the motherboard info:

Gigabyte Technology Co., Ltd. Z790M AORUS ELITE AX motherboard with BIOS version F10 dated 09/27/2024
https://www.gigabyte.com/Motherboard/Z790M-AORUS-ELITE-rev-10#kf

 

Up to 20 PCI Express 4.0 lanes of 16 GT/s for access to peripheral devices and networking. Up to 8 PCI Express 3.0 lanes of 8 GT/s for access to peripheral devices and networking. Integrated digital signal processor (DSP) for audio offload and audio/voice features.

 

https://www.intel.com/content/www/us/en/products/docs/chipsets/desktop-chipsets/z790-chipset-brief.html#:~:text=Up to 20 PCI Express,to peripheral devices and networking.&text=Up to 8 PCI Express,to peripheral devices and networking.&text=Integrated digital signal processor (DSP,offload and audio%2Fvoice features.

so no this could be a valid concern as any card in the top x16 slot would need to run in x8 mode (bifurcating)...

Leaving you 8 for the nvem and bottom slot... you could be hitting pce lane caps...

Edited November 25, 20241 yr by bmartino1
typo

I can't find that exact reference to the total number of lanes, but that doesn't match the block diagram....and there's nothing in the manual about using a M.2 slots and losing the PCIe slot.

 

EDIT: Found the table:

Processor PCIe Lane Configuration Support 	1x16+1x4
Chipset PCIe Express5 4.0 Lanes6 	        Up to 20

 

The 20 PCI Express 4.0 lanes I expect are the x16 + x4 connected to the CPU.

 

The Z790 chipset allows for up to 20 PCI Express 4.0 lanes - that means no contraint.

 

Even if that isn't the case, if everything is populated I don't think we don't get bifurcation (telling a slot that there are multiple devices installed in a single slot), rather we would get lane/bandwidth sharing. This means slower performance when there is congestion....but the chipset should manage that....it hsould generate errors and kernel crashes?

 

 

Edited November 25, 20241 yr by ChirpyTurnip

Exact opposite your +4 is the dmi link from the cpu to the south bus bridge. (Usually on amd, the 4 is the top m.2 slot direct to cpu)
# of lanes is based on chipset and cpu and dedicated my the board manufacture.

the 16 at the number of lanes left and if you populate everything you hit or maxed the lane and are over. Bios may be handling some irq and sharing lanes...

 

if only using the top slot all 16 lanes are there. Because you populate the m.2 and the bottom slot the top x16 slot became x8 (not a bad thing this is hardware bifurcating normaly)

the issues the m.2 slots and nic at bottom are more than 8 total lanes. a m.2 can run in 2x mode. this usually disables a sata 5/6 function and is directed within internal/hidden bios options. as this will use the chipsets sata pcie lanes for the m.2 (likely what happened in your case). The chipset wil aut negtiate at POST. I doesn't dictate lane bandwidth sharing nor handle ups/down when the OS/kerneal level hits and functions... other softwrae load balencors... hence

 

the chipset can only mange the # of lanes asinged at boot via IRQ request. This would explain the potential kernel error due to # of lanes asinged but not the actual # in use to the correct irq devce.

Especial since things returned to a working state when you removed the nic.
the use of nvme top may be required to see the bifurcating error due to # of lanes and resource aviablity. otehrwsies contact gigabyte as this is a complicated and hard to explin topic that took me qute a few think tanks to fully understand why the manfautelre went one way vs the other... Thank pcie....

So after going backwards and forwards with Gigabyte for a while it appears that there is nothing wrong with my use of the PCI slots or the M.2 slots - it should all just work. 

 

With no NIC in the extra slot I had no errors for about three days (or at least no fatal errors).

With the use of a USB NIC I got 24 hours of stable operation (or at least no fatal errors).

With a new NIC in the slot (an Intel 210 chipset) I got almost 19 hours of stable operation...but now I have got an error!

 

Nov 28 10:41:19 Svalbard kernel: WARNING: CPU: 12 PID: 25398 at fs/dcache.c:430 retain_dentry+0x52/0xa5
Nov 28 10:41:19 Svalbard kernel: Modules linked in: xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net vhost vhost_iotlb tap ipvlan wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha tun nft_compat nf_tables xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs md_mod tcp_diag inet_diag it87(O) hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs 8021q garp mrp bridge stp llc bonding tls igb r8169 realtek zfs(PO) i915 zunicode(PO) zzstd(O) intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel zlua(O) iosf_mbi drm_buddy kvm ttm zavl(PO) icp(PO) btusb drm_display_helper btrtl btbcm btintel drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel bluetooth ghash_clmulni_intel sha512_ssse3
Nov 28 10:41:19 Svalbard kernel: drm sha256_ssse3 sha1_ssse3 aesni_intel ecdh_generic ecc crypto_simd cryptd zcommon(PO) i2c_i801 intel_gtt rapl znvpair(PO) intel_cstate spl(O) mei_hdcp mei_pxp gigabyte_wmi wmi_bmof mpt3sas intel_uncore nvme agpgart i2c_algo_bit i2c_smbus mei_me nvme_core i2c_core ahci mei raid_class libahci scsi_transport_sas syscopyarea sysfillrect sysimgblt fb_sys_fops thermal fan tpm_crb video tpm_tis tpm_tis_core wmi tpm backlight intel_pmc_core acpi_tad acpi_pad button unix [last unloaded: igb]
Nov 28 10:41:19 Svalbard kernel: CPU: 12 PID: 25398 Comm: lsof Tainted: P           O       6.1.106-Unraid #1
Nov 28 10:41:19 Svalbard kernel: Hardware name: Gigabyte Technology Co., Ltd. Z790M AORUS ELITE AX/Z790M AORUS ELITE AX, BIOS F10 09/27/2024
Nov 28 10:41:19 Svalbard kernel: RIP: 0010:retain_dentry+0x52/0xa5
Nov 28 10:41:19 Svalbard kernel: Code: 74 18 eb e9 48 8b 43 60 48 89 df 48 8b 40 20 ff d0 0f 1f 00 85 c0 74 e4 eb d3 ff 4b 5c 0f ba e0 13 72 49 a9 00 04 08 00 74 02 <0f> 0b 0d 00 00 08 00 89 03 65 48 ff 05 b7 fb dc 7e f7 03 00 00 70
Nov 28 10:41:19 Svalbard kernel: RSP: 0018:ffffc900518ebd98 EFLAGS: 00010206
Nov 28 10:41:19 Svalbard kernel: RAX: 0000000000600c00 RBX: ffff888119253080 RCX: 0000000000000064
Nov 28 10:41:19 Svalbard kernel: RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff888119253080
Nov 28 10:41:19 Svalbard kernel: RBP: ffffc900518ebe65 R08: 000000000bbc5130 R09: 000000000000000a
Nov 28 10:41:19 Svalbard kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003
Nov 28 10:41:19 Svalbard kernel: R13: ffffffff812b089e R14: ffff888119253080 R15: ffff888119253080
Nov 28 10:41:19 Svalbard kernel: FS:  0000149a3d71fe00(0000) GS:ffff88907f700000(0000) knlGS:0000000000000000
Nov 28 10:41:19 Svalbard kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 28 10:41:19 Svalbard kernel: CR2: 0000149615b52000 CR3: 0000000533e1c004 CR4: 0000000000770ee0
Nov 28 10:41:19 Svalbard kernel: PKRU: 55555554
Nov 28 10:41:19 Svalbard kernel: Call Trace:
Nov 28 10:41:19 Svalbard kernel: <TASK>
Nov 28 10:41:19 Svalbard kernel: ? __warn+0xab/0x122
Nov 28 10:41:19 Svalbard kernel: ? report_bug+0x109/0x17e
Nov 28 10:41:19 Svalbard kernel: ? retain_dentry+0x52/0xa5
Nov 28 10:41:19 Svalbard kernel: ? handle_bug+0x41/0x6f
Nov 28 10:41:19 Svalbard kernel: ? exc_invalid_op+0x13/0x60
Nov 28 10:41:19 Svalbard kernel: ? asm_exc_invalid_op+0x16/0x20
Nov 28 10:41:19 Svalbard kernel: ? tid_fd_update_inode+0x4d/0x4d
Nov 28 10:41:19 Svalbard kernel: ? retain_dentry+0x52/0xa5
Nov 28 10:41:19 Svalbard kernel: dput+0x41/0x17b
Nov 28 10:41:19 Svalbard kernel: proc_fill_cache+0x110/0x156
Nov 28 10:41:19 Svalbard kernel: ? compat_filldir+0x17a/0x17a
Nov 28 10:41:19 Svalbard kernel: proc_readfd_common+0x16b/0x1bc
Nov 28 10:41:19 Svalbard kernel: ? tid_fd_update_inode+0x4d/0x4d
Nov 28 10:41:19 Svalbard kernel: iterate_dir+0x94/0x149
Nov 28 10:41:19 Svalbard kernel: __do_sys_getdents64+0x6b/0xd8
Nov 28 10:41:19 Svalbard kernel: ? compat_filldir+0x17a/0x17a
Nov 28 10:41:19 Svalbard kernel: do_syscall_64+0x65/0x7b
Nov 28 10:41:19 Svalbard kernel: entry_SYSCALL_64_after_hwframe+0x6e/0xd8
Nov 28 10:41:19 Svalbard kernel: RIP: 0033:0x149a3d97f283
Nov 28 10:41:19 Svalbard kernel: Code: 89 df e8 20 05 fb ff 48 83 c4 08 48 89 e8 5b 5d c3 66 0f 1f 44 00 00 b8 ff ff ff 7f 48 39 c2 48 0f 47 d0 b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 61 0b 11 00 f7 d8
Nov 28 10:41:19 Svalbard kernel: RSP: 002b:00007ffdea84a808 EFLAGS: 00000293 ORIG_RAX: 00000000000000d9
Nov 28 10:41:19 Svalbard kernel: RAX: ffffffffffffffda RBX: 00000000004c5650 RCX: 0000149a3d97f283
Nov 28 10:41:19 Svalbard kernel: RDX: 0000000000008000 RSI: 00000000004c5680 RDI: 0000000000000004
Nov 28 10:41:19 Svalbard kernel: RBP: 00000000004c5654 R08: 0000149a3da91310 R09: 0000149a3da91310
Nov 28 10:41:19 Svalbard kernel: R10: 0000000000000000 R11: 0000000000000293 R12: ffffffffffffff88
Nov 28 10:41:19 Svalbard kernel: R13: 0000000000000002 R14: 0000000000433dd0 R15: 0000149a3db12000
Nov 28 10:41:19 Svalbard kernel: </TASK>
Nov 28 10:41:19 Svalbard kernel: ---[ end trace 0000000000000000 ]---
Nov 28 10:41:20 Svalbard kernel: ------------[ cut here ]------------
Nov 28 10:41:20 Svalbard kernel: WARNING: CPU: 17 PID: 310 at fs/dcache.c:483 d_lru_shrink_move+0x17/0x38
Nov 28 10:41:20 Svalbard kernel: Modules linked in: xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net vhost vhost_iotlb tap ipvlan wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha tun nft_compat nf_tables xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs md_mod tcp_diag inet_diag it87(O) hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs 8021q garp mrp bridge stp llc bonding tls igb r8169 realtek zfs(PO) i915 zunicode(PO) zzstd(O) intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel zlua(O) iosf_mbi drm_buddy kvm ttm zavl(PO) icp(PO) btusb drm_display_helper btrtl btbcm btintel drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel bluetooth ghash_clmulni_intel sha512_ssse3
Nov 28 10:41:20 Svalbard kernel: drm sha256_ssse3 sha1_ssse3 aesni_intel ecdh_generic ecc crypto_simd cryptd zcommon(PO) i2c_i801 intel_gtt rapl znvpair(PO) intel_cstate spl(O) mei_hdcp mei_pxp gigabyte_wmi wmi_bmof mpt3sas intel_uncore nvme agpgart i2c_algo_bit i2c_smbus mei_me nvme_core i2c_core ahci mei raid_class libahci scsi_transport_sas syscopyarea sysfillrect sysimgblt fb_sys_fops thermal fan tpm_crb video tpm_tis tpm_tis_core wmi tpm backlight intel_pmc_core acpi_tad acpi_pad button unix [last unloaded: igb]
Nov 28 10:41:20 Svalbard kernel: CPU: 17 PID: 310 Comm: kswapd0 Tainted: P        W  O       6.1.106-Unraid #1
Nov 28 10:41:20 Svalbard kernel: Hardware name: Gigabyte Technology Co., Ltd. Z790M AORUS ELITE AX/Z790M AORUS ELITE AX, BIOS F10 09/27/2024
Nov 28 10:41:20 Svalbard kernel: RIP: 0010:d_lru_shrink_move+0x17/0x38
Nov 28 10:41:20 Svalbard kernel: Code: e7 e8 31 c2 62 00 5a 48 89 d8 5b 5d 41 5c c3 cc cc cc cc 0f 1f 44 00 00 8b 06 89 c1 81 e1 00 04 08 00 81 f9 00 00 08 00 74 02 <0f> 0b 89 c1 80 cd 04 a9 00 00 70 00 89 0e 75 08 65 48 ff 0d 44 f9
Nov 28 10:41:20 Svalbard kernel: RSP: 0018:ffffc90000c2fab0 EFLAGS: 00010206
Nov 28 10:41:20 Svalbard kernel: RAX: 0000000000680c00 RBX: ffff888119253100 RCX: 0000000000080400
Nov 28 10:41:20 Svalbard kernel: RDX: ffffc90000c2fb78 RSI: ffff888119253080 RDI: ffff888103838108
Nov 28 10:41:20 Svalbard kernel: RBP: ffff8881192530d8 R08: 0000000000000000 R09: 000000000000025f
Nov 28 10:41:20 Svalbard kernel: R10: ffff888119253ac0 R11: ffff888129c2c920 R12: ffff888103838108
Nov 28 10:41:20 Svalbard kernel: R13: ffffc90000c2fb78 R14: ffffffff8125be2d R15: ffff888119253100
Nov 28 10:41:20 Svalbard kernel: FS:  0000000000000000(0000) GS:ffff88907f840000(0000) knlGS:0000000000000000
Nov 28 10:41:20 Svalbard kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 28 10:41:20 Svalbard kernel: CR2: 000014f7e0055d58 CR3: 00000010160bc003 CR4: 0000000000770ee0
Nov 28 10:41:20 Svalbard kernel: PKRU: 55555554
Nov 28 10:41:20 Svalbard kernel: Call Trace:
Nov 28 10:41:20 Svalbard kernel: <TASK>
Nov 28 10:41:20 Svalbard kernel: ? __warn+0xab/0x122
Nov 28 10:41:20 Svalbard kernel: ? report_bug+0x109/0x17e
Nov 28 10:41:20 Svalbard kernel: ? d_lru_shrink_move+0x17/0x38
Nov 28 10:41:20 Svalbard kernel: ? handle_bug+0x41/0x6f
Nov 28 10:41:20 Svalbard kernel: ? exc_invalid_op+0x13/0x60
Nov 28 10:41:20 Svalbard kernel: ? asm_exc_invalid_op+0x16/0x20
Nov 28 10:41:20 Svalbard kernel: ? d_lru_shrink_move+0x38/0x38
Nov 28 10:41:20 Svalbard kernel: ? d_lru_shrink_move+0x17/0x38
Nov 28 10:41:20 Svalbard kernel: dentry_lru_isolate+0x9c/0xb1
Nov 28 10:41:20 Svalbard kernel: __list_lru_walk_one+0x90/0x123
Nov 28 10:41:20 Svalbard kernel: list_lru_walk_one+0x60/0x7d
Nov 28 10:41:20 Svalbard kernel: ? d_lru_shrink_move+0x38/0x38
Nov 28 10:41:20 Svalbard kernel: prune_dcache_sb+0x46/0x73
Nov 28 10:41:20 Svalbard kernel: super_cache_scan+0xf4/0x17c
Nov 28 10:41:20 Svalbard kernel: do_shrink_slab+0x188/0x2a1
Nov 28 10:41:20 Svalbard kernel: shrink_slab+0x1f9/0x267
Nov 28 10:41:20 Svalbard kernel: shrink_node+0x334/0x588
Nov 28 10:41:20 Svalbard kernel: balance_pgdat+0x4e9/0x6a2
Nov 28 10:41:20 Svalbard kernel: ? _raw_spin_lock+0x13/0x1c
Nov 28 10:41:20 Svalbard kernel: ? raw_spin_rq_lock_nested+0x1b/0x26
Nov 28 10:41:20 Svalbard kernel: ? newidle_balance+0x289/0x30a
Nov 28 10:41:20 Svalbard kernel: kswapd+0x2f0/0x333
Nov 28 10:41:20 Svalbard kernel: ? _raw_spin_rq_lock_irqsave+0x20/0x20
Nov 28 10:41:20 Svalbard kernel: ? balance_pgdat+0x6a2/0x6a2
Nov 28 10:41:20 Svalbard kernel: kthread+0xe4/0xef
Nov 28 10:41:20 Svalbard kernel: ? kthread_complete_and_exit+0x1b/0x1b
Nov 28 10:41:20 Svalbard kernel: ret_from_fork+0x1f/0x30
Nov 28 10:41:20 Svalbard kernel: </TASK>
Nov 28 10:41:20 Svalbard kernel: ---[ end trace 0000000000000000 ]---

 

So that's not great. 😞

 

 

New diagnostics attached. 

 

Any new thoughts or clues in the diagnostic?

 

 

svalbard-diagnostics-20241128-1149.zip

Your lattest Diag:

The filtered syslog contains initialization logs and some general hardware information but lacks specific errors or warnings related to the Intel NIC or networking. Here’s how we can proceed:

Recommendations for Investigation

Focus on PCIe and Networking Errors:

Expand the search for entries related to "eth," "PCIe," or "driver" to isolate potential issues with the Intel NIC or PCI slot.

Review Specific NIC Driver Issues:

Intel NIC drivers (e1000e or similar) might log issues during initialization or operation.

Check for Kernel Panic or Fatal Errors:

Look for entries with "kernel bug," "fatal," or "panic" to identify instability.

 

your posted trace logs reveal a critical kernel panic related to dentry cache management and memory operations. The key points from the kernel panic trace are:

 

Observations

Root Cause in Dentry Cache:

Errors like retain_dentry+0x52/0xa5 and d_lru_shrink_move+0x17/0x38 point to issues in managing the dentry cache. This suggests a problem with filesystem operations or memory management.

Involvement of Kswapd:

The kswapd process, which handles memory reclaiming, was active during the crash

kswapd+0x2f0/0x333

This could indicate memory pressure or issues with reclaiming filesystem caches.

 

Hardware and Software Context:

The system is running on a Gigabyte Z790M AORUS ELITE AX motherboard with BIOS version F10 (dated 09/27/2024).

There are traces of third-party modules such as zfs, zzstd, and spl. These may influence kernel behavior.

Unloaded NIC Driver:

A line in the trace shows last unloaded: igb, which corresponds to the Intel NIC driver. It’s unclear if this is directly related, but it's worth investigating.

 

Unraid does have driver plugin and issues can occur with out them and with them...

 

Recommendations

 

Filesystem and Cache Issues:

Consider reducing memory pressure by reviewing and optimizing workloads.

Check for filesystem integrity using appropriate tools (e.g., fsck for ext4 or equivalent for other filesystems).

 

NIC Driver Update:

Update the Intel NIC driver (igb) to the latest version or try a different compatible version.

Verify if the NIC is causing resource contention or unexpected conflicts.

 

Kernel and BIOS Updates:

Update to the latest Unraid version and Linux kernel, as this might fix bugs related to memory or dentry management.

Update the BIOS to the latest version available from Gigabyte to address any hardware-related issues.

 

Check Third-Party Modules:

Ensure all third-party modules (zfs, zzstd, etc.) are up-to-date and compatible with the kernel version.

 

Hardware Diagnostics:

Test memory for errors using memtest86+.

Inspect the NIC and other hardware components for physical or configuration issues.

we may need to inact udev rules or a 3rd part Slackware driver install.

this may be caused with gigabyts onbaord nic drvier clash with the pcie nic

it is a gift that keeps on giving!

 

Here's what I know for sure now (more or less):

1. It's not the RAM, passes memtest, behaves the same if running on individual sticks or both. Probability of both stick being faulty but also passing memtest is very low.
2. Memory pressure is also unlikely - there 64GB, so RAM to burn....I have plans for this in the future, but not yet...so not constrained.
3. It's not specific to the NIC or NIC driver - I've tried with both RTL8125 PCIe, I210 PCIe, and AX88179A USB-Ethernet adapters. Fails every time, which I think rules a PCIe thing (because USB also failed) and a driver thing (they all failed). It could be a weird combination issue, but again pretty unlikely.
4. It's also not the plugins - booting in safe mode with no plugins and it still dies.

Now it gets interesting.....it looks like I can run in non-safe mode (with plugins) but with no additional network card and docker DISABLED. So far it has been up rebuilding parity for >18 hours....normally within 3 hours it has crashed...so we are close to a record (here's hoping!)

 

So maybe Docker is where I should be looking?

1. I am running ipVLAN, so there should be no macVLAN dramas. That's the first item ticked off.
2. I was running 802.1 VLANs across the second adapter, I've disabled that, but it made no difference....still crashed
3. My dockers fall into three config types:
   - My *arr dockers are all linked to a GlueTun instance that connects to NordVPN. There might be a user-defined network behind that...my memory is vague
   - Two dockers are connected directly into another LAN via the second adapter (bridged, they have their own IP addresses)
   - The others are all bridged as well, everything has it's own IP address

My next steps are:

1. Restart docker without an extra NIC and with the extra NIC config removed.
   - Some dockers will complain, I'll move them to br0, bit of a pain, but nothing some firewall changes can't fix.
   - If it doesn't crash then we know it is a docker multiple-networks thing. If it does crash then I will need to look at stopping everything and working through docker by docker.
   - I'll probably start by unpicking GlueTun...it seems to me this is most likely thing to cause problems than other standalone dockers.

 

Thoughts?

 

Highly doubt just throwing it out there...

as weird as this is going to sound. a trial boot of a fresh unarid on a different USB. It could be usb wear out issues form inability to read a config....

 

You have done a lot of troubleshooting. I'm not quite out of ideas, but running out of potential, if not this then xyz.
For dockers, I prefer macvlan for dockers myself. unraid had a update 6.12.14 while troubleshooting.

*lets ignore the second nic for the time being then and look at libvrit corruption and network setups.

so we think it may be docker related. You might have a corrupted netwrok db file.

Giving your vpn choice I would actually recommend macvlan. no bridgin no bonding.

 

Edited December 1, 20241 yr by bmartino1