Skip to content
View in the app

A better way to browse. Learn more.
Unraid

A full-screen app on your home screen with push notifications, badges and more.

To install this app on iOS and iPadOS
  1. Tap the Share icon in Safari
  2. Scroll the menu and tap Add to Home Screen.
  3. Tap Add in the top-right corner.
To install this app on Android
  1. Tap the 3-dot menu (⋮) in the top-right corner of the browser.
  2. Tap Add to Home screen or Install app.
  3. Confirm by tapping Install.
Unraid Summer Sale in live: 20% off Starter and Unleashed + buy one, get one 50% off

[6.12.x] Random Server crash, XFS and BTRFS corruptions, Data loss. Tested on 3 different Computers

Crash_Override
in General Support

Featured Replies

Hi guys,

 

I know there are many other threads with crashing issues. Some of them think they found issue (C-States, Mover, RAM, CPU, firmware issues). But I am sure this is a software-hardware (kernel and/or driver) incompatibilty problem.

 

First of all I have two rack servers which working rockstable like I know unRAID for.

A HP ProLiant DL380 G9 and a even older Fujitsu Primergy of some sort, both full of drives, multiple HBAs, 10G networking and so on. Both servers has a big amount of RAM for many Docker containers and VMs.

 

My homeserver(s) is not working stable since 6.12.0 till 6.12.4 and I think it is a issue of networking <-> docker related things.

 

My home unRAID is setup is fairly simple, because I mirrored my data to my datacenter servers mentioned above and my focus is power consumption.

 

First hardware I used since 6.12 was my old mini itx gaming rig:

MB: X570 I AORUS PRO WIFI (of course latest bios)

CPU: Ryzen 3800X (Overdrive is disabled only standard speeds because of power consumption)

RAM: 2x16GB G.Skill Trident Z RGB DDR4-3600 (running on JEDEC speed (2400), no XMP)

HDD: 1x 8TB Seagate Ironwulf NAS 7200rpm (SATA)

SSD: 2x Samsung PM883 960GB (Datacenter-grade)

 

I get multiple crashes with BTRFS (SSD Pool) corruptions, mostly reboot and btrfsck fixes them, but one day not.
I changed these two ssds to a 1TB Samsung 970 EVO NVME.

Crash and corruptions same like above.

 

Ok I thougt there must be an memory issue of some kind, but I dont want to spent money so I use an older but working Rig with following specs:

 

MB: Gigabyte Z170X-Gaming 3

CPU: Intel I7-7700 (without K, Intel Speedstep disabled, power cons....)

RAM: 4x8GB Corsair Vengeance LPX 16GB DDR4 3200MHz (running on JEDEC speed, and "maximum stability" in BIOS selected)

HDD: (SAME) 1x 8TB Seagate Ironwulf NAS 7200rpm (SATA)

SSD: (SAME) 1x 1TB Samsung 970 EVO NVME

 

Nothing changed really. Of course I secure erase my ssd and reformatted my HDD and copy all files from backups too it. It will working normally like the previous system, then there is a kernel crash with nf_conntrack and some nat related thing same as previous hardware and then corruptions will starting till nothing is working anymore.

 

I thought okay maybe my harddrive was really bad even SMART values are OK, because on this rig, XFS crashes often. I replaced the HDD with the same model (I have 3 of them lying around from an HDD upgrade). Reformat and copied files from old drive and some from backup because of corruption.

 

Now okay I give it a last chance with new hardware, because power consumption was about half lower than the amd system.

I bought following components NEW:

 

MB: ASUSTeK PRIME B550M-A (BIOS from 2021 and latest from a month ago tested)

CPU: AMD Ryzen 5 PRO 4650G

RAM: 2x 16GB Kingston Server Premier 16 GB DDR4-3200 (with Multi Bit ECC!!!!!! which is supported by MB and CPU)

HDD: 2x 8TB Seagate Ironwulf NAS 7200rpm (SATA) (the last 2 drives I had lying around)

SSD: 2x 1TB Samsung 1TB Samsung 970 EVO NVME (both new, other I placed in my laptop)

 

Restarted fully from scratch, copied files, docker and vms and .............. nothing. Some XFS corruptions there some BTRFS read faults there with fully disconnecting NVMEs (/dev/nvmeXnX disapperaed). Mostly reboot fixed it but......

Now both filesystem XFS and BTRFS crashed at same time. Data lost, It is not recoverable anymore without many scans with tools and things. I have backups so I dont care about time consuming restoring.

 

Uptime was from about 1 to 8 days till crash and corruptions occured.

 

First crash in syslog was always the same like this one network related:

  

Oct 14 04:52:01 frameRAID kernel: kvm: vcpu 3: requested 134909 ns lapic timer period limited to 200000 ns
Oct 14 04:52:01 frameRAID kernel: kvm: vcpu 6: requested 134909 ns lapic timer period limited to 200000 ns
Oct 14 04:52:01 frameRAID kernel: kvm: vcpu 4: requested 134909 ns lapic timer period limited to 200000 ns
Oct 14 04:52:01 frameRAID kernel: kvm: vcpu 1: requested 134909 ns lapic timer period limited to 200000 ns
Oct 14 04:52:01 frameRAID kernel: kvm: vcpu 5: requested 134909 ns lapic timer period limited to 200000 ns
Oct 14 04:52:01 frameRAID kernel: kvm: vcpu 2: requested 134909 ns lapic timer period limited to 200000 ns
Oct 14 04:52:01 frameRAID kernel: kvm: vcpu 7: requested 134909 ns lapic timer period limited to 200000 ns
Oct 14 06:07:02 frameRAID emhttpd: spinning down /dev/sdb
Oct 14 06:34:46 frameRAID kernel: ------------[ cut here ]------------
Oct 14 06:34:46 frameRAID kernel: WARNING: CPU: 5 PID: 18034 at net/netfilter/nf_conntrack_core.c:1210 __nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
Oct 14 06:34:46 frameRAID kernel: Modules linked in: af_packet bluetooth ecdh_generic ecc vhost_net tun vhost tap kvm_amd ccp kvm md_mod rpcsec_gss_krb5 ftdi_sio usbserial cmac cifs asn1_decoder cifs_arc4 cifs_md4 dns_resolver veth xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_nat xt_tcpudp ip6table_mangle iptable_mangle vhost_iotlb macvlan xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat xt_addrtype br_netfilter xfs xt_MASQUERADE ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nfsd auth_rpcgss oid_registry lockd grace sunrpc zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge 8021q garp mrp stp llc amdgpu amd64_edac edac_mce_amd edac_core intel_rapl_msr intel_rapl_common iosf_mbi gpu_sched drm_buddy video i2c_algo_bit drm_ttm_helper ttm drm_display_helper drm_kms_helper drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
Oct 14 06:34:46 frameRAID kernel: sha512_ssse3 aesni_intel crypto_simd cryptd backlight wmi_bmof agpgart i2c_piix4 rapl nvme i2c_core r8169 k10temp syscopyarea nvme_core sysfillrect sysimgblt fb_sys_fops realtek input_leds led_class ahci libahci wmi tpm_crb tpm_tis tpm_tis_core tpm acpi_cpufreq button unix [last unloaded: md_mod]
Oct 14 06:34:46 frameRAID kernel: CPU: 5 PID: 18034 Comm: kworker/u64:0 Tainted: P           O       6.1.49-Unraid #1
Oct 14 06:34:46 frameRAID kernel: Hardware name: ASUS System Product Name/PRIME B550M-A, BIOS 3205 08/14/2023
Oct 14 06:34:46 frameRAID kernel: Workqueue: events_unbound macvlan_process_broadcast [macvlan]
Oct 14 06:34:46 frameRAID kernel: RIP: 0010:__nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
Oct 14 06:34:46 frameRAID kernel: Code: 44 24 10 e8 e2 e1 ff ff 8b 7c 24 04 89 ea 89 c6 89 04 24 e8 7e e6 ff ff 84 c0 75 a2 48 89 df e8 9b e2 ff ff 85 c0 89 c5 74 18 <0f> 0b 8b 34 24 8b 7c 24 04 e8 18 dd ff ff e8 93 e3 ff ff e9 72 01
Oct 14 06:34:46 frameRAID kernel: RSP: 0018:ffffc90000380d98 EFLAGS: 00010202
Oct 14 06:34:46 frameRAID kernel: RAX: 0000000000000001 RBX: ffff888285c52f00 RCX: df3c1e17515edd13
Oct 14 06:34:46 frameRAID kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff888285c52f00
Oct 14 06:34:46 frameRAID kernel: RBP: 0000000000000001 R08: ef190138c8277ed0 R09: 731c57c5524097e7
Oct 14 06:34:46 frameRAID kernel: R10: e18fad964382a371 R11: ffffc90000380d60 R12: ffffffff82a11d00
Oct 14 06:34:46 frameRAID kernel: R13: 000000000003838a R14: ffff8881082bd700 R15: 0000000000000000
Oct 14 06:34:46 frameRAID kernel: FS:  0000000000000000(0000) GS:ffff888805d40000(0000) knlGS:0000000000000000
Oct 14 06:34:46 frameRAID kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 14 06:34:46 frameRAID kernel: CR2: 000055a572636098 CR3: 000000000520a000 CR4: 0000000000350ee0
Oct 14 06:34:46 frameRAID kernel: Call Trace:
Oct 14 06:34:46 frameRAID kernel: <IRQ>
Oct 14 06:34:46 frameRAID kernel: ? __warn+0xab/0x122
Oct 14 06:34:46 frameRAID kernel: ? report_bug+0x109/0x17e
Oct 14 06:34:46 frameRAID kernel: ? __nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
Oct 14 06:34:46 frameRAID kernel: ? handle_bug+0x41/0x6f
Oct 14 06:34:46 frameRAID kernel: ? exc_invalid_op+0x13/0x60
Oct 14 06:34:46 frameRAID kernel: ? asm_exc_invalid_op+0x16/0x20
Oct 14 06:34:46 frameRAID kernel: ? __nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
Oct 14 06:34:46 frameRAID kernel: ? __nf_conntrack_confirm+0x9e/0x2b0 [nf_conntrack]
Oct 14 06:34:46 frameRAID kernel: ? nf_nat_inet_fn+0x60/0x1a8 [nf_nat]
Oct 14 06:34:46 frameRAID kernel: nf_conntrack_confirm+0x25/0x54 [nf_conntrack]
Oct 14 06:34:46 frameRAID kernel: nf_hook_slow+0x3d/0x96
Oct 14 06:34:46 frameRAID kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Oct 14 06:34:46 frameRAID kernel: NF_HOOK.constprop.0+0x79/0xd9
Oct 14 06:34:46 frameRAID kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Oct 14 06:34:46 frameRAID kernel: __netif_receive_skb_one_core+0x77/0x9c
Oct 14 06:34:46 frameRAID kernel: process_backlog+0x8c/0x116
Oct 14 06:34:46 frameRAID kernel: __napi_poll.constprop.0+0x2b/0x124
Oct 14 06:34:46 frameRAID kernel: net_rx_action+0x159/0x24f
Oct 14 06:34:46 frameRAID kernel: __do_softirq+0x129/0x288
Oct 14 06:34:46 frameRAID kernel: do_softirq+0x7f/0xab
Oct 14 06:34:46 frameRAID kernel: </IRQ>
Oct 14 06:34:46 frameRAID kernel: <TASK>
Oct 14 06:34:46 frameRAID kernel: __local_bh_enable_ip+0x4c/0x6b
Oct 14 06:34:46 frameRAID kernel: netif_rx+0x52/0x5a
Oct 14 06:34:46 frameRAID kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Oct 14 06:34:46 frameRAID kernel: ? _raw_spin_unlock+0x14/0x29
Oct 14 06:34:46 frameRAID kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]
Oct 14 06:34:46 frameRAID kernel: process_one_work+0x1ab/0x295
Oct 14 06:34:46 frameRAID kernel: worker_thread+0x18b/0x244
Oct 14 06:34:46 frameRAID kernel: ? rescuer_thread+0x281/0x281
Oct 14 06:34:46 frameRAID kernel: kthread+0xe7/0xef
Oct 14 06:34:46 frameRAID kernel: ? kthread_complete_and_exit+0x1b/0x1b
Oct 14 06:34:46 frameRAID kernel: ret_from_fork+0x22/0x30
Oct 14 06:34:46 frameRAID kernel: </TASK>
Oct 14 06:34:46 frameRAID kernel: ---[ end trace 0000000000000000 ]---
Oct 14 09:41:08 frameRAID emhttpd: read SMART /dev/sdb
Oct 14 12:30:49 frameRAID emhttpd: spinning down /dev/sdb
Oct 14 12:35:43 frameRAID emhttpd: read SMART /dev/sdb
Oct 14 13:30:34 frameRAID emhttpd: spinning down /dev/sdb

 

After this it will run from about 6h to 24h then something nat related (Docker!) will occur: 

Oct 15 12:52:30 frameRAID emhttpd: read SMART /dev/sdb
Oct 15 14:39:17 frameRAID kernel: ------------[ cut here ]------------
Oct 15 14:39:17 frameRAID kernel: WARNING: CPU: 8 PID: 0 at net/netfilter/nf_nat_core.c:594 nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Oct 15 14:39:17 frameRAID kernel: Modules linked in: af_packet bluetooth ecdh_generic ecc vhost_net tun vhost tap kvm_amd ccp kvm md_mod rpcsec_gss_krb5 ftdi_sio usbserial cmac cifs asn1_decoder cifs_arc4 cifs_md4 dns_resolver veth xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_nat xt_tcpudp ip6table_mangle iptable_mangle vhost_iotlb macvlan xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat xt_addrtype br_netfilter xfs xt_MASQUERADE ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nfsd auth_rpcgss oid_registry lockd grace sunrpc zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge 8021q garp mrp stp llc amdgpu amd64_edac edac_mce_amd edac_core intel_rapl_msr intel_rapl_common iosf_mbi gpu_sched drm_buddy video i2c_algo_bit drm_ttm_helper ttm drm_display_helper drm_kms_helper drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
Oct 15 14:39:17 frameRAID kernel: sha512_ssse3 aesni_intel crypto_simd cryptd backlight wmi_bmof agpgart i2c_piix4 rapl nvme i2c_core r8169 k10temp syscopyarea nvme_core sysfillrect sysimgblt fb_sys_fops realtek input_leds led_class ahci libahci wmi tpm_crb tpm_tis tpm_tis_core tpm acpi_cpufreq button unix [last unloaded: md_mod]
Oct 15 14:39:17 frameRAID kernel: CPU: 8 PID: 0 Comm: swapper/8 Tainted: P        W  O       6.1.49-Unraid #1
Oct 15 14:39:17 frameRAID kernel: Hardware name: ASUS System Product Name/PRIME B550M-A, BIOS 3205 08/14/2023
Oct 15 14:39:17 frameRAID kernel: RIP: 0010:nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Oct 15 14:39:17 frameRAID kernel: Code: a8 80 75 26 48 8d 73 58 48 8d 7c 24 20 e8 18 bb fd ff 48 8d 43 0c 4c 8b bb 88 00 00 00 48 89 44 24 18 eb 54 0f ba e0 08 73 07 <0f> 0b e9 75 06 00 00 48 8d 73 58 48 8d 7c 24 20 e8 eb ba fd ff 48
Oct 15 14:39:17 frameRAID kernel: RSP: 0018:ffffc9000041c788 EFLAGS: 00010282
Oct 15 14:39:17 frameRAID kernel: RAX: 0000000000000180 RBX: ffff88811c27da00 RCX: ffff888104f0e940
Oct 15 14:39:17 frameRAID kernel: RDX: 0000000000000000 RSI: ffffc9000041c86c RDI: ffff88811c27da00
Oct 15 14:39:17 frameRAID kernel: RBP: ffffc9000041c850 R08: 000000001900010a R09: 0000000000000000
Oct 15 14:39:17 frameRAID kernel: R10: 0000000000000158 R11: 0000000000000000 R12: ffffc9000041c86c
Oct 15 14:39:17 frameRAID kernel: R13: 0000000000000000 R14: ffffc9000041c950 R15: 0000000000000001
Oct 15 14:39:17 frameRAID kernel: FS:  0000000000000000(0000) GS:ffff888805e00000(0000) knlGS:0000000000000000
Oct 15 14:39:17 frameRAID kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 15 14:39:17 frameRAID kernel: CR2: 000014dd80ef2000 CR3: 000000000520a000 CR4: 0000000000350ee0
Oct 15 14:39:17 frameRAID kernel: Call Trace:
Oct 15 14:39:17 frameRAID kernel: <IRQ>
Oct 15 14:39:17 frameRAID kernel: ? __warn+0xab/0x122
Oct 15 14:39:17 frameRAID kernel: ? report_bug+0x109/0x17e
Oct 15 14:39:17 frameRAID kernel: ? nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Oct 15 14:39:17 frameRAID kernel: ? handle_bug+0x41/0x6f
Oct 15 14:39:17 frameRAID kernel: ? exc_invalid_op+0x13/0x60
Oct 15 14:39:17 frameRAID kernel: ? asm_exc_invalid_op+0x16/0x20
Oct 15 14:39:17 frameRAID kernel: ? nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Oct 15 14:39:17 frameRAID kernel: ? nf_nat_setup_info+0x44/0x7d1 [nf_nat]
Oct 15 14:39:17 frameRAID kernel: ? percpu_counter_add_batch+0x85/0xa2
Oct 15 14:39:17 frameRAID kernel: ? xt_write_recseq_end+0xf/0x1c [ip_tables]
Oct 15 14:39:17 frameRAID kernel: ? __local_bh_enable_ip+0x56/0x6b
Oct 15 14:39:17 frameRAID kernel: ? ipt_do_table+0x57a/0x5bf [ip_tables]
Oct 15 14:39:17 frameRAID kernel: ? rt_dst_alloc+0x2f/0xa0
Oct 15 14:39:17 frameRAID kernel: ? xt_write_recseq_end+0xf/0x1c [ip_tables]
Oct 15 14:39:17 frameRAID kernel: __nf_nat_alloc_null_binding+0x66/0x81 [nf_nat]
Oct 15 14:39:17 frameRAID kernel: nf_nat_inet_fn+0xc0/0x1a8 [nf_nat]
Oct 15 14:39:17 frameRAID kernel: nf_nat_ipv4_local_in+0x2a/0xaa [nf_nat]
Oct 15 14:39:17 frameRAID kernel: nf_hook_slow+0x3d/0x96
Oct 15 14:39:17 frameRAID kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Oct 15 14:39:17 frameRAID kernel: NF_HOOK.constprop.0+0x79/0xd9
Oct 15 14:39:17 frameRAID kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Oct 15 14:39:17 frameRAID kernel: ip_sabotage_in+0x52/0x60 [br_netfilter]
Oct 15 14:39:17 frameRAID kernel: nf_hook_slow+0x3d/0x96
Oct 15 14:39:17 frameRAID kernel: ? ip_rcv_finish_core.constprop.0+0x3e8/0x3e8
Oct 15 14:39:17 frameRAID kernel: NF_HOOK.constprop.0+0x79/0xd9
Oct 15 14:39:17 frameRAID kernel: ? ip_rcv_finish_core.constprop.0+0x3e8/0x3e8
Oct 15 14:39:17 frameRAID kernel: __netif_receive_skb_one_core+0x77/0x9c
Oct 15 14:39:17 frameRAID kernel: netif_receive_skb+0xbf/0x127
Oct 15 14:39:17 frameRAID kernel: br_handle_frame_finish+0x438/0x472 [bridge]
Oct 15 14:39:17 frameRAID kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
Oct 15 14:39:17 frameRAID kernel: br_nf_hook_thresh+0xe5/0x109 [br_netfilter]
Oct 15 14:39:17 frameRAID kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
Oct 15 14:39:17 frameRAID kernel: br_nf_pre_routing_finish+0x2c1/0x2ec [br_netfilter]
Oct 15 14:39:17 frameRAID kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
Oct 15 14:39:17 frameRAID kernel: ? NF_HOOK.isra.0+0xe4/0x140 [br_netfilter]
Oct 15 14:39:17 frameRAID kernel: ? br_nf_hook_thresh+0x109/0x109 [br_netfilter]
Oct 15 14:39:17 frameRAID kernel: br_nf_pre_routing+0x236/0x24a [br_netfilter]
Oct 15 14:39:17 frameRAID kernel: ? br_nf_hook_thresh+0x109/0x109 [br_netfilter]
Oct 15 14:39:17 frameRAID kernel: br_handle_frame+0x27a/0x2e0 [bridge]
Oct 15 14:39:17 frameRAID kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
Oct 15 14:39:17 frameRAID kernel: __netif_receive_skb_core.constprop.0+0x4fd/0x6e9
Oct 15 14:39:17 frameRAID kernel: ? get_page_from_freelist+0x755/0x89f
Oct 15 14:39:17 frameRAID kernel: ? __skb_checksum+0x5c/0x283
Oct 15 14:39:17 frameRAID kernel: __netif_receive_skb_list_core+0x8a/0x11e
Oct 15 14:39:17 frameRAID kernel: netif_receive_skb_list_internal+0x1d2/0x20b
Oct 15 14:39:17 frameRAID kernel: gro_normal_list+0x1d/0x3f
Oct 15 14:39:17 frameRAID kernel: napi_complete_done+0x7b/0x11a
Oct 15 14:39:17 frameRAID kernel: rtl8169_poll+0x3f0/0x414 [r8169]
Oct 15 14:39:17 frameRAID kernel: __napi_poll.constprop.0+0x2b/0x124
Oct 15 14:39:17 frameRAID kernel: net_rx_action+0x159/0x24f
Oct 15 14:39:17 frameRAID kernel: __do_softirq+0x129/0x288
Oct 15 14:39:17 frameRAID kernel: __irq_exit_rcu+0x5e/0xb8
Oct 15 14:39:17 frameRAID kernel: common_interrupt+0x9b/0xc1
Oct 15 14:39:17 frameRAID kernel: </IRQ>
Oct 15 14:39:17 frameRAID kernel: <TASK>
Oct 15 14:39:17 frameRAID kernel: asm_common_interrupt+0x22/0x40
Oct 15 14:39:17 frameRAID kernel: RIP: 0010:cpuidle_enter_state+0x11d/0x202
Oct 15 14:39:17 frameRAID kernel: Code: 20 22 a0 ff 45 84 ff 74 1b 9c 58 0f 1f 40 00 0f ba e0 09 73 08 0f 0b fa 0f 1f 44 00 00 31 ff e8 4c e3 a4 ff fb 0f 1f 44 00 00 <45> 85 e4 0f 88 ba 00 00 00 48 8b 04 24 49 63 cc 48 6b d1 68 49 29
Oct 15 14:39:17 frameRAID kernel: RSP: 0018:ffffc900001b7e98 EFLAGS: 00000246
Oct 15 14:39:17 frameRAID kernel: RAX: ffff888805e00000 RBX: ffff888108252800 RCX: 0000000000000000
Oct 15 14:39:17 frameRAID kernel: RDX: 0001783d0c329ee7 RSI: ffffffff820ed4af RDI: ffffffff820ed9b8
Oct 15 14:39:17 frameRAID kernel: RBP: 0000000000000003 R08: 0000000000000002 R09: 0000000000000002
Oct 15 14:39:17 frameRAID kernel: R10: 0000000000000010 R11: 000000000000002a R12: 0000000000000003
Oct 15 14:39:17 frameRAID kernel: R13: ffffffff82323720 R14: 0001783d0c329ee7 R15: 0000000000000000
Oct 15 14:39:17 frameRAID kernel: ? cpuidle_enter_state+0xf7/0x202
Oct 15 14:39:17 frameRAID kernel: cpuidle_enter+0x2a/0x38
Oct 15 14:39:17 frameRAID kernel: do_idle+0x18d/0x1fb
Oct 15 14:39:17 frameRAID kernel: cpu_startup_entry+0x1d/0x1f
Oct 15 14:39:17 frameRAID kernel: start_secondary+0x101/0x101
Oct 15 14:39:17 frameRAID kernel: secondary_startup_64_no_verify+0xce/0xdb
Oct 15 14:39:17 frameRAID kernel: </TASK>
Oct 15 14:39:17 frameRAID kernel: ---[ end trace 0000000000000000 ]---
Oct 15 15:00:31 frameRAID emhttpd: spinning down /dev/sdb
### [PREVIOUS LINE REPEATED 1 TIMES] ###
Oct 15 16:42:10 frameRAID emhttpd: read SMART /dev/sdb

 

And then goodbye filesystem, in this last crashlog it was my XFS again. 

Oct 16 00:20:04 frameRAID sSMTP[30444]: Sent mail for [email protected] (221 goodbye) uid=0 username=xxx outbytes=1073
Oct 16 00:22:19 frameRAID kernel: XFS (md1p1): Internal error !xfs_dir2_namecheck(dep->name, dep->namelen) at line 466 of file fs/xfs/xfs_dir2_readdir.c.  Caller xfs_dir2_leaf_getdents+0x220/0x322 [xfs]
Oct 16 00:22:19 frameRAID kernel: CPU: 3 PID: 1589 Comm: find Tainted: P        W  O       6.1.49-Unraid #1
Oct 16 00:22:19 frameRAID kernel: Hardware name: ASUS System Product Name/PRIME B550M-A, BIOS 3205 08/14/2023
Oct 16 00:22:19 frameRAID kernel: Call Trace:
Oct 16 00:22:19 frameRAID kernel: <TASK>
Oct 16 00:22:19 frameRAID kernel: dump_stack_lvl+0x44/0x5c
Oct 16 00:22:19 frameRAID kernel: xfs_corruption_error+0x63/0x83 [xfs]
Oct 16 00:22:19 frameRAID kernel: ? filldir64+0x94/0x173
Oct 16 00:22:19 frameRAID kernel: xfs_dir2_leaf_getdents+0x253/0x322 [xfs]
Oct 16 00:22:19 frameRAID kernel: ? xfs_dir2_leaf_getdents+0x220/0x322 [xfs]
Oct 16 00:22:19 frameRAID kernel: xfs_readdir+0x14e/0x190 [xfs]
Oct 16 00:22:19 frameRAID kernel: iterate_dir+0x97/0x146
Oct 16 00:22:19 frameRAID kernel: __do_sys_getdents64+0x6b/0xd8
Oct 16 00:22:19 frameRAID kernel: ? compat_filldir+0x17a/0x17a
Oct 16 00:22:19 frameRAID kernel: do_syscall_64+0x6b/0x81
Oct 16 00:22:19 frameRAID kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce
Oct 16 00:22:19 frameRAID kernel: RIP: 0033:0x14d8ed624283
Oct 16 00:22:19 frameRAID kernel: Code: 89 df e8 20 05 fb ff 48 83 c4 08 48 89 e8 5b 5d c3 66 0f 1f 44 00 00 b8 ff ff ff 7f 48 39 c2 48 0f 47 d0 b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 61 0b 11 00 f7 d8
Oct 16 00:22:19 frameRAID kernel: RSP: 002b:00007ffc67faf5b8 EFLAGS: 00000293 ORIG_RAX: 00000000000000d9
Oct 16 00:22:19 frameRAID kernel: RAX: ffffffffffffffda RBX: 000000000046ff90 RCX: 000014d8ed624283
Oct 16 00:22:19 frameRAID kernel: RDX: 0000000000008000 RSI: 000000000046ffc0 RDI: 0000000000000007
Oct 16 00:22:19 frameRAID kernel: RBP: 000000000046ff94 R08: 00000000004872c0 R09: 000014d8ed735db0
Oct 16 00:22:19 frameRAID kernel: R10: 00000000004d34e0 R11: 0000000000000293 R12: ffffffffffffff88
Oct 16 00:22:19 frameRAID kernel: R13: 0000000000000000 R14: 00000000004d33e0 R15: 00000000000010b7
Oct 16 00:22:19 frameRAID kernel: </TASK>
Oct 16 00:22:19 frameRAID kernel: XFS (md1p1): Corruption detected. Unmount and run xfs_repair
Oct 16 00:22:19 frameRAID kernel: XFS (md1p1): Internal error !xfs_dir2_namecheck(dep->name, dep->namelen) at line 466 of file fs/xfs/xfs_dir2_readdir.c.  Caller xfs_dir2_leaf_getdents+0x220/0x322 [xfs]
Oct 16 00:22:19 frameRAID kernel: CPU: 9 PID: 1589 Comm: find Tainted: P        W  O       6.1.49-Unraid #1
Oct 16 00:22:19 frameRAID kernel: Hardware name: ASUS System Product Name/PRIME B550M-A, BIOS 3205 08/14/2023
Oct 16 00:22:19 frameRAID kernel: Call Trace:
Oct 16 00:22:19 frameRAID kernel: <TASK>
Oct 16 00:22:19 frameRAID kernel: dump_stack_lvl+0x44/0x5c
Oct 16 00:22:19 frameRAID kernel: xfs_corruption_error+0x63/0x83 [xfs]
Oct 16 00:22:19 frameRAID kernel: ? preempt_latency_start+0x1e/0x46
Oct 16 00:22:19 frameRAID kernel: xfs_dir2_leaf_getdents+0x253/0x322 [xfs]
Oct 16 00:22:19 frameRAID kernel: ? xfs_dir2_leaf_getdents+0x220/0x322 [xfs]
Oct 16 00:22:19 frameRAID kernel: xfs_readdir+0x14e/0x190 [xfs]
Oct 16 00:22:19 frameRAID kernel: ? mas_destroy+0xa8/0xbb
Oct 16 00:22:19 frameRAID kernel: iterate_dir+0x97/0x146
Oct 16 00:22:19 frameRAID kernel: __do_sys_getdents64+0x6b/0xd8
Oct 16 00:22:19 frameRAID kernel: ? compat_filldir+0x17a/0x17a
Oct 16 00:22:19 frameRAID kernel: do_syscall_64+0x6b/0x81
Oct 16 00:22:19 frameRAID kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce
Oct 16 00:22:19 frameRAID kernel: RIP: 0033:0x14d8ed624283
Oct 16 00:22:19 frameRAID kernel: Code: 89 df e8 20 05 fb ff 48 83 c4 08 48 89 e8 5b 5d c3 66 0f 1f 44 00 00 b8 ff ff ff 7f 48 39 c2 48 0f 47 d0 b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 61 0b 11 00 f7 d8
Oct 16 00:22:19 frameRAID kernel: RSP: 002b:00007ffc67faf5b8 EFLAGS: 00000293 ORIG_RAX: 00000000000000d9
Oct 16 00:22:19 frameRAID kernel: RAX: ffffffffffffffda RBX: 000000000046ff90 RCX: 000014d8ed624283
Oct 16 00:22:19 frameRAID kernel: RDX: 0000000000008000 RSI: 000000000046ffc0 RDI: 0000000000000007
Oct 16 00:22:19 frameRAID kernel: RBP: 000000000046ff94 R08: 0000000000021071 R09: 0000000000000000
Oct 16 00:22:19 frameRAID kernel: R10: 00000000004e6870 R11: 0000000000000293 R12: ffffffffffffff88
Oct 16 00:22:19 frameRAID kernel: R13: 0000000000000000 R14: 00000000004e6770 R15: 00000000000010b7
Oct 16 00:22:19 frameRAID kernel: </TASK>
Oct 16 00:22:19 frameRAID kernel: XFS (md1p1): Corruption detected. Unmount and run xfs_repair
Oct 16 00:24:58 frameRAID kernel: XFS (md1p1): Internal error !xfs_dir2_namecheck(dep->name, dep->namelen) at line 466 of file fs/xfs/xfs_dir2_readdir.c.  Caller xfs_dir2_leaf_getdents+0x220/0x322 [xfs]
Oct 16 00:24:58 frameRAID kernel: CPU: 8 PID: 5490 Comm: find Tainted: P        W  O       6.1.49-Unraid #1
Oct 16 00:24:58 frameRAID kernel: Hardware name: ASUS System Product Name/PRIME B550M-A, BIOS 3205 08/14/2023
Oct 16 00:24:58 frameRAID kernel: Call Trace:
Oct 16 00:24:58 frameRAID kernel: <TASK>
Oct 16 00:24:58 frameRAID kernel: dump_stack_lvl+0x44/0x5c
Oct 16 00:24:58 frameRAID kernel: xfs_corruption_error+0x63/0x83 [xfs]
Oct 16 00:24:58 frameRAID kernel: ? filldir64+0x94/0x173
Oct 16 00:24:58 frameRAID kernel: xfs_dir2_leaf_getdents+0x253/0x322 [xfs]
Oct 16 00:24:58 frameRAID kernel: ? xfs_dir2_leaf_getdents+0x220/0x322 [xfs]
Oct 16 00:24:58 frameRAID kernel: xfs_readdir+0x14e/0x190 [xfs]
Oct 16 00:24:58 frameRAID kernel: iterate_dir+0x97/0x146
Oct 16 00:24:58 frameRAID kernel: __do_sys_getdents64+0x6b/0xd8
Oct 16 00:24:58 frameRAID kernel: ? compat_filldir+0x17a/0x17a
Oct 16 00:24:58 frameRAID kernel: do_syscall_64+0x6b/0x81
Oct 16 00:24:58 frameRAID kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce
Oct 16 00:24:58 frameRAID kernel: RIP: 0033:0x145de4472283
Oct 16 00:24:58 frameRAID kernel: Code: 89 df e8 20 05 fb ff 48 83 c4 08 48 89 e8 5b 5d c3 66 0f 1f 44 00 00 b8 ff ff ff 7f 48 39 c2 48 0f 47 d0 b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 61 0b 11 00 f7 d8
Oct 16 00:24:58 frameRAID kernel: RSP: 002b:00007ffcf6011978 EFLAGS: 00000293 ORIG_RAX: 00000000000000d9
Oct 16 00:24:58 frameRAID kernel: RAX: ffffffffffffffda RBX: 000000000046ff90 RCX: 0000145de4472283
Oct 16 00:24:58 frameRAID kernel: RDX: 0000000000008000 RSI: 000000000046ffc0 RDI: 0000000000000007
Oct 16 00:24:58 frameRAID kernel: RBP: 000000000046ff94 R08: 00000000004872c0 R09: 0000145de4583db0
Oct 16 00:24:58 frameRAID kernel: R10: 00000000004d34e0 R11: 0000000000000293 R12: ffffffffffffff88
Oct 16 00:24:58 frameRAID kernel: R13: 0000000000000000 R14: 00000000004d33e0 R15: 00000000000010b7
Oct 16 00:24:58 frameRAID kernel: </TASK>
Oct 16 00:24:58 frameRAID kernel: XFS (md1p1): Corruption detected. Unmount and run xfs_repair
Oct 16 00:24:58 frameRAID kernel: XFS (md1p1): Internal error !xfs_dir2_namecheck(dep->name, dep->namelen) at line 466 of file fs/xfs/xfs_dir2_readdir.c.  Caller xfs_dir2_leaf_getdents+0x220/0x322 [xfs]
Oct 16 00:24:58 frameRAID kernel: CPU: 1 PID: 5490 Comm: find Tainted: P        W  O       6.1.49-Unraid #1
Oct 16 00:24:58 frameRAID kernel: Hardware name: ASUS System Product Name/PRIME B550M-A, BIOS 3205 08/14/2023
Oct 16 00:24:58 frameRAID kernel: Call Trace:
Oct 16 00:24:58 frameRAID kernel: <TASK>

 

I will give a downgrade to 6.11 a really really last chance for using unRAID. And then I will stop using unRAID at home. And no I am not using an FRITZ!Box, my home network is entirely mikrotik device based. I moved my "mission critical" containers to my mikrotik devices since a few months they support containers, so a unRAID crash is not a big showstopper anymore.

 

I hope this bringing some light into the darkness. So if you think you have a hardware issue, it is probably not if you running 6.12.x.

 

The problem must be a mix of kernel, network-drivers, docker based incompatibility issues, which crashed parts of the ip stack and some other processes in the linux kernel must write then to other memory addresses which leads to metadata corruptions, which should not occur but it does.

  • Quote
    • Community Expert
    10 hours ago, Crash_Override said: 
    Oct 14 06:34:46 frameRAID kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Oct 14 06:34:46 frameRAID kernel: ? _raw_spin_unlock+0x14/0x29
Oct 14 06:34:46 frameRAID kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

    Macvlan call traces will usually end up crashing the server, switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

  • Quote
    • Upvote 1
    • Author

    Hmmmm strange my two Datacenter servers HP and Fujitsu dont crash with macvlan. They have broadcom NICs maybe driver is different than intel.

    I will give it a try.

  • Quote

    • Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Reply to this topic...
    Go to topic listing

    Account

    Navigation

    Search

    Search

    Configure browser push notifications
    Chrome (Android)
    1. Tap the lock icon next to the address bar.
    2. Tap Permissions → Notifications.
    3. Adjust your preference.
    Chrome (Desktop)
    1. Click the padlock icon in the address bar.
    2. Select Site settings.
    3. Find Notifications and adjust your preference.