Hi there - thanks in advance for the help.

 

For the past few days, I've been pulling my hair out at an assortment of BTRFS errors and issues with my Unraid server.

 

Originally, the messages were like the below, something about a corrupt leaf.

 

BTRFS critical (device dm-9): corrupt leaf: block=825281921024 slot=40 extent bytenr=20484227072 len=4096 unknown inline ref type: 18

 

As I troubleshot these issues, I kept getting more and different BTRFS issues. Was also getting ATA errors, but I'm not sure if they were related.

 

I'm now at the point where Unraid won't even present me with shares and getting constant errors like the below.

 

Nov 12 13:53:21 Zebra emhttpd: error: get_filesystem_status, 7329: Invalid argument (22): scandir Invalid argument
Nov 12 13:53:21 Zebra kernel: BTRFS critical (device dm-9): unable to find logical 6410751739243216896 length 4096
Nov 12 13:53:21 Zebra kernel: BTRFS critical (device dm-9): unable to find logical 6410751739243216896 length 16384

 

I've done a lot of reading through the forums and tried a bunch of commands/BTRFS troubleshooting to no avail. Additional things I've tried:

1. Memtest - 4 passes, no errors, all pass
2. Move all of the cache data off a SATA SSD (Samsung 860 EVO) to an NVME SSD (Samsung 970 EVO Plus)
3. Unplug the SATA SSD physically from the system

 

I'm attaching my latest diagnostics, as well as one of the original syslogs I saved off from when these problems originated.

 

I assume this is has to be hardware related, but am at a total loss how to continue troubleshooting or diagnosing this. Any help is super appreciated. Thank you so much!

syslog-combined-11-11-2023.txt zebra-diagnostics-20231112-1322.zip

At this point I would recommend backup and re-format the pool, if it happens again you likely have an underlying hardware issue, you should also run a scrub on disk8 since data corruption has been detected.

Thanks! The NVME cache went offline due to errors and so I plugged the SATA cache drive back in, reformatted it to XFS (via this post) and everything seemed to work much better last night with minimal errors in the logs. However, woke up to the server unresponsive and even direct console access not responsive. Hard reset and everything seems ok again, but I don't know what caused the issue. I enabled syslog server so will see if it happens again.

 

Are these errors anything to be concerned about?

 

Nov 10 22:37:41 Zebra kernel: WARNING: CPU: 6 PID: 0 at net/netfilter/nf_nat_core.c:594 nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Nov 10 22:37:41 Zebra kernel: Modules linked in: bluetooth ecdh_generic ecc nvidia_uvm(PO) xt_connmark xt_mark xt_comment iptable_raw udp_diag cmac cifs asn1_decoder cifs_arc4 cifs_md4 oid_registry dns_resolver xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap xt_nat xt_tcpudp veth macvlan xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xt_addrtype br_netfilter dm_crypt dm_mod md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6687(O) nct6683 iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc ixgbe xfrm_algo mdio r8169 realtek intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp
Nov 10 22:37:41 Zebra kernel: coretemp kvm_intel nvidia_drm(PO) nvidia_modeset(PO) i915 kvm nvidia(PO) crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 aesni_intel iosf_mbi drm_buddy crypto_simd i2c_algo_bit cryptd ttm drm_display_helper drm_kms_helper mei_hdcp mei_pxp intel_gtt rapl intel_cstate drm wmi_bmof intel_wmi_thunderbolt mxm_wmi nvme agpgart i2c_i801 intel_uncore mei_me apex(O) i2c_smbus ahci syscopyarea sr_mod input_leds sysfillrect sysimgblt nvme_core i2c_core gasket(O) mei libahci cdrom led_class joydev intel_pch_thermal fb_sys_fops thermal fan video wmi backlight intel_pmc_core acpi_pad acpi_tad button unix [last unloaded: xfrm_algo]
Nov 10 22:37:41 Zebra kernel: CPU: 6 PID: 0 Comm: swapper/6 Tainted: P        W  O       6.1.49-Unraid #1
Nov 10 22:37:41 Zebra kernel: Hardware name: Micro-Star International Co., Ltd. MS-7C75/Z490-A PRO (MS-7C75), BIOS 2.80 01/30/2021
Nov 10 22:37:41 Zebra kernel: RIP: 0010:nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Nov 10 22:37:41 Zebra kernel: Code: a8 80 75 26 48 8d 73 58 48 8d 7c 24 20 e8 18 9b 07 00 48 8d 43 0c 4c 8b bb 88 00 00 00 48 89 44 24 18 eb 54 0f ba e0 08 73 07 <0f> 0b e9 75 06 00 00 48 8d 73 58 48 8d 7c 24 20 e8 eb 9a 07 00 48
Nov 10 22:37:41 Zebra kernel: RSP: 0018:ffffc900002f8708 EFLAGS: 00010282
Nov 10 22:37:41 Zebra kernel: RAX: 0000000000000180 RBX: ffff8884d9db6200 RCX: ffff888106349e00
Nov 10 22:37:41 Zebra kernel: RDX: 0000000000000000 RSI: ffffc900002f87ec RDI: ffff8884d9db6200
Nov 10 22:37:41 Zebra kernel: RBP: ffffc900002f87d0 R08: 000000003d01a8c0 R09: 0000000000000000
Nov 10 22:37:41 Zebra kernel: R10: 0000000000000158 R11: 0000000000000000 R12: ffffc900002f87ec
Nov 10 22:37:41 Zebra kernel: R13: 0000000000000000 R14: ffffc900002f88d0 R15: 0000000000000001
Nov 10 22:37:41 Zebra kernel: FS:  0000000000000000(0000) GS:ffff88883e180000(0000) knlGS:0000000000000000
Nov 10 22:37:41 Zebra kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 10 22:37:41 Zebra kernel: CR2: 000014d1d49f3000 CR3: 00000001d7f42001 CR4: 00000000007706e0
Nov 10 22:37:41 Zebra kernel: PKRU: 55555554
Nov 10 22:37:41 Zebra kernel: Call Trace:
Nov 10 22:37:41 Zebra kernel: <IRQ>
Nov 10 22:37:41 Zebra kernel: ? __warn+0xab/0x122
Nov 10 22:37:41 Zebra kernel: ? report_bug+0x109/0x17e
Nov 10 22:37:41 Zebra kernel: ? nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Nov 10 22:37:41 Zebra kernel: ? handle_bug+0x41/0x6f
Nov 10 22:37:41 Zebra kernel: ? exc_invalid_op+0x13/0x60
Nov 10 22:37:41 Zebra kernel: ? asm_exc_invalid_op+0x16/0x20
Nov 10 22:37:41 Zebra kernel: ? nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Nov 10 22:37:41 Zebra kernel: ? nf_nat_setup_info+0x44/0x7d1 [nf_nat]
Nov 10 22:37:41 Zebra kernel: ? xt_write_recseq_end+0xf/0x1c [ip_tables]
Nov 10 22:37:41 Zebra kernel: ? __local_bh_enable_ip+0x56/0x6b
Nov 10 22:37:41 Zebra kernel: ? ipt_do_table+0x57a/0x5bf [ip_tables]
Nov 10 22:37:41 Zebra kernel: ? xt_write_recseq_end+0xf/0x1c [ip_tables]
Nov 10 22:37:41 Zebra kernel: __nf_nat_alloc_null_binding+0x66/0x81 [nf_nat]
Nov 10 22:37:41 Zebra kernel: nf_nat_inet_fn+0xc0/0x1a8 [nf_nat]
Nov 10 22:37:41 Zebra kernel: nf_nat_ipv4_local_in+0x2a/0xaa [nf_nat]
Nov 10 22:37:41 Zebra kernel: nf_hook_slow+0x3a/0x96
Nov 10 22:37:41 Zebra kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Nov 10 22:37:41 Zebra kernel: NF_HOOK.constprop.0+0x79/0xd9
Nov 10 22:37:41 Zebra kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Nov 10 22:37:41 Zebra kernel: ip_sabotage_in+0x4f/0x60 [br_netfilter]
Nov 10 22:37:41 Zebra kernel: nf_hook_slow+0x3a/0x96
Nov 10 22:37:41 Zebra kernel: ? ip_rcv_finish_core.constprop.0+0x3e8/0x3e8
Nov 10 22:37:41 Zebra kernel: NF_HOOK.constprop.0+0x79/0xd9
Nov 10 22:37:41 Zebra kernel: ? ip_rcv_finish_core.constprop.0+0x3e8/0x3e8
Nov 10 22:37:41 Zebra kernel: __netif_receive_skb_one_core+0x77/0x9c
Nov 10 22:37:41 Zebra kernel: netif_receive_skb+0xbf/0x127
Nov 10 22:37:41 Zebra kernel: br_handle_frame_finish+0x438/0x472 [bridge]
Nov 10 22:37:41 Zebra kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
Nov 10 22:37:41 Zebra kernel: br_nf_hook_thresh+0xe2/0x109 [br_netfilter]
Nov 10 22:37:41 Zebra kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
Nov 10 22:37:41 Zebra kernel: br_nf_pre_routing_finish+0x2c1/0x2ec [br_netfilter]
Nov 10 22:37:41 Zebra kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
Nov 10 22:37:41 Zebra kernel: ? NF_HOOK.isra.0+0xe4/0x140 [br_netfilter]
Nov 10 22:37:41 Zebra kernel: ? br_nf_hook_thresh+0x109/0x109 [br_netfilter]
Nov 10 22:37:41 Zebra kernel: br_nf_pre_routing+0x236/0x24a [br_netfilter]
Nov 10 22:37:41 Zebra kernel: ? br_nf_hook_thresh+0x109/0x109 [br_netfilter]
Nov 10 22:37:41 Zebra kernel: br_handle_frame+0x277/0x2e0 [bridge]
Nov 10 22:37:41 Zebra kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
Nov 10 22:37:41 Zebra kernel: __netif_receive_skb_core.constprop.0+0x4fa/0x6e9
Nov 10 22:37:41 Zebra kernel: ? br_nf_pre_routing_finish+0x2c1/0x2ec [br_netfilter]
Nov 10 22:37:41 Zebra kernel: __netif_receive_skb_list_core+0x8a/0x11e
Nov 10 22:37:41 Zebra kernel: netif_receive_skb_list_internal+0x1d2/0x20b
Nov 10 22:37:41 Zebra kernel: gro_normal_list+0x1d/0x3f
Nov 10 22:37:41 Zebra kernel: napi_complete_done+0x7b/0x11a
Nov 10 22:37:41 Zebra kernel: ixgbe_poll+0xdb6/0xe7d [ixgbe]
Nov 10 22:37:41 Zebra kernel: __napi_poll.constprop.0+0x28/0x124
Nov 10 22:37:41 Zebra kernel: net_rx_action+0x159/0x24f
Nov 10 22:37:41 Zebra kernel: __do_softirq+0x126/0x288
Nov 10 22:37:41 Zebra kernel: __irq_exit_rcu+0x5e/0xb8
Nov 10 22:37:41 Zebra kernel: common_interrupt+0x9b/0xc1
Nov 10 22:37:41 Zebra kernel: </IRQ>
Nov 10 22:37:41 Zebra kernel: <TASK>
Nov 10 22:37:41 Zebra kernel: asm_common_interrupt+0x22/0x40
Nov 10 22:37:41 Zebra kernel: RIP: 0010:cpuidle_enter_state+0x11d/0x202
Nov 10 22:37:41 Zebra kernel: Code: 20 22 a0 ff 45 84 ff 74 1b 9c 58 0f 1f 40 00 0f ba e0 09 73 08 0f 0b fa 0f 1f 44 00 00 31 ff e8 4c e3 a4 ff fb 0f 1f 44 00 00 <45> 85 e4 0f 88 ba 00 00 00 48 8b 04 24 49 63 cc 48 6b d1 68 49 29
Nov 10 22:37:41 Zebra kernel: RSP: 0018:ffffc90000153e98 EFLAGS: 00000246
Nov 10 22:37:41 Zebra kernel: RAX: ffff88883e180000 RBX: ffff88883e1b6470 RCX: 0000000000000000
Nov 10 22:37:41 Zebra kernel: RDX: 000008e2b1a77de2 RSI: ffffffff820ed4af RDI: ffffffff820ed9b8
Nov 10 22:37:41 Zebra kernel: RBP: 0000000000000002 R08: 0000000000000002 R09: 0000000000000002
Nov 10 22:37:41 Zebra kernel: R10: 0000000000000020 R11: 000000000000028e R12: 0000000000000002
Nov 10 22:37:41 Zebra kernel: R13: ffffffff823205c0 R14: 000008e2b1a77de2 R15: 0000000000000000
Nov 10 22:37:41 Zebra kernel: ? cpuidle_enter_state+0xf7/0x202
Nov 10 22:37:41 Zebra kernel: cpuidle_enter+0x2a/0x38
Nov 10 22:37:41 Zebra kernel: do_idle+0x18d/0x1fb
Nov 10 22:37:41 Zebra kernel: cpu_startup_entry+0x1d/0x1f
Nov 10 22:37:41 Zebra kernel: start_secondary+0x101/0x101
Nov 10 22:37:41 Zebra kernel: secondary_startup_64_no_verify+0xce/0xdb
Nov 10 22:37:41 Zebra kernel: </TASK>
Nov 10 22:37:41 Zebra kernel: ---[ end trace 0000000000000000 ]---

 

Change the docker network to ipvlan and reboot.

All this seemed to work; I think the NVME SSD either was bad or the file system was corrupt. I replaced it with a new SSD (formatted with XFS) and no more errors. Swapped the docker network to ipvlan and have had a solid 24h of clean logs. Thanks for the help, JorgeB!