I have a 1 drive cache (500GB Samsung SSD). It’s relatively new, about a month old or so, and the whole rig is also quite new, maybe three or four months.

 

About a week ago, I started getting BTRFS checksum errors every other day or so. Specifically:

 

`BTRFS error (device nvme0n1p1): block=274071552 write time tree block corruption detected`

 

I’m attaching the diagnostics in case that’s helpful for anyone. 
 

Things I’ve tried so far:

 

- Reseating my memory sticks. I actually had them in the wrong slots for a two stick config, so I also resoled that

- Running a full memtest. All green, no errors

 

I… unfortunately have no idea what to try next, but if anyone has ideas I’m definitely all ears!

r710-diagnostics-20231115-1857.zip

9 hours ago, smoores said:

write time tree block corruption detected`

This usually meant bad RAM, but with this kernel there are possibly some false positives for some users, I would recommend trying zfs instead, if if there are also issues with that then likely there's a hardware problem.

I haven’t tried ZFS, but I did try reformatting to xfs a few days ago, and eventually got a kernel panic (after about a day), which I assumed was the result of having written the bad data? Though now that I’m looking more closely at the kernel dump, I’m less certain that that makes sense. 
 

The error message was “Kernel panic - not syncing: Fatal exception in interrupt“, and the stack trace looked like:

 

ip_finish_output2+0x36a/0x398

ip_send_skb+0x15/0x3b

udp_send_skb+0x278/0x2e6

udp_sendnsg+0x756/0x9bd

? filenap_get_read_batch+9x8d/0x16c

? ip neigh_gu4+0×8b/0x86

1 nod_ob jog_state+0x11e/0x153

slab_post_alloc_hook+0x11c/0x15e

1 sock_sendmsg_nosec+0x2b/0x40

sock_sendnsg_nosec+0x2b/0x40

sock_write_iter+0x89/0xb8

uts_write+0x10c/0x169

ksys_write+0x76/0xc2

do_sysca 11_64+0x68/0x81

entry_SYSCALL_64_after_huframe+0x64/Oxce
 

Which looks like a networking event?

 

If you think ZFS will give more information, I can give that a shot, for sure.

Difficult to say based on that alone, enable the syslog server and post that after a some issues, there may be some more clues there.

Alright, thanks, I'll give that a shot. I've just enabled the syslog server: I'll try switching back to xfs and report back if another kernel panic comes up, with the syslogs. Thanks so much for your help, Jorge!

 

EDIT: Trying ZFS, actually, as you recommended.

Edited November 16, 20232 yr by smoores

@JorgeB This same kernel panic happened again this morning. This is what was in the syslog (the actual kernel panic message doesn't seem to be there, but the kernel dump does):

 

Nov 17 09:20:05 R710 kernel: ------------[ cut here ]------------
Nov 17 09:20:05 R710 kernel: WARNING: CPU: 1 PID: 0 at net/netfilter/nf_nat_core.c:594 nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Nov 17 09:20:05 R710 kernel: Modules linked in: xt_nat veth nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter vhost_net tun vhost kvm_intel kvm md_mod xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 vhost_iotlb xfs zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp macvtap macvlan tap bridge stp llc bonding tls e1000e r8169 realtek i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 drm aesni_intel btusb
Nov 17 09:20:05 R710 kernel: crypto_simd btrtl cryptd btbcm rapl intel_cstate mei_hdcp mei_pxp wmi_bmof intel_uncore btintel nvme i2c_i801 intel_gtt bluetooth thunderbolt i2c_smbus agpgart mei_me ahci i2c_core mei input_leds nvme_core ecdh_generic led_class joydev libahci ecc syscopyarea sysfillrect sysimgblt fb_sys_fops tpm_crb video tpm_tis tpm_tis_core wmi tpm backlight intel_pmc_core acpi_pad acpi_tad button unix [last unloaded: md_mod]
Nov 17 09:20:05 R710 kernel: CPU: 1 PID: 0 Comm: swapper/1 Tainted: P        W  O       6.1.49-Unraid #1
Nov 17 09:20:05 R710 kernel: Hardware name: To Be Filled By O.E.M. Z690 Taichi/Z690 Taichi, BIOS 12.01 06/17/2022
Nov 17 09:20:05 R710 kernel: RIP: 0010:nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Nov 17 09:20:05 R710 kernel: Code: a8 80 75 26 48 8d 73 58 48 8d 7c 24 20 e8 18 db fc ff 48 8d 43 0c 4c 8b bb 88 00 00 00 48 89 44 24 18 eb 54 0f ba e0 08 73 07 <0f> 0b e9 75 06 00 00 48 8d 73 58 48 8d 7c 24 20 e8 eb da fc ff 48
Nov 17 09:20:05 R710 kernel: RSP: 0018:ffffc900000fc788 EFLAGS: 00010282
Nov 17 09:20:05 R710 kernel: RAX: 0000000000000180 RBX: ffff8881567c7200 RCX: ffff8881043779c0
Nov 17 09:20:05 R710 kernel: RDX: 0000000000000000 RSI: ffffc900000fc86c RDI: ffff8881567c7200
Nov 17 09:20:05 R710 kernel: RBP: ffffc900000fc850 R08: 0000000000000000 R09: 0000000000000000
Nov 17 09:20:05 R710 kernel: R10: 0000000000000158 R11: ffff888100400128 R12: ffffc900000fc86c
Nov 17 09:20:05 R710 kernel: R13: 0000000000000000 R14: ffffc900000fc950 R15: 0000000000000001
Nov 17 09:20:05 R710 kernel: FS:  0000000000000000(0000) GS:ffff88888f440000(0000) knlGS:0000000000000000
Nov 17 09:20:05 R710 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 17 09:20:05 R710 kernel: CR2: 00001516eedd9000 CR3: 000000000420a000 CR4: 0000000000750ee0
Nov 17 09:20:05 R710 kernel: PKRU: 55555554
Nov 17 09:20:05 R710 kernel: Call Trace:
Nov 17 09:20:05 R710 kernel: <IRQ>
Nov 17 09:20:05 R710 kernel: ? __warn+0xab/0x122
Nov 17 09:20:05 R710 kernel: ? report_bug+0x109/0x17e
Nov 17 09:20:05 R710 kernel: ? nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Nov 17 09:20:05 R710 kernel: ? handle_bug+0x41/0x6f
Nov 17 09:20:05 R710 kernel: ? exc_invalid_op+0x13/0x60
Nov 17 09:20:05 R710 kernel: ? asm_exc_invalid_op+0x16/0x20
Nov 17 09:20:05 R710 kernel: ? nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Nov 17 09:20:05 R710 kernel: ? nf_nat_setup_info+0x44/0x7d1 [nf_nat]
Nov 17 09:20:05 R710 kernel: ? percpu_counter_add_batch+0x85/0xa2
Nov 17 09:20:05 R710 kernel: ? xt_write_recseq_end+0xf/0x1c [ip_tables]
Nov 17 09:20:05 R710 kernel: ? __local_bh_enable_ip+0x56/0x6b
Nov 17 09:20:05 R710 kernel: ? ipt_do_table+0x57a/0x5bf [ip_tables]
Nov 17 09:20:05 R710 kernel: ? rt_dst_alloc+0x2f/0xa0
Nov 17 09:20:05 R710 kernel: ? xt_write_recseq_end+0xf/0x1c [ip_tables]
Nov 17 09:20:05 R710 kernel: __nf_nat_alloc_null_binding+0x66/0x81 [nf_nat]
Nov 17 09:20:05 R710 kernel: nf_nat_inet_fn+0xc0/0x1a8 [nf_nat]
Nov 17 09:20:05 R710 kernel: nf_nat_ipv4_local_in+0x2a/0xaa [nf_nat]
Nov 17 09:20:05 R710 kernel: nf_hook_slow+0x3a/0x96
Nov 17 09:20:05 R710 kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Nov 17 09:20:05 R710 kernel: NF_HOOK.constprop.0+0x79/0xd9
Nov 17 09:20:05 R710 kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Nov 17 09:20:05 R710 kernel: ip_sabotage_in+0x4f/0x60 [br_netfilter]
Nov 17 09:20:05 R710 kernel: nf_hook_slow+0x3a/0x96
Nov 17 09:20:05 R710 kernel: ? ip_rcv_finish_core.constprop.0+0x3e8/0x3e8
Nov 17 09:20:05 R710 kernel: NF_HOOK.constprop.0+0x79/0xd9
Nov 17 09:20:05 R710 kernel: ? ip_rcv_finish_core.constprop.0+0x3e8/0x3e8
Nov 17 09:20:05 R710 kernel: __netif_receive_skb_one_core+0x77/0x9c
Nov 17 09:20:05 R710 kernel: netif_receive_skb+0xbf/0x127
Nov 17 09:20:05 R710 kernel: br_handle_frame_finish+0x438/0x472 [bridge]
Nov 17 09:20:05 R710 kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
Nov 17 09:20:05 R710 kernel: br_nf_hook_thresh+0xe2/0x109 [br_netfilter]
Nov 17 09:20:05 R710 kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
Nov 17 09:20:05 R710 kernel: br_nf_pre_routing_finish+0x2c1/0x2ec [br_netfilter]
Nov 17 09:20:05 R710 kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
Nov 17 09:20:05 R710 kernel: ? NF_HOOK.isra.0+0xe4/0x140 [br_netfilter]
Nov 17 09:20:05 R710 kernel: ? br_nf_hook_thresh+0x109/0x109 [br_netfilter]
Nov 17 09:20:05 R710 kernel: br_nf_pre_routing+0x236/0x24a [br_netfilter]
Nov 17 09:20:05 R710 kernel: ? br_nf_hook_thresh+0x109/0x109 [br_netfilter]
Nov 17 09:20:05 R710 kernel: br_handle_frame+0x277/0x2e0 [bridge]
Nov 17 09:20:05 R710 kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
Nov 17 09:20:05 R710 kernel: __netif_receive_skb_core.constprop.0+0x4fa/0x6e9
Nov 17 09:20:05 R710 kernel: ? __skb_checksum+0x5c/0x283
Nov 17 09:20:05 R710 kernel: __netif_receive_skb_list_core+0x8a/0x11e
Nov 17 09:20:05 R710 kernel: netif_receive_skb_list_internal+0x1d2/0x20b
Nov 17 09:20:05 R710 kernel: gro_normal_list+0x1d/0x3f
Nov 17 09:20:05 R710 kernel: napi_complete_done+0x7b/0x11a
Nov 17 09:20:05 R710 kernel: rtl8169_poll+0x3f0/0x414 [r8169]
Nov 17 09:20:05 R710 kernel: __napi_poll.constprop.0+0x28/0x124
Nov 17 09:20:05 R710 kernel: net_rx_action+0x159/0x24f
Nov 17 09:20:05 R710 kernel: __do_softirq+0x126/0x288
Nov 17 09:20:05 R710 kernel: __irq_exit_rcu+0x5e/0xb8
Nov 17 09:20:05 R710 kernel: common_interrupt+0x9b/0xc1
Nov 17 09:20:05 R710 kernel: </IRQ>
Nov 17 09:20:05 R710 kernel: <TASK>
Nov 17 09:20:05 R710 kernel: asm_common_interrupt+0x22/0x40
Nov 17 09:20:05 R710 kernel: RIP: 0010:cpuidle_enter_state+0x11d/0x202
Nov 17 09:20:05 R710 kernel: Code: 20 22 a0 ff 45 84 ff 74 1b 9c 58 0f 1f 40 00 0f ba e0 09 73 08 0f 0b fa 0f 1f 44 00 00 31 ff e8 4c e3 a4 ff fb 0f 1f 44 00 00 <45> 85 e4 0f 88 ba 00 00 00 48 8b 04 24 49 63 cc 48 6b d1 68 49 29
Nov 17 09:20:05 R710 kernel: RSP: 0018:ffffc900000efe98 EFLAGS: 00000246
Nov 17 09:20:05 R710 kernel: RAX: ffff88888f440000 RBX: ffff88888f476400 RCX: 0000000000000000
Nov 17 09:20:05 R710 kernel: RDX: 0000488a462161ae RSI: ffffffff820ed4af RDI: ffffffff820ed9b8
Nov 17 09:20:05 R710 kernel: RBP: 0000000000000004 R08: 0000000000000000 R09: 0000000000000000
Nov 17 09:20:05 R710 kernel: R10: 0000000000000020 R11: 0000000000000606 R12: 0000000000000004
Nov 17 09:20:05 R710 kernel: R13: ffffffff823205c0 R14: 0000488a462161ae R15: 0000000000000000
Nov 17 09:20:05 R710 kernel: ? cpuidle_enter_state+0xf7/0x202
Nov 17 09:20:05 R710 kernel: cpuidle_enter+0x2a/0x38
Nov 17 09:20:05 R710 kernel: do_idle+0x18d/0x1fb
Nov 17 09:20:05 R710 kernel: cpu_startup_entry+0x1d/0x1f
Nov 17 09:20:05 R710 kernel: start_secondary+0x101/0x101
Nov 17 09:20:05 R710 kernel: secondary_startup_64_no_verify+0xce/0xdb
Nov 17 09:20:05 R710 kernel: </TASK>
Nov 17 09:20:05 R710 kernel: ---[ end trace 0000000000000000 ]---

 

Change the docker network to ipvlan.

Alright, done! Do you think there's a relationship between this kernel panic and the btrfs write error? If this seems stable for a week, would it make sense to switch back to btrfs, or was that somehow unrelated?

1 hour ago, smoores said:

Do you think there's a relationship between this kernel panic and the btrfs write error?

Nope, ipvlan if for the last panic.

I got another kernel panic this morning; this one seemed more likely to be related to the btrfs stuff. Oddly it didn't get written to the syslog, but here's what I could grab from the monitor before I restarted:

 

rcu_do_batch+0x27b/0x52c
rcu_core+0x265/0x2ac
? tinekeeping_get_ns+0x19/0x33
_do_sortirq+0x126/0x288
_irg_exit_rcu+0×5/0xb8
sysuec_apic_tiner_interrupt +0x85/Oxa6
</IRQ>
<TASK>
asn_sysuec_apic_timer_interrupt+0x16/0x20
RIP: 0010: _raw_spin_unlock_irqrestore+0x1a/0x3a
Code: 7a 7e 85 c0 75 05 of 1f 44 00 00 3 cc cc cc cc 0f 1f 44 00 00 53 48 89 13 e8 e1 cd 83 ff Of ba e3 09 73 06 fb Of 1f 44 00 00 <bf> 01 00 00 00 68 72 8 81 ff 65 85 05 d RSP: 0018:ffffc90004767a60 EFLAGS: 00000283
RAX: ffffc9000477a0 RBX: 0000000000000282 RCX: 0000000000000005
RDX: fffF8888af718d80 RSI: 0000000000000282 RDI:
fffF888881432040
RBP: ffff8888af7{8d80 ROB: 0000000000000000 R09: 0000000000000000
R10: ffffc90004767ad8 R11: ffffc9000477a0 R12: ffffc90004767a0
R13: ffffc90004767ad8 R14: ffffea0012c8c0 R15: 0000000000000001
free_unref_page_list+0x20e/0x23b
release_pages+0x2e6/0x30e pageuec_release+0x24/0x2d
invalidate_inode_pages2_range+0x294/0x2fc
fuse_open_connon+0x15/0x186
? fuse_open_common+0x18b/0x18b
do_dentry_open+0x192/0x304
path_openat+0x8f4/Oxa4d
7 filenap_map_pages+0x3d0/0x406
do_filp_open+0x55/0xb8
? getnane_flags+0x29/0x152
? knem_cache_a 11oc+0x122/0x14d
_raw_spin_unlock+0x14/0x29
do_sys_openat2+0x6c/0xd9
do_sys_open+0x3a/0x5a
do_sysca 11_64+0x68/0x81
entry_SYSCALL_64_after_hufrane+0x64/Oxce
RIP: 0033:0x154bd?d6c739
Code:
RSP: 002:00007ffedca13768 EFLAGS: 00000246 ORIG_RAX:
RAX:
00 Of 85 24 00 00 00 49 89 b 48 89 10 48 89 7 48 89 ce 4c 89 c2 4d 89 ca 4c 8b 44 24 08 4c 8b 4c 24 10 4c 89 ffffffffffffffda RBX: 0000000000000002 RCX:
0000000000000002
RDX: 0000000000000000 RSI: 0000000000088000
RBP:
0000154bd7d6c739
08 of 05 <c3› e9 d5 cb ff ff 41 57 41 56 53 48 81 ec 90 00
R10:
0000000000000000 ROB: 0000000000000000
RDI: 0000154bd2a1a260
R13:
0000000000000000 R11: 0000000000000246 R12:
R09: 0000000000000000
</TASK>
0000000000000030 R14: 0000000000000000
0000154d7d7a808
R15:
0000154d787a844
Modules linked in: ipulan xt_CHECKSUM ipt_REJECT nf_reject_ipu4 iptable_mangle iptable_nat iptable mangle uhost_net tun uhost uhost_iotlb veth xt_nat xt_tcpudp xt_conntrack xt_MASQI
iptable_nat nf_nat nf_contrack nf_defrag_ipu6 nf_defrag_ipu4 xt_addrtype
le_filter ip tables iptable_filter ip tables x_tables zun icode (PO)
br_netfilter xfs nd_nod wireguard curve25519_x86_64 1ibcurve25519_generic libchacha20poly1305 chacha x86_64 1
mrp macutap naculan tap bridge stp 11c bonding tis e1000e r8169 realtek zfs(PO) 1915 intel_rapl_ms
re led_class joydeu libahci ecc
Bode 01018 our 55 get a c c ca
Code:
48 89
d8 5b
RSP: 0018: ffffc90000003c08 EFLAGS: 00010046
RAX: IFTI8882dcfaB690
RDX:
FFF r8881140a4000
RBX: 0000000000000001 RCX: 0000000000000000
RSI:
cC cC 55 48 89 Fd 53 48 89 13 e8 c1 2c ca ff 48 86 45 00 a8 03 74 0a 48 8b 45 08 10 48 01 18 eb 04 <65> 48 01 18 5b 5d e9 83 79 ea ff 16 47 54 01 75 0b
ffff88816692de30
R08:
0000000000000001 RDI: CET88816692de30
R10:
LET+88832a70d240
ffff88832a70eb58 R09:
R13:
ffFfc90000003F18 R12:
ffffffff810473a6
FS:
ffC88816692de30
0000000900000216 1: 11110402093000 315: 00000011118
CS:
0000154b&747a808(0000) GS: TET/888881400000(0000) knIGS: 0000000000000
00
CR2: 0000154bd7375150 CR3: 000000035eedc000 CR4: 0000000000750efe
PKRU: 55555554
Kernel
Kernel Offset: disabled
panic - not syncing: Fatal exception in interrupt

 

Edited November 18, 20232 yr by smoores

This I don't know where it comes from, possibly a hardware issue.

Just wanted to follow up on this (thanks again for all of the support Jorge!).

Following a hunch that this was some sort of motherboard firmware/PCIe bus bug/edge case, I disabled some unnecessary hardware (namely, the HD Audio module, the WiFi module, and the "Killer" Ethernet port), and... everything now seems stable. I'm going to try switching back to btrfs this weekend, now that it's been over a week with no kernel panics, will follow up if anything interesting comes out of that.