nmethod

Some other relevant outputs root@Unraid:/dev# btrfs fi show Label: none uuid: a492518f-0ef3-4da9-a8c5-595aabe5ed90 Total devices 1 FS bytes used 1.71TiB devid 1 size 9.09TiB used 2.18TiB path /dev/sdg1 Label: none uuid: 469f8c77-5201-48d5-aec1-4b8352e8fe6e Total devices 2 FS bytes used 157.32GiB devid 2 size 476.94GiB used 246.03GiB path /dev/sdc1 devid 3 size 465.76GiB used 246.03GiB path /dev/nvme0n1p1 root@Unraid:/dev# dmesg | grep -i error [ 2.729353] ERST: Error Record Serialization Table (ERST) support is initialized. [ 49.300727] sas: ata20: end_device-1:0: dev error handler [ 49.509393] sas: ata20: end_device-1:0: dev error handler [ 49.509417] sas: ata21: end_device-1:1: dev error handler [ 50.055105] sas: ata20: end_device-1:0: dev error handler [ 50.055132] sas: ata21: end_device-1:1: dev error handler [ 50.055142] sas: ata22: end_device-1:2: dev error handler [ 50.495069] sas: ata20: end_device-1:0: dev error handler [ 50.495091] sas: ata21: end_device-1:1: dev error handler [ 50.495096] sas: ata22: end_device-1:2: dev error handler [ 50.495101] sas: ata23: end_device-1:3: dev error handler [ 161.302219] BTRFS error (device sdc1): unable to find ref byte nr 48157478912 parent 0 root 5 owner 20581559 offset 5049610240 [ 161.302223] BTRFS: Transaction aborted (error -2) [ 161.302446] BTRFS: error (device sdc1: state A) in __btrfs_free_extent:3067: errno=-2 No such entry [ 161.302509] BTRFS error (device sdc1: state EA): failed to run delayed ref for logical 48157478912 num_bytes 4096 type 178 action 2 ref_mod 1: -2 [ 161.302513] BTRFS: error (device sdc1: state EA) in btrfs_run_delayed_refs:2144: errno=-2 No such entry [ 161.302577] BTRFS: error (device sdc1: state EA) in btrfs_replay_log:2415: errno=-2 No such entry (Failed to recover log tree) [ 161.363233] BTRFS error (device sdc1: state EA): open_ctree faile root@Unraid:/dev# blkid /dev/sda1: LABEL_FATBOOT="UNRAID" LABEL="UNRAID" UUID="7B3E-14A4" BLOCK_SIZE="512" TYPE="vfat" /dev/loop1: TYPE="squashfs" /dev/sdf1: UUID="908ca018-2be7-44a7-8597-85dd60865998" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="37b823d5-1304-47bc-ac7e-c24ed4780150" /dev/nvme0n1p1: UUID="469f8c77-5201-48d5-aec1-4b8352e8fe6e" UUID_SUB="74173125-b356-45aa-bf14-de414c77bfea" BLOCK_SIZE="4096" TYPE="btrfs" /dev/sdd1: UUID="84d48648-5ede-487a-96c2-4eb71341faff" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="47a8e4d2-fc25-45d4-a99a-cd8fbfcdc0fb" /dev/md2p1: UUID="eafcd747-8b58-4eae-8eba-e5694b12b2be" BLOCK_SIZE="512" TYPE="xfs" /dev/sdm1: UUID="68515c7e-62f0-4605-a46e-1e0c072162a0" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="83195a03-0bf0-4c2c-a57d-37029ae710d8" /dev/md5p1: UUID="00972d14-8046-4172-87c4-9b9db1c0d2bc" BLOCK_SIZE="512" TYPE="xfs" /dev/sdk1: UUID="76101465-dd25-45b8-8335-3470f22c2d9d" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="ffb3d289-34c3-2394-e00e-2fbd77de907c" /dev/md8p1: UUID="84f20336-6d87-446d-8ff7-5571b652be76" BLOCK_SIZE="512" TYPE="xfs" /dev/md1p1: UUID="908ca018-2be7-44a7-8597-85dd60865998" BLOCK_SIZE="512" TYPE="xfs" /dev/sdg1: UUID="a492518f-0ef3-4da9-a8c5-595aabe5ed90" UUID_SUB="428bfeb1-4b3e-47b3-b4e1-d83d8b99cf07" BLOCK_SIZE="4096" TYPE="btrfs" PARTUUID="ab3cf074-6261-43ed-8e01-5eac5c901825" /dev/md4p1: UUID="76101465-dd25-45b8-8335-3470f22c2d9d" BLOCK_SIZE="512" TYPE="xfs" /dev/loop0: TYPE="squashfs" /dev/sde1: UUID="eafcd747-8b58-4eae-8eba-e5694b12b2be" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="02d2ca2f-c36a-48d4-8e2d-da463129e2c2" /dev/md7p1: UUID="84d48648-5ede-487a-96c2-4eb71341faff" BLOCK_SIZE="512" TYPE="xfs" /dev/sdc1: UUID="469f8c77-5201-48d5-aec1-4b8352e8fe6e" UUID_SUB="1d216435-f1f3-4f46-9ec1-0648ccd576ab" BLOCK_SIZE="4096" TYPE="btrfs" /dev/sdl1: UUID="00972d14-8046-4172-87c4-9b9db1c0d2bc" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="0acdb824-044f-4174-8450-b0e166739e1d" /dev/sdj1: UUID="d4596a58-09f3-4dee-8cba-424ef33d48fa" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="9bcb74d7-7855-4ee6-b1ce-317cbfce987b" /dev/md3p1: UUID="d4596a58-09f3-4dee-8cba-424ef33d48fa" BLOCK_SIZE="512" TYPE="xfs" /dev/sdh1: UUID="84f20336-6d87-446d-8ff7-5571b652be76" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="21c002d3-d4f7-4c9e-beaa-6522713e95fd" /dev/md6p1: UUID="68515c7e-62f0-4605-a46e-1e0c072162a0" BLOCK_SIZE="512" TYPE="xfs" /dev/sdi1: PARTUUID="6eaffde1-fb72-4725-bee2-01f7b5285a96"

Hi Folks, Over the past few days I've been troubleshooting some BRTFS errors that have appeared in my Syslogs. Previously, a simple restart would restore my ability to mount the filesystem and run my dockers, but as of this afternoon, the filesystem is unmountable. I've attached a diagnostic from this afternoon (21st) and yesterday (20th). From what I see searching online, it seems either there is an issue with a memory module which lead to some corruption issues or there is an issue with one of my drives. My cache setup is a RAID 1 BTRFS of an NVME drive and a SSD (in my logs, /dev/sdc1 and /dev/nvme0n1). Notably, on September 20th (Diagnostic attached), I see the following: Sep 20 00:45:28 Unraid kernel: BTRFS critical (device sdc1): corrupt leaf: root=5 block=1161025175552 slot=143 ino=20996272 file_offset=65536, invalid type for file extent, have 164 expect range [0, 2] Sep 20 00:45:28 Unraid kernel: BTRFS info (device sdc1): leaf 1161025175552 gen 1029363 total ptrs 197 free space 749 owner 5 Today, I see the following: Sep 21 04:03:17 Unraid kernel: ------------[ cut here ]------------ Sep 21 04:03:17 Unraid kernel: WARNING: CPU: 16 PID: 34855 at fs/btrfs/extent-tree.c:3061 __btrfs_free_extent+0x466/0xc02 Sep 21 04:03:17 Unraid kernel: Modules linked in: veth wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter nvidia_uvm(PO) xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) ip6table_filter ip6_tables iptable_filter ip_tables x_tables macvtap macvlan tap bridge 8021q garp mrp stp llc ixgbe xfrm_algo mdio e1000e nvidia_drm(PO) nvidia_modeset(PO) intel_rapl_msr intel_rapl_common iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm nvidia(PO) crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 video aesni_intel Sep 21 04:03:17 Unraid kernel: drm_kms_helper crypto_simd ipmi_ssif cryptd drm rapl isci i2c_i801 intel_cstate nvme backlight mei_me i2c_smbus ahci cp210x apex(O) libsas syscopyarea input_leds sysfillrect sysimgblt i2c_core joydev led_class usbserial mei intel_uncore nvme_core scsi_transport_sas libahci gasket(O) fb_sys_fops acpi_ipmi wmi ipmi_si button unix [last unloaded: xfrm_algo] Sep 21 04:03:17 Unraid kernel: CPU: 16 PID: 34855 Comm: kworker/u82:5 Tainted: P W O 6.1.49-Unraid #1 Sep 21 04:03:17 Unraid kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./EP2C602-4L/D16, BIOS P1.90 04/11/2018 Sep 21 04:03:17 Unraid kernel: Workqueue: events_unbound btrfs_preempt_reclaim_metadata_space Sep 21 04:03:17 Unraid kernel: RIP: 0010:__btrfs_free_extent+0x466/0xc02 Sep 21 04:03:17 Unraid kernel: Code: 41 b4 01 45 89 e0 89 d9 ba f0 0b 00 00 41 83 e0 01 e9 93 00 00 00 31 db 45 8b 75 40 e9 dc 00 00 00 83 f8 fe 0f 85 93 00 00 00 <0f> 0b 49 8b 7d 00 e8 48 45 00 00 48 c7 c6 c4 8c 10 82 41 56 4c 8b Sep 21 04:03:17 Unraid kernel: RSP: 0018:ffffc90038d57bd8 EFLAGS: 00010246 Sep 21 04:03:17 Unraid kernel: RAX: 00000000fffffffe RBX: 00000000fffffffe RCX: 0000000000000000 Sep 21 04:03:17 Unraid kernel: RDX: 0000000000000000 RSI: ffff88943ed4434b RDI: ffffc90038d57b80 Sep 21 04:03:17 Unraid kernel: RBP: ffff88a0c7a97000 R08: 0000000000001000 R09: 0000160000000000 Sep 21 04:03:17 Unraid kernel: R10: ffff888000000000 R11: ffff88a50da34c70 R12: 0000000000000000 Sep 21 04:03:17 Unraid kernel: R13: ffff88a0afe7d620 R14: 000000012cfaf000 R15: 0000000000000361 Sep 21 04:03:17 Unraid kernel: FS: 0000000000000000(0000) GS:ffff88bfff980000(0000) knlGS:0000000000000000 Sep 21 04:03:17 Unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Sep 21 04:03:17 Unraid kernel: CR2: 000000c000422000 CR3: 000000000220a006 CR4: 00000000001726e0 Sep 21 04:03:17 Unraid kernel: Call Trace: Sep 21 04:03:17 Unraid kernel: <TASK> Sep 21 04:03:17 Unraid kernel: ? __warn+0xab/0x122 Sep 21 04:03:17 Unraid kernel: ? report_bug+0x109/0x17e Sep 21 04:03:17 Unraid kernel: ? __btrfs_free_extent+0x466/0xc02 Sep 21 04:03:17 Unraid kernel: ? handle_bug+0x41/0x6f Sep 21 04:03:17 Unraid kernel: ? exc_invalid_op+0x13/0x60 Sep 21 04:03:17 Unraid kernel: ? asm_exc_invalid_op+0x16/0x20 Sep 21 04:03:17 Unraid kernel: ? __btrfs_free_extent+0x466/0xc02 Sep 21 04:03:17 Unraid kernel: ? sysfs_slab_add+0x14d/0x1c8 Sep 21 04:03:17 Unraid kernel: __btrfs_run_delayed_refs+0x698/0xbe2 Sep 21 04:03:17 Unraid kernel: btrfs_run_delayed_refs+0x65/0x146 Sep 21 04:03:17 Unraid kernel: flush_space+0x37e/0x491 Sep 21 04:03:17 Unraid kernel: ? newidle_balance+0x289/0x30a Sep 21 04:03:17 Unraid kernel: ? _raw_spin_lock+0x13/0x1c Sep 21 04:03:17 Unraid kernel: ? _raw_spin_unlock+0x14/0x29 Sep 21 04:03:17 Unraid kernel: ? btrfs_bg_type_to_factor+0xa/0x1c Sep 21 04:03:17 Unraid kernel: ? calc_available_free_space.isra.0+0x33/0x59 Sep 21 04:03:17 Unraid kernel: btrfs_preempt_reclaim_metadata_space+0xe8/0x172 Sep 21 04:03:17 Unraid kernel: process_one_work+0x1ab/0x295 Sep 21 04:03:17 Unraid kernel: worker_thread+0x18b/0x244 Sep 21 04:03:17 Unraid kernel: ? rescuer_thread+0x281/0x281 Sep 21 04:03:17 Unraid kernel: kthread+0xe7/0xef Sep 21 04:03:17 Unraid kernel: ? kthread_complete_and_exit+0x1b/0x1b Sep 21 04:03:17 Unraid kernel: ret_from_fork+0x22/0x30 Sep 21 04:03:17 Unraid kernel: </TASK> Sep 21 04:03:17 Unraid kernel: ---[ end trace 0000000000000000 ]--- Sep 21 04:03:17 Unraid kernel: BTRFS info (device sdc1): leaf 1958068224 gen 1031631 total ptrs 179 free space 2321 owner 2 I've run extended SMART tests on /dev/sdc1 and /dev/nvme0n1, and both pass. Any guidance on what I can look at next? The cache is currently unmountable -- do I have to format again and restore backups (appdata for Dockers, VMs), or is there a way to re-mount just a single (ex: the nvme drive) file system? If I do have to rebuild, I'm concerned I haven't found the root cause and will be back in this situation shortly. TIA for any thoughts unraid-diagnostics-20230921-1640.zip unraid-diagnostics-20230920-0837.zip Samsung_SSD_970_EVO_500GB_S466NX0K904535J-20230921-1905.txt Crucial_CT512MX100SSD1_14240D1D6D98-20230921-1907.txt

This is definitely still an issue -- currently reading through past threads after upgrading to 6.10.2 and being flooded with the following messages and an unusable ConBee II: May 29 10:32:30 Unraid kernel: usb 2-1.4: usbfs: process 32050 (qemu-system-x86) did not claim interface 1 before use Solution as posted in this thread (and others) was to obtain the proper device path for the ConBee II and manually add this in to the virtual machine XML as per https://community.home-assistant.io/t/unraid-vm-conbee-ii-stick-dev-path-change-configuration-problem/411424.