slayernine

The server seems to be running good now. USB drive errors are gone. Docker is running great. Web interface loads faster than ever before. Scrubs are still clean. UUID: 3d1c23bb-8e22-4ab9-9595-c5e4fd71041b Scrub started: Fri Jan 24 11:36:31 2025 Status: finished Duration: 0:06:02 Total to scrub: 641.57GiB Rate: 1.77GiB/s Error summary: no errors found The only lingering issue is the USB 2.0 ports on my motherboard might be having some issues. Thanks for the help and advice.

Ok, I'm starting to get very suspect of my motherboard/CPU as it is Intel 12th gen. USB 2.0 ports on the motherboard seem to be dying or not working well. Unraid would not successfully boot off the USB 2.0 port. I even started making a new USB drive because I thought the drive was dying. Plugged the unraid USB drive into a USB 3.0 port and it booted faster than ever and the interface even feels snappier than before.

And now I randomly got this message about the flash drive being corrupted... Just when I think everything is working again. " Your flash drive is corrupted or offline. Post your diagnostics in the forum for help."

Yes, thank you for the link. I've done this before, so I'm familiar with it. Seems good so far, I've ran another scrub and it has found nothing. Plex and other docker apps seem to be running normally. My PiHole server is failing to start up with a custom IP so I'm trying to figure out how I got that to work before.

Docker disabled, and image deleted. Restarted the array and scrubbed. No errors. UUID: 3d1c23bb-8e22-4ab9-9595-c5e4fd71041b Scrub started: Wed Jan 22 17:44:32 2025 Status: finished Duration: 0:05:19 Total to scrub: 638.64GiB Rate: 2.00GiB/s Error summary: no errors found. I'm going to start on reinstalling docker apps.

@JorgeB After you previously asked me to run Memtest I ran it for 12 hours and nothing detected. It is possible there were memory errors previously, I updated the BIOS which switched my memory speed to "auto" and with those settings it is fine. It is likely I was running the memory on the XMP profile before I updated the BIOS. Are these errors new errors or perhaps they happened before on the old memory settings???

Do I need to delete the docker image and re-download the containers? The Minecraft server my kids use PaperMC is no longer available so I'm hesitant to delete as I'm not sure I can recover their Minecraft worlds.

@JorgeBI've attached diagnostics as you requested. I enabled docker again but it immediately threw the cache into read only mode: Jan 22 11:32:12 hodor kernel: BTRFS: error (device nvme0n1p1: state A) in __btrfs_free_extent:3072: errno=-2 No such entry Jan 22 11:32:12 hodor kernel: BTRFS info (device nvme0n1p1: state EA): forced readonly Jan 22 11:32:12 hodor kernel: BTRFS error (device nvme0n1p1: state EA): failed to run delayed ref for logical 564075810816 num_bytes 4096 type 178 action 2 ref_mod 1: -2 Jan 22 11:32:12 hodor kernel: BTRFS: error (device nvme0n1p1: state EA) in btrfs_run_delayed_refs:2149: errno=-2 No such entry Jan 22 11:32:12 hodor kernel: loop: Write error at byte offset 1562750976, length 4096. Jan 22 11:32:12 hodor kernel: I/O error, dev loop3, sector 3052248 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 2 Jan 22 11:32:12 hodor kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 1, rd 0, flush 0, corrupt 11, gen 0 Jan 22 11:32:13 hodor kernel: loop: Write error at byte offset 1446375424, length 4096. Jan 22 11:32:13 hodor kernel: I/O error, dev loop3, sector 2824952 op 0x1:(WRITE) flags 0x100000 phys_seg 2 prio class 2 Jan 22 11:32:13 hodor kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 2, rd 0, flush 0, corrupt 11, gen 0 Jan 22 11:32:13 hodor kernel: loop: Write error at byte offset 1822388224, length 4096. Jan 22 11:32:13 hodor kernel: loop: Write error at byte offset 1067728896, length 4096. Jan 22 11:32:13 hodor kernel: loop: Write error at byte offset 1848250368, length 4096. Jan 22 11:32:13 hodor kernel: loop: Write error at byte offset 1529765888, length 4096. Jan 22 11:32:13 hodor kernel: I/O error, dev loop3, sector 3559352 op 0x1:(WRITE) flags 0x100000 phys_seg 19 prio class 2 Jan 22 11:32:13 hodor kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 3, rd 0, flush 0, corrupt 11, gen 0 Jan 22 11:32:13 hodor kernel: I/O error, dev loop3, sector 2085408 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 2 Jan 22 11:32:13 hodor kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 4, rd 0, flush 0, corrupt 11, gen 0 Jan 22 11:32:13 hodor kernel: I/O error, dev loop3, sector 3609864 op 0x1:(WRITE) flags 0x100000 phys_seg 21 prio class 2 Jan 22 11:32:13 hodor kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 5, rd 0, flush 0, corrupt 11, gen 0 Jan 22 11:32:13 hodor kernel: I/O error, dev loop3, sector 2987824 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 2 Jan 22 11:32:13 hodor kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 6, rd 0, flush 0, corrupt 11, gen 0 Jan 22 11:32:14 hodor kernel: loop: Write error at byte offset 1562750976, length 4096. Jan 22 11:32:14 hodor kernel: I/O error, dev loop3, sector 3052248 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2 Jan 22 11:32:14 hodor kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 7, rd 0, flush 0, corrupt 11, gen 0 Jan 22 11:32:14 hodor kernel: veth8f2c58d: renamed from eth0 Jan 22 11:32:14 hodor kernel: loop: Write error at byte offset 1406246912, length 4096. Jan 22 11:32:14 hodor kernel: loop: Write error at byte offset 1405751296, length 4096. Jan 22 11:32:14 hodor kernel: loop: Write error at byte offset 1529765888, length 4096. Jan 22 11:32:14 hodor kernel: I/O error, dev loop3, sector 2746576 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 2 Jan 22 11:32:14 hodor kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 8, rd 0, flush 0, corrupt 11, gen 0 Jan 22 11:32:14 hodor kernel: I/O error, dev loop3, sector 2745608 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2 Jan 22 11:32:14 hodor kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 9, rd 0, flush 0, corrupt 11, gen 0 Jan 22 11:32:14 hodor kernel: I/O error, dev loop3, sector 2987824 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2 Jan 22 11:32:14 hodor kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 10, rd 0, flush 0, corrupt 11, gen 0 Jan 22 11:32:14 hodor kernel: BTRFS: error (device loop3) in btrfs_commit_transaction:2494: errno=-5 IO failure (Error while writing out transaction) Jan 22 11:32:14 hodor kernel: BTRFS info (device loop3: state E): forced readonly Jan 22 11:32:14 hodor kernel: BTRFS warning (device loop3: state E): Skipping commit of aborted transaction. Jan 22 11:32:14 hodor kernel: BTRFS: error (device loop3: state EA) in cleanup_transaction:1992: errno=-5 IO failure Jan 22 11:32:14 hodor kernel: docker0: port 3(veth2401cad) entered disabled state hodor-diagnostics-20250122-1149.zip

I've disabled docker for now. Ran this commmand: btrfs rescue zero-log /dev/nvme0n1p1 and restarted the array. Trying to run mover to get temp files off but got a bunch of errors: Jan 22 11:00:02 hodor shfs: copy_file: /mnt/cache/REDACTED.partial (5) Input/output error Jan 22 11:00:02 hodor kernel: btrfs_print_data_csum_error: 5 callbacks suppressed Jan 22 11:00:02 hodor kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 231457946 off 804892672 csum 0xcc9072f5 expected csum 0xbf33199b mirror 1 Jan 22 11:00:02 hodor kernel: btrfs_dev_stat_inc_and_print: 5 callbacks suppressed Jan 22 11:00:02 hodor kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 37032, gen 0 Jan 22 11:00:02 hodor kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 231457946 off 804892672 csum 0xcc9072f5 expected csum 0xbf33199b mirror 1 Jan 22 11:00:02 hodor kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 37033, gen 0 Jan 22 11:00:02 hodor kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 231457946 off 804892672 csum 0xcc9072f5 expected csum 0xbf33199b mirror 1 Jan 22 11:00:02 hodor kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 37034, gen 0 Jan 22 11:00:02 hodor kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 231457946 off 804892672 csum 0xcc9072f5 expected csum 0xbf33199b mirror 1 Jan 22 11:00:02 hodor kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 37035, gen 0 Jan 22 11:00:02 hodor kernel: br0: received packet on bond0 with own address as source address (addr:50:eb:f6:56:32:ac, vlan:0) Jan 22 11:00:03 hodor kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 231586144 off 2849775616 csum 0x0bef57a6 expected csum 0x2b988132 mirror 1 Jan 22 11:00:03 hodor kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 73365, gen 0 Jan 22 11:00:03 hodor kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 231586144 off 2849775616 csum 0x0bef57a6 expected csum 0x2b988132 mirror 1 Jan 22 11:00:03 hodor kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 73366, gen 0 Jan 22 11:00:03 hodor shfs: copy_file: /mnt/cache/REDACTED.partial (5) Input/output error Jan 22 11:00:03 hodor kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 231586144 off 2849775616 csum 0x0bef57a6 expected csum 0x2b988132 mirror 1 Jan 22 11:00:03 hodor kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 73367, gen 0 Jan 22 11:00:03 hodor kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 231586144 off 2849775616 csum 0x0bef57a6 expected csum 0x2b988132 mirror 1 Jan 22 11:00:03 hodor kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 73368, gen 0 Jan 22 11:00:03 hodor kernel: br0: received packet on bond0 with own address as source address (addr:50:eb:f6:56:32:ac, vlan:0) Jan 22 11:00:03 hodor shfs: copy_file: /mnt/cache/REDACTED.partial (5) Input/output error Jan 22 11:00:03 hodor kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 231605081 off 210542592 csum 0x6a4df49f expected csum 0x19ee9ff1 mirror 1 Jan 22 11:00:03 hodor kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 73369, gen 0 Jan 22 11:00:03 hodor kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 231605081 off 210542592 csum 0x6a4df49f expected csum 0x19ee9ff1 mirror 1 Jan 22 11:00:03 hodor kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 73370, gen 0 Running a scrub now. Jan 22 11:05:43 hodor kernel: BTRFS warning (device nvme0n1p1): checksum error at logical 6847963668480 on dev /dev/nvme0n1p1, physical 224332947456, root 5, inode 231586144, offset 2849775616, length 4096, links 1 (path: REDACTED) Jan 22 11:05:43 hodor kernel: btrfs_dev_stat_inc_and_print: 2 callbacks suppressed Jan 22 11:05:43 hodor kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 73373, gen 0 Jan 22 11:05:43 hodor kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 6847963668480 on dev /dev/nvme0n1p1 Jan 22 11:05:44 hodor kernel: BTRFS warning (device nvme0n1p1): checksum error at logical 6850557026304 on dev /dev/nvme0n1p1, physical 224778821632, root 5, inode 231605081, offset 210542592, length 4096, links 1 (path: REDACTED) Jan 22 11:05:44 hodor kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 73374, gen 0 Jan 22 11:05:44 hodor kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 6850557026304 on dev /dev/nvme0n1p1 Jan 22 11:06:17 hodor root: Fix Common Problems: Other Warning: Unassigned Devices Plus not installed Jan 22 11:07:32 hodor kernel: BTRFS warning (device nvme0n1p1): checksum error at logical 6755676602368 on dev /dev/nvme1n1p1, physical 454214565888, root 5, inode 231457946, offset 804892672, length 4096, links 1 (path: REDACTED) Jan 22 11:07:32 hodor kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 37036, gen 0 Jan 22 11:07:32 hodor kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 6755676602368 on dev /dev/nvme1n1p1 Jan 22 11:07:58 hodor kernel: BTRFS info (device nvme0n1p1): scrub: finished on devid 1 with status: 0 Jan 22 11:08:01 hodor kernel: BTRFS info (device nvme0n1p1): scrub: finished on devid 2 with status: 0 Scrub complete: UUID: 3d1c23bb-8e22-4ab9-9595-c5e4fd71041b Scrub started: Wed Jan 22 11:01:43 2025 Status: finished Duration: 0:06:18 Total to scrub: 670.83GiB Rate: 1.77GiB/s Error summary: csum=3 Corrected: 0 Uncorrectable: 3 Unverified: 0 Deleted folder and files that contained the errors. Started another scrub to see if the errors go away. Jan 22 11:15:17 hodor kernel: BTRFS info (device nvme0n1p1): scrub: finished on devid 1 with status: 0 Jan 22 11:15:25 hodor kernel: BTRFS info (device nvme0n1p1): scrub: finished on devid 2 with status: 0 UUID: 3d1c23bb-8e22-4ab9-9595-c5e4fd71041b Scrub started: Wed Jan 22 11:09:56 2025 Status: finished Duration: 0:05:29 Total to scrub: 657.67GiB Rate: 2.00GiB/s Error summary: no errors found Looks like errors are gone. @JorgeB I very much appreciate your advice. What should I do next? Should I be trying to change my cache drive away from BTRFS to XFS or ZFS? I don't want to keep having these issues, it is causing me a lot of stress

I can't even run mover right now so I'm going to try stopping the array and starting again. I got a bunch of errors during that process: Jan 22 10:46:36 hodor kernel: loop: Write error at byte offset 92766208, length 4096. Jan 22 10:46:36 hodor kernel: loop: Write error at byte offset 39092224, length 4096. Jan 22 10:46:36 hodor kernel: loop: Write error at byte offset 92717056, length 4096. Jan 22 10:46:36 hodor kernel: loop: Write error at byte offset 39043072, length 4096. Jan 22 10:46:36 hodor kernel: I/O error, dev loop3, sector 181184 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 2 Jan 22 10:46:36 hodor kernel: I/O error, dev loop3, sector 76352 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 2 Jan 22 10:46:36 hodor kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0 Jan 22 10:46:36 hodor kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0 Jan 22 10:46:36 hodor kernel: I/O error, dev loop3, sector 181088 op 0x1:(WRITE) flags 0x1800 phys_seg 8 prio class 2 Jan 22 10:46:36 hodor kernel: I/O error, dev loop3, sector 76256 op 0x1:(WRITE) flags 0x1800 phys_seg 8 prio class 2 Jan 22 10:46:36 hodor kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0 Jan 22 10:46:36 hodor kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0 Jan 22 10:46:36 hodor kernel: BTRFS: error (device loop3) in btrfs_commit_transaction:2494: errno=-5 IO failure (Error while writing out transaction) Jan 22 10:46:36 hodor kernel: BTRFS info (device loop3: state E): forced readonly Jan 22 10:46:36 hodor kernel: BTRFS warning (device loop3: state E): Skipping commit of aborted transaction. Jan 22 10:46:36 hodor kernel: BTRFS: error (device loop3: state EA) in cleanup_transaction:1992: errno=-5 IO failure Jan 22 10:46:36 hodor kernel: BTRFS error (device loop3: state EA): commit super ret -5 .... Jan 22 10:48:37 hodor kernel: BTRFS: error (device nvme0n1p1: state A) in __btrfs_free_extent:3072: errno=-2 No such entry Jan 22 10:48:37 hodor kernel: BTRFS error (device nvme0n1p1: state EA): failed to run delayed ref for logical 564075810816 num_bytes 4096 type 178 action 2 ref_mod 1: -2 Jan 22 10:48:37 hodor kernel: BTRFS: error (device nvme0n1p1: state EA) in btrfs_run_delayed_refs:2149: errno=-2 No such entry Jan 22 10:48:37 hodor kernel: BTRFS: error (device nvme0n1p1: state EA) in btrfs_replay_log:2417: errno=-2 No such entry (Failed to recover log tree) Jan 22 10:48:37 hodor root: mount: /mnt/cache: mount(2) system call failed: No such file or directory. Jan 22 10:48:37 hodor root: dmesg(1) may have more information after failed mount system call. Jan 22 10:48:37 hodor emhttpd: shcmd (776): exit status: 32 Jan 22 10:48:37 hodor emhttpd: /mnt/cache mount error: Unsupported or no file system Jan 22 10:48:37 hodor emhttpd: shcmd (777): rmdir /mnt/cache Jan 22 10:48:37 hodor kernel: BTRFS error (device nvme0n1p1: state EA): open_ctree failed And... yeah cache disks are back to an unmountable state

I didn't catch the log in time, I'll need to re-run the scrub. But I did have a problem Today where docker entirely crashed. It looks like my cache is read only right now. It seems to be failing to even start a scrub. Jan 22 10:16:35 hodor kernel: I/O error, dev loop3, sector 26768 op 0x3:(DISCARD) flags 0x800 phys_seg 1 prio class 2 Jan 22 10:16:35 hodor kernel: I/O error, dev loop3, sector 43008 op 0x3:(DISCARD) flags 0x800 phys_seg 1 prio class 2 Jan 22 10:16:35 hodor kernel: I/O error, dev loop3, sector 75776 op 0x3:(DISCARD) flags 0x800 phys_seg 1 prio class 2 Jan 22 10:16:35 hodor kernel: BTRFS warning (device loop3): failed to trim 3 block group(s), last error -5 Jan 22 10:16:35 hodor kernel: I/O error, dev loop3, sector 2048 op 0x3:(DISCARD) flags 0x800 phys_seg 1 prio class 2 Jan 22 10:16:35 hodor kernel: BTRFS warning (device loop3): failed to trim 1 device(s), last error -5

The scrub found three uncorrectable errors: UUID: 3d1c23bb-8e22-4ab9-9595-c5e4fd71041b Scrub started: Sun Jan 19 13:44:04 2025 Status: finished Duration: 0:06:28 Total to scrub: 667.60GiB Rate: 1.72GiB/s Error summary: csum=3 Corrected: 0 Uncorrectable: 3 Unverified: 0

I ran memtest86 overnight and it passed. Running a scrub with the repair option right now.

I'm not quite certain if I was running XMP or not previously, it's been quite awhile since I set up this server.

I have not yet run a memory test but I did update the BIOS while troubleshooting the initial BTRFS issue. The memory is most likely running at a default speed and possibly not XMP profile after that update. So I will do a memtest when time allows Today. Currently trying to get a full backup of the Appdata, Plex is giving an error about files changing during backup. [18.01.2025 03:01:53][❌][plex] tar creation failed! Tar said: tar: /mnt/user/appdata/plex/Library/Application Support/Plex Media Server: file changed as we read it ASUSTeK COMPUTER INC. PRIME H670-PLUS D4 , Version Rev 1.xx American Megatrends Inc., Version 3601 BIOS dated: Mon 14 Oct 2024 12:00:00 AM MDT 12th Gen Intel® Core™ i5-12400 @ 3960 MHz Memory: 64 GiB DDR4 I did run a scrub Yesterday and this was the result: UUID: 3d1c23bb-8e22-4ab9-9595-c5e4fd71041b Scrub started: Fri Jan 17 20:44:37 2025 Status: finished Duration: 0:06:07 Total to scrub: 667.45GiB Rate: 934.47MiB/s Error summary: csum=2 Corrected: 0 Uncorrectable: 0 Unverified: 0 There were no scheduled scrub, so I scheduled it to run once per week. Should I run a scrub with the "Repair corrupted blocks" checked off?

Update, I've run this command "btrfs rescue zero-log /dev/nvme0n1p1" after reading this post: And I restarted the array and BOOM, the drives mounted!!!! But I did see this come up in the log saying there is major corruption? Jan 17 19:17:01 hodor kernel: BTRFS info (device nvme0n1p1): using crc32c (crc32c-intel) checksum algorithm Jan 17 19:17:01 hodor kernel: BTRFS info (device nvme0n1p1): using free space tree Jan 17 19:17:01 hodor kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 72648, gen 0 Jan 17 19:17:01 hodor kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 36670, gen 0 At least that is what I think is going on. I'm going to back up my appdata right now but after that, what should I do next to prevent this issue from coming up again??? Again, if anyone has any helpful info, I would be very appreciative. I've got thousands of hours sunk into this data and I really want to take care of it.

Uploaded diagnostics file. hodor-diagnostics-20250117-1900.zip

Here is the excerpt from the syslog with the errors it gets when trying to mount the cache: Jan 17 14:31:27 hodor kernel: BTRFS: error (device nvme0n1p1: state A) in __btrfs_free_extent:3072: errno=-2 No such entry Jan 17 14:31:27 hodor kernel: BTRFS error (device nvme0n1p1: state EA): failed to run delayed ref for logical 564075810816 num_bytes 4096 type 178 action 2 ref_mod 1: -2 Jan 17 14:31:27 hodor kernel: BTRFS: error (device nvme0n1p1: state EA) in btrfs_run_delayed_refs:2149: errno=-2 No such entry Jan 17 14:31:27 hodor kernel: BTRFS: error (device nvme0n1p1: state EA) in btrfs_replay_log:2417: errno=-2 No such entry (Failed to recover log tree) Jan 17 14:31:27 hodor kernel: BTRFS error (device nvme0n1p1: state EA): open_ctree failed Jan 17 14:31:27 hodor root: mount: /mnt/cache: mount(2) system call failed: No such file or directory. Jan 17 14:31:27 hodor root: dmesg(1) may have more information after failed mount system call. Jan 17 14:31:27 hodor emhttpd: shcmd (97): exit status: 32 Jan 17 14:31:27 hodor emhttpd: /mnt/cache mount error: Unsupported or no file system I've been following the instructions by JorgeB on fixing BTRFS but I'm not having any luck. I've never tried fixing BTRFS or manually mounting so I'm wondering if I'm just doing something wrong. root@hodor:/dev# mount -o rescue=all,ro /dev/nvme0 /temp mount: /temp: /dev/nvme0 is not a block device. dmesg(1) may have more information after failed mount system call. root@hodor:/dev# mount -o rescue=all,ro /dev/nvme0^Ctemp root@hodor:/dev# mount -o rescue=all,ro /dev/nvme1n1 /temp mount: /temp: wrong fs type, bad option, bad superblock on /dev/nvme1n1, missing codepage or helper program, or other error. dmesg(1) may have more information after failed mount system call. root@hodor:/dev# ^C root@hodor:/dev# mount -o rescue=all,ro /dev/nvme0n1 /temp mount: /temp: wrong fs type, bad option, bad superblock on /dev/nvme0n1, missing codepage or helper program, or other error. dmesg(1) may have more information after failed mount system call. root@hodor:/dev# mount -o rescue=all,ro /dev/nvme0 /temp mount: /temp: /dev/nvme0 is not a block device. dmesg(1) may have more information after failed mount system call. root@hodor:/dev# mount -o degraded,rescue=all,ro /dev/nvme0n1 /temp mount: /temp: wrong fs type, bad option, bad superblock on /dev/nvme0n1, missing codepage or helper program, or other error. dmesg(1) may have more information after failed mount system call. root@hodor:/dev# cd .. root@hodor:/# cd mnt root@hodor:/mnt# cd disk1 root@hodor:/mnt/disk1# ls downloads/ family\ movies/ isos/ movies/ music/ television/ root@hodor:/mnt/disk1# mkdir restore root@hodor:/mnt/disk1# ls downloads/ family\ movies/ isos/ movies/ music/ restore/ television/ root@hodor:/mnt/disk1# btrfs restore -v /dev/nvme0n1 /mnt/disk2/restore No valid Btrfs found on /dev/nvme0n1 Could not open root, trying backup super No valid Btrfs found on /dev/nvme0n1 Could not open root, trying backup super No valid Btrfs found on /dev/nvme0n1 Could not open root, trying backup super root@hodor:/mnt/disk1# btrfs restore -v /dev/nvme1n1 /mnt/disk2/restore No valid Btrfs found on /dev/nvme1n1 Could not open root, trying backup super No valid Btrfs found on /dev/nvme1n1 Could not open root, trying backup super No valid Btrfs found on /dev/nvme1n1 Could not open root, trying backup super root@hodor:/mnt/disk1# btrfs restore -vi /dev/nvme1n1 /mnt/disk2/restore No valid Btrfs found on /dev/nvme1n1 Could not open root, trying backup super No valid Btrfs found on /dev/nvme1n1 Could not open root, trying backup super No valid Btrfs found on /dev/nvme1n1 Could not open root, trying backup super root@hodor:/mnt/disk1# btrfs restore -v /dev/nvme0 /mnt/disk2/restore ERROR: not a regular file or block device: /dev/nvme0 Could not open root, trying backup super ERROR: not a regular file or block device: /dev/nvme0 Could not open root, trying backup super ERROR: not a regular file or block device: /dev/nvme0 Could not open root, trying backup super root@hodor:/mnt/disk1#

Last month I upgraded to 6.12.6 and my docker froze up around noon Today and it had something about a loop2 BTRFS error. So I soft rebooted the server and after the nvme cache drives both failed to mount. I checked the SMART data and they are both in good health. It seems like BTRFS has corrupted. Unfortunately, it looks like my appdata backup was not running correctly and I really need to try and recover my data. I've been reading old posts on here but not getting anywhere. Any advice or direction would be appreciated. I've shortened my post as it was saying "banned" when I tried to post the last two times.

Just rebuilt my UNRAID server on Ryzen and I'm loving it!