thatnovaguy Posted September 12, 2023 Share Posted September 12, 2023 Not sure if the update caused it but last night I updated to 6.12.4 and the this morning my NVME cache drive was in read-only mode. Stop array hung trying to unmount the nvme drive and in the end I had to use 'umount /var/lib/docker' to get it to fully stop. When I restarted the array the drive was shown as unmountable file system. Disk log shows: Sep 12 09:53:29 HTPC kernel: BTRFS: error (device nvme0n1p1: state A) in __btrfs_free_extent:3070: errno=-22 unknown Sep 12 09:53:29 HTPC kernel: BTRFS error (device nvme0n1p1: state EA): failed to run delayed ref for logical 4078200520704 num_bytes 12288 type 178 action 2 ref_mod 1: -22 Sep 12 09:53:29 HTPC kernel: BTRFS: error (device nvme0n1p1: state EA) in btrfs_run_delayed_refs:2144: errno=-22 unknown Sep 12 09:53:29 HTPC kernel: BTRFS: error (device nvme0n1p1: state EA) in btrfs_replay_log:2414: errno=-22 unknown (Failed to recover log tree) Sep 12 09:53:29 HTPC root: mount: /mnt/nvme: wrong fs type, bad option, bad superblock on /dev/nvme0n1p1, missing codepage or helper program, or other error. Sep 12 09:53:29 HTPC kernel: BTRFS error (device nvme0n1p1: state EA): open_ctree failed BTRFS check gives me: root@HTPC:~# btrfs check /dev/nvme0n1 Opening filesystem to check... No valid Btrfs found on /dev/nvme0n1 ERROR: cannot open file system I'm planning to replace these drives with standard sata SSDs but would like to recover my appdata and docker image if possible. Diagnostics attached. htpc-diagnostics-20230912-1000.zip Quote Link to comment
JorgeB Posted September 12, 2023 Share Posted September 12, 2023 If the log tree is the only problem this should help: btrfs rescue zero-log /dev/nvme0n1p1 Then restart array. Quote Link to comment
thatnovaguy Posted September 12, 2023 Author Share Posted September 12, 2023 Ran that and restarted the array. Disk is readable but almost immediately went into read-only. Sep 12 17:48:30 HTPC emhttpd: read SMART /dev/nvme0n1 Sep 12 17:48:34 HTPC emhttpd: INTEL_SSDPE2KE016T8_BTLN84730E6A1P6AGN (nvme0n1) 512 3125627568 Sep 12 17:48:34 HTPC emhttpd: import 31 cache device: (nvme0n1) INTEL_SSDPE2KE016T8_BTLN84730E6A1P6AGN Sep 12 17:48:34 HTPC emhttpd: read SMART /dev/nvme0n1 Sep 12 17:49:00 HTPC emhttpd: devid 2 size 1.46TiB used 888.09GiB path /dev/nvme0n1p1 Sep 12 17:49:00 HTPC kernel: BTRFS info (device nvme0n1p1): using crc32c (crc32c-intel) checksum algorithm Sep 12 17:49:00 HTPC kernel: BTRFS info (device nvme0n1p1): allowing degraded mounts Sep 12 17:49:00 HTPC kernel: BTRFS info (device nvme0n1p1): using free space tree Sep 12 17:49:00 HTPC kernel: BTRFS warning (device nvme0n1p1): devid 1 uuid 4f00e3be-cb73-4615-aa54-2311591e7bfa is missing Sep 12 17:49:00 HTPC kernel: BTRFS info (device nvme0n1p1): bdev (efault) errs: wr 0, rd 0, flush 0, corrupt 8, gen 0 Sep 12 17:49:00 HTPC kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 59, gen 0 Sep 12 17:49:00 HTPC kernel: BTRFS info (device nvme0n1p1): enabling ssd optimizations Sep 12 17:49:00 HTPC kernel: BTRFS info (device nvme0n1p1: state M): allowing degraded mounts Sep 12 17:49:00 HTPC kernel: BTRFS info (device nvme0n1p1: state M): turning on async discard Sep 12 17:49:00 HTPC kernel: BTRFS info (device nvme0n1p1): balance: start -f -dconvert=single -mconvert=dup -sconvert=dup Sep 12 17:49:00 HTPC kernel: BTRFS info (device nvme0n1p1): relocating block group 6094656110592 flags data Sep 12 17:49:02 HTPC kernel: BTRFS info (device nvme0n1p1): found 8 extents, stage: move data extents Sep 12 17:49:02 HTPC kernel: BTRFS info (device nvme0n1p1): found 8 extents, stage: update data pointers Sep 12 17:49:02 HTPC kernel: BTRFS info (device nvme0n1p1): relocating block group 6093582368768 flags data Sep 12 17:49:03 HTPC kernel: BTRFS info (device nvme0n1p1): found 8 extents, stage: move data extents Sep 12 17:49:03 HTPC kernel: BTRFS info (device nvme0n1p1): found 8 extents, stage: update data pointers Sep 12 17:49:03 HTPC kernel: BTRFS info (device nvme0n1p1): relocating block group 6092508626944 flags data Sep 12 17:49:04 HTPC kernel: BTRFS info (device nvme0n1p1): found 8 extents, stage: move data extents Sep 12 17:49:04 HTPC kernel: BTRFS info (device nvme0n1p1): found 8 extents, stage: update data pointers Sep 12 17:49:04 HTPC kernel: BTRFS info (device nvme0n1p1): relocating block group 6091434885120 flags data Sep 12 17:49:06 HTPC kernel: BTRFS info (device nvme0n1p1): found 6 extents, stage: move data extents Sep 12 17:49:06 HTPC kernel: BTRFS info (device nvme0n1p1): found 6 extents, stage: update data pointers Sep 12 17:49:06 HTPC kernel: BTRFS info (device nvme0n1p1): relocating block group 6090361143296 flags data Sep 12 17:49:06 HTPC kernel: BTRFS info (device nvme0n1p1): found 4 extents, stage: move data extents Sep 12 17:49:06 HTPC kernel: BTRFS info (device nvme0n1p1): found 4 extents, stage: update data pointers Sep 12 17:49:06 HTPC kernel: BTRFS info (device nvme0n1p1): relocating block group 6089287401472 flags data Sep 12 17:49:06 HTPC kernel: BTRFS info (device nvme0n1p1): found 2 extents, stage: move data extents Sep 12 17:49:06 HTPC kernel: BTRFS info (device nvme0n1p1): found 2 extents, stage: update data pointers Sep 12 17:49:06 HTPC kernel: BTRFS info (device nvme0n1p1): relocating block group 6088213659648 flags data Sep 12 17:49:07 HTPC kernel: BTRFS info (device nvme0n1p1): found 6 extents, stage: move data extents Sep 12 17:49:07 HTPC kernel: BTRFS info (device nvme0n1p1): found 6 extents, stage: update data pointers Sep 12 17:49:07 HTPC kernel: BTRFS info (device nvme0n1p1): relocating block group 6087139917824 flags data Sep 12 17:49:08 HTPC kernel: BTRFS info (device nvme0n1p1): found 8 extents, stage: move data extents Sep 12 17:49:08 HTPC kernel: BTRFS info (device nvme0n1p1): found 8 extents, stage: update data pointers Sep 12 17:49:08 HTPC kernel: BTRFS info (device nvme0n1p1): relocating block group 6086066176000 flags data Sep 12 17:49:10 HTPC kernel: BTRFS info (device nvme0n1p1): found 9 extents, stage: move data extents Sep 12 17:49:10 HTPC kernel: BTRFS info (device nvme0n1p1): found 9 extents, stage: update data pointers Sep 12 17:49:10 HTPC kernel: BTRFS info (device nvme0n1p1): relocating block group 6084992434176 flags data Sep 12 17:49:11 HTPC kernel: BTRFS info (device nvme0n1p1): found 5 extents, stage: move data extents Sep 12 17:49:11 HTPC kernel: BTRFS info (device nvme0n1p1): found 5 extents, stage: update data pointers Sep 12 17:49:11 HTPC kernel: BTRFS info (device nvme0n1p1): relocating block group 6083918692352 flags data Sep 12 17:49:12 HTPC kernel: BTRFS info (device nvme0n1p1): found 28 extents, stage: move data extents Sep 12 17:49:12 HTPC kernel: BTRFS info (device nvme0n1p1): found 28 extents, stage: update data pointers Sep 12 17:49:12 HTPC kernel: BTRFS info (device nvme0n1p1): found 1 extents, stage: update data pointers Sep 12 17:49:12 HTPC kernel: BTRFS info (device nvme0n1p1): found 1 extents, stage: update data pointers Sep 12 17:49:12 HTPC kernel: BTRFS info (device nvme0n1p1): found 1 extents, stage: update data pointers Sep 12 17:49:12 HTPC kernel: BTRFS info (device nvme0n1p1): found 1 extents, stage: update data pointers Sep 12 17:49:12 HTPC kernel: BTRFS info (device nvme0n1p1): relocating block group 6082844950528 flags data Sep 12 17:49:14 HTPC kernel: BTRFS info (device nvme0n1p1): found 8 extents, stage: move data extents Sep 12 17:49:14 HTPC kernel: BTRFS info (device nvme0n1p1): found 8 extents, stage: update data pointers Sep 12 17:49:14 HTPC kernel: BTRFS info (device nvme0n1p1): relocating block group 6081771208704 flags data Sep 12 17:49:15 HTPC kernel: BTRFS info (device nvme0n1p1): found 7 extents, stage: move data extents Sep 12 17:49:15 HTPC kernel: BTRFS info (device nvme0n1p1): found 7 extents, stage: update data pointers Sep 12 17:49:15 HTPC kernel: BTRFS info (device nvme0n1p1): relocating block group 6080697466880 flags data Sep 12 17:49:16 HTPC kernel: BTRFS info (device nvme0n1p1): found 7 extents, stage: move data extents Sep 12 17:49:16 HTPC kernel: BTRFS info (device nvme0n1p1): found 7 extents, stage: update data pointers Sep 12 17:49:17 HTPC kernel: BTRFS info (device nvme0n1p1): relocating block group 6079623725056 flags data Sep 12 17:49:18 HTPC kernel: BTRFS info (device nvme0n1p1): found 8 extents, stage: move data extents Sep 12 17:49:18 HTPC kernel: BTRFS info (device nvme0n1p1): found 8 extents, stage: update data pointers Sep 12 17:49:18 HTPC kernel: BTRFS info (device nvme0n1p1): relocating block group 6078549983232 flags data Sep 12 17:49:19 HTPC kernel: BTRFS critical (device nvme0n1p1): unable to find logical 13893481886742069248 length 4096 Sep 12 17:49:19 HTPC kernel: BTRFS critical (device nvme0n1p1): unable to find logical 13893481886742069248 length 16384 Sep 12 17:49:19 HTPC kernel: BTRFS error (device nvme0n1p1): failed to run delayed ref for logical 4078200520704 num_bytes 12288 type 184 action 1 ref_mod 1: -22 Sep 12 17:49:19 HTPC kernel: BTRFS: error (device nvme0n1p1: state A) in btrfs_run_delayed_refs:2144: errno=-22 unknown Sep 12 17:49:19 HTPC kernel: BTRFS info (device nvme0n1p1: state EA): forced readonly Sep 12 17:49:19 HTPC kernel: BTRFS warning (device nvme0n1p1: state EA): Skipping commit of aborted transaction. Sep 12 17:49:19 HTPC kernel: BTRFS: error (device nvme0n1p1: state EA) in cleanup_transaction:1958: errno=-22 unknown Sep 12 17:49:20 HTPC kernel: BTRFS info (device nvme0n1p1: state EA): balance: ended with status: -5 Quote Link to comment
JorgeB Posted September 13, 2023 Share Posted September 13, 2023 10 hours ago, thatnovaguy said: Sep 12 17:49:00 HTPC kernel: BTRFS info (device nvme0n1p1): bdev (efault) errs: wr 0, rd 0, flush 0, corrupt 8, gen 0 Sep 12 17:49:00 HTPC kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 59, gen 0 There's a missing device and corruption on the other one, do you know what happened to the other device? Quote Link to comment
thatnovaguy Posted September 13, 2023 Author Share Posted September 13, 2023 10 hours ago, JorgeB said: There's a missing device and corruption on the other one, do you know what happened to the other device? Had to pull the missing device to RMA it. I truly regret getting these drives. They've been nothing but trouble. Quote Link to comment
Solution JorgeB Posted September 13, 2023 Solution Share Posted September 13, 2023 Since it's still mounted read-only suggest to backup what you can, then re-format. Quote Link to comment
thatnovaguy Posted September 14, 2023 Author Share Posted September 14, 2023 19 hours ago, JorgeB said: Since it's still mounted read-only suggest to backup what you can, then re-format. I did as you said. I plan to replace the drive once the replacements come in but it's working for now. However, after formatting and copying everything back I noticed it ran trim. Sep 14 00:00:13 HTPC root: /mnt/nvme: 735.9 GiB (790133469184 bytes) trimmed on /dev/nvme0n1p1 Is it normal to trim that much? Quote Link to comment
wayner Posted September 14, 2023 Share Posted September 14, 2023 A bunch of us have had issues with cache drives using BTRFS after upgrading to 6.12.4. I have changed to XFS on my cache drive, but I will likely go to ZFS so that I can have mirrored cache drives. I reformatted and corruption on my BTRFS cache drive re-occurred. Quote Link to comment
JorgeB Posted September 14, 2023 Share Posted September 14, 2023 38 minutes ago, thatnovaguy said: Is it normal to trim that much? It should TRIM the current free space. Quote Link to comment
thatnovaguy Posted September 14, 2023 Author Share Posted September 14, 2023 42 minutes ago, JorgeB said: It should TRIM the current free space. Ok. Sorry I wasn't exactly sure how trim worked. 1 hour ago, wayner said: A bunch of us have had issues with cache drives using BTRFS after upgrading to 6.12.4. I have changed to XFS on my cache drive, but I will likely go to ZFS so that I can have mirrored cache drives. I reformatted and corruption on my BTRFS cache drive re-occurred. I'll keep an eye on it. I tried using zfs shortly after it was officially released but ran into corruption issues. I believe, however, it was the drives that caused my problems. Quote Link to comment
wayner Posted September 14, 2023 Share Posted September 14, 2023 Here is the thread - I meant to put this in my earlier post Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.