I have a BTRFS pool with two SDD, i was seeing a lot of errors on one of the disks, so I replaced it with a new disk. (shutdown, replaced disk, added new disk to pool). I let the pool rebuild and the next day i was getting those same errors, so i shut it back down and replaced the sata cable. Now i get the following errors when i start the array.
I have tried just about everything from all the existing posts on here about this kind of issue and have not made any progress at all. Does anyone have any suggestions of other things i could try?
Syslog Output
Mar 14 20:58:03 HYATTS emhttpd: import 32 pool device: (sdl) SAMSUNG_MZ7LM1T9HMJP-00005_S2TVNX0JB08456
Mar 14 20:58:03 HYATTS emhttpd: import 33 pool device: (sdk) Samsung_SSD_860_EVO_500GB_S3Z1NB0K814596V
Mar 14 20:58:03 HYATTS emhttpd: ssd_cache: recover volume
Mar 14 20:58:03 HYATTS emhttpd: /sbin/blkid /dev/sdl1 2>&1
Mar 14 20:58:03 HYATTS emhttpd: /dev/sdl1: UUID="ac0daf63-c73c-4130-90a7-65bb7beeb45c" UUID_SUB="c912ab9f-d56d-4890-83f7-f2621615d435" BLOCK_SIZE="4096" TYPE="btrfs"
Mar 14 20:58:03 HYATTS emhttpd: ssd_cache: btrfs recover pool
Mar 14 20:58:03 HYATTS emhttpd: ssd_cache: btrfs verify devices
Mar 14 20:58:03 HYATTS emhttpd: /sbin/btrfs filesystem show ac0daf63-c73c-4130-90a7-65bb7beeb45c 2>&1
Mar 14 20:58:03 HYATTS emhttpd: Label: none uuid: ac0daf63-c73c-4130-90a7-65bb7beeb45c
Mar 14 20:58:03 HYATTS emhttpd: Total devices 2 FS bytes used 210.78GiB
Mar 14 20:58:03 HYATTS emhttpd: devid 0 size 476.94GiB used 48.03GiB path /dev/sdl1
Mar 14 20:58:03 HYATTS emhttpd: devid 2 size 465.76GiB used 216.03GiB path /dev/sdk1
Mar 14 20:58:03 HYATTS emhttpd: shcmd (778): mount -t btrfs -o ro,degraded -U ac0daf63-c73c-4130-90a7-65bb7beeb45c /tmp/emhttp/mnt
Mar 14 20:58:03 HYATTS kernel: BTRFS info (device sdl1): first mount of filesystem ac0daf63-c73c-4130-90a7-65bb7beeb45c
Mar 14 20:58:03 HYATTS kernel: BTRFS info (device sdl1): using crc32c (crc32c-intel) checksum algorithm
Mar 14 20:58:03 HYATTS kernel: BTRFS info (device sdl1): allowing degraded mounts
Mar 14 20:58:03 HYATTS kernel: BTRFS info (device sdl1): using free space tree
Mar 14 20:58:03 HYATTS kernel: BTRFS warning (device sdl1): devid 1 uuid b7d94211-9749-40bd-adb2-f44797b585e6 is missing
Mar 14 20:58:03 HYATTS root: mount: /tmp/emhttp/mnt: mount system call failed: Structure needs cleaning.
Mar 14 20:58:03 HYATTS root: dmesg(1) may have more information after failed mount system call.
Mar 14 20:58:03 HYATTS kernel: BTRFS info (device sdl1): bdev /dev/sdl1 errs: wr 591963, rd 0, flush 208, corrupt 0, gen 0
Mar 14 20:58:03 HYATTS kernel: BTRFS info (device sdl1): bdev <missing disk> errs: wr 12196108, rd 2443, flush 8144, corrupt 0, gen 0
Mar 14 20:58:03 HYATTS kernel: BTRFS error (device sdl1): replace without active item, run 'device scan --forget' on the target device
Mar 14 20:58:03 HYATTS kernel: BTRFS error (device sdl1): failed to init dev_replace: -117
Mar 14 20:58:03 HYATTS kernel: BTRFS error (device sdl1): open_ctree failed
When i do a btrfs fi show, the new disk which is a 2TB disk (i bought two to upgrade that pool since i though the drive was failing)
Label: none uuid: ac0daf63-c73c-4130-90a7-65bb7beeb45c
Total devices 2 FS bytes used 210.78GiB
devid 0 size 476.94GiB used 48.03GiB path /dev/sdl1 <--- this is the new disk, that should be a 2TB disk.
devid 2 size 465.76GiB used 216.03GiB path /dev/sdk1
If i attach the old disk via a usb adapter the drive size is the same as the new 2tb disk
Label: none uuid: ac0daf63-c73c-4130-90a7-65bb7beeb45c
Total devices 2 FS bytes used 210.78GiB
devid 0 size 476.94GiB used 48.03GiB path /dev/sdl1
devid 1 size 476.94GiB used 216.03GiB path /dev/sda1 <-- old disk
devid 2 size 465.76GiB used 216.03GiB path /dev/sdk1
The PARTUUID is missing from the pool that wont mount.
# blkid
/dev/sdb1: LABEL_FATBOOT="UNRAID" LABEL="UNRAID" UUID="272C-EBE2" BLOCK_SIZE="512" TYPE="vfat"
/dev/loop1: BLOCK_SIZE="131072" TYPE="squashfs"
/dev/sdk1: UUID="ac0daf63-c73c-4130-90a7-65bb7beeb45c" UUID_SUB="642602af-b04f-416a-aa0b-0e3fd1d31175" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sdg1: UUID="1b6b0121-607d-40da-abba-3066b1df6eb4" UUID_SUB="75fa5d29-33f9-4ed9-bfef-ccee577abdc4" BLOCK_SIZE="4096" TYPE="btrfs" PARTUUID="136c4b ad-a7e5-4237-a238-2e72c0f3701c"
/dev/loop0: BLOCK_SIZE="131072" TYPE="squashfs"
/dev/sdl1: UUID="ac0daf63-c73c-4130-90a7-65bb7beeb45c" UUID_SUB="c912ab9f-d56d-4890-83f7-f2621615d435" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sdh1: UUID="1b6b0121-607d-40da-abba-3066b1df6eb4" UUID_SUB="98f7ee37-7c2e-4f37-ab16-8b1639cc88e2" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/md2p1: UUID="a9433759-3dd6-4469-bf20-5b12a84c0f01" BLOCK_SIZE="512" TYPE="xfs"
/dev/md5p1: UUID="9398a7c0-202a-4469-8d84-ae3b45b0c2e9" BLOCK_SIZE="512" TYPE="xfs"
/dev/md1p1: UUID="e694944e-4ce4-4e5b-8520-8d0e2a4e057c" BLOCK_SIZE="512" TYPE="xfs"
/dev/md4p1: UUID="fb32da95-1beb-4fbb-ade4-d69cdd4643ee" BLOCK_SIZE="512" TYPE="xfs"
/dev/md3p1: UUID="41abf08d-1e8e-4b13-9435-18e0619b608b" BLOCK_SIZE="512" TYPE="xfs"
/dev/loop2: UUID="0be7b92c-f210-4892-8b94-c6bc67aeaa2f" UUID_SUB="7cf8ed06-8d54-41a3-984f-9b1e3610faed" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/loop3: UUID="3f086c01-7c91-489e-b831-c2a8eb9c6f6a" UUID_SUB="177f97d1-d783-4088-af05-f7a5debaa0d6" BLOCK_SIZE="4096" TYPE="btrfs"
btrfs check /dev/sdl1
Opening filesystem to check...
warning, device 1 is missing
Checking filesystem on /dev/sdl1
UUID: ac0daf63-c73c-4130-90a7-65bb7beeb45c
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
[4/8] checking free space tree
[5/8] checking fs roots
[6/8] checking only csums items (without verifying data)
[7/8] checking root refs
[8/8] checking quota groups skipped (not enabled on this FS)
found 226325237760 bytes used, no error found
total csum bytes: 220170504
total tree bytes: 336019456
total fs tree bytes: 58064896
total extent tree bytes: 48136192
btree space waste bytes: 37366285
file data blocks allocated: 243617456128
referenced 224019578880
# btrfs check /dev/sdk1
Opening filesystem to check...
warning, device 1 is missing
Checking filesystem on /dev/sdk1
UUID: ac0daf63-c73c-4130-90a7-65bb7beeb45c
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
[4/8] checking free space tree
[5/8] checking fs roots
[6/8] checking only csums items (without verifying data)
[7/8] checking root refs
[8/8] checking quota groups skipped (not enabled on this FS)
found 226325237760 bytes used, no error found
total csum bytes: 220170504
total tree bytes: 336019456
total fs tree bytes: 58064896
total extent tree bytes: 48136192
btree space waste bytes: 37366285
file data blocks allocated: 243617456128
referenced 224019578880
When i try to mount it with rescue, ro
# mount -o rescue=all,ro /dev/sdk1 /mnt/ssd_cache
mount: /mnt/ssd_cache: mount system call failed: Structure needs cleaning.
dmesg(1) may have more information after failed mount system call.
diagnostics-20250315-0824.zip