Part of the array mounted Read Only

[ruepel]

hi there, after a couple of weeks of backing up and switching drives around I am still stuck with a RO drive in my array.

• Fix Common Issues tells me that disk3 is mounted read only.
• btrfs scrub of disk3 results in it being aborted after a few hours.
• SMART looks good on all the drives.

What I have tried so far:

1. backup up disk and test files on backup: worked
2. resilver disk in place: worked
3. mount array in production environment: worked for a couple of hours and then failed again (edited)

 

Dec  3 17:01:06 X kernel: BTRFS warning (device dm-2): dm-2 checksum verify failed on 8243588595712 wanted 0x20266073 found 0x54c99e5c level 0
Dec  3 17:01:06 X kernel: BTRFS: error (device dm-2) in __btrfs_free_extent:3095: errno=-5 IO failure
Dec  3 17:01:06 X kernel: BTRFS info (device dm-2): forced readonly
Dec  3 17:01:06 X kernel: BTRFS: error (device dm-2) in btrfs_run_delayed_refs:2144: errno=-5 IO failure
Dec  3 17:01:07 X kernel: BTRFS warning (device dm-2): failed setting block group ro: -30

Dec  4 04:40:25 X kernel: BTRFS error (device dm-2): parent transid verify failed on 865140736 wanted 133651 found 130063
Dec  4 04:40:27 X kernel: BTRFS error (device dm-2): error loading props for ino 194637 (root 5): -5
Dec  4 04:40:31 X kernel: BTRFS error (device dm-2): parent transid verify failed on 865140736 wanted 133651 found 130063

Dec  4 05:31:02 X kernel: BTRFS warning (device dm-2): csum failed root 5 ino 568257 off 0 csum 0xe9a722c6 expected csum 0x00000000 mirror 1
Dec  4 05:31:05 X kernel: BTRFS error (device dm-2): bdev /dev/mapper/md3 errs: wr 0, rd 0, flush 0, corrupt 4455, gen 0

Dec  4 05:31:12 X kernel: verify_parent_transid: 92 callbacks suppressed
Dec  4 05:31:12 X kernel: BTRFS error (device dm-2): parent transid verify failed on 940867584 wanted 133651 found 130308
Dec  4 05:31:12 X kernel: btrfs_lookup_bio_sums: 92 callbacks suppressed
Dec  4 05:31:12 X kernel: BTRFS info (device dm-2): no csum found for inode 568263 start 0
Dec  4 05:31:12 X kernel: BTRFS error (device dm-2): parent transid verify failed on 940867584 wanted 133651 found 130308
Dec  4 05:31:12 X kernel: BTRFS info (device dm-2): no csum found for inode 568263 start 4096

Dec  4 05:31:13 X kernel: BTRFS error (device dm-2): bdev /dev/mapper/md3 errs: wr 0, rd 0, flush 0, corrupt 4472, gen 0
Dec  4 05:31:13 X kernel: BTRFS warning (device dm-2): csum failed root 5 ino 568262 off 0 csum 0x95a1df60 expected csum 0x00000000 mirror 1
Dec  4 05:31:13 X kernel: BTRFS error (device dm-2): bdev /dev/mapper/md3 errs: wr 0, rd 0, flush 0, corrupt 4473, gen 0
Dec  4 05:31:36 X kernel: verify_parent_transid: 62 callbacks suppressed
Dec  4 05:31:36 X kernel: BTRFS error (device dm-2): parent transid verify failed on 841433088 wanted 133651 found 130013
Dec  4 05:31:36 X kernel: btrfs_lookup_bio_sums: 62 callbacks suppressed

All I can gleam from the log is dm-2 which I'm sure is disk3 or /dev/mapper/md3 throws btrfs errors and I don't quite know why. Granted the drive is 99% full (ie. has 250GB space left) but others are at 120GB or 90GB even and those disks don't have any problems.

Has anyone any idea what might cause this issue? Is it the drive that is broken or is it the data on the drive?

vulcan-diagnostics-20211204-1251.zip

[JorgeB]

Looks like some writes were lost to that device, best bet it to backup the disk, re-format to create a new filesystem then restore the data back.