BTRFS ERROR, CACHE possiblly corrupted

[Swagnoor19]

Hey guys,

 

I'm getting errors on my unRAID server:

Feb 25 18:26:03 SwagServer kernel: BTRFS warning (device sdd1): csum failed root 5 ino 11057991 off 73400320 csum 0x491c2044 expected csum 0xf2770f0e mirror 1
Feb 25 18:26:03 SwagServer kernel: BTRFS error (device sdd1): bdev /dev/sdc1 errs: wr 3341754, rd 454753, flush 228253, corrupt 5, gen 0
Feb 25 18:26:03 SwagServer kernel: BTRFS warning (device sdd1): csum failed root 5 ino 11057991 off 73404416 csum 0x1b593b06 expected csum 0x8941f998 mirror 1
Feb 25 18:26:03 SwagServer kernel: BTRFS error (device sdd1): bdev /dev/sdc1 errs: wr 3341754, rd 454753, flush 228253, corrupt 6, gen 0
Feb 25 18:26:03 SwagServer kernel: BTRFS warning (device sdd1): csum failed root 5 ino 11057991 off 73408512 csum 0x722766e4 expected csum 0x8941f998 mirror 1
Feb 25 18:26:03 SwagServer kernel: BTRFS error (device sdd1): bdev /dev/sdc1 errs: wr 3341754, rd 454753, flush 228253, corrupt 7, gen 0
Feb 25 18:26:03 SwagServer kernel: BTRFS warning (device sdd1): csum failed root 5 ino 11057991 off 73412608 csum 0x09d94116 expected csum 0x8941f998 mirror 1
Feb 25 18:26:03 SwagServer kernel: BTRFS error (device sdd1): bdev /dev/sdc1 errs: wr 3341754, rd 454753, flush 228253, corrupt 8, gen 0
Feb 25 18:26:03 SwagServer kernel: BTRFS error (device sdd1): parent transid verify failed on 4369711218688 wanted 794631 found 791060
Feb 25 18:26:03 SwagServer kernel: BTRFS error (device sdd1): parent transid verify failed on 4368831709184 wanted 794326 found 791150
Feb 25 18:26:03 SwagServer kernel: BTRFS warning (device sdd1): csum failed root 5 ino 11057991 off 73629696 csum 0xb4e5e826 expected csum 0xd2b663e6 mirror 1
Feb 25 18:26:03 SwagServer kernel: BTRFS error (device sdd1): bdev /dev/sdc1 errs: wr 3341754, rd 454753, flush 228253, corrupt 9, gen 0

 

I've been trying to solve this on my own. I tried to see which file is at 11057991 and I've found that it's the docker.img file. I have tried to delete this file from the GUI, by following the guide, but I'm still getting the error and the file is not deleting.

 

I then tried to delete it manually from command line but I get another error that says the cache is a read-only filesystem.

 

I have also tried to restart the server and when I did I saw this error (see screenshot.)

 

My sever has been down for over a week so any help would be greatly appreciated. I'm attaching diagnostics as well.

 

swagserver-diagnostics-20220225-1844.zip

[JorgeB]

Feb 25 18:21:38 SwagServer kernel: BTRFS info (device sdd1): bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 67623, gen 0
Feb 25 18:21:38 SwagServer kernel: BTRFS info (device sdd1): bdev /dev/sdc1 errs: wr 3341754, rd 454753, flush 228253, corrupt 0, gen 0
Feb 25 18:21:38 SwagServer kernel: BTRFS info (device sdd1): bdev /dev/sdb1 errs: wr 284514652, rd 245960197, flush 2035012, corrupt 37140664, gen 57

 

All pool devices show errors, read/write errors suggest two of them dropped offline in the past, suggest you backup and re-format the pool, then check here for better pool monitoring.

[Swagnoor19]

Thanks for your response. I was able to follow your guide for mounting the cache to a different mount point and copied over the data from each drive (although I think all the drives had the same data since they where in a pool.)

 

But I'm having a bit of trouble finding how to format the drives. Could you point me in the direction where I can find that?

[JorgeB]

With the array stopped wipe all pool devices with:

 

blkdiscard /dev/sdX

 

Replace X with the correct letters, after all 3 are wiped start the array and you'll have the option to format the pool.

[Swagnoor19]

Thanks for this. I was able to format the pool but when I checked the array size in the ui, it's showing the total capacity as 756GB and the free capacity as 501GB. I have 1 512GB SSD and 2 Crucial 500GB SSDs. If I set the balance to raid 1, shouldn't the capacity bee like 1000GB?

[ChatNoir]

7 hours ago, Swagnoor19 said:

If I set the balance to raid 1, shouldn't the capacity bee like 1000GB?

How would you have two copies of the data (RAID1) on a 1512GB total capacity and be able to use 1000GB ?

 

756GB is in line with BTRFS RAID1 : https://carfax.org.uk/btrfs-usage/?c=2&slo=1&shi=1&p=0&dg=1&d=500&d=500&d=512

Also, consider that the free space displayed is not very reliable on an odd number of drives. (available space is good though)

[Swagnoor19]

Okay I see. So I was thinking of changing the pool to raid 5, but after reading around, it might not be such a good idea? Do you know why it's recommended against a raid 5 on unRAID?

Also, what would you recommend for a pool that's used for downloading and storing appdata?

[JorgeB]

Btrfs raid5 has some known issues, it can be used, and I do, but I also have good backups.