(SOLVED) Input/output error or btrfs leaf corrupt

[DexTroN]

So, I got those 3 foldera on disk3 that "don't exist" anymore and I've tried deleting them with rm -r /path/ but then I get input/output error.

They look like this when I do a find or ls -al /path/

find: `/mnt/disk3/UnRaid/ssss': Input/output error
d????????? ? ?      ?        ?            ? ssss/

at the same time my log shows me this

Nov 4 11:12:41 KL kernel: BTRFS critical (device md3): corrupt leaf: root=5 block=695271424 slot=112 ino=87804, name hash mismatch with key, have 0x0000000024f576f0 expect 0x000000008115b321

So at the moment I am unable to access my files in the Share "UnRaid" because of this.

I did a filesystem check, first read only, then repair and then read only again and it looks like this

checking extents
checking free space cache
block group 941692551168 has wrong amount of free space, free space cache has 595787776 block group has 594493440
ERROR: free space cache has more free space than block group item, this could leads to serious corruption, please contact btrfs developers
failed to load free space cache for block group 941692551168
ERROR: errors found in free space cache
Checking filesystem on /dev/md3
UUID: 89b48731-b52c-40a3-9cae-2e756f837851
found 1428185309184 bytes used, error(s) found
total csum bytes: 1392536424
total tree bytes: 1758773248
total fs tree bytes: 47366144
total extent tree bytes: 50397184
btree space waste bytes: 250423815
file data blocks allocated: 1796143235072
 referenced 1401989853184


Fixed 0 roots.
checking extents
checking free space cache
checking fs roots
ERROR: DIR_ITEM[87804 2165682977] name Plex Media Scanner Chapter Thumbnailw.2.logions)_60_TVOON_DE.mpg.avi1).mpgd trimmings.mpgRay.x264.HQ-TUSAHDm namelen 43 filetype 1 mismatch with its hash, wanted 2165682977 have 620066544
root 5 inode 87804 errors 10, odd dir item
	unresolved ref dir 87804 index 0 namelen 43 name Plex Media Scanner Chapter Thumbnailw.2.log filetype 1 errors 6, no dir index, no inode ref
	unresolved ref dir 87804 index 3364 namelen 43 name Plex Media Scanner Chapter Thumbnails.2.log filetype 1 errors 1, no dir item
checking only csum items (without verifying data)
checking root refs
enabling repair mode
Checking filesystem on /dev/md3
UUID: 89b48731-b52c-40a3-9cae-2e756f837851
No device size related problem found
cache and super generation don't match, space cache will be invalidated
found 1428185309184 bytes used, no error found
total csum bytes: 1392536424
total tree bytes: 1758773248
total fs tree bytes: 47366144
total extent tree bytes: 50397184
btree space waste bytes: 250423815
file data blocks allocated: 1796143235072
 referenced 1401989853184


checking extents
checking free space cache
checking fs roots
ERROR: DIR_ITEM[87804 2165682977] name Plex Media Scanner Chapter Thumbnailw.2.logions)_60_TVOON_DE.mpg.avi1).mpgd trimmings.mpgRay.x264.HQ-TUSAHD namelen 43 filetype 1 mismatch with its hash, wanted 2165682977 have 620066544
root 5 inode 87804 errors 10, odd dir item
	unresolved ref dir 87804 index 0 namelen 43 name Plex Media Scanner Chapter Thumbnailw.2.log filetype 1 errors 6, no dir index, no inode ref
	unresolved ref dir 87804 index 3364 namelen 43 name Plex Media Scanner Chapter Thumbnails.2.log filetype 1 errors 1, no dir item
ERROR: errors found in fs roots
Checking filesystem on /dev/md3
UUID: 89b48731-b52c-40a3-9cae-2e756f837851
cache and super generation don't match, space cache will be invalidated
found 1428185325568 bytes used, error(s) found
total csum bytes: 1392536424
total tree bytes: 1758789632
total fs tree bytes: 47366144
total extent tree bytes: 50413568
btree space waste bytes: 250440007
file data blocks allocated: 1796143235072
 referenced 1401989853184

Then I did a btrfs scrub that looks like this

scrub status for 89b48731-b52c-40a3-9cae-2e756f837851
	scrub started at Sun Nov  4 06:41:13 2018 and finished after 02:06:48
	total bytes scrubbed: 1.30TiB with 3 errors
	error details: csum=3
	corrected errors: 0, uncorrectable errors: 0, unverified errors: 0

The log showed this during the scrub

Nov 4 06:51:16 KL kernel: BTRFS error (device md3): bdev /dev/md3 errs: wr 0, rd 0, flush 0, corrupt 94, gen 0
Nov 4 07:47:42 KL kernel: BTRFS error (device md3): bdev /dev/md3 errs: wr 0, rd 0, flush 0, corrupt 95, gen 0
Nov 4 08:47:59 KL kernel: BTRFS error (device md3): bdev /dev/md3 errs: wr 0, rd 0, flush 0, corrupt 96, gen 0

My fs df looks like this

Data, single: total=1.77TiB, used=1.30TiB
System, single: total=36.00MiB, used=224.00KiB
Metadata, single: total=4.01GiB, used=1.64GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

And lastly my btrfs dev stats /mnt/disk3

btrfs dev stats /mnt/disk3
[/dev/md3].write_io_errs    0
[/dev/md3].read_io_errs     0
[/dev/md3].flush_io_errs    0
[/dev/md3].corruption_errs  96
[/dev/md3].generation_errs  0

So can you please help me make sense of this all?

I've googled and searched the forums but couldn't find anyting and I'm unsure if those 2 issues are connected or not.

Where do I go from here to fix this? I work in IT as support and mostly MS platforms so my *nix knowledge is limited and therefore I don't know how or what to ask.

If I happened to miss anything pleas tell me and I shall provide. And no I do not have ECC ram but I've done memtest86+ for a period of 43 hours please see picture, my PSU is good as well 750w 80+ gold.

kl-diagnostics-20181104-1109.zip

Edited November 12, 2018 by DexTroN

[JorgeB]

Btrfs check --repair should not be used in most cases, only if told so by a btrfs maintainer, for now when there are filesystem issues with btrfs best be prepared to backup, reformat and restore the data, you can use the link below if help is needed with the backup:

 

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=543490

 

You should also convert the metadata on all disks to dup, this should be the default on Unradi for data disks, it can be done like so:

 

btrfs balance start -mconvert=dup /mnt/diskX

 

[DexTroN]

Thanks a lot, I will keep you updated if I need more help, but I would also like to see if it would be fixable and if it isn't fixable then maybe put up a guide saying that there is no point in trying to fix this (insert good searchable term/keywords) and then follow the info you provided me with and some kind of explanation?

[JorgeB]

57 minutes ago, DexTroN said:

and if it isn't fixable then maybe put up a guide saying that there is no point in trying to fix this (insert good searchable term/keywords)

A guide were? on the GUI? You'd need to ask LT for that, though IMO btrfs check --repair should not be available on the GUI.

[DexTroN]

1 hour ago, johnnie.black said:

A guide were? on the GUI? You'd need to ask LT for that, though IMO btrfs check --repair should not be available on the GUI.

Yeah the --repair shouldn't be shown when you press "help".

A guide on the forum where it says "btrfs leaf corrupt or similar issue like that, do this cause X is not easily fixable and therefore this is the easiest/best solution".