BTRFS scrub uncorrectable errors

[ipreferpie 0]

Hi, after a kernel panic and a hard reset, I conducted a BTRFS scrub on my RAID1 cache pool. What I got were 4 uncorrectable errors. I deleted the docker.img and conducted SMART tests on both SSDs without issue. But upon running the scrub again, I still get the same 4 errors. I'm not sure if it's a hardware problem and how I should proceed to fix this. Below are the logs and attached is the diagnostics. Many thanks.

 

Cache 1:

Nov 10 16:13:09 HK-HomeLab kernel: BTRFS warning (device dm-10): checksum error at logical 530594066432 on dev /dev/mapper/sdn1, physical 458653364224, root 5, inode 118530799, offset 28672, length 4096, links 1 (path: appdata/binhex-plexpass/Plex Media Server/Metadata/TV Shows/2/902b34d01d1d00699de74ef41aa4a378087be30.bundle/Contents/com.plexapp.agents.thetvdb/seasons/5/episodes/13/thumbs/66948d605b6a303889512aaaad8668cb4b5d7106)
Nov 10 16:13:09 HK-HomeLab kernel: BTRFS error (device dm-10): bdev /dev/mapper/sdn1 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
Nov 10 16:13:09 HK-HomeLab kernel: BTRFS error (device dm-10): unable to fixup (regular) error at logical 530594066432 on dev /dev/mapper/sdn1
Nov 10 16:13:09 HK-HomeLab kernel: BTRFS warning (device dm-10): checksum error at logical 530594070528 on dev /dev/mapper/sdn1, physical 458653368320, root 5, inode 118530799, offset 32768, length 4096, links 1 (path: appdata/binhex-plexpass/Plex Media Server/Metadata/TV Shows/2/902b34d01d1d00699de74ef41aa4a378087be30.bundle/Contents/com.plexapp.agents.thetvdb/seasons/5/episodes/13/thumbs/66948d605b6a303889512aaaad8668cb4b5d7106)
Nov 10 16:13:09 HK-HomeLab kernel: BTRFS error (device dm-10): bdev /dev/mapper/sdn1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
Nov 10 16:13:09 HK-HomeLab kernel: BTRFS error (device dm-10): unable to fixup (regular) error at logical 530594070528 on dev /dev/mapper/sdn1

 

Cache 2:

 

Nov 10 15:59:58 HK-HomeLab kernel: BTRFS warning (device dm-10): checksum error at logical 530594066432 on dev /dev/mapper/sdd1, physical 36651855872, root 5, inode 118530799, offset 28672, length 4096, links 1 (path: appdata/binhex-plexpass/Plex Media Server/Metadata/TV Shows/2/902b34d01d1d00699de74ef41aa4a378087be30.bundle/Contents/com.plexapp.agents.thetvdb/seasons/5/episodes/13/thumbs/66948d605b6a303889512aaaad8668cb4b5d7106)
Nov 10 15:59:58 HK-HomeLab kernel: BTRFS error (device dm-10): bdev /dev/mapper/sdd1 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0
Nov 10 15:59:58 HK-HomeLab kernel: BTRFS error (device dm-10): unable to fixup (regular) error at logical 530594066432 on dev /dev/mapper/sdd1
Nov 10 15:59:58 HK-HomeLab kernel: BTRFS warning (device dm-10): checksum error at logical 530594070528 on dev /dev/mapper/sdd1, physical 36651859968, root 5, inode 118530799, offset 32768, length 4096, links 1 (path: appdata/binhex-plexpass/Plex Media Server/Metadata/TV Shows/2/902b34d01d1d00699de74ef41aa4a378087be30.bundle/Contents/com.plexapp.agents.thetvdb/seasons/5/episodes/13/thumbs/66948d605b6a303889512aaaad8668cb4b5d7106)
Nov 10 15:59:58 HK-HomeLab kernel: BTRFS error (device dm-10): bdev /dev/mapper/sdd1 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0
Nov 10 15:59:58 HK-HomeLab kernel: BTRFS error (device dm-10): unable to fixup (regular) error at logical 530594070528 on dev /dev/mapper/sdd1
 

hk-homelab-diagnostics-20191110-0837.zip

[johnnie.black 2326]

If there are uncorrectable errors you should backup cache data, reformat and restore data

[ipreferpie 0]

15 hours ago, johnnie.black said:

If there are uncorrectable errors you should backup cache data, reformat and restore data

thanks! so basically, can I 1) move my files on to the array, 2) take the pair of cache drives offline, 3) format, 4) then reintroduce it back as new cache drives, 5) move the data back from array to cache?

[johnnie.black 2326]

Yep, to format you don't need to offline the cache devices, once data is moved just wipe both with:

 

blkdiscard /dev/sdX

 

 

[ipreferpie 0]

7 hours ago, johnnie.black said:

Yep, to format you don't need to offline the cache devices, once data is moved just wipe both with:

 

blkdiscard /dev/sdX

 

 

got! thanks

[ipreferpie 0]

so I gave it a try, but I got:

BLKDISCARD ioctl failed: Remote I/O error

 

should I take it offline and run BLKDISCARD again?

[johnnie.black 2326]

That means the controllers where the SSDs are connected doesn't support trim.

[ipreferpie 0]

got it, thanks! reformatted and all looks good again after a scrub