Cache drive corrupt

[ayfeQJXvPcdhrEzY]

Hi there, my cache drives seem corrupt, the mover and midnight commander won't copy the files to the main array.

 

This is after I foolishly broke things during a server upgrade. I moved to a new server with the same disks, and I wanted to rebuild the cache with different drives. So I turned off docker, vms, set all shares to Cache: Yes, and invoked the mover. Mover fails with lots of issues from one of the files (my VM disk ), and a scrub shows unrecoverable errors - how can they be unrecoverable if I have a pool of two cache disks?!

 

I get the following errors:

 

Jun 17 19:44:53 unraid kernel: BTRFS warning (device sdg1: state EA): csum failed root 5 ino 27798 off 5299212288 csum 0x8941f998 expected csum 0x5327b8b3 mirror 1
Jun 17 19:44:53 unraid kernel: BTRFS error (device sdg1: state EA): bdev /dev/sdg1 errs: wr 0, rd 0, flush 0, corrupt 11659, gen 0

 

Is there any way to save the files?

 

etestes-diagnostics-20230617-1946.zip

[JorgeB]

Jun 17 19:41:13 eTestes kernel: BTRFS info (device sdg1): bdev /dev/sdg1 errs: wr 0, rd 0, flush 0, corrupt 11658, gen 0
Jun 17 19:41:13 eTestes kernel: BTRFS info (device sdg1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 17055, gen 0

 

There is a lot of corruption detected on both devices, you should start by running memtest

[ayfeQJXvPcdhrEzY]

Memtest is okay so far, but I'll leave it running a while longer. I did turn on my XMP profile previously, so perhaps that caused it, but I've since disabled it Just In Case. If I had corrupted things using XMP briefly, can I still recover?

[JorgeB]

Look here for some recovery options, depending on the amount of corruption it might or not be recoverable.

 

 

[ayfeQJXvPcdhrEzY]

Thank you so much for your replies JorgeB!

 

Update is: Even with XMP memory profile enabled, an all-night memtest shows fine. So I've got no idea what's caused this... None of the recovery options worked, even the destructive one just looped for ages printing the same error message about the same block.

 

I've lost my HomeAssistant VM image (but I have a config backup, so we'll see how well that restores...), but I think everything else is safe.

 

I'm kinda concerned about what the cause of this might have been, I thought having two drives in a pool would make me safe...

[JorgeB]

8 hours ago, ayfeQJXvPcdhrEzY said:

I thought having two drives in a pool would make me safe...

It won't help with filesystem or data corruption, see here so you can better monitor the pool in case it finds new corruptions.

[ayfeQJXvPcdhrEzY]

Much later update (back from overseas): I was able to use:

 

dd if=fileWithBadBlocks of=recoveredFile bs=4k conv=noerror,sync

 

to copy the corrupted file, which was a VM ova file. Somehow the VM still boots and works fine! I was able to save everything I needed from it.