Disk errors on 1 XFS disk in the array with NO parity. Can files be recovered somehow?

[shEiD]

I have been running unRAID for years now and I love it. But I have never used parity. Finally it seems, I'm gonna be hit with consequences of not using the parity 😔

 

Some days ago, I ran a simple bash script that deletes all empty directories on all disks on my unraid array, and I got an error on disks12. I thought - that's weird... because the script is very simple:

for ((number=1;number < 29;number++))
{
    disk="disk$number"
    folder="/mnt/$disk/media"
    echo "deleting in... $disk"
    find $folder -type d -empty -delete
}

I checked the unraid webui and saw that disk12 had 32 errors. I checked the syslog and found that disk12 had XFS errors.

The folder, that errored out when I ran the script, was empty. All the files and subfolders in that folder where gone. That folder was only on this one disk12. So basically everything in that folder was gone.

 

Like I said - my array has no parity protection.

I am a Linux newb, and have only modest experience with unRAID. But I vaguely remember reading somewhere that XFS has some magic-like recovery abilities. I maybe misremembering, but I hope I'm not 😉Secondly, I remember reading here on the forums, that in case of a problem and before doing anything - I should ask for an advice.

 

So, here I am, asking for help. Is there any chance/way to try and recover the missing files? One hopefully good thing - I think no new files have been written or updated on disk12 since these errors happened. So I am hoping, that if there is any way to recover the files, they have not been corrupted by over-writing the blocks.

 

TBH, the most painful part is the sour irony, that I have 7 brand new 10TB drives that have been tested and precleared. They have been sitting in a drawer for almost 6 months. The plan was to replace 5 smaller 3TB array drives and then finally add 2 parity drives 🤣
The only thing I was waiting for was 6.9.0 release candidate. I wanted stable multiple cache pools to make my migration easier.
So I had the disks, the 6.9.0 RC came out, and on Christmas Day I got a present - these disks errors... 🤣

 

I've attached the diagnostics and some screenshots.

 

Please, help.

 

thepit-diagnostics-20210107-0044.zip

Edited January 7, 2021 by shEiD

[JorgeB]

Without parity best bet is to use ddrescue, after the clone run a filesystem check.

[JorgeB]

Note that for the clone to mount in Unraid it needs to be the same size, if you use a larger disk you can still mount with UD.

[shEiD]

I finally finished moving out all the files from this disk using unBALANCE. I did not touch/move that directory, which gives disk errors.

 

Interesting - the drive should be completely empty, as the only thing left is that borked folder, which shows up empty over network. But unRAID webUI shows drive is still using 321 GB, which is actually the size of that borked folder, ie: the size of all teh files inside that folder, that are now gone... 🤔

Does this mean, that unRAID somehow sees the actual files?

Maybe a silly question - would it be ok for me to move this drive to another machine to do the ddresque? I have an old rig, that I use as a secondary unRAID to preclear new drives. I would prefer to do the rescue on that machine, instead of my main unRAID server.

Also, I have never done ddrescue before - anything important that I have to look out for, or know about - to at least try and not mess this attempt of recovery by my own stupid hands? 😄

 

A screenshot - notice the 321 GB used by the ghost files on disk12 👀

[ChatNoir]

Did you check on the recycle bin plugin ? I see that it is present on your initial diagnostics.

[JorgeB]

There's filesystem corruption on disk12, but it might not be fixable on a failing disk, hence why I suggested ddrescue.