Hi all,

 

I'm having a bit of an odd issue that I'm hoping you can shed some light on.

 

To quickly summarize, my dual NVME Raid1 cache drives are now both showing "Unmountable: No file system" (screenshot attached) and I can't load dockers or my VMs as a result. I also can't balance or scrub them to look for errors.

 

Taking a step back slightly, recently my VM which I keep on the cache randomly stopped working and my docker service wouldn't load after a system reboot. I looked at the terminal and saw a few btrfs errors (BTRFS error (device loop3): open_ctree failed). After some searching I did a balance and scrub which ideally was going to reveal any corrupted files so I could restore them and move on. Sadly I came back with zero errors after multiple scrubs and the terminal was no longer showing errors. So I rebuilt deleted my docker file and rebuilt my dockers and everything seemed fine outside of the fact that my original Windows VM still wouldn't boot but I was kinda assuming it was corrupted and was still hoping to fix it, so it just sat there waiting.

 

Now a month later, I randomly am seeing this "Unmountable: No file system" as shown in the screenshot. I will attach diagnostics but does anyone have any ideas?

 

Thanks all!

 

Side note, I know I have a dying/disabled hard drive in my array. It was already completely empty when it started dying and I have dual parity, so I'm not too worried about it. I have 2 new large parity drives I'm going to switch to, but I wanted to clear up this cache drive issue first before messing with the array and rebuilding my parity.

unraid-diagnostics-20221212-1148.zip

Look like serious corrupted >> data on-disk, but cannot tell you what caused it based on just that, there are some recovery options here you can try but in the way it's corrupted they probably won't work, then it's format and restore the data.

Thanks I will look into these recovery options.

 

Considering I saw a few BTRFS errors initially a month ago, but the scrub showed no files with errors. Any thoughts? I was expecting some corrupted files including my docker file and VM file, but it was clear there were no errors, but then only a month later it comes back with unmountable. Maybe 1 of the 2 NVME drives are dying? They are still under warranty.

15 minutes ago, Vaggeto said:

Maybe 1 of the 2 NVME drives are dying?

If the issue is device related I would suspect more a firmware issue that result in corrupted metadata, not dying device(s).