Hello All,
I apologize in advance for the long message. I'm trying to be concise but at same time provide details needed.
The TLDR is I had a power outage at home and multiple data disks and my sole cache drive have failed or had issues during the various data rebuild processes. Dockers don't work and one of my media shares (TV is completely empty now).
Detailed explanation below.
I recently had a power outage at home. My unraid server lost power. I have a DAS configured and it's connected via an external sas.
Both were plugged into an APC unit but only the DAS was on the backup and the primary server surge only. My big mistake.
I powered up the server and it seemed to come up fine. I then noticed a couple data drives disabled. Not ideal but not too big a deal I thought as I have two parity drives and a few spares.
Shut down server replace disk 8 (14TB for 14TB) and parity rebuilds successfully in a little over a day.
Once that's done shut down again and replace second drive, disk 12. I upgraded this disk from 8TB to 12TB. I put the new disk in and all of a sudden disk 11 is completely missing. I checked the connections and all looked good. I decide to let the disk 12 replacement finish rebuilding and then will visit disk 11.
I then try to access my plex instance which is running as a docker on my cache drive. It does not work. I tried to restart the docker but it fails to load the page. I check my shares out of curiosity and I only see my media shares on the data disks but no app data, domain, system, etc. My single cache drive has a crc error but I tested many times before when it first reported that a while back. I again decide to revisit this after the second disk finished rebuilding and disk 11 is dealt with.
The second disk finished rebuilding and returns to normal operation.
I had one more spare and replaced disk 11 and let the data rebuild process start. This was a like for like 12TB for 12TB.
While this starts I get a error message from the Fix Common Problems plugin. It shows that my disk 12 which was recently replaced shows as being unable to write to, ie read only. I will investigate after disk 11 finishes data rebuild.
Before disk 11 finishes rebuild disk 13 goes to error state. I again decide to wait to investigate after disk 11 finishes data rebuild.
Disk 11 finishes rebuild but with many errors.
I stopped the array and disks 8 (replaced) and 13 (not replaced) are both disabled. These are the drives where all the errors are coming from.
I decide to cut my losses for the day and shut down the server and DAS.
I order a new pci sata port expander and sata cables. They came in today and i replaced both. These drives that failed for the most part seem to have come from one pci card. My cache drive was also connected here.
I power my das up first then server. Disks 8 and 12 are still disabled but now show unmountable not mounted.
I may have jumped the gun and tried a few things to fix the problems after reading the forums. I check the filesystem for disks 8 and 13 via the gui. First without the -n and errors are found in both. I try to restart array but they still are not working. I recheck filesystem with -n and errors are found but the disks are still not mountable. Finally I use the -vL when checking filesystem. It seems to partially work as both disks mount but are still disabled.
I then try to focus on my cache drive and ignore the data array for now. I try steps 1 and 2 from this link and they did not work for me. I go with the nuclear 3rd step, BTRFS check --repair. No change.
I have a second ssd and added to my cache in the hopes that it will let me access the shares such as appdata, system, domain. The second ssd adds fine but still no access to dockers.
Finally one of my shares setup for Plex shows as being completely empty. I have two setup, one for movies and one for tv shows. Most of the disks that had issues are in the tv shows share but one is used for movies. I am able to view the contents of the movies share for the most part. For the tv show share in the main view in the console I can see data on the disks allocated for the tv shows share.
At this point is there any thoughts as to what I can do to repair my unraid instance? I am totally lost. I don't know where to begin/continue. Ideally I'd like to not lose any media data but at this point accept I may lose most if not all. I'd like to save whatever data I can as I never made a full backup.
Thanks in advance.
I almost forgot to add the diagnostics file.
tower-diagnostics-20211115-2049.zip