I have posted here before about this problem. It happened again. I have Unraid 6.1+ with 7 drives connected to an SAS expander. The server is connected to a rackmount APC UPS. The issue Ive had three times now is the server will become unresponsive. No webgui and SSH is unresponsive also. I can login via SSH but as soon as I run a command it locks up. Powerdown, shutdown, and reboot commands do not work when its like this. The only way to shut it down is by holding the power button. When it comes back up one of the drives will show "unmountable". The last two times I replaced the drive but the procedure didn't work. It starts to do a data rebuild and finishes, but will still say "unmountable". The only way I can get it to use the drive is to format it first then do a data rebuild. This results in losing the data that was on that disk. I'm hoping I'm just doing something wrong. This last time I followed the drive replacement procedure to a T. I'm not really worried about a hardware issue if there is one, I will figure it out eventually, what I'm worried about is I keep losing data.  Shouldn't Unraid protect against this kind of stuff? I've had drives fail in the past and the data rebuild worked like a charm. Anyone have any ideas for me?

 

p.s This last time I loaded the failed drive into my Linux desktop and tried to recover data off it but the drives partition was corrupted : (

A disk showing as 'unmountable' normally means that some sort of file system corruption has occurred and the repair tool appropriate to the file system type needs running to correct it.  Most of the time this will fix such corruption with no data loss, although if the corruption is very severe this may not always be the case.  Experience has shown that reiserfs is the best system at recovering from severe corruption with minimal loss, but XFS is normally OK as well and the repair process is much faster.  BTRFS seems the worst in such scenarios which is probably a reflection on its maturity.

 

If a disk is shown as 'unmountable' before a rebuild then it will still show as 'unmountable' after the rebuild.  This is because the rebuild is nor aware of file systems and works purely at the physical sector level and thus does not fix file system corruption.  You therefore still need to run the repair tool.  Doing a 'format' is rarely the right thing to do as it guarantees any data on the disk will be lost.

If this has happened multiple times, I'd say a very large chance your RAM isn't stable. Run memtest overnight. unRAID can't protect your data if a piece of your hardware is corrupting filesystems. You need to make sure the drive is mountable and in working order before attempting a rebuild. I'd assume it would still be recoverable by repairing the filesystem. If that fails, format the disk to wipe the corrupt filesystem, and then attempt repair again. If that fails, your data is RIP.

 

I assume you are doing monthly parity checks. Do you get random sync errors during them? If you do that also points at faulty RAM.

 

For the record, this is not unRAID's fault, three times now your server has became unresponsive, you knew you had faulty hardware somewhere and choose to ignore it. Even now, it seems you still have the attitude "not worried about hardware failure, i'll find it later". Not a good idea to ignore problems on a data server.

Thank you, I have had this Unraid server since 2011 and never knew there was a repair tool. Where is it located in the webgui? Also is this not in the Unraid manual? I have read the manual several times and haven't seen this info in there? If its not maybe we could get it added? I feel like this could help other people with the same issue.

 

I am not blaming Unraid, its a great OS that I have used for some time now. I am not that upset about the data loss either. I have everything I really care about backed up in two other places. Its mostly just annoying. On the hardware issue, yes I actually do weekly parity checks and there are never sync errors. I don't think its a RAM issue, I mean I guess its possible but it seems like a small possibility. I will do a memtest just to be sure. What I am leaning towards is just old drives. I know it seems silly, but they were all 4+ year old Seagate 2TB drives probably from the same batch. I still have one left so I guess I will see...

Probably not in anything officially labelled a manual, but here is the wiki:

 

Check Disk Filesystems

Probably not in anything officially labelled a manual, but here is the wiki:

 

Check Disk Filesystems

Can I just run the repair tool without rebuilding?

 

Just thought - let's say I copy over a not-so-important file and the drive drops - presumably would cause an error prompt. Then I notice a disk has gone unmountable. Since I know the lost file isn't that important, should I simply run the repar tool instead of waiting for 2 days worth of slow rebuild?

Probably not in anything officially labelled a manual, but here is the wiki:

 

Check Disk Filesystems

Can I just run the repair tool without rebuilding?

 

Just thought - let's say I copy over a not-so-important file and the drive drops - presumably would cause an error prompt. Then I notice a disk has gone unmountable. Since I know the lost file isn't that important, should I simply run the repar tool instead of waiting for 2 days worth of slow rebuild?

 

There is no need to rebuild a disk before attempting file system repair, file system repairs can be made to emulated disks. Keep in mind that rebuilding a disk won't repair the file system, it will just recreate the emulated disk identically on a physical disk. If the disk is unmountable before the rebuild, it will be unmountable after the rebuild.

 

Regarding your thought, the short answer is "No!". Repairing the file system on the unmountable disk might bring that one back, but the disabled disk is still out of sync with the array and needs, as you have been told, to be rebuilt.

 

In all seriousness, if a disk gets disabled when you write to it and another one goes unmountable, you most likely have hardware problems that will make you lose data whatever you do...

Can I still mount an array-unmountable but repaired drive outside of array (via Unassigned Devices) though?

 

So the idea is to run repair => mount it outside array => copy all the data onto a spare HDD (also mounted outside of array) => investigate and fix any issue => put the dropped drive back on array (assuming no issue with the drive) => format => copy data from spare HDD back to the mounted drive.

 

Would that work?

I reckon that would still be faster than trying to rebuild.

Can I still mount an array-unmountable but repaired drive outside of array (via Unassigned Devices) though?

 

So the idea is to run repair => mount it outside array => copy all the data onto a spare HDD (also mounted outside of array) => investigate and fix any issue => put the dropped drive back on array (assuming no issue with the drive) => format => copy data from spare HDD back to the mounted drive.

 

Would that work?

I reckon that would still be faster than trying to rebuild.

 

I think you should start your own thread. And, no, your idea would not work.