Hoping someone can set me straight. I helped someone get setup with UNRAID and their setup is a mini-PC a single internal SSD as a cache drive and then an external 4 drive USB 3 enclosure. The enclosure doesn't do any sort of RAID nonsense and does present each disk as a single volume but we did find that it does not pass all of the drive information across because on my UNRAID setup the drives display as "Manufacturer_Model#_SerialNumber" but on my friends setup the devices show up as "Manufacturer_Model#" making discerning one drive from the others difficult since we have 2 sets of identical drives (2x6Tb Hitachi and 2x8Tb Western Digital).

 

All of that info aside we had setup unRAID and all was working with 3 data drives (2x6tb & 1x8Tb), a single 8Tb parity drive, and a SSD cache drive. We had migrated all of the data onto the XFS array and a parity sync completed but then a few days later we noticed all of the dockers had stopped running and when I logged in remotely to lend a hand I saw errors about one of the 6Tb drives and have attached screenshots of that. I imagine this is a disk failure and that installing a new 6 or 8Tb drive and doing a parity rebuild is what we need to do but I am not 100% sure.

 

I looked through the forums and some people say "if the drive is okay rebuild on top of it" but I don't know how to confirm if that drive is healthy and I do not have physical access to it. At the time of posting this I did run the Diagnostics tool and have attached those. The system is running UNRAID 6.12.4 but it is a trial version expiring in 7 days so hoping to figure this out by then

gertgen-server-diagnostics-20231213-1419.zip

USB not recommended for array or pools for many reasons including unreliable connections. 

 

If you didn't use parity there would be nothing to get out of sync, and so no need (or ability) to rebuild. 

Trust me the USB enclosure is being used by necessity not choice. I'm actively looking for a cheap desktop that he can migrate to. That being said the 4 drives were mounted in this enclosure and had 15+ Tb of data copied onto them and remaining online and functional for days afterwards. We powered everything down and reseated all of the drives. So again I understand, and agree, with the "USB is bad" sentiment but for the moment that's what my friend has available.

 

I'm just wondering if I should rebuild over the existing drive or mail him an 8Tb drive and do a rebuild.

Do you have another copy of anything important and irreplaceable?

 

SMART for disabled disk2 looks OK though no SMART self-tests have been run on it.

 

But, we can't tell from the diagnostics if the emulated drive is mountable. Better to fix the emulated drive if necessary before rebuilding.

 

Start the array and post new diagnostics so we can see if the emulated drive is mountable.

I started the array in Maintenance mode, ran the diagnostic tool, and have that file attached. I've also attached a photo of what i see in the "Main" area.

gertgen-server-diagnostics-20231213-1613.zip

3 minutes ago, PeaInAPod said:

started the array in Maintenance mode

That mode doesn't attempt to mount any drives, so we still don't know if the emulated disk2 has a mountable filesystem.

 

Start the array in Normal mode and post new diagnostics.

The array was stopped, brought online normally, and I've posted those diagnostics.

gertgen-server-diagnostics-20231213-1628.zip

Emulated disk2 is mounted and seems to have plenty of contents.

 

48 minutes ago, trurl said:

Do you have another copy of anything important and irreplaceable?

 

We could feasibly replace everything if needed and there are backups present. I've been using UNRAID for some time but never encountered this sort out of issue. Based on what you're seeing in the diagnostics what do you think happened and what is your suggestion on us resolving this?

Unraid disables a disk when a write to it fails. This is because parity is updated anyway so the write (and any subsequent writes to the disk) can be recovered. After a disk is disabled, Unraid doesn't use it anymore. Instead, it emulates the disk from the parity calculation. Reads from the disk instead read all other disks and calculate the data for the read. Writes for the disk update parity as if the disk had been written, so the write can be recovered by rebuilding.

 

Also, if a disk can't be read, Unraid will get its data from the parity calculation by reading all other disks, and try to write it back to the disk, which can fail and disable the disk.

 

So the data disk is out of sync with the array.

 

SInce SMART for the disk looks good, almost certainly USB caused the disk to disconnect, which lead to this failure. But the syslog in those initial diagnostics doesn't include any I/O errors that I would expect to see. Possibly the system had been rebooted after the disk became disabled, and the system was allowed to run for several days like that since that syslog goes back to Dec 2.

 

Rebuild is the normal way to get the array back in sync. Since the disabled disk is the disk out-of-sync (due to write failure) that is the disk you would rebuild.

 

Another way to get the array back in sync is to forget about that failed write, trust the contents of the disabled disk, and rebuild parity instead. But there is some chance the failed write could be filesystem metadata which might mean corruption.

 

If this weren't all on USB I wouldn't hesitate to recommend rebuilding the data disk on top of the same drive since it seems to be OK. Might be safer to not trust USB to rebuild on top, but instead rebuild onto another disk and keep the original as is in case of problems.

 

Or rebuild parity instead since it contains none of your data. But that would mean any writes to the data disk since it became disabled would be lost.

Thank you for that detailed breakdown of how UNRAID handles disk disconnects and the rebuild options it clarified a lot of things. I agree that rebuilding parity only was probably safe given what we know but not 100% guarenteed safe so since I'm fairly confident the disk is healthy I went with a full rebuild on the existing disk. That is running now with an estimated completion of 18 hours for the 6Tb of data it has to rebuild so I'll see what happens then.

 

Assuming that all goes well I'll report back once that finishes just to provide some closure on this

Again thanks so much for the help. I know enough to troubleshoot my own UNRAID system but with this goofy setup and being that I can't be there in person to troubleshoot things I didn't want to make an assumption and make my friends issue worse

Well the rebuild finished without issue and everything appears functional. Logs are attached not because I suspect anything is wrong but I figured might as well attach diagnostics at this point just to be thorough.

 

I really appreciate your help and explanations. I definitely left with a deeper understanding of UNRAID. Fingers crossed that Santa finds my buddy something else to use so we can dump this USB enclosure. I will say that we got 100Mb/s sustained writes so at least we have that going for us.

gertgen-server-diagnostics-20231215-1052.zip

Edited December 15, 20232 yr by PeaInAPod
Didn't attach logs

Your "default" shares are not configured ideally.

 

appdata, domains, system shares should be on fast pool such as cache with no files on the array, and configured to stay there. so Docker/VM will perform better, and so array disks can spin down since these files are always open.

 

Your cache is a bit small so maybe only those shares should use cache.

 

Nothing can move open files so in addition to setting these shares to move to cache, you have to disable Docker and VM Manager in Settings.

 

If you think you will be getting new hardware soon you might wait until you have all that working then we can revisit this.

Yeah we'll revisit this once my friend ends up with better hardware. Again I appreciate all of your help!