Array Disks repeatedly listed as 'Wrong' and unmountable after reboot

[CelestialTeaPot]

Hi folks. I'm new to Unraid and starting moving a Plex server from OMV. I spend a few days copying terabytes of data across but I think I must be doing something terribly wrong because two nights in a row I've restarted my server to find my second Array disk listed as 'Wrong'. After searching I saw a common solution was to set a New Config which I did, but then Unraid could not see my second disk's (8tb Iron Wolf Pro) xsf file system and listed it as auto, with the only option to format.

 

I ran an xfs_repair command but it ran for 24 hours and finally said it could not find superblock and finished. I ran a scan using the free version of Disk Drill which revealed lots of video files still there but most not previewable and missing file info. So not much hope there.

 

Now today the same thing has happened with my second disk which in this case was a 4TB Iron Wolf Pro. The Array Devices list still says it has an xfs file system but I do not want to try the new config route again. Really don't want to lose another 4TB of data. Running xfs_repair now but looks to be the same result as with the previous drive.

 

I've posted my diagnostics here. Any help would be greatly appreciated. My only thought is that maybe I'm filling the drives up too much, leaving only about 100gb free. Thanks!

unraidtm-diagnostics-20230814-2046.zip

[JorgeB]

Disk is being detected with a different capacity, was it always on the current SATA port or did you use it with a different controller or USB enclosure?

[CelestialTeaPot]

Thanks for the reply. Nothing changed. Everything was working fine, then I rebooted the machine and suddnely the second disk was appearing as wrong. I've since moved it to a SATA USB adapter on an Ubuntu machine and managed to salvage all the contents using testdisk. But still not sure why this is happening. My only option right now is to reformat the drives and start copying data again, and hope it doesn't happen a third time!

[JorgeB]

This is very strange, if used with the same controller a disk should always report the exact same capacity.

[CelestialTeaPot]

Interesting, I wonder where I’m going wrong. I assume user error as the same thing has happened twice in a row. I’ve been creating shares spanning multiple disks using the fill up allocation method. Once both disks are basically at capacity, next time I restart the array the second array disk appears as wrong. From there the only thing I’ve managed to do is run new config, run testdisk  on another machine to make disk mountable, format and try again. 
 

Thanks so much your time and input.

[JorgeB]

Used capacity won't cause this, if it's happening with the disk always in the same SATA port I would guess a disk problem, but never seen this before.

[CelestialTeaPot]

Thanks Jorge. I don't think the disks themselves are the issue here, given this has happened to two different disks now (8tb IWP and 4tb IWP) in different NAS bays, two days in a row, both times with array disk 2 (I've had the same 8tb IWP as disk 1 the whole time without problem). The first time was with the 8tb disk in drive bay 3, the second time with with the 4tb disk in drive bay 4.

 

I've been using all these disks for a while now with either openmediavault or proxmox without issue. This is the first time I've tried to create a share folder spanning multiple disks, so I figure I must be doing something wrong.

[JorgeB]

I don't think it's anything you are doing wrong, if it's not the disk it could be the controller, first what kind of hardware is this? Is this a normal board or some kind NAS/lower power board?

 

There are two SATA controllers, two ports each, Intel and Asmedia, disk2 is connected to the Asmedia controller, first test would be to connect for example parity to the Asmedia and disk1 and disk2 to the Intel controller, then try a gain and see if the problem is now with parity.

[CelestialTeaPot]

Thanks Jorge. I'm using a Terramaster F4-423, plus a Terramaster 4-bay USB chassis. All the Array Disks are in the F4-423 main unit, but I have an SSD as a second cache pool attached via USB. Had no problem with the cache pool. Problem is always with array disk 2. The first time it was an 8tb drive in bay #3 of my F4-423, the second time it was a 4tb drive in bay #4. 

 

I just tried this again without changing anything and the same problem happened again. Restarted array and disk 2 is listed as wrong. Have to run testdisk again and copy all the files again. 

[itimpi]

1 minute ago, CelestialTeaPot said:

I just tried this again without changing anything and the same problem happened again. Restarted array and disk 2 is listed as wrong. Have to run testdisk again and copy all the files again. 

This implies that the drive serial is not being reported consistently or its size is being reported differently as this combination is what Unraid uses to identify the drive.

 

It might be worth checking if you get the same issue if you power cycle the hardware rather than simply reboot or stop/start array.

[CelestialTeaPot]

Thanks itimpi. I'll have to go though a bit of a process to test that. To salvage my last failed attempt a ran a New Config which let me add the two drives to the array, but now the second one is showing as Unmountable: Unsupported or no file system. The only way i've found to get around that is to run testdisk on another machine, then copy all the data off, format the drive, then copy all the data back. But then I'll just end up in this spot again next time I restart the array. I will try to power cycle next time though.

 

Edited August 23, 2023 by CelestialTeaPot

[JorgeB]

28 minutes ago, CelestialTeaPot said:

All the Array Disks are in the F4-423 main unit

OK, but it does use two different controllers, try leaving disk1 where it is, swap disk2 with parity, then copy just a few files to disk2 and reboot, see if the problem is with parity now, IMHO this has to be hardware related.

 

 

[CelestialTeaPot]

Interesting. I don't currently have a parity disk but I do have an 8tb drive in bay #1 which i intend to use as parity once i've copied everything across. I just haven't made it that far yet. Currently that drive is being used as temporary storage to move things across once disks are formatted in Unraid.

 

But I can try moving disk 2 from bay #3 into the bay #1 where my intended parity disk is and see how that goes. 

[JorgeB]

1 minute ago, CelestialTeaPot said:

But I can try moving disk 2 from bay #3 into the bay #1 where my intended parity disk is and see how that goes. 

Yes, please try that.

[CelestialTeaPot]

OK, rebooting with that disk in bay #1 it still appears as Unmountable: Unsupported or no file system. I'll have to revive the drive with testdisk, reformat and try again with this new drive order. Will see how it goes. Thanks again for your help here.

[CelestialTeaPot]

Thanks JorgeB, I think you're right about the SATA controllers. Reading a review of my Terramaster F4-423 unit it appears SATA drive bays 1 and 2 use the Intel controller and drive bays 3 and 4 use the ASMedia controller. Perhaps this means the unit only has 2 integrated SATA ports, and the other two are PCI-SATA?

 

I've put disks 1 and 2 into the first two Intel controlled bays and managed to restart the array without issue. I'm copying some files across now to test what happens when I add a 3rd disk to one of the ASMedia bays, but I have a feeling it won't end well. I have heard of lots of people running Unraid on this same unit, so maybe there is still hope.

 

But we'll see. I really appreciate your insight and support here. Thanks!

[JorgeB]

I suspect that it's some issue with the Asmedia controller, but it's kind of strange since those controllers are regularly used with Unraid (onboard or as addon) without issues, so possibly something specific to that controller/firmware.

[CelestialTeaPot]

Well I now have 3 disks working without issue in my array. Disk 1 and 2 in the intel bays and disk 3 in the first Asmedia bay. Array restarts without issue, so far so good! I guess the issue was indeed something related to the Asmedia controller, but it seems to be handling my 3rd disk fine now. We'll see how the parity disk goes in the 2nd Asmedia bay once I've finished copying files from it. Currently mounted as an unassigned device.

 

So thanks again very much for your help. Definitely wouldn't have gotten things going without your insight. Much appreciated!

[Poprin]

I hope you have managed to get this resolved. I have recently migrated my unraid install from an ageing AMD FX machine into a Terramaster f4-423. I had some teething issues but now I'm getting used to new features I wasn't using before and some of the limitations of the hardware it is running absolutely great with Plex docker (a few other low resource ones) and a Win10 VM on cores 3&4. Even Plex 4k encoding with quicksync (which I was unable to do before) works fantastic.

 

Maybe you had a faulty PCIe riser? I believe all the drives in the hotswap bays are connected to the PCIe connector on the board. The separate SATA controllers are present as hardware but I didn't see the ports on the board (but I didn't remove the board from the chassis so could be hidden).