After Rebuild Disk "Unmountable: not mounted"

[Dalewn]

Hi together,

 

a couple of days ago I had a drive fail. No big deal I thought; That thing is still in warranty.

So I went ahead and pulled the drive in order to RMA it.

 

I might have made a mistake here and did not stop the arrays beforehand. The server is a Primergy RX300, so pulling the drive from the hot-swappable enclosure should have been fine. At least I thought so.

 

Once the replacement drive finally returned (fast forward like 3 weeks... Mistake number two incoming) I went ahead and pre-cleared it. In the meantime I was still using the array like normal and had quite a few big writes to it.

After pre-clearing was done, I stopped the array, assigned the drive and started it again to start the rebuild process. This all went without any hiccups.

The notification showed something like:

 

GLaDOS: Notice [GLADOS] - Parity sync / Data rebuild started
Unraid Parity sync / Data rebuild
Size: 2 TB

 

GLaDOS: Notice [GLADOS] - Disk 4 returned to normal operation
Unraid Disk 4 message
ST2000VN004-2E4164_Z52CCYFX (sdi)

 

But when I checked to see what the status is in the webUI, I found the disk to be to be unmountable. (Maybe the notification messages here are a little misleading?)

Well, I went ahead and stopped and started the array once more in the hopes of fixing the issue, but to no avail.

 

I also went and checked the FS with

xfs_repair -n

in Maintenance Mode in order to see what might be the issue. The log of that is attached too.

 

I am not all too scared about data-loss since I have backups, but am a little puzzled as to what caused this issue and how to most elegantly fix it. The data in the "Media" folder referenced in the xfs_repair log is not backed up, but can easily be re-downloaded. Most of it is *arr stuff anyway.

 

I was going to follow along this guide here Checking a File System, but figured it wouldn't hurt to ask before taking any actions that might not be undone.

 

Any advice as to how to fix and how to prevent this scenario is very welcomed and thanks in advance!

xfs_repair_log.txt glados-diagnostics-20230329-1109.zip

Edited March 29, 2023 by Dalewn

[JorgeB]

Run xfs_repair again without -n, and if it asks for -L use it.

[Dalewn]

Thanks for the quick reply!

I will do so and report back.

 

At the same time I was hoping to get a little more insight in what I failed at in order to prevent me from repeating my mistake.

[itimpi]

The diagnostics posted did not have the array started in normal mode so we could not see what the state of the array was at that point.  Although it should not have mattered it is never a good idea to pull array drives with the array running as since Unraid is not hot-swap aware you may get undesirable side-effects.

 

Did the drive show as unmountable before the rebuild?    If so then the rebuild would also leave it unmountable (this is mentioned here in the section of the online documentation about rebuilding disks).  In such a case we normally recommend doing the repair on the emulated drive before doing the rebuild .

[Dalewn]

I agree with you on the hot-swap-part. That was a brainfart.

 

IIRC the drive was fine (mountable) until it had failed enough for UnRaid to determine it was bad and disable it. The emulated drive was seemingly working fine too.

 

Anyways, xfs_repair saved the FS on the disk and as far as I can tell some of the files are recovered in the lost+found dir. I guess I'll have to manually sort them in again...
It also created a few files with just plain numbers as names (metadata got lost). Is there a way to determine what those were before?

EDIT: While starting everything back up, I just remembered something. A Scheduled parity check started while the drive was absent from the array and it was started with correction enabled. I cancelled this as soon as I realised, but maybe it was too late... Could this have caused the problems?

Edited March 29, 2023 by Dalewn

[JorgeB]

No, that would not be a problem.