So had a first prolonged power outage in a long while.  On a UPS but the comms cable must've slipped out at some point and the machine just died.  After power came back and Unraid came up, I saw that one disk moved from the array to Unassigned Devices (while appearing under Historical Devices too).  Parity seemed fine.

 

Anyway, I though I had a choice between rebuilding from parity or trying new config and hope parity was fine.  So I reassigned the drive back into its slot, started the array and let it about clearing the disk (which felt slightly different to the process I remembered but ok...)

Well, 18h later it's done clearing but FS is auto, shows as unmountable, and I get the scary Format warning.  Is this normal?  Will this just format the drive and start rebuilding its data from parity?  Cuz now it feels extra unusual as if my rebuild attempt has gone sideways and this is somehow truly adding a new drive to the array as opposed to replacing a drive that I thought should've shown us as emulated and simply go through a parity check...

 

What's my best bet right now if I just want to restore Disk 8 from parity and ensure no data loss?  

 

8 minutes ago, another_hoarder said:

Will this just format the drive and start rebuilding its data from parity?

It will update the parity drive to show that the drive is blank, overwriting the corrupt filesystem with a blank one, removing any data.

 

What you are seeing is what is on the drive, a corrupt filesystem. You need to run a filesystem check in the GUI for that drive.

 

The drive is not disabled, it doesn't have a red X.

 

https://docs.unraid.net/unraid-os/troubleshooting/data-recovery/#unmountable-disks

https://docs.unraid.net/unraid-os/manual/storage-management/#checking-a-file-system

@JonathanM thanks for this, glad I didn't format then.  So at this point, because I didn't format, could I just try to rebuild this drive from parity since it hasn't yet been updated? Wouldn't that be one way of getting this data back?

Alternatively, if I run a check now on this filesystem and attempt a fix, wouldn't it be too late for that form of repair since the disk has been cleared?  

I guess, what's the more certain path to getting this disk restored regardless if it takes longer?

5 minutes ago, another_hoarder said:

could I just try to rebuild this drive from parity since it hasn't yet been updated? Wouldn't that be one way of getting this data back?

If you want to, you could physically disconnect that disk, then when Unraid is started it will show what is emulated by parity. Be aware that doing that will put you in a riskier spot, if one of your other disks fails you would be at a greater risk of losing both drives.

 

Just be very very sure you remove the correct disk, and once you remove it, don't put it back in the system until the emulated disk slot has a repaired filesystem. That way if the emulated disk is also corrupt, you at least get another shot at fixing the filesystem on the physical disk.

I see. Thanks, will attempt that next.  Although it was weird in first place upon booting up that the drive in question showed up in Unassigned Devices and Array didn't show as degraded. 🤨

10 hours ago, another_hoarder said:

I see. Thanks, will attempt that next.  Although it was weird in first place upon booting up that the drive in question showed up in Unassigned Devices and Array didn't show as degraded. 🤨

That makes me think something went terribly wrong. I hope you have good backups, sounds like parity may not be valid with that disk. Hopefully it's still valid from before the disk dropped.

Well, just verified my backups and looks like my main rsync to my backup unit has been failing since June but not notifying me via email or Discord.  So 'something went terribly wrong' in every sense of that phrase.

Also, disk check has been running on the emulated disk8 since last night (and doing the endless dots which I suspect is a full scan), didn't see the first Superblock as expected but I'm not sure if there's any hope since it's been cleared first while still sitting in a normally started array, which I suspect was my biggest mistake of two nights ago.

 

 

 

Crucial question - how would I know if parity is still valid from before the disk dropped as opposed to anything done since having updated parity?

7 minutes ago, another_hoarder said:

Also, disk check has been running on the emulated disk8

Did you run the check from the GUI or at the command line? It's easy to get wrong at the command line.

 

It is the emulated disk (red X instead of green dot) that you are checking, correct?

9 minutes ago, another_hoarder said:

Crucial question - how would I know if parity is still valid from before the disk dropped as opposed to anything done since having updated parity?

When was your last parity check that completed with zero errors?

4 minutes ago, JonathanM said:

Did you run the check from the GUI or at the command line? It's easy to get wrong at the command line.

I ran it from the CLI as I couldn't find the option from the GUI.  Trying xfs_repair -v /dev/md8 as it's Disk8 that's awol.
And it's just red x --> disabled/emulated since I yanked it out as you recommended.

 

7 minutes ago, JonathanM said:

When was your last parity check that completed with zero errors?

Looks like yesterday, aligning with the Clear operation on Disk 8 ending right before I started this post.

If you are running a recent Unraid release then disk8 is NOT /dev/md8 - it is instead /dev/md8p1 (i.e. It now includes the partition number), and can be something else if using encrypted drives).

 

you should have been able to do it from the GUI by clicking on the drive on the Main tab.

5 minutes ago, another_hoarder said:

I couldn't find the option from the GUI.

If the file system type is set as Auto, and it can't figure out what type it was, it won't offer an option.

 

Set the file system type to XFS and the check filesystem section should show up.

@itimpi I'm running 6.9.2 and not using an encrypted drive.  Should I try to run that command on the emulated /md8p1 ?

 

@JonathanM The file system is set to Auto on the emulated drive same as it was on the physical drive.  However via GUI, changing it from Auto to XFS doesn't seem to do anything.  Apply is greyed out and just clicking Done keeps it at Auto.

2 minutes ago, another_hoarder said:

However via GUI, changing it from Auto to XFS doesn't seem to do anything.  Apply is greyed out and just clicking Done keeps it at Auto.

Pretty sure the array must be stopped to change it.

2 minutes ago, JonathanM said:

Pretty sure the array must be stopped to change it.

Stopped the array, went into emulated Disk 8.  Changed from Auto to XFS.  Apply still greyed out.  Clicked Done.  Started array in Maintenance mode, went into Disk 8, still shows as Auto.

 

Should I try from the CLI against /md8p1 ?

Actually, that device doesn't exist.

 

 

 

Confirmed, not there.

The p1 bit was added sometime after 6.9.2

 

That version is VERY old, IIRC around 4 years ago. It's tough to offer good support on such an old version.

3 minutes ago, JonathanM said:

The p1 bit was added sometime after 6.9.2

So should I go back and retry the xfs_repair -v /dev/md8 and let it run till it finishes then?

You also asked about the parity check / history - does that give us an idea if I could try popping in another disk and rebuilding from parity?  Or has parity been indeed rebuilt yesterday and I'm out of luck on that front?

 

7 minutes ago, JonathanM said:

That version is VERY old

Doesn't feel that way to an end user where NAS software just feels a lot more "LTS" by design - eg if it works, one tends to not touch it for a while.  All my apps and dockers have worked flawlessly.  But I hear you and an upgrade has been on my radar for a while.  I just know that everything will break and it'll take a month of time I don't have to get everything feeling perfect again

Just now, another_hoarder said:

You also asked about the parity check / history - does that give us an idea if I could try popping in another disk and rebuilding from parity?  Or has parity been indeed rebuilt yesterday and I'm out of luck on that front?

The emulated disk is being emulated from parity. What you see is what would be rebuilt on another disk.

 

Normally when a disk is detected as new and set to be cleared, it shows up with a blue square, at least on current versions. You didn't mention that, which struck me as strange. Once a disk is cleared, data recovery from the physical disk is practically impossible. That's why I was hoping against hope that parity wasn't changed since the incident, as the clearing process is normally done to AVOID the need to update parity when adding a new disk.

 

Before you throw in the towel I'd at least let the check run to completion.

8 minutes ago, JonathanM said:

Normally when a disk is detected as new and set to be cleared, it shows up with a blue square, at least on current versions. You didn't mention that, which struck me as strange.

Ah, yes it did show with a blue square.  I thought I described it clearly enough but didn't mention that.

 

10 minutes ago, JonathanM said:

Once a disk is cleared, data recovery from the physical disk is practically impossible

OK, so def looks like a grave mistake in rushing to clear that drive.

 

11 minutes ago, JonathanM said:

That's why I was hoping against hope that parity wasn't changed since the incident, as the clearing process is normally done to AVOID the need to update parity when adding a new disk.

This must be some fundamental flaw in how I understood Unraid to work this entire time.  I thought that with one drive dropping out from the array (due to whatever kind of failure - physical, filesystem, or otherwise), I could always reliably pop in another disk (or same one if it's just a filesystem corruption), and allow Unraid to Clear + Rebuild from Parity. 

But it somehow seems like something else has transpired initially that made the whole system look just fine without Disk 8 present so when I moved it from Unassigned Devices to Disk 8 slot, it thought I'm simply adding a new drive to the array, eg just adding space.  Which I just don't follow.

 

17 minutes ago, JonathanM said:

What you see is what would be rebuilt on another disk.

So what I see is this.  Does this imply that my Clear of the physical disk (plus the evidence from Parity History) before I yanked it out did in fact write it to Parity so I'm totally out of luck on both trying to get data from the physical drive AND the parity drive? 

 

Additionally, is there a way to find out what I lost?  Eg a file diff between my 11/02 and 11/12 parity checks?

 

22 minutes ago, JonathanM said:

Before you throw in the towel I'd at least let the check run to completion.

Yeah, guess I'll restart it now and let it run till it finishes.

 

3 minutes ago, another_hoarder said:

a filesystem corruption), and allow Unraid to Clear + Rebuild from Parity. 

There basic idea is correct but there is no Clear phase if things are operating correctly.   The fact you mentioned Clear means that something not mentioned must have happened as U raid will only clear what it thinks is a new drive and not just a replacement for a failed one.

1 minute ago, itimpi said:

There basic idea is correct but there is no Clear phase if things are operating correctly.   The fact you mentioned Clear means that something not mentioned must have happened.

@itimpi that's why I mentioned in my initial post that I found it odd (and that's where I should've stopped).  But there wasn't anything unmentioned.  Power went out, came back up, array was started and showed no failure, but missing Disk 8, which was in Unassigned Devices.  And assigning it back to it Disk 8 slot presented the blue box and Clear as the requisite step to add it back to the array.  *sigh*

1 hour ago, another_hoarder said:

array was started and showed no failure, but missing Disk 8,

Did Unraid show that it thought there was a disk assigned to Slot 8 that was missing (which was what should have been the case) and it was emulating disk8, or did it not think a drive had been assigned at all to slot 8?  

1 hour ago, itimpi said:

Did Unraid show that it thought there was a disk assigned to Slot 8 that was missing (which was what should have been the case) and it was emulating disk8, or did it not think a drive had been assigned at all to slot 8?

It was the latter, it didn't show a missing drive, just an empty slot.

17 minutes ago, another_hoarder said:

It was the latter, it didn't show a missing drive, just an empty slot.

That would mean when you added the drive in it would get treated as a new drive rather than a replacement.    Sounds as if somehow when you previously added the drive the configuration information stored on the flash drive was not updated.   

That is truly bizarre @itimpi as that drive was added back in August when I swapped it for a drive that was showing signs of old age and it's been working perfectly since, including a few controlled reboots.  It's too bad Unraid doesn't maintain some sort of a long term persistent log so that I could learn what actually happened and learn from it somehow.  Now my confidence in Unraid is somewhat shaken, mostly by not being able to fully diagnose, debug and understand what could've happened.