another_hoarder

OK, end of suspense. After parity rebuild ended, it was still unmountable. Formatted and the array is back minus whatever data was on that disk. I'll be upgrading to 7.0 over the Christmas break and in the meantime I'm enjoying a fixed UPS connection, a working backup solution, updated notifications to work Slack and a remote syslog. Thank you both @JonathanM and @itimpi for your help, I learned a bunch of new things. LMK if I should close this thread somehow.

Another interesting update. Even though while it was being emulated only, parity seemed totally fine, upon popping the disk back in, even though (as expected) it still shows up as Unmountable: No file system, and wants to be formatted, a parity rebuild has also kicked off and that hasn't happened before. I'm confused but curious as to what will happen after it's rebuilt. And I've a temporary syslog server up to log it all this time.

Well, 3 days later, with the whole remaining array backed up 🙃 and another attempt at a running xfs repair, the darn browser crashed as the xfs repair tab ran out of memory, going up to to 8GB+. I guess the endless [...] doesn't get cleared but just keeps filling the buffer. *sigh* The official Unraid instructions for using it in CLI, given much larger disks, should be updated, imho, to have that command ran via nohup / screen so that the tab can be closed. Now I'm debating starting that one last time from scratch or just popping the cleared disk in the array and letting it rebuild. It's been lonely without my dockers for almost a week. (This also reminds me that while simplifying a while ago to Unraid has made my homelab very hands off, running a small separate HA proxmox cluster for important dockers meant that nothing was ever offline. Some life decisions to revisit after upgrading from 6.9.2 to 7.0, I guess).

That's also going on my list of post-repair, post-upgrade tasks. Thanks!

That is truly bizarre @itimpi as that drive was added back in August when I swapped it for a drive that was showing signs of old age and it's been working perfectly since, including a few controlled reboots. It's too bad Unraid doesn't maintain some sort of a long term persistent log so that I could learn what actually happened and learn from it somehow. Now my confidence in Unraid is somewhat shaken, mostly by not being able to fully diagnose, debug and understand what could've happened.

It was the latter, it didn't show a missing drive, just an empty slot.

@itimpi that's why I mentioned in my initial post that I found it odd (and that's where I should've stopped). But there wasn't anything unmentioned. Power went out, came back up, array was started and showed no failure, but missing Disk 8, which was in Unassigned Devices. And assigning it back to it Disk 8 slot presented the blue box and Clear as the requisite step to add it back to the array. *sigh*

Ah, yes it did show with a blue square. I thought I described it clearly enough but didn't mention that. OK, so def looks like a grave mistake in rushing to clear that drive. This must be some fundamental flaw in how I understood Unraid to work this entire time. I thought that with one drive dropping out from the array (due to whatever kind of failure - physical, filesystem, or otherwise), I could always reliably pop in another disk (or same one if it's just a filesystem corruption), and allow Unraid to Clear + Rebuild from Parity. But it somehow seems like something else has transpired initially that made the whole system look just fine without Disk 8 present so when I moved it from Unassigned Devices to Disk 8 slot, it thought I'm simply adding a new drive to the array, eg just adding space. Which I just don't follow. So what I see is this. Does this imply that my Clear of the physical disk (plus the evidence from Parity History) before I yanked it out did in fact write it to Parity so I'm totally out of luck on both trying to get data from the physical drive AND the parity drive? Additionally, is there a way to find out what I lost? Eg a file diff between my 11/02 and 11/12 parity checks? Yeah, guess I'll restart it now and let it run till it finishes.

So should I go back and retry the xfs_repair -v /dev/md8 and let it run till it finishes then? You also asked about the parity check / history - does that give us an idea if I could try popping in another disk and rebuilding from parity? Or has parity been indeed rebuilt yesterday and I'm out of luck on that front? Doesn't feel that way to an end user where NAS software just feels a lot more "LTS" by design - eg if it works, one tends to not touch it for a while. All my apps and dockers have worked flawlessly. But I hear you and an upgrade has been on my radar for a while. I just know that everything will break and it'll take a month of time I don't have to get everything feeling perfect again

Actually, that device doesn't exist. Confirmed, not there.

Stopped the array, went into emulated Disk 8. Changed from Auto to XFS. Apply still greyed out. Clicked Done. Started array in Maintenance mode, went into Disk 8, still shows as Auto. Should I try from the CLI against /md8p1 ?

@itimpi I'm running 6.9.2 and not using an encrypted drive. Should I try to run that command on the emulated /md8p1 ? @JonathanM The file system is set to Auto on the emulated drive same as it was on the physical drive. However via GUI, changing it from Auto to XFS doesn't seem to do anything. Apply is greyed out and just clicking Done keeps it at Auto.

I ran it from the CLI as I couldn't find the option from the GUI. Trying xfs_repair -v /dev/md8 as it's Disk8 that's awol. And it's just red x --> disabled/emulated since I yanked it out as you recommended. Looks like yesterday, aligning with the Clear operation on Disk 8 ending right before I started this post.

Well, just verified my backups and looks like my main rsync to my backup unit has been failing since June but not notifying me via email or Discord. So 'something went terribly wrong' in every sense of that phrase. Also, disk check has been running on the emulated disk8 since last night (and doing the endless dots which I suspect is a full scan), didn't see the first Superblock as expected but I'm not sure if there's any hope since it's been cleared first while still sitting in a normally started array, which I suspect was my biggest mistake of two nights ago. Crucial question - how would I know if parity is still valid from before the disk dropped as opposed to anything done since having updated parity?

I see. Thanks, will attempt that next. Although it was weird in first place upon booting up that the drive in question showed up in Unassigned Devices and Array didn't show as degraded. 🤨