u0126

Disk shares are really night and day. That shfs overhead is a killer. Seems like my system gets bogged down possibly with I/O having to pass through the shfs layer and that locks things up from the SMB server reading from it... because right now mounting a disk share directly is like it's directly attached to my Windows system. At least for now so I don't have to worry about data corruption weird stuff I'm only going to do activities inside of the specific disk share itself. Not move things in and out of it. It'll let me at least do a lot of cleanup on the specific disk, stuff that was sometimes super painful when trying to go through the user share.

I've got some fun things I've noticed (without any in-depth research) but simple anecdota - if things are "clean" - I haven't done anything to lock up the samba connection, I can get 100-200MB/sec between my Windows system and Unraid (2.5G onboard ethernet on both connected to the same 2.5G switch) and that's great. What sucks is when samba locks up (and seems to happen frequently enough to go to Google once again) and everything stalls out for what feels like an eternity. Just minutes ago I tried to move one folder to another inside of the same share (/mnt/user/foo) and same mapped drive and all, not even that much data (~5 gig) and my entire Windows explorer process wound up locking up for well over 5 minutes. It never timed out or gave up, it just sat there. I can't figure out a discernable pattern so far, other than shfs processes do seem to be busier at the moment (I am doing some other stuff on the array, usually, but nothing that should be completely freezing up simple samba operations)

Yeah, I had set it as ready to restore to. what I'm still curious about is if parity is restoring things back as they were isn't it restoring/emulating a corrupted xfs filesystem? If it's sector-based?

It is rebuilding while the array is active, out of curiosity I looked at /mnt/disk9 and seeing new downloads are hitting that disk. I'm not really concerned about performance (it's still performing well enough) and I'm on vacation not in any hurry for that to finish... it's not at top speed but seems at least 50% last I checked. Previous post seemed shocked that new stuff is being sent to that disk, but I'm just letting unraid do its thing. I'd expect if that was "crazy" it'd leave the disk out of the array while it did that. I'm just confused what it might be emulating (or maybe it's because I'm conceptually thinking it's emulating a "disk", but really it's just emulating "missing sectors" overall?) - when it says a disk is being emulated does it not mean the disk but rather simply the "missing data" (sectors) are being filled in?

How would that work? As of right now /mnt/disk9 is already putting in fresh data. Is that both being rebuilt and able to function at the same time, or is there some version of disk9 that's being rebuilt? Struggling to see how it can rebuild something at the same time it's adding to it. Stopping the rebuild, putting into maintenance, then xfs_repair on dm8? If /mnt/disk9 is mounted/available right now but is being emulated (and emulating the xfs corruption) how is it available and adding new data to it already? Shouldn't it still be corrupt?

Funny enough Unraid is still saying it's emulating the contents of disk9. which is just the *current state* of that drive, right? The drive 42.5% being rebuilt from parity? It's still confusing to me what it's restoring and how it knows it's wrong, if parity can tell, can't I simply xfs_repair what's still there? What exactly is it restoring?

yeah, I understand all that. Like I said most could be re downloaded I just don't know what I lost. so building a file list (unencrypted) is at least the most basic thing. I am shipping backups off (as fast as I can) but it's too late for that disk issue sadly. Literally was doing it this weekend while I was bored on vacation.

If the rebuild to this disk is 42.5% completed does it make sense to stop it and see if there's any way to recover/find anything on it at this point?

ehh. Simply using a different fuel type feels a little bit of a stretch here. Mostly, I misunderstood exactly how parity applied. I did understand destroying the disk would be but I also did not have time to wait for some sort of "recovery" process that I wasn't sure entailed, and still again, thought parity worked like I thought it did from using other "parity" tools 😛 Ultimately I skimmed it and did not fully comprehend it; I'm used to so many other systems (parity tools, ZFS, etc, etc) and did not understand the Unraid application of parity which ultimately can be summed up as above "it comes in only when a disk is missing/failed/unavailable and will emulate the data" and that is it. Also I learned that dual parity doesn't actually provide 2x the amount of parity (which I know other people have thought too), but rather a second copy of a single disk parity using a different mechanism (so that up to 2 disks can be emulated when unavailable) I'll take a look at the script. Even a find -type f is all I'd need in the end. Most I could redownload but I need to know what to redownload. I'm over 50% capacity on ~300TB on Unraid alone, with a bunch of USB drives I need to move into Unraid + possibly shuck the drives and add to the array. Glad this came up _before_ that then, ultimately. I'm sorry for raging, I'm just super annoyed at how quick this came up and ultimately it's just my fault. This was one reason it took a while to decide on Unraid vs. SnapRAID/MergerFS/etc. vs. ZFS, was "do I want to use someone else's management style for my system" but Unraid seemed "hands off" enough... I actually tried SnapRAID/Merger before this and switched off of it in favor of some newer hardware and Unraid as it seemed like it had enough community/support/etc but I misunderstood how some of the internals worked. If I didn't have to leave I might have spent more time exploring options instead of applying my usual ZFS "replace it in place for now" approach, thinking parity provided something else. It would have been nice if there was something that popped up and ran xfs_repair for me or notified me. I didn't fully understand what disabled disk or whatever was and Googled quick and saw some "here's how to fix it" with only a couple of them mentioning gotchas/losing data (but again I thought that applied only if parity hadn't been built yet)

Well sadly that's how I took it, because I thought parity acted like I'm used to with par2 and such. A safeguard statement that if parity wasn't available you'd lose data, something like that. So there we go. What a pisser. All this because I moved things into another room before I left for a trip and wanted to make sure shit was stable before I left for weeks without the ability to physically do anything with it. I didn't see the xfs_repair until much later and I'm used to ZFS failing a disk and being able to put it back right in place. I understand ZFS raidz is actual RAID and this isn't, but as stated, I thought parity worked like I've experienced it with other apps (and maybe it does under the hood somehow, but not in the same portable fashion) The real shitty thing is I don't even know what was on that disk. If I even had a list of files that would have been something. I'm going to setup a daily job to make an entire file list of my system now so worst case any further stuff I at least know what was lost.

If the underlying disk goes corrupt, the parity data about the disk is corrupt, right? So it can't emulate the good data? Bit confused when it jumps in. If a disk goes corrupt, Unraid unmounts it and parity patches the missing data, yes?

yeah, that's just standard "if you format a disk it will delete all data" - not a very important "hey if this was previously a data disk try a repair first!" at that point. Using parity with tools such as par2 it's able to see it figure out that files don't match it's checksums / the parity it originally built and rebuild. That's what I "assumed" it did. Here's the issue I have with your statement. Do you use mysql? Do you know what it does end to end under the hood? No. Do you drive a car? Can you explain everything it does to go from point A to B? No. (And don't be snarky with "actually I do" you can get the idea of the examples) - the selling point of Unraid is its relative simplicity. Even though I know it's not RAID I had to apparently feel the pain of what parity is and isn't as it applies to unraid to learn that. Which is awesome. Parity to me was amazing when I saw how par2 worked; in this case I'm still at a loss of what it really provides - an "emulation layer" for a missing disk basically and no "knowledge" of the data - just bits - which is what data is, but apparently it somehow created parity of some disk that failed/was incorrect just minutes prior. It was fine until I turned it off and turned it back on a couple minutes later and it came up as disk uncountable or whatever.

I didn't have the luxury of time to go through the logs as I was leaving for 3 weeks, I wanted to make sure the system was going to be stable. I mistakenly thought parity worked like par2, where it had knowledge of what was corrupt and would repair. Maybe it does that exact thing but still wondering why it didn't in this case. Apparently the parity on unraid is only good if nothing is corrupt and won't detect/fix corruption (or only does that on a parity "check"?) I just don't understand what the point is. I have 15x 20tb drives - it's a lot of data to have backups immediately available, and it took weeks to get the data centralized so I could begin shipping backups of it.

which is sad because I literally was setting up 2-3 different offsite backup mechanisms to begin this weekend. Like I said I don't even have a list of what I lost now. Unraid really needs to make this shit clearer. Detect the corruption and make it more obvious what next steps to do. Also it seems like everyone is confused how parity works and expects that to provide a loose "backup" So you're saying disk9 started corrupting so the parity of that disk was added in as corrupt?

I guess maybe you did answer that. Any data on disk9 is gone? What the fuck was the point of parity then? disk9 needing a repair should be the same as disk9 being gone or a failure. Where was the 20tb of data being patched in through parity?