Juliet

tortuga-diagnostics-20260529-2302.zip

Thanks - tried that, docker still doesn't seem to want to start. I'm noticing my rebuild is running impossibly slow. At this point, estimated time of completion is going to be 12 years from now...lol. I still feel like things are really unhappy.

Following an OS upgrade, my system came back up with two drives disabled. I checked the SMART status and they both looked good, so I mounted in maintenance mode and tried running an xfs_repair on each from the UI and it said it fixed the corruption. Unfortunately, they were still in a disabled state. Per some info I found on the forums, I heard the correct way to handle this was to let them rebuild from parity by removing them in maintenance mode, starting, then stopping an re-adding in the same slots - so I attempted to do so. I still have one showing a read error, but it the rebuild has started. I don't know if there's anything else I need to do, but my docker service doesn't seem to want to start - so I've attached my diagnostics.zip to see if any of the wizards here might be able to help. tortuga-diagnostics-20260529-1704.zip

Yeah, I figured as much. Short of some sort of forensic recovery, I doubt I'm getting anything off of Disk4. No, I didn't have any of the stuff on there backed up. Currently rsyncing the most important stuff from the array to an external drive (which was something on my list of things to do this week, before all the trouble started), but I lost all my docker containers and a bunch of media. We didn't lose power or anything...I have no idea what happened to cause this initially, but I'm sure I trashed the FS when I inadvertently started a rebuild of Disk 4.

I've reassembled the array and restarted it. Here's the current Diagnostic tortuga-diagnostics-20250305-0832.zip

Does order matter? Do they have to be assigned the same drive numbers? I took a screenshot, so I can reassemble it.

I did with my last update, but since then, the parity rebuild failed, and the current state is that the array is stopped and all drives are unassigned so I could try and mount them individually to recover whatever I could. If you think there's any chance of fixing it, let me know, I guess. tortuga-diagnostics-20250305-0809.zip

On a related note, initially both my parity drives showed rebuilding, but assigned - however it seems Parity1 just went disabled again 😞

Well, I'm pretty sure I just boned it. I tried to stop the array, but had a terminal open in the background where the CWD was within /mnt/user, so the array never stopped. I didn't realize this, so I tried to go ahead with the New Config, but as the docs state "After doing this there is no obvious feedback that anything has been done.", I assumed the "apply" button had done its job. So I headed back to Main, kept trying to remember where I needed to be to reassign Disk 4 to the array, but then realized it was still stuck in "Stopping". After a bit of googling, I saw the answer, realized I had the terminal open, and I exited it - the array stopped. I went ahead and assigned Disk4 and started it again, only to see Disk4 was rebuilding (which you mentioned we were trying to avoid). I panicked and stopped the rebuild. Then I went and tried the "New Config" again, and it was more obvious that something had actually happened. When I returned to Main, I tried starting the array with Disk4 assigned, and then it started trying to rebuild Parity (which we originally wanted) - but Disk4 was now unmountable. By starting that rebuild, I am pretty sure I trashed the filesystem. I tried running xfs_repair on it, and it completed successfully, but Disk4 shows "Unmountable: Unsupported or no file system" and the disk log shows XFS corruption errors. So the FS on that drive is in a bad way, and my parity has been wiped - so I'm pretty sure I'm boned. None of it is your fault...just one of those "oh sh!t" moments and I did a stupid in my panic. Here's a fresh diagnostics zip in case theres any chance of salvaging the data on Drive4. Don't mind the new "sdj" - that's just a 16tb USB drive I plugged in to backup whatever I'm able of my most critical data in case things go further south. 😕 tortuga-diagnostics-20250304-2109.zip

Don't know if it's relevant/useful, but with drive 4 merely emulated, a ton of data is missing from my Music folder in my Audio share. However, that data does appear to be present on the mounted, unassigned Drive 4. Thank you for all your help thus far, btw. I'm a DevOps Engineer by trade, but I spend my days in the clouds, and I don't get this close to the metal too often.

Unassigned devices seems to have mounted it.

Sorry, I'm not clear here. I can click "Compute All", which seems to churn for a very, very long time. When I click on lost+found, I do indeed see many, many folders in there for which the location is "Disk 4". Is there a way to recover these? Edit: Nevermind - Compute All is completed:

Here you go. Thanks. tortuga-diagnostics-20250304-1127.zip

I tried running xfs_repair -L (via the Check button in the UI) and it seemed like it finished successfully. Unfortunately, the drive isn't coming back, and I'm reluctant to do anything with the parity drive (I have two). For context, I had an array of 16tb & 8tb drives - two 16TB parity drives, one SSD cache drive. I removed a dead 8TB, replaced with a 16TB, added it to the array, and let it rebuild for two days. This morning, I logged into my machine and found a bunch of missing files, so I started investigating and saw all the xfs corruption errors. That's when I attempted the repair after reading some forum threads. Diagnostic zip attached. tortuga-diagnostics-20250304-0749.zip