teslap4

So parity sync has finished after a new config. the sync had 0 errors. Yay! So it looks like my problem is resolved. Thank you tremendously for your assistance JorgeB. For those that come through this thread looking for assistance on a similar issue, I will summarize this event. At the start of a scheduled parity check, one of my data disks was put into a disabled state. When this happened, the disk was put offline and the data was not being emulated correctly. The array had to be stopped and started for the disk to appear in an emulated state. The server has two HBA cards with 2 sets of 4 breakout cables on each hba, for a total of 16 sata ports for 10 data and 2 parity disks (and some spares and other pools such as cache, etc). While attempting to verify the state of the disk by running an extended SMART test, the server hard-locked/crashed. After running a second extended smart test on the disk, which completed without issue, the disk was determined to be ok. At this time, the emulated contents of the disk were not mounted/viewable. To fix the emulation problem, we ran xfs-repair with the array started in maintenance mode. I first ran xfs-repair with -n to verify if there were errors to be fixed. There were. I then ran xfs-repair -L after being unable to run xfs-repair with no options. When xfs-repair -L ran, a second disk went into disabled state. I believed this might have been caused by a failing HBA or damaged breakout cables. I had a spare hba on hand that can handle 16 disks on its own, so i replaced the two HBAs and breakout cables with a different, known-working HBA with brand-new cables. The second disabled disk was being properly emulated. xfs-repair -L was run against the first failing disk, the first time corrected errors but gave output that it recommended running again to complete successfully. After the second run, the xfs-repair made the emulated contents mountable and viewable. However, the contents of the disk were incomplete - parity was only emulating about half of the disk contents. So this suggested that parity was corrupt/bad/incorrect. When mounting both of the disabled disks as unassigned devices (while their contents were being emulated by the 2 parity disks), the disabled disks content was verified to be correct and intact. With this in mind, both disabled disks did not have any issues with their extended smart reports after replacing the HBA, suggesting that the HBA/cables might have been the culprit. So to get my data back to a fully accessible state, a "New Config" was created while preserving disk assignments. A parity sync was performed to rebuild the parity. That completed with 0 errors. I have yet to perform a visual inspection or other verification of the suspect hardware, but given my current chassis is not hot swap and thus necessitates opening the chassis to swap disks, it is very possible one of the sata breakouts was damaged during previous disk swaps as the cables appear to be very brittle/easy to damage. Thanks again to JorgeB for all their help with this issue!

Newconfig, preserved all disk assignments, array is started and parity is syncing. Thanks for your help throughout this whole process, greatly appreciated. I'll follow up once sync is completed and post a summary for those who run into the same/similar in the future.

Diags attached

Alright, I've got the xfs-repair -L completed on disk 4, stopped the array, started it in regular mode (after being in maintenance mode to complete xfs-repair). Disk 4 is now properly being emulated. I think.... when I mount the physical drive as an unassigned device, its showing 13.1/14tb in use, but the emulated drive is showing 6.33tb/14tb used. I haven't compared the disk contents to the emulated contents but this has me a little alarmed. Disk6 was being properly emulated after a reboot of the system after it first went disabled. So at this point I am left with 2 disks in an disabled/emulated state. Both disks appear to be fine and likely were put into a disabled state because of either damaged sata cables or failing HBAs (both have been replaced with spares i had on-hand). What would be my best course of action to get these disks back into the array? Am I forced to copy the contents of these disks to *somewhere else* (rest of the array, other storage media, etc) prior to rebuilding both at the same time via parity check? Is there a rebuild option that just verifies the contents of the drive without wiping and rebuilding from parity? Thanks so much again for your help thus far. I don't think I need to upload a diag, but if so, let me know.

Ended up stopping array and shutting down via CLI. Will update on the xfs-repair.

The second disabled disk is showing in unassigned devices (with the mount button greyed out with "ARRAY" in its place. I don't know if this happened with disk4 when it was set to disabled. Additionally, I appear to be unable to stop the array. The button gives me the typical prompt about being sure i want to stop the array, which i proceed with. But then the array does not appear to be wanting to stop. The syslogs show nothing added to indicate an error or anything. I've refreshed the browser, verified webGUI is response, etc. The bottom left corner just shows "Array started".

The fun never ends! I ran xfs-repair -n at first, it gave a whole long log that i can upload if needed. Then I ran xfs-repair with no other inputs, which failed due to a log to replay existing. I reran with -L to ignore said log (the doc you linked suggests that it was safe tod o so). That ran and seemed to fix most everything up, but ended with error 117 and a line about re-running it. HOWEVER at the same time I ran this, another disk entered disabled state (disk6). I then re-ran xfs-repair -n on disk4 which came back looking clean, but disk4 is not able to be accessed from emulated content. I have gathered a new diag export. I'm currently away from the device but i have strong suspicions that one or more of my breakout cables from my HBAs are damaged. I have had issues with them in the past, and the chassis I currently use does not have hotswap bays so drive replacements/additions are a PITA and it is quite easy to accidentally pinch or bend one of the breakout sata cables while swapping disks around (rosewill LSV-4500 chassis). I'm going to power down the system, trace the failing disks back to their respective breakout cables/HBAs to see if there is any commonality. Then I will swap in a different single HBA (lsi-9300-16i) to replace the two lsi-92xx-8i HBAs and their assorted breakout cables. This way I will be using a known-good HBA with completely brand new breakout cables. After that is complete, my plan is to mount the array again in maintenance mode, re-run xfs-repair -L on disk4, and see if it is able to be accessed via emulation. That is, unless there are any other suggetions of what I should do first prior to those actions.

Diags attached.

Extended test succeeded on disk5, no *new* errors. Array is still showing the disable disk (disk4) as unmountable when being emulated, so i cannot access the emulated contents. Is there any way to resolve this or does it mean the parity needs to be rebuilt once i have re-added disk 4 into the array? I'm anticipating disk4 will get wiped so im just copying the data from it onto another storage device right now, might take some time.

So an interesting wrinkle to this - while running the extended self test on disk 5, my system appears to have hard locked and crashed. I was unable to get any sort of response or image output. No webgui response. IPMI would load, but the remote console would not display anything. Ended up having to force-shutdown the machine. This happened with approximately 2-3 hours left (estimated) on the extended self test. I've gathered a new set of diagnostics but I don't think they will be of much value since they will have come after the crash and likely do not contain anything useful. I also have the smart report from disk5 post-crash. I note that it lists 2 errors that have occurred. The first error was almost 2 months ago (i dont think i noticed any detrimental effects at the time). The second error was ~55 hours ago, which is something like 12 hours after disk4 went offline (i had probably started to investigate via webgui around that timeframe, possibly had stopped the array and restarted it after). I've restarted the long test on disk5. The array is not currently running as it wants to initiate a parity check when I do so because of the unclean shutdown.

Thanks, I've got the extended test running on 5, will report back with diags once complete and array restarted.

I've stopped the array, shut down the system, moved the affected disk to a different, known-good cable. Booted server back up, started array. Disk4 now shows "Unmountable". I stopped the array, removed disk4 from the array. I mounted it using unassigned devices, and was able to browse its files. SMART short test comes back clean. Smart report seems ok to me. I suspect that because this device went offline at the start of a parity check, that it has since caused unraid to generate a new parity that is un-usable. My plan is to copy all of the data off this disk to somewhere else as a backup, then re-assign the device into the array, and force rebuild overtop of it. I expect what will happen is that the disk will have all usable data wiped and be left unformatted, requiring me to wipe it and rebuild parity again. Once that is complete, I should be able to copy data back onto the drive. Open to any thoughts/feedback on this.

The parity check completed, reporting the following results: Last check completed on Mon 6 Jan 2025 02:08:13 PM CST (today) Duration: 1 day, 12 hours, 8 minutes, 12 seconds. Average speed: 107.6 MB/s Finding 1902608354 errors

Sorry I just realized I left out some useful info - The parity check started at 2am on January 5th. Immediately after it started, disk4 was set to disabled due to read errors. When i look through the diag file, I can see that there were read errors, and then later on, write errors to that disk. I run nxlog as a docker container to forward syslogs from unraid (and other systems) to. That docker is hosted on the unraid server, which is where we see the errors from later on about being unable to contact a server on port 514 - that container was not able to successfully restart (likely a file dependency on the missing disk). This unraid is running on a supermicro Supermicro X9DRH with both sockets popualted with an intel e2670-v2. Drives are connected via 2 HBA cards (there are 16 drives connected total, 3 of them are unassigned spares and 1 is a sata SSD). I am suspecting possibly a failed cable or loose connection, but was looking for thoughts/feedback prior to shutting the server down, checking, then attempting to rebuild.

diags attached