gabemcg

Ok thank you. I will try this

They share a 4 drive backplane cage which is powered by 2 molex (off of a splitter) the entire backplane cage and splitter and power//data cables were recently replaced to rule out similar issues I had a few months ago with drives in these slots. I ended up leaving them empty at the time but perhaps falsely assumed they had been resolved. I can move the parity drives back to their previous positions on the onboard controller or put two drives from then 24 bay disk shelf on the onboard controller and put the parity drives in the disk shelf, but I want to be very careful about order of operations to avoid any data loss or loss of parity if possible. I do believe all the drives are healthy and the issue is related to signal integrity, but I can't say for certain that multiple hard resets during parity checks with docker running did not create issues with the parity. Appreciate your help!

Hello-- I was recently having issues with my motherboard's onboard sata controllers. (some data drives were showing link degradation and dropping link speed. My system was also experiencing hard reboots with increasing frequency which may have been memory related (I was pushing my onboard memory controller with 3x64gb DDR5 ECC UDIMMs, when I had it powered down I eased it back to 2x64. part of the issue was every time it would hard reboot it would come back up and start a new parity check which stressed the already struggling SATA controller situation so I couldn't complete a parity check before another hard reboot. both of my parity drives were on the one of the motherboards onboard sata controllers (there is a ASM1064 and a PCH controller) they were on the ASM1064 and the drives with issues were on the PCH. I decided to move both of the drives having issues and the two parity drives to connect via my HBA instead. I powered down, moved the drives, and when I powered up it the two data drives I'd moved to the HBA connected ports came up fine, and the two parity drives came up as disabled due to errors. I have attached the diagnostics and have not done anything else to this point. I'm about to go to bed and my plan is to stop docker and VMs to minimize any writes/reads to the unprotected array and check back here for further guidance, which is much appreciateed! mother-diagnostics-20260615-0059.zip

Thank you for your help!

looks like that fixed it. Now to figure out how to catalog what was in the ~5tb of data I lost... mother-diagnostics-20260331-1439.zip

Thanks, it found filesystem corruption. Should I click "Fix"? xfs_repair status.txt

Rebuild on disk8 and disk9 at the same time was successful. Array is back online, and appears healthy. However in the logs I'm seeing a repeated metadata error on what I believe is disk9 Updated diagnostics attached. Log error: Mar 31 05:39:06 mother kernel: XFS (md9p1): Metadata CRC error detected at xfs_dir3_block_read_verify+0x7e/0x100, xfs_dir3_block block 0x280000048 Mar 31 05:39:06 mother kernel: XFS (md9p1): Unmount and run xfs_repair Mar 31 05:39:06 mother kernel: XFS (md9p1): First 128 bytes of corrupted metadata buffer: Mar 31 05:39:06 mother kernel: 00000000: 90 60 d3 a2 cd 3b 61 2a 3f 00 00 bf aa 20 38 e4 .`...;a*?.... 8. Mar 31 05:39:06 mother kernel: 00000010: 6d bd 00 b6 4c 00 11 7f 37 4d 54 77 d4 33 83 1c m...L...7MTw.3.. Mar 31 05:39:06 mother kernel: 00000020: 88 0b ad aa e9 ca e2 e9 00 00 00 02 7d f4 15 98 ............}... Mar 31 05:39:06 mother kernel: 00000030: 6b b8 00 32 fd f4 15 3f 49 79 66 ce bd f6 01 3f k..2...?Iyf....? Mar 31 05:39:06 mother kernel: 00000040: 62 73 0d 4d 12 4f 89 a7 01 2e 02 00 02 67 00 40 bs.M.O.......g.@ Mar 31 05:39:06 mother kernel: 00000050: 00 00 00 00 00 00 00 a1 d2 2e 2e 02 00 00 00 ed ................ Mar 31 05:39:06 mother kernel: 00000060: 00 00 00 03 00 00 00 a6 d9 4a a6 f3 38 59 75 7d .........J..8Yu} Mar 31 05:39:06 mother kernel: 00000070: 72 69 63 61 a0 73 20 28 32 30 31 33 29 20 53 65 rica.s (2013) Se mother-diagnostics-20260331-0637.zip

So it seems I have two choices, appreciate any thoughts on which is best 1. Rebuilding Drive 9 onto itself, 2. New Config and rebuild parity At what point should I add the "replacement" Disk 8 back into the array? (this is the original disk 8 that I used UFS Explorer to extract some of the data from) It will need to be cleaned and then I'll copy the recovered data back to the array. I'm not concerned with preserving any of the failed writes that may have happened to disable disk 9 as they probably happened only in the minute or two the array was trying to start before I shut down. Either way the process will take a long time, and I think it would be best to wait for Disk 9 to be back online with valid parity before I try and copy the data from Disk 8 back to the array from the recovery PC. Thanks for any suggestions!

So this initial issue was caused by what I thought was a failed backplane, but now I suspect one of the onboard SFF-8643 ports is having conflict issues communicating over the crowded PCIe bus. I replaced the entire 4-drive SATA backplane/cage and installed the 2 drives that were operating fine previously but in the slots that had been giving me issues in the old cage and started to see similar issues immediately (upon boot the array was taking a long time in "starting" status) so I shut down and disconnected that cage from the SFF-8643 port and rewired it to the internal port of my HBA. I booted back up and the drive errors were gone, but now one of the remaining two drives in this cage is showing as disabled (Disk 9). I don't believe there is any issue with the disk or the current backplane or connection to the HBA. I believe it entered this state due to the failed Array start and shutdown when it was hooked up to the SFF-8643 port. I assume the correct course of action is to do a new configuration and check "parity is good". I have a feeling I'll need to install and assign a disk to the Disk 8 slot as well before I do this right? updated diagnostics before I make any further changes. appreciate the help! On a side note the recovery of Disk 8 with UFS Explorer found 7.43TB of data and is recovering on a separate system now. mother-diagnostics-20260328-1016.zip

running the scan now, thanks!

As I feared. Claude is very sorry for leading me down this destructive path at least. Is there consensus on trying the UFS Explorer on the physical drive to see what can be recovered? Any next steps on this path would be appreciated (AI has lost my trust). Thank you, gabe

Thank you JorgeB - Updated diagnostics attached after completing your procedure. mother-diagnostics-20260326-1128.zip

mother-diagnostics-20260325-2321.zip Attached, thank you!

Somehow I let what should have been a simple rebuild get away from me and rather than stop and post here first I foolishly trusted Claude Opus 4.6 to try and fix the problem. Below is its summary of want it had me try, which I think has resulted in data loss. Appreciate any help the community can provide in recovering any data from my drive TL;DRBad backplane slot caused two disks (Disk 8 and Disk 10) to throw SATA link errors. During recovery, Disk 8 was rebuilt while Disk 3 was disabled/absent, producing a corrupt rebuild. Attempts to fix via New Config and xfs_repair destroyed the XFS metadata on the emulated Disk 8. Raw file data blocks are confirmed still present on the emulated device, but the filesystem index trees are gone. Dual parity, 28 data disks, ~12.4 TB of data on Disk 8 including irreplaceable photos, videos, and documents. Diagnostic file: https://drive.google.com/file/d/1uXJcs0d84-5kCR-PITG4yUgwhpT8SiJN/view?usp=drive_link HardwareUnraid 7.2.4 2x 24 TB parity drives (ST24000DM001) 28 data disks (mixed 12–24 TB) 5x WD Black SN7100 4 TB NVMe cache pool Motherboard has 3x SFF-8643 ports; ports 1–2 are native, port 3 goes through an ASM1164 chip 4-bay hot-swap backplane cage connected via SFF-8643 breakout cable to one of the motherboard ports Disk 8: WDC WD140EDFZ-11A0VA0 (serial 9MG7WRXJ) — 14 TB Timeline of EventsMarch 19 — Initial FailureAt 04:40 AM, Disk 8 (ata1) began throwing rapid SATA link errors: ata1.00: exception Emask 0x10 SAct 0x38000 SErr 0x4890000 action 0xe frozen ata1.00: irq_stat 0x0c400040, interface fatal error, connection status changed ata1: SError: { PHYRdyChg 10B8B LinkSeq DevExch }Nine minutes later, Disk 10 (ata3) started throwing identical errors. Both drives renegotiated at 1.5 Gbps instead of 6.0 Gbps. By 06:01, ata1 went fully offline (SATA link down → disable device). ata3 continued throwing sporadic errors through March 21. Error counts: ~1,721 on ata1, ~743 on ata3. Zero UNC or reallocated sector errors — all physical-layer SATA link instability (PHYRdyChg, 10B8B, DevExch). March 21–23 — DiagnosisInvestigation determined ata2 and ata4 (the other two ports on the same SFF-8643 breakout and same backplane) were completely clean. Replaced the breakout cable and molex Y-splitter — errors persisted on the same two ports. Swapped the suspect drives into the good backplane slots — errors followed the slots, not the drives. Conclusion: two bad connectors/traces on the backplane. March 23 — First Rebuild Attempt (Bad Slot)Before the backplane was identified as the root cause, I attempted to rebuild Disk 8: Stopped array, unassigned Disk 8, started array, stopped again, reassigned same drive, started to trigger rebuild Rebuild ran overnight in the bad backplane slot, throwing continuous errors, not progressing Canceled the rebuild after ~7 hours March 23 — Second Rebuild Attempt (Disk 3 Absent)Moved the drives out of the bad backplane slots into an external disk shelf. During this process, Disk 3 was inadvertently removed/disabled. I then ran a second rebuild of Disk 8, believing dual parity would compensate for Disk 3's absence during the rebuild. This assumption was wrong — Unraid's rebuild uses a simple XOR of Parity 1 and all other present data disks; it does not use Parity 2 to fill in for a second missing disk. The rebuild completed (March 25, 19:16) with 0 I/O errors but the resulting data was mathematically incorrect — every block was the true Disk 8 data XORed with Disk 3's data. Disk 8 showed only 268 GB used instead of the expected ~12.4 TB. March 25 — New Config AttemptUsed Tools → New Config (Preserve All) to restore Disk 3 and fix the array: First attempt: Unassigned Disk 8 before starting the array, intending to add it back afterward for rebuild Started the array — Disk 8 was registered as a new disk since it wasn't in the config When Disk 8 was assigned and array started, Unraid initiated a Disk-Clear (zeroing) instead of a Data-Rebuild Canceled the Disk-Clear after ~1 minute / 17 GB / 0.1% The Disk-Clear zeroed the first ~17 GB of Disk 8, which included the XFS primary superblock, root inode, AG 0 metadata, and the beginning of the journal log. March 25 — XFS Repair AttemptsWith Disk 8 showing "Not installed" / "Unmountable: unsupported or no file system", attempted recovery: Confirmed the emulated block device /dev/md8p1 existed Verified the first block was all zeros (from Disk-Clear) but data existed deeper on the device xfs_repair -n found and verified a backup superblock at AG 1 Write test confirmed reads/writes to the emulated device worked correctly (md5sum round-trip) Ran xfs_repair /dev/md8p1 — it found the backup superblock, rebuilt metadata, moved files to lost+found, but failed at Phase 7 with: Maximum metadata LSN ahead of log. Format log to cycle X and fatal error -- File system metadata writeout failed, err=117 Ran xfs_repair -L /dev/md8p1 to zero the log and retry — this generated a new UUID, saw all AG headers as having bad UUIDs (from the first repair's different UUID), reset all 13 AGs, rebuilt empty trees, and completed Mounted the filesystem — completely empty, 0 bytes used, no lost+found Current StateArray is running with 29 disks (Disk 8 showing "Not installed") Disk 3 is back and healthy with all its data Both parity drives are present — but parity has been modified by the xfs_repair writes to the emulated device /dev/md8p1 exists and contains the emulated Disk 8 Raw file data blocks are confirmed present beyond the zeroed region (hexdump shows non-zero data at various offsets) XFS metadata (AG headers, inode btrees, free space btrees) has been overwritten by the two xfs_repair runs with empty/rebuilt structures The filesystem mounts but shows 0 bytes / no files mdcmd status shows rdevStatus.8=DISK_NP_DSBL and rdevId.8= (empty) What I Need Help WithThe raw data blocks for ~12.4 TB of files (including irreplaceable photos, videos, and documents) are still on the emulated device. The XFS metadata maps that point to those blocks have been destroyed. Questions: Is there a way to recover the XFS inode blocks that may still exist on the device? xfs_repair rebuilt the index trees but shouldn't have zeroed the actual inode blocks themselves, which contain per-file extent maps and filenames. Would data carving tools (photorec, testdisk) be the best approach for the irreplaceable files? Is there any way to reconstruct the original parity state, given that the only modifications to parity came from (a) the brief Disk-Clear (~17 GB) and (b) the two xfs_repair runs? Any other recovery paths I'm not considering? The emulated device is currently unmounted and I'm not writing to it. Happy to provide any additional diagnostics.