Somehow I let what should have been a simple rebuild get away from me and rather than stop and post here first I foolishly trusted Claude Opus 4.6 to try and fix the problem. Below is its summary of want it had me try, which I think has resulted in data loss. Appreciate any help the community can provide in recovering any data from my drive

TL;DR

Bad backplane slot caused two disks (Disk 8 and Disk 10) to throw SATA link errors. During recovery, Disk 8 was rebuilt while Disk 3 was disabled/absent, producing a corrupt rebuild. Attempts to fix via New Config and xfs_repair destroyed the XFS metadata on the emulated Disk 8. Raw file data blocks are confirmed still present on the emulated device, but the filesystem index trees are gone. Dual parity, 28 data disks, ~12.4 TB of data on Disk 8 including irreplaceable photos, videos, and documents.

Diagnostic file: https://drive.google.com/file/d/1uXJcs0d84-5kCR-PITG4yUgwhpT8SiJN/view?usp=drive_link

---

Hardware

• Unraid 7.2.4

• 2x 24 TB parity drives (ST24000DM001)

• 28 data disks (mixed 12–24 TB)

• 5x WD Black SN7100 4 TB NVMe cache pool

• Motherboard has 3x SFF-8643 ports; ports 1–2 are native, port 3 goes through an ASM1164 chip

• 4-bay hot-swap backplane cage connected via SFF-8643 breakout cable to one of the motherboard ports

• Disk 8: WDC WD140EDFZ-11A0VA0 (serial 9MG7WRXJ) — 14 TB

---

Timeline of Events

March 19 — Initial Failure

At 04:40 AM, Disk 8 (ata1) began throwing rapid SATA link errors:

ata1.00: exception Emask 0x10 SAct 0x38000 SErr 0x4890000 action 0xe frozen
ata1.00: irq_stat 0x0c400040, interface fatal error, connection status changed
ata1: SError: { PHYRdyChg 10B8B LinkSeq DevExch }

Nine minutes later, Disk 10 (ata3) started throwing identical errors. Both drives renegotiated at 1.5 Gbps instead of 6.0 Gbps. By 06:01, ata1 went fully offline (SATA link down → disable device). ata3 continued throwing sporadic errors through March 21.

Error counts: ~1,721 on ata1, ~743 on ata3. Zero UNC or reallocated sector errors — all physical-layer SATA link instability (PHYRdyChg, 10B8B, DevExch).

March 21–23 — Diagnosis

Investigation determined ata2 and ata4 (the other two ports on the same SFF-8643 breakout and same backplane) were completely clean. Replaced the breakout cable and molex Y-splitter — errors persisted on the same two ports. Swapped the suspect drives into the good backplane slots — errors followed the slots, not the drives. Conclusion: two bad connectors/traces on the backplane.

March 23 — First Rebuild Attempt (Bad Slot)

Before the backplane was identified as the root cause, I attempted to rebuild Disk 8:

• Stopped array, unassigned Disk 8, started array, stopped again, reassigned same drive, started to trigger rebuild

• Rebuild ran overnight in the bad backplane slot, throwing continuous errors, not progressing

• Canceled the rebuild after ~7 hours

March 23 — Second Rebuild Attempt (Disk 3 Absent)

Moved the drives out of the bad backplane slots into an external disk shelf. During this process, Disk 3 was inadvertently removed/disabled.

I then ran a second rebuild of Disk 8, believing dual parity would compensate for Disk 3's absence during the rebuild. This assumption was wrong — Unraid's rebuild uses a simple XOR of Parity 1 and all other present data disks; it does not use Parity 2 to fill in for a second missing disk.

The rebuild completed (March 25, 19:16) with 0 I/O errors but the resulting data was mathematically incorrect — every block was the true Disk 8 data XORed with Disk 3's data. Disk 8 showed only 268 GB used instead of the expected ~12.4 TB.

March 25 — New Config Attempt

Used Tools → New Config (Preserve All) to restore Disk 3 and fix the array:

1. First attempt: Unassigned Disk 8 before starting the array, intending to add it back afterward for rebuild

2. Started the array — Disk 8 was registered as a new disk since it wasn't in the config

3. When Disk 8 was assigned and array started, Unraid initiated a Disk-Clear (zeroing) instead of a Data-Rebuild

4. Canceled the Disk-Clear after ~1 minute / 17 GB / 0.1%

The Disk-Clear zeroed the first ~17 GB of Disk 8, which included the XFS primary superblock, root inode, AG 0 metadata, and the beginning of the journal log.

March 25 — XFS Repair Attempts

With Disk 8 showing "Not installed" / "Unmountable: unsupported or no file system", attempted recovery:

1. Confirmed the emulated block device /dev/md8p1 existed

2. Verified the first block was all zeros (from Disk-Clear) but data existed deeper on the device

3. xfs_repair -n found and verified a backup superblock at AG 1

4. Write test confirmed reads/writes to the emulated device worked correctly (md5sum round-trip)

5. Ran xfs_repair /dev/md8p1 — it found the backup superblock, rebuilt metadata, moved files to lost+found, but failed at Phase 7 with: Maximum metadata LSN ahead of log. Format log to cycle X and fatal error -- File system metadata writeout failed, err=117

6. Ran xfs_repair -L /dev/md8p1 to zero the log and retry — this generated a new UUID, saw all AG headers as having bad UUIDs (from the first repair's different UUID), reset all 13 AGs, rebuilt empty trees, and completed

7. Mounted the filesystem — completely empty, 0 bytes used, no lost+found

Current State

• Array is running with 29 disks (Disk 8 showing "Not installed")

• Disk 3 is back and healthy with all its data

• Both parity drives are present — but parity has been modified by the xfs_repair writes to the emulated device

• /dev/md8p1 exists and contains the emulated Disk 8

• Raw file data blocks are confirmed present beyond the zeroed region (hexdump shows non-zero data at various offsets)

• XFS metadata (AG headers, inode btrees, free space btrees) has been overwritten by the two xfs_repair runs with empty/rebuilt structures

• The filesystem mounts but shows 0 bytes / no files

• mdcmd status shows rdevStatus.8=DISK_NP_DSBL and rdevId.8= (empty)

What I Need Help With

The raw data blocks for ~12.4 TB of files (including irreplaceable photos, videos, and documents) are still on the emulated device. The XFS metadata maps that point to those blocks have been destroyed.

Questions:

1. Is there a way to recover the XFS inode blocks that may still exist on the device? xfs_repair rebuilt the index trees but shouldn't have zeroed the actual inode blocks themselves, which contain per-file extent maps and filenames.

2. Would data carving tools (photorec, testdisk) be the best approach for the irreplaceable files?

3. Is there any way to reconstruct the original parity state, given that the only modifications to parity came from (a) the brief Disk-Clear (~17 GB) and (b) the two xfs_repair runs?

4. Any other recovery paths I'm not considering?

The emulated device is currently unmounted and I'm not writing to it. Happy to provide any additional diagnostics.

3 hours ago, gabemcg said:

Diagnostic file: https://drive.google.com/file/d/1uXJcs0d84-5kCR-PITG4yUgwhpT8SiJN/view?usp=drive_link

This is not working; please attach the diags here.

mother-diagnostics-20260325-2321.zip

3 hours ago, JorgeB said:

This is not working; please attach the diags here.

Attached, thank you!

9 hours ago, gabemcg said:

I then ran a second rebuild of Disk 8, believing dual parity would compensate for Disk 3's absence during the rebuild. This assumption was wrong — Unraid's rebuild uses a simple XOR of Parity 1 and all other present data disks; it does not use Parity 2 to fill in for a second missing disk.

FYI, this is incorrect, with dual parity, you can rebuild any two missing/disabled disks.

9 hours ago, gabemcg said:

The rebuild completed (March 25, 19:16) with 0 I/O errors but the resulting data was mathematically incorrect — every block was the true Disk 8 data XORed with Disk 3's data. Disk 8 showed only 268 GB used instead of the expected ~12.4 TB.

This doesn't make any sense.

9 hours ago, gabemcg said:

New Config Attempt

Used Tools → New Config (Preserve All) to restore Disk 3 and fix the array:

1. First attempt: Unassigned Disk 8 before starting the array, intending to add it back afterward for rebuild

2. Started the array — Disk 8 was registered as a new disk since it wasn't in the config

3. When Disk 8 was assigned and array started, Unraid initiated a Disk-Clear (zeroing) instead of a Data-Rebuild

4. Canceled the Disk-Clear after ~1 minute / 17 GB / 0.1%

This should have never been done like this. New config is not used to rebuild disks; it can be used in some specific situations but needs to follow a specific procedure to force emulated the disk(s) you want to rebuild.

9 hours ago, gabemcg said:

March 25 — XFS Repair Attempts

With Disk 8 showing "Not installed" / "Unmountable: unsupported or no file system", attempted recovery:

1. Confirmed the emulated block device /dev/md8p1 existed

2. Verified the first block was all zeros (from Disk-Clear) but data existed deeper on the device

3. xfs_repair -n found and verified a backup superblock at AG 1

4. Write test confirmed reads/writes to the emulated device worked correctly (md5sum round-trip)

5. Ran xfs_repair /dev/md8p1 — it found the backup superblock, rebuilt metadata, moved files to lost+found, but failed at Phase 7 with: Maximum metadata LSN ahead of log. Format log to cycle X and fatal error -- File system metadata writeout failed, err=117

6. Ran xfs_repair -L /dev/md8p1 to zero the log and retry — this generated a new UUID, saw all AG headers as having bad UUIDs (from the first repair's different UUID), reset all 13 AGs, rebuilt empty trees, and completed

7. Mounted the filesystem — completely empty, 0 bytes used, no lost+found

Since most of the disk was cleared, it's not surprising xfs_reapair was unable to recover any usable data.

After all this, not sure there's much that can be done, but there's one thing you can try. It will only work if parity is still valid, or at least mostly valid, but it won't hurt to try:

-Tools -> New Config -> Preserve current assignments: All -> Apply

-Check all assignments and assign any missing disk(s) if needed, including disk 8. A physical disk needs to be assigned to this slot, it can be the old disk or a new one. If it is a new one, it should be same size or larger than the old one

-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)

-Stop array

-Unassign disk8

-Start array (in normal mode now), and post new diagnostics

2 hours ago, JorgeB said:

FYI, this is incorrect, with dual parity, you can rebuild any two missing/disabled disks.

This doesn't make any sense.

This should have never been done like this. New config is not used to rebuild disks; it can be used in some specific situations but needs to follow a specific procedure to force emulated the disk(s) you want to rebuild.

Since most of the disk was cleared, it's not surprising xfs_reapair was unable to recover any usable data.

After all this, not sure there's much that can be done, but there's one thing you can try. It will only work if parity is still valid, or at least mostly valid, but it won't hurt to try:

-Tools -> New Config -> Preserve current assignments: All -> Apply

-Check all assignments and assign any missing disk(s) if needed, including disk 8. A physical disk needs to be assigned to this slot, it can be the old disk or a new one. If it is a new one, it should be same size or larger than the old one

-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)

-Stop array

-Unassign disk8

-Start array (in normal mode now), and post new diagnostics

Thank you JorgeB - Updated diagnostics attached after completing your procedure.

mother-diagnostics-20260326-1128.zip

Emulated disk8 is mounted but empty.

Maybe UFS Explorer can recover something

Yep, so basically parity is in sync with the current disk8 status, i.e., an empty disk.

7 minutes ago, JorgeB said:

Yep, so basically parity is in sync with the current disk8 status, i.e., an empty disk.

As I feared. Claude is very sorry for leading me down this destructive path at least.

Is there consensus on trying the UFS Explorer on the physical drive to see what can be recovered? Any next steps on this path would be appreciated (AI has lost my trust).

Thank you,

gabe

It's worth a try, the free trial should show what it can recover, then you can decide if it's worth the $60

2 hours ago, JorgeB said:

It's worth a try, the free trial should show what it can recover, then you can decide if it's worth the $60

running the scan now, thanks!

So this initial issue was caused by what I thought was a failed backplane, but now I suspect one of the onboard SFF-8643 ports is having conflict issues communicating over the crowded PCIe bus. I replaced the entire 4-drive SATA backplane/cage and installed the 2 drives that were operating fine previously but in the slots that had been giving me issues in the old cage and started to see similar issues immediately (upon boot the array was taking a long time in "starting" status) so I shut down and disconnected that cage from the SFF-8643 port and rewired it to the internal port of my HBA. I booted back up and the drive errors were gone, but now one of the remaining two drives in this cage is showing as disabled (Disk 9).

I don't believe there is any issue with the disk or the current backplane or connection to the HBA. I believe it entered this state due to the failed Array start and shutdown when it was hooked up to the SFF-8643 port. I assume the correct course of action is to do a new configuration and check "parity is good". I have a feeling I'll need to install and assign a disk to the Disk 8 slot as well before I do this right? updated diagnostics before I make any further changes. appreciate the help!

On a side note the recovery of Disk 8 with UFS Explorer found 7.43TB of data and is recovering on a separate system now.

mother-diagnostics-20260328-1016.zip

Edited March 28Mar 28 by gabemcg

33 minutes ago, gabemcg said:

I believe it entered this state

Not entirely clear what happened, except for the fact that the drive was disabled because a write to it failed. This is the only thing that disables a disk because it is the only thing that makes it out-of-sync with parity. The failed write that disables the disk, and any subsequent write to that (now emulated) disk, updates parity so those writes can be recovered by rebuild.

Sometimes if a read fails Unraid will try to write the calculated data back to the disk, and if that write fails the disk is disabled. But ultimately, a write failed so the disk was disabled.

Emulated disk9 is mounted, and SMART for the Unassigned Device that was disk9 look OK. It hasn't had an extended test.

Should be OK to rebuild on top if you get the problem that caused this sorted out.

If you decide to New Config instead, parity is not valid because the disabled disk is out-of-sync.

8 hours ago, trurl said:

If you decide to New Config instead, parity is not valid because the disabled disk is out-of-sync.

So it seems I have two choices, appreciate any thoughts on which is best

1. Rebuilding Drive 9 onto itself,
2. New Config and rebuild parity

At what point should I add the "replacement" Disk 8 back into the array? (this is the original disk 8 that I used UFS Explorer to extract some of the data from) It will need to be cleaned and then I'll copy the recovered data back to the array.

I'm not concerned with preserving any of the failed writes that may have happened to disable disk 9 as they probably happened only in the minute or two the array was trying to start before I shut down. Either way the process will take a long time, and I think it would be best to wait for Disk 9 to be back online with valid parity before I try and copy the data from Disk 8 back to the array from the recovery PC.

Thanks for any suggestions!

Edited March 29Mar 29 by gabemcg

1 hour ago, gabemcg said:

add the "replacement" Disk 8 back into the array?

You can rebuild disk8 and disk9 at the same time. Since emulated disk8 is already an empty XFS disk that is what rebuild will give you. Then you can copy the recovered data back to it.

On 3/28/2026 at 9:21 PM, trurl said:

You can rebuild disk8 and disk9 at the same time. Since emulated disk8 is already an empty XFS disk that is what rebuild will give you. Then you can copy the recovered data back to it.

Rebuild on disk8 and disk9 at the same time was successful. Array is back online, and appears healthy. However in the logs I'm seeing a repeated metadata error on what I believe is disk9

Updated diagnostics attached.

Log error:

Mar 31 05:39:06 mother kernel: XFS (md9p1): Metadata CRC error detected at xfs_dir3_block_read_verify+0x7e/0x100, xfs_dir3_block block 0x280000048

Mar 31 05:39:06 mother kernel: XFS (md9p1): Unmount and run xfs_repair

Mar 31 05:39:06 mother kernel: XFS (md9p1): First 128 bytes of corrupted metadata buffer:

Mar 31 05:39:06 mother kernel: 00000000: 90 60 d3 a2 cd 3b 61 2a 3f 00 00 bf aa 20 38 e4 .`...;a*?.... 8.

Mar 31 05:39:06 mother kernel: 00000010: 6d bd 00 b6 4c 00 11 7f 37 4d 54 77 d4 33 83 1c m...L...7MTw.3..

Mar 31 05:39:06 mother kernel: 00000020: 88 0b ad aa e9 ca e2 e9 00 00 00 02 7d f4 15 98 ............}...

Mar 31 05:39:06 mother kernel: 00000030: 6b b8 00 32 fd f4 15 3f 49 79 66 ce bd f6 01 3f k..2...?Iyf....?

Mar 31 05:39:06 mother kernel: 00000040: 62 73 0d 4d 12 4f 89 a7 01 2e 02 00 02 67 00 40 bs.M.O.......g.@

Mar 31 05:39:06 mother kernel: 00000050: 00 00 00 00 00 00 00 a1 d2 2e 2e 02 00 00 00 ed ................

Mar 31 05:39:06 mother kernel: 00000060: 00 00 00 03 00 00 00 a6 d9 4a a6 f3 38 59 75 7d .........J..8Yu}

Mar 31 05:39:06 mother kernel: 00000070: 72 69 63 61 a0 73 20 28 32 30 31 33 29 20 53 65 rica.s (2013) Se

mother-diagnostics-20260331-0637.zip

Check filesystem on that disk.

2 hours ago, JorgeB said:

Check filesystem on that disk.

Thanks, it found filesystem corruption. Should I click "Fix"?

xfs_repair status.txt

Yep

Then post new diagnostics after restarting the array in normal (not maintenance) mode.

35 minutes ago, trurl said:

Then post new diagnostics after restarting the array in normal (not maintenance) mode.

looks like that fixed it. Now to figure out how to catalog what was in the ~5tb of data I lost...

mother-diagnostics-20260331-1439.zip

Good fix! Nothing got put in lost+found for that disk. Sometimes fixing "forgets" folders and filenames for some files.

2 hours ago, trurl said:

Good fix! Nothing got put in lost+found for that disk. Sometimes fixing "forgets" folders and filenames for some files.Good fix! Nothing got put in lost+found for that disk. Sometimes fixing "forgets" folders and filenames for some files.

Thank you for your help!

Edited April 1Apr 1 by gabemcg