I recently replaced one of my old 6TB drives that was showing a few errors with a 14TB drive and during the data rebuild a second drive started throwing millions of errors. After the rebuild I shut everything down to check connections but now the second failed drive cannot be read at all.

 

I have a few questions about the current state of my data after this experience.
 

1) Is the data on the first drive rebuilt still considered good based on the bit state across all of my other drives other than the one that failed during the rebuild?

2) Is there any way to get back my data on the second failed drive? I assume the answer here is no and it's lost without a second parity drive in place.

3) If the answer to #2 is no, would my next steps be to replace the failed drive without a data rebuild somehow, and then rebuild parity data from there once everything is stable once again?

2 hours ago, sidezero said:

After the rebuild I shut everything down

Did you save the diagnostics before shutting down?

I unfortunately did not in this instance.

If the rebuild continued with read errors, and assuming no parity2, the rebuilt disk will likely have corruption, post current diags after array start, but don't write anything else to the array.

You called it. The original rebuilt drive is unmountable as well. I do have the original drive still that I could throw back in for that data potentially. Diagnostics attached. Thank you for all the help.

 

 

orbit-diagnostics-20240605-1238.zip

Sorry forgot to ask, which was the disk that had the read errors during the rebuild? I see currently two unmountable disks, which one was replaced?

sdb is the drive that was originally replaced. Disk 9 is the one that is showing missing and is the one that failed during the rebuild of sdb.

You can try this, but it will only work if parity is still valid:

 

-Connect old disk16

-Tools -> New Config -> Retain current configuration: All -> Apply
-Check all assignments and assign any missing disk(s) if needed, including old disk 16 and a new disk9, new disk9 should be same size or larger than the old one
-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)
-Stop array
-Unassign disk9
-Start array (in normal mode now), and post new diags.

 

 

Ran through all of that and updated diag attached. Thanks again.

orbit-diagnostics-20240606-1758.zip

Now check filesystem on the emulated disk9, run it without -n

It doesn't appear that I can. Disk 9 is showing Unmountable after the last steps I took prior to the most recent diag.

Forgot to mention, first with the array stopped, click on that disk and set the fs to xfs, you should then be able to run it, without -n first, and only if it asks for it, use -L.

After setting the fs to xfs, should the array be re-started in maintenance mode or normal?

5 minutes ago, sidezero said:

After setting the fs to xfs, should the array be re-started in maintenance mode or normal?

It has to be started in maintenance mode to be able to run xfs_repair.

 

You restart it in normal mode after completing the repair.

Thank you.

 

xfs_repair log is below:

 

Phase 1 - find and verify superblock...
bad primary superblock - inconsistent filesystem geometry information !!!

attempting to find secondary superblock...
.found candidate secondary superblock...
verified secondary superblock...
writing modified primary superblock
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
out-of-order bno btree record 2 (6748572 2327) block 0/1
out-of-order bno btree record 3 (5362660 132532) block 0/1
out-of-order bno btree record 4 (4115676 31148) block 0/1
out-of-order bno btree record 6 (8126233 70) block 0/1
block (0,8126233-8126302) multiply claimed by bno space tree, state - 1
out-of-order bno btree record 11 (1060775 12557) block 0/1
out-of-order bno btree record 13 (18475539 6235) block 0/1
out-of-order bno btree record 15 (11267389 8288) block 0/1
out-of-order bno btree record 18 (26538296 2973704) block 0/1
out-of-order bno btree record 23 (63104919 5300) block 0/1
out-of-order bno btree record 25 (72231270 47347) block 0/1
out-of-order bno btree record 29 (83185556 7456) block 0/1
out-of-order bno btree record 34 (100843496 38056) block 0/1
out-of-order bno btree record 38 (23689307 61750) block 0/1
invalid start block 124128315 in record 41 of bno btree block 0/1
out-of-order bno btree record 42 (121173937 5887) block 0/1
out-of-order cnt btree record 8 (96564608 2094) block 0/2
out-of-order cnt btree record 12 (41000970 4425) block 0/2
out-of-order cnt btree record 15 (88332932 5159) block 0/2
out-of-order cnt btree record 17 (100743044 7148) block 0/2
invalid start block 123296326 in record 19 of cnt btree block 0/2
out-of-order cnt btree record 21 (68927866 10155) block 0/2
out-of-order cnt btree record 23 (18503749 14117) block 0/2
out-of-order cnt btree record 24 (82492973 14469) block 0/2
block (0,28085137-28088502) multiply claimed by cnt space tree, state - 2
invalid start block 133026027 in record 27 of cnt btree block 0/2
out-of-order cnt btree record 28 (101294130 18502) block 0/2
out-of-order cnt btree record 32 (9320005 70835) block 0/2
invalid start block 234338807 in record 35 of cnt btree block 0/2
invalid start block 216810126 in record 40 of cnt btree block 0/2
invalid start block 145965057 in record 41 of cnt btree block 0/2
agf_freeblks 6589319, counted 6468671 in ag 0
agf_longest 3071743, counted 4756377 in ag 0
inode chunk claims used block, inobt block - agno 0, bno 16, inopb 8
ir_freecount/free mismatch, inode chunk 0/66647872, freecount 19 nfree 29
inode chunk claims used block, finobt block - agno 0, bno 16, inopb 8
undiscovered finobt record, ino 192 (0/192)
finobt ir_freecount/free mismatch, inode chunk 0/192, freecount 12 nfree 14
undiscovered finobt record, ino 66647872 (0/66647872)
undiscovered finobt record, ino 67040000 (0/67040000)
undiscovered finobt record, ino 67106816 (0/67106816)
agi_freecount 32, counted 76 in ag 0
agi_freecount 32, counted 94 in ag 0 finobt
sb_icount 1024, counted 2048
sb_ifree 114, counted 628
sb_fdblocks 4762656, counted 29615893
root inode chunk not found
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
found inodes not in the inode allocation tree
found inodes not in the inode allocation tree
        - process known inodes and perform inode discovery...
        - agno = 0
1512067d7680: Badness in key lookup (length)
bp=(bno 0xc0, len 4096 bytes) key=(bno 0xc0, len 16384 bytes)
1512067d7680: Badness in key lookup (length)
bp=(bno 0xe0, len 4096 bytes) key=(bno 0xe0, len 16384 bytes)
1512067d7680: Badness in key lookup (length)
bp=(bno 0x3f8f740, len 4096 bytes) key=(bno 0x3f8f740, len 16384 bytes)
1512067d7680: Badness in key lookup (length)
bp=(bno 0x3f8f760, len 4096 bytes) key=(bno 0x3f8f760, len 16384 bytes)
1512067d7680: Badness in key lookup (length)
bp=(bno 0x3fef300, len 4096 bytes) key=(bno 0x3fef300, len 16384 bytes)
1512067d7680: Badness in key lookup (length)
bp=(bno 0x3fef320, len 4096 bytes) key=(bno 0x3fef320, len 16384 bytes)
1512067d7680: Badness in key lookup (length)
bp=(bno 0x3fff800, len 4096 bytes) key=(bno 0x3fff800, len 16384 bytes)
1512067d7680: Badness in key lookup (length)
bp=(bno 0x3fff820, len 4096 bytes) key=(bno 0x3fff820, len 16384 bytes)
data fork in ino 131 claims free block 15
data fork in ino 131 claims free block 14
data fork in ino 131 claims free block 13
data fork in ino 131 claims free block 12
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
data fork in ino 154 claims free block 13033146
data fork in ino 154 claims free block 37305503
data fork in ino 154 claims free block 37995151
data fork in ino 154 claims free block 61359149
data fork in ino 154 claims free block 71336771
data fork in ino 154 claims free block 71459262
correcting imap
data fork in ino 155 claims free block 7806721
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
data fork in ino 164 claims free block 7806724
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
data fork in ino 207 claims free block 116927339
data fork in ino 207 claims free block 117252879
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
data fork in ino 227 claims free block 13308957
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
data fork in ino 245 claims free block 11
correcting imap
correcting imap
correcting imap
correcting imap
data fork in ino 249 claims free block 32
data fork in ino 249 claims free block 1060775
data fork in ino 249 claims free block 4115676
correcting imap
data fork in ino 250 claims free block 5362660
correcting imap
correcting imap
data fork in ino 252 claims free block 6748572
correcting imap
data fork in ino 253 claims free block 7803443
data fork in ino 253 claims free block 8390412
data fork in ino 253 claims free block 8398494
data fork in ino 253 claims free block 8824247
data fork in ino 253 claims free block 9320005
data fork in ino 253 claims free block 11267389
correcting imap
imap claims a free inode 66647872 is in use, correcting imap and clearing inode
cleared inode 66647872
correcting imap
data fork in ino 66647875 claims free block 18475539
data fork in ino 66647875 claims free block 18503749
data fork in ino 66647875 claims free block 19122726
correcting imap
data fork in ino 66647877 claims free block 21826214
correcting imap
data fork in ino 66647878 claims free block 23689307
correcting imap
data fork in ino 66647880 claims free block 25404468
data fork in ino 66647880 claims free block 26538296
correcting imap
data fork in ino 66647881 claims free block 26854724
data fork in ino 66647881 claims free block 28084569
data fork in ino 66647881 claims free block 28108248
data fork in ino 66647881 claims free block 28951860
data fork in ino 66647881 claims free block 32130417
correcting imap
correcting imap
data fork in ino 66647887 claims free block 49535651
data fork in ino 66647887 claims free block 49790474
correcting imap
correcting imap
data fork in ino 66647895 claims free block 63104919
data fork in ino 66647895 claims free block 63475252
correcting imap
correcting imap
data fork in ino 66647897 claims free block 63897028
correcting imap
correcting imap
data fork in ino 66647901 claims free block 72798090
data fork in ino 66647901 claims free block 73346810
data fork in ino 66647901 claims free block 73407799
data fork in ino 66647901 claims free block 73415260
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
data fork in ino 66647909 claims free block 88332932
correcting imap
correcting imap
correcting imap
correcting imap
data fork in ino 66647915 claims free block 95889725
data fork in ino 66647915 claims free block 96564608
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
data fork in ino 66647924 claims free block 15633220
correcting imap
data fork in ino 66647925 claims free block 101308540
data fork in ino 66647925 claims free block 101318331
data fork in ino 66647925 claims free block 101696011
correcting imap
data fork in ino 66647926 claims free block 102407986
data fork in ino 66647926 claims free block 102868064
correcting imap
correcting imap
data fork in ino 66647930 claims free block 106685975
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
data fork in ino 67040030 claims free block 104754479
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
correcting imap
        - agno = 1
data fork in ino 1073742022 claims free block 92027186
        - agno = 2
data fork in ino 2200047805 claims free block 109063324
data fork in ino 2447259523 claims free block 105065314
data fork in ino 2447259551 claims free block 121473289
data fork in ino 2447259551 claims free block 99548026
data fork in ino 2447259551 claims free block 100743044
data fork in ino 2447259551 claims free block 100843496
data fork in ino 2447259554 claims free block 83185556
        - agno = 3
data fork in ino 3407200641 claims free block 106003630
data fork in ino 3407200641 claims free block 93652025
data fork in ino 3407200641 claims free block 94066445
data fork in ino 3407200701 claims free block 83696670
data fork in ino 3407200701 claims free block 83931983
        - agno = 4
data fork in ino 5264909527 claims free block 82492973
        - agno = 5
        - agno = 6
        - agno = 7
data fork in ino 7516192937 claims free block 12144174
        - agno = 8
        - agno = 9
data fork in ino 9663676597 claims free block 101166741
data fork in ino 9663676604 claims free block 38135217
data fork in ino 9663676604 claims free block 38188254
data fork in ino 9663676604 claims free block 38306395
data fork in ino 9663676605 claims free block 101294130
data fork in ino 9663676606 claims free block 21803650
data fork in ino 9663676606 claims free block 38791169
data fork in ino 9663676607 claims free block 40853351
data fork in ino 9663676607 claims free block 48321003
data fork in ino 9686061382 claims free block 15284998
data fork in ino 9686061383 claims free block 72231270
data fork in ino 9686061386 claims free block 68927866
data fork in ino 9686061386 claims free block 56831453
data fork in ino 9686061386 claims free block 40957911
data fork in ino 9686061386 claims free block 41000970
data fork in ino 9686061386 claims free block 41005395
        - agno = 10
data fork in ino 10737418405 claims free block 101300155
        - agno = 11
data fork in ino 11811160223 claims free block 121173937
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 4
        - agno = 9
        - agno = 3
        - agno = 6
        - agno = 5
        - agno = 7
        - agno = 8
        - agno = 10
        - agno = 1
        - agno = 11
        - agno = 2
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
Note - stripe unit (0) and width (0) were copied from a backup superblock.
Please reset with mount -o sunit=,swidth= if necessary
Maximum metadata LSN (1:79786) is ahead of log (1:78114).
Format log to cycle 4.
done

 

No mention of -L in the output.

From here, would I re-assign the drive and re-start the array as normal?

4 minutes ago, sidezero said:

From here, would I re-assign the drive and re-start the array as normal?

Before doing that I would suggest restarting the array in Normal mode and seeing if you have a lost+found folder on the drive with numeric cryptic names where the repair process could not figure out the correct name.  If you have this it needs sorting out manually (and that can be a lot of work), or restoring from backups.

 

It is always possible that doing the repair process on the physical drive rather than the emulated one may be more successful in getting files back with their original names so do not rush to overwrite the physical drive.

I don't see a lost+found folder on the emulated disk9 after starting the array normally with the drive still unassigned. It appears I may have gotten lucky this time around. When re-assigning the drive and bringing the array back up, will a parity rebuild kick off after all of this?

 

Thank you both again for all of the help with this matter.

1 minute ago, sidezero said:

When re-assigning the drive and bringing the array back up, will a parity rebuild kick off after all of this?

Yes - and it will make the physical drive you assign match the contents of the emulated drive which was why you want to check the contents of the emulated drive before overwriting the physical drive.

Everything is looking great. Thanks again!