sidezero

Everything looks great after the rebuild. Thanks again.

Rebuild is running. Thank you again for all the help.

Ran a repair on the emulated disk and re-started the array in normal mode to confirm there doesn't appear to be a lost+found directory created on the emulated Disk 8. At this point am I good to re-add the new drive in the disk8 slot and start the array/rebuild?

Drive set to xfs and updated diags attached. orbit-diagnostics-20250827-0336.zip

Process followed and updated diagnostics attached. Thank you. orbit-diagnostics-20250826-1253.zip

The failures seem pretty consistent with Disk 8 despite it being in three different slots now. With the current issues is there any process where I can re-install the original Disk 13 and then attempt to rebuild Disk 8 instead with the existing parity data?

Yep, a different vertical and horizontal row.

There are no power splitters in place. When I performed a rebuild I had to purposely purchase an overkill PSU in order to get enough connections for the backplane. I moved Disk 8 to another slot as requested and I saw the same thing upon boot. Disk 8 and Disk 13 are unable to be mounted and the parity operation kicked off. Disk 8 is currently accumulating millions of errors. Updated diagnostic attached. orbit-diagnostics-20250826-0847.zip

I use an old Norco 4220 case. Since the cables are shared to a backplane I swapped slots on the backplane with another drive before it became unmountable.

Updated diagnostics are attached. I haven't stopped the array to re-add Disk 13 yet at this point from the initial slot swap and boot up/auto start. orbit-diagnostics-20250825-1546.zip

I mistakenly removed Disk 13 from the Unraid GUI and had auto start enabled without realizing. Upon boot up Disk 8 is showing additional read errors and unmountable. Disk 13 is also showing the same but that is expected due to my previous manual removal. I still have the original Disk 13 available.

Thank you. Rebuild cancelled and will report back.

Looks like this accidentally got marked as solved early morning when submitted. That’s been corrected. Apologies for the mixup.

Diagnostics attached. Since the failure I've disabled the docker and vm services to limit writes to the array. Disk 13, sdt is the drive originally replaced for the upgrade. Disk 8, sdm is the drive that started throwing errors during the rebuild of sdt. Thank you orbit-diagnostics-20250825-0911.zip

I replaced one of my 6TB drives yesterday to upgrade and during the rebuild overnight a second drive started reporting over 800,000 read errors which have stopped while the rebuild still continues. What should my next steps be? I have the original 6TB drive I replaced still, but data has still been written to the array while the rebuild was in progress overnight.

Everything is looking great. Thanks again!

I don't see a lost+found folder on the emulated disk9 after starting the array normally with the drive still unassigned. It appears I may have gotten lucky this time around. When re-assigning the drive and bringing the array back up, will a parity rebuild kick off after all of this? Thank you both again for all of the help with this matter.

From here, would I re-assign the drive and re-start the array as normal?

Thank you. xfs_repair log is below: Phase 1 - find and verify superblock... bad primary superblock - inconsistent filesystem geometry information !!! attempting to find secondary superblock... .found candidate secondary superblock... verified secondary superblock... writing modified primary superblock Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... out-of-order bno btree record 2 (6748572 2327) block 0/1 out-of-order bno btree record 3 (5362660 132532) block 0/1 out-of-order bno btree record 4 (4115676 31148) block 0/1 out-of-order bno btree record 6 (8126233 70) block 0/1 block (0,8126233-8126302) multiply claimed by bno space tree, state - 1 out-of-order bno btree record 11 (1060775 12557) block 0/1 out-of-order bno btree record 13 (18475539 6235) block 0/1 out-of-order bno btree record 15 (11267389 8288) block 0/1 out-of-order bno btree record 18 (26538296 2973704) block 0/1 out-of-order bno btree record 23 (63104919 5300) block 0/1 out-of-order bno btree record 25 (72231270 47347) block 0/1 out-of-order bno btree record 29 (83185556 7456) block 0/1 out-of-order bno btree record 34 (100843496 38056) block 0/1 out-of-order bno btree record 38 (23689307 61750) block 0/1 invalid start block 124128315 in record 41 of bno btree block 0/1 out-of-order bno btree record 42 (121173937 5887) block 0/1 out-of-order cnt btree record 8 (96564608 2094) block 0/2 out-of-order cnt btree record 12 (41000970 4425) block 0/2 out-of-order cnt btree record 15 (88332932 5159) block 0/2 out-of-order cnt btree record 17 (100743044 7148) block 0/2 invalid start block 123296326 in record 19 of cnt btree block 0/2 out-of-order cnt btree record 21 (68927866 10155) block 0/2 out-of-order cnt btree record 23 (18503749 14117) block 0/2 out-of-order cnt btree record 24 (82492973 14469) block 0/2 block (0,28085137-28088502) multiply claimed by cnt space tree, state - 2 invalid start block 133026027 in record 27 of cnt btree block 0/2 out-of-order cnt btree record 28 (101294130 18502) block 0/2 out-of-order cnt btree record 32 (9320005 70835) block 0/2 invalid start block 234338807 in record 35 of cnt btree block 0/2 invalid start block 216810126 in record 40 of cnt btree block 0/2 invalid start block 145965057 in record 41 of cnt btree block 0/2 agf_freeblks 6589319, counted 6468671 in ag 0 agf_longest 3071743, counted 4756377 in ag 0 inode chunk claims used block, inobt block - agno 0, bno 16, inopb 8 ir_freecount/free mismatch, inode chunk 0/66647872, freecount 19 nfree 29 inode chunk claims used block, finobt block - agno 0, bno 16, inopb 8 undiscovered finobt record, ino 192 (0/192) finobt ir_freecount/free mismatch, inode chunk 0/192, freecount 12 nfree 14 undiscovered finobt record, ino 66647872 (0/66647872) undiscovered finobt record, ino 67040000 (0/67040000) undiscovered finobt record, ino 67106816 (0/67106816) agi_freecount 32, counted 76 in ag 0 agi_freecount 32, counted 94 in ag 0 finobt sb_icount 1024, counted 2048 sb_ifree 114, counted 628 sb_fdblocks 4762656, counted 29615893 root inode chunk not found Phase 3 - for each AG... - scan and clear agi unlinked lists... found inodes not in the inode allocation tree found inodes not in the inode allocation tree - process known inodes and perform inode discovery... - agno = 0 1512067d7680: Badness in key lookup (length) bp=(bno 0xc0, len 4096 bytes) key=(bno 0xc0, len 16384 bytes) 1512067d7680: Badness in key lookup (length) bp=(bno 0xe0, len 4096 bytes) key=(bno 0xe0, len 16384 bytes) 1512067d7680: Badness in key lookup (length) bp=(bno 0x3f8f740, len 4096 bytes) key=(bno 0x3f8f740, len 16384 bytes) 1512067d7680: Badness in key lookup (length) bp=(bno 0x3f8f760, len 4096 bytes) key=(bno 0x3f8f760, len 16384 bytes) 1512067d7680: Badness in key lookup (length) bp=(bno 0x3fef300, len 4096 bytes) key=(bno 0x3fef300, len 16384 bytes) 1512067d7680: Badness in key lookup (length) bp=(bno 0x3fef320, len 4096 bytes) key=(bno 0x3fef320, len 16384 bytes) 1512067d7680: Badness in key lookup (length) bp=(bno 0x3fff800, len 4096 bytes) key=(bno 0x3fff800, len 16384 bytes) 1512067d7680: Badness in key lookup (length) bp=(bno 0x3fff820, len 4096 bytes) key=(bno 0x3fff820, len 16384 bytes) data fork in ino 131 claims free block 15 data fork in ino 131 claims free block 14 data fork in ino 131 claims free block 13 data fork in ino 131 claims free block 12 correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap data fork in ino 154 claims free block 13033146 data fork in ino 154 claims free block 37305503 data fork in ino 154 claims free block 37995151 data fork in ino 154 claims free block 61359149 data fork in ino 154 claims free block 71336771 data fork in ino 154 claims free block 71459262 correcting imap data fork in ino 155 claims free block 7806721 correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap data fork in ino 164 claims free block 7806724 correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap data fork in ino 207 claims free block 116927339 data fork in ino 207 claims free block 117252879 correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap data fork in ino 227 claims free block 13308957 correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap data fork in ino 245 claims free block 11 correcting imap correcting imap correcting imap correcting imap data fork in ino 249 claims free block 32 data fork in ino 249 claims free block 1060775 data fork in ino 249 claims free block 4115676 correcting imap data fork in ino 250 claims free block 5362660 correcting imap correcting imap data fork in ino 252 claims free block 6748572 correcting imap data fork in ino 253 claims free block 7803443 data fork in ino 253 claims free block 8390412 data fork in ino 253 claims free block 8398494 data fork in ino 253 claims free block 8824247 data fork in ino 253 claims free block 9320005 data fork in ino 253 claims free block 11267389 correcting imap imap claims a free inode 66647872 is in use, correcting imap and clearing inode cleared inode 66647872 correcting imap data fork in ino 66647875 claims free block 18475539 data fork in ino 66647875 claims free block 18503749 data fork in ino 66647875 claims free block 19122726 correcting imap data fork in ino 66647877 claims free block 21826214 correcting imap data fork in ino 66647878 claims free block 23689307 correcting imap data fork in ino 66647880 claims free block 25404468 data fork in ino 66647880 claims free block 26538296 correcting imap data fork in ino 66647881 claims free block 26854724 data fork in ino 66647881 claims free block 28084569 data fork in ino 66647881 claims free block 28108248 data fork in ino 66647881 claims free block 28951860 data fork in ino 66647881 claims free block 32130417 correcting imap correcting imap data fork in ino 66647887 claims free block 49535651 data fork in ino 66647887 claims free block 49790474 correcting imap correcting imap data fork in ino 66647895 claims free block 63104919 data fork in ino 66647895 claims free block 63475252 correcting imap correcting imap data fork in ino 66647897 claims free block 63897028 correcting imap correcting imap data fork in ino 66647901 claims free block 72798090 data fork in ino 66647901 claims free block 73346810 data fork in ino 66647901 claims free block 73407799 data fork in ino 66647901 claims free block 73415260 correcting imap correcting imap correcting imap correcting imap correcting imap data fork in ino 66647909 claims free block 88332932 correcting imap correcting imap correcting imap correcting imap data fork in ino 66647915 claims free block 95889725 data fork in ino 66647915 claims free block 96564608 correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap data fork in ino 66647924 claims free block 15633220 correcting imap data fork in ino 66647925 claims free block 101308540 data fork in ino 66647925 claims free block 101318331 data fork in ino 66647925 claims free block 101696011 correcting imap data fork in ino 66647926 claims free block 102407986 data fork in ino 66647926 claims free block 102868064 correcting imap correcting imap data fork in ino 66647930 claims free block 106685975 correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap data fork in ino 67040030 claims free block 104754479 correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap - agno = 1 data fork in ino 1073742022 claims free block 92027186 - agno = 2 data fork in ino 2200047805 claims free block 109063324 data fork in ino 2447259523 claims free block 105065314 data fork in ino 2447259551 claims free block 121473289 data fork in ino 2447259551 claims free block 99548026 data fork in ino 2447259551 claims free block 100743044 data fork in ino 2447259551 claims free block 100843496 data fork in ino 2447259554 claims free block 83185556 - agno = 3 data fork in ino 3407200641 claims free block 106003630 data fork in ino 3407200641 claims free block 93652025 data fork in ino 3407200641 claims free block 94066445 data fork in ino 3407200701 claims free block 83696670 data fork in ino 3407200701 claims free block 83931983 - agno = 4 data fork in ino 5264909527 claims free block 82492973 - agno = 5 - agno = 6 - agno = 7 data fork in ino 7516192937 claims free block 12144174 - agno = 8 - agno = 9 data fork in ino 9663676597 claims free block 101166741 data fork in ino 9663676604 claims free block 38135217 data fork in ino 9663676604 claims free block 38188254 data fork in ino 9663676604 claims free block 38306395 data fork in ino 9663676605 claims free block 101294130 data fork in ino 9663676606 claims free block 21803650 data fork in ino 9663676606 claims free block 38791169 data fork in ino 9663676607 claims free block 40853351 data fork in ino 9663676607 claims free block 48321003 data fork in ino 9686061382 claims free block 15284998 data fork in ino 9686061383 claims free block 72231270 data fork in ino 9686061386 claims free block 68927866 data fork in ino 9686061386 claims free block 56831453 data fork in ino 9686061386 claims free block 40957911 data fork in ino 9686061386 claims free block 41000970 data fork in ino 9686061386 claims free block 41005395 - agno = 10 data fork in ino 10737418405 claims free block 101300155 - agno = 11 data fork in ino 11811160223 claims free block 121173937 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 4 - agno = 9 - agno = 3 - agno = 6 - agno = 5 - agno = 7 - agno = 8 - agno = 10 - agno = 1 - agno = 11 - agno = 2 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Note - stripe unit (0) and width (0) were copied from a backup superblock. Please reset with mount -o sunit=,swidth= if necessary Maximum metadata LSN (1:79786) is ahead of log (1:78114). Format log to cycle 4. done No mention of -L in the output.

After setting the fs to xfs, should the array be re-started in maintenance mode or normal?

It doesn't appear that I can. Disk 9 is showing Unmountable after the last steps I took prior to the most recent diag.

Ran through all of that and updated diag attached. Thanks again. orbit-diagnostics-20240606-1758.zip

sdb is the drive that was originally replaced. Disk 9 is the one that is showing missing and is the one that failed during the rebuild of sdb.

You called it. The original rebuilt drive is unmountable as well. I do have the original drive still that I could throw back in for that data potentially. Diagnostics attached. Thank you for all the help. orbit-diagnostics-20240605-1238.zip

I unfortunately did not in this instance.