scrathe Posted October 6, 2022 Share Posted October 6, 2022 (edited) Hi Community! Looking for some guidance on how best to proceed with my issue. Thank you in advance for your review and any suggestions shared! Diagnostic logs hoth-diagnostics-20221006-0923.zip Scenario: Older/established server experienced a mini-SAS to 4-drive cable failure. (SFF-8643 to SFF-8482) Two drives were knocked offline. Parity2 and Disk1. Powered down. Replaced cable. Removed Parity2. Left healthy Parity1 in place. Powered up. Started array, Data-Rebuild started automatically if I recall correctly. Note: During the rebuild, the disk was labeled as "Unmountable: not mounted". I didn't know if this was to be expected. Left it be during the rebuild. Rebuild completed. Disk1 is marked "green, in operation", but still shows "Unmountable: not mounted". Did some research on the community forums and https://wiki.unraid.net/index.php/Replacing_a_Data_Drive. Checked Disk1 xfs filesystem. Stopped array, mounted in maintenance, performed an xfs_repair -n. (see Spoiler below). Note I only performed "-n" and have not taken any further actions. Ideas: New Config, Keep Parity, and let Data Rebuild again? Let xfs_repair try and repair? There were 100s and 100s of errors. Data loss more likely? Spoiler Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... Metadata CRC error detected at 0x439496, xfs_agf block 0x1/0x200 agf has bad CRC for ag 0 Metadata CRC error detected at 0x439496, xfs_agf block 0x17fffffe9/0x200 agf has bad CRC for ag 3 Metadata CRC error detected at 0x43cfad, xfs_cntbt block 0x17ffffff8/0x1000 btree block 3/2 is suspect, error -74 bad magic # 0x477512e2 in btcnt block 3/2 Metadata CRC error detected at 0x43cfad, xfs_cntbt block 0x18045aeb0/0x1000 btree block 3/570841 is suspect, error -74 invalid start block 3077311351 in record 0 of cnt btree block 3/570841 ... snip invalid length 2516062702 in record 174 of cnt btree block 3/570841 Metadata CRC error detected at 0x43cfad, xfs_cntbt block 0x10/0x1000 btree block 0/2 is suspect, error -74 invalid start block 1794696543 in record 0 of cnt btree block 0/2 ... snip block (0,195903802-195903802) multiply claimed by cnt space tree, state - 2 out-of-order cnt btree record 340 (195903811 1) block 0/2 block (0,195903811-195903811) multiply claimed by cnt space tree, state - 2 Metadata CRC error detected at 0x43cfad, xfs_cntbt block 0xc620c0/0x1000 btree block 0/1623064 is suspect, error -74 invalid start block 3232662982 in record 0 of cnt btree block 0/1623064 block (0,195903829-195903829) multiply claimed by cnt space tree, state - 2 block (0,195903838-195903838) multiply claimed by cnt space tree, state - 2 ... snip undiscovered finobt record, ino 1605135936 (0/1605135936) undiscovered finobt record, ino 1605141632 (0/1605141632) block (3,145568353-145568358) multiply claimed by cnt space tree, state - 2 agf_freeblks 6413616, counted 6413157 in ag 3 agf_btreeblks 7, counted 6 in ag 3 sb_fdblocks 54204876, counted 178044926 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... found inodes not in the inode allocation tree - process known inodes and perform inode discovery... - agno = 0 data fork in ino 6069124 claims free block 147373298 data fork in ino 6069133 claims free block 191522022 ... snip data fork in ino 27234786432 claims free block 193693988 data fork in ino 27234786435 claims free block 193715928 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... free space (0,757565-757565) only seen by one free space btree free space (0,1120013-1120017) only seen by one free space btree ... snip free space (3,228297223-228297224) only seen by one free space btree - check for inodes claiming duplicate blocks... - agno = 0 - agno = 6 - agno = 1 - agno = 4 - agno = 2 - agno = 5 - agno = 7 - agno = 9 - agno = 8 - agno = 10 - agno = 3 - agno = 11 entry ".." at block 0 offset 80 in directory inode 10737418373 references non-existent inode 1517850785 entry "Media Content - S26E01 - Media Content-thumb.jpg" at block 0 offset 96 in directory inode 4314053987 references non-existent inode 1542695104 would clear inode number in entry at offset 96... ... snip entry "objects" in shortform directory 27234786453 references non-existent inode 1567232496 would have junked entry "objects" in directory inode 27234786453 entry "Season 01" in shortform directory 27234786459 references non-existent inode 1517850787 would have junked entry "Season 01" in directory inode 27234786459 entry "Season 07" in shortform directory 27234786461 references non-existent inode 1517850794 would have junked entry "Season 07" in directory inode 27234786461 No modify flag set, skipping phase 5 Inode allocation btrees are too corrupted, skipping phases 6 and 7 Maximum metadata LSN (1697619147:916413618) is ahead of log (5:2875030). Would format log to cycle 1697619150. No modify flag set, skipping filesystem flush and exiting. Edited October 9, 2022 by scrathe fixed attachment Quote Link to comment
JorgeB Posted October 6, 2022 Share Posted October 6, 2022 2 minutes ago, scrathe said: Diagnostic logs attached. Forgot that. Quote Link to comment
trurl Posted October 6, 2022 Share Posted October 6, 2022 3 minutes ago, scrathe said: Diagnostic logs attached 1 minute ago, JorgeB said: Forgot that. Buried in the middle of the check output Quote Link to comment
trurl Posted October 6, 2022 Share Posted October 6, 2022 6 minutes ago, scrathe said: Data-Rebuild started automatically if I recall correctly No that doesn't happen Quote Link to comment
trurl Posted October 6, 2022 Share Posted October 6, 2022 9 minutes ago, scrathe said: During the rebuild, the disk was labeled as "Unmountable: not mounted". I didn't know if this was to be expected. Left it be during the rebuild. We usually recommend repairing filesystem BEFORE rebuilding on top of the same disk. 7 minutes ago, scrathe said: New Config, Keep Parity, and let Data Rebuild again? You can't rebuild again unless you disable the disk again, and New Config isn't necessary to disable the disk again. Quote Link to comment
scrathe Posted October 6, 2022 Author Share Posted October 6, 2022 (edited) While array was in maintenance mode. Ran xfs_repair with options set to blank (see Spoiler) Spoiler Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... Metadata CRC error detected at 0x439496, xfs_agf block 0x17fffffe9/0x200 agf has bad CRC for ag 3 Metadata CRC error detected at 0x439496, xfs_agf block 0x1/0x200 agf has bad CRC for ag 0 Metadata CRC error detected at 0x43cfad, xfs_cntbt block 0x17ffffff8/0x1000 btree block 3/2 is suspect, error -74 bad magic # 0x477512e2 in btcnt block 3/2 Metadata CRC error detected at 0x43cfad, xfs_cntbt block 0x18045aeb0/0x1000 btree block 3/570841 is suspect, error -74 invalid start block 3077311351 in record 0 of cnt btree block 3/570841 ... undiscovered finobt record, ino 1605141632 (0/1605141632) block (3,145568353-145568358) multiply claimed by cnt space tree, state - 2 agf_freeblks 6413616, counted 6413157 in ag 3 agf_btreeblks 7, counted 6 in ag 3 sb_fdblocks 54204876, counted 178044926 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... found inodes not in the inode allocation tree - process known inodes and perform inode discovery... - agno = 0 data fork in ino 6069124 claims free block 147373298 ... data fork in ino 27234786435 claims free block 193715928 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 8 - agno = 11 - agno = 7 - agno = 4 - agno = 5 - agno = 1 - agno = 9 - agno = 10 - agno = 6 - agno = 3 ... Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... bad hash table for directory inode 18088873 (no data entry): rebuilding rebuilding directory inode 18088873 bad hash table for directory inode 172686908 (no data entry): rebuilding rebuilding directory inode 172686908 bad hash table for directory inode 176069144 (no data entry): rebuilding rebuilding directory inode 176069144 rebuilding directory inode 750251314 bad hash table for directory inode 750251316 (no data entry): rebuilding rebuilding directory inode 750251316 entry ".." in directory inode 2147483790 points to non-existent inode 1584954317 bad hash table for directory inode 2147483790 (no data entry): rebuilding rebuilding directory inode 2147483790 entry ".." in directory inode 2304313552 points to non-existent inode 1542714484 bad hash table for directory inode 2304313552 (no data entry): rebuilding rebuilding directory inode 2304313552 entry ".." in directory inode 2304313586 points to non-existent inode 1542714484 bad hash table for directory inode 2304313586 (no data entry): rebuilding rebuilding directory inode 2304313586 entry ".." in directory inode 2306984403 points to non-existent inode 1450743945 bad hash table for directory inode 2306984403 (no data entry): rebuilding rebuilding directory inode 2306984403 entry ".." in directory inode 2371901417 points to non-existent inode 1153943658 bad hash table for directory inode 2371901417 (no data entry): rebuilding rebuilding directory inode 2371901417 entry ".." in directory inode 3235472096 points to non-existent inode 1139914224 bad hash table for directory inode 3235472096 (no data entry): rebuilding rebuilding directory inode 3235472096 entry ".." in directory inode 3297180352 points to non-existent inode 1203155028 bad hash table for directory inode 3297180352 (no data entry): rebuilding rebuilding directory inode 3297180352 entry ".." in directory inode 3297180380 points to non-existent inode 1203155041 bad hash table for directory inode 3297180380 (no data entry): rebuilding rebuilding directory inode 3297180380 entry ".." in directory inode 3297191765 points to non-existent inode 1206333596 bad hash table for directory inode 3297191765 (no data entry): rebuilding rebuilding directory inode 3297191765 entry ".." in directory inode 3297191805 points to non-existent inode 1206333606 bad hash table for directory inode 3297191805 (no data entry): rebuilding rebuilding directory inode 3297191805 entry ".." in directory inode 3304800712 points to non-existent inode 1273442179 bad hash table for directory inode 3304800712 (no data entry): rebuilding rebuilding directory inode 3304800712 entry ".." in directory inode 3489132583 points to non-existent inode 1502482720 bad hash table for directory inode 3489132583 (no data entry): rebuilding rebuilding directory inode 3489132583 entry ".." in directory inode 3556217221 points to non-existent inode 1542704231 bad hash table for directory inode 3556217221 (no data entry): rebuilding rebuilding directory inode 3556217221 entry ".." in directory inode 3556230505 points to non-existent inode 1542714454 bad hash table for directory inode 3556230505 (no data entry): rebuilding rebuilding directory inode 3556230505 entry ".." in directory inode 3556240074 points to non-existent inode 1567232477 bad hash table for directory inode 3556240074 (no data entry): rebuilding rebuilding directory inode 3556240074 entry ".." in directory inode 3565501164 points to non-existent inode 1584940743 bad hash table for directory inode 3565501164 (no data entry): rebuilding rebuilding directory inode 3565501164 entry ".." in directory inode 3565503355 points to non-existent inode 1584946721 bad hash table for directory inode 3565503355 (no data entry): rebuilding rebuilding directory inode 3565503355 entry ".." in directory inode 3967762003 points to non-existent inode 1584954317 bad hash table for directory inode 3967762003 (no data entry): rebuilding rebuilding directory inode 3967762003 bad hash table for directory inode 4314053987 (no data entry): rebuilding rebuilding directory inode 4314053987 entry ".." in directory inode 5421314820 points to non-existent inode 1542710310 bad hash table for directory inode 5421314820 (no data entry): rebuilding rebuilding directory inode 5421314820 entry ".." in directory inode 5490442895 points to non-existent inode 1203155028 bad hash table for directory inode 5490442895 (no data entry): rebuilding rebuilding directory inode 5490442895 entry ".." in directory inode 5490442905 points to non-existent inode 1203155041 bad hash table for directory inode 5490442905 (no data entry): rebuilding rebuilding directory inode 5490442905 entry ".." in directory inode 5549049986 points to non-existent inode 1273442179 bad hash table for directory inode 5549049986 (no data entry): rebuilding rebuilding directory inode 5549049986 entry ".." in directory inode 5674007502 points to non-existent inode 1567232477 bad hash table for directory inode 5674007502 (no data entry): rebuilding rebuilding directory inode 5674007502 entry ".." in directory inode 5674013384 points to non-existent inode 1584940743 bad hash table for directory inode 5674013384 (no data entry): rebuilding rebuilding directory inode 5674013384 entry ".." in directory inode 6456700682 points to non-existent inode 1584946721 bad hash table for directory inode 6456700682 (no data entry): rebuilding rebuilding directory inode 6456700682 entry ".." in directory inode 7609547865 points to non-existent inode 1203155028 bad hash table for directory inode 7609547865 (no data entry): rebuilding rebuilding directory inode 7609547865 entry ".." in directory inode 7609547892 points to non-existent inode 1203155041 bad hash table for directory inode 7609547892 (no data entry): rebuilding rebuilding directory inode 7609547892 entry ".." in directory inode 8199692778 points to non-existent inode 1567230795 bad hash table for directory inode 8199692778 (no data entry): rebuilding rebuilding directory inode 8199692778 entry ".." in directory inode 8199694971 points to non-existent inode 1567232477 bad hash table for directory inode 8199694971 (no data entry): rebuilding rebuilding directory inode 8199694971 entry ".." in directory inode 8199702619 points to non-existent inode 1584946721 bad hash table for directory inode 8199702619 (no data entry): rebuilding rebuilding directory inode 8199702619 bad hash table for directory inode 8200043305 (no data entry): rebuilding rebuilding directory inode 8200043305 entry ".." in directory inode 8593141652 points to non-existent inode 1517850785 bad hash table for directory inode 8593141652 (no data entry): rebuilding rebuilding directory inode 8593141652 entry ".." in directory inode 9713284470 points to non-existent inode 1203155028 bad hash table for directory inode 9713284470 (no data entry): rebuilding rebuilding directory inode 9713284470 entry ".." in directory inode 9713288738 points to non-existent inode 1203155041 bad hash table for directory inode 9713288738 (no data entry): rebuilding rebuilding directory inode 9713288738 entry ".." in directory inode 9766072063 points to non-existent inode 1273442179 bad hash table for directory inode 9766072063 (no data entry): rebuilding rebuilding directory inode 9766072063 entry ".." in directory inode 9780331904 points to non-existent inode 1273442179 bad hash table for directory inode 9780331904 (no data entry): rebuilding rebuilding directory inode 9780331904 bad hash table for directory inode 10039246444 (no data entry): rebuilding rebuilding directory inode 10039246444 entry ".." in directory inode 10219013065 points to non-existent inode 1542714454 bad hash table for directory inode 10219013065 (no data entry): rebuilding rebuilding directory inode 10219013065 entry ".." in directory inode 10388696706 points to non-existent inode 1567232477 bad hash table for directory inode 10388696706 (no data entry): rebuilding rebuilding directory inode 10388696706 entry ".." in directory inode 10737418373 points to non-existent inode 1517850785 bad hash table for directory inode 10737418373 (no data entry): rebuilding rebuilding directory inode 10737418373 entry ".." in directory inode 10801183553 points to non-existent inode 1134285346 bad hash table for directory inode 10801183553 (no data entry): rebuilding rebuilding directory inode 10801183553 entry ".." in directory inode 11107244971 points to non-existent inode 1203155041 bad hash table for directory inode 11107244971 (no data entry): rebuilding rebuilding directory inode 11107244971 entry ".." in directory inode 12006758532 points to non-existent inode 1273442179 bad hash table for directory inode 12006758532 (no data entry): rebuilding rebuilding directory inode 12006758532 bad hash table for directory inode 12134728027 (no data entry): rebuilding rebuilding directory inode 12134728027 entry ".." in directory inode 12187770442 points to non-existent inode 1542714454 bad hash table for directory inode 12187770442 (no data entry): rebuilding rebuilding directory inode 12187770442 entry ".." in directory inode 12226632126 points to non-existent inode 1567232477 bad hash table for directory inode 12226632126 (no data entry): rebuilding rebuilding directory inode 12226632126 bad hash table for directory inode 12388519269 (no data entry): rebuilding rebuilding directory inode 12388519269 entry ".." in directory inode 12902824970 points to non-existent inode 1134285345 bad hash table for directory inode 12902824970 (no data entry): rebuilding rebuilding directory inode 12902824970 entry ".." in directory inode 12915323528 points to non-existent inode 1517850785 bad hash table for directory inode 12915323528 (no data entry): rebuilding rebuilding directory inode 12915323528 entry ".." in directory inode 12937884407 points to non-existent inode 1567232477 bad hash table for directory inode 12937884407 (no data entry): rebuilding rebuilding directory inode 12937884407 bad hash table for directory inode 13906006073 (no data entry): rebuilding rebuilding directory inode 13906006073 entry ".." in directory inode 14716596603 points to non-existent inode 1567232477 bad hash table for directory inode 14716596603 (no data entry): rebuilding rebuilding directory inode 14716596603 entry ".." in directory inode 15069040967 points to non-existent inode 1273442179 bad hash table for directory inode 15069040967 (no data entry): rebuilding rebuilding directory inode 15069040967 bad hash table for directory inode 15097673199 (no data entry): rebuilding rebuilding directory inode 15097673199 bad hash table for directory inode 16749649900 (no data entry): rebuilding rebuilding directory inode 16749649900 entry ".." in directory inode 16766380496 points to non-existent inode 1567232477 bad hash table for directory inode 16766380496 (no data entry): rebuilding rebuilding directory inode 16766380496 bad hash table for directory inode 18333421901 (no data entry): rebuilding rebuilding directory inode 18333421901 entry ".." in directory inode 18349269021 points to non-existent inode 1203155041 bad hash table for directory inode 18349269021 (no data entry): rebuilding rebuilding directory inode 18349269021 bad hash table for directory inode 18416108544 (no data entry): rebuilding rebuilding directory inode 18416108544 bad hash table for directory inode 18416376109 (no data entry): rebuilding rebuilding directory inode 18416376109 entry ".." in directory inode 18483202927 points to non-existent inode 1273442179 bad hash table for directory inode 18483202927 (no data entry): rebuilding rebuilding directory inode 18483202927 bad hash table for directory inode 18483207010 (no data entry): rebuilding rebuilding directory inode 18483207010 bad hash table for directory inode 18809401984 (no data entry): rebuilding rebuilding directory inode 18809401984 bad hash table for directory inode 18839408527 (no data entry): rebuilding rebuilding directory inode 18839408527 bad hash table for directory inode 18839479276 (no data entry): rebuilding rebuilding directory inode 18839479276 bad hash table for directory inode 18906490495 (no data entry): rebuilding rebuilding directory inode 18906490495 bad hash table for directory inode 19397345049 (no data entry): rebuilding rebuilding directory inode 19397345049 entry ".." in directory inode 19472006774 points to non-existent inode 1517850785 bad hash table for directory inode 19472006774 (no data entry): rebuilding rebuilding directory inode 19472006774 entry ".." in directory inode 19719890151 points to non-existent inode 1134285345 bad hash table for directory inode 19719890151 (no data entry): rebuilding rebuilding directory inode 19719890151 bad hash table for directory inode 20448154744 (no data entry): rebuilding rebuilding directory inode 20448154744 bad hash table for directory inode 20460449994 (no data entry): rebuilding rebuilding directory inode 20460449994 entry ".." in directory inode 20460453916 points to non-existent inode 1203155041 bad hash table for directory inode 20460453916 (no data entry): rebuilding rebuilding directory inode 20460453916 entry ".." in directory inode 20503064094 points to non-existent inode 1273442179 bad hash table for directory inode 20503064094 (no data entry): rebuilding rebuilding directory inode 20503064094 bad hash table for directory inode 20503064098 (no data entry): rebuilding rebuilding directory inode 20503064098 entry ".." in directory inode 21474852821 points to non-existent inode 1542714484 bad hash table for directory inode 21474852821 (no data entry): rebuilding rebuilding directory inode 21474852821 bad hash table for directory inode 22655123743 (no data entry): rebuilding rebuilding directory inode 22655123743 bad hash table for directory inode 22655123755 (no data entry): rebuilding rebuilding directory inode 22655123755 entry ".." in directory inode 22659807927 points to non-existent inode 1203155041 bad hash table for directory inode 22659807927 (no data entry): rebuilding rebuilding directory inode 22659807927 bad hash table for directory inode 22788740985 (no data entry): rebuilding rebuilding directory inode 22788740985 entry ".." in directory inode 22794983378 points to non-existent inode 1273442179 bad hash table for directory inode 22794983378 (no data entry): rebuilding rebuilding directory inode 22794983378 bad hash table for directory inode 23179807737 (no data entry): rebuilding rebuilding directory inode 23179807737 bad hash table for directory inode 23246908615 (no data entry): rebuilding rebuilding directory inode 23246908615 entry ".." in directory inode 23623361055 points to non-existent inode 1542714484 bad hash table for directory inode 23623361055 (no data entry): rebuilding rebuilding directory inode 23623361055 entry ".." in directory inode 23623399301 points to non-existent inode 1517850785 bad hash table for directory inode 23623399301 (no data entry): rebuilding rebuilding directory inode 23623399301 entry ".." in directory inode 23629527674 points to non-existent inode 1206333596 bad hash table for directory inode 23629527674 (no data entry): rebuilding rebuilding directory inode 23629527674 bad hash table for directory inode 24665371904 (no data entry): rebuilding rebuilding directory inode 24665371904 bad hash table for directory inode 24665371958 (no data entry): rebuilding rebuilding directory inode 24665371958 entry ".." in directory inode 24710945757 points to non-existent inode 1203155041 bad hash table for directory inode 24710945757 (no data entry): rebuilding rebuilding directory inode 24710945757 bad hash table for directory inode 25145636343 (no data entry): rebuilding rebuilding directory inode 25145636343 bad hash table for directory inode 25442121811 (no data entry): rebuilding rebuilding directory inode 25442121811 bad hash table for directory inode 25442146715 (no data entry): rebuilding rebuilding directory inode 25442146715 bad hash table for directory inode 25442146727 (no data entry): rebuilding rebuilding directory inode 25442146727 bad hash table for directory inode 25649929886 (no data entry): rebuilding rebuilding directory inode 25649929886 entry ".." in directory inode 25770375968 points to non-existent inode 1206333596 bad hash table for directory inode 25770375968 (no data entry): rebuilding rebuilding directory inode 25770375968 entry ".." in directory inode 27167686926 points to non-existent inode 1273442179 bad hash table for directory inode 27167686926 (no data entry): rebuilding rebuilding directory inode 27167686926 bad hash table for directory inode 27176147095 (no data entry): rebuilding rebuilding directory inode 27176147095 bad hash table for directory inode 27234784587 (no data entry): rebuilding rebuilding directory inode 27234784587 bad hash table for directory inode 27234784590 (no data entry): rebuilding rebuilding directory inode 27234784590 - traversal finished ... - moving disconnected inodes to lost+found ... disconnected inode 6069120, moving to lost+found disconnected inode 6069121, moving to lost+found disconnected inode 6069122, moving to lost+found disconnected inode 15764032, moving to lost+found disconnected inode 15764033, moving to lost+found disconnected inode 15764034, moving to lost+found disconnected inode 15764035, moving to lost+found disconnected inode 15764036, moving to lost+found disconnected inode 15764037, moving to lost+found disconnected inode 15764038, moving to lost+found disconnected inode 15764039, moving to lost+found disconnected inode 15764040, moving to lost+found disconnected inode 15764041, moving to lost+found disconnected inode 15764042, moving to lost+found disconnected inode 15764043, moving to lost+found disconnected inode 15764044, moving to lost+found disconnected inode 15764045, moving to lost+found disconnected inode 15764046, moving to lost+found disconnected inode 15842999, moving to lost+found disconnected inode 15843000, moving to lost+found disconnected inode 15843001, moving to lost+found disconnected inode 15843002, moving to lost+found disconnected inode 15843003, moving to lost+found disconnected inode 15843004, moving to lost+found disconnected inode 15843005, moving to lost+found disconnected inode 15843006, moving to lost+found disconnected inode 15843007, moving to lost+found disconnected dir inode 18088834, moving to lost+found disconnected inode 40984972, moving to lost+found disconnected inode 137995191, moving to lost+found disconnected inode 137995192, moving to lost+found disconnected inode 137995193, moving to lost+found disconnected inode 137995586, moving to lost+found disconnected inode 137995587, moving to lost+found disconnected inode 137995588, moving to lost+found disconnected inode 137995589, moving to lost+found disconnected inode 137995590, moving to lost+found disconnected inode 137995591, moving to lost+found disconnected inode 137995592, moving to lost+found disconnected inode 137995593, moving to lost+found disconnected inode 137995594, moving to lost+found disconnected inode 137995595, moving to lost+found disconnected inode 137995596, moving to lost+found disconnected inode 137995597, moving to lost+found disconnected inode 137995598, moving to lost+found disconnected inode 137995599, moving to lost+found disconnected inode 137995600, moving to lost+found disconnected inode 137995601, moving to lost+found disconnected inode 171570604, moving to lost+found disconnected inode 171570605, moving to lost+found disconnected inode 171570606, moving to lost+found disconnected inode 171570607, moving to lost+found disconnected inode 171570608, moving to lost+found disconnected inode 171572225, moving to lost+found disconnected inode 171572232, moving to lost+found disconnected inode 171572234, moving to lost+found disconnected inode 172686856, moving to lost+found disconnected inode 172686857, moving to lost+found disconnected inode 172686858, moving to lost+found disconnected inode 172686883, moving to lost+found disconnected inode 172686884, moving to lost+found disconnected inode 172686885, moving to lost+found disconnected inode 172686889, moving to lost+found disconnected inode 172686890, moving to lost+found disconnected inode 172686891, moving to lost+found disconnected inode 172686892, moving to lost+found disconnected inode 172686897, moving to lost+found disconnected inode 172686901, moving to lost+found disconnected inode 172686902, moving to lost+found disconnected inode 172686903, moving to lost+found disconnected inode 172686904, moving to lost+found disconnected inode 172686905, moving to lost+found disconnected inode 176069127, moving to lost+found disconnected inode 176069128, moving to lost+found disconnected inode 176069129, moving to lost+found disconnected inode 496117636, moving to lost+found disconnected inode 496117637, moving to lost+found disconnected inode 496117638, moving to lost+found disconnected inode 496117639, moving to lost+found disconnected inode 496117640, moving to lost+found disconnected inode 496117641, moving to lost+found disconnected inode 496117642, moving to lost+found disconnected inode 496117684, moving to lost+found disconnected inode 496117685, moving to lost+found disconnected inode 496117686, moving to lost+found disconnected inode 496117687, moving to lost+found disconnected inode 496117688, moving to lost+found disconnected inode 727822723, moving to lost+found disconnected inode 727822724, moving to lost+found disconnected inode 727822725, moving to lost+found disconnected inode 727822726, moving to lost+found disconnected inode 727822734, moving to lost+found disconnected inode 727822735, moving to lost+found disconnected inode 727822736, moving to lost+found disconnected inode 727822737, moving to lost+found disconnected inode 750197043, moving to lost+found disconnected inode 750197044, moving to lost+found disconnected inode 750197045, moving to lost+found disconnected inode 750197046, moving to lost+found disconnected inode 750197051, moving to lost+found disconnected inode 750197052, moving to lost+found disconnected inode 750197053, moving to lost+found disconnected inode 750197054, moving to lost+found disconnected inode 750228936, moving to lost+found disconnected inode 750228937, moving to lost+found disconnected inode 750228938, moving to lost+found disconnected inode 750228943, moving to lost+found disconnected inode 750228973, moving to lost+found disconnected inode 750228989, moving to lost+found disconnected inode 750228990, moving to lost+found disconnected inode 750228991, moving to lost+found disconnected inode 750251264, moving to lost+found disconnected inode 750251293, moving to lost+found disconnected inode 750251294, moving to lost+found disconnected inode 750251295, moving to lost+found disconnected inode 750251297, moving to lost+found disconnected inode 750251298, moving to lost+found disconnected inode 750251299, moving to lost+found disconnected inode 750251300, moving to lost+found disconnected inode 750251301, moving to lost+found disconnected inode 750251302, moving to lost+found disconnected dir inode 2147483790, moving to lost+found disconnected dir inode 2161125158, moving to lost+found disconnected dir inode 2161914451, moving to lost+found disconnected dir inode 2161914453, moving to lost+found disconnected dir inode 2161914455, moving to lost+found disconnected dir inode 2161914457, moving to lost+found disconnected dir inode 2180270920, moving to lost+found disconnected dir inode 2180270960, moving to lost+found disconnected dir inode 2304313552, moving to lost+found disconnected dir inode 2304313586, moving to lost+found disconnected dir inode 2306984403, moving to lost+found disconnected dir inode 2371901417, moving to lost+found disconnected dir inode 3235472096, moving to lost+found disconnected dir inode 3297180352, moving to lost+found disconnected dir inode 3297180380, moving to lost+found disconnected dir inode 3297191765, moving to lost+found disconnected dir inode 3297191805, moving to lost+found disconnected dir inode 3304800711, moving to lost+found disconnected dir inode 3304800712, moving to lost+found disconnected dir inode 3364286566, moving to lost+found disconnected dir inode 3489132583, moving to lost+found disconnected dir inode 3498499462, moving to lost+found disconnected dir inode 3498499485, moving to lost+found disconnected dir inode 3498499487, moving to lost+found disconnected dir inode 3498499490, moving to lost+found disconnected dir inode 3556198253, moving to lost+found disconnected dir inode 3556198255, moving to lost+found disconnected dir inode 3556198257, moving to lost+found disconnected dir inode 3556198260, moving to lost+found disconnected dir inode 3556217221, moving to lost+found disconnected dir inode 3556222329, moving to lost+found disconnected dir inode 3556230505, moving to lost+found disconnected dir inode 3556236493, moving to lost+found disconnected dir inode 3556240074, moving to lost+found disconnected dir inode 3565501164, moving to lost+found disconnected dir inode 3565503355, moving to lost+found disconnected dir inode 3967762003, moving to lost+found disconnected inode 4009212729, moving to lost+found disconnected inode 4009212730, moving to lost+found disconnected inode 4009214146, moving to lost+found disconnected dir inode 4009215815, moving to lost+found disconnected dir inode 4009215816, moving to lost+found disconnected dir inode 4294967498, moving to lost+found disconnected dir inode 4314055888, moving to lost+found disconnected dir inode 4328229861, moving to lost+found disconnected dir inode 4328229863, moving to lost+found disconnected dir inode 4328229866, moving to lost+found disconnected dir inode 4328229868, moving to lost+found disconnected dir inode 5421314820, moving to lost+found disconnected dir inode 5490442895, moving to lost+found disconnected dir inode 5490442905, moving to lost+found disconnected dir inode 5549049985, moving to lost+found disconnected dir inode 5549049986, moving to lost+found disconnected dir inode 5673983115, moving to lost+found disconnected dir inode 5673984340, moving to lost+found disconnected dir inode 5673993118, moving to lost+found disconnected dir inode 5674007502, moving to lost+found disconnected dir inode 5674013384, moving to lost+found disconnected dir inode 5674013396, moving to lost+found disconnected dir inode 5674019646, moving to lost+found disconnected dir inode 6442451103, moving to lost+found disconnected dir inode 6447009699, moving to lost+found disconnected dir inode 6447009701, moving to lost+found disconnected dir inode 6447009704, moving to lost+found disconnected dir inode 6447009706, moving to lost+found disconnected dir inode 6456700682, moving to lost+found disconnected dir inode 6458320559, moving to lost+found disconnected dir inode 6476714226, moving to lost+found disconnected dir inode 7609547865, moving to lost+found disconnected dir inode 7609547892, moving to lost+found disconnected dir inode 7715054956, moving to lost+found disconnected dir inode 7715054957, moving to lost+found disconnected dir inode 8199687755, moving to lost+found disconnected dir inode 8199687800, moving to lost+found disconnected dir inode 8199692778, moving to lost+found disconnected dir inode 8199694971, moving to lost+found disconnected dir inode 8199702619, moving to lost+found disconnected dir inode 8590768095, moving to lost+found disconnected dir inode 8590768098, moving to lost+found disconnected dir inode 8590768112, moving to lost+found disconnected dir inode 8590768114, moving to lost+found disconnected dir inode 8590768578, moving to lost+found disconnected dir inode 8593141652, moving to lost+found disconnected dir inode 8611686511, moving to lost+found disconnected dir inode 9713284470, moving to lost+found disconnected dir inode 9713288738, moving to lost+found disconnected dir inode 9766072063, moving to lost+found disconnected dir inode 9780331904, moving to lost+found disconnected dir inode 10219013064, moving to lost+found disconnected dir inode 10219013065, moving to lost+found disconnected dir inode 10388696706, moving to lost+found disconnected dir inode 10388712750, moving to lost+found disconnected dir inode 10737418373, moving to lost+found disconnected dir inode 10745981952, moving to lost+found disconnected dir inode 10745981955, moving to lost+found disconnected dir inode 10745981960, moving to lost+found disconnected dir inode 10745981963, moving to lost+found disconnected dir inode 10745981965, moving to lost+found disconnected dir inode 10801183553, moving to lost+found disconnected dir inode 11107244971, moving to lost+found disconnected dir inode 12006758531, moving to lost+found disconnected dir inode 12006758532, moving to lost+found disconnected dir inode 12187768315, moving to lost+found disconnected dir inode 12187770442, moving to lost+found disconnected dir inode 12226632126, moving to lost+found disconnected dir inode 12900190310, moving to lost+found disconnected dir inode 12900190312, moving to lost+found disconnected dir inode 12900190314, moving to lost+found disconnected dir inode 12900190316, moving to lost+found disconnected dir inode 12900190318, moving to lost+found disconnected dir inode 12902824970, moving to lost+found disconnected dir inode 12902837913, moving to lost+found disconnected dir inode 12915323528, moving to lost+found disconnected dir inode 12937884407, moving to lost+found disconnected dir inode 13906002509, moving to lost+found disconnected dir inode 13973099911, moving to lost+found disconnected dir inode 13973099912, moving to lost+found disconnected dir inode 14527035195, moving to lost+found disconnected dir inode 14716596603, moving to lost+found disconnected dir inode 15032385711, moving to lost+found disconnected dir inode 15069014859, moving to lost+found disconnected dir inode 15069016589, moving to lost+found disconnected dir inode 15069016626, moving to lost+found disconnected dir inode 15069016629, moving to lost+found disconnected dir inode 15069016631, moving to lost+found disconnected dir inode 15069016633, moving to lost+found disconnected dir inode 15069040967, moving to lost+found disconnected dir inode 15109354794, moving to lost+found disconnected dir inode 16172183548, moving to lost+found disconnected dir inode 16239252522, moving to lost+found disconnected dir inode 16766380496, moving to lost+found disconnected dir inode 17184690593, moving to lost+found disconnected dir inode 17184690596, moving to lost+found disconnected dir inode 17184690598, moving to lost+found disconnected dir inode 17184690605, moving to lost+found disconnected dir inode 17184690607, moving to lost+found disconnected dir inode 17202158696, moving to lost+found disconnected dir inode 18349269021, moving to lost+found disconnected dir inode 18483202927, moving to lost+found disconnected dir inode 19360581601, moving to lost+found disconnected dir inode 19360581604, moving to lost+found disconnected dir inode 19360581606, moving to lost+found disconnected dir inode 19360581608, moving to lost+found disconnected dir inode 19360581610, moving to lost+found disconnected dir inode 19472006774, moving to lost+found disconnected dir inode 19719890151, moving to lost+found disconnected dir inode 20460453916, moving to lost+found disconnected dir inode 20503064094, moving to lost+found disconnected dir inode 21474852821, moving to lost+found disconnected dir inode 21474965147, moving to lost+found disconnected dir inode 21489210666, moving to lost+found disconnected dir inode 21489213578, moving to lost+found disconnected dir inode 21501520125, moving to lost+found disconnected dir inode 21501520192, moving to lost+found disconnected dir inode 21501520209, moving to lost+found disconnected dir inode 21501520211, moving to lost+found disconnected dir inode 21501520213, moving to lost+found disconnected dir inode 22659807927, moving to lost+found disconnected dir inode 22794983378, moving to lost+found disconnected dir inode 23623361055, moving to lost+found disconnected dir inode 23623399301, moving to lost+found disconnected dir inode 23623570172, moving to lost+found disconnected dir inode 23623570174, moving to lost+found disconnected dir inode 23623571200, moving to lost+found disconnected dir inode 23623571202, moving to lost+found disconnected dir inode 23623571204, moving to lost+found disconnected dir inode 23629527674, moving to lost+found disconnected dir inode 24710945757, moving to lost+found disconnected dir inode 25080686505, moving to lost+found disconnected dir inode 25770370775, moving to lost+found disconnected dir inode 25770370777, moving to lost+found disconnected dir inode 25770370779, moving to lost+found disconnected dir inode 25770370781, moving to lost+found disconnected dir inode 25770370783, moving to lost+found disconnected dir inode 25770375968, moving to lost+found disconnected dir inode 25777464650, moving to lost+found disconnected dir inode 25791151900, moving to lost+found disconnected dir inode 27167686926, moving to lost+found Phase 7 - verify and correct link counts... resetting inode 6069183 nlinks from 2 to 178 resetting inode 24645427729 nlinks from 5 to 4 resetting inode 18333421901 nlinks from 7 to 6 resetting inode 24665371904 nlinks from 7 to 6 resetting inode 25775024404 nlinks from 3 to 2 resetting inode 18416108544 nlinks from 13 to 12 resetting inode 11104651132 nlinks from 10 to 9 resetting inode 18416376109 nlinks from 8 to 7 resetting inode 18483207010 nlinks from 7 to 6 resetting inode 18809401984 nlinks from 9 to 8 resetting inode 26226346691 nlinks from 3 to 2 resetting inode 12134728027 nlinks from 21 to 20 resetting inode 18839408527 nlinks from 14 to 13 resetting inode 750251314 nlinks from 744 to 715 resetting inode 18839479276 nlinks from 8 to 6 resetting inode 12226629543 nlinks from 7 to 6 resetting inode 26226358548 nlinks from 3 to 2 resetting inode 15097673199 nlinks from 30 to 29 resetting inode 12388519269 nlinks from 9 to 8 resetting inode 18906490495 nlinks from 18 to 16 resetting inode 24665371958 nlinks from 12 to 11 resetting inode 22534506087 nlinks from 7 to 6 resetting inode 26226364386 nlinks from 4 to 3 resetting inode 25049644139 nlinks from 5 to 4 resetting inode 12673846970 nlinks from 5 to 4 resetting inode 19397345049 nlinks from 484 to 482 resetting inode 10039246444 nlinks from 35 to 34 resetting inode 13906006073 nlinks from 16 to 15 resetting inode 25145587097 nlinks from 5 to 4 resetting inode 22655123743 nlinks from 6 to 5 resetting inode 25145636343 nlinks from 11 to 10 resetting inode 22655123755 nlinks from 13 to 12 resetting inode 16749649900 nlinks from 21 to 19 resetting inode 27101091506 nlinks from 4 to 3 resetting inode 22788740961 nlinks from 5 to 4 resetting inode 27109040590 nlinks from 4 to 3 resetting inode 25429718986 nlinks from 5 to 4 resetting inode 22788740985 nlinks from 6 to 5 resetting inode 20448154744 nlinks from 12 to 11 resetting inode 27142776353 nlinks from 3 to 2 resetting inode 27152032045 nlinks from 4 to 3 resetting inode 25442121811 nlinks from 8 to 7 resetting inode 23179807737 nlinks from 23 to 21 resetting inode 27167688327 nlinks from 3 to 2 resetting inode 8200043305 nlinks from 41 to 35 resetting inode 23179923732 nlinks from 5 to 4 resetting inode 27176139883 nlinks from 3 to 2 resetting inode 25442146703 nlinks from 5 to 4 resetting inode 27176147094 nlinks from 8 to 7 resetting inode 23246908615 nlinks from 6 to 5 resetting inode 27176147095 nlinks from 11 to 10 resetting inode 25442146715 nlinks from 6 to 5 resetting inode 23428201952 nlinks from 9 to 7 resetting inode 27214960756 nlinks from 10 to 9 resetting inode 23428201953 nlinks from 5 to 3 resetting inode 20460449994 nlinks from 13 to 12 resetting inode 25442146727 nlinks from 8 to 7 resetting inode 27214962170 nlinks from 4 to 3 resetting inode 23428275653 nlinks from 8 to 6 resetting inode 25649929886 nlinks from 8 to 7 resetting inode 27234777490 nlinks from 4 to 3 resetting inode 20460454822 nlinks from 8 to 7 resetting inode 27234784587 nlinks from 6 to 5 resetting inode 27234786453 nlinks from 5 to 4 resetting inode 27234786459 nlinks from 3 to 2 resetting inode 20503064098 nlinks from 6 to 5 resetting inode 27234786461 nlinks from 6 to 5 Maximum metadata LSN (1697619147:916413618) is ahead of log (5:2875030). Format log to cycle 1697619150. done Stopped / Started array still in maintenance mode. Ran xfs_repair -n (see Spoiler) Disk1 still says "Unmountable: not mounted" Spoiler Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 8 - agno = 3 - agno = 10 - agno = 4 - agno = 1 - agno = 9 - agno = 7 - agno = 11 - agno = 6 - agno = 5 - agno = 12 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting. Edited October 6, 2022 by scrathe Quote Link to comment
trurl Posted October 7, 2022 Share Posted October 7, 2022 You sort of did that backward. The first time you actually committed changes from filesystem repair without checking what it was going to do first by using -n (nomodify). So it should have already been repaired the first run, and nothing needed repair when you ran it -n. Start the array in normal mode and post new diagnostics Quote Link to comment
Solution scrathe Posted October 9, 2022 Author Solution Share Posted October 9, 2022 On 10/6/2022 at 8:03 PM, trurl said: You sort of did that backward. The first time you actually committed changes from filesystem repair without checking what it was going to do first by using -n (nomodify). So it should have already been repaired the first run, and nothing needed repair when you ran it -n. Start the array in normal mode and post new diagnostics I did a poor job of explaining what steps I took previously. I did xfs_repair -n, no -n (repaired), and another -n which reported no errors. Despite no errors, it still reported "Unmountable: not mounted". Thank you trurl 🙏 for taking the time on this thread. I was able to save my data. Posting the *cough solution below?... may some other poor soul who's thinking they lost data find help here. TLDR; Run xfs_repair (without -n) before you run Data-Rebuild. Data-Rebuild didn't work the first time. Removed/readded the disk to the array and re-ran Data-Rebuild. Disk and array are back to normal. Scenario: Older/established server experienced a mini-SAS to 4-drive cable failure. (SFF-8643 to SFF-8482) Two drives were knocked offline. Parity2 and Disk1. unRAID array status reported: Disk1 was emulated. However the data, and total array space was missing. Disk1 was "Unmountable: not mounted". Powered down. Replaced cable. Removed Parity2. Left healthy Parity1 in place. Powered up. Started array, started Data-Rebuild. unRAID array status reported: Disk1 was emulated. Data and space still missing. Disk1 was "Unmountable: not mounted". Data-Rebuild completed: Disk1 is marked "green, in operation". Data and space still missing. Disk1 was "Unmountable: not mounted". Stopped array. Solution? It worked... not entirely sure if all steps are needed. Started array (maintenance). Disk1: xfs_repair -n (100s and 100s of errors) Disk1: xfs_repair (repair without -n) Disk1: xfs_repair -n (no errors) Stopped array. Started array (normal). unRAID array status reported: Disk1 is marked "green, in operation". Data and space still missing. Disk1 was "Unmountable: not mounted". Stopped array. Removed Disk1 from array. Started array (maintenance). Stopped array. Added Disk1 to array. Started array (normal). unRAID array status reported: Disk1 was emulated. Data APPEARED, and total array space was BACK. Disk1 was "Unmountable: not mounted". Started Data-Rebuild. Data-Rebuild completed: Disk1 back to NORMAL. Quote Link to comment
trurl Posted October 9, 2022 Share Posted October 9, 2022 Not sure what happened, but your "solution" is unlikely to be a solution for anyone. The reason we recommend repair before rebuilding on top of the same disk is so you can determine if the repair of the emulated disk will give good results, since the emulated disk is exactly what the rebuild will result in. If the emulated repair isn't good, you can decide whether to trust the contents of the original disk instead of rebuilding it, or other possible options to get your data from that original disk. If you rebuild to a spare, you already have all those options with the original disk so it doesn't matter whether you repair before rebuild or not. Quote Link to comment
trurl Posted October 9, 2022 Share Posted October 9, 2022 Just now, trurl said: Not sure what happened Possibly you didn't start the array in normal mode after the repair. It won't tell you the repair is mountable until you start in normal mode Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.