December 7, 20241 yr My server received a sudden power outage (bad cable) and when it booted up one of the disks when to unmountable. The server has been restarted a couple of times since without any change in status and it is currently trying to run a parity check at extremely slow speeds (will take a decade or more to complete). I'm tempted to stop the array, set the unmountable disk to no drive, start the array, then stop once again and add it back in and start. But I am paranoid of losing the data. I don't have a spare disk at the moment that I 100% trust to swap out. Hopefully someone here can suggest a good procedure of recovering this disk with at least a good chance of not losing the data. oden-diagnostics-20241207-1637.zip
December 8, 20241 yr Author Using this thread to catalog what I am doing to fix things, hopefully helping someone in the future or to give people more information to help me out when I mess things up. I came across the unraid documentation on how to recover from this error: https://docs.unraid.net/unraid-os/manual/storage-management/#drive-shows-as-unmountable Starting with stopping the array then starting it in maintenance mode and running the xfs_repair from the ui (button in the disk page of the one that is having a problem, detailed in the link above) Here is my output. Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being ignored because the -n option was used. Expect spurious inconsistencies which may be resolved by first mounting the filesystem to replay the log. - scan filesystem freespace and inode maps... sb_fdblocks 34958915, counted 36912366 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 data fork in ino 1036865 claims free block 131018 data fork in ino 1036866 claims free block 154703 data fork in ino 1036878 claims free block 154847 data fork in ino 1036879 claims free block 154874 data fork in ino 1036880 claims free block 154877 Bad mtime nsec 1040198661 on inode 18449155, would reset to zero Bad ctime nsec 1040198661 on inode 18449155, would reset to zero data fork in regular inode 18449155 claims used block 2306194 correcting nextents for inode 18449155 bad data fork in inode 18449155 would have cleared inode 18449155 data fork in ino 34267989 claims free block 4283550 data fork in ino 47864192 claims free block 5983126 - agno = 1 - agno = 2 - agno = 3 data fork in regular inode 6484102721 claims used block 810512573 correcting nextents for inode 6484102721 bad data fork in inode 6484102721 would have cleared inode 6484102721 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 1 - agno = 5 - agno = 2 - agno = 3 - agno = 4 - agno = 6 - agno = 7 - agno = 0 entry "Mythic Quest - S03E04 - The Two Joes WEBDL-1080p.nfo" at block 0 offset 1024 in directory inode 4300967801 references free inode 6484102721 would clear inode number in entry at offset 1024... data fork in regular inode 6484102721 claims used block 810512573 correcting nextents for inode 6484102721 bad data fork in inode 6484102721 would have cleared inode 6484102721 entry "HTTPCookies" in shortform directory 18449153 references free inode 18449155 would have junked entry "HTTPCookies" in directory inode 18449153 Bad mtime nsec 1040198661 on inode 18449155, would reset to zero Bad ctime nsec 1040198661 on inode 18449155, would reset to zero data fork in regular inode 18449155 claims used block 2306194 correcting nextents for inode 18449155 bad data fork in inode 18449155 would have cleared inode 18449155 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... entry "HTTPCookies" in shortform directory inode 18449153 points to free inode 18449155 would junk entry entry "Mythic Quest - S03E04 - The Two Joes WEBDL-1080p.nfo" in directory inode 4300967801 points to free inode 6484102721, would junk entry bad hash table for directory inode 4300967801 (no data entry): would rebuild would rebuild directory inode 4300967801 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... Maximum metadata LSN (3:3811639) is ahead of log (3:3811361). Would format log to cycle 6. No modify flag set, skipping filesystem flush and exiting. Going to run it with out the "-u" flag next.
December 8, 20241 yr Author When I ran it with out the -u it gave me: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this. So stopping the array and restarting it without the maintenance mode flag. aaaaaand still unmountable, going to stop/maintenance mode start to run it with the -L option.
December 8, 20241 yr Author Not encouraging... Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. fatal error -- failed to clear log Trying again (definition of insanity)
December 8, 20241 yr Author incremental improvement and got some results Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... clearing needsrepair flag and regenerating metadata sb_fdblocks 34958915, counted 36912366 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 data fork in ino 1036865 claims free block 131018 data fork in ino 1036866 claims free block 154703 data fork in ino 1036878 claims free block 154847 data fork in ino 1036879 claims free block 154874 data fork in ino 1036880 claims free block 154877 data fork in ino 18449155 claims free block 2306193 data fork in ino 34267989 claims free block 4283550 data fork in ino 47864192 claims free block 5983126 - agno = 1 Bad mtime nsec 1043052783 on inode 2148134387, resetting to zero Bad ctime nsec 1043052783 on inode 2148134387, resetting to zero data fork in regular inode 2148134387 claims used block 268516601 correcting nextents for inode 2148134387 bad data fork in inode 2148134387 cleared inode 2148134387 data fork in regular inode 2148137103 claims used block 268516954 correcting nextents for inode 2148137103 bad data fork in inode 2148137103 cleared inode 2148137103 data fork in regular inode 2148137117 claims used block 268516959 correcting nextents for inode 2148137117 bad data fork in inode 2148137117 cleared inode 2148137117 - agno = 2 - agno = 3 data fork in regular inode 6484102721 claims used block 810512573 correcting nextents for inode 6484102721 bad data fork in inode 6484102721 cleared inode 6484102721 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 6 - agno = 1 - agno = 3 - agno = 7 - agno = 5 - agno = 4 entry "Mythic Quest - S03E04 - The Two Joes WEBDL-1080p.nfo" at block 0 offset 1024 in directory inode 4300967801 references free inode 6484102721 clearing inode number in entry at offset 1024... entry "Sleepy Hollow - S01E06 - The Sin Eater Bluray-720p.nfo" at block 0 offset 1576 in directory inode 2147990936 references free inode 2148134387 clearing inode number in entry at offset 1576... entry "24 - S09E10 - Day 9- 8-00 P.M. - 9-00 P.M Bluray-1080p.srt" at block 0 offset 1928 in directory inode 2148149515 references free inode 2148137117 clearing inode number in entry at offset 1928... entry "Good Eats - S07E18 - I Pie SDTV.nfo" at block 0 offset 3208 in directory inode 18462929 references free inode 2148137103 clearing inode number in entry at offset 3208... Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... bad hash table for directory inode 18462929 (no data entry): rebuilding rebuilding directory inode 18462929 bad hash table for directory inode 2147990936 (no data entry): rebuilding rebuilding directory inode 2147990936 bad hash table for directory inode 2148149515 (no data entry): rebuilding rebuilding directory inode 2148149515 bad hash table for directory inode 4300967801 (no data entry): rebuilding rebuilding directory inode 4300967801 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (3:3811660) is ahead of log (1:2). Format log to cycle 6. done Restarted and mounted the hard drives but it is still showing up as unmountable. Going back to Maintenance mode and running the check again this time without any flags
December 8, 20241 yr Author Another round of scanning results: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... Metadata CRC error detected at 0x44228d, xfs_bnobt block 0x74725738/0x1000 btree block 1/17652 is suspect, error -74 bad magic # 0x3c657069 in btbno block 1/17652 Metadata CRC error detected at 0x44228d, xfs_cntbt block 0x7477a3e0/0x1000 btree block 1/61065 is suspect, error -74 bad magic # 0x3c657069 in btcnt block 1/61065 agf_freeblks 3859698, counted 0 in ag 1 agf_longest 268674, counted 0 in ag 1 agf_btreeblks 4, counted 0 in ag 1 Metadata CRC error detected at 0x44228d, xfs_cntbt block 0x2470faa80/0x1000 btree block 5/1824657 is suspect, error -74 bad magic # 0x7b226974 in btcnt block 5/1824657 agf_freeblks 4255488, counted 0 in ag 5 agf_longest 384296, counted 0 in ag 5 Metadata CRC error detected at 0x47191d, xfs_inobt block 0x2470fa930/0x1000 btree block 5/1824615 is suspect, error -74 bad magic # 0x7b226974 in inobt block 5/1824615 Metadata CRC error detected at 0x47191d, xfs_finobt block 0x2470fa988/0x1000 btree block 5/1824626 is suspect, error -74 bad magic # 0x7b226974 in finobt block 5/1824626 agi_count 4096, counted 0 in ag 5 agi_freecount 1271, counted 0 in ag 5 agi_freecount 1271, counted 0 in ag 5 finobt Metadata CRC error detected at 0x47191d, xfs_inobt block 0x2baa11df0/0x1000 btree block 6/12 is suspect, error -74 bad magic # 0x7b227374 in inobt block 6/12 Metadata CRC error detected at 0x47191d, xfs_finobt block 0x2bac3a398/0x1000 btree block 6/282817 is suspect, error -74 bad magic # 0x3c747673 in finobt block 6/282817 agi_count 3648, counted 0 in ag 6 agi_freecount 1101, counted 0 in ag 6 agi_freecount 1101, counted 0 in ag 6 finobt Metadata CRC error detected at 0x47191d, xfs_inobt block 0x32f114da0/0x1000 btree block 7/15 is suspect, error -74 bad magic # 0x23204e65 in inobt block 7/15 Metadata corruption detected at 0x441fa0, xfs_cntbt block 0x1d1efff18/0x1000 btree block 4/387095 is suspect, error -117 bad magic # 0x49414233 in btcnt block 4/387095 agf_freeblks 4230033, counted 7514 in ag 4 agf_longest 229744, counted 261 in ag 4 agf_btreeblks 4, counted 3 in ag 4 Metadata CRC error detected at 0x44228d, xfs_bnobt block 0x15d509990/0x1000 btree block 3/345 is suspect, error -74 bad magic # 0x7b226974 in btbno block 3/345 Metadata CRC error detected at 0x44228d, xfs_cntbt block 0x15d7346f0/0x1000 btree block 3/284421 is suspect, error -74 bad magic # 0x7b227374 in btcnt block 3/284421 agf_freeblks 7102899, counted 0 in ag 3 agf_longest 738752, counted 0 in ag 3 agf_btreeblks 4, counted 0 in ag 3 Metadata CRC error detected at 0x44228d, xfs_bnobt block 0xe9448770/0x1000 btree block 2/820488 is suspect, error -74 bad magic # 0x7b226974 in btbno block 2/820488 agf_btreeblks 4, counted 2 in ag 2 Metadata CRC error detected at 0x47191d, xfs_finobt block 0xe91aa0d8/0x1000 btree block 2/477237 is suspect, error -74 bad magic # 0x3c657069 in finobt block 2/477237 agi_freecount 1201, counted 0 in ag 2 finobt Metadata CRC error detected at 0x47191d, xfs_finobt block 0x32f307190/0x1000 btree block 7/255117 is suspect, error -74 bad magic # 0x3c657069 in finobt block 7/255117 agi_count 4224, counted 0 in ag 7 agi_freecount 1108, counted 0 in ag 7 agi_freecount 1108, counted 0 in ag 7 finobt Metadata CRC error detected at 0x44228d, xfs_cntbt block 0x173200/0x1000 btree block 0/190016 is suspect, error -74 bad magic # 0x3c657069 in btcnt block 0/190016 agf_freeblks 4253107, counted 0 in ag 0 agf_longest 493152, counted 0 in ag 0 agf_btreeblks 4, counted 2 in ag 0 Metadata CRC error detected at 0x47191d, xfs_inobt block 0x58/0x1000 btree block 0/11 is suspect, error -74 bad magic # 0x3c657069 in inobt block 0/11 Metadata CRC error detected at 0x47191d, xfs_finobt block 0x12eec0/0x1000 btree block 0/155096 is suspect, error -74 bad magic # 0x3c657069 in finobt block 0/155096 agi_count 10560, counted 0 in ag 0 agi_freecount 1306, counted 0 in ag 0 agi_freecount 1306, counted 0 in ag 0 finobt sb_icount 52832, counted 30304 sb_ifree 9714, counted 4928 sb_fdblocks 36912384, counted 13218660 root inode chunk not found Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 correcting imap correcting imap correcting imap correcting imap correcting imap - agno = 1 correcting imap correcting imap correcting imap - agno = 2 Metadata CRC error detected at 0x45fb48, xfs_dir3_block block 0xe940fec8/0x1000 bad directory block magic # 0x3c657069 in block 0 for directory inode 4300967801 corrupt block 0 in directory inode 4300967801 will junk block no . entry for directory 4300967801 no .. entry for directory 4300967801 problem with directory contents in inode 4300967801 cleared inode 4300967801 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... ... A Huge set of these just with diffrent addresses correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap 152f93588680: Badness in key lookup (length) bp=(bno 0xe6a40, len 4096 bytes) key=(bno 0xe6a40, len 16384 bytes) 152f93588680: Badness in key lookup (length) bp=(bno 0xe6a60, len 4096 bytes) key=(bno 0xe6a60, len 16384 bytes) correcting imap correcting imap correcting imap correcting imap ... found inodes not in the inode allocation tree Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 5 - agno = 2 - agno = 3 - agno = 4 - agno = 7 - agno = 6 entry "Season 03" in shortform directory 46999073 references free inode 4300967801 junking entry "Season 03" in directory inode 46999073 corrected i8 count in directory 46999073, was 1, now 0 corrected directory 46999073 size, was 171, now 139 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... fixing ftype mismatch (0/2) in directory/child inode 46999073/2147990948 fixing ftype mismatch (0/1) in directory/child inode 46999073/46999074 fixing ftype mismatch (0/1) in directory/child inode 46999073/46999075 fixing ftype mismatch (0/1) in directory/child inode 46999073/46999076 fixing ftype mismatch (0/1) in directory/child inode 46999073/46999091 fixing ftype mismatch (0/2) in directory/child inode 46999073/2147700745 fixing ftype mismatch (0/1) in directory/child inode 46999073/256299545 - traversal finished ... - moving disconnected inodes to lost+found ... disconnected inode 2148137103, moving to lost+found disconnected inode 2148137117, moving to lost+found disconnected inode 4300967750, moving to lost+found disconnected inode 4300967751, moving to lost+found disconnected inode 4300967754, moving to lost+found disconnected inode 4300967757, moving to lost+found disconnected inode 4300967758, moving to lost+found disconnected inode 4300967769, moving to lost+found disconnected inode 4300967772, moving to lost+found disconnected inode 4300967773, moving to lost+found disconnected inode 4300967776, moving to lost+found disconnected inode 4300967802, moving to lost+found disconnected inode 4300967803, moving to lost+found disconnected inode 4300967804, moving to lost+found disconnected inode 4300967805, moving to lost+found disconnected inode 4300969398, moving to lost+found disconnected inode 4300969400, moving to lost+found disconnected inode 6484042804, moving to lost+found disconnected inode 6484102720, moving to lost+found disconnected inode 6484103352, moving to lost+found disconnected inode 6484103353, moving to lost+found disconnected inode 6484103354, moving to lost+found disconnected inode 6484103889, moving to lost+found disconnected inode 6484103911, moving to lost+found disconnected inode 6484103914, moving to lost+found disconnected inode 6484103917, moving to lost+found disconnected inode 6484104914, moving to lost+found disconnected inode 6484104915, moving to lost+found disconnected inode 6484104916, moving to lost+found disconnected inode 6484105620, moving to lost+found disconnected inode 6484105621, moving to lost+found Phase 7 - verify and correct link counts... resetting inode 46999073 nlinks from 5 to 4 Maximum metadata LSN (1819239275:1933324148) is ahead of log (6:4). Format log to cycle 1819239278. done still unmountable.....
December 8, 20241 yr Author and another run.... Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... Log inconsistent (didn't find previous header) failed to find log head zero_log: cannot find log head/tail (xlog_find_tail=5) - scan filesystem freespace and inode maps... block (2,476935-476936) multiply claimed by bno space tree, state - 2 Metadata CRC error detected at 0x44228d, xfs_cntbt block 0x173200/0x1000 btree block 0/190016 is suspect, error -74 bad magic # 0x3c657069 in btcnt block 0/190016 agf_freeblks 4253106, counted 0 in ag 0 agf_longest 493152, counted 0 in ag 0 agf_btreeblks 4, counted 2 in ag 0 block (1,8649-8650) multiply claimed by bno space tree, state - 2 Metadata CRC error detected at 0x44228d, xfs_cntbt block 0x7477a3e0/0x1000 btree block 1/61065 is suspect, error -74 bad magic # 0x3c657069 in btcnt block 1/61065 agf_freeblks 3859683, counted 0 in ag 1 agf_longest 268674, counted 0 in ag 1 agf_btreeblks 4, counted 2 in ag 1 agi_freecount 1290, counted 1293 in ag 1 agi_freecount 1290, counted 1293 in ag 1 finobt agf_freeblks 3009942, counted 3009941 in ag 2 Metadata CRC error detected at 0x47191d, xfs_finobt block 0xe91aa0d8/0x1000 btree block 2/477237 is suspect, error -74 bad magic # 0x3c657069 in finobt block 2/477237 agi_freecount 1202, counted 0 in ag 2 finobt sb_ifree 9711, counted 9714 sb_fdblocks 36912369, counted 28799575 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 Metadata CRC error detected at 0x45fb48, xfs_dir3_block block 0xe219b8/0x1000 bad directory block magic # 0x494e81fd in block 0 for directory inode 14817728 corrupt block 0 in directory inode 14817728 would junk block no . entry for directory 14817728 no .. entry for directory 14817728 problem with directory contents in inode 14817728 would have cleared inode 14817728 - agno = 1 imap claims in-use inode 2148134387 is free, would correct imap imap claims in-use inode 2148137103 is free, would correct imap imap claims in-use inode 2148137117 is free, would correct imap - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... free space (0,1-8) only seen by one free space btree free space (0,161980-161982) only seen by one free space btree free space (0,189649-189650) only seen by one free space btree ... a lot of these ... free space (1,244179761-244179823) only seen by one free space btree free space (2,791539-791539) only seen by one free space btree - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 1 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 entry "lost+found" in shortform directory 128 references free inode 14817728 would have junked entry "lost+found" in directory inode 128 bad directory block magic # 0x494e81fd in block 0 for directory inode 14817728 corrupt block 0 in directory inode 14817728 would junk block no . entry for directory 14817728 no .. entry for directory 14817728 problem with directory contents in inode 14817728 would have cleared inode 14817728 No modify flag set, skipping phase 5 Inode allocation btrees are too corrupted, skipping phases 6 and 7 Maximum metadata LSN (1698583137:1769171310) is ahead of log (0:0). Would format log to cycle 1698583140. No modify flag set, skipping filesystem flush and exiting.
December 8, 20241 yr Community Expert I would run memtest just to make there aren't any obvious RAM issues, if no errors, run xfs_repair again and take the opportunity to make sure the backups are up-to-date, if issues persist with just that filesystem, I would backup and re-format.
December 8, 20241 yr Author Latest run resulted in: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... Log inconsistent (didn't find previous header) failed to find log head zero_log: cannot find log head/tail (xlog_find_tail=5) - scan filesystem freespace and inode maps... clearing needsrepair flag and regenerating metadata block (2,476935-476936) multiply claimed by bno space tree, state - 2 Metadata CRC error detected at 0x44228d, xfs_cntbt block 0x173200/0x1000 btree block 0/190016 is suspect, error -74 bad magic # 0x3c657069 in btcnt block 0/190016 agf_freeblks 4253106, counted 0 in ag 0 agf_longest 493152, counted 0 in ag 0 agf_btreeblks 4, counted 2 in ag 0 block (1,8649-8650) multiply claimed by bno space tree, state - 2 agf_freeblks 3009942, counted 3009941 in ag 2 Metadata CRC error detected at 0x47191d, xfs_finobt block 0xe91aa0d8/0x1000 btree block 2/477237 is suspect, error -74 bad magic # 0x3c657069 in finobt block 2/477237 agi_freecount 1202, counted 0 in ag 2 finobt Metadata CRC error detected at 0x44228d, xfs_cntbt block 0x7477a3e0/0x1000 btree block 1/61065 is suspect, error -74 bad magic # 0x3c657069 in btcnt block 1/61065 agf_freeblks 3859683, counted 0 in ag 1 agf_longest 268674, counted 0 in ag 1 agf_btreeblks 4, counted 2 in ag 1 agi_freecount 1290, counted 1293 in ag 1 agi_freecount 1290, counted 1293 in ag 1 finobt sb_ifree 9711, counted 9714 sb_fdblocks 36912369, counted 28799575 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 Metadata CRC error detected at 0x45fb48, xfs_dir3_block block 0xe219b8/0x1000 bad directory block magic # 0x494e81fd in block 0 for directory inode 14817728 corrupt block 0 in directory inode 14817728 will junk block no . entry for directory 14817728 no .. entry for directory 14817728 problem with directory contents in inode 14817728 cleared inode 14817728 - agno = 1 correcting imap correcting imap correcting imap - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 4 entry "lost+found" in shortform directory 128 references free inode 14817728 - agno = 1 - agno = 3 - agno = 5 - agno = 2 - agno = 6 - agno = 7 junking entry "lost+found" in directory inode 128 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... disconnected inode 2148137103, moving to lost+found disconnected inode 2148137117, moving to lost+found disconnected inode 4300967750, moving to lost+found disconnected inode 4300967751, moving to lost+found disconnected inode 4300967754, moving to lost+found disconnected inode 4300967757, moving to lost+found disconnected inode 4300967758, moving to lost+found disconnected inode 4300967769, moving to lost+found disconnected inode 4300967772, moving to lost+found disconnected inode 4300967773, moving to lost+found disconnected inode 4300967776, moving to lost+found disconnected inode 4300967802, moving to lost+found disconnected inode 4300967803, moving to lost+found disconnected inode 4300967804, moving to lost+found disconnected inode 4300967805, moving to lost+found disconnected inode 4300969398, moving to lost+found disconnected inode 4300969400, moving to lost+found disconnected inode 6484042804, moving to lost+found disconnected inode 6484102720, moving to lost+found disconnected inode 6484103352, moving to lost+found disconnected inode 6484103353, moving to lost+found disconnected inode 6484103354, moving to lost+found disconnected inode 6484103889, moving to lost+found disconnected inode 6484103911, moving to lost+found disconnected inode 6484103914, moving to lost+found disconnected inode 6484103917, moving to lost+found disconnected inode 6484104914, moving to lost+found disconnected inode 6484104915, moving to lost+found disconnected inode 6484104916, moving to lost+found disconnected inode 6484105620, moving to lost+found disconnected inode 6484105621, moving to lost+found Phase 7 - verify and correct link counts... resetting inode 128 nlinks from 11 to 10 Maximum metadata LSN (1698583137:1769171310) is ahead of log (1:2). Format log to cycle 1698583140. done
December 8, 20241 yr Author That last run seems to have done it, everything is mounted now and only about 30 files in the lost+found, most of which look to be some "linux ios's" that I should be able to recover. Thanks JorgeB for the suggestions, I'll put through a memtest next to see if there is anything else going on. Any comments as to why I needed to run the fix so many times to get the file system backup? is that (semi) normal or is it a sign of other issues? Just want to know if or where I should look for any other issues that may be hiding under the surface. Edited December 8, 20241 yr by armagedon41
December 9, 20241 yr Community Expert Could have just been a very corrupt filesystem, but if it keeps happening to that one I would reformat the disk, and if it happens with more disks there could be an underlying hardware issue
December 9, 20241 yr Author There does seem to be an issue with this drive/file system, I tried to get a parity check going but it ended up running just as slow as before, running in the KB/s range. I paused it and I have been using unbalance to move things off of the drive but it to is transferring very slow, I have managed to get everything that I was worried about (things that didn't have proper backups or would be hard to replace), There is still a lot of data and would be annoying to recover from but not devastating. By reformatting the disk are you referring to pulling the drive, then clearing it outside of the array, then popping it back in to rebuild? I was hoping to be able to recover without degrading the array while things rebuild but if that is what it will take....
December 9, 20241 yr Community Expert 17 minutes ago, armagedon41 said: By reformatting the disk are you referring to pulling the drive, then clearing it outside of the array, then popping it back in to rebuild? No, you can re-format in the array, but if it's slow, it may be a good idea run a preclear first.
December 17, 20241 yr Author Alright still on this journey, the disk unmounted again and I was able to scan and get it up again. So I started to play with the drive and things were still going slow until it went to emulated, I took this opportunity to move everything off the drive so it is now empty. Once the drive was empty and everything was still emulated I dropped the disk and re-added, the build ran for a bit but ran a bit slower then I would expected (estimated about 32-48 hours for a 12 tb array) and when it was 80% complete it failed with read errors and when back to emulated. I then pulled the drive and tried to rebuild with a different drive (I don't 100% trust this this new drive but it was the only other drive I have on hand) and this one was rebuilding faster (about 24-28 hours) but at only 15% it failed with read errors as well. Bellow is a chunk of the disk log that is sitting on that drive now. I'm just trying to confirm what might be the best course of action, does the read errors look like a failing drive, this would be 2 drives both showing up as faulty on the same area, not fully unexpected on the second drive since that was an untrusted disk but still odd. If I have to pick up another drive then so be it, but rather not have to throw cash at it right now if it could be likely resolved with existing hardware. Right now I am thinking of just swapping out the cable and port on the mobo and trying to rebuild once more (remove and re-add). ... Dec 17 09:26:02 Oden kernel: ata4.00: exception Emask 0x10 SAct 0x780 SErr 0x90200 action 0xe frozen Dec 17 09:26:02 Oden kernel: ata4.00: irq_stat 0x00400000, PHY RDY changed Dec 17 09:26:02 Oden kernel: ata4: SError: { Persist PHYRdyChg 10B8B } Dec 17 09:26:02 Oden kernel: ata4.00: failed command: READ FPDMA QUEUED Dec 17 09:26:02 Oden kernel: ata4.00: cmd 60/10:38:68:00:00/00:00:00:00:00/40 tag 7 ncq dma 8192 in Dec 17 09:26:02 Oden kernel: ata4.00: status: { DRDY } Dec 17 09:26:02 Oden kernel: ata4.00: failed command: READ FPDMA QUEUED Dec 17 09:26:02 Oden kernel: ata4.00: cmd 60/30:40:88:00:00/00:00:00:00:00/40 tag 8 ncq dma 24576 in Dec 17 09:26:02 Oden kernel: ata4.00: status: { DRDY } Dec 17 09:26:02 Oden kernel: ata4.00: failed command: READ FPDMA QUEUED Dec 17 09:26:02 Oden kernel: ata4.00: cmd 60/78:48:c8:00:00/00:00:00:00:00/40 tag 9 ncq dma 61440 in Dec 17 09:26:02 Oden kernel: ata4.00: status: { DRDY } Dec 17 09:26:02 Oden kernel: ata4.00: failed command: READ FPDMA QUEUED Dec 17 09:26:02 Oden kernel: ata4.00: cmd 60/f8:50:48:01:00/00:00:00:00:00/40 tag 10 ncq dma 126976 in Dec 17 09:26:02 Oden kernel: ata4.00: status: { DRDY } Dec 17 09:26:02 Oden kernel: ata4: hard resetting link Dec 17 09:26:08 Oden kernel: ata4: link is slow to respond, please be patient (ready=0) Dec 17 09:26:12 Oden kernel: ata4: found unknown device (class 0) Dec 17 09:26:12 Oden kernel: ata4: found unknown device (class 0) Dec 17 09:26:12 Oden kernel: ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Dec 17 09:26:15 Oden kernel: ata4.00: configured for UDMA/33 Dec 17 09:26:15 Oden kernel: sd 4:0:0:0: [sdg] tag#7 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=12s Dec 17 09:26:15 Oden kernel: sd 4:0:0:0: [sdg] tag#7 Sense Key : 0x5 [current] Dec 17 09:26:15 Oden kernel: sd 4:0:0:0: [sdg] tag#7 ASC=0x21 ASCQ=0x4 Dec 17 09:26:15 Oden kernel: sd 4:0:0:0: [sdg] tag#7 CDB: opcode=0x88 88 00 00 00 00 00 00 00 00 68 00 00 00 10 00 00 Dec 17 09:26:15 Oden kernel: I/O error, dev sdg, sector 104 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2 Dec 17 09:26:15 Oden kernel: sd 4:0:0:0: [sdg] tag#8 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=12s Dec 17 09:26:15 Oden kernel: sd 4:0:0:0: [sdg] tag#8 Sense Key : 0x5 [current] Dec 17 09:26:15 Oden kernel: sd 4:0:0:0: [sdg] tag#8 ASC=0x21 ASCQ=0x4 Dec 17 09:26:15 Oden kernel: sd 4:0:0:0: [sdg] tag#8 CDB: opcode=0x88 88 00 00 00 00 00 00 00 00 88 00 00 00 30 00 00 Dec 17 09:26:15 Oden kernel: I/O error, dev sdg, sector 136 op 0x0:(READ) flags 0x80700 phys_seg 6 prio class 2 Dec 17 09:26:15 Oden kernel: sd 4:0:0:0: [sdg] tag#9 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=12s Dec 17 09:26:15 Oden kernel: sd 4:0:0:0: [sdg] tag#9 Sense Key : 0x5 [current] Dec 17 09:26:15 Oden kernel: sd 4:0:0:0: [sdg] tag#9 ASC=0x21 ASCQ=0x4 Dec 17 09:26:15 Oden kernel: sd 4:0:0:0: [sdg] tag#9 CDB: opcode=0x88 88 00 00 00 00 00 00 00 00 c8 00 00 00 78 00 00 Dec 17 09:26:15 Oden kernel: I/O error, dev sdg, sector 200 op 0x0:(READ) flags 0x80700 phys_seg 15 prio class 2 Dec 17 09:26:15 Oden kernel: sd 4:0:0:0: [sdg] tag#10 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=12s Dec 17 09:26:15 Oden kernel: sd 4:0:0:0: [sdg] tag#10 Sense Key : 0x5 [current] Dec 17 09:26:15 Oden kernel: sd 4:0:0:0: [sdg] tag#10 ASC=0x21 ASCQ=0x4 Dec 17 09:26:15 Oden kernel: sd 4:0:0:0: [sdg] tag#10 CDB: opcode=0x88 88 00 00 00 00 00 00 00 01 48 00 00 00 f8 00 00 Dec 17 09:26:15 Oden kernel: I/O error, dev sdg, sector 328 op 0x0:(READ) flags 0x80700 phys_seg 30 prio class 2 Dec 17 09:26:15 Oden kernel: ata4: EH complete Dec 17 09:26:15 Oden kernel: ata4.00: exception Emask 0x10 SAct 0x2000 SErr 0x90200 action 0xe frozen Dec 17 09:26:15 Oden kernel: ata4.00: irq_stat 0x00400000, PHY RDY changed Dec 17 09:26:15 Oden kernel: ata4: SError: { Persist PHYRdyChg 10B8B } Dec 17 09:26:15 Oden kernel: ata4.00: failed command: READ FPDMA QUEUED Dec 17 09:26:15 Oden kernel: ata4.00: cmd 60/00:68:40:26:81/02:00:a3:03:00/40 tag 13 ncq dma 262144 in Dec 17 09:26:15 Oden kernel: ata4.00: status: { DRDY } Dec 17 09:26:15 Oden kernel: ata4: hard resetting link Dec 17 09:26:21 Oden kernel: ata4: link is slow to respond, please be patient (ready=0) Dec 17 09:26:25 Oden kernel: ata4: found unknown device (class 0) Dec 17 09:26:25 Oden kernel: ata4: found unknown device (class 0) Dec 17 09:26:25 Oden kernel: ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Dec 17 09:26:28 Oden kernel: ata4.00: configured for UDMA/33 Dec 17 09:26:28 Oden kernel: sd 4:0:0:0: [sdg] tag#13 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=12s Dec 17 09:26:28 Oden kernel: sd 4:0:0:0: [sdg] tag#13 Sense Key : 0x5 [current] Dec 17 09:26:28 Oden kernel: sd 4:0:0:0: [sdg] tag#13 ASC=0x21 ASCQ=0x4 Dec 17 09:26:28 Oden kernel: sd 4:0:0:0: [sdg] tag#13 CDB: opcode=0x88 88 00 00 00 00 03 a3 81 26 40 00 00 02 00 00 00 Dec 17 09:26:28 Oden kernel: I/O error, dev sdg, sector 15628052032 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 2 Dec 17 09:26:28 Oden kernel: ata4: EH complete Dec 17 09:40:02 Oden kernel: ata4.00: exception Emask 0x10 SAct 0x8000000 SErr 0x90200 action 0xe frozen Dec 17 09:40:02 Oden kernel: ata4.00: irq_stat 0x00400000, PHY RDY changed Dec 17 09:40:02 Oden kernel: ata4: SError: { Persist PHYRdyChg 10B8B } Dec 17 09:40:02 Oden kernel: ata4.00: failed command: READ FPDMA QUEUED Dec 17 09:40:02 Oden kernel: ata4.00: cmd 60/08:d8:80:00:00/00:00:00:00:00/40 tag 27 ncq dma 4096 in Dec 17 09:40:02 Oden kernel: ata4.00: status: { DRDY } Dec 17 09:40:02 Oden kernel: ata4: hard resetting link Dec 17 09:40:08 Oden kernel: ata4: link is slow to respond, please be patient (ready=0) Dec 17 09:40:12 Oden kernel: ata4: found unknown device (class 0) Dec 17 09:40:12 Oden kernel: ata4: found unknown device (class 0) Dec 17 09:40:12 Oden kernel: ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Dec 17 09:40:14 Oden kernel: ata4.00: configured for UDMA/33 Dec 17 09:40:14 Oden kernel: sd 4:0:0:0: [sdg] tag#27 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=12s Dec 17 09:40:14 Oden kernel: sd 4:0:0:0: [sdg] tag#27 Sense Key : 0x5 [current] Dec 17 09:40:14 Oden kernel: sd 4:0:0:0: [sdg] tag#27 ASC=0x21 ASCQ=0x4 Dec 17 09:40:14 Oden kernel: sd 4:0:0:0: [sdg] tag#27 CDB: opcode=0x88 88 00 00 00 00 00 00 00 00 80 00 00 00 08 00 00 Dec 17 09:40:14 Oden kernel: I/O error, dev sdg, sector 128 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 Dec 17 09:40:14 Oden kernel: ata4: EH complete Dec 17 09:40:15 Oden kernel: ata4.00: exception Emask 0x10 SAct 0x10000 SErr 0x90200 action 0xe frozen Dec 17 09:40:15 Oden kernel: ata4.00: irq_stat 0x00400000, PHY RDY changed Dec 17 09:40:15 Oden kernel: ata4: SError: { Persist PHYRdyChg 10B8B } Dec 17 09:40:15 Oden kernel: ata4.00: failed command: READ FPDMA QUEUED Dec 17 09:40:15 Oden kernel: ata4.00: cmd 60/00:80:40:26:81/02:00:a3:03:00/40 tag 16 ncq dma 262144 in Dec 17 09:40:15 Oden kernel: ata4.00: status: { DRDY } Dec 17 09:40:15 Oden kernel: ata4: hard resetting link Dec 17 09:40:21 Oden kernel: ata4: link is slow to respond, please be patient (ready=0) Dec 17 09:40:25 Oden kernel: ata4: found unknown device (class 0) Dec 17 09:40:25 Oden kernel: ata4: found unknown device (class 0) Dec 17 09:40:25 Oden kernel: ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Dec 17 09:40:28 Oden kernel: ata4.00: configured for UDMA/33 Dec 17 09:40:28 Oden kernel: sd 4:0:0:0: [sdg] tag#16 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=13s Dec 17 09:40:28 Oden kernel: sd 4:0:0:0: [sdg] tag#16 Sense Key : 0x5 [current] Dec 17 09:40:28 Oden kernel: sd 4:0:0:0: [sdg] tag#16 ASC=0x21 ASCQ=0x4 Dec 17 09:40:28 Oden kernel: sd 4:0:0:0: [sdg] tag#16 CDB: opcode=0x88 88 00 00 00 00 03 a3 81 26 40 00 00 02 00 00 00 Dec 17 09:40:28 Oden kernel: I/O error, dev sdg, sector 15628052032 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 2 Dec 17 09:40:28 Oden kernel: ata4: EH complete Dec 17 09:45:02 Oden kernel: ata4.00: exception Emask 0x10 SAct 0x400000 SErr 0x40d0202 action 0xe frozen Dec 17 09:45:02 Oden kernel: ata4.00: irq_stat 0x00400000, PHY RDY changed Dec 17 09:45:02 Oden kernel: ata4: SError: { RecovComm Persist PHYRdyChg CommWake 10B8B DevExch } Dec 17 09:45:02 Oden kernel: ata4.00: failed command: READ FPDMA QUEUED Dec 17 09:45:02 Oden kernel: ata4.00: cmd 60/08:b0:78:00:00/00:00:00:00:00/40 tag 22 ncq dma 4096 in Dec 17 09:45:02 Oden kernel: ata4.00: status: { DRDY } Dec 17 09:45:02 Oden kernel: ata4: hard resetting link Dec 17 09:45:08 Oden kernel: ata4: link is slow to respond, please be patient (ready=0) Dec 17 09:45:12 Oden kernel: ata4: found unknown device (class 0) Dec 17 09:45:12 Oden kernel: ata4: found unknown device (class 0) Dec 17 09:45:12 Oden kernel: ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Dec 17 09:45:14 Oden kernel: ata4.00: configured for UDMA/33 Dec 17 09:45:14 Oden kernel: sd 4:0:0:0: [sdg] tag#22 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=12s Dec 17 09:45:14 Oden kernel: sd 4:0:0:0: [sdg] tag#22 Sense Key : 0x5 [current] Dec 17 09:45:14 Oden kernel: sd 4:0:0:0: [sdg] tag#22 ASC=0x21 ASCQ=0x4 Dec 17 09:45:14 Oden kernel: sd 4:0:0:0: [sdg] tag#22 CDB: opcode=0x88 88 00 00 00 00 00 00 00 00 78 00 00 00 08 00 00 Dec 17 09:45:14 Oden kernel: I/O error, dev sdg, sector 120 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 Dec 17 09:45:14 Oden kernel: ata4: EH complete Dec 17 09:45:15 Oden kernel: ata4.00: exception Emask 0x10 SAct 0x1000 SErr 0x90200 action 0xe frozen Dec 17 09:45:15 Oden kernel: ata4.00: irq_stat 0x00400000, PHY RDY changed Dec 17 09:45:15 Oden kernel: ata4: SError: { Persist PHYRdyChg 10B8B } Dec 17 09:45:15 Oden kernel: ata4.00: failed command: READ FPDMA QUEUED Dec 17 09:45:15 Oden kernel: ata4.00: cmd 60/00:60:40:26:81/02:00:a3:03:00/40 tag 12 ncq dma 262144 in Dec 17 09:45:15 Oden kernel: ata4.00: status: { DRDY } Dec 17 09:45:15 Oden kernel: ata4: hard resetting link Dec 17 09:45:21 Oden kernel: ata4: link is slow to respond, please be patient (ready=0) Dec 17 09:45:25 Oden kernel: ata4: found unknown device (class 0) Dec 17 09:45:25 Oden kernel: ata4: found unknown device (class 0) Dec 17 09:45:25 Oden kernel: ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Dec 17 09:45:28 Oden kernel: ata4.00: configured for UDMA/33 Dec 17 09:45:28 Oden kernel: sd 4:0:0:0: [sdg] tag#12 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=13s Dec 17 09:45:28 Oden kernel: sd 4:0:0:0: [sdg] tag#12 Sense Key : 0x5 [current] Dec 17 09:45:28 Oden kernel: sd 4:0:0:0: [sdg] tag#12 ASC=0x21 ASCQ=0x4 Dec 17 09:45:28 Oden kernel: sd 4:0:0:0: [sdg] tag#12 CDB: opcode=0x88 88 00 00 00 00 03 a3 81 26 40 00 00 02 00 00 00 Dec 17 09:45:28 Oden kernel: I/O error, dev sdg, sector 15628052032 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 2 Dec 17 09:45:28 Oden kernel: ata4: EH complete Dec 17 10:35:02 Oden kernel: ata4.00: exception Emask 0x10 SAct 0x2000 SErr 0x90200 action 0xe frozen Dec 17 10:35:02 Oden kernel: ata4.00: irq_stat 0x00400000, PHY RDY changed Dec 17 10:35:02 Oden kernel: ata4: SError: { Persist PHYRdyChg 10B8B } Dec 17 10:35:02 Oden kernel: ata4.00: failed command: READ FPDMA QUEUED Dec 17 10:35:02 Oden kernel: ata4.00: cmd 60/00:68:40:26:81/02:00:a3:03:00/40 tag 13 ncq dma 262144 in Dec 17 10:35:02 Oden kernel: ata4.00: status: { DRDY } Dec 17 10:35:02 Oden kernel: ata4: hard resetting link Dec 17 10:35:08 Oden kernel: ata4: link is slow to respond, please be patient (ready=0) Dec 17 10:35:13 Oden kernel: ata4: found unknown device (class 0) Dec 17 10:35:13 Oden kernel: ata4: found unknown device (class 0) Dec 17 10:35:13 Oden kernel: ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Dec 17 10:35:15 Oden kernel: ata4.00: configured for UDMA/33 Dec 17 10:35:15 Oden kernel: sd 4:0:0:0: [sdg] tag#13 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=12s Dec 17 10:35:15 Oden kernel: sd 4:0:0:0: [sdg] tag#13 Sense Key : 0x5 [current] Dec 17 10:35:15 Oden kernel: sd 4:0:0:0: [sdg] tag#13 ASC=0x21 ASCQ=0x4 Dec 17 10:35:15 Oden kernel: sd 4:0:0:0: [sdg] tag#13 CDB: opcode=0x88 88 00 00 00 00 03 a3 81 26 40 00 00 02 00 00 00 Dec 17 10:35:15 Oden kernel: I/O error, dev sdg, sector 15628052032 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 2 Dec 17 10:35:15 Oden kernel: ata4: EH complete
December 17, 20241 yr Community Expert Solution Looks more like a power/connection issue, replace both cables and retry, if the same try with a different disk.
December 19, 20241 yr Author I did end up swapping cables and rebuilt once again with the untrusted drive which failed at the same % so fairly sure that drive is just toast. I then swapped out the drive back to the original drive (with the new cable and port) and rebuilt, it succeeded and only took 14 hours so I think I am all up and running again. Thanks again for the help, there was not any one smoking gun on this one so I'll just mark the last comment as the solution.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.