Hello Everyone,

 

I am having lots of issues.  it started with running my parity check, parity check failed due to a bad drive.  I replaced the drive ran Preclear for 4 days, then  I proceeded with the data rebuild with lots of errors, I then stopped the array and rebooted due to the log file stuck at100%.  Now the drive is showing Unmountable.  I am not sure what my next step should be.uss-enterprise-syslog-20240730-0426.zip

 

Thank you.

Check filesystem on disk2, run it without -n

I restarted in array in maintenance mode and execute the Check Filesystem with -n and this is the output.  Not sure if it is running.

Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being ignored because the -n option was used. Expect spurious inconsistencies which may be resolved by first mounting the filesystem to replay the log. - scan filesystem freespace and inode maps... Metadata CRC error detected at 0x44228d, xfs_bnobt block 0x2ffffffd8/0x1000 btree block 6/1 is suspect, error -74 bad magic # 0 in btbno block 6/1 Metadata CRC error detected at 0x44228d, xfs_bnobt block 0x1ffffffe8/0x1000 btree block 4/1 is suspect, error -74 bad magic # 0 in btbno block 4/1 Metadata CRC error detected at 0x44228d, xfs_cntbt block 0x2ffffffe0/0x1000 btree block 6/2 is suspect, error -74 bad magic # 0 in btcnt block 6/2 Metadata CRC error detected at 0x47494d, xfs_refcountbt block 0x2fffffff8/0x1000 btree block 6/5 is suspect, error -74 bad magic # 0 in refcount btree block 6/5 Metadata CRC error detected at 0x44228d, xfs_cntbt block 0x1fffffff0/0x1000 btree block 4/2 is suspect, error -74 bad magic # 0 in btcnt block 4/2 bad refcountbt block count 0, saw 1 agf_freeblks 226278958, counted 0 in ag 6 agf_longest 226252057, counted 0 in ag 6 Metadata CRC error detected at 0x47494d, xfs_refcountbt block 0x200000008/0x1000 btree block 4/5 is suspect, error -74 bad magic # 0 in refcount btree block 4/5 bad refcountbt block count 0, saw 1 agf_freeblks 245229779, counted 0 in ag 4 agf_longest 244949437, counted 0 in ag 4 Metadata CRC error detected at 0x47191d, xfs_inobt block 0x2ffffffe8/0x1000 btree block 6/3 is suspect, error -74 bad magic # 0 in inobt block 6/3 Metadata CRC error detected at 0x47191d, xfs_inobt block 0x1fffffff8/0x1000 btree block 4/3 is suspect, error -74 bad magic # 0 in inobt block 4/3 inode chunk claims untracked block, finobt block - agno 6, bno 355592, inopb 8 inode chunk claims untracked block, finobt block - agno 6, bno 355593, inopb 8 inode chunk claims untracked block, finobt block - agno 6, bno 355594, inopb 8 inode chunk claims untracked block, finobt block - agno 6, bno 355595, inopb 8 inode chunk claims untracked block, finobt block - agno 4, bno 320944, inopb 8 inode chunk claims untracked block, finobt block - agno 4, bno 320945, inopb 8 inode chunk claims untracked block, finobt block - agno 4, bno 320946, inopb 8 inode chunk claims untracked block, finobt block - agno 4, bno 320947, inopb 8 inode chunk claims untracked block, finobt block - agno 6, bno 355596, inopb 8 inode chunk claims untracked block, finobt block - agno 6, bno 355597, inopb 8 inode chunk claims untracked block, finobt block - agno 6, bno 355598, inopb 8 inode chunk claims untracked block, finobt block - agno 6, bno 355599, inopb 8 undiscovered finobt record, ino 12887746624 (6/2844736) inode chunk claims untracked block, finobt block - agno 4, bno 320948, inopb 8 inode chunk claims untracked block, finobt block - agno 4, bno 320949, inopb 8 inode chunk claims untracked block, finobt block - agno 4, bno 320950, inopb 8 agi_count 64, counted 0 in ag 6 agi_freecount 29, counted 0 in ag 6 inode chunk claims untracked block, finobt block - agno 4, bno 320951, inopb 8 undiscovered finobt record, ino 8592502144 (4/2567552) agi unlinked bucket 34 is 2844770 in ag 6 (inode=12887746658) agi_count 128, counted 0 in ag 4 agi_freecount 57, counted 0 in ag 4 sb_icount 1920, counted 1728 sb_ifree 489, counted 397 sb_fdblocks 2240515018, counted 1787708866 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... found inodes not in the inode allocation tree found inodes not in the inode allocation tree found inodes not in the inode allocation tree - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 1496b9ebb680: Badness in key lookup (length) bp=(bno 0x3002b6810, len 4096 bytes) key=(bno 0x3002b6810, len 16384 bytes) 1496b9ebb680: Badness in key lookup (length) bp=(bno 0x3002b6830, len 4096 bytes) key=(bno 0x3002b6830, len 16384 bytes) - agno = 6 imap claims in-use inode 12887746625 is free, would correct imap imap claims in-use inode 12887746626 is free, would correct imap imap claims in-use inode 12887746627 is free, would correct imap imap claims in-use inode 12887746628 is free, would correct imap imap claims in-use inode 12887746629 is free, would correct imap imap claims in-use inode 12887746630 is free, would correct imap imap claims in-use inode 12887746631 is free, would correct imap imap claims in-use inode 12887746632 is free, would correct imap imap claims in-use inode 12887746633 is free, would correct imap imap claims in-use inode 12887746634 is free, would correct imap imap claims in-use inode 12887746635 is free, would correct imap imap claims in-use inode 12887746636 is free, would correct imap imap claims in-use inode 12887746637 is free, would correct imap imap claims in-use inode 12887746638 is free, would correct imap imap claims in-use inode 12887746639 is free, would correct imap imap claims in-use inode 12887746640 is free, would correct imap imap claims in-use inode 12887746641 is free, would correct imap imap claims in-use inode 12887746642 is free, would correct imap imap claims in-use inode 12887746643 is free, would correct imap imap claims in-use inode 12887746644 is free, would correct imap imap claims in-use inode 12887746645 is free, would correct imap imap claims in-use inode 12887746646 is free, would correct imap imap claims in-use inode 12887746647 is free, would correct imap imap claims in-use inode 12887746648 is free, would correct imap imap claims in-use inode 12887746649 is free, would correct imap imap claims in-use inode 12887746650 is free, would correct imap imap claims in-use inode 12887746651 is free, would correct imap imap claims in-use inode 12887746652 is free, would correct imap imap claims in-use inode 12887746653 is free, would correct imap imap claims in-use inode 12887746654 is free, would correct imap imap claims in-use inode 12887746655 is free, would correct imap imap claims in-use inode 12887746656 is free, would correct imap imap claims in-use inode 12887746657 is free, would correct imap imap claims in-use inode 12887746658 is free, would correct imap - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - agno = 13 - agno = 14 - agno = 15 - agno = 16 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 1 - agno = 12 - agno = 3 entry "Fresh Off the Boat" at block 0 offset 224 in directory inode 2147483776 references non-existent inode 8592502080 entry ".." at block 0 offset 80 in directory inode 6442451010 references non-existent inode 8592502080 - agno = 8 - agno = 9 - agno = 10 - agno = 4 - agno = 16 - agno = 13 - agno = 14 entry "Season 01" in shortform directory 27917287552 references non-existent inode 8592502083 would have junked entry "Season 01" in directory inode 27917287552 would have corrected i8 count in directory 27917287552 from 1 to 0 entry ".." at block 0 offset 80 in directory inode 4312399554 references non-existent inode 8592502080 - agno = 5 - agno = 6 would clear inode number in entry at offset 224... entry "Season 06" in shortform directory 12887746625 references non-existent inode 8592502084 would have junked entry "Season 06" in directory inode 12887746625 would have corrected i8 count in directory 12887746625 from 4 to 3 entry "TV Shows" in shortform directory 136 references non-existent inode 8592502150 would have junked entry "TV Shows" in directory inode 136 would have corrected i8 count in directory 136 from 1 to 0 - agno = 11 - agno = 15 - agno = 7 entry "The Mosquito Coast" at block 0 offset 696 in directory inode 2147483776 references non-existent inode 8592502081 would clear inode number in entry at offset 696... entry "Billions" at block 0 offset 1024 in directory inode 2147483776 references non-existent inode 8592502145 would clear inode number in entry at offset 1024... entry ".." at block 0 offset 80 in directory inode 2705898631 references non-existent inode 8592502081 would rebuild corrupt refcount btrees. No modify flag set, skipping phase 5 Inode allocation btrees are too corrupted, skipping phases 6 and 7 No modify flag set, skipping filesystem flush and exiting.

7 hours ago, JorgeB said:

run it without -n

 

I ran it without -n  and here are the results.

 

 

Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this.

Use -L

Here is the output after using the  -L 

 

Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... clearing needsrepair flag and regenerating metadata Metadata CRC error detected at 0x44228d, xfs_bnobt block 0x2ffffffd8/0x1000 btree block 6/1 is suspect, error -74 bad magic # 0 in btbno block 6/1 Metadata CRC error detected at 0x44228d, xfs_cntbt block 0x2ffffffe0/0x1000 btree block 6/2 is suspect, error -74 bad magic # 0 in btcnt block 6/2 Metadata CRC error detected at 0x47494d, xfs_refcountbt block 0x2fffffff8/0x1000 btree block 6/5 is suspect, error -74 bad magic # 0 in refcount btree block 6/5 bad refcountbt block count 0, saw 1 agf_freeblks 226278958, counted 0 in ag 6 agf_longest 226252057, counted 0 in ag 6 Metadata CRC error detected at 0x47191d, xfs_inobt block 0x2ffffffe8/0x1000 btree block 6/3 is suspect, error -74 bad magic # 0 in inobt block 6/3 inode chunk claims untracked block, finobt block - agno 6, bno 355592, inopb 8 inode chunk claims untracked block, finobt block - agno 6, bno 355593, inopb 8 inode chunk claims untracked block, finobt block - agno 6, bno 355594, inopb 8 inode chunk claims untracked block, finobt block - agno 6, bno 355595, inopb 8 inode chunk claims untracked block, finobt block - agno 6, bno 355596, inopb 8 inode chunk claims untracked block, finobt block - agno 6, bno 355597, inopb 8 inode chunk claims untracked block, finobt block - agno 6, bno 355598, inopb 8 inode chunk claims untracked block, finobt block - agno 6, bno 355599, inopb 8 undiscovered finobt record, ino 12887746624 (6/2844736) agi_count 64, counted 0 in ag 6 agi_freecount 29, counted 0 in ag 6 agi unlinked bucket 34 is 2844770 in ag 6 (inode=12887746658) Metadata CRC error detected at 0x44228d, xfs_bnobt block 0x1ffffffe8/0x1000 btree block 4/1 is suspect, error -74 bad magic # 0 in btbno block 4/1 Metadata CRC error detected at 0x44228d, xfs_cntbt block 0x1fffffff0/0x1000 btree block 4/2 is suspect, error -74 bad magic # 0 in btcnt block 4/2 Metadata CRC error detected at 0x47494d, xfs_refcountbt block 0x200000008/0x1000 btree block 4/5 is suspect, error -74 bad magic # 0 in refcount btree block 4/5 bad refcountbt block count 0, saw 1 agf_freeblks 245229779, counted 0 in ag 4 agf_longest 244949437, counted 0 in ag 4 Metadata CRC error detected at 0x47191d, xfs_inobt block 0x1fffffff8/0x1000 btree block 4/3 is suspect, error -74 bad magic # 0 in inobt block 4/3 inode chunk claims untracked block, finobt block - agno 4, bno 320944, inopb 8 inode chunk claims untracked block, finobt block - agno 4, bno 320945, inopb 8 inode chunk claims untracked block, finobt block - agno 4, bno 320946, inopb 8 inode chunk claims untracked block, finobt block - agno 4, bno 320947, inopb 8 inode chunk claims untracked block, finobt block - agno 4, bno 320948, inopb 8 inode chunk claims untracked block, finobt block - agno 4, bno 320949, inopb 8 inode chunk claims untracked block, finobt block - agno 4, bno 320950, inopb 8 inode chunk claims untracked block, finobt block - agno 4, bno 320951, inopb 8 undiscovered finobt record, ino 8592502144 (4/2567552) agi_count 128, counted 0 in ag 4 agi_freecount 57, counted 0 in ag 4 sb_icount 1920, counted 1728 sb_ifree 489, counted 397 sb_fdblocks 2240515018, counted 1787708866 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... found inodes not in the inode allocation tree found inodes not in the inode allocation tree found inodes not in the inode allocation tree - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 14741cc36680: Badness in key lookup (length) bp=(bno 0x3002b6810, len 4096 bytes) key=(bno 0x3002b6810, len 16384 bytes) 14741cc36680: Badness in key lookup (length) bp=(bno 0x3002b6830, len 4096 bytes) key=(bno 0x3002b6830, len 16384 bytes) - agno = 6 correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - agno = 13 - agno = 14 - agno = 15 - agno = 16 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 5 - agno = 9 - agno = 4 - agno = 12 - agno = 3 - agno = 8 - agno = 7 - agno = 15 - agno = 1 - agno = 10 entry "Fresh Off the Boat" at block 0 offset 224 in directory inode 2147483776 references non-existent inode 8592502080 clearing inode number in entry at offset 224... entry "The Mosquito Coast" at block 0 offset 696 in directory inode 2147483776 references non-existent inode 8592502081 clearing inode number in entry at offset 696... entry "Billions" at block 0 offset 1024 in directory inode 2147483776 references non-existent inode 8592502145 clearing inode number in entry at offset 1024... entry ".." at block 0 offset 80 in directory inode 4312399554 references non-existent inode 8592502080 - agno = 6 entry ".." at block 0 offset 80 in directory inode 6442451010 references non-existent inode 8592502080 - agno = 13 entry "Season 01" in shortform directory 27917287552 references non-existent inode 8592502083 junking entry "Season 01" in directory inode 27917287552 corrected i8 count in directory 27917287552, was 1, now 0 corrected directory 27917287552 size, was 10, now 6 - agno = 14 - agno = 16 entry "TV Shows" in shortform directory 136 references non-existent inode 8592502150 junking entry "TV Shows" in directory inode 136 corrected i8 count in directory 136, was 1, now 0 - agno = 11 entry "Season 06" in shortform directory 12887746625 references non-existent inode 8592502084 junking entry "Season 06" in directory inode 12887746625 corrected i8 count in directory 12887746625, was 4, now 3 corrected directory 136 size, was 10, now 6 entry ".." at block 0 offset 80 in directory inode 2705898631 references non-existent inode 8592502081 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... bad hash table for directory inode 2147483776 (no data entry): rebuilding rebuilding directory inode 2147483776 entry ".." in directory inode 2705898631 points to non-existent inode 8592502081 bad hash table for directory inode 2705898631 (no data entry): rebuilding rebuilding directory inode 2705898631 entry ".." in directory inode 4312399554 points to non-existent inode 8592502080 bad hash table for directory inode 4312399554 (no data entry): rebuilding rebuilding directory inode 4312399554 entry ".." in directory inode 6442451010 points to non-existent inode 8592502080 bad hash table for directory inode 6442451010 (no data entry): rebuilding rebuilding directory inode 6442451010 - traversal finished ... - moving disconnected inodes to lost+found ... disconnected dir inode 2705898631, moving to lost+found disconnected dir inode 4312399554, moving to lost+found disconnected dir inode 6442451010, moving to lost+found disconnected dir inode 10741604802, moving to lost+found disconnected dir inode 10741604846, moving to lost+found disconnected dir inode 12887746658, moving to lost+found Phase 7 - verify and correct link counts... resetting inode 136 nlinks from 3 to 2 resetting inode 2147483776 nlinks from 52 to 49 resetting inode 12887746625 nlinks from 6 to 5 resetting inode 27917287552 nlinks from 3 to 2 resetting inode 12887746658 nlinks from 0 to 2 resetting inode 11302117 nlinks from 2 to 8 Maximum metadata LSN (2:116367) is ahead of log (1:2). Format log to cycle 5. done

How should I proceed to fix my issue.  I have not been down this road with the issues I am experiencing.

 

Thank you.

Edited July 31, 20241 yr by Scuba_Steve

Start the array in normal mode now, the disk should mount, look for a lost+found folder

Thank you so much JorgeB.  Should I run a parity check in non-correcting?  Are the files in the lost+found files corrupt?

Not necessarily, they can just be lost files, without the original name, not always easy to identify them sometimes.

 

You can a parity check, but if all was done correctly parity should still be valid.

How it started was when I was doing my parity check with correcting was selected and then I lost a hdd during the process.  I then precleared a replacement drive and the data rebuild resulted in thousands of error from the rebuild.  Not sure how valid the data on my parity drive.

Should reflect the current parity, but you can do a check to confirm.

Thank you so much for everyone that has given me good information.

Edited August 1, 20241 yr by Scuba_Steve