2/3 drives showing as unmountable after power outage.

[Crocs]

I'm in need of some big help here...

I had a power outage that caused 2 of my 3 drives to show as "unmountable" after a reboot. I'm not sure what next steps to take to insure I don't loose data.

 

I have sense ordered a UPS

motherlode-diagnostics-20230815-0927.zip

[JorgeB]

Diags after array start please.

[Crocs]

motherlode-diagnostics-20230815-1419.zip

[JorgeB]

Check filesystem on disks 1 and 2.

[Crocs]

Phase 1 - find and verify superblock...
        - block cache size set to 1415808 entries
Phase 2 - using internal log
        - zero log...
* ERROR: mismatched uuid in log
*            SB : edfe3834-877a-426a-82ce-7b77aafb082e
*            log: d68381a6-f261-4fd4-aca3-446415ce8102
zero_log: head block 163732 tail block 163732
        - scan filesystem freespace and inode maps...
sb_icount 64, counted 32
sb_ifree 61, counted 29
sb_fdblocks 3905948049, counted 3905948053
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 9
        - agno = 5
        - agno = 13
        - agno = 4
        - agno = 3
        - agno = 7
        - agno = 11
        - agno = 8
        - agno = 12
        - agno = 10
        - agno = 14
        - agno = 6
Phase 5 - rebuild AG headers and trees...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
SB summary counter sanity check failed
Metadata corruption detected at 0x47518b, xfs_sb block 0x0/0x200
libxfs_bwrite: write verifier failed on xfs_sb bno 0x0/0x200
xfs_repair: Releasing dirty buffer to free list!
xfs_repair: Refusing to write a corrupt buffer to the data device!
xfs_repair: Lost a write to the data device!

fatal error -- File system metadata writeout failed, err=117.  Re-run xfs_repair.

 This is drive1's first run with the "-v" tag

 

Phase 1 - find and verify superblock...
        - block cache size set to 1415808 entries
Phase 2 - using internal log
        - zero log...
* ERROR: mismatched uuid in log
*            SB : edfe3834-877a-426a-82ce-7b77aafb082e
*            log: d68381a6-f261-4fd4-aca3-446415ce8102
zero_log: head block 163732 tail block 163732
        - scan filesystem freespace and inode maps...
sb_icount 64, counted 32
sb_ifree 61, counted 29
sb_fdblocks 3905948049, counted 3905948053
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 2
        - agno = 4
        - agno = 5
        - agno = 3
        - agno = 10
        - agno = 11
        - agno = 1
        - agno = 7
        - agno = 6
        - agno = 8
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 9
Phase 5 - rebuild AG headers and trees...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
SB summary counter sanity check failed
Metadata corruption detected at 0x47518b, xfs_sb block 0x0/0x200
libxfs_bwrite: write verifier failed on xfs_sb bno 0x0/0x200
xfs_repair: Releasing dirty buffer to free list!
xfs_repair: Refusing to write a corrupt buffer to the data device!
xfs_repair: Lost a write to the data device!

fatal error -- File system metadata writeout failed, err=117.  Re-run xfs_repair.

This is drive1's second run.

I'm getting a similar output from drive2

[Crocs]

I am on 6.9.2. I've heard others say to upgrade to 6.1.0 but wasn't sure.

[JorgeB]

Try upgrading to v6.12 since it includes newer xfsprogs and try again, if it still fails post new diags after running xfs_repair.

[Crocs]

Still failed....

motherlode-diagnostics-20230816-1319.zip

[Crocs]

Never mind, reran and we're back.

 

Thank you!!!