Thanks for the direction!
Here's the summary of the steps I took, but I'm stuck on the scrub part. The button is disabled while it's unmounted.
Started the array in maintenance mode. I ran Fix on Disk 6 and Disk 7 (see the output sections below for details)
Stopped the array
Started the array. Disk 6 and Disk 7 now appear mounted
Selected cache drive settings, and "Scrub" button is disabled with the message, "Scrub is only available when the filesyestem is mounted." The File system status:
Unmountable: wrong or no file system
More details below:
Disk 6
In maintenance mode, I ran Fix, and this is the output:
Phase 1 - find and verify superblock...
bad primary superblock - bad sector size !!!
attempting to find secondary superblock...
.found candidate secondary superblock...
verified secondary superblock...
writing modified primary superblock
sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128
resetting superblock root inode pointer to 128
sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129
resetting superblock realtime bitmap inode pointer to 129
sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130
resetting superblock realtime summary inode pointer to 130
Phase 2 - using internal log
- zero log...
- scan filesystem freespace and inode maps...
Metadata CRC error detected at 0x447820, xfs_agf block 0x1/0x200
agf has bad CRC for ag 0
Metadata CRC error detected at 0x474fd0, xfs_agi block 0x2/0x200
agi has bad CRC for ag 0
bad magic # 0x0 for agf 0
bad version # 0 for agf 0
bad length 0 for agf 0, should be 268435455
bad uuid 00000000-0000-0000-0000-000000000000 for agf 0
bad magic # 0x0 for agi 0
bad version # 0 for agi 0
bad length # 0 for agi 0, should be 268435455
bad uuid 00000000-0000-0000-0000-000000000000 for agi 0
reset bad agf for ag 0
reset bad agi for ag 0
bad levels 0 for btbno root, agno 0
bad agbno 0 for btbno root, agno 0
bad levels 0 for btbcnt root, agno 0
bad agbno 0 for btbcnt root, agno 0
bad levels 0 for rmapbt root, agno 0
bad agbno 0 for rmapbt root, agno 0
bad levels 0 for refcountbt root, agno 0
bad agbno 0 for refcntbt root, agno 0
bad levels 0 for inobt root, agno 0
bad agbno 0 for inobt root, agno 0
bad levels 0 for finobt root, agno 0
bad agbno 0 for finobt root, agno 0
agi unlinked bucket 0 is 0 in ag 0 (inode=0)
agi unlinked bucket 1 is 0 in ag 0 (inode=0)
agi unlinked bucket 2 is 0 in ag 0 (inode=0)
agi unlinked bucket 3 is 0 in ag 0 (inode=0)
agi unlinked bucket 4 is 0 in ag 0 (inode=0)
agi unlinked bucket 5 is 0 in ag 0 (inode=0)
agi unlinked bucket 6 is 0 in ag 0 (inode=0)
agi unlinked bucket 7 is 0 in ag 0 (inode=0)
agi unlinked bucket 8 is 0 in ag 0 (inode=0)
agi unlinked bucket 9 is 0 in ag 0 (inode=0)
agi unlinked bucket 10 is 0 in ag 0 (inode=0)
agi unlinked bucket 11 is 0 in ag 0 (inode=0)
agi unlinked bucket 12 is 0 in ag 0 (inode=0)
agi unlinked bucket 13 is 0 in ag 0 (inode=0)
agi unlinked bucket 14 is 0 in ag 0 (inode=0)
agi unlinked bucket 15 is 0 in ag 0 (inode=0)
agi unlinked bucket 16 is 0 in ag 0 (inode=0)
agi unlinked bucket 17 is 0 in ag 0 (inode=0)
agi unlinked bucket 18 is 0 in ag 0 (inode=0)
agi unlinked bucket 19 is 0 in ag 0 (inode=0)
agi unlinked bucket 20 is 0 in ag 0 (inode=0)
agi unlinked bucket 21 is 0 in ag 0 (inode=0)
agi unlinked bucket 22 is 0 in ag 0 (inode=0)
agi unlinked bucket 23 is 0 in ag 0 (inode=0)
agi unlinked bucket 24 is 0 in ag 0 (inode=0)
agi unlinked bucket 25 is 0 in ag 0 (inode=0)
agi unlinked bucket 26 is 0 in ag 0 (inode=0)
agi unlinked bucket 27 is 0 in ag 0 (inode=0)
agi unlinked bucket 28 is 0 in ag 0 (inode=0)
agi unlinked bucket 29 is 0 in ag 0 (inode=0)
agi unlinked bucket 30 is 0 in ag 0 (inode=0)
agi unlinked bucket 31 is 0 in ag 0 (inode=0)
agi unlinked bucket 32 is 0 in ag 0 (inode=0)
agi unlinked bucket 33 is 0 in ag 0 (inode=0)
agi unlinked bucket 34 is 0 in ag 0 (inode=0)
agi unlinked bucket 35 is 0 in ag 0 (inode=0)
agi unlinked bucket 36 is 0 in ag 0 (inode=0)
agi unlinked bucket 37 is 0 in ag 0 (inode=0)
agi unlinked bucket 38 is 0 in ag 0 (inode=0)
agi unlinked bucket 39 is 0 in ag 0 (inode=0)
agi unlinked bucket 40 is 0 in ag 0 (inode=0)
agi unlinked bucket 41 is 0 in ag 0 (inode=0)
agi unlinked bucket 42 is 0 in ag 0 (inode=0)
agi unlinked bucket 43 is 0 in ag 0 (inode=0)
agi unlinked bucket 44 is 0 in ag 0 (inode=0)
agi unlinked bucket 45 is 0 in ag 0 (inode=0)
agi unlinked bucket 46 is 0 in ag 0 (inode=0)
agi unlinked bucket 47 is 0 in ag 0 (inode=0)
agi unlinked bucket 48 is 0 in ag 0 (inode=0)
agi unlinked bucket 49 is 0 in ag 0 (inode=0)
agi unlinked bucket 50 is 0 in ag 0 (inode=0)
agi unlinked bucket 51 is 0 in ag 0 (inode=0)
agi unlinked bucket 52 is 0 in ag 0 (inode=0)
agi unlinked bucket 53 is 0 in ag 0 (inode=0)
agi unlinked bucket 54 is 0 in ag 0 (inode=0)
agi unlinked bucket 55 is 0 in ag 0 (inode=0)
agi unlinked bucket 56 is 0 in ag 0 (inode=0)
agi unlinked bucket 57 is 0 in ag 0 (inode=0)
agi unlinked bucket 58 is 0 in ag 0 (inode=0)
agi unlinked bucket 59 is 0 in ag 0 (inode=0)
agi unlinked bucket 60 is 0 in ag 0 (inode=0)
agi unlinked bucket 61 is 0 in ag 0 (inode=0)
agi unlinked bucket 62 is 0 in ag 0 (inode=0)
agi unlinked bucket 63 is 0 in ag 0 (inode=0)
sb_icount 0, counted 320
sb_ifree 0, counted 282
sb_fdblocks 1952984849, counted 1633657855
root inode chunk not found
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
imap claims in-use inode 131 is free, correcting imap
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 1
- agno = 2
- agno = 5
- agno = 7
- agno = 4
- agno = 6
- agno = 3
Phase 5 - rebuild AG headers and trees...
- reset superblock...
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
- traversal finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
Note - stripe unit (0) and width (0) were copied from a backup superblock.
Please reset with mount -o sunit=,swidth= if necessary
done
after running fix, it says, "File system corruption fixed"
Ran another check:
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
- scan filesystem freespace and inode maps...
- found root inode chunk
Phase 3 - for each AG...
- scan (but don't clear) agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 3
- agno = 6
- agno = 2
- agno = 5
- agno = 1
- agno = 7
- agno = 4
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
- traversing filesystem ...
- traversal finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.
...and "No file system corruption detected.
Disk 7
In maintenance mode, I ran a Fix and this is the output:
Phase 1 - find and verify superblock...
bad primary superblock - bad sector size !!!
attempting to find secondary superblock...
.found candidate secondary superblock...
verified secondary superblock...
writing modified primary superblock
sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 128
resetting superblock root inode pointer to 128
sb realtime bitmap inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 129
resetting superblock realtime bitmap inode pointer to 129
sb realtime summary inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 130
resetting superblock realtime summary inode pointer to 130
Phase 2 - using internal log
- zero log...
- scan filesystem freespace and inode maps...
Metadata CRC error detected at 0x447820, xfs_agf block 0x1/0x200
agf has bad CRC for ag 0
Metadata CRC error detected at 0x474fd0, xfs_agi block 0x2/0x200
agi has bad CRC for ag 0
bad magic # 0x0 for agf 0
bad version # 0 for agf 0
bad length 0 for agf 0, should be 268435455
bad uuid 00000000-0000-0000-0000-000000000000 for agf 0
bad magic # 0x0 for agi 0
bad version # 0 for agi 0
bad length # 0 for agi 0, should be 268435455
bad uuid 00000000-0000-0000-0000-000000000000 for agi 0
reset bad agf for ag 0
reset bad agi for ag 0
bad levels 0 for btbno root, agno 0
bad agbno 0 for btbno root, agno 0
bad levels 0 for btbcnt root, agno 0
bad agbno 0 for btbcnt root, agno 0
bad levels 0 for rmapbt root, agno 0
bad agbno 0 for rmapbt root, agno 0
bad levels 0 for refcountbt root, agno 0
bad agbno 0 for refcntbt root, agno 0
bad levels 0 for inobt root, agno 0
bad agbno 0 for inobt root, agno 0
bad levels 0 for finobt root, agno 0
bad agbno 0 for finobt root, agno 0
agi unlinked bucket 0 is 0 in ag 0 (inode=0)
agi unlinked bucket 1 is 0 in ag 0 (inode=0)
agi unlinked bucket 2 is 0 in ag 0 (inode=0)
agi unlinked bucket 3 is 0 in ag 0 (inode=0)
agi unlinked bucket 4 is 0 in ag 0 (inode=0)
agi unlinked bucket 5 is 0 in ag 0 (inode=0)
agi unlinked bucket 6 is 0 in ag 0 (inode=0)
agi unlinked bucket 7 is 0 in ag 0 (inode=0)
agi unlinked bucket 8 is 0 in ag 0 (inode=0)
agi unlinked bucket 9 is 0 in ag 0 (inode=0)
agi unlinked bucket 10 is 0 in ag 0 (inode=0)
agi unlinked bucket 11 is 0 in ag 0 (inode=0)
agi unlinked bucket 12 is 0 in ag 0 (inode=0)
agi unlinked bucket 13 is 0 in ag 0 (inode=0)
agi unlinked bucket 14 is 0 in ag 0 (inode=0)
agi unlinked bucket 15 is 0 in ag 0 (inode=0)
agi unlinked bucket 16 is 0 in ag 0 (inode=0)
agi unlinked bucket 17 is 0 in ag 0 (inode=0)
agi unlinked bucket 18 is 0 in ag 0 (inode=0)
agi unlinked bucket 19 is 0 in ag 0 (inode=0)
agi unlinked bucket 20 is 0 in ag 0 (inode=0)
agi unlinked bucket 21 is 0 in ag 0 (inode=0)
agi unlinked bucket 22 is 0 in ag 0 (inode=0)
agi unlinked bucket 23 is 0 in ag 0 (inode=0)
agi unlinked bucket 24 is 0 in ag 0 (inode=0)
agi unlinked bucket 25 is 0 in ag 0 (inode=0)
agi unlinked bucket 26 is 0 in ag 0 (inode=0)
agi unlinked bucket 27 is 0 in ag 0 (inode=0)
agi unlinked bucket 28 is 0 in ag 0 (inode=0)
agi unlinked bucket 29 is 0 in ag 0 (inode=0)
agi unlinked bucket 30 is 0 in ag 0 (inode=0)
agi unlinked bucket 31 is 0 in ag 0 (inode=0)
agi unlinked bucket 32 is 0 in ag 0 (inode=0)
agi unlinked bucket 33 is 0 in ag 0 (inode=0)
agi unlinked bucket 34 is 0 in ag 0 (inode=0)
agi unlinked bucket 35 is 0 in ag 0 (inode=0)
agi unlinked bucket 36 is 0 in ag 0 (inode=0)
agi unlinked bucket 37 is 0 in ag 0 (inode=0)
agi unlinked bucket 38 is 0 in ag 0 (inode=0)
agi unlinked bucket 39 is 0 in ag 0 (inode=0)
agi unlinked bucket 40 is 0 in ag 0 (inode=0)
agi unlinked bucket 41 is 0 in ag 0 (inode=0)
agi unlinked bucket 42 is 0 in ag 0 (inode=0)
agi unlinked bucket 43 is 0 in ag 0 (inode=0)
agi unlinked bucket 44 is 0 in ag 0 (inode=0)
agi unlinked bucket 45 is 0 in ag 0 (inode=0)
agi unlinked bucket 46 is 0 in ag 0 (inode=0)
agi unlinked bucket 47 is 0 in ag 0 (inode=0)
agi unlinked bucket 48 is 0 in ag 0 (inode=0)
agi unlinked bucket 49 is 0 in ag 0 (inode=0)
agi unlinked bucket 50 is 0 in ag 0 (inode=0)
agi unlinked bucket 51 is 0 in ag 0 (inode=0)
agi unlinked bucket 52 is 0 in ag 0 (inode=0)
agi unlinked bucket 53 is 0 in ag 0 (inode=0)
agi unlinked bucket 54 is 0 in ag 0 (inode=0)
agi unlinked bucket 55 is 0 in ag 0 (inode=0)
agi unlinked bucket 56 is 0 in ag 0 (inode=0)
agi unlinked bucket 57 is 0 in ag 0 (inode=0)
agi unlinked bucket 58 is 0 in ag 0 (inode=0)
agi unlinked bucket 59 is 0 in ag 0 (inode=0)
agi unlinked bucket 60 is 0 in ag 0 (inode=0)
agi unlinked bucket 61 is 0 in ag 0 (inode=0)
agi unlinked bucket 62 is 0 in ag 0 (inode=0)
agi unlinked bucket 63 is 0 in ag 0 (inode=0)
sb_fdblocks 1952984849, counted 1684549401
root inode chunk not found
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 1
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 2
- agno = 7
Phase 5 - rebuild AG headers and trees...
- reset superblock...
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
- traversal finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
Note - stripe unit (0) and width (0) were copied from a backup superblock.
Please reset with mount -o sunit=,swidth= if necessary
done
Afterwards, it says, "File system corruption fixed."
Ran another check:
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
- scan filesystem freespace and inode maps...
- found root inode chunk
Phase 3 - for each AG...
- scan (but don't clear) agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 1
- agno = 5
- agno = 4
- agno = 7
- agno = 6
- agno = 3
- agno = 2
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
- traversing filesystem ...
- traversal finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.
...and "No file system corruption detected."
Started the array and the Disks appear mounted! Success with that. The cache, however, is still unmounted, and I cannot run the scrub. Stopping/starting in maintenance mode, I ran a new check on cache:
[1/8] checking log
[2/8] checking root items
[3/8] checking extents
metadata level mismatch on [51691520, 16384]
metadata level mismatch on [101482496, 16384]
metadata level mismatch on [778158080, 16384]
ERROR: errors found in extent allocation tree or chunk allocation
[4/8] checking free space tree
[5/8] checking fs roots
[6/8] checking only csums items (without verifying data)
[7/8] checking root refs
[8/8] checking quota groups skipped (not enabled on this FS)
Opening filesystem to check...
Checking filesystem on /dev/nvme1n1p1
UUID: 2053cffe-1ca1-4777-a2a7-206317bde466
found 320820903936 bytes used, error(s) found
total csum bytes: 289544368
total tree bytes: 371916800
total fs tree bytes: 53035008
total extent tree bytes: 12828672
btree space waste bytes: 27503735
file data blocks allocated: 979578281984
referenced 315945730048