thebedivere

No errors showing up:

Shut it down and checked the cables on the drives, removed the disk that wasn't in use and had errors. Restarted and things are looking better. Should I be worried about it leaking helium? I'll probably start the warranty process on it, since I doubt it can be recovered if that's the problem. orthanc-diagnostics-20220810-1636.zip

Rebuild is running, but it's going real slow. Docker and VMs are off. orthanc-diagnostics-20220810-1002.zip

Is this the process of stopping the array, unassigning the drive, starting the array, stopping the array, and reassigning the drive to the same slot?

Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... Metadata CRC error detected at 0x439496, xfs_agf block 0x1ffffffe1/0x200 agf has bad CRC for ag 4 block (2,184178199-184178280) multiply claimed by bno space tree, state - 1 block (2,226384586-226384833) multiply claimed by cnt space tree, state - 2 block (2,123204880-123205051) multiply claimed by cnt space tree, state - 2 block (2,40225617-40225622) multiply claimed by cnt space tree, state - 2 block (2,40225773-40225859) multiply claimed by cnt space tree, state - 2 block (2,102377537-102377552) multiply claimed by cnt space tree, state - 2 block (2,36413631-36413735) multiply claimed by cnt space tree, state - 2 block (2,61136756-61136865) multiply claimed by cnt space tree, state - 2 block (2,174373505-174373514) multiply claimed by cnt space tree, state - 2 block (2,9748180-9748193) multiply claimed by cnt space tree, state - 2 block (2,9748274-9748287) multiply claimed by cnt space tree, state - 2 block (2,104799337-104799352) multiply claimed by cnt space tree, state - 2 block (2,123180332-123180475) multiply claimed by cnt space tree, state - 2 block (2,162984111-162984288) multiply claimed by cnt space tree, state - 2 block (2,60006119-60006216) multiply claimed by cnt space tree, state - 2 block (2,102524589-102524613) multiply claimed by cnt space tree, state - 2 block (2,40224317-40224326) multiply claimed by cnt space tree, state - 2 block (2,40224419-40224432) multiply claimed by cnt space tree, state - 2 block (2,40224497-40224575) multiply claimed by cnt space tree, state - 2 block (2,107092002-107092006) multiply claimed by cnt space tree, state - 2 block (2,236311683-236311689) multiply claimed by cnt space tree, state - 2 block (2,107087496-107087499) multiply claimed by cnt space tree, state - 2 block (2,48692963-48693032) multiply claimed by cnt space tree, state - 2 block (2,106727218-106727508) multiply claimed by cnt space tree, state - 2 block (2,104837401-104837680) multiply claimed by cnt space tree, state - 2 block (2,9514183-9514243) multiply claimed by cnt space tree, state - 2 block (2,48042435-48042473) multiply claimed by cnt space tree, state - 2 block (2,40238304-40238370) multiply claimed by cnt space tree, state - 2 block (2,162968554-162968627) multiply claimed by cnt space tree, state - 2 block (2,47965621-47965668) multiply claimed by cnt space tree, state - 2 block (2,178636310-178636383) multiply claimed by cnt space tree, state - 2 block (2,102523728-102523738) multiply claimed by cnt space tree, state - 2 block (2,19627766-19627977) multiply claimed by cnt space tree, state - 2 block (2,95752502-95752512) multiply claimed by cnt space tree, state - 2 block (2,127294648-127294666) multiply claimed by cnt space tree, state - 2 agf_freeblks 2038004, counted 2048802 in ag 2 agf_freeblks 2116416, counted 2106811 in ag 4 agi unlinked bucket 24 is 344682520 in ag 1 (inode=2492166168) sb_fdblocks 793886, counted 11056513 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 data fork in ino 1841992 claims free block 722810940 data fork in ino 1841992 claims free block 722818433 data fork in ino 1841992 claims free block 722766205 data fork in ino 1841992 claims free block 722745607 - agno = 1 data fork in ino 2358538759 claims free block 721347041 data fork in ino 2358538759 claims free block 721149027 - agno = 2 data fork in ino 4296520813 claims free block 556498438 data fork in ino 4296520813 claims free block 578099252 data fork in ino 4296520813 claims free block 773182227 data fork in ino 4296520813 claims free block 773182467 data fork in ino 4296520817 claims free block 573284307 data fork in ino 4296520817 claims free block 639394219 data fork in ino 4296520817 claims free block 639395162 data fork in ino 4296733237 claims free block 616345471 data fork in ino 4296733237 claims free block 639394507 data fork in ino 4296733237 claims free block 639395362 data fork in ino 4296739755 claims free block 641669917 data fork in ino 4296739755 claims free block 699854846 data fork in ino 4296739755 claims free block 699839130 data fork in ino 4296739755 claims free block 632622970 data fork in ino 4296739755 claims free block 632630486 data fork in ino 4296739755 claims free block 643598034 data fork in ino 4296739755 claims free block 632630142 data fork in ino 4296739755 claims free block 585572404 data fork in ino 4296739755 claims free block 585563566 data fork in ino 4296739755 claims free block 585563626 data fork in ino 4296744535 claims free block 711244085 data fork in ino 4296744537 claims free block 578099040 data fork in ino 4296744541 claims free block 596876767 data fork in ino 4296744541 claims free block 763252191 data fork in ino 4296744541 claims free block 763255439 data fork in ino 4296744541 claims free block 763255426 data fork in ino 4296744541 claims free block 763255442 data fork in ino 4296744541 claims free block 641708205 data fork in ino 4296744541 claims free block 584836169 data fork in ino 4296744541 claims free block 643965222 data fork in ino 4296744541 claims free block 643962849 data fork in ino 4296744541 claims free block 643962780 data fork in ino 4296744541 claims free block 643958036 data fork in ino 4296744541 claims free block 643962552 data fork in ino 4296744541 claims free block 639248129 data fork in ino 4296744541 claims free block 715506878 data fork in ino 4296744541 claims free block 660075640 data fork in ino 4296744541 claims free block 660051040 data fork in ino 4296744541 claims free block 584913266 data fork in ino 4296744541 claims free block 722709888 data fork in ino 4296744541 claims free block 722710684 data fork in ino 4296744541 claims free block 584912986 data fork in ino 4296744544 claims free block 546384767 data fork in ino 4296744546 claims free block 546619004 data fork in ino 4296744546 claims free block 546619106 data fork in ino 4296744546 claims free block 546619200 data fork in ino 4296744547 claims free block 577095121 data fork in ino 4296744547 claims free block 577095239 data fork in ino 4296744547 claims free block 577095345 data fork in ino 4296744547 claims free block 577096441 data fork in ino 4296744547 claims free block 577096535 data fork in ino 4296744547 claims free block 577108880 data fork in ino 4296744547 claims free block 585468837 data fork in ino 4296744553 claims free block 546619278 data fork in ino 4296744553 claims free block 598007436 - agno = 3 data fork in ino 6446982893 claims free block 721391881 - agno = 4 data fork in ino 8590740692 claims free block 664165116 - agno = 5 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 2 - agno = 0 - agno = 3 - agno = 1 - agno = 5 - agno = 4 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... disconnected inode 2492166168, moving to lost+found Phase 7 - verify and correct link counts... done

Disk 7 still shows up as unmountable.

Ok, checked disk 7. Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 2 - agno = 1 - agno = 3 - agno = 0 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting. Ran again without the -n flag and attached the diagnostics. orthanc-diagnostics-20220808-1844.zip

Anything important is backed up to another machine and to backblaze, so I'm not too worried about missing files at this point. Is it safe to ignore for now, or do I need to address it before I move on?

The audio book files I found are also on the correct share for Media/audiobook files.

Scratch that, after clicking into more folders I found some audiobook files.

I have not looked in that share. I have probably repair the other disks at some point in time, yes. What should I do to restore that data? Looking in the share I just see a bunch of folder and files with alpha-numeric names that don't have any meaning to me.

Restarted the array in normal mode orthanc-diagnostics-20220808-1556.zip

Ok, xfs_repair on disk 6 finished: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... Metadata CRC error detected at 0x439496, xfs_agf block 0x1ffffffe1/0x200 agf has bad CRC for ag 4 block (2,184178199-184178280) multiply claimed by bno space tree, state - 1 block (2,226384586-226384833) multiply claimed by cnt space tree, state - 2 block (2,123204880-123205051) multiply claimed by cnt space tree, state - 2 block (2,40225617-40225622) multiply claimed by cnt space tree, state - 2 block (2,40225773-40225859) multiply claimed by cnt space tree, state - 2 block (2,102377537-102377552) multiply claimed by cnt space tree, state - 2 block (2,36413631-36413735) multiply claimed by cnt space tree, state - 2 block (2,61136756-61136865) multiply claimed by cnt space tree, state - 2 block (2,174373505-174373514) multiply claimed by cnt space tree, state - 2 block (2,9748180-9748193) multiply claimed by cnt space tree, state - 2 block (2,9748274-9748287) multiply claimed by cnt space tree, state - 2 block (2,104799337-104799352) multiply claimed by cnt space tree, state - 2 block (2,123180332-123180475) multiply claimed by cnt space tree, state - 2 block (2,162984111-162984288) multiply claimed by cnt space tree, state - 2 block (2,60006119-60006216) multiply claimed by cnt space tree, state - 2 block (2,102524589-102524613) multiply claimed by cnt space tree, state - 2 block (2,40224317-40224326) multiply claimed by cnt space tree, state - 2 block (2,40224419-40224432) multiply claimed by cnt space tree, state - 2 block (2,40224497-40224575) multiply claimed by cnt space tree, state - 2 block (2,107092002-107092006) multiply claimed by cnt space tree, state - 2 block (2,236311683-236311689) multiply claimed by cnt space tree, state - 2 block (2,107087496-107087499) multiply claimed by cnt space tree, state - 2 block (2,48692963-48693032) multiply claimed by cnt space tree, state - 2 block (2,106727218-106727508) multiply claimed by cnt space tree, state - 2 block (2,104837401-104837680) multiply claimed by cnt space tree, state - 2 block (2,9514183-9514243) multiply claimed by cnt space tree, state - 2 block (2,48042435-48042473) multiply claimed by cnt space tree, state - 2 block (2,40238304-40238370) multiply claimed by cnt space tree, state - 2 block (2,162968554-162968627) multiply claimed by cnt space tree, state - 2 block (2,47965621-47965668) multiply claimed by cnt space tree, state - 2 block (2,178636310-178636383) multiply claimed by cnt space tree, state - 2 block (2,102523728-102523738) multiply claimed by cnt space tree, state - 2 block (2,19627766-19627977) multiply claimed by cnt space tree, state - 2 block (2,95752502-95752512) multiply claimed by cnt space tree, state - 2 block (2,127294648-127294666) multiply claimed by cnt space tree, state - 2 agf_freeblks 2038004, counted 2048802 in ag 2 agf_freeblks 2116416, counted 2106811 in ag 4 agi unlinked bucket 24 is 344682520 in ag 1 (inode=2492166168) sb_fdblocks 793886, counted 11056513 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 data fork in ino 1841992 claims free block 722810940 data fork in ino 1841992 claims free block 722818433 data fork in ino 1841992 claims free block 722766205 data fork in ino 1841992 claims free block 722745607 - agno = 1 data fork in ino 2358538759 claims free block 721347041 data fork in ino 2358538759 claims free block 721149027 - agno = 2 data fork in ino 4296520813 claims free block 556498438 data fork in ino 4296520813 claims free block 578099252 data fork in ino 4296520813 claims free block 773182227 data fork in ino 4296520813 claims free block 773182467 data fork in ino 4296520817 claims free block 573284307 data fork in ino 4296520817 claims free block 639394219 data fork in ino 4296520817 claims free block 639395162 data fork in ino 4296733237 claims free block 616345471 data fork in ino 4296733237 claims free block 639394507 data fork in ino 4296733237 claims free block 639395362 data fork in ino 4296739755 claims free block 641669917 data fork in ino 4296739755 claims free block 699854846 data fork in ino 4296739755 claims free block 699839130 data fork in ino 4296739755 claims free block 632622970 data fork in ino 4296739755 claims free block 632630486 data fork in ino 4296739755 claims free block 643598034 data fork in ino 4296739755 claims free block 632630142 data fork in ino 4296739755 claims free block 585572404 data fork in ino 4296739755 claims free block 585563566 data fork in ino 4296739755 claims free block 585563626 data fork in ino 4296744535 claims free block 711244085 data fork in ino 4296744537 claims free block 578099040 data fork in ino 4296744541 claims free block 596876767 data fork in ino 4296744541 claims free block 763252191 data fork in ino 4296744541 claims free block 763255439 data fork in ino 4296744541 claims free block 763255426 data fork in ino 4296744541 claims free block 763255442 data fork in ino 4296744541 claims free block 641708205 data fork in ino 4296744541 claims free block 584836169 data fork in ino 4296744541 claims free block 643965222 data fork in ino 4296744541 claims free block 643962849 data fork in ino 4296744541 claims free block 643962780 data fork in ino 4296744541 claims free block 643958036 data fork in ino 4296744541 claims free block 643962552 data fork in ino 4296744541 claims free block 639248129 data fork in ino 4296744541 claims free block 715506878 data fork in ino 4296744541 claims free block 660075640 data fork in ino 4296744541 claims free block 660051040 data fork in ino 4296744541 claims free block 584913266 data fork in ino 4296744541 claims free block 722709888 data fork in ino 4296744541 claims free block 722710684 data fork in ino 4296744541 claims free block 584912986 data fork in ino 4296744544 claims free block 546384767 data fork in ino 4296744546 claims free block 546619004 data fork in ino 4296744546 claims free block 546619106 data fork in ino 4296744546 claims free block 546619200 data fork in ino 4296744547 claims free block 577095121 data fork in ino 4296744547 claims free block 577095239 data fork in ino 4296744547 claims free block 577095345 data fork in ino 4296744547 claims free block 577096441 data fork in ino 4296744547 claims free block 577096535 data fork in ino 4296744547 claims free block 577108880 data fork in ino 4296744547 claims free block 585468837 data fork in ino 4296744553 claims free block 546619278 data fork in ino 4296744553 claims free block 598007436 - agno = 3 data fork in ino 6446982893 claims free block 721391881 - agno = 4 data fork in ino 8590740692 claims free block 664165116 - agno = 5 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 2 - agno = 0 - agno = 3 - agno = 1 - agno = 5 - agno = 4 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... disconnected inode 2492166168, moving to lost+found Phase 7 - verify and correct link counts... done

Running xfs_repair without the -n flag on disk 6 now.

xfs_repair for disk 6 finished: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... Metadata CRC error detected at 0x439496, xfs_agf block 0x1ffffffe1/0x200 agf has bad CRC for ag 4 block (2,184178199-184178280) multiply claimed by bno space tree, state - 1 block (2,226384586-226384833) multiply claimed by cnt space tree, state - 2 block (2,123204880-123205051) multiply claimed by cnt space tree, state - 2 block (2,40225617-40225622) multiply claimed by cnt space tree, state - 2 block (2,40225773-40225859) multiply claimed by cnt space tree, state - 2 block (2,102377537-102377552) multiply claimed by cnt space tree, state - 2 block (2,36413631-36413735) multiply claimed by cnt space tree, state - 2 block (2,61136756-61136865) multiply claimed by cnt space tree, state - 2 block (2,174373505-174373514) multiply claimed by cnt space tree, state - 2 block (2,9748180-9748193) multiply claimed by cnt space tree, state - 2 block (2,9748274-9748287) multiply claimed by cnt space tree, state - 2 block (2,104799337-104799352) multiply claimed by cnt space tree, state - 2 block (2,123180332-123180475) multiply claimed by cnt space tree, state - 2 block (2,162984111-162984288) multiply claimed by cnt space tree, state - 2 block (2,60006119-60006216) multiply claimed by cnt space tree, state - 2 block (2,102524589-102524613) multiply claimed by cnt space tree, state - 2 block (2,40224317-40224326) multiply claimed by cnt space tree, state - 2 block (2,40224419-40224432) multiply claimed by cnt space tree, state - 2 block (2,40224497-40224575) multiply claimed by cnt space tree, state - 2 block (2,107092002-107092006) multiply claimed by cnt space tree, state - 2 block (2,236311683-236311689) multiply claimed by cnt space tree, state - 2 block (2,107087496-107087499) multiply claimed by cnt space tree, state - 2 block (2,48692963-48693032) multiply claimed by cnt space tree, state - 2 block (2,106727218-106727508) multiply claimed by cnt space tree, state - 2 block (2,104837401-104837680) multiply claimed by cnt space tree, state - 2 block (2,9514183-9514243) multiply claimed by cnt space tree, state - 2 block (2,48042435-48042473) multiply claimed by cnt space tree, state - 2 block (2,40238304-40238370) multiply claimed by cnt space tree, state - 2 block (2,162968554-162968627) multiply claimed by cnt space tree, state - 2 block (2,47965621-47965668) multiply claimed by cnt space tree, state - 2 block (2,178636310-178636383) multiply claimed by cnt space tree, state - 2 block (2,102523728-102523738) multiply claimed by cnt space tree, state - 2 block (2,19627766-19627977) multiply claimed by cnt space tree, state - 2 block (2,95752502-95752512) multiply claimed by cnt space tree, state - 2 block (2,127294648-127294666) multiply claimed by cnt space tree, state - 2 agf_freeblks 2038004, counted 2048802 in ag 2 agf_freeblks 2116416, counted 2106811 in ag 4 agi unlinked bucket 24 is 344682520 in ag 1 (inode=2492166168) sb_fdblocks 793886, counted 11056513 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 data fork in ino 1841992 claims free block 722810940 data fork in ino 1841992 claims free block 722818433 data fork in ino 1841992 claims free block 722766205 data fork in ino 1841992 claims free block 722745607 - agno = 1 data fork in ino 2358538759 claims free block 721347041 data fork in ino 2358538759 claims free block 721149027 - agno = 2 data fork in ino 4296520813 claims free block 556498438 data fork in ino 4296520813 claims free block 578099252 data fork in ino 4296520813 claims free block 773182227 data fork in ino 4296520813 claims free block 773182467 data fork in ino 4296520817 claims free block 573284307 data fork in ino 4296520817 claims free block 639394219 data fork in ino 4296520817 claims free block 639395162 data fork in ino 4296733237 claims free block 616345471 data fork in ino 4296733237 claims free block 639394507 data fork in ino 4296733237 claims free block 639395362 data fork in ino 4296739755 claims free block 641669917 data fork in ino 4296739755 claims free block 699854846 data fork in ino 4296739755 claims free block 699839130 data fork in ino 4296739755 claims free block 632622970 data fork in ino 4296739755 claims free block 632630486 data fork in ino 4296739755 claims free block 643598034 data fork in ino 4296739755 claims free block 632630142 data fork in ino 4296739755 claims free block 585572404 data fork in ino 4296739755 claims free block 585563566 data fork in ino 4296739755 claims free block 585563626 data fork in ino 4296744535 claims free block 711244085 data fork in ino 4296744537 claims free block 578099040 data fork in ino 4296744541 claims free block 596876767 data fork in ino 4296744541 claims free block 763252191 data fork in ino 4296744541 claims free block 763255439 data fork in ino 4296744541 claims free block 763255426 data fork in ino 4296744541 claims free block 763255442 data fork in ino 4296744541 claims free block 641708205 data fork in ino 4296744541 claims free block 584836169 data fork in ino 4296744541 claims free block 643965222 data fork in ino 4296744541 claims free block 643962849 data fork in ino 4296744541 claims free block 643962780 data fork in ino 4296744541 claims free block 643958036 data fork in ino 4296744541 claims free block 643962552 data fork in ino 4296744541 claims free block 639248129 data fork in ino 4296744541 claims free block 715506878 data fork in ino 4296744541 claims free block 660075640 data fork in ino 4296744541 claims free block 660051040 data fork in ino 4296744541 claims free block 584913266 data fork in ino 4296744541 claims free block 722709888 data fork in ino 4296744541 claims free block 722710684 data fork in ino 4296744541 claims free block 584912986 data fork in ino 4296744544 claims free block 546384767 data fork in ino 4296744546 claims free block 546619004 data fork in ino 4296744546 claims free block 546619106 data fork in ino 4296744546 claims free block 546619200 data fork in ino 4296744547 claims free block 577095121 data fork in ino 4296744547 claims free block 577095239 data fork in ino 4296744547 claims free block 577095345 data fork in ino 4296744547 claims free block 577096441 data fork in ino 4296744547 claims free block 577096535 data fork in ino 4296744547 claims free block 577108880 data fork in ino 4296744547 claims free block 585468837 data fork in ino 4296744553 claims free block 546619278 data fork in ino 4296744553 claims free block 598007436 - agno = 3 data fork in ino 6446982893 claims free block 721391881 - agno = 4 data fork in ino 8590740692 claims free block 664165116 - agno = 5 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... free space (2,36216381-36216733) only seen by one free space btree free space (2,40529923-40530269) only seen by one free space btree free space (2,47525503-47525835) only seen by one free space btree free space (2,48366886-48367196) only seen by one free space btree free space (2,82534521-82534831) only seen by one free space btree free space (2,100630810-100631194) only seen by one free space btree free space (2,106722377-106722817) only seen by one free space btree free space (2,119468999-119469400) only seen by one free space btree free space (2,119830899-119831347) only seen by one free space btree free space (2,187768370-187768613) only seen by one free space btree free space (2,187768626-187768701) only seen by one free space btree free space (2,187785086-187785185) only seen by one free space btree - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 1 - agno = 3 - agno = 4 - agno = 5 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... disconnected inode 2492166168, would move to lost+found Phase 7 - verify link counts... would have reset inode 2492166168 nlinks from 0 to 1 No modify flag set, skipping filesystem flush and exiting.

It's the one that I just replaced with the 14 TB drive. I was hoping to verify that the drive was dead before I removed it from the server. I ran a pre-clear on it last night which failed, so it's not looking good.

That drive isn't in use, and I am fairly certain that the drive is bad.

Running xfs_repair from the UI with the `-n` flag and it has been hanging for about 20 minutes. Here is the output so far: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... Metadata CRC error detected at 0x439496, xfs_agf block 0x1ffffffe1/0x200 agf has bad CRC for ag 4 block (2,184178199-184178280) multiply claimed by bno space tree, state - 1 block (2,226384586-226384833) multiply claimed by cnt space tree, state - 2 block (2,123204880-123205051) multiply claimed by cnt space tree, state - 2 block (2,40225617-40225622) multiply claimed by cnt space tree, state - 2 block (2,40225773-40225859) multiply claimed by cnt space tree, state - 2 block (2,102377537-102377552) multiply claimed by cnt space tree, state - 2 block (2,36413631-36413735) multiply claimed by cnt space tree, state - 2 block (2,61136756-61136865) multiply claimed by cnt space tree, state - 2 block (2,174373505-174373514) multiply claimed by cnt space tree, state - 2 block (2,9748180-9748193) multiply claimed by cnt space tree, state - 2 block (2,9748274-9748287) multiply claimed by cnt space tree, state - 2 block (2,104799337-104799352) multiply claimed by cnt space tree, state - 2 block (2,123180332-123180475) multiply claimed by cnt space tree, state - 2 block (2,162984111-162984288) multiply claimed by cnt space tree, state - 2 block (2,60006119-60006216) multiply claimed by cnt space tree, state - 2 block (2,102524589-102524613) multiply claimed by cnt space tree, state - 2 block (2,40224317-40224326) multiply claimed by cnt space tree, state - 2 block (2,40224419-40224432) multiply claimed by cnt space tree, state - 2 block (2,40224497-40224575) multiply claimed by cnt space tree, state - 2 block (2,107092002-107092006) multiply claimed by cnt space tree, state - 2 block (2,236311683-236311689) multiply claimed by cnt space tree, state - 2 block (2,107087496-107087499) multiply claimed by cnt space tree, state - 2 block (2,48692963-48693032) multiply claimed by cnt space tree, state - 2 block (2,106727218-106727508) multiply claimed by cnt space tree, state - 2 block (2,104837401-104837680) multiply claimed by cnt space tree, state - 2 block (2,9514183-9514243) multiply claimed by cnt space tree, state - 2 block (2,48042435-48042473) multiply claimed by cnt space tree, state - 2 block (2,40238304-40238370) multiply claimed by cnt space tree, state - 2 block (2,162968554-162968627) multiply claimed by cnt space tree, state - 2 block (2,47965621-47965668) multiply claimed by cnt space tree, state - 2 block (2,178636310-178636383) multiply claimed by cnt space tree, state - 2 block (2,102523728-102523738) multiply claimed by cnt space tree, state - 2 block (2,19627766-19627977) multiply claimed by cnt space tree, state - 2 block (2,95752502-95752512) multiply claimed by cnt space tree, state - 2 block (2,127294648-127294666) multiply claimed by cnt space tree, state - 2 agf_freeblks 2038004, counted 2048802 in ag 2 agf_freeblks 2116416, counted 2106811 in ag 4 agi unlinked bucket 24 is 344682520 in ag 1 (inode=2492166168) sb_fdblocks 793886, counted 11056513 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0

Here's my update. I let the rebuild and parity checks finish, and I still had a dead drive. I decided to just remove the 6 TB drive and replace it with a larger one. I got a good deal on a 16 TB seagate, so I swapped out the 14 TB drive in parity for the larger 16 TB drive after a preclear. Both parity drives are now 16tb. I let parity rebuild onto the new drive, and swapped out the dead 6 TB for the 14 TB drive (that has always worked fine). The data rebuilt and I had all drives green! But, the last 2 drives still show up as unmountable. I did some digging into the Marvell raid controller. I initially thought it had to be the motherboard since it says it has 4 ports, but nope, it's the expansion card which has 6 ports on it. I decided to stop the array and see if I could troubleshoot, and this is what I saw next. Only 2 drives showed up as attached and the rest were missing. I downloaded the diagnostics and rebooted, now the drives are all back. Disks 6 and 7 still show up as unmountable. My thoughts are that I should replace the sata controller and see if that helps. My question at this point is do I need to do something with the unmountable drives? I know disk 6 is a good disk since it's been in use as a parity drive and I did not unplug or move the port it is connected to, just assigned it in the UI. I also attached the diagnostics for after I rebooted if that helps. It's the one that ends in "-2". Thanks! orthanc-diagnostics-20220807-2206.zip orthanc-diagnostics-20220807-2219-2.zip

Gotcha, I'll report back in once the rebuild is done.

No errors are showing up. I'm freeing some more storage space from disk 4 before I stop the array and try and do some file system repair. Current view on the UI: Disk 7 did eventually show up after I swapped out the sata cable for one that connected better. I tried moving disks 3, 5, and 6 to other sata ports on the motherboard instead of the expansion card, and I also tried different power connectors, and nothing changed. I will attempt to do some file system repair and see how it goes.

I was able to get some time to take the server out of the rack and up to my office (thank god the kids are napping lol). I opened it up and disconnected all the sata cables and power. I carefully reconnected everything and double checked the connections. I still have multiple drives that show up as unmountable. orthanc-diagnostics-20220717-1849.zip

OK, moved some cables around to see what happened, and some drives are not showing up at all. It's in a small server rack in the basement, so it'll take me some effort to pull the case fully open to get to the inner connectors. The case also has some quick replace slots in the front, I'll also try plugging directly to the drives. Thanks for the help so far, it might be a day or two before I get to the next step.