Hey guys, I goofed up, I see that now. For starters I went from 6.12.x (I think it was 11) to 6.12.14 but I didn't reboot. So time went on, parity check. then I shutdown and installed a HBA, I only attached a unassigned drive at first. when I booted it up over half the array was missing. I moved drives on then off the HBA. Finally I rebooted with no drives on the HBA. almost everything came back but still not enough to start the array. Giving up I just pulled the HBA out of the PCIe slot after shutting it down. When I turned it on again it was good enough to start the array. Now I have 2 drives that are disabled (dual parity). I started with drive 3. I tried a xfs_repair but it failed. So I just dropped it and did a in place rebuild. rebuilding the data from parity onto itself. However it is now a green dot, but where the gauges should be it says it is "Unmountable: unsupported or no file system" drive 4 is currently rebuilding in place to fix it as I do not like having even 1 drive emulated due to my luck. In krusader disk 3 is just gone. I got the email that said disk 3 returned to normal operation but it is still not functioning as expected.

In summary drive 3 and 4 are corrupted or something. I tried xfs_repair -L on both, and have done a in place rebuild for disk 3 but it's still not working.

Anyone here have further advice? I am beginning to wonder if it is a kernel issue but as upgrading got me here I am afraid to go near that button or the downgrade button.

1 hour ago, ericup said:

tried xfs_repair -L on both

Did you do this from the webUI? Easy to do it wrong from the command line.

 

Attach Diagnostics to your NEXT post in this thread.

25 minutes ago, trurl said:

Did you do this from the webUI? Easy to do it wrong from the command line.

 

Attach Diagnostics to your NEXT post in this thread.

Yes i used webUI, I don't trust myself to run something so potentially destructive from command line.

tower-diagnostics-20250113-2216.zip

Disk2 is also showing filesystem issues, fix that one, then reboot to clear the logs and post new diags after array start.

I am 22 hours into a data rebuild on disk 4. Should I pause that and start messing with disk 2?

Looks like emulated disk4 was mountable in those last diagnostics so let it finish rebuild.

Cool, I will let it run. another day to complete rebuild. I get 12tb is a lot of data and takes time but is there any way to decrease the time needed in the future? I hate having my array unprotected for days at a time.

2 hours ago, ericup said:

any way to decrease the time

no

2 hours ago, ericup said:

12tb is a lot of data and takes time

 

Another day to complete rebuild?  12TB should not take a full day let alone another day.  I would take a look at your syslog and see if you have any disk errors.

4 hours ago, ericup said:

Cool, I will let it run. another day to complete rebuild. I get 12tb is a lot of data and takes time but is there any way to decrease the time needed in the future? I hate having my array unprotected for days at a time.

I would normally expect something like 1.5 to 2 hours per TB being rebuilt so it sounds as if the rebuild is taking longer than expected.    Another way of looking at it  is that it should be comparable to doing 12TB of parity check on your system.

17 hours ago, itimpi said:

I would normally expect something like 1.5 to 2 hours per TB being rebuilt so it sounds as if the rebuild is taking longer than expected.    Another way of looking at it  is that it should be comparable to doing 12TB of parity check on your system.

Yeah, that's how long my parity checks take too. 

 

Jan 15 10:12:39 Tower kernel: md: sync done. time=136794sec

Jan 15 10:12:40 Tower kernel: md: recovery thread: exit status: 0

I'm not seeing any disk errors since 9 pm last night. Those were "metadata I/O error" on disk 3 which is "unmountable" anyway.

 

On 1/14/2025 at 3:08 AM, JorgeB said:

Disk2 is also showing filesystem issues, fix that one, then reboot to clear the logs and post new diags after array start.

Anything I should do before a reboot now that 4 is good? For refrence 3 is unmountable and you say you saw an error on 2 though it seems to be working.

You can just reboot and post new diags after array start

Ok, it rebooted. now disk 2 and 3 are unmountable. Fresh Diags attached. should I run a xfs_repair on them?

tower-diagnostics-20250116-1021.zip

Yep, run it without -n, and if it asks for -L use it.

 

Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this.

 

so next i run -L correct?

Just now, ericup said:

so next i run -L correct?

Yes - without -n and adding -L

1 hour ago, JorgeB said:

if it asks for -L use it

2 minutes ago, ericup said:

If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair.

 

Unraid has already determined the disk is unmountable, but the XFS tool doesn't know this. So it tells you to try so it can replay the log.

last few lines of the log:

cache_purge: shake on cache 0x50c660 left 1 nodes!?

cache_purge: shake on cache 0x50c660 left 1 nodes!?

cache_zero_check: refcount is 1, not zero (node=0x1545d3251a10) done

 

i dont know what that means but i imagine that noting it closed at 1 not 0 is is important.

Should I reboot or try disk 3 first?

1 hour ago, ericup said:

last few lines of the log:

 

Post the complete output of that check filesystem.

2 minutes ago, trurl said:

 

Post the complete output of that check filesystem.

Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... clearing needsrepair flag and regenerating metadata agi unlinked bucket 7 is 33586183 in ag 3 (inode=6476037127) agi_freecount 23650, counted 23651 in ag 8 agi unlinked bucket 44 is 205868 in ag 8 (inode=17180075052) agi_freecount 71, counted 64 in ag 5 agi_freecount 71, counted 64 in ag 5 finobt agi_freecount 83, counted 77 in ag 9 agi_freecount 83, counted 77 in ag 9 finobt sb_ifree 155973, counted 156485 sb_fdblocks 732288671, counted 749316530 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 imap claims a free inode 6477553055 is in use, correcting imap and clearing inode cleared inode 6477553055 - agno = 4 Metadata CRC error detected at 0x4626c8, xfs_dir3_data block 0x201449fa0/0x1000 bad directory block magic # 0x1000010 in block 17 for directory inode 8611209547 corrupt block 17 in directory inode 8611209547 will junk block Metadata CRC error detected at 0x45fb48, xfs_dir3_block block 0x20144ba80/0x1000 bad directory block magic # 0xffffffff in block 0 for directory inode 8611214513 corrupt block 0 in directory inode 8611214513 will junk block no . entry for directory 8611214513 no .. entry for directory 8611214513 problem with directory contents in inode 8611214513 cleared inode 8611214513 - agno = 5 imap claims a free inode 10748652669 is in use, correcting imap and clearing inode cleared inode 10748652669 imap claims a free inode 10748652670 is in use, correcting imap and clearing inode cleared inode 10748652670 imap claims a free inode 10748652671 is in use, correcting imap and clearing inode cleared inode 10748652671 imap claims a free inode 10748652800 is in use, correcting imap and clearing inode cleared inode 10748652800 - agno = 6 imap claims a free inode 12884965786 is in use, correcting imap and clearing inode cleared inode 12884965786 imap claims a free inode 12885109022 is in use, correcting imap and clearing inode cleared inode 12885109022 imap claims a free inode 12885111982 is in use, correcting imap and clearing inode cleared inode 12885111982 - agno = 7 imap claims a free inode 15032386727 is in use, correcting imap and clearing inode cleared inode 15032386727 imap claims a free inode 15032779199 is in use, correcting imap and clearing inode cleared inode 15032779199 - agno = 8 imap claims a free inode 17179997713 is in use, correcting imap and clearing inode cleared inode 17179997713 imap claims a free inode 17179997721 is in use, correcting imap and clearing inode cleared inode 17179997721 imap claims a free inode 17180000180 is in use, correcting imap and clearing inode cleared inode 17180000180 - agno = 9 imap claims a free inode 19327353235 is in use, correcting imap and clearing inode cleared inode 19327353235 imap claims a free inode 19327353237 is in use, correcting imap and clearing inode cleared inode 19327353237 imap claims a free inode 19327353238 is in use, correcting imap and clearing inode cleared inode 19327353238 imap claims a free inode 19327362283 is in use, correcting imap and clearing inode cleared inode 19327362283 imap claims a free inode 19328594331 is in use, correcting imap and clearing inode cleared inode 19328594331 imap claims a free inode 19328594332 is in use, correcting imap and clearing inode cleared inode 19328594332 Metadata CRC error detected at 0x45fb48, xfs_dir3_block block 0x48012bae0/0x1000 bad directory block magic # 0x7475726e in block 0 for directory inode 19328616862 corrupt block 0 in directory inode 19328616862 will junk block no . entry for directory 19328616862 no .. entry for directory 19328616862 problem with directory contents in inode 19328616862 cleared inode 19328616862 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 5 - agno = 9 - agno = 3 - agno = 4 - agno = 2 - agno = 7 - agno = 6 - agno = 8 bad directory block magic # 0x1000010 in block 17 for directory inode 8611209547 corrupt block 17 in directory inode 8611209547 will junk block entry "readable-stream" in shortform directory 17180875851 references free inode 19328594331 junking entry "readable-stream" in directory inode 17180875851 corrected i8 count in directory 17180875851, was 2, now 1 entry "wg-meetings" in shortform directory 17180875852 references free inode 19328594332 junking entry "wg-meetings" in directory inode 17180875852 corrected i8 count in directory 17180875852, was 2, now 1 entry "gravitar" in shortform directory 4310164634 references free inode 6477553055 junking entry "gravitar" in directory inode 4310164634 corrected i8 count in directory 4310164634, was 1, now 0 corrected directory 4310164634 size, was 10, now 6 entry "gui" in shortform directory 17180879987 references free inode 19328616862 junking entry "gui" in directory inode 17180879987 corrected i8 count in directory 17180879987, was 2, now 1 entry "Genesis 8 Male" in shortform directory 17180879993 references free inode 15032779199 junking entry "Genesis 8 Male" in directory inode 17180879993 corrected i8 count in directory 17180879993, was 4, now 3 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... Metadata CRC error detected at 0x4626c8, xfs_dir3_data block 0x201449fa0/0x1000 corrupt block 17 in directory inode 8611209547: junking block rebuilding directory inode 8611209547 entry "https_www.fandom.com_0.indexeddb.leveldb" in directory inode 10748652659 references already connected inode 2150699336. entry "Default" in directory inode 12884965745 references already connected inode 8606594693. entry "TpcdMetadata" in directory inode 12884965745 references already connected inode 8606594699. fixing i8count in inode 12884965745 entry "9374" in directory inode 15032386281 not consistent with .. value (19327353227) in inode 1625, entry "_metadata" in directory inode 17179997674 references already connected inode 8606594692. fixing i8count in inode 17179997674 entry "Users" in directory inode 19327353219 not consistent with .. value (19327353221) in inode 578, Metadata corruption detected at 0x45acb0, xfs_dir3_leaf1 block 0x483ffff80/0x1000 leaf block 8388608 for directory inode 19328506324 bad CRC rebuilding directory inode 19328506324 Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected Warning: recursive buffer locking at block 19394461568 detected - traversal finished ... - moving disconnected inodes to lost+found ... disconnected dir inode 2094240, moving to lost+found disconnected dir inode 4310164613, moving to lost+found disconnected inode 10748650463, moving to lost+found disconnected inode 10748650467, moving to lost+found disconnected dir inode 10748651142, moving to lost+found disconnected dir inode 10748652659, moving to lost+found disconnected dir inode 10748652661, moving to lost+found disconnected inode 15032459409, moving to lost+found disconnected inode 15032459411, moving to lost+found disconnected dir inode 17180075052, moving to lost+found disconnected dir inode 17180879994, moving to lost+found Phase 7 - verify and correct link counts... resetting inode 17179997674 nlinks from 3 to 2 resetting inode 15032386281 nlinks from 5 to 4 resetting inode 19327353219 nlinks from 3 to 2 resetting inode 6476037127 nlinks from 0 to 1 resetting inode 6476038946 nlinks from 15 to 16 resetting inode 6476038963 nlinks from 15 to 16 resetting inode 6476038971 nlinks from 15 to 16 resetting inode 6476038991 nlinks from 15 to 16 resetting inode 6476039064 nlinks from 15 to 16 resetting inode 6476045693 nlinks from 15 to 16 resetting inode 4306809799 nlinks from 9 to 8 resetting inode 4306809800 nlinks from 9 to 8 resetting inode 10748652659 nlinks from 4 to 3 resetting inode 15032459749 nlinks from 16 to 15 resetting inode 15032459750 nlinks from 16 to 15 resetting inode 15032459753 nlinks from 16 to 15 resetting inode 17180075052 nlinks from 0 to 2 resetting inode 4310164634 nlinks from 3 to 2 resetting inode 2070111 nlinks from 2 to 9 resetting inode 17180875851 nlinks from 3 to 2 resetting inode 17180875852 nlinks from 3 to 2 resetting inode 17180879987 nlinks from 3 to 2 resetting inode 17180879993 nlinks from 5 to 4 resetting inode 19327728586 nlinks from 13 to 12 resetting inode 12884965745 nlinks from 5 to 3 resetting inode 12885110582 nlinks from 18 to 19 resetting inode 4429722 nlinks from 19 to 20 resetting inode 19328376990 nlinks from 15 to 16 resetting inode 19328376992 nlinks from 16 to 17 resetting inode 8611764884 nlinks from 9 to 10 resetting inode 15032740356 nlinks from 14 to 13 resetting inode 15032740474 nlinks from 16 to 15 resetting inode 19328645801 nlinks from 18 to 19 resetting inode 17200702783 nlinks from 19 to 20 resetting inode 17375650216 nlinks from 8 to 9 resetting inode 17375650219 nlinks from 8 to 9 resetting inode 19346776545 nlinks from 16 to 17 resetting inode 4375175321 nlinks from 4 to 5 resetting inode 4375175343 nlinks from 4 to 5 resetting inode 9520157981 nlinks from 10 to 11 resetting inode 9574597494 nlinks from 10 to 11 resetting inode 13024127441 nlinks from 8 to 7 resetting inode 13024127452 nlinks from 8 to 7 resetting inode 13024127457 nlinks from 8 to 7 resetting inode 13024127458 nlinks from 8 to 7 resetting inode 13047722429 nlinks from 8 to 7 resetting inode 13047722431 nlinks from 8 to 7 resetting inode 13047722433 nlinks from 8 to 7 resetting inode 13047722436 nlinks from 8 to 7 resetting inode 13047722437 nlinks from 8 to 7 resetting inode 13047722440 nlinks from 8 to 7 resetting inode 13047722441 nlinks from 8 to 7 resetting inode 13047722442 nlinks from 8 to 7 resetting inode 13047722443 nlinks from 8 to 7 resetting inode 13047722444 nlinks from 8 to 7 resetting inode 13047722452 nlinks from 8 to 7 resetting inode 13047722453 nlinks from 8 to 7 resetting inode 11067771035 nlinks from 7 to 6 resetting inode 7014637248 nlinks from 6 to 5 resetting inode 11071244289 nlinks from 7 to 6 resetting inode 11071244290 nlinks from 7 to 6 resetting inode 7014637249 nlinks from 6 to 5 resetting inode 7014637252 nlinks from 6 to 5 resetting inode 11071244291 nlinks from 7 to 6 resetting inode 7014637254 nlinks from 6 to 5 resetting inode 11071244293 nlinks from 7 to 6 resetting inode 7014637255 nlinks from 6 to 5 resetting inode 7014637260 nlinks from 6 to 5 resetting inode 11071244295 nlinks from 7 to 6 resetting inode 7014637261 nlinks from 6 to 5 resetting inode 11071244304 nlinks from 7 to 6 resetting inode 11071244305 nlinks from 7 to 6 resetting inode 7014637262 nlinks from 6 to 5 resetting inode 11071244306 nlinks from 7 to 6 resetting inode 7014637266 nlinks from 6 to 5 resetting inode 11071244311 nlinks from 7 to 6 resetting inode 7014637268 nlinks from 6 to 5 resetting inode 11071244313 nlinks from 7 to 6 resetting inode 7014637269 nlinks from 6 to 5 resetting inode 11071244314 nlinks from 7 to 6 resetting inode 11071244315 nlinks from 7 to 6 resetting inode 7014637270 nlinks from 6 to 5 resetting inode 11071244320 nlinks from 7 to 6 resetting inode 7014637273 nlinks from 6 to 5 resetting inode 7014637274 nlinks from 6 to 5 resetting inode 7014637275 nlinks from 6 to 5 resetting inode 11071244323 nlinks from 7 to 6 resetting inode 11071244325 nlinks from 7 to 6 resetting inode 11071244326 nlinks from 7 to 6 resetting inode 11071244339 nlinks from 7 to 6 resetting inode 11071244341 nlinks from 7 to 6 resetting inode 11071244345 nlinks from 7 to 6 resetting inode 11071244349 nlinks from 7 to 6 resetting inode 11071244350 nlinks from 7 to 6 resetting inode 11071244355 nlinks from 7 to 6 resetting inode 7015457805 nlinks from 11 to 12 resetting inode 11071244360 nlinks from 7 to 6 resetting inode 11071244365 nlinks from 7 to 6 resetting inode 11071244374 nlinks from 7 to 6 resetting inode 11071244376 nlinks from 7 to 6 resetting inode 11071244379 nlinks from 7 to 6 resetting inode 11071244385 nlinks from 7 to 6 resetting inode 11071244390 nlinks from 7 to 6 resetting inode 11071244391 nlinks from 7 to 6 resetting inode 11071244392 nlinks from 7 to 6 resetting inode 11071244393 nlinks from 7 to 6 resetting inode 11071244396 nlinks from 7 to 6 resetting inode 11071244399 nlinks from 7 to 6 resetting inode 11071244405 nlinks from 7 to 6 resetting inode 11071244412 nlinks from 7 to 6 resetting inode 11071244414 nlinks from 7 to 6 resetting inode 11071244422 nlinks from 7 to 6 resetting inode 11071244423 nlinks from 7 to 6 resetting inode 11071244426 nlinks from 7 to 6 resetting inode 11071244428 nlinks from 7 to 6 resetting inode 11071244431 nlinks from 7 to 6 resetting inode 11071244432 nlinks from 7 to 6 resetting inode 11071244437 nlinks from 7 to 6 resetting inode 11071244439 nlinks from 7 to 6 resetting inode 11071244442 nlinks from 7 to 6 resetting inode 11071244443 nlinks from 7 to 6 resetting inode 11071244444 nlinks from 7 to 6 resetting inode 11071244449 nlinks from 7 to 6 resetting inode 11071244453 nlinks from 7 to 6 resetting inode 11071244454 nlinks from 7 to 6 resetting inode 11109477152 nlinks from 7 to 6 resetting inode 11109477154 nlinks from 7 to 6 resetting inode 11109477160 nlinks from 7 to 6 resetting inode 11109477164 nlinks from 7 to 6 resetting inode 11109477171 nlinks from 7 to 6 resetting inode 11109477179 nlinks from 7 to 6 resetting inode 11109477180 nlinks from 7 to 6 resetting inode 11109477182 nlinks from 7 to 6 resetting inode 11109477184 nlinks from 7 to 6 resetting inode 11109477187 nlinks from 7 to 6 resetting inode 11109477189 nlinks from 7 to 6 resetting inode 11109477190 nlinks from 7 to 6 resetting inode 11109477194 nlinks from 7 to 6 resetting inode 11109477203 nlinks from 7 to 6 resetting inode 11109477207 nlinks from 7 to 6 resetting inode 11109477208 nlinks from 7 to 6 resetting inode 11109477212 nlinks from 7 to 6 resetting inode 11109477215 nlinks from 7 to 6 resetting inode 11109477219 nlinks from 7 to 6 resetting inode 11109477222 nlinks from 7 to 6 resetting inode 11109477225 nlinks from 7 to 6 resetting inode 11109477226 nlinks from 7 to 6 resetting inode 11109477230 nlinks from 7 to 6 resetting inode 11109477232 nlinks from 7 to 6 resetting inode 11109477233 nlinks from 7 to 6 resetting inode 11109477237 nlinks from 7 to 6 resetting inode 11109477238 nlinks from 7 to 6 resetting inode 11109477240 nlinks from 7 to 6 resetting inode 11109477244 nlinks from 7 to 6 resetting inode 11109477246 nlinks from 7 to 6 resetting inode 11109477249 nlinks from 7 to 6 resetting inode 11109477253 nlinks from 7 to 6 resetting inode 11109477257 nlinks from 7 to 6 resetting inode 11109477258 nlinks from 7 to 6 resetting inode 11109477261 nlinks from 7 to 6 resetting inode 11109477264 nlinks from 7 to 6 resetting inode 11109477265 nlinks from 7 to 6 resetting inode 11109477266 nlinks from 7 to 6 resetting inode 11109477269 nlinks from 7 to 6 resetting inode 11109477270 nlinks from 7 to 6 resetting inode 11109477272 nlinks from 7 to 6 resetting inode 11109477275 nlinks from 7 to 6 resetting inode 11109477280 nlinks from 7 to 6 resetting inode 11109477286 nlinks from 7 to 6 resetting inode 11109477287 nlinks from 7 to 6 resetting inode 11109477288 nlinks from 7 to 6 resetting inode 11109477295 nlinks from 7 to 6 resetting inode 11109477297 nlinks from 7 to 6 resetting inode 11109477304 nlinks from 7 to 6 resetting inode 11109477307 nlinks from 7 to 6 resetting inode 11109477309 nlinks from 7 to 6 resetting inode 1502387105 nlinks from 10 to 11 resetting inode 1635625385 nlinks from 19 to 20 Maximum metadata LSN (19:896092) is ahead of log (1:2). Format log to cycle 22. cache_purge: shake on cache 0x50c660 left 1 nodes!? cache_purge: shake on cache 0x50c660 left 1 nodes!? cache_zero_check: refcount is 1, not zero (node=0x1545d3251a10) done

 

Text formatting is weird... not sure how to fix it.

Rebooted with array not auto spun up. everything looks ok in maintenance mode...

 

But I definitly do NOT know what I am looking for, as shown by @trurl seeing a issue with disk 2 before I did. so here is a new diag. Is my server fixed then?

tower-diagnostics-20250116-2000.zip

Start the array in normal mode and post new diagnostics.

Drive 3 is still unmountable but 2 and 4 are fixed. I'm guessing rinse and repeat? xfs_repair then xfs_repair -L if needed?

tower-diagnostics-20250116-2029.zip

Go ahead with disk3.

 

Then post new diagnostics after starting the array in normal mode.