hermy65

Yes, without the -n sorry. Appreciate the help!

@JorgeB Thanks, this is what the -n gave me Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - agno = 13 bad CRC for inode 27918115649 bad CRC for inode 27918115649, will rewrite cleared inode 27918115649 - agno = 14 - agno = 15 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 8 - agno = 13 - agno = 4 - agno = 5 - agno = 7 - agno = 6 - agno = 9 - agno = 10 - agno = 12 - agno = 3 - agno = 14 - agno = 15 - agno = 11 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... done

Woke up this morning and my machine was completely unresponsive, monitor showed some sort of XFS errors but i didnt see a disk number on the screen. Managed to get in via putty and kick of diagnostics before rebooting. Not seeing any errors on the dashboard, no idea what happened or how to fix/prevent it. Diagnostics attached Thanks in advance! storage-diagnostics-20240415-0859.zip

@trurl ok so if i move it to cache and update the path i should be good to go, correct? Thanks again!

@trurl Interesting, checked the system/libvrt folder on cache and its empty, the one on disk17 has the .img in it and it was last modified after i started the array when you told me to. Is there a reason that .img file would be on disk17 and being used when the setting is as you posted above?

@trurl appreciate the help! The lost+found folders have been taken care of, appreciate the alert on that. As for the system folder, it looks like it has my libvrt.img in it for some reason, i assume i can move that to cache and then update the location in VM settings or will that cause an issue?

@trurl updated diagnostics are attached! storage-diagnostics-20240228-0945.zip

@trurl without -n, i see no mention of -L in this output. Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 bad CRC for inode 87062 inode identifier 16149622513293918207 mismatch on inode 87062 bad CRC for inode 87062, will rewrite inode identifier 16149622513293918207 mismatch on inode 87062 cleared inode 87062 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 1 - agno = 3 - agno = 4 - agno = 7 - agno = 6 - agno = 5 entry "11 - The Amity Affliction - Stairway to Hell.mp3" at block 0 offset 736 in directory inode 87051 references free inode 87062 clearing inode number in entry at offset 736... Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... bad hash table for directory inode 87051 (no data entry): rebuilding rebuilding directory inode 87051 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... done

@trurl here is the output: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 bad CRC for inode 87062 inode identifier 16149622513293918207 mismatch on inode 87062 bad CRC for inode 87062, would rewrite inode identifier 16149622513293918207 mismatch on inode 87062 would have cleared inode 87062 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 4 - agno = 2 - agno = 3 - agno = 5 - agno = 6 - agno = 7 - agno = 1 entry "11 - The Amity Affliction - Stairway to Hell.mp3" at block 0 offset 736 in directory inode 87051 references free inode 87062 would clear inode number in entry at offset 736... bad CRC for inode 87062, would rewrite inode identifier 16149622513293918207 mismatch on inode 87062 would have cleared inode 87062 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... entry "11 - The Amity Affliction - Stairway to Hell.mp3" in directory inode 87051 points to free inode 87062, would junk entry bad hash table for directory inode 87051 (no data entry): would rebuild would rebuild directory inode 87051 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting.

Popped open my syslog this morning and its full of this but it doesnt list what disk its detecting the corruption on so i have no idea what to do. None of the disks show errors on the dashboard either. Diagnostics are attached. Feb 28 08:55:51 Storage kernel: XFS (md18p1): Metadata corruption detected at xfs_dinode_verify+0xa0/0x732 [xfs], inode 0x15416 dinode Feb 28 08:55:51 Storage kernel: XFS (md18p1): Unmount and run xfs_repair Feb 28 08:55:51 Storage kernel: XFS (md18p1): First 128 bytes of corrupted metadata buffer: Feb 28 08:55:51 Storage kernel: 00000000: 49 4e 81 ff 03 02 00 00 00 00 00 63 00 00 00 64 IN.........c...d Feb 28 08:55:51 Storage kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................ Feb 28 08:55:51 Storage kernel: 00000020: 5c e8 88 37 21 21 f6 d0 5c e8 88 12 1e 16 0a a8 \..7!!..\....... Feb 28 08:55:51 Storage kernel: 00000030: 5c e8 93 98 08 a2 4f 36 00 00 00 00 00 b7 90 e7 \.....O6........ Feb 28 08:55:51 Storage kernel: 00000040: 00 00 00 00 00 00 0b 7a 00 00 00 00 00 00 00 01 .......z........ Feb 28 08:55:51 Storage kernel: 00000050: 00 00 18 01 00 00 00 00 00 00 00 00 d5 4c 13 f1 .............L.. Feb 28 08:55:51 Storage kernel: 00000060: ff ff ff ff fa 0a 06 c8 00 00 00 00 00 00 00 0d ................ Feb 28 08:55:51 Storage kernel: 00000070: 00 00 00 01 00 18 ff 66 00 00 00 00 00 00 00 00 .......f........ Feb 28 08:55:51 Storage kernel: XFS (md18p1): Metadata corruption detected at xfs_dinode_verify+0xa0/0x732 [xfs], inode 0x15416 dinode Feb 28 08:55:51 Storage kernel: XFS (md18p1): Unmount and run xfs_repair Feb 28 08:55:51 Storage kernel: XFS (md18p1): First 128 bytes of corrupted metadata buffer: Feb 28 08:55:51 Storage kernel: 00000000: 49 4e 81 ff 03 02 00 00 00 00 00 63 00 00 00 64 IN.........c...d Feb 28 08:55:51 Storage kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................ Feb 28 08:55:51 Storage kernel: 00000020: 5c e8 88 37 21 21 f6 d0 5c e8 88 12 1e 16 0a a8 \..7!!..\....... Feb 28 08:55:51 Storage kernel: 00000030: 5c e8 93 98 08 a2 4f 36 00 00 00 00 00 b7 90 e7 \.....O6........ Feb 28 08:55:51 Storage kernel: 00000040: 00 00 00 00 00 00 0b 7a 00 00 00 00 00 00 00 01 .......z........ Feb 28 08:55:51 Storage kernel: 00000050: 00 00 18 01 00 00 00 00 00 00 00 00 d5 4c 13 f1 .............L.. Feb 28 08:55:51 Storage kernel: 00000060: ff ff ff ff fa 0a 06 c8 00 00 00 00 00 00 00 0d ................ Feb 28 08:55:51 Storage kernel: 00000070: 00 00 00 01 00 18 ff 66 00 00 00 00 00 00 00 00 .......f........ storage-diagnostics-20240228-0851.zip

@JorgeB @itimpi The array started fine and Disk 1 is back and operational. Disk 8 though is still marked as disabled and emulated . I assume my next step is to force a rebuild on that disk? storage-diagnostics-20231207-1045.zip

@JorgeB Done - logs are below, what should i do next? disk 1 with -L did this: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... clearing needsrepair flag and regenerating metadata sb_icount 23808, counted 132032 sb_ifree 2334, counted 35228 sb_fdblocks 10484039, counted 33519164 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 9 - agno = 3 - agno = 1 - agno = 6 - agno = 7 - agno = 5 - agno = 8 - agno = 4 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (3:1193189) is ahead of log (1:2). Format log to cycle 6. done Disk 8 did: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... clearing needsrepair flag and regenerating metadata sb_icount 61696, counted 312768 sb_ifree 9362, counted 47709 sb_fdblocks 47064461, counted 92818335 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - agno = 13 - agno = 14 - agno = 15 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 7 - agno = 12 - agno = 5 - agno = 3 - agno = 6 - agno = 8 - agno = 2 - agno = 9 - agno = 11 - agno = 10 - agno = 13 - agno = 14 - agno = 15 - agno = 4 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (7:742200) is ahead of log (1:2). Format log to cycle 10. done

@JorgeB im getting this on both Phase 1 - find and verify superblock... bad primary superblock - bad CRC in superblock !!! attempting to find secondary superblock... .found candidate secondary superblock... verified secondary superblock... writing modified primary superblock Phase 2 - using internal log - zero log... ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this.

Tuesday i started to hear a sound from my server that sounded like a fan going bad then about 5 minutes later i got notification that some drives were hot so i pulled the lid off to check the fan and it was good. Put things back together then went back to the dashboard and was missing 4 drives from one backplane. Shut down, reseated the 4 drives and powered back on, 2 came back and 2 were marked as missing. Thought maybe a backplane went bad so i powered down, swapped a new one in and then the 2 missing drives showed back up but were disabled so i kicked off a rebuild on the first one. Today the rebuild finished but now it says its unmountable, same with the other drive that was missing. What do i do now? Diagnostics are attached storage-diagnostics-20231207-0936.zip

I just added another USB device to passthru into a VM today and now im getting all kinds of issues with USB devices and an error saying No free USB ports. I found another post saying to bump the USB controller to 3.0 (qemu XHCI) and that would fix the issue but it did not. I have 4 devices im trying to pass thru to a Home Assistant VM. Diagnostics are attached Thanks for the assistance! storage-diagnostics-20230707-1527.zip