Endda Posted July 16 Share Posted July 16 For a while over noticed the log file getting full and I'm having difficult figuring out what is filling it up. I just found a command to expand it to 512MB, but I doubt that will fix the issue (and it needs to be executed after each reboot). So I'm hoping to help analyze the server to figure out what is causing it Quote Link to comment
itimpi Posted July 16 Share Posted July 16 You should provide at least a copy of the log file (probably easiest to do it as part of diagnostics) to get anything than random guesses. 1 Quote Link to comment
Endda Posted July 17 Author Share Posted July 17 3 hours ago, itimpi said: You should provide at least a copy of the log file (probably easiest to do it as part of diagnostics) to get anything than random guesses. thanks for the suggestion. here is what the webui gave me tower-diagnostics-20240716-2042.zip Quote Link to comment
Solution itimpi Posted July 17 Solution Share Posted July 17 Your syslog appears to be ful of Jul 16 20:46:03 Tower kernel: XFS (nvme0n1p1): Metadata corruption detected at xfs_dinode_verify+0xa0/0x732 [xfs], inode 0x6017cf8d dinode Jul 16 20:46:03 Tower kernel: XFS (nvme0n1p1): Unmount and run xfs_repair Jul 16 20:46:03 Tower kernel: XFS (nvme0n1p1): First 128 bytes of corrupted metadata buffer: Jul 16 20:46:03 Tower kernel: 00000000: 49 4e 41 ed 03 01 00 00 00 00 00 63 00 00 00 64 INA........c...d Jul 16 20:46:03 Tower kernel: 00000010: 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 ................ Jul 16 20:46:03 Tower kernel: 00000020: 61 71 64 89 0d 84 32 3f 61 71 64 89 0d 84 32 3f aqd...2?aqd...2? Jul 16 20:46:03 Tower kernel: 00000030: 63 dd c3 85 1e f8 8b 7c 00 00 00 00 00 00 00 54 c......|.......T Jul 16 20:46:03 Tower kernel: 00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ Jul 16 20:46:03 Tower kernel: 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 77 39 4a 6b ............w9Jk Jul 16 20:46:03 Tower kernel: 00000060: ff ff ff ff 78 8b 49 98 00 00 00 00 00 00 00 04 ....x.I......... Jul 16 20:46:03 Tower kernel: 00000070: 00 00 06 34 00 0b a9 ed 00 00 00 00 00 00 00 00 ...4............ which I believe is your cache pool. You would need to run a check filesystem on this to repair the file system. Quote Link to comment
Endda Posted July 17 Author Share Posted July 17 9 hours ago, itimpi said: Your syslog appears to be ful of Jul 16 20:46:03 Tower kernel: XFS (nvme0n1p1): Metadata corruption detected at xfs_dinode_verify+0xa0/0x732 [xfs], inode 0x6017cf8d dinode Jul 16 20:46:03 Tower kernel: XFS (nvme0n1p1): Unmount and run xfs_repair Jul 16 20:46:03 Tower kernel: XFS (nvme0n1p1): First 128 bytes of corrupted metadata buffer: Jul 16 20:46:03 Tower kernel: 00000000: 49 4e 41 ed 03 01 00 00 00 00 00 63 00 00 00 64 INA........c...d Jul 16 20:46:03 Tower kernel: 00000010: 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 ................ Jul 16 20:46:03 Tower kernel: 00000020: 61 71 64 89 0d 84 32 3f 61 71 64 89 0d 84 32 3f aqd...2?aqd...2? Jul 16 20:46:03 Tower kernel: 00000030: 63 dd c3 85 1e f8 8b 7c 00 00 00 00 00 00 00 54 c......|.......T Jul 16 20:46:03 Tower kernel: 00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ Jul 16 20:46:03 Tower kernel: 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 77 39 4a 6b ............w9Jk Jul 16 20:46:03 Tower kernel: 00000060: ff ff ff ff 78 8b 49 98 00 00 00 00 00 00 00 04 ....x.I......... Jul 16 20:46:03 Tower kernel: 00000070: 00 00 06 34 00 0b a9 ed 00 00 00 00 00 00 00 00 ...4............ which I believe is your cache pool. You would need to run a check filesystem on this to repair the file system. so I did this on the cache drive. and this was the result. . . Quote Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 bad CRC for inode 19620313 inode identifier 1743622489441370111 mismatch on inode 19620313 bad CRC for inode 19620313, would rewrite inode identifier 1743622489441370111 mismatch on inode 19620313 would have cleared inode 19620313 - agno = 1 bad CRC for inode 536904871 bad CRC for inode 536904871, would rewrite would have cleared inode 536904871 bad CRC for inode 538445791 bad CRC for inode 538445791, would rewrite would have cleared inode 538445791 - agno = 2 bad CRC for inode 1078394027 inode identifier 10445839294802755583 mismatch on inode 1078394027 bad CRC for inode 1078394027, would rewrite inode identifier 10445839294802755583 mismatch on inode 1078394027 would have cleared inode 1078394027 bad CRC for inode 1079984215 inode identifier 12731651323838660607 mismatch on inode 1079984215 bad CRC for inode 1079984215, would rewrite inode identifier 12731651323838660607 mismatch on inode 1079984215 would have cleared inode 1079984215 bad CRC for inode 1087944725 inode identifier 12721527003590230015 mismatch on inode 1087944725 bad CRC for inode 1087944725, would rewrite inode identifier 12721527003590230015 mismatch on inode 1087944725 would have cleared inode 1087944725 bad CRC for inode 1088295528 bad CRC for inode 1088295528, would rewrite would have cleared inode 1088295528 - agno = 3 bad CRC for inode 1611941443 inode identifier 10382906509075480575 mismatch on inode 1611941443 bad CRC for inode 1611941443, would rewrite inode identifier 10382906509075480575 mismatch on inode 1611941443 would have cleared inode 1611941443 bad CRC for inode 1612074453 bad CRC for inode 1612074453, would rewrite would have cleared inode 1612074453 bad CRC for inode 1612173197 bad CRC for inode 1612173197, would rewrite would have cleared inode 1612173197 bad CRC for inode 1612987978 bad CRC for inode 1612987978, would rewrite would have cleared inode 1612987978 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 3 - agno = 1 bad CRC for inode 536904871, would rewrite would have cleared inode 536904871 bad CRC for inode 538445791, would rewrite would have cleared inode 538445791 bad CRC for inode 1611941443, would rewrite inode identifier 10382906509075480575 mismatch on inode 1611941443 would have cleared inode 1611941443 bad CRC for inode 1612074453, would rewrite would have cleared inode 1612074453 bad CRC for inode 1612173197, would rewrite would have cleared inode 1612173197 entry "7b5eed1456ec3a6c209652a06c8a33d7382ea2b6_exiftool.json" at block 0 offset 96 in directory inode 1078394026 references free inode 1078394027 would clear inode number in entry at offset 96... bad CRC for inode 1078394027, would rewrite inode identifier 10445839294802755583 mismatch on inode 1078394027 would have cleared inode 1078394027 entry "86541d26abf95c98afbc6a19682ac40552b72bff_224x224_left.jpg" at block 4 offset 496 in directory inode 4656745 references free inode 19620313 would clear inode number in entry at offset 496... bad CRC for inode 1612987978, would rewrite would have cleared inode 1612987978 bad CRC for inode 1079984215, would rewrite inode identifier 12731651323838660607 mismatch on inode 1079984215 would have cleared inode 1079984215 entry "Contents" in shortform directory 541659744 references free inode 1079984215 would have junked entry "Contents" in directory inode 541659744 entry "bb75d06b171a605eb59c80b8ea655a39d12932e.bundle" at block 33 offset 1472 in directory inode 1080164943 references free inode 1611941443 would clear inode number in entry at offset 1472... bad CRC for inode 1087944725, would rewrite inode identifier 12721527003590230015 mismatch on inode 1087944725 would have cleared inode 1087944725 entry "b7884ad1b4f4252a8356c1160fa040cf26db6fe1.jpg" at block 0 offset 3856 in directory inode 1087947980 references free inode 1087944725 would clear inode number in entry at offset 3856... bad CRC for inode 1088295528, would rewrite would have cleared inode 1088295528 bad CRC for inode 19620313, would rewrite inode identifier 1743622489441370111 mismatch on inode 19620313 would have cleared inode 19620313 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... entry "86541d26abf95c98afbc6a19682ac40552b72bff_224x224_left.jpg" in directory inode 4656745 points to free inode 19620313, would junk entry bad hash table for directory inode 4656745 (no data entry): would rebuild would rebuild directory inode 4656745 Metadata corruption detected at 0x46f860, inode 0x200084a7 dinode couldn't map inode 536904871, err = 117 Metadata corruption detected at 0x46f860, inode 0x201807df dinode couldn't map inode 538445791, err = 117 entry "Contents" in shortform directory inode 541659744 points to free inode 1079984215 would junk entry entry "7b5eed1456ec3a6c209652a06c8a33d7382ea2b6_exiftool.json" in directory inode 1078394026 points to free inode 1078394027, would junk entry bad hash table for directory inode 1078394026 (no data entry): would rebuild would rebuild directory inode 1078394026 entry "bb75d06b171a605eb59c80b8ea655a39d12932e.bundle" in directory inode 1080164943 points to free inode 1611941443, would junk entry would rebuild directory inode 1080164943 entry "b7884ad1b4f4252a8356c1160fa040cf26db6fe1.jpg" in directory inode 1087947980 points to free inode 1087944725, would junk entry would rebuild directory inode 1087947980 Metadata corruption detected at 0x46f860, inode 0x60164dd5 dinode couldn't map inode 1612074453, err = 117 Metadata corruption detected at 0x46f860, inode 0x6017cf8d dinode couldn't map inode 1612173197, err = 117 - traversal finished ... - moving disconnected inodes to lost+found ... disconnected dir inode 1054979, would move to lost+found disconnected dir inode 1375650, would move to lost+found disconnected dir inode 1390142, would move to lost+found disconnected dir inode 1395799, would move to lost+found disconnected dir inode 1398231, would move to lost+found disconnected dir inode 1401157, would move to lost+found disconnected dir inode 5526332, would move to lost+found disconnected dir inode 538330100, would move to lost+found disconnected dir inode 538552768, would move to lost+found disconnected dir inode 538913248, would move to lost+found disconnected dir inode 1076124556, would move to lost+found disconnected dir inode 1076149588, would move to lost+found disconnected dir inode 1076208487, would move to lost+found disconnected dir inode 1077199060, would move to lost+found disconnected inode 1079903738, would move to lost+found disconnected inode 1079984217, would move to lost+found disconnected inode 1611955154, would move to lost+found disconnected dir inode 1612074454, would move to lost+found disconnected dir inode 1612091453, would move to lost+found disconnected dir inode 1612098287, would move to lost+found disconnected dir inode 1613860942, would move to lost+found disconnected dir inode 1613860943, would move to lost+found Phase 7 - verify link counts... Metadata corruption detected at 0x46f860, inode 0x200084a7 dinode couldn't map inode 536904871, err = 117, can't compare link counts Metadata corruption detected at 0x46f860, inode 0x201807df dinode couldn't map inode 538445791, err = 117, can't compare link counts Metadata corruption detected at 0x46f860, inode 0x60164dd5 dinode couldn't map inode 1612074453, err = 117, can't compare link counts would have reset inode 541659744 nlinks from 3 to 2 Metadata corruption detected at 0x46f860, inode 0x6017cf8d dinode couldn't map inode 1612173197, err = 117, can't compare link counts would have reset inode 1080164943 nlinks from 3756 to 3755 No modify flag set, skipping filesystem flush and exiting. is that all that is needed. . .or do I need to add a modify flag to the command? Quote Link to comment
Endda Posted July 17 Author Share Posted July 17 I have since removed the flags in that field (in an attempt to actually repair it) and got this response Quote Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 bad CRC for inode 19620313 inode identifier 1743622489441370111 mismatch on inode 19620313 bad CRC for inode 19620313, will rewrite inode identifier 1743622489441370111 mismatch on inode 19620313 cleared inode 19620313 - agno = 1 bad CRC for inode 536904871 bad CRC for inode 536904871, will rewrite cleared inode 536904871 bad CRC for inode 538445791 bad CRC for inode 538445791, will rewrite cleared inode 538445791 - agno = 2 bad CRC for inode 1078394027 inode identifier 10445839294802755583 mismatch on inode 1078394027 bad CRC for inode 1078394027, will rewrite inode identifier 10445839294802755583 mismatch on inode 1078394027 cleared inode 1078394027 bad CRC for inode 1079984215 inode identifier 12731651323838660607 mismatch on inode 1079984215 bad CRC for inode 1079984215, will rewrite inode identifier 12731651323838660607 mismatch on inode 1079984215 cleared inode 1079984215 bad CRC for inode 1087944725 inode identifier 12721527003590230015 mismatch on inode 1087944725 bad CRC for inode 1087944725, will rewrite inode identifier 12721527003590230015 mismatch on inode 1087944725 cleared inode 1087944725 bad CRC for inode 1088295528 bad CRC for inode 1088295528, will rewrite cleared inode 1088295528 - agno = 3 bad CRC for inode 1611941443 inode identifier 10382906509075480575 mismatch on inode 1611941443 bad CRC for inode 1611941443, will rewrite inode identifier 10382906509075480575 mismatch on inode 1611941443 cleared inode 1611941443 bad CRC for inode 1612074453 bad CRC for inode 1612074453, will rewrite cleared inode 1612074453 bad CRC for inode 1612173197 bad CRC for inode 1612173197, will rewrite cleared inode 1612173197 bad CRC for inode 1612987978 bad CRC for inode 1612987978, will rewrite cleared inode 1612987978 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 3 - agno = 2 - agno = 1 entry "7b5eed1456ec3a6c209652a06c8a33d7382ea2b6_exiftool.json" at block 0 offset 96 in directory inode 1078394026 references free inode 1078394027 clearing inode number in entry at offset 96... entry "86541d26abf95c98afbc6a19682ac40552b72bff_224x224_left.jpg" at block 4 offset 496 in directory inode 4656745 references free inode 19620313 clearing inode number in entry at offset 496... entry "Contents" in shortform directory 541659744 references free inode 1079984215 junking entry "Contents" in directory inode 541659744 entry "bb75d06b171a605eb59c80b8ea655a39d12932e.bundle" at block 33 offset 1472 in directory inode 1080164943 references free inode 1611941443 clearing inode number in entry at offset 1472... entry "b7884ad1b4f4252a8356c1160fa040cf26db6fe1.jpg" at block 0 offset 3856 in directory inode 1087947980 references free inode 1087944725 clearing inode number in entry at offset 3856... Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... bad hash table for directory inode 4656745 (no data entry): rebuilding rebuilding directory inode 4656745 bad hash table for directory inode 1078394026 (no data entry): rebuilding rebuilding directory inode 1078394026 rebuilding directory inode 1080164943 rebuilding directory inode 1087947980 - traversal finished ... - moving disconnected inodes to lost+found ... disconnected dir inode 1054979, moving to lost+found disconnected dir inode 1401157, moving to lost+found disconnected dir inode 5526332, moving to lost+found disconnected inode 1079903738, moving to lost+found disconnected inode 1079984217, moving to lost+found disconnected dir inode 1613860942, moving to lost+found disconnected dir inode 1613860943, moving to lost+found Phase 7 - verify and correct link counts... resetting inode 63909 nlinks from 2 to 7 resetting inode 541659744 nlinks from 3 to 2 resetting inode 1080164943 nlinks from 3756 to 3755 done so that seems to be what was needed? Quote Link to comment
Endda Posted July 17 Author Share Posted July 17 9 hours ago, JorgeB said: Yes, should be fixed now. Thanks so much! Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.