Jump to content

Needing help with log file getting full

Endda

By Endda
in General Support

Go to solution Solved by itimpi,

Recommended Posts

Endda Rookie

Endda

Posted
Posted

For a while over noticed the log file getting full and I'm having difficult figuring out what is filling it up. 

 

I just found a command to expand it to 512MB, but I doubt that will fix the issue (and it needs to be executed after each reboot).

 

So I'm hoping to help analyze the server to figure out what is causing it

Link to comment
  • Solution
itimpi Grand Master

itimpi

Posted
  • Solution
Posted

Your syslog appears to be ful of 

Jul 16 20:46:03 Tower kernel: XFS (nvme0n1p1): Metadata corruption detected at xfs_dinode_verify+0xa0/0x732 [xfs], inode 0x6017cf8d dinode
Jul 16 20:46:03 Tower kernel: XFS (nvme0n1p1): Unmount and run xfs_repair
Jul 16 20:46:03 Tower kernel: XFS (nvme0n1p1): First 128 bytes of corrupted metadata buffer:
Jul 16 20:46:03 Tower kernel: 00000000: 49 4e 41 ed 03 01 00 00 00 00 00 63 00 00 00 64  INA........c...d
Jul 16 20:46:03 Tower kernel: 00000010: 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul 16 20:46:03 Tower kernel: 00000020: 61 71 64 89 0d 84 32 3f 61 71 64 89 0d 84 32 3f  aqd...2?aqd...2?
Jul 16 20:46:03 Tower kernel: 00000030: 63 dd c3 85 1e f8 8b 7c 00 00 00 00 00 00 00 54  c......|.......T
Jul 16 20:46:03 Tower kernel: 00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul 16 20:46:03 Tower kernel: 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 77 39 4a 6b  ............w9Jk
Jul 16 20:46:03 Tower kernel: 00000060: ff ff ff ff 78 8b 49 98 00 00 00 00 00 00 00 04  ....x.I.........
Jul 16 20:46:03 Tower kernel: 00000070: 00 00 06 34 00 0b a9 ed 00 00 00 00 00 00 00 00  ...4............

which I believe is your cache pool.     You would need to run a check filesystem on this to repair the file system. 

Link to comment
Endda Rookie

Endda

Posted
  • Author
Posted
9 hours ago, itimpi said:

Your syslog appears to be ful of 

Jul 16 20:46:03 Tower kernel: XFS (nvme0n1p1): Metadata corruption detected at xfs_dinode_verify+0xa0/0x732 [xfs], inode 0x6017cf8d dinode
Jul 16 20:46:03 Tower kernel: XFS (nvme0n1p1): Unmount and run xfs_repair
Jul 16 20:46:03 Tower kernel: XFS (nvme0n1p1): First 128 bytes of corrupted metadata buffer:
Jul 16 20:46:03 Tower kernel: 00000000: 49 4e 41 ed 03 01 00 00 00 00 00 63 00 00 00 64  INA........c...d
Jul 16 20:46:03 Tower kernel: 00000010: 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul 16 20:46:03 Tower kernel: 00000020: 61 71 64 89 0d 84 32 3f 61 71 64 89 0d 84 32 3f  aqd...2?aqd...2?
Jul 16 20:46:03 Tower kernel: 00000030: 63 dd c3 85 1e f8 8b 7c 00 00 00 00 00 00 00 54  c......|.......T
Jul 16 20:46:03 Tower kernel: 00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul 16 20:46:03 Tower kernel: 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 77 39 4a 6b  ............w9Jk
Jul 16 20:46:03 Tower kernel: 00000060: ff ff ff ff 78 8b 49 98 00 00 00 00 00 00 00 04  ....x.I.........
Jul 16 20:46:03 Tower kernel: 00000070: 00 00 06 34 00 0b a9 ed 00 00 00 00 00 00 00 00  ...4............

which I believe is your cache pool.     You would need to run a check filesystem on this to repair the file system. 

so I did this on the cache drive. and this was the result. . .

 

Quote

Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
bad CRC for inode 19620313
inode identifier 1743622489441370111 mismatch on inode 19620313
bad CRC for inode 19620313, would rewrite
inode identifier 1743622489441370111 mismatch on inode 19620313
would have cleared inode 19620313
        - agno = 1
bad CRC for inode 536904871
bad CRC for inode 536904871, would rewrite
would have cleared inode 536904871
bad CRC for inode 538445791
bad CRC for inode 538445791, would rewrite
would have cleared inode 538445791
        - agno = 2
bad CRC for inode 1078394027
inode identifier 10445839294802755583 mismatch on inode 1078394027
bad CRC for inode 1078394027, would rewrite
inode identifier 10445839294802755583 mismatch on inode 1078394027
would have cleared inode 1078394027
bad CRC for inode 1079984215
inode identifier 12731651323838660607 mismatch on inode 1079984215
bad CRC for inode 1079984215, would rewrite
inode identifier 12731651323838660607 mismatch on inode 1079984215
would have cleared inode 1079984215
bad CRC for inode 1087944725
inode identifier 12721527003590230015 mismatch on inode 1087944725
bad CRC for inode 1087944725, would rewrite
inode identifier 12721527003590230015 mismatch on inode 1087944725
would have cleared inode 1087944725
bad CRC for inode 1088295528
bad CRC for inode 1088295528, would rewrite
would have cleared inode 1088295528
        - agno = 3
bad CRC for inode 1611941443
inode identifier 10382906509075480575 mismatch on inode 1611941443
bad CRC for inode 1611941443, would rewrite
inode identifier 10382906509075480575 mismatch on inode 1611941443
would have cleared inode 1611941443
bad CRC for inode 1612074453
bad CRC for inode 1612074453, would rewrite
would have cleared inode 1612074453
bad CRC for inode 1612173197
bad CRC for inode 1612173197, would rewrite
would have cleared inode 1612173197
bad CRC for inode 1612987978
bad CRC for inode 1612987978, would rewrite
would have cleared inode 1612987978
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 2
        - agno = 3
        - agno = 1
bad CRC for inode 536904871, would rewrite
would have cleared inode 536904871
bad CRC for inode 538445791, would rewrite
would have cleared inode 538445791
bad CRC for inode 1611941443, would rewrite
inode identifier 10382906509075480575 mismatch on inode 1611941443
would have cleared inode 1611941443
bad CRC for inode 1612074453, would rewrite
would have cleared inode 1612074453
bad CRC for inode 1612173197, would rewrite
would have cleared inode 1612173197
entry "7b5eed1456ec3a6c209652a06c8a33d7382ea2b6_exiftool.json" at block 0 offset 96 in directory inode 1078394026 references free inode 1078394027
    would clear inode number in entry at offset 96...
bad CRC for inode 1078394027, would rewrite
inode identifier 10445839294802755583 mismatch on inode 1078394027
would have cleared inode 1078394027
entry "86541d26abf95c98afbc6a19682ac40552b72bff_224x224_left.jpg" at block 4 offset 496 in directory inode 4656745 references free inode 19620313
    would clear inode number in entry at offset 496...
bad CRC for inode 1612987978, would rewrite
would have cleared inode 1612987978
bad CRC for inode 1079984215, would rewrite
inode identifier 12731651323838660607 mismatch on inode 1079984215
would have cleared inode 1079984215
entry "Contents" in shortform directory 541659744 references free inode 1079984215
would have junked entry "Contents" in directory inode 541659744
entry "bb75d06b171a605eb59c80b8ea655a39d12932e.bundle" at block 33 offset 1472 in directory inode 1080164943 references free inode 1611941443
    would clear inode number in entry at offset 1472...
bad CRC for inode 1087944725, would rewrite
inode identifier 12721527003590230015 mismatch on inode 1087944725
would have cleared inode 1087944725
entry "b7884ad1b4f4252a8356c1160fa040cf26db6fe1.jpg" at block 0 offset 3856 in directory inode 1087947980 references free inode 1087944725
    would clear inode number in entry at offset 3856...
bad CRC for inode 1088295528, would rewrite
would have cleared inode 1088295528
bad CRC for inode 19620313, would rewrite
inode identifier 1743622489441370111 mismatch on inode 19620313
would have cleared inode 19620313
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
entry "86541d26abf95c98afbc6a19682ac40552b72bff_224x224_left.jpg" in directory inode 4656745 points to free inode 19620313, would junk entry
bad hash table for directory inode 4656745 (no data entry): would rebuild
would rebuild directory inode 4656745
Metadata corruption detected at 0x46f860, inode 0x200084a7 dinode
couldn't map inode 536904871, err = 117
Metadata corruption detected at 0x46f860, inode 0x201807df dinode
couldn't map inode 538445791, err = 117
entry "Contents" in shortform directory inode 541659744 points to free inode 1079984215
would junk entry
entry "7b5eed1456ec3a6c209652a06c8a33d7382ea2b6_exiftool.json" in directory inode 1078394026 points to free inode 1078394027, would junk entry
bad hash table for directory inode 1078394026 (no data entry): would rebuild
would rebuild directory inode 1078394026
entry "bb75d06b171a605eb59c80b8ea655a39d12932e.bundle" in directory inode 1080164943 points to free inode 1611941443, would junk entry
would rebuild directory inode 1080164943
entry "b7884ad1b4f4252a8356c1160fa040cf26db6fe1.jpg" in directory inode 1087947980 points to free inode 1087944725, would junk entry
would rebuild directory inode 1087947980
Metadata corruption detected at 0x46f860, inode 0x60164dd5 dinode
couldn't map inode 1612074453, err = 117
Metadata corruption detected at 0x46f860, inode 0x6017cf8d dinode
couldn't map inode 1612173197, err = 117
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
disconnected dir inode 1054979, would move to lost+found
disconnected dir inode 1375650, would move to lost+found
disconnected dir inode 1390142, would move to lost+found
disconnected dir inode 1395799, would move to lost+found
disconnected dir inode 1398231, would move to lost+found
disconnected dir inode 1401157, would move to lost+found
disconnected dir inode 5526332, would move to lost+found
disconnected dir inode 538330100, would move to lost+found
disconnected dir inode 538552768, would move to lost+found
disconnected dir inode 538913248, would move to lost+found
disconnected dir inode 1076124556, would move to lost+found
disconnected dir inode 1076149588, would move to lost+found
disconnected dir inode 1076208487, would move to lost+found
disconnected dir inode 1077199060, would move to lost+found
disconnected inode 1079903738, would move to lost+found
disconnected inode 1079984217, would move to lost+found
disconnected inode 1611955154, would move to lost+found
disconnected dir inode 1612074454, would move to lost+found
disconnected dir inode 1612091453, would move to lost+found
disconnected dir inode 1612098287, would move to lost+found
disconnected dir inode 1613860942, would move to lost+found
disconnected dir inode 1613860943, would move to lost+found
Phase 7 - verify link counts...
Metadata corruption detected at 0x46f860, inode 0x200084a7 dinode
couldn't map inode 536904871, err = 117, can't compare link counts
Metadata corruption detected at 0x46f860, inode 0x201807df dinode
couldn't map inode 538445791, err = 117, can't compare link counts
Metadata corruption detected at 0x46f860, inode 0x60164dd5 dinode
couldn't map inode 1612074453, err = 117, can't compare link counts
would have reset inode 541659744 nlinks from 3 to 2
Metadata corruption detected at 0x46f860, inode 0x6017cf8d dinode
couldn't map inode 1612173197, err = 117, can't compare link counts
would have reset inode 1080164943 nlinks from 3756 to 3755
No modify flag set, skipping filesystem flush and exiting.

 

is that all that is needed. . .or do I need to add a modify flag to the command?

Link to comment
Endda Rookie

Endda

Posted
  • Author
Posted

I have since removed the flags in that field (in an attempt to actually repair it) and got this response

 

Quote

Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
bad CRC for inode 19620313
inode identifier 1743622489441370111 mismatch on inode 19620313
bad CRC for inode 19620313, will rewrite
inode identifier 1743622489441370111 mismatch on inode 19620313
cleared inode 19620313
        - agno = 1
bad CRC for inode 536904871
bad CRC for inode 536904871, will rewrite
cleared inode 536904871
bad CRC for inode 538445791
bad CRC for inode 538445791, will rewrite
cleared inode 538445791
        - agno = 2
bad CRC for inode 1078394027
inode identifier 10445839294802755583 mismatch on inode 1078394027
bad CRC for inode 1078394027, will rewrite
inode identifier 10445839294802755583 mismatch on inode 1078394027
cleared inode 1078394027
bad CRC for inode 1079984215
inode identifier 12731651323838660607 mismatch on inode 1079984215
bad CRC for inode 1079984215, will rewrite
inode identifier 12731651323838660607 mismatch on inode 1079984215
cleared inode 1079984215
bad CRC for inode 1087944725
inode identifier 12721527003590230015 mismatch on inode 1087944725
bad CRC for inode 1087944725, will rewrite
inode identifier 12721527003590230015 mismatch on inode 1087944725
cleared inode 1087944725
bad CRC for inode 1088295528
bad CRC for inode 1088295528, will rewrite
cleared inode 1088295528
        - agno = 3
bad CRC for inode 1611941443
inode identifier 10382906509075480575 mismatch on inode 1611941443
bad CRC for inode 1611941443, will rewrite
inode identifier 10382906509075480575 mismatch on inode 1611941443
cleared inode 1611941443
bad CRC for inode 1612074453
bad CRC for inode 1612074453, will rewrite
cleared inode 1612074453
bad CRC for inode 1612173197
bad CRC for inode 1612173197, will rewrite
cleared inode 1612173197
bad CRC for inode 1612987978
bad CRC for inode 1612987978, will rewrite
cleared inode 1612987978
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 3
        - agno = 2
        - agno = 1
entry "7b5eed1456ec3a6c209652a06c8a33d7382ea2b6_exiftool.json" at block 0 offset 96 in directory inode 1078394026 references free inode 1078394027
    clearing inode number in entry at offset 96...
entry "86541d26abf95c98afbc6a19682ac40552b72bff_224x224_left.jpg" at block 4 offset 496 in directory inode 4656745 references free inode 19620313
    clearing inode number in entry at offset 496...
entry "Contents" in shortform directory 541659744 references free inode 1079984215
junking entry "Contents" in directory inode 541659744
entry "bb75d06b171a605eb59c80b8ea655a39d12932e.bundle" at block 33 offset 1472 in directory inode 1080164943 references free inode 1611941443
    clearing inode number in entry at offset 1472...
entry "b7884ad1b4f4252a8356c1160fa040cf26db6fe1.jpg" at block 0 offset 3856 in directory inode 1087947980 references free inode 1087944725
    clearing inode number in entry at offset 3856...
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
bad hash table for directory inode 4656745 (no data entry): rebuilding
rebuilding directory inode 4656745
bad hash table for directory inode 1078394026 (no data entry): rebuilding
rebuilding directory inode 1078394026
rebuilding directory inode 1080164943
rebuilding directory inode 1087947980
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
disconnected dir inode 1054979, moving to lost+found
disconnected dir inode 1401157, moving to lost+found
disconnected dir inode 5526332, moving to lost+found
disconnected inode 1079903738, moving to lost+found
disconnected inode 1079984217, moving to lost+found
disconnected dir inode 1613860942, moving to lost+found
disconnected dir inode 1613860943, moving to lost+found
Phase 7 - verify and correct link counts...
resetting inode 63909 nlinks from 2 to 7
resetting inode 541659744 nlinks from 3 to 2
resetting inode 1080164943 nlinks from 3756 to 3755
done

 

so that seems to be what was needed?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing
×
×
  • Create New...