hermy65 Posted February 28 Share Posted February 28 Popped open my syslog this morning and its full of this but it doesnt list what disk its detecting the corruption on so i have no idea what to do. None of the disks show errors on the dashboard either. Diagnostics are attached. Feb 28 08:55:51 Storage kernel: XFS (md18p1): Metadata corruption detected at xfs_dinode_verify+0xa0/0x732 [xfs], inode 0x15416 dinode Feb 28 08:55:51 Storage kernel: XFS (md18p1): Unmount and run xfs_repair Feb 28 08:55:51 Storage kernel: XFS (md18p1): First 128 bytes of corrupted metadata buffer: Feb 28 08:55:51 Storage kernel: 00000000: 49 4e 81 ff 03 02 00 00 00 00 00 63 00 00 00 64 IN.........c...d Feb 28 08:55:51 Storage kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................ Feb 28 08:55:51 Storage kernel: 00000020: 5c e8 88 37 21 21 f6 d0 5c e8 88 12 1e 16 0a a8 \..7!!..\....... Feb 28 08:55:51 Storage kernel: 00000030: 5c e8 93 98 08 a2 4f 36 00 00 00 00 00 b7 90 e7 \.....O6........ Feb 28 08:55:51 Storage kernel: 00000040: 00 00 00 00 00 00 0b 7a 00 00 00 00 00 00 00 01 .......z........ Feb 28 08:55:51 Storage kernel: 00000050: 00 00 18 01 00 00 00 00 00 00 00 00 d5 4c 13 f1 .............L.. Feb 28 08:55:51 Storage kernel: 00000060: ff ff ff ff fa 0a 06 c8 00 00 00 00 00 00 00 0d ................ Feb 28 08:55:51 Storage kernel: 00000070: 00 00 00 01 00 18 ff 66 00 00 00 00 00 00 00 00 .......f........ Feb 28 08:55:51 Storage kernel: XFS (md18p1): Metadata corruption detected at xfs_dinode_verify+0xa0/0x732 [xfs], inode 0x15416 dinode Feb 28 08:55:51 Storage kernel: XFS (md18p1): Unmount and run xfs_repair Feb 28 08:55:51 Storage kernel: XFS (md18p1): First 128 bytes of corrupted metadata buffer: Feb 28 08:55:51 Storage kernel: 00000000: 49 4e 81 ff 03 02 00 00 00 00 00 63 00 00 00 64 IN.........c...d Feb 28 08:55:51 Storage kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................ Feb 28 08:55:51 Storage kernel: 00000020: 5c e8 88 37 21 21 f6 d0 5c e8 88 12 1e 16 0a a8 \..7!!..\....... Feb 28 08:55:51 Storage kernel: 00000030: 5c e8 93 98 08 a2 4f 36 00 00 00 00 00 b7 90 e7 \.....O6........ Feb 28 08:55:51 Storage kernel: 00000040: 00 00 00 00 00 00 0b 7a 00 00 00 00 00 00 00 01 .......z........ Feb 28 08:55:51 Storage kernel: 00000050: 00 00 18 01 00 00 00 00 00 00 00 00 d5 4c 13 f1 .............L.. Feb 28 08:55:51 Storage kernel: 00000060: ff ff ff ff fa 0a 06 c8 00 00 00 00 00 00 00 0d ................ Feb 28 08:55:51 Storage kernel: 00000070: 00 00 00 01 00 18 ff 66 00 00 00 00 00 00 00 00 .......f........ storage-diagnostics-20240228-0851.zip Quote Link to comment
trurl Posted February 28 Share Posted February 28 Check filesystem on disk18, use the webUI not the command line, post the output. Quote Link to comment
trurl Posted February 28 Share Posted February 28 Notes for future discussion, looks like disks 12 and 16 already have lost+found from previous repair. Also, system share has files on disk17. Quote Link to comment
hermy65 Posted February 28 Author Share Posted February 28 @trurl here is the output: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 bad CRC for inode 87062 inode identifier 16149622513293918207 mismatch on inode 87062 bad CRC for inode 87062, would rewrite inode identifier 16149622513293918207 mismatch on inode 87062 would have cleared inode 87062 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 4 - agno = 2 - agno = 3 - agno = 5 - agno = 6 - agno = 7 - agno = 1 entry "11 - The Amity Affliction - Stairway to Hell.mp3" at block 0 offset 736 in directory inode 87051 references free inode 87062 would clear inode number in entry at offset 736... bad CRC for inode 87062, would rewrite inode identifier 16149622513293918207 mismatch on inode 87062 would have cleared inode 87062 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... entry "11 - The Amity Affliction - Stairway to Hell.mp3" in directory inode 87051 points to free inode 87062, would junk entry bad hash table for directory inode 87051 (no data entry): would rebuild would rebuild directory inode 87051 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting. Quote Link to comment
trurl Posted February 28 Share Posted February 28 Do it again without -n. If it asks for it, use -L 12 minutes ago, trurl said: Check filesystem on disk18, use the webUI not the command line, post the output. Quote Link to comment
hermy65 Posted February 28 Author Share Posted February 28 @trurl without -n, i see no mention of -L in this output. Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 bad CRC for inode 87062 inode identifier 16149622513293918207 mismatch on inode 87062 bad CRC for inode 87062, will rewrite inode identifier 16149622513293918207 mismatch on inode 87062 cleared inode 87062 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 1 - agno = 3 - agno = 4 - agno = 7 - agno = 6 - agno = 5 entry "11 - The Amity Affliction - Stairway to Hell.mp3" at block 0 offset 736 in directory inode 87051 references free inode 87062 clearing inode number in entry at offset 736... Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... bad hash table for directory inode 87051 (no data entry): rebuilding rebuilding directory inode 87051 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... done Quote Link to comment
trurl Posted February 28 Share Posted February 28 Start the array in normal (not maintenance) mode and post new diagnostics. Quote Link to comment
hermy65 Posted February 28 Author Share Posted February 28 @trurl updated diagnostics are attached! storage-diagnostics-20240228-0945.zip Quote Link to comment
trurl Posted February 28 Share Posted February 28 Looks good, nothing from disk18 in lost+found. Looks like you need to start planning for increased capacity. 44 minutes ago, trurl said: disks 12 and 16 already have lost+found Have you examined this share? 44 minutes ago, trurl said: system share has files on disk17. Ideally, appdata, domains, and system shares, would have all files on fast pool such as cache, with nothing on the array, so Dockers/VMs will perform better, and so array disks can spin down since these files are always open Nothing can move open files, so you have to disable Docker and VM Manager in Settings before these can be moved. And mover won't replace files so if there are duplicates you will have to decide which to keep. Quote Link to comment
hermy65 Posted February 28 Author Share Posted February 28 @trurl appreciate the help! The lost+found folders have been taken care of, appreciate the alert on that. As for the system folder, it looks like it has my libvrt.img in it for some reason, i assume i can move that to cache and then update the location in VM settings or will that cause an issue? Quote Link to comment
trurl Posted February 28 Share Posted February 28 VM Manager already has IMAGE_FILE="/mnt/user/system/libvirt/libvirt.img" which includes cache Quote Link to comment
hermy65 Posted February 28 Author Share Posted February 28 @trurl Interesting, checked the system/libvrt folder on cache and its empty, the one on disk17 has the .img in it and it was last modified after i started the array when you told me to. Is there a reason that .img file would be on disk17 and being used when the setting is as you posted above? Quote Link to comment
trurl Posted February 28 Share Posted February 28 The setting I posted above just says it is in the system user share, which includes all pools and disks. If it already exists on another disk, it will use that one. Probably you had VM Manager enabled when you were missing cache for some reason, so it got created on the array. Quote Link to comment
hermy65 Posted February 28 Author Share Posted February 28 (edited) @trurl ok so if i move it to cache and update the path i should be good to go, correct? Thanks again! Edited February 28 by hermy65 Quote Link to comment
trurl Posted February 28 Share Posted February 28 No need to update path. That path already includes cache as I mentioned. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.