b34r

hm, it wouldn't let me run xfs_repair on md1p1. It's thoroughly possible (heck, it's downright likely) I did something wrong there, though. At any rate, it's been stable since I did have to do a parity rebuild, but that's a small price to pay compared to unraid randomly crashing every 2-3 days. That said, If these weren't near-new drives and I didn't have backups of everything, my math would have changed.

Ok, for the curious, here's the output from repairing sdb1: xfs_repair sdb1 Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 bad CRC for inode 21474861914 inode identifier 15589994527963283455 mismatch on inode 21474861914 bad CRC for inode 21474861914, will rewrite inode identifier 15589994527963283455 mismatch on inode 21474861914 cleared inode 21474861914 - agno = 11 - agno = 12 - agno = 13 - agno = 14 - agno = 15 - agno = 16 bad CRC for inode 34359738500 bad CRC for inode 34359738500, will rewrite cleared inode 34359738500 - agno = 17 - agno = 18 - agno = 19 - agno = 20 - agno = 21 - agno = 22 - agno = 23 - agno = 24 - agno = 25 - agno = 26 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 3 - agno = 10 - agno = 4 - agno = 13 - agno = 14 - agno = 5 - agno = 12 - agno = 2 - agno = 1 - agno = 6 - agno = 9 - agno = 7 - agno = 8 - agno = 15 - agno = 11 entry "310618-video.mp4" in shortform directory 21474861913 references free inode 21474861914 junking entry "310618-video.mp4" in directory inode 21474861913 corrected i8 count in directory 21474861913, was 4, now 3 - agno = 16 - agno = 19 - agno = 18 - agno = 17 - agno = 20 - agno = 21 - agno = 22 - agno = 23 - agno = 24 - agno = 25 - agno = 26 clearing reflink flag on inodes when possible Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... done Looks like if found and fixed some CRC errors as well as an errant mp4. I'm hoping this will do the trick. Going to start the array back up. Will report back if I see this again, or by EOW if it seems I'm in the clear!

Found this: Checking and fixing file systems in another post and learned to run xfs_repair on partitions instead of disks. Running xfs_repair against sdb1 looks like it did, indeed, run some repairs. Going to continue on the other 4 data drives.

Hi All, My Info logs are getting spammed with this message: Nov 26 06:48:16 Dissident kernel: XFS (md1p1): Metadata corruption detected at xfs_dinode_verify+0xa0/0x732 [xfs], inode 0x50000635a dinode Nov 26 06:48:16 Dissident kernel: XFS (md1p1): Unmount and run xfs_repair Nov 26 06:48:16 Dissident kernel: XFS (md1p1): First 128 bytes of corrupted metadata buffer: Nov 26 06:48:16 Dissident kernel: 00000000: 49 4e 81 b4 03 02 00 00 00 00 00 21 00 00 00 21 IN.........!...! Nov 26 06:48:16 Dissident kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................ Nov 26 06:48:16 Dissident kernel: 00000020: 65 53 40 dc 12 f9 5f c2 65 53 40 e6 03 98 d7 2f eS@..._.eS@..../ Nov 26 06:48:16 Dissident kernel: 00000030: 65 53 40 e6 03 98 d7 2f 00 00 00 00 00 06 92 e1 eS@..../........ Nov 26 06:48:16 Dissident kernel: 00000040: 00 00 00 00 00 00 00 6a 00 00 00 00 00 00 00 01 .......j........ Nov 26 06:48:16 Dissident kernel: 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 77 ba 92 3a ............w..: Nov 26 06:48:16 Dissident kernel: 00000060: ff ff ff ff d0 d9 3e 0a 00 00 00 00 00 00 00 0a ......>......... Nov 26 06:48:16 Dissident kernel: 00000070: 00 00 00 53 00 0f e4 5f 00 00 00 00 00 00 00 00 ...S..._........ It usually takes between 2 and 4 days before I start seeing this, but once it happens it absolutely floods the logs - outputting this same block every few seconds. How do I figure out which drive this is? md1p1 disappears from /dev once I spin down the array. I do have this from the logs: Nov 27 15:21:26 Dissident kernel: mdcmd (1): import 0 sdf 64 19531825100 0 ST20000NM007D-3DJ103_ZVT5LH8Y Nov 27 15:21:26 Dissident kernel: md: import disk0: (sdf) ST20000NM007D-3DJ103_ZVT5LH8Y size: 19531825100 Nov 27 15:21:26 Dissident kernel: mdcmd (2): import 1 sdb 64 19531825100 0 ST20000NM007D-3DJ103_ZVT5KP6S Nov 27 15:21:26 Dissident kernel: md: import disk1: (sdb) ST20000NM007D-3DJ103_ZVT5KP6S size: 19531825100 Nov 27 15:21:26 Dissident kernel: mdcmd (3): import 2 sde 64 19531825100 0 ST20000NM007D-3DJ103_ZVT5JDG8 Nov 27 15:21:26 Dissident kernel: md: import disk2: (sde) ST20000NM007D-3DJ103_ZVT5JDG8 size: 19531825100 Nov 27 15:21:26 Dissident kernel: mdcmd (4): import 3 sdc 64 19531825100 0 ST20000NM007D-3DJ103_ZVT5HH7D Nov 27 15:21:26 Dissident kernel: md: import disk3: (sdc) ST20000NM007D-3DJ103_ZVT5HH7D size: 19531825100 Nov 27 15:21:26 Dissident kernel: mdcmd (5): import 4 sdd 64 19531825100 0 ST20000NM007D-3DJ103_ZVT5MSH2 Nov 27 15:21:26 Dissident kernel: md: import disk4: (sdd) ST20000NM007D-3DJ103_ZVT5MSH2 size: 19531825100 which has me thinking it should be 'sdf' but that's a parity disk, which doesn't seem right to me (though, I have no basis for that) I started running xfs_repair on sdb, but I have no idea if that's the right drive or how long I should expect that to take on 20TB. when I start it up, it reports this: Phase 1 - find and verify superblock... bad primary superblock - bad magic number !!! attempting to find secondary superblock... and then just prints '.......'s forever. UPDATE: If you're in my boat and seeing endless '...'s, you're likely running xfs_repair on a non-xfs partition (or on a drive instead of a partition). Kill it and read this: Checking and fixing file systems So, I guess I have 3 questions 1. how do I figure out what maps to md1p1? 2. do I need to wait for the issue to occur again before xfs_repair can find & fix it? (and is xfs_repair even the right tool?) 3. did I miss anything else obvious? I've attached diagnostics, but these are post-reboot, so I'm not sure if that'll be helpful. Also checked the logs, but there's nothing for over 2 hours before this message starts spamming, so it seems to me like that won't help much either. Thanks in advance for any help! dissident-diagnostics-20231127-1544.zip