[6.9.1] Metadata corruption detected at xfs_dinode_verify

Hello,

I have upgraded to 6.9.1 a few days ago and everything seemed to be working okay so far.

However, I noticed that my 'memory log' was at 100% today.

(This bar was red at 100%)

I rebooted my server and since then I am facing errors regarding one of my array-disks:

Mar 15 11:34:35 Server kernel: XFS (dm-2): Metadata corruption detected at xfs_dinode_verify+0xa3/0x581 [xfs], inode 0x3112d7016 dinode
Mar 15 11:34:35 Server kernel: XFS (dm-2): Unmount and run xfs_repair
Mar 15 11:34:35 Server kernel: XFS (dm-2): First 128 bytes of corrupted metadata buffer:
Mar 15 11:34:35 Server kernel: 00000000: 49 4e 81 80 03 02 00 00 00 00 00 63 00 00 00 64  IN.........c...d
Mar 15 11:34:35 Server kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
Mar 15 11:34:35 Server kernel: 00000020: 5e ab 64 67 25 cd 6b cc 60 4e 9b 47 00 29 29 3e  ^.dg%.k.`N.G.))>
Mar 15 11:34:35 Server kernel: 00000030: 60 4e 9b 47 00 29 29 3e 00 00 00 00 02 cc a0 00  `N.G.))>........
Mar 15 11:34:35 Server kernel: 00000040: 00 00 00 00 00 00 2c ca 00 00 00 00 00 00 00 01  ......,.........
Mar 15 11:34:35 Server kernel: 00000050: 00 00 00 02 00 00 00 00 00 00 00 00 2d 32 ff fb  ............-2..
Mar 15 11:34:35 Server kernel: 00000060: ff ff ff ff aa 5c ad 41 00 00 00 00 00 00 00 5a  .....\.A.......Z
Mar 15 11:34:35 Server kernel: 00000070: 00 00 00 14 00 3e c7 7f 00 00 00 00 00 00 00 00  .....>..........

If I understand it correctly, this means the affected disk is 'disk3'. That is where the appdata and docker files + other shares are currently located at.

I don't know if this is related to 6.9.1 or that memory log problem at all. However, those XFS metadata corruption errors started showing right after the restart which at least solved the memory log at 100% error..

How can I repair those corrupted metadata files properly?

I have not yet restarted the server after those errors came up.

---

I have attached the newest diagnostics and a full /flash/logs/syslog file.

The diagnostics file only shows the current errors related to 'metadata corruption detected...', the other syslog shows:

• Mar 13 - 'BTRFS critical (device loop2): corrupt leaf: ...' error (lines 30029 - 30653)
• Mar 15 - /var/log/syslog spam (lines 30715 - 41391)
• Mar 15 - xfs metadata corruption (lines 44133 - 54109)
• Feb 17, Mar 2, Mar 12 - various kernel panics which seem to be related to my previous problem regarding macvlan(?) kernel panics (lines 3184 - 7121, 16962+, 29687+)

(As this is probably/hopefully not related to my new problem, I'll keep using the other thread for the network-related errors though.)

edit: uploaded missing diagnostics/syslog files

Thanks in advance,

jnk

server-diagnostics-20210315-1826.zip syslog_02-17_03-15.zip