BTRFS corrupt leaf - share guide please

[Freender]

May  5 07:03:52 Tower root: /var/lib/docker: 3.6 GiB (3892211712 bytes) trimmed on /dev/loop2
May  5 07:03:52 Tower root: /mnt/cache: 121 GiB (129878523904 bytes) trimmed on /dev/nvme0n1p1
May  5 07:55:12 Tower kernel: BTRFS critical (device nvme0n1p1): corrupt leaf: root=10 block=2644209385472 slot=296, unexpected item end, have 2140948741 expect 16259
May  5 07:55:12 Tower kernel: BTRFS info (device nvme0n1p1): leaf 2644209385472 gen 635469 total ptrs 590 free space 1493 owner 10
May  5 07:55:12 Tower kernel: 	item 0 key (2493317468160 199 167936) itemoff 16283 itemsize 0
May  5 07:55:12 Tower kernel: 	item 1 key (2493317644288 198 1073741824) itemoff 16275 itemsize 8
May  5 07:55:12 Tower kernel: 	item 2 key (2493330280448 199 4096) itemoff 16275 itemsize 0
May  5 07:55:12 Tower kernel: 	item 3 key (2493330288640 199 4096) itemoff 16275 itemsize 0
May  5 07:55:12 Tower kernel: 	item 4 key (2493330300928 199 4096) itemoff 16275 itemsize 0
May  5 07:55:12 Tower kernel: 	item 5 key (2493330849792 199 114688) itemoff 16275 itemsize 0
May  5 07:55:12 Tower kernel: 	item 6 key (2493339525120 199 24576) itemoff 16275 itemsize 0
May  5 07:55:12 Tower kernel: 	item 7 key (2493339557888 199 49152) itemoff 16275 itemsize 0
May  5 07:55:12 Tower kernel: 	item 8 key (2493339623424 199 40960) itemoff 16275 itemsize 0
...
May  5 07:55:12 Tower kernel: 	item 586 key (2498537791488 199 49152) itemoff 16243 itemsize 0
May  5 07:55:12 Tower kernel: 	item 587 key (2498546405376 199 135168) itemoff 16243 itemsize 0
May  5 07:55:12 Tower kernel: 	item 588 key (2498671570944 199 12288) itemoff 16243 itemsize 0
May  5 07:55:12 Tower kernel: 	item 589 key (2498671603712 199 4096) itemoff 16243 itemsize 0
May  5 07:55:12 Tower kernel: BTRFS error (device nvme0n1p1): block=2644209385472 write time tree block corruption detected
May  5 07:55:12 Tower kernel: BTRFS: error (device nvme0n1p1) in btrfs_commit_transaction:2438: errno=-5 IO failure (Error while writing out transaction)
May  5 07:55:12 Tower kernel: BTRFS info (device nvme0n1p1): forced readonly

 

Hi All,

do we have any guide I can follow for similar type of issues?

Diagnostics attached

tower-diagnostics-20220505-1235.zip

[JorgeB]

17 minutes ago, Freender said:

write time tree block corruption detected

This usually means memory problems or some other kernel memory corruption, start by running memtest, the filesystem should be OK after that's fixed, since it went read-only to prevent corruption, though if it's bad RAM there could be some data corruption, etc.

[Freender]

The RAM was running on not standard timings, reverted to auto and started Memtest. Thanks for your tips

[JorgeB]

Yep, IMHO servers and overclocking never really go well together, and running 2133 and 2400 rated DIMMs @ 3200MT/s is pushing your luck

 

[Freender]

 

Memory test was successful, might also run it again overnight. 

 

Cache pool looks healthy after rebooting

 

Topic can be closed
 

UUID: 2dd181dd-c2fb-4c7f-b9e7-af5b707f89f2

Scrub started: Thu May 5 15:57:19 2022

Status: finished

Duration: 0:03:08

Total to scrub: 580.62GiB

Rate: 3.09GiB/s

Error summary: no errors found