Hello, I've been looking around for the solution I'm still unsure of how to proceed. I got an error from unraid on my disk 1: "ERRORS on disk 1
No description".

 

On investigation, the log gave 10 errors:

Dec 24 22:04:55 Tower kernel: BTRFS warning (device md1): csum failed root 5 ino 13270418 off 2265088 csum 0x006f85b2 expected csum 0x0fba521a mirror 1 Dec 24 22:04:55 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0 Dec 24 22:04:55 Tower kernel: BTRFS warning (device md1): csum failed root 5 ino 13270418 off 2269184 csum 0x8941f998 expected csum 0x56f66b68 mirror 1 Dec 24 22:04:55 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 Dec 24 22:04:55 Tower kernel: BTRFS warning (device md1): csum failed root 5 ino 13270418 off 2273280 csum 0x8941f998 expected csum 0x6f4cb633 mirror 1 Dec 24 22:04:55 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0 Dec 24 22:04:55 Tower kernel: BTRFS warning (device md1): csum failed root 5 ino 13270418 off 2277376 csum 0x8941f998 expected csum 0x611605c0 mirror 1 Dec 24 22:04:55 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0 Dec 24 22:04:55 Tower kernel: BTRFS warning (device md1): csum failed root 5 ino 13270418 off 2281472 csum 0x6b9109a4 expected csum 0xefb5e681 mirror 1 Dec 24 22:04:55 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0 Dec 24 22:04:55 Tower kernel: BTRFS warning (device md1): csum failed root 5 ino 13270418 off 2265088 csum 0x006f85b2 expected csum 0x0fba521a mirror 1 Dec 24 22:04:55 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 6, gen 0 Dec 24 22:04:55 Tower kernel: BTRFS warning (device md1): csum failed root 5 ino 13270418 off 2265088 csum 0x006f85b2 expected csum 0x0fba521a mirror 1 Dec 24 22:04:55 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 Dec 24 22:04:55 Tower kernel: BTRFS warning (device md1): csum failed root 5 ino 13270418 off 2265088 csum 0x006f85b2 expected csum 0x0fba521a mirror 1 Dec 24 22:04:55 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0 Dec 24 22:04:55 Tower kernel: BTRFS warning (device md1): csum failed root 5 ino 13270418 off 2265088 csum 0x006f85b2 expected csum 0x0fba521a mirror 1 Dec 24 22:04:55 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 9, gen 0 Dec 24 22:04:55 Tower kernel: BTRFS warning (device md1): csum failed root 5 ino 13270418 off 2265088 csum 0x006f85b2 expected csum 0x0fba521a mirror 1 Dec 24 22:04:55 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0

 

Disk and SMART reported healthy..

So I found that I could initiate a btrfs scrub with repair from the disk page. Did that and it seems there are 5 errors left, all in the same file.

UUID: 0fd6d4ef-feb6-4ae4-bfe1-4fc322633363 Scrub started: Mon Dec 26 17:16:23 2022 Status: finished Duration: 15:10:33 Total to scrub: 4.87TiB Rate: 93.50MiB/s Error summary: csum=5 Corrected: 0 Uncorrectable: 5 Unverified: 0

 

Dec 26 21:19:01 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0 Dec 26 21:19:01 Tower kernel: BTRFS error (device md1): unable to fixup (regular) error at logical 3898780192768 on dev /dev/md1 Dec 26 21:19:01 Tower kernel: BTRFS warning (device md1): checksum error at logical 3898780196864 on dev /dev/md1, physical 1603128565760, root 5, inode 13270418, offset 2269184, length 4096, links 1 (path: Photos/2017.05 - Taiwan/DSCN6177.JPG)

 

I tough I could repair the corrupted file from a backup, but the restore fails because the file is not accessible (logical if its not fixed I guess).

 

My questions are:

1. How to proceed from here: I see replies on the forum where people mention formatting the whole drive, but could I not repair this file/ sector on the disk alone? This is the first time I've encountered this, do I have to check the RAM for faults?

2. Should I do more to prevent this corruption in the future? I only got the error because of a backup I was running which stopped due to the file not being accessible.

 

Attached diagnostics + syslog

 

 

 

tower-diagnostics-20221227-0959.zip

tower-syslog-20221227-0859.zip

Edited December 27, 20223 yr by Madfox

18 minutes ago, Madfox said:

I tough I could repair the corrupted file from a backup, but the restore fails because the file is not accessible (logical if its not fixed I guess).

Not sure what you mean, you should be able to just delete the file and restore from a backup.

I see, When I did it via restore option in Duplicati, it could not edit the file. Removing it by hand worked!

 

However, there are still errors added to the log. Even from this morning I see now:

Dec 27 09:25:49 Tower kernel: btrfs_print_data_csum_error: 3 callbacks suppressed Dec 27 09:25:49 Tower kernel: BTRFS warning (device md1): csum failed root 5 ino 13270418 off 2265088 csum 0x006f85b2 expected csum 0x0fba521a mirror 1 Dec 27 09:25:49 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 6, gen 0 Dec 27 09:25:49 Tower kernel: BTRFS warning (device md1): csum failed root 5 ino 13270418 off 2269184 csum 0x8941f998 expected csum 0x56f66b68 mirror 1 Dec 27 09:25:49 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 Dec 27 09:25:49 Tower kernel: BTRFS warning (device md1): csum failed root 5 ino 13270418 off 2273280 csum 0x8941f998 expected csum 0x6f4cb633 mirror 1 Dec 27 09:25:49 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0 Dec 27 09:25:49 Tower kernel: BTRFS warning (device md1): csum failed root 5 ino 13270418 off 2277376 csum 0x8941f998 expected csum 0x611605c0 mirror 1 Dec 27 09:25:49 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 9, gen 0 Dec 27 09:25:49 Tower kernel: BTRFS warning (device md1): csum failed root 5 ino 13270418 off 2281472 csum 0x6b9109a4 expected csum 0xefb5e681 mirror 1 Dec 27 09:25:49 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0 Dec 27 09:25:49 Tower kernel: BTRFS warning (device md1): csum failed root 5 ino 13270418 off 2265088 csum 0x006f85b2 expected csum 0x0fba521a mirror 1 Dec 27 09:25:49 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 11, gen 0 Dec 27 09:25:49 Tower kernel: BTRFS warning (device md1): csum failed root 5 ino 13270418 off 2265088 csum 0x006f85b2 expected csum 0x0fba521a mirror 1 Dec 27 09:25:49 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 12, gen 0 Dec 27 09:25:49 Tower kernel: BTRFS warning (device md1): csum failed root 5 ino 13270418 off 2265088 csum 0x006f85b2 expected csum 0x0fba521a mirror 1 Dec 27 09:25:49 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 13, gen 0 Dec 27 09:25:49 Tower kernel: BTRFS warning (device md1): csum failed root 5 ino 13270418 off 2265088 csum 0x006f85b2 expected csum 0x0fba521a mirror 1 Dec 27 09:25:49 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 14, gen 0 Dec 27 09:25:49 Tower kernel: BTRFS warning (device md1): csum failed root 5 ino 13270418 off 2265088 csum 0x006f85b2 expected csum 0x0fba521a mirror 1 Dec 27 09:25:49 Tower kernel: BTRFS error (device md1): bdev /dev/md1 errs: wr 0, rd 0, flush 0, corrupt 15, gen 0

 

Those error should be in the logs attached in the previous post. What to do about this?

Clear the fs stats then run another scrub.

Thank you, I've ran:

btrfs dev stats -z /mnt/disk1

Ouput was:

[/dev/md1].write_io_errs    0
[/dev/md1].read_io_errs     0
[/dev/md1].flush_io_errs    0
[/dev/md1].corruption_errs  18
[/dev/md1].generation_errs  0

Now scrubbing again with repair errors on, will report back after this is done.

 

Thank you @JorgeB. This indeed worked, no more mentions of errors on the disk. Happy to have had the script to even be notified of the corruption in the first place.

 

So I guess if I do not encounter any more errors on this disc, there is no HW to replace?

 

 

3 minutes ago, Madfox said:

So I guess if I do not encounter any more errors on this disc, there is no HW to replace?

Not for now, good idea to run memtest to see if there are any glaring RAM issues, if it happens again in the future and only that disk is affected it could also be a disk problem.