BTFS errors in syslog

[-Daedalus]

Google-fu not great here. BTRFS and what look like errors from the controller from the drive (Samsung 970 Evo Plus). It's in a 2 drive mirror.

I wonder if someone who's better with BTRFS can have a look, or point me in the right direction.

 

Scrutiny is showing a ton of medium errors, but it's been doing this on both drives, from day 1. I updated firmware at the time, but it didn't seem to have any affect, and the drives have been running fine for well over a year at this point.

 

 

Snippet:

Apr 26 07:03:37 server kernel: BTRFS warning (device nvme1n1p1): i/o error at logical 812252712960 on dev /dev/nvme1n1p1, physical 469695516672, root 5, inode 1072832, offset 4437626880, length 4096, links 1 (path: domains/eco/vdisk1.img)
Apr 26 07:03:37 server kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 21284, flush 0, corrupt 0, gen 0
Apr 26 07:03:37 server kernel: BTRFS warning (device nvme1n1p1): i/o error at logical 812249567232 on dev /dev/nvme1n1p1, physical 469692370944, root 5, inode 1072832, offset 4434481152, length 4096, links 1 (path: domains/eco/vdisk1.img)
Apr 26 07:03:37 server kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 21285, flush 0, corrupt 0, gen 0
Apr 26 07:03:41 server kernel: nvme_log_error: 22 callbacks suppressed
Apr 26 07:03:41 server kernel: nvme1n1: I/O Cmd(0x2) @ LBA 917376120, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR 
Apr 26 07:03:41 server kernel: blk_print_req_error: 22 callbacks suppressed
Apr 26 07:03:41 server kernel: critical medium error, dev nvme1n1, sector 917376120 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
Apr 26 07:03:41 server kernel: scrub_handle_errored_block: 14 callbacks suppressed
Apr 26 07:03:41 server kernel: BTRFS warning (device nvme1n1p1): i/o error at logical 812252721152 on dev /dev/nvme1n1p1, physical 469695524864, root 5, inode 1072832, offset 4437635072, length 4096, links 1 (path: domains/eco/vdisk1.img)
Apr 26 07:03:41 server kernel: nvme1n1: I/O Cmd(0x2) @ LBA 917369976, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR 
Apr 26 07:03:41 server kernel: critical medium error, dev nvme1n1, sector 917369976 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
Apr 26 07:03:41 server kernel: btrfs_dev_stat_print_on_error: 14 callbacks suppressed
Apr 26 07:03:41 server kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 21300, flush 0, corrupt 0, gen 0
Apr 26 07:03:41 server kernel: nvme1n1: I/O Cmd(0x2) @ LBA 917379192, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR 
Apr 26 07:03:41 server kernel: critical medium error, dev nvme1n1, sector 917379192 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
Apr 26 07:03:41 server kernel: BTRFS warning (device nvme1n1p1): i/o error at logical 812249575424 on dev /dev/nvme1n1p1, physical 469692379136, root 5, inode 1072832, offset 4434489344, length 4096, links 1 (path: domains/eco/vdisk1.img)
Apr 26 07:03:41 server kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 21301, flush 0, corrupt 0, gen 0
Apr 26 07:03:41 server kernel: BTRFS warning (device nvme1n1p1): i/o error at logical 812254294016 on dev /dev/nvme1n1p1, physical 469697097728, root 5, inode 1072832, offset 4439207936, length 4096, links 1 (path: domains/eco/vdisk1.img)
Apr 26 07:03:41 server kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 21302, flush 0, corrupt 0, gen 0
Apr 26 07:03:41 server kernel: nvme1n1: I/O Cmd(0x2) @ LBA 917382264, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR 
Apr 26 07:03:41 server kernel: critical medium error, dev nvme1n1, sector 917382264 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
Apr 26 07:03:41 server kernel: nvme1n1: I/O Cmd(0x2) @ LBA 917385336, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR 
Apr 26 07:03:41 server kernel: critical medium error, dev nvme1n1, sector 917385336 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
Apr 26 07:03:41 server kernel: BTRFS warning (device nvme1n1p1): i/o error at logical 812255866880 on dev /dev/nvme1n1p1, physical 469698670592, root 5, inode 1072832, offset 4440780800, length 4096, links 1 (path: domains/eco/vdisk1.img)
Apr 26 07:03:41 server kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 21303, flush 0, corrupt 0, gen 0
Apr 26 07:03:41 server kernel: nvme1n1: I/O Cmd(0x2) @ LBA 917388408, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR 
Apr 26 07:03:41 server kernel: critical medium error, dev nvme1n1, sector 917388408 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
Apr 26 07:03:41 server kernel: nvme1n1: I/O Cmd(0x2) @ LBA 917373048, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR 
Apr 26 07:03:41 server kernel: critical medium error, dev nvme1n1, sector 917373048 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0

 

Diags attached as well.

server-diagnostics-20230506-1116.zip

[JorgeB]

Apr  3 13:51:19 server kernel: BTRFS info (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 21071, flush 0, corrupt 0, gen 0
Apr  3 13:51:19 server kernel: BTRFS info (device nvme1n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 3232, flush 0, corrupt 0, gen 0

 

Both devices are reporting read errors, these look like device problem for both, though one is worse than the other, from SMART:

 

cache1

Media and Data Integrity Errors:    23,329

cache2

Media and Data Integrity Errors:    4,105

 

[-Daedalus]

Thanks for the quick response!

 

They were overheating originally, so I added some heatsinks, though my understanding is that the SSDs with throttle performance to avoid errors just like this, so I'm not quite sure what's going on.

[JorgeB]

Look for a firmware update, IIRC there are knwon issues with some devices from this model that an update can fix, not sure if it's this one though.

[-Daedalus]

Updated. I'll keep an eye on it and report back. Cheers.