Google-fu not great here. BTRFS and what look like errors from the controller from the drive (Samsung 970 Evo Plus). It's in a 2 drive mirror.

I wonder if someone who's better with BTRFS can have a look, or point me in the right direction.

 

Scrutiny is showing a ton of medium errors, but it's been doing this on both drives, from day 1. I updated firmware at the time, but it didn't seem to have any affect, and the drives have been running fine for well over a year at this point.

 

 

Snippet:

Apr 26 07:03:37 server kernel: BTRFS warning (device nvme1n1p1): i/o error at logical 812252712960 on dev /dev/nvme1n1p1, physical 469695516672, root 5, inode 1072832, offset 4437626880, length 4096, links 1 (path: domains/eco/vdisk1.img)
Apr 26 07:03:37 server kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 21284, flush 0, corrupt 0, gen 0
Apr 26 07:03:37 server kernel: BTRFS warning (device nvme1n1p1): i/o error at logical 812249567232 on dev /dev/nvme1n1p1, physical 469692370944, root 5, inode 1072832, offset 4434481152, length 4096, links 1 (path: domains/eco/vdisk1.img)
Apr 26 07:03:37 server kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 21285, flush 0, corrupt 0, gen 0
Apr 26 07:03:41 server kernel: nvme_log_error: 22 callbacks suppressed
Apr 26 07:03:41 server kernel: nvme1n1: I/O Cmd(0x2) @ LBA 917376120, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR 
Apr 26 07:03:41 server kernel: blk_print_req_error: 22 callbacks suppressed
Apr 26 07:03:41 server kernel: critical medium error, dev nvme1n1, sector 917376120 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
Apr 26 07:03:41 server kernel: scrub_handle_errored_block: 14 callbacks suppressed
Apr 26 07:03:41 server kernel: BTRFS warning (device nvme1n1p1): i/o error at logical 812252721152 on dev /dev/nvme1n1p1, physical 469695524864, root 5, inode 1072832, offset 4437635072, length 4096, links 1 (path: domains/eco/vdisk1.img)
Apr 26 07:03:41 server kernel: nvme1n1: I/O Cmd(0x2) @ LBA 917369976, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR 
Apr 26 07:03:41 server kernel: critical medium error, dev nvme1n1, sector 917369976 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
Apr 26 07:03:41 server kernel: btrfs_dev_stat_print_on_error: 14 callbacks suppressed
Apr 26 07:03:41 server kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 21300, flush 0, corrupt 0, gen 0
Apr 26 07:03:41 server kernel: nvme1n1: I/O Cmd(0x2) @ LBA 917379192, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR 
Apr 26 07:03:41 server kernel: critical medium error, dev nvme1n1, sector 917379192 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
Apr 26 07:03:41 server kernel: BTRFS warning (device nvme1n1p1): i/o error at logical 812249575424 on dev /dev/nvme1n1p1, physical 469692379136, root 5, inode 1072832, offset 4434489344, length 4096, links 1 (path: domains/eco/vdisk1.img)
Apr 26 07:03:41 server kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 21301, flush 0, corrupt 0, gen 0
Apr 26 07:03:41 server kernel: BTRFS warning (device nvme1n1p1): i/o error at logical 812254294016 on dev /dev/nvme1n1p1, physical 469697097728, root 5, inode 1072832, offset 4439207936, length 4096, links 1 (path: domains/eco/vdisk1.img)
Apr 26 07:03:41 server kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 21302, flush 0, corrupt 0, gen 0
Apr 26 07:03:41 server kernel: nvme1n1: I/O Cmd(0x2) @ LBA 917382264, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR 
Apr 26 07:03:41 server kernel: critical medium error, dev nvme1n1, sector 917382264 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
Apr 26 07:03:41 server kernel: nvme1n1: I/O Cmd(0x2) @ LBA 917385336, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR 
Apr 26 07:03:41 server kernel: critical medium error, dev nvme1n1, sector 917385336 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
Apr 26 07:03:41 server kernel: BTRFS warning (device nvme1n1p1): i/o error at logical 812255866880 on dev /dev/nvme1n1p1, physical 469698670592, root 5, inode 1072832, offset 4440780800, length 4096, links 1 (path: domains/eco/vdisk1.img)
Apr 26 07:03:41 server kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 21303, flush 0, corrupt 0, gen 0
Apr 26 07:03:41 server kernel: nvme1n1: I/O Cmd(0x2) @ LBA 917388408, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR 
Apr 26 07:03:41 server kernel: critical medium error, dev nvme1n1, sector 917388408 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
Apr 26 07:03:41 server kernel: nvme1n1: I/O Cmd(0x2) @ LBA 917373048, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR 
Apr 26 07:03:41 server kernel: critical medium error, dev nvme1n1, sector 917373048 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0

 

Diags attached as well.

server-diagnostics-20230506-1116.zip

Apr  3 13:51:19 server kernel: BTRFS info (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 21071, flush 0, corrupt 0, gen 0
Apr  3 13:51:19 server kernel: BTRFS info (device nvme1n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 3232, flush 0, corrupt 0, gen 0

 

Both devices are reporting read errors, these look like device problem for both, though one is worse than the other, from SMART:

 

cache1

Media and Data Integrity Errors:    23,329

cache2

Media and Data Integrity Errors:    4,105

 

Thanks for the quick response!

 

They were overheating originally, so I added some heatsinks, though my understanding is that the SSDs with throttle performance to avoid errors just like this, so I'm not quite sure what's going on.

Look for a firmware update, IIRC there are knwon issues with some devices from this model that an update can fix, not sure if it's this one though.

Updated. I'll keep an eye on it and report back. Cheers.