Hi all I just saw that my log were filled with errors mentioning BTRFS. I am bit lost and hope someone could help me identify the problem.
The NVME drive is using XFS and is set as "cache", and I have 2 SATA SSDs using BTRFS set up as a separate cache "sata-raid" pool.
Note that for now, TRIM fails on the 2 SATA SSDs because my HBA card does not support it.
Below are the first occurrences of the errors. The diagnostics files are also attached.
Thank you so much for you help.
OP
Jan 31 08:44:57 MOZART emhttpd: spinning down /dev/sdb
Jan 31 08:45:28 MOZART kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
Jan 31 08:45:28 MOZART kernel: nvme nvme0: Abort status: 0x371
Jan 31 08:45:28 MOZART kernel: nvme nvme0: Abort status: 0x371
Jan 31 08:45:28 MOZART kernel: nvme nvme0: Abort status: 0x371
Jan 31 08:45:28 MOZART kernel: nvme nvme0: Abort status: 0x371
Jan 31 08:46:29 MOZART kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
Jan 31 08:46:29 MOZART kernel: nvme nvme0: Removing after probe failure status: -19
Jan 31 08:47:29 MOZART kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
Jan 31 08:47:29 MOZART kernel: nvme0n1: detected capacity change from 2000409264 to 0
Jan 31 08:47:29 MOZART kernel: blk_update_request: I/O error, dev nvme0n1, sector 1061649312 op 0x1:(WRITE) flags 0x8800 phys_seg 12 prio class 0
Jan 31 08:47:29 MOZART kernel: blk_update_request: I/O error, dev nvme0n1, sector 1062183136 op 0x1:(WRITE) flags 0x8800 phys_seg 12 prio class 0
Jan 31 08:47:29 MOZART kernel: blk_update_request: I/O error, dev nvme0n1, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0
Jan 31 08:47:29 MOZART kernel: blk_update_request: I/O error, dev nvme0n1, sector 1061659456 op 0x1:(WRITE) flags 0x8800 phys_seg 12 prio class 0
Jan 31 08:47:29 MOZART kernel: blk_update_request: I/O error, dev nvme0n1, sector 18836248 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
Jan 31 08:47:29 MOZART kernel: blk_update_request: I/O error, dev nvme0n1, sector 1509999680 op 0x1:(WRITE) flags 0x800 phys_seg 7 prio class 0
Jan 31 08:47:29 MOZART kernel: blk_update_request: I/O error, dev nvme0n1, sector 1013529432 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
Jan 31 08:47:29 MOZART kernel: blk_update_request: I/O error, dev nvme0n1, sector 7064120 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
Jan 31 08:47:29 MOZART kernel: blk_update_request: I/O error, dev nvme0n1, sector 7064080 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
Jan 31 08:47:29 MOZART kernel: blk_update_request: I/O error, dev nvme0n1, sector 1013686704 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
Jan 31 08:47:29 MOZART kernel: dm-12: writeback error on inode 15528150, offset 1048576, sector 18801432
Jan 31 08:47:29 MOZART kernel: dm-12: writeback error on inode 15528150, offset 1052672, sector 18801440
Jan 31 08:47:29 MOZART kernel: dm-12: writeback error on inode 15528150, offset 1056768, sector 18801448
Jan 31 08:47:29 MOZART kernel: dm-12: writeback error on inode 1620304068, offset 0, sector 1509964864
Jan 31 08:47:29 MOZART kernel: dm-12: writeback error on inode 7026833, offset 49152, sector 7029264
Jan 31 08:47:29 MOZART kernel: dm-12: writeback error on inode 7026833, offset 53248, sector 7029304
Jan 31 08:47:29 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
Jan 31 08:47:29 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
Jan 31 08:47:29 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
Jan 31 08:47:29 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
Jan 31 08:47:29 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
Jan 31 08:47:29 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
Jan 31 08:47:29 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
Jan 31 08:47:29 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
Jan 31 08:47:29 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
Jan 31 08:47:29 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0
Jan 31 08:47:29 MOZART kernel: dm-12: writeback error on inode 15053696, offset 581632, sector 24232304
Jan 31 08:47:29 MOZART kernel: dm-12: writeback error on inode 547072497, offset 4096, sector 510551288
Jan 31 08:47:29 MOZART kernel: XFS (dm-12): log I/O error -5
Jan 31 08:47:29 MOZART kernel: BTRFS: error (device loop2) in btrfs_commit_transaction:2438: errno=-5 IO failure (Error while writing out transaction)
Jan 31 08:47:29 MOZART kernel: BTRFS: error (device loop2) in __btrfs_free_extent:3069: errno=-5 IO failure
Jan 31 08:47:29 MOZART kernel: BTRFS info (device loop2): forced readonly
Jan 31 08:47:29 MOZART kernel: BTRFS: error (device loop2) in btrfs_run_delayed_refs:2150: errno=-5 IO failure
Jan 31 08:47:29 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [34803916800, 34803920896)
Jan 31 08:47:29 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [34803916800, 34803920896)
Jan 31 08:47:29 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [34803916800, 34803920896)
Jan 31 08:47:29 MOZART rsyslogd: file '/mnt/user/logs/syslog/syslog-192.168.92.100.log'[10] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: Input/output error [v8.2102.0 try https://www.rsyslog.com/e/2027 ]
Jan 31 08:47:29 MOZART rsyslogd: file '/mnt/user/logs/syslog/syslog-192.168.92.100.log': open error: Input/output error [v8.2102.0 try https://www.rsyslog.com/e/2433 ]
Jan 31 08:47:29 MOZART kernel: XFS (dm-12): Log I/O Error (0x2) detected at xlog_ioend_work+0x50/0x6e [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
Jan 31 08:47:29 MOZART kernel: BTRFS warning (device loop2): Skipping commit of aborted transaction.
Jan 31 08:47:29 MOZART kernel: XFS (dm-12): Please unmount the filesystem and rectify the problem(s)
Jan 31 08:47:29 MOZART kernel: dm-12: writeback error on inode 15880868, offset 5332992, sector 511744744
Jan 31 08:47:29 MOZART kernel: BTRFS: error (device loop2) in cleanup_transaction:2011: errno=-5 IO failure
Jan 31 08:47:29 MOZART kernel: dm-12: writeback error on inode 15880868, offset 5337088, sector 511744752
Jan 31 08:47:29 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [20687400960, 20687405056)
Jan 31 08:47:29 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [20687405056, 20687409152)
Jan 31 08:47:29 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [20687409152, 20687413248)
Jan 31 08:47:29 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [20687400960, 20687405056)
Jan 31 08:47:29 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [17793519616, 17793523712)
Jan 31 08:47:29 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [17793523712, 17793527808)
Jan 31 08:47:29 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [17793519616, 17793523712)
Jan 31 08:47:29 MOZART rsyslogd: file '/mnt/user/logs/syslog/syslog-192.168.92.10.log'[2] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: Input/output error [v8.2102.0 try https://www.rsyslog.com/e/2027 ]
Jan 31 08:47:29 MOZART rsyslogd: file '/mnt/user/logs/syslog/syslog-192.168.92.10.log': open error: Input/output error [v8.2102.0 try https://www.rsyslog.com/e/2433 ]
Jan 31 08:48:04 MOZART kernel: print_req_error: 99 callbacks suppressed
Jan 31 08:48:04 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 1249656 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
Jan 31 08:48:04 MOZART kernel: btrfs_dev_stat_print_on_error: 77 callbacks suppressed
Jan 31 08:48:04 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 47, rd 41, flush 0, corrupt 0, gen 0
Jan 31 08:48:04 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 1249672 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
Jan 31 08:48:04 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 41, flush 0, corrupt 0, gen 0
Jan 31 08:48:12 MOZART kernel: btrfs_lookup_bio_sums: 18 callbacks suppressed
Jan 31 08:48:12 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [21088358400, 21088362496)
Jan 31 08:48:12 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 36485992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jan 31 08:48:12 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 42, flush 0, corrupt 0, gen 0
Jan 31 08:48:12 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [21088358400, 21088362496)
Jan 31 08:48:12 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 36485992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jan 31 08:48:12 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 43, flush 0, corrupt 0, gen 0
Jan 31 08:48:12 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [21088358400, 21088362496)
Jan 31 08:48:12 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 36485992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jan 31 08:48:12 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 44, flush 0, corrupt 0, gen 0
Jan 31 08:48:12 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [21088358400, 21088362496)
Jan 31 08:48:12 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 36485992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jan 31 08:48:12 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 45, flush 0, corrupt 0, gen 0
Jan 31 08:48:12 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [21088358400, 21088362496)
Jan 31 08:48:12 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 36485992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jan 31 08:48:12 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 46, flush 0, corrupt 0, gen 0
Jan 31 08:48:12 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [21088358400, 21088362496)
Jan 31 08:48:12 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 36485992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jan 31 08:48:12 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 47, flush 0, corrupt 0, gen 0
Jan 31 08:48:12 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [21088358400, 21088362496)
Jan 31 08:48:12 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 36485992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jan 31 08:48:12 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 48, flush 0, corrupt 0, gen 0
Jan 31 08:48:12 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [21088358400, 21088362496)
Jan 31 08:48:12 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 36485992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jan 31 08:48:12 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 49, flush 0, corrupt 0, gen 0
Jan 31 08:48:12 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [21088358400, 21088362496)
Jan 31 08:48:12 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 36485992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jan 31 08:48:12 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 50, flush 0, corrupt 0, gen 0
Jan 31 08:48:12 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [21088358400, 21088362496)
Jan 31 08:48:12 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 36485992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jan 31 08:48:12 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 51, flush 0, corrupt 0, gen 0
Jan 31 08:50:07 MOZART kernel: print_req_error: 2 callbacks suppressed
Jan 31 08:50:07 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 37026528 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
Jan 31 08:50:07 MOZART kernel: btrfs_dev_stat_print_on_error: 2 callbacks suppressed
Jan 31 08:50:07 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 54, flush 0, corrupt 0, gen 0
Jan 31 08:50:07 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 37550816 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
Jan 31 08:50:07 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 55, flush 0, corrupt 0, gen 0
Jan 31 08:50:07 MOZART kernel: btrfs_lookup_bio_sums: 2 callbacks suppressed
Jan 31 08:50:07 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [17946841088, 17946845184)
Jan 31 08:50:07 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [17946845184, 17946849280)
Jan 31 08:50:07 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [17946849280, 17946853376)
Jan 31 08:50:07 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [17946853376, 17946857472)
Jan 31 08:50:07 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [17946857472, 17946861568)
Jan 31 08:50:07 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [17946861568, 17946865664)
Jan 31 08:50:07 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [17946865664, 17946869760)
Jan 31 08:50:07 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [17946869760, 17946873856)
Jan 31 08:50:07 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [17946873856, 17946877952)
Jan 31 08:50:07 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [17946877952, 17946882048)
Jan 31 08:50:07 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 34544520 op 0x0:(READ) flags 0x80700 phys_seg 14 prio class 0
Jan 31 08:50:07 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 34544520 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jan 31 08:50:07 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 56, flush 0, corrupt 0, gen 0
Jan 31 08:50:07 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 37182624 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
Jan 31 08:50:07 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 57, flush 0, corrupt 0, gen 0
Jan 31 08:50:07 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 37706912 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
Jan 31 08:50:07 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 58, flush 0, corrupt 0, gen 0
Jan 31 08:50:07 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 37184224 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
Jan 31 08:50:07 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 59, flush 0, corrupt 0, gen 0
Jan 31 08:50:07 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 37708512 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
Jan 31 08:50:07 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 60, flush 0, corrupt 0, gen 0
Jan 31 08:50:07 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 37186080 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
Jan 31 08:50:07 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 61, flush 0, corrupt 0, gen 0
Jan 31 08:50:07 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 37710368 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
Jan 31 08:50:07 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 62, flush 0, corrupt 0, gen 0
Jan 31 08:50:07 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 63, flush 0, corrupt 0, gen 0
Jan 31 08:51:22 MOZART kernel: btrfs_lookup_bio_sums: 5 callbacks suppressed
Jan 31 08:51:22 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [21088358400, 21088362496)
Jan 31 08:51:22 MOZART kernel: print_req_error: 4542 callbacks suppressed
Jan 31 08:51:22 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 36485992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jan 31 08:51:22 MOZART kernel: btrfs_dev_stat_print_on_error: 4541 callbacks suppressed
Jan 31 08:51:22 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 4605, flush 0, corrupt 0, gen 0
Jan 31 08:51:22 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [21088358400, 21088362496)
Jan 31 08:51:22 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 36485992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jan 31 08:51:22 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 4606, flush 0, corrupt 0, gen 0
Jan 31 08:51:22 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [21088358400, 21088362496)
Jan 31 08:51:22 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 36485992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jan 31 08:51:22 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 4607, flush 0, corrupt 0, gen 0
Jan 31 08:51:22 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [21088358400, 21088362496)
Jan 31 08:51:22 MOZART kernel: blk_update_request: I/O error, dev loop2, sector 36485992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jan 31 08:51:22 MOZART kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 48, rd 4608, flush 0, corrupt 0, gen 0
Jan 31 08:51:22 MOZART kernel: BTRFS warning (device loop2): csum hole found for disk bytenr range [21088358400, 21088362496)