Jump to content

Cache drive errors forced drive into readonly

veri745

By veri745
in General Support

Recommended Posts

veri745 Rookie

veri745

Posted
Posted

I found all my docker containers unresponsive this morning.  It appears my NVMe cache drive encountered some errors and went into read-only mode.  What steps should I take to investigate/resolve the issue?

  

Apr 27 00:00:01 Tower Plugin Auto Update: Checking for available plugin updates
Apr 27 00:00:01 Tower Docker Auto Update: Community Applications Docker Autoupdate running
Apr 27 00:00:01 Tower Docker Auto Update: Checking for available updates
Apr 27 00:00:06 Tower Plugin Auto Update: community.applications.plg version 2023.04.26b does not meet age requirements to update - 1 days old
Apr 27 00:00:06 Tower Plugin Auto Update: compose.manager.plg version 2023.04.27 does not meet age requirements to update - 0 days old
Apr 27 00:00:07 Tower Plugin Auto Update: Checking for language updates
Apr 27 00:00:07 Tower Plugin Auto Update: Community Applications Plugin Auto Update finished
Apr 27 00:00:28 Tower Docker Auto Update: No updates will be installed
Apr 27 00:19:08 Tower  emhttpd: spinning down /dev/sdd
Apr 27 00:19:08 Tower  emhttpd: spinning down /dev/sde
Apr 27 00:19:37 Tower  emhttpd: spinning down /dev/sdb
Apr 27 00:19:51 Tower  emhttpd: spinning down /dev/sdf
Apr 27 00:23:53 Tower  emhttpd: read SMART /dev/sdg
Apr 27 00:23:53 Tower  emhttpd: read SMART /dev/sde
Apr 27 00:36:07 Tower  emhttpd: read SMART /dev/sdb
Apr 27 00:40:07 Tower  emhttpd: read SMART /dev/sdf
Apr 27 00:55:24 Tower  emhttpd: spinning down /dev/sde
Apr 27 01:09:05 Tower  emhttpd: spinning down /dev/sdb
Apr 27 01:13:05 Tower  emhttpd: spinning down /dev/sdg
Apr 27 01:13:05 Tower  emhttpd: spinning down /dev/sdf
Apr 27 02:07:29 Tower  emhttpd: read SMART /dev/sdb
Apr 27 02:07:48 Tower  emhttpd: read SMART /dev/sde
Apr 27 02:08:24 Tower  emhttpd: read SMART /dev/sdd
Apr 27 02:38:25 Tower  emhttpd: spinning down /dev/sdb
Apr 27 03:14:24 Tower kernel: nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
Apr 27 03:14:24 Tower kernel: nvme nvme0: Does your device have a faulty power saving mode enabled?
Apr 27 03:14:24 Tower kernel: nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug
Apr 27 03:14:24 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 639968960, 8 blocks, I/O Error (sct 0x3 / sc 0x71) 
Apr 27 03:14:24 Tower kernel: I/O error, dev nvme0n1, sector 639968960 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Apr 27 03:14:24 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 639968976, 8 blocks, I/O Error (sct 0x3 / sc 0x71) 
Apr 27 03:14:24 Tower kernel: I/O error, dev nvme0n1, sector 639968976 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Apr 27 03:14:24 Tower kernel: nvme 0000:09:00.0: enabling device (0000 -> 0002)
Apr 27 03:14:24 Tower kernel: nvme nvme0: Removing after probe failure status: -19
Apr 27 03:14:24 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 1, flush 0, corrupt 0, gen 0
Apr 27 03:14:24 Tower kernel: nvme0n1: detected capacity change from 976773168 to 0
Apr 27 03:14:24 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1, rd 2, flush 0, corrupt 0, gen 0
Apr 27 03:14:24 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1, rd 1, flush 0, corrupt 0, gen 0
Apr 27 03:14:24 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 2, rd 2, flush 0, corrupt 0, gen 0
Apr 27 03:14:24 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 3, rd 2, flush 0, corrupt 0, gen 0
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 1,34817 sector 0xe028bc8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 1,34817 sector 0x8b0868 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 4, rd 2, flush 0, corrupt 0, gen 0
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 1,34817 sector 0x9c4af0 len 0 err no 10
Apr 27 03:14:24 Tower kernel: I/O error, dev loop2, sector 9709456 op 0x1:(WRITE) flags 0x8800 phys_seg 10 prio class 0
Apr 27 03:14:24 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 5, rd 2, flush 0, corrupt 0, gen 0
Apr 27 03:14:24 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 6, rd 2, flush 0, corrupt 0, gen 0
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 1,34817 sector 0x9c5350 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
Apr 27 03:14:24 Tower kernel: BTRFS warning (device loop2): direct IO failed ino 9757 rw 1,34817 sector 0x942790 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 7, rd 2, flush 0, corrupt 0, gen 0
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 1,34817 sector 0x871130 len 0 err no 10
Apr 27 03:14:24 Tower kernel: I/O error, dev loop2, sector 894240 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 0
Apr 27 03:14:24 Tower kernel: I/O error, dev loop2, sector 2991392 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 0
Apr 27 03:14:24 Tower kernel: I/O error, dev loop2, sector 897600 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 0
Apr 27 03:14:24 Tower kernel: I/O error, dev loop2, sector 2994752 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 0
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xdc21160 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xe00b1a0 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xe00b1a8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xe00b1b0 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xe00b1b8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xe00b1c0 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xe00b1c8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xe00b1d0 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xe00b1d8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x524a508 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x524a510 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x524a518 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x524a520 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x524a528 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x524a530 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x524a538 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x524a540 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x524a548 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x524a550 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x524a558 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x524a560 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xe00f6d8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xe00f6e0 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xe00f6e8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xe00f6f0 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xe00f6f8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xe00f700 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xe00f708 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xe00f710 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xe00f718 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xe00f720 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xe00f728 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xdd564b8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xdd564c0 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xdd564c8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xdd564d0 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xdd564d8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xdd564e0 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xdd564e8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xdd564f0 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xdd564f8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xdd56500 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xdd56508 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xdd56510 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xdd56518 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x95b1fd0 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x95b1fd8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x95b1fe0 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x95b1fe8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x95b1ff0 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x95b1ff8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x95b2000 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x95b2008 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x95b2010 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5238150 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5238158 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5238160 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5238168 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5238170 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5238178 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5238180 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5238188 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5238190 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5238198 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x52381a0 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x52381a8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5239868 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5239870 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5239878 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5239880 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5239888 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5239890 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5239898 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5236c60 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5236c68 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5236c70 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5236c78 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5236c80 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5236c88 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5236c90 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5236c98 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0x5236ca0 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xddef1b8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xddef1c0 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xddef1c8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xddef1d0 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xddef1d8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xddef1e0 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xddef1e8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xddef1f0 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xddef1f8 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xddef200 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xddef208 len 0 err no 10
Apr 27 03:14:24 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 0,0 sector 0xde00880 len 0 err no 10
Apr 27 03:14:24 Tower kernel: I/O error, dev loop2, sector 8742272 op 0x0:(READ) flags 0x0 phys_seg 9 prio class 0
Apr 27 03:14:24 Tower kernel: I/O error, dev loop2, sector 9336344 op 0x0:(READ) flags 0x0 phys_seg 8 prio class 0
Apr 27 03:14:24 Tower kernel: I/O error, dev loop2, sector 10100888 op 0x0:(READ) flags 0x0 phys_seg 12 prio class 0
Apr 27 03:14:24 Tower kernel: BTRFS: error (device loop2: state A) in btrfs_run_delayed_refs:2157: errno=-5 IO failure
Apr 27 03:14:24 Tower kernel: BTRFS info (device loop2: state EA): forced readonly
Apr 27 03:14:24 Tower kernel: BTRFS: error (device loop2: state EA) in cleanup_transaction:1982: errno=-5 IO failure
Apr 27 03:14:24 Tower kernel: BTRFS: error (device nvme0n1p1) in btrfs_commit_transaction:2418: errno=-5 IO failure (Error while writing out transaction)
Apr 27 03:14:24 Tower kernel: BTRFS info (device nvme0n1p1: state E): forced readonly
Apr 27 03:14:24 Tower kernel: BTRFS: error (device nvme0n1p1: state EA) in cleanup_transaction:1982: errno=-5 IO failure
Apr 27 03:14:24 Tower  rsyslogd: file '/mnt/user/syslog/syslog-192.168.1.200.log'[10] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: Read-only file system [v8.2102.0 try https://www.rsyslog.com/e/2027 ]
### [PREVIOUS LINE REPEATED 3 TIMES] ###

 

tower-diagnostics-20230427-0726.zip

Link to comment
JorgeB Grand Master

JorgeB

Posted
Posted
1 hour ago, veri745 said: 
Apr 27 03:14:24 Tower kernel: nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug

Try this, it helps sometimes, on the main GUI page click on the flash drive, scroll down to "Syslinux Configuration", make sure it's set to "menu view" (top right) and add this to your default boot option, after "append initrd=/bzroot" 

nvme_core.default_ps_max_latency_us=0 pcie_aspm=off

e.g.: 

append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off


Reboot and see if it makes a difference.

 

 

  • Thanks 1
Link to comment
veri745 Rookie

veri745

Posted
  • Author
Posted
On 4/27/2023 at 9:30 AM, JorgeB said:

Try this, it helps sometimes, on the main GUI page click on the flash drive, scroll down to "Syslinux Configuration", make sure it's set to "menu view" (top right) and add this to your default boot option, after "append initrd=/bzroot" 

nvme_core.default_ps_max_latency_us=0 pcie_aspm=off

e.g.: 

append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off


Reboot and see if it makes a difference.

 

 

Not sure if this is what did it, but the issue has not reoccurred yet.

  • Like 1
Link to comment
veri745 Rookie

veri745

Posted
  • Author
Posted
On 5/2/2023 at 9:43 PM, veri745 said:

Not sure if this is what did it, but the issue has not reoccurred yet.

 

No joy after all, same error last night: 

May  6 02:40:05 Tower kernel: nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
May  6 02:40:05 Tower kernel: nvme nvme0: Does your device have a faulty power saving mode enabled?
May  6 02:40:05 Tower kernel: nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug
May  6 02:40:06 Tower kernel: nvme 0000:09:00.0: enabling device (0000 -> 0002)
May  6 02:40:06 Tower kernel: nvme nvme0: Removing after probe failure status: -19
May  6 02:40:06 Tower kernel: nvme0n1: detected capacity change from 976773168 to 0
May  6 02:40:06 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
### [PREVIOUS LINE REPEATED 1 TIMES] ###
May  6 02:40:06 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 1,34817 sector 0x1455f8f0 len 0 err no 10
May  6 02:40:06 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
May  6 02:40:06 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
May  6 02:40:06 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 1,34817 sector 0x5f0bac0 len 0 err no 10
May  6 02:40:06 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 4, rd 1, flush 0, corrupt 0, gen 0
May  6 02:40:06 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 5, rd 1, flush 0, corrupt 0, gen 0
May  6 02:40:06 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 1,34817 sector 0x14dbad78 len 0 err no 10
May  6 02:40:06 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 6, rd 1, flush 0, corrupt 0, gen 0
May  6 02:40:06 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 7, rd 1, flush 0, corrupt 0, gen 0
May  6 02:40:06 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 1,34817 sector 0x799ede0 len 0 err no 10
May  6 02:40:06 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 1,34817 sector 0x14f257e8 len 0 err no 10
May  6 02:40:06 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 8, rd 1, flush 0, corrupt 0, gen 0
May  6 02:40:06 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 1,34817 sector 0x14dbad58 len 0 err no 10
May  6 02:40:06 Tower kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 9, rd 1, flush 0, corrupt 0, gen 0
May  6 02:40:06 Tower kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 425294 rw 1,34817 sector 0x14f257c8 len 0 err no 10
May  6 02:40:06 Tower kernel: I/O error, dev loop2, sector 6879848 op 0x1:(WRITE) flags 0x8800 phys_seg 8 prio class 0
May  6 02:40:06 Tower kernel: BTRFS warning (device loop2): direct IO failed ino 10754 rw 1,34817 sector 0x68fa68 len 0 err no 10
May  6 02:40:06 Tower kernel: I/O error, dev loop2, sector 206432 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 0
May  6 02:40:06 Tower kernel: I/O error, dev loop2, sector 2303584 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 0
May  6 02:40:06 Tower kernel: I/O error, dev loop2, sector 206400 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 0
May  6 02:40:06 Tower kernel: I/O error, dev loop2, sector 2303552 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 0
May  6 02:40:06 Tower kernel: BTRFS: error (device nvme0n1p1: state A) in btrfs_create_new_inode:6442: errno=-5 IO failure
May  6 02:40:06 Tower kernel: BTRFS info (device nvme0n1p1: state EA): forced readonly
May  6 02:40:06 Tower kernel: BTRFS warning (device nvme0n1p1: state EA): Skipping commit of aborted transaction.
May  6 02:40:06 Tower kernel: I/O error, dev loop2, sector 8740264 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
May  6 02:40:06 Tower kernel: BTRFS warning (device nvme0n1p1: state EA): direct IO failed ino 425294 rw 0,0 sector 0x1137f180 len 4096 err no 10

 

Link to comment
itimpi Grand Master

itimpi

Posted
Posted

There definitely seems to be an issue with the nvme controller because you are still getting: 

May  6 02:40:05 Tower kernel: nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
May  6 02:40:05 Tower kernel: nvme nvme0: Does your device have a faulty power saving mode enabled?
May  6 02:40:05 Tower kernel: nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug

Not sure what to suggest.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing
×
×
  • Create New...