paulmorabi Posted June 28, 2022 Share Posted June 28, 2022 Hi, I noticed my VM's were all down and this in the logs: Jun 27 22:19:42 FatBoy kernel: nvme nvme0: I/O 44 QID 1 timeout, aborting Jun 27 22:19:42 FatBoy kernel: nvme nvme0: I/O 45 QID 1 timeout, aborting Jun 27 22:19:42 FatBoy kernel: nvme nvme0: I/O 224 QID 7 timeout, aborting Jun 27 22:19:42 FatBoy kernel: nvme nvme0: I/O 225 QID 7 timeout, aborting Jun 27 22:20:11 FatBoy kernel: nvme nvme0: I/O 29 QID 0 timeout, reset controller Jun 27 22:20:12 FatBoy kernel: nvme nvme0: I/O 44 QID 1 timeout, reset controller Jun 27 22:22:12 FatBoy kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1 Jun 27 22:22:12 FatBoy kernel: blk_update_request: I/O error, dev nvme0n1, sector 1230365384 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Jun 27 22:22:12 FatBoy kernel: blk_update_request: I/O error, dev nvme0n1, sector 1230238336 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 Jun 27 22:22:12 FatBoy kernel: nvme nvme0: Abort status: 0x371 Jun 27 22:22:12 FatBoy kernel: nvme nvme0: Abort status: 0x371 Jun 27 22:22:12 FatBoy kernel: nvme nvme0: Abort status: 0x371 Jun 27 22:22:12 FatBoy kernel: nvme nvme0: Abort status: 0x371 Jun 27 22:23:12 FatBoy kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1 Jun 27 22:23:12 FatBoy kernel: nvme nvme0: Removing after probe failure status: -19 Jun 27 22:24:13 FatBoy kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1 Jun 27 22:24:13 FatBoy kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 1, flush 0, corrupt 0, gen 0 Jun 27 22:24:13 FatBoy kernel: nvme0n1: detected capacity change from 2000409264 to 0 Jun 27 22:24:13 FatBoy kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1, rd 1, flush 0, corrupt 0, gen 0 Jun 27 22:24:13 FatBoy kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1, rd 2, flush 0, corrupt 0, gen 0 Jun 27 22:24:13 FatBoy kernel: blk_update_request: I/O error, dev nvme0n1, sector 1195311008 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0 On reboot, my cache disk is not detected. I'm guessing it is completely dead? Is there any chance it could be recovered? Anything I can do? Thanks, Quote Link to comment
trurl Posted June 28, 2022 Share Posted June 28, 2022 attach diagnostics to your NEXT post in this thread Quote Link to comment
paulmorabi Posted June 28, 2022 Author Share Posted June 28, 2022 Attached. fatboy-diagnostics-20220628-0547.zip Quote Link to comment
paulmorabi Posted June 28, 2022 Author Share Posted June 28, 2022 Unplugged everything and reconnected the cables, cleaned connection. It seems to be working fine. Taking a backup of everything now just in case. Quote Link to comment
JorgeB Posted June 28, 2022 Share Posted June 28, 2022 NVMe device dropped offline, if it's a reoccurring issue this can sometimes help: Some NVMe devices have issues with power states on Linux, try this, on the main GUI page click on flash, scroll down to "Syslinux Configuration", make sure it's set to "menu view" (on the top right) and add this to your default boot option, after "append initrd=/bzroot" nvme_core.default_ps_max_latency_us=0 e.g.: append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 Reboot and see if it makes a difference. Quote Link to comment
paulmorabi Posted June 28, 2022 Author Share Posted June 28, 2022 28 minutes ago, JorgeB said: NVMe device dropped offline, if it's a reoccurring issue this can sometimes help: Some NVMe devices have issues with power states on Linux, try this, on the main GUI page click on flash, scroll down to "Syslinux Configuration", make sure it's set to "menu view" (on the top right) and add this to your default boot option, after "append initrd=/bzroot" nvme_core.default_ps_max_latency_us=0 e.g.: append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 Reboot and see if it makes a difference. Thanks. I'll give it a try. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.