Hi,

 

I noticed my VM's were all down and this in the logs:

 

Jun 27 22:19:42 FatBoy kernel: nvme nvme0: I/O 44 QID 1 timeout, aborting
Jun 27 22:19:42 FatBoy kernel: nvme nvme0: I/O 45 QID 1 timeout, aborting
Jun 27 22:19:42 FatBoy kernel: nvme nvme0: I/O 224 QID 7 timeout, aborting
Jun 27 22:19:42 FatBoy kernel: nvme nvme0: I/O 225 QID 7 timeout, aborting
Jun 27 22:20:11 FatBoy kernel: nvme nvme0: I/O 29 QID 0 timeout, reset controller
Jun 27 22:20:12 FatBoy kernel: nvme nvme0: I/O 44 QID 1 timeout, reset controller
Jun 27 22:22:12 FatBoy kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
Jun 27 22:22:12 FatBoy kernel: blk_update_request: I/O error, dev nvme0n1, sector 1230365384 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Jun 27 22:22:12 FatBoy kernel: blk_update_request: I/O error, dev nvme0n1, sector 1230238336 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Jun 27 22:22:12 FatBoy kernel: nvme nvme0: Abort status: 0x371
Jun 27 22:22:12 FatBoy kernel: nvme nvme0: Abort status: 0x371
Jun 27 22:22:12 FatBoy kernel: nvme nvme0: Abort status: 0x371
Jun 27 22:22:12 FatBoy kernel: nvme nvme0: Abort status: 0x371
Jun 27 22:23:12 FatBoy kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
Jun 27 22:23:12 FatBoy kernel: nvme nvme0: Removing after probe failure status: -19
Jun 27 22:24:13 FatBoy kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
Jun 27 22:24:13 FatBoy kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 1, flush 0, corrupt 0, gen 0
Jun 27 22:24:13 FatBoy kernel: nvme0n1: detected capacity change from 2000409264 to 0
Jun 27 22:24:13 FatBoy kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1, rd 1, flush 0, corrupt 0, gen 0
Jun 27 22:24:13 FatBoy kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1, rd 2, flush 0, corrupt 0, gen 0
Jun 27 22:24:13 FatBoy kernel: blk_update_request: I/O error, dev nvme0n1, sector 1195311008 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0

 

On reboot, my cache disk is not detected. I'm guessing it is completely dead? Is there any chance it could be recovered? Anything I can do?

 

Thanks,

attach diagnostics to your NEXT post in this thread

Attached.

fatboy-diagnostics-20220628-0547.zip

Unplugged everything and reconnected the cables, cleaned connection. It seems to be working fine. Taking a backup of everything now just in case.

NVMe device dropped offline, if it's a reoccurring issue this can sometimes help:

 

Some NVMe devices have issues with power states on Linux, try this, on the main GUI page click on flash, scroll down to "Syslinux Configuration", make sure it's set to "menu view" (on the top right) and add this to your default boot option, after "append initrd=/bzroot"

nvme_core.default_ps_max_latency_us=0

e.g.:

append initrd=/bzroot nvme_core.default_ps_max_latency_us=0

Reboot and see if it makes a difference.

 

 

28 minutes ago, JorgeB said:

NVMe device dropped offline, if it's a reoccurring issue this can sometimes help:

 

Some NVMe devices have issues with power states on Linux, try this, on the main GUI page click on flash, scroll down to "Syslinux Configuration", make sure it's set to "menu view" (on the top right) and add this to your default boot option, after "append initrd=/bzroot"

nvme_core.default_ps_max_latency_us=0

e.g.:

append initrd=/bzroot nvme_core.default_ps_max_latency_us=0

Reboot and see if it makes a difference.

 

 

 

Thanks. I'll give it a try.