Noticed one of my VMs was unresponsive. It looks like my cache drive just dropped offline? Never had a cache drive just drop out suddenly like this. I have had zero issues with this drive since it was installed years ago so this is totally out of left field.

Feb 7 14:53:43 VOID kernel: nvme nvme0: I/O tag 453 (31c5) opcode 0x0 (I/O Cmd) QID 11 timeout, aborting req_op:FLUSH(2) size:0

Feb 7 14:53:43 VOID kernel: nvme nvme0: Abort status: 0x0

Feb 7 14:53:45 VOID kernel: nvme nvme0: I/O tag 129 (b081) opcode 0x2 (I/O Cmd) QID 1 timeout, aborting req_op:READ(0) size:65536

Feb 7 14:53:45 VOID kernel: nvme nvme0: Abort status: 0x0

Feb 7 14:53:49 VOID kernel: nvme nvme0: I/O tag 792 (5318) opcode 0x2 (I/O Cmd) QID 5 timeout, aborting req_op:READ(0) size:8192

Feb 7 14:53:49 VOID kernel: nvme nvme0: I/O tag 793 (5319) opcode 0x2 (I/O Cmd) QID 5 timeout, aborting req_op:READ(0) size:12288

Feb 7 14:53:49 VOID kernel: nvme nvme0: Abort status: 0x0

### [PREVIOUS LINE REPEATED 1 TIMES] ###

Feb 7 14:53:53 VOID kernel: nvme nvme0: I/O tag 454 (e1c6) opcode 0x2 (I/O Cmd) QID 11 timeout, aborting req_op:READ(0) size:16384

Feb 7 14:53:53 VOID kernel: nvme nvme0: Abort status: 0x0

Feb 7 14:53:58 VOID kernel: nvme nvme0: I/O tag 130 (7082) opcode 0x9 (I/O Cmd) QID 1 timeout, aborting req_op:DISCARD(3) size:32505856

Feb 7 14:53:58 VOID kernel: nvme nvme0: Abort status: 0x0

Feb 7 14:54:13 VOID kernel: nvme nvme0: I/O tag 453 (31c5) opcode 0x0 (I/O Cmd) QID 11 timeout, reset controller

Feb 7 14:55:26 VOID kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1

Feb 7 14:55:26 VOID kernel: nvme0n1: I/O Cmd(0x2) @ LBA 153442520, 128 blocks, I/O Error (sct 0x3 / sc 0x71)

Feb 7 14:55:26 VOID kernel: I/O error, dev nvme0n1, sector 153442520 op 0x0:(READ) flags 0x80700 phys_seg 8 prio class 2

Feb 7 14:55:26 VOID kernel: nvme0n1: I/O Cmd(0x2) @ LBA 225096848, 16 blocks, I/O Error (sct 0x3 / sc 0x71)

Feb 7 14:55:26 VOID kernel: I/O error, dev nvme0n1, sector 225096848 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2

Feb 7 14:55:26 VOID kernel: nvme0n1: I/O Cmd(0x2) @ LBA 246795088, 24 blocks, I/O Error (sct 0x3 / sc 0x71)

Feb 7 14:55:26 VOID kernel: I/O error, dev nvme0n1, sector 246795088 op 0x0:(READ) flags 0x80700 phys_seg 3 prio class 2

Feb 7 14:55:26 VOID kernel: nvme0n1: I/O Cmd(0x2) @ LBA 597332760, 1024 blocks, I/O Error (sct 0x3 / sc 0x71)

Feb 7 14:55:26 VOID kernel: I/O error, dev nvme0n1, sector 597332760 op 0x0:(READ) flags 0x84700 phys_seg 126 prio class 2

Feb 7 14:55:26 VOID kernel: nvme0n1: I/O Cmd(0x2) @ LBA 597496360, 1024 blocks, I/O Error (sct 0x3 / sc 0x71)

Feb 7 14:55:26 VOID kernel: I/O error, dev nvme0n1, sector 597496360 op 0x0:(READ) flags 0x84700 phys_seg 127 prio class 2

Feb 7 14:55:26 VOID kernel: nvme0n1: I/O Cmd(0x2) @ LBA 597497384, 24 blocks, I/O Error (sct 0x3 / sc 0x71)

Feb 7 14:55:26 VOID kernel: I/O error, dev nvme0n1, sector 597497384 op 0x0:(READ) flags 0x80700 phys_seg 3 prio class 2

Feb 7 14:55:26 VOID kernel: nvme0n1: I/O Cmd(0x2) @ LBA 597333784, 456 blocks, I/O Error (sct 0x3 / sc 0x71)

Feb 7 14:55:26 VOID kernel: I/O error, dev nvme0n1, sector 597333784 op 0x0:(READ) flags 0x80700 phys_seg 57 prio class 2

Feb 7 14:55:36 VOID kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1

Feb 7 14:55:38 VOID kernel: nvme 0000:02:00.0: not ready 1023ms after FLR; waiting

Feb 7 14:55:39 VOID kernel: nvme 0000:02:00.0: not ready 2047ms after FLR; waiting

Feb 7 14:55:41 VOID kernel: nvme 0000:02:00.0: not ready 4095ms after FLR; waiting

Feb 7 14:55:46 VOID kernel: nvme 0000:02:00.0: not ready 8191ms after FLR; waiting

Feb 7 14:55:54 VOID kernel: nvme 0000:02:00.0: not ready 16383ms after FLR; waiting

Feb 7 14:56:11 VOID kernel: nvme 0000:02:00.0: not ready 32767ms after FLR; waiting

Feb 7 14:56:47 VOID kernel: nvme 0000:02:00.0: not ready 65535ms after FLR; giving up

Feb 7 14:56:47 VOID kernel: nvme nvme0: Disabling device after reset failure: -25

Feb 7 14:56:47 VOID kernel: I/O error, dev nvme0n1, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 2

Feb 7 14:56:47 VOID kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 1, flush 0, corrupt 0, gen 0

Feb 7 14:56:47 VOID kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 2, flush 0, corrupt 0, gen 0

Feb 7 14:56:47 VOID kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 2, flush 1, corrupt 0, gen 0

Feb 7 14:56:47 VOID kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 1, rd 2, flush 1, corrupt 0, gen 0

Feb 7 14:56:47 VOID kernel: BTRFS warning (device nvme0n1p1): chunk 13631488 missing 1 devices, max tolerance is 0 for writable mount

Feb 7 14:56:47 VOID kernel: BTRFS: error (device nvme0n1p1) in write_all_supers:4044: errno=-5 IO failure (errors while submitting device barriers.)

Feb 7 14:56:47 VOID kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 2, rd 2, flush 1, corrupt 0, gen 0

Feb 7 14:56:47 VOID kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 2, rd 3, flush 1, corrupt 0, gen 0

Feb 7 14:56:47 VOID kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 2, rd 4, flush 1, corrupt 0, gen 0

Feb 7 14:56:47 VOID kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 2, rd 5, flush 1, corrupt 0, gen 0

Feb 7 14:56:47 VOID kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 2, rd 6, flush 1, corrupt 0, gen 0

Feb 7 14:56:47 VOID kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 2, rd 7, flush 1, corrupt 0, gen 0

Feb 7 14:56:47 VOID kernel: I/O error, dev loop2, sector 80732704 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 2

Can't view anything in the webui about it. Going to try and reboot to see if it comes back up.

EDIT: well it seems to have come back up ok after a reboot? I've stopped all dockers and VMs for now. I'm going to do an FS check and a BTRFS scrub to look for errors.

BTRFS scrub passed, the FS check did find errors:

[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
[4/8] checking free space tree
We have a space info key for a block group that doesn't exist
[5/8] checking fs roots
[6/8] checking only csums items (without verifying data)
[7/8] checking root refs
[8/8] checking quota groups skipped (not enabled on this FS)
Opening filesystem to check...
Checking filesystem on /dev/nvme0n1p1
UUID: 93f1bb6c-be8a-4014-a762-2755d1e2e690
found 914831544320 bytes used, error(s) found
total csum bytes: 818051900
total tree bytes: 2522759168
total fs tree bytes: 1412333568
total extent tree bytes: 200622080
btree space waste bytes: 393034216
file data blocks allocated: 3480582864896
 referenced 888944390144

Currently have the array in maintenance mode while I wait for next steps.

void-diagnostics-20260208-0610.zip

Edit: Could it be related to power saving features? https://wiki.archlinux.org/title/Solid_state_drive/NVMe#Troubleshooting https://forums.unraid.net/topic/135766-seems-one-of-my-nvme-drives-threw-up-on-itself-overnight-help-diagnostics-attached/#comment-1234771

Seems odd that this would just now suddenly become an issue after years of trouble free operation if this was a power saving issue...I've not updated the drive firmware or changed any bios settings recently...

Edited February 8Feb 8 by weirdcrap

If it keeps happening, you can try this, on Main click on the flash drive, scroll down to "Syslinux Configuration", make sure it's set to "menu view" and add this to your default boot option, after "append initrd=/bzroot"

nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off

e.g.:

append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off

Reboot (or power cycle the server if just a reboot doesn't bring the device back) and then see if it makes a difference.

5 minutes ago, JorgeB said:

If it keeps happening, you can try this, on Main click on the flash drive, scroll down to "Syslinux Configuration", make sure it's set to "menu view" and add this to your default boot option, after "append initrd=/bzroot"

nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off

e.g.:

append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off

Reboot (or power cycle the server if just a reboot doesn't bring the device back) and then see if it makes a difference.

Ok and what about the BTRFS errors? Should I run a repair on the file system before I mess with the power saving features? I'm hesitant to run a repair without guidance as the help suggests.

This is the first and ONLY time this has ever happened so I find it hard to believe that the power saving features just suddenly became problematic after years of trouble free operation.

I'll try as you suggested, just strikes me as unlikely that without a firmware update or bios setting change that it would just suddenly become an issue on it's own. I'll also check the bios to make sure any PCIE/NVME power saving features are off.

EDIT: or could the "We have a space info key for a block group that doesn't exist" be a benign error? https://www.spinics.net/lists/linux-btrfs/msg160026.html https://bbs.archlinux.org/viewtopic.php?id=310626

Edited February 8Feb 8 by weirdcrap

You can try repairing, but make sure it's all backed up, btrfs check repair can sometimes make things worse.

29 minutes ago, JorgeB said:

You can try repairing, but make sure it's all backed up, btrfs check repair can sometimes make things worse.

Ok I'll get the drive back up first and make a backup before i attempt any repairs.

I double checked the bios and all PCI/PCIE ASPM options are disabled.

I'm fairly certain the btrfs error is ignorable so I didn't end up doing anything about it. The drive has stayed online and I have no reasonable reference on a test window to wait for so I'll just mark this as solved and bump it if it ends up returning later. I did add the lines recommended in the thread I linked/what JorgeB recommended.

Edited February 16Feb 16 by weirdcrap