I keep having an issue with my storage and cache pools where one of the nvme drop connection while the other stays running. It kept happening when I was running a btrfs pools so I switched to zfs and its still happening. I have no idea what to do at this point, the drive in zfs shows that its missing but it doesn't have any errors.

athena-syslog-20260309-0207.zip

Attach Diagnostics ZIP to your NEXT post in this thread.

Does it not work when I attach it to the main post?

athena-syslog-20260309-0207.zip

I'mbarely literate

athena-diagnostics-20260308-2220.zip

Mar 6 19:13:28 Athena kernel: nvme nvme0: I/O tag 411 (e19b) opcode 0x1 (I/O Cmd) QID 11 timeout, aborting req_op:WRITE(1) size:8192

Mar 6 19:13:45 Athena kernel: nvme nvme0: I/O tag 412 (d19c) opcode 0x1 (I/O Cmd) QID 11 timeout, aborting req_op:WRITE(1) size:12288

Mar 6 19:13:56 Athena kernel: nvme nvme0: I/O tag 4 (2004) opcode 0x6 (Admin Cmd) QID 0 timeout, reset controller

Mar 6 19:15:19 Athena kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1

Mar 6 19:15:19 Athena kernel: nvme nvme0: Abort status: 0x371

Mar 6 19:15:39 Athena kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1

Mar 6 19:15:39 Athena kernel: nvme nvme0: Disabling device after reset failure: -19

They are dropping offline, you can try this, on Main click on the flash drive, scroll down to "Syslinux Configuration", make sure it's set to "menu view" and add this to your default boot option, after "append initrd=/bzroot"

nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off

e.g.:

append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off

Reboot (or power cycle the server if just a reboot doesn't bring the device back) and then see if it makes a difference.

All right, I will give this a try. I also am going to update the firmware on all the nvme as well. In your opinion for cache and storage pools, would you do btrfs or zfs?

Thanks

11 minutes ago, LocksOfChaos said:

In your opinion for cache and storage pools, would you do btrfs or zfs?

For mirrors, recommend zfs, it's better at handling of dropped devices, and for single device pools, recommend both.

Note that this issue is unrelated to the fs used.

Thanks, I figured it wasn't an fs specific issue since its happened with both, I just wanted to make sure that I was using the best file system for my use case. I added that line to my boot options and did a shutdown and it brought the drives back. I've attached a screenshot below, I ran a new zpool status and the drive that dropped in both pools have 3 check sum errors, but it says that a scrub was run with no errors. Is there anything else I have to do for the check sum errors or just clear the errors from the pools?

5 minutes ago, LocksOfChaos said:

Is there anything else I have to do for the check sum errors or just clear the errors from the pools?

Just hit CLEAR