I have been struggling for this issue for months. My server crashes every few days. Last time this happened, I REPLACED my NVME Cache drive (w/ a brand new Samsung drive) thinking the issue was a bad drive. After a week or so, the problems started again. I am pasting the diagnostics zip file right after the latest crash. I ran the diagnostics thru ChatGPT, and it reported the issue:

the ZFS cache pool (‘cache’) experienced repeated I/O errors on the NVMe device and the pool was repeatedly suspended. When ZFS hits uncorrectable I/O (EIO) on a vdev that’s part of a pool the pool will be suspended to protect data — that will cause services to hang, errors in file access, and generally what looks like “crashes”.

My cache drive is the only NVME drive in my server. I have two other cache pools that are SSD drives. Back with the original drive, I tried the NVME slot in my motherboard, and I also tried an PCIe NVME adapter card. Thinking was that the slot was bad in the motherboard. What could be causing these ZFS i/o errors? Should I rebuild my cache drive with a different file system? XFS?

Just now I ran a scrub on the drive and it did not find any errors.

Every night I run the Backup/Restore Appdata to back up my appdata folder to a non array spinner drive. This usually takes a good 4+ hours. I imagine a good deal of reading during the process.

What should I do please...?

Thank you.

H.

tower-diagnostics-20251006-1119.zip

NVMe device is dropping offline:

Oct 6 06:25:29 Tower kernel: nvme nvme0: Does your device have a faulty power saving mode enabled?

Oct 6 06:25:29 Tower kernel: nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug

Oct 6 06:25:29 Tower kernel: nvme 0000:86:00.0: enabling device (0000 -> 0002)

Oct 6 06:25:29 Tower kernel: nvme nvme0: Disabling device after reset failure: -19

Try adding those kernel options to syslinux.cfg

Thank you @JorgeB

Just to be clear, my syslinux should read like this:

default menu.c32

menu title Lime Technology, Inc.

prompt 0

timeout 50

label Unraid OS

menu default

kernel /bzimage

append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off

label Unraid OS GUI Mode

kernel /bzimage

append initrd=/bzroot,/bzroot-gui

label Unraid OS Safe Mode (no plugins, no GUI)

kernel /bzimage

append initrd=/bzroot unraidsafemode

label Unraid OS GUI Safe Mode (no plugins)

kernel /bzimage

append initrd=/bzroot,/bzroot-gui unraidsafemode

label Memtest86+

kernel /memtest

Yep, then reboot

2 hours ago, JorgeB said:

Yep, then reboot

Thank you... Fingers crossed.

Hi @JorgeB

Unfortunately I am getting new crashes after a couple days after I added append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off to my syslinux.cfg. I am attaching the latest diagnostics.

Any ideas? I have tried the NVME drive in the motherboard directly, and on a PCIe adapter card. I also bought a new NVME drive. The issues persists.

Thank you.

H.

.tower-diagnostics-20251008-0754.zip

You can try updating to 7.2.0-rc.1, there's a kernel change that's supposed to help with dropping NVMe devices.