Hi all,
I’m running into a frustrating issue with my Unraid box and hoping for some advice or insight.
I am consistently getting the following kernel errors when the system is under heavy load (in particular, running Ollama and an AI model seems to trigger it):
nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
nvme nvme0: Does your device have a faulty power saving mode enabled?
nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug I/O error, dev nvme0n1, sector XXXX op READ/WRITE XFS (nvme0n1p1): log I/O error -5 XFS (nvme0n1p1): Filesystem has been shut down due to log error (0x2).
Details logs:
May 2 10:30:15 Tower kernel: nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
May 2 10:30:15 Tower kernel: nvme nvme0: Does your device have a faulty power saving mode enabled?
May 2 10:30:15 Tower kernel: nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug
May 2 10:30:15 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 2932062696, 8 blocks, I/O Error (sct 0x3 / sc 0x71)
May 2 10:30:15 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 1953422024, 32 blocks, I/O Error (sct 0x3 / sc 0x71)
May 2 10:30:15 Tower kernel: I/O error, dev nvme0n1, sector 2932062696 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
May 2 10:30:15 Tower kernel: I/O error, dev nvme0n1, sector 1953422024 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
May 2 10:30:15 Tower kernel: nvme 0000:01:00.0: enabling device (0000 -> 0002)
May 2 10:30:15 Tower kernel: nvme nvme0: Disabling device after reset failure: -19
May 2 10:30:15 Tower kernel: I/O error, dev nvme0n1, sector 1267670888 op 0x1:(WRITE) flags 0x100000 phys_seg 4 prio class 2
May 2 10:30:15 Tower kernel: I/O error, dev nvme0n1, sector 1955369608 op 0x1:(WRITE) flags 0x29800 phys_seg 1 prio class 2
May 2 10:30:15 Tower kernel: I/O error, dev nvme0n1, sector 1227912264 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
May 2 10:30:15 Tower kernel: I/O error, dev nvme0n1, sector 874028944 op 0x1:(WRITE) flags 0x800 phys_seg 3 prio class 2
May 2 10:30:15 Tower kernel: I/O error, dev nvme0n1, sector 715024 op 0x1:(WRITE) flags 0x0 phys_seg 2 prio class 2
May 2 10:30:15 Tower kernel: XFS (nvme0n1p1): log I/O error -5
May 2 10:30:15 Tower kernel: nvme0n1p1: writeback error on inode 1342730911, offset 23631253504, sector 1267670888
May 2 10:30:15 Tower kernel: nvme0n1p1: writeback error on inode 1342730911, offset 3274838016, sector 1227912264
May 2 10:30:15 Tower kernel: nvme0n1p1: writeback error on inode 408041943, offset 0, sector 371701112
May 2 10:30:15 Tower kernel: nvme0n1p1: writeback error on inode 562218, offset 0, sector 715024
May 2 10:30:15 Tower kernel: nvme0n1p1: writeback error on inode 957483739, offset 665518080, sector 874028944
May 2 10:30:15 Tower kernel: nvme0n1p1: writeback error on inode 1342730911, offset 3281526784, sector 1227925328
May 2 10:30:15 Tower kernel: nvme0n1p1: writeback error on inode 671088792, offset 831488, sector 610523984
May 2 10:30:15 Tower kernel: nvme0n1p1: writeback error on inode 1342730911, offset 3444932608, sector 1228244480
May 2 10:30:15 Tower kernel: nvme0n1p1: writeback error on inode 4089207642, offset 618496, sector 3725472440
May 2 10:30:15 Tower kernel: XFS (nvme0n1p1): Filesystem has been shut down due to log error (0x2).
May 2 10:30:15 Tower kernel: nvme0n1p1: writeback error on inode 1342730911, offset 3940171776, sector 1229211744
This causes the filesystem to unmount and require intervention.
I have already tried the suggested kernel parameters to disable power saving modes:
nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off
Additionally, I have changed both the motherboard and the NVMe drive itself, yet the issue still persists in the same way.
The only thing that seems to correlate with the failures is heavy load, particularly when Ollama and AI models are running. However, nothing in the Ollama logs looks unusual (just normal request processing, then unexpected EOF right when the crash happens).
time=2025-05-02T00:59:47.165Z level=DEBUG source=sched.go:409 msg="context for request finished" time=2025-05-02T00:59:47.165Z level=DEBUG source=sched.go:341 msg="runner with non-zero duration has gone idle, adding timer" ... unexpected EOF
Summary of what I’ve tried so far:
Changed motherboard
Changed NVMe drive
Upgraded my BIOS
Applied nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off kernel parameters
Verified no obvious errors from Ollama side besides normal shutdown / EOF after the drive failure
At this point, I’m wondering if:
There could still be some PCIe power management issue at the BIOS level? Where do I look for this?
There is a compatibility issue with Unraid + my hardware + heavy I/O + AI workloads?
Something else (PSU/power stability, memory/cpu, or NVMe firmware bug)?
Has anyone seen similar NVMe controller resets triggered under load despite having disabled the power saving options?
Any advice would be greatly appreciated — I’m running out of ideas.
What other logs can I provide?