Hi all,

 

I’m running into a frustrating issue with my Unraid box and hoping for some advice or insight.

 

I am consistently getting the following kernel errors when the system is under heavy load (in particular, running Ollama and an AI model seems to trigger it):

 

nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 
nvme nvme0: Does your device have a faulty power saving mode enabled? 
nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug I/O error, dev nvme0n1, sector XXXX op READ/WRITE XFS (nvme0n1p1): log I/O error -5 XFS (nvme0n1p1): Filesystem has been shut down due to log error (0x2).

 

Details logs:

May  2 10:30:15 Tower kernel: nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
May  2 10:30:15 Tower kernel: nvme nvme0: Does your device have a faulty power saving mode enabled?
May  2 10:30:15 Tower kernel: nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug
May  2 10:30:15 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 2932062696, 8 blocks, I/O Error (sct 0x3 / sc 0x71)
May  2 10:30:15 Tower kernel: nvme0n1: I/O Cmd(0x2) @ LBA 1953422024, 32 blocks, I/O Error (sct 0x3 / sc 0x71)
May  2 10:30:15 Tower kernel: I/O error, dev nvme0n1, sector 2932062696 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
May  2 10:30:15 Tower kernel: I/O error, dev nvme0n1, sector 1953422024 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
May  2 10:30:15 Tower kernel: nvme 0000:01:00.0: enabling device (0000 -> 0002)
May  2 10:30:15 Tower kernel: nvme nvme0: Disabling device after reset failure: -19
May  2 10:30:15 Tower kernel: I/O error, dev nvme0n1, sector 1267670888 op 0x1:(WRITE) flags 0x100000 phys_seg 4 prio class 2
May  2 10:30:15 Tower kernel: I/O error, dev nvme0n1, sector 1955369608 op 0x1:(WRITE) flags 0x29800 phys_seg 1 prio class 2
May  2 10:30:15 Tower kernel: I/O error, dev nvme0n1, sector 1227912264 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
May  2 10:30:15 Tower kernel: I/O error, dev nvme0n1, sector 874028944 op 0x1:(WRITE) flags 0x800 phys_seg 3 prio class 2
May  2 10:30:15 Tower kernel: I/O error, dev nvme0n1, sector 715024 op 0x1:(WRITE) flags 0x0 phys_seg 2 prio class 2
May  2 10:30:15 Tower kernel: XFS (nvme0n1p1): log I/O error -5
May  2 10:30:15 Tower kernel: nvme0n1p1: writeback error on inode 1342730911, offset 23631253504, sector 1267670888
May  2 10:30:15 Tower kernel: nvme0n1p1: writeback error on inode 1342730911, offset 3274838016, sector 1227912264
May  2 10:30:15 Tower kernel: nvme0n1p1: writeback error on inode 408041943, offset 0, sector 371701112
May  2 10:30:15 Tower kernel: nvme0n1p1: writeback error on inode 562218, offset 0, sector 715024
May  2 10:30:15 Tower kernel: nvme0n1p1: writeback error on inode 957483739, offset 665518080, sector 874028944
May  2 10:30:15 Tower kernel: nvme0n1p1: writeback error on inode 1342730911, offset 3281526784, sector 1227925328
May  2 10:30:15 Tower kernel: nvme0n1p1: writeback error on inode 671088792, offset 831488, sector 610523984
May  2 10:30:15 Tower kernel: nvme0n1p1: writeback error on inode 1342730911, offset 3444932608, sector 1228244480
May  2 10:30:15 Tower kernel: nvme0n1p1: writeback error on inode 4089207642, offset 618496, sector 3725472440
May  2 10:30:15 Tower kernel: XFS (nvme0n1p1): Filesystem has been shut down due to log error (0x2).
May  2 10:30:15 Tower kernel: nvme0n1p1: writeback error on inode 1342730911, offset 3940171776, sector 1229211744

 

This causes the filesystem to unmount and require intervention.

I have already tried the suggested kernel parameters to disable power saving modes:

nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off

 

Additionally, I have changed both the motherboard and the NVMe drive itself, yet the issue still persists in the same way.

 

The only thing that seems to correlate with the failures is heavy load, particularly when Ollama and AI models are running. However, nothing in the Ollama logs looks unusual (just normal request processing, then unexpected EOF right when the crash happens).

 

time=2025-05-02T00:59:47.165Z level=DEBUG source=sched.go:409 msg="context for request finished" time=2025-05-02T00:59:47.165Z level=DEBUG source=sched.go:341 msg="runner with non-zero duration has gone idle, adding timer" ... unexpected EOF

 

Summary of what I’ve tried so far:

• Changed motherboard
• Changed NVMe drive
• Upgraded my BIOS
• Applied nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off kernel parameters
• Verified no obvious errors from Ollama side besides normal shutdown / EOF after the drive failure

 

At this point, I’m wondering if:

There could still be some PCIe power management issue at the BIOS level? Where do I look for this?

There is a compatibility issue with Unraid + my hardware + heavy I/O + AI workloads?

Something else (PSU/power stability, memory/cpu, or NVMe firmware bug)?

Has anyone seen similar NVMe controller resets triggered under load despite having disabled the power saving options?

 

Any advice would be greatly appreciated — I’m running out of ideas.

 

What other logs can I provide?

Edited May 2, 20251 yr by AndyLively

If possible, I would recommend trying a different brand/model NVMe device.

Thanks @JorgeB - I have tried that too. I had a samsung in there and now I have a crucial NVME. Not too keen to buy another one.

If the same thing happens with different devices and those kernel boot options, then I would recommend using a different brand/model board

So there is nothing I can do from a software point of view? Do you think its just hardware?

Based on what I see, it looks like some boards and NVMe devices just don't work correctly with Linux, sometimes the kernel boot options help, other times a BIOS update or kernel update helps, other times, only changing part of the hardware.

Should a M.2 sata add in board potentially fix the issue? Less expensive than a new motherboard and related components.

It may be worth a try.

NVME gets hot, very hot under hevy load. Did you try to cool them down / monitor their temperature?

I used a m.2 PCIE expansion board - and it appears to be working fine now without issue. Will see if it stays stable over time.

 

I was checking the temp in the past and it wasnt getting very high.

Hey @AndyLively Is yours still stable? What pcie board did you go with?