Unraid Support,

My unraid server has crashed several times in the last 24 hours. I’ve not made any recent hardware changes & the server has been very stable for the last 6-12 months. I've pasted these logs into ChatGPT & it stated that there might be an NVMe PCIe Bus Error. Is this correct? I've no idea how to 'read' or 'review' logs so I don't have the first clue what I'm looking for here or what the root cauyse might be.

Does this mean that my m.2 cache drive may be failing? Would that cause the system to crash/ power down multiple times? If anyone is able to help me with a diagnosis/ help me narrow down this issue to a likely culprit I'd greatly apprecaite it. I can add the current logs (& may be able to offer additional logs if it crashes again (as I've temporarily turned on continuous logging). 

Any advise/ support would be greatly appreciated. 

Thanks,

syslog-192.168.50.17.log

Attach Diagnostics ZIP to your NEXT post in this thread.

Hi @trurl

I've tried several diagnostic steps with Unraid Support so far but I've not had any luck identifying the root cause of my sustem crashing (after a long period of sustained uptime). I've attached 4 x diagnostics logs files. The most recent of these is named 'flyingdutchman-diagnostics-20251115-1933.'

The support steps I have so far completed with Unraid Support are as follows:

• Unraid supports initial response was as follows: 'a PCIe error doesn't typically make the server crash, but in most cases adding the kernel option mentioned in the link below resolves the issue: https://forums.unraid.net/topic/118286-nvme-drives-throwing-errors-filling-logs-instantly-how-to-resolve/?do=findComment&comment=1165009. I do see what looks like a container constantly restarting, and this can cause issues. Please go to the Docker page and check the containers uptimes to see if you can find the problem container; it would be the one with just a few seconds/minutes of uptime.

  In response to this I added the code: 'append initrd=/bzroot pcie_aspm=off' to the system config & performed a system restart. I also checked for any dockers that were constantly restarting & confirmed that I don't have any docker containers with that behaviour; as they all had consistent uptime. After a very short period of time the system crashed. I was able to get diagnostic logs before the system crashed.

• Having informed Unraid support of this they've further advised: 'the new diags are just after a reboot, but everything looks normal for now, strange that no container shows with a recent uptime, the previous diags suggested that. Please try running the server for some time with the Docker service disabled, and see if it still crashes, if it doesn't, re-enable half the containers and retest, then the other half, and keep drilling down to see if a container is causing the issues, because other than that, there isn't anything abnormal logged. If it still crashes without Docker running, please send us a new perisitent syslog.'

  I've tried running the server with the Docker service completely disabled. The system lasted less than 24 hours & then crashed again. I've attached a full set of diagnostic logs after this crash 'flyingdutchman-diagnostics-20251115-1933' & I'm hoping that someone can help me diagnose what might be causing this...

  Are there any other software checks that I can carry out to identify if there may be a software issue here. Is it even possible to set up unraid from scratch, using a new/ clean copy of unraid on a Flash Drive & remove all existing appdata? I don't know if its even possible to do that, to then connect to an existing HDD array. Is this a hardare failure & I need to start swapping out individual components, such as the PSU/ Cache Drive to identify an intermittent hardware fault... I'm a bit at my wits end here as I've no idea where to start or what's even wrong...

flyingdutchman-diagnostics-20251115-1933.zip flyingdutchman-diagnostics-20251114-1307.zip flyingdutchman-diagnostics-20251114-1317.zip flyingdutchman-diagnostics-20251114-1331.zip

I have also had a 'Warning - API is offline' error come up briefly once or twice previously; but its since disappeared. I've no idea if this is even relevant or not but thought I'd mention it.

13 hours ago, MatthewT said:

The system lasted less than 24 hours & then crashed again

That and the fact there's nothing relevant logged suggests a hardware issue, since you have multiple RAM sticks, try using the server with just one pair, if the same try with the other one, that will basically rule out bad RAM.

So I thought I might have a dodgy SSD (Cache Drive), so I reformated that to ZFS, I also uninstalled nzbget (binhex) & since I've done that I've not had it crash once (which leads me to suspect it was the NZBGET container rather than a hardware fault).