Lately I've been having stability issues with unraid after upgrading to 6.12.14. Its sporadic and seems to occur every few days. Basically the entire server becomes unresponsive. VMs and dockers are killed, web access no longer works, and SSH also doesn't work.

 

However, If I hit the power button on my server, I can see in the console with a connected monitor that the server does shut down after the scripts execute. The screenshot shows the console and the shutdown process. I see some errors but I do not know what they mean. Also, it never gracefully shuts down. After 90 seconds it goes into forced shutdown.

I couldn't find anything in the logs that aligns with the timing of the issue. The most recent occurrence was around 8:00 PM ET 12/29. The photos are from a few days ago, but the behavior seems to be the same every time. Diagnostics attached.

The only thing I can think of is a failing USB drive since its very old and I ran into another issue a few weeks ago where it said there was no USB device connected but the server was still running from RAM. I haven't been able to reproduce that issue, but I havent tried transferring my license to a new device yet.

 

Anyone have any ideas?

tower-diagnostics-20241229-2013.zip

The diags posted are after a reboot, do you have the syslog from the crash? There may be ones from the unclean shutdown in the logs folder, if not, enable the syslog server and post that after a crash.

Thanks @JorgeB. Syslog server was on, just not mirrored to flash. A copy of logs are also not saved to flash in this shutdown scenario. I've attached the syslog from the system folder which resides on my cache drive. I wasn't aware that that diagnostics didn't include this.

syslog-johnsanc.zip

Seeing dome issues wit ata2, looks more like a power/connection problem, this should not crash the server, but may make it appear to be unresponsive during the errors, check/replace cables for that device.

 

Also seeing many of these:

 

Dec 29 20:08:20 Tower kernel: pcieport 0000:00:03.1: PME: Spurious native interrupt!
Dec 29 20:08:20 Tower kernel: pcieport 0000:00:03.1: PME: Spurious native interrupt!
Dec 29 20:08:20 Tower kernel: pcieport 0000:00:03.1: PME: Spurious native interrupt!
Dec 29 20:08:20 Tower kernel: pcieport 0000:00:03.1: PME: Spurious native interrupt!

 

But not sure if they are serious or not

Yeah I suspected the ATA2 could be an issue so I moved that drive to a totally different controller. The latest crash yesterday was after that move so I suspect its unrelated.

 

I have no idea what the "Spurious native interrupt" errors are. Based on the syslog i provided those are present all the way back in August. I never had this crash issue until after upgrading to 6.12.14 which was end of November.

 

Is there anything else I can do to enable better logging when the issue happens again? I saw literally nothing captured around the time of the crash. Does mirroring to flash include more info than if I were just writing to `system` on cache drive?

 

Some issues don't leave anything logged, like some hardware or docker related issues for example, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one, including the docker containers. 

Darn. Well, perhaps an enhancement request would be to save diagnostics on the "forced shutdown" scenario like my photos above. Don't know why its only saved on clean shutdown. Perhaps there are clues in other logs outside of syslog.

Edited December 30, 20241 yr by johnsanc

13 hours ago, johnsanc said:

Perhaps there are clues in other logs outside of syslog.

Syslog is where any relevant info would be logged.

So one thing I've noticed is this most commonly seems to happen in the middle of the night, sometime after the trim jobs happen at 5:30 AM. No idea if its related, but I may try changing the schedule to see if the crashes start to align with a different time.

Quick update... I tried replacing my key to a new flash drive, and in the process of the key validation using the web replacement method it failed and threw an error saying there was an issue communicating with the flash device. I manually replaced the key and it booted fine... but then after a few minutes the flash was completely dropped again. I tried one more time in safe mode, and the same thing happened. I tried switching USB ports and so far so good... Maybe I have a USB port that has gone bad. I'll monitor to see if that fixed the issue.

Edited January 1, 20251 yr by johnsanc