My server has recently become unstable, - as evidence by services being unreachable (including the WebUI or via SSH) requiring a reboot to get it up and running. Today I had a situation where i logged into the WebUI and saw that a has cpu core pinned at 100% and then the WebUI returned 500 error. I was able to still ssh into the box and grab a diagnostic file. But a little while later it was unreachable again (either way) and I had to reboot it. I have turned off docker services and vms with no really success. Yesterday I turned on the mirror syslogs to boot drive in hopes of capturing more logs about why the lock up was happening. Im wondering if that potentially wore out my bootdrive (which was ~6 years old?)

unraid-diagnostics.zip

I'm having similar issues and from what I can tell my logs shows something with some pcie devices and power saving. Something about the new kernel changed power saving settings.

This looks more like a platform instability, not primarily a WebGUI or QEMU problem.

Key sequence from the diagnostics:

- Machine-check hardware errors begin first.

- find then page-faults inside XFS directory handling.

- QEMU page-faults in kernel timekeeping and exits with interrupts disabled.

- A media scanner hits an invalid opcode in memory-management code.

- BANK0 CMCI storms are reported on four logical CPUs.

- The resulting zombie/blocked processes persist for almost two days.

- PHP-FPM eventually times out, producing WebGUI 504 errors. That is downstream damage, not the initial failure.

The apparent pegged core was the dead QEMU process attributed 110.9% CPU. However, the overall system was 92.4% idle, suggesting corrupted

task/accounting state rather than a normally busy VM.

Most likely cause: marginal CPU, RAM, motherboard, voltage, cooling, or power stability. Linux defines these machine checks as

CPU-detected internal hardware conditions.

Suggested next steps:

1. Preserve /var/log/mcelog before the next reboot. It existed and contained 23,739 bytes at the crash time, that's not included in the diags

2. Load BIOS defaults; disable XMP, CPU overclock/undervolt, and multicore enhancement.

3. Run boot-time Memtest for multiple passes

4. Check CPU cooling, PSU condition, and power delivery.

5. Update the BIOS, looks like there's a newer version than the one you're using.

Any advice on how to perserve /var/loog/mcelog in the event the machine becomes unresponsive again? (I don't have a monitor keyboard attached to this machine so i rely on SSH or the WebGUI).

Good call out on the BIOS its been a while since i have updated it though I could have sworn it was more recent in 2021.

You should be able to copy that over SSH, to the flash drive, for example, cp /var/log/mcelog /boot/mcelog.txt

So this just happened again. I had interacted with the WebUI and opened up a few docker containers. And when i came back to the tab, the UI was down and refreshing didn't solve it (This site can’t be reached message). i have tried to ssh into the machine but get and operation timed out when trying to connect. So i end up having to reboot the machine.

When i last checked the Log Viewer tool (yesterday) there weren't any logs in the mcelog file.

If the syslog server is running, post that log, if it's not, enable it and post it after the next crash.

Below is what i pulled from the boot drive after reboot

syslog-previous.txt

Jul 27 09:55:02 BuellVault emhttpd: read SMART /dev/sdn

Jul 27 09:55:51 BuellVault kernel: mce: [Hardware Error]: Machine check events logged

Jul 27 09:55:52 BuellVault kernel: mce: [Hardware Error]: Machine check events logged

The last couple of lines are MCEs, confirming a hardware issue, but if there's nothing on the log, it may be a trial and error situation; RAM, CPU and board would be my main suspects.