Unraid 6.8.3 hanging randomly forcing unclean shutdown

[Krzaku]

Unraid has been acting very unreliably, hanging to the point of me having to restart the system forcibly. When it hangs most of the time the screen is blank (screensaver?), keyboard is unresponsive and there is no possibility to connect to the server, and since unraid doesn't save its logs anywhere persistent, the logs from those crashes are lost. Just today a VM hanged (screen stuck on last frame), but the server was left (semi) running. Semi, because I still couldn't shutdown cleanly. I tried force stopping the VM, but that hanged as well. After a few minutes and refreshing the page, it appeared to have stopped it (it did not fully, the process was gone but memory usage was still high with no processes showing usage above 1%, so the VM process must have been left in some zombie like state), so I tried rebooting the server, but that hanged as well on "stopping services", at which point the server became even more unresponsive, the webui barely responded loading half of the pages, ssh was not working. This kind of issues have been going for over a year, half of parity checks in the log were because of unraid becoming unresponsive. Anyway, back to the point. This time I managed to capture some more information in the form of dmesg logs. They mention KVM and MMU but that's about as far as I can understand from them. I'm passing my GPU and NVMe drives to the VM.

 

dmesg: https://pastebin.com/YXseGUWp

vm definition: https://pastebin.com/PvZXmuCr

kernel params: 

append pcie_acs_override=downstream,multifunction iommu=pt isolcpus=1-6,9-14 initrd=/bzroot vfio-pci.ids=8086:a370,8087:0aaa,10de:1e87,10de:10f8,10de:1ad8,10de:1ad9,1033:0194,144d:a808,1987:5012

 

[Krzaku]

It happened again recently, unfortunately I have nothing more than a picture of the error:

After this happened, cpu usage was more or less frozen in the following state: