Debugging system stability issues

[harperhendee]

I have been dealing with some system stability issues since my first installation 6 months ago.  They fall into a couple of different buckets:

Bucket 1 - Windows 10 VM fails to launch

1a) Hangs with no display from VM

1b) Hangs with VM static display of windows logo, but no spinning dots

1c) Goes to Windows recovery

1d) Kill and relaunch fixes it about 50% of the time

 

Bucket 2 - Windows 10 VM hangs during usage

2a) VM becomes unresponsive when USB devices plugged/unplugged

2b) Other VMs continue to function

 

Bucket 3 - Unraid hard hang

3a) Sometimes bucket 2 problems also hang the Unraid server.  

3b) Screen is frozen for Unraid and VMs

3c) Server will not respond to SSH, ping, or short HW power button press

 

Debugging these things is difficult:

 

Bucket 1 - There isn't any debug trail that I can find.  I don't see any errors associated with the failure in syslog or virtlogd.  I can usually spot that the issue has occurred based on CPU usage.  Normal behavior is all CPUs at 100% for a time, then one CPU at 80-100% while others are idling from 5-50%.  Failure modes are all CPUs 100% or 1 CPU at 100% and all others at 0%.

 

Bucket 2 - I occasionally get a "fatal error" message in VM log.  Sometimes nothing.  When I try to restart the VM, I usually get some execution error pop up.  All CPUs are at 0%.

 

Bucket 3 - I have no idea how to debug this.  Once it hangs, I reboot the system and lose my logs from previous run.  I was thinking that I might use a second computer to ssh and run "tail -f" on the syslogd file.  

 

Are there other debug messages I can get to?  

I read about MCE logs as a possible debug path.  I'm not sure if those are already going to show up in syslogd or my remote SSH console.  

What low level information is exposed with unraid?  

Is there a HW observation point where I could get lower level debug information over and above what unraid supplies?

 

HW diagram of system is attached.

 

--Brad