Stokkes

Alright what an ordeal (esp as I don't have a monitor, but managed to use a USB HDMI Capture card with my iPad and Orion app). 
 
Putting solution here in case someone comes across this.
 
Issue is related to how Supermicro boards / IPMI works with an external GPU. Even though there are settings set, if you have any monitor (or in my case a dummy HDMI adapter) plugged in, it will switch to use that, which effectively means IPMI becomes useless. 
 
I had put the dummy adapter for my RTX 4060 in Windows to allow custom resolutions, I also wanted the ability to use VNC within Unraid in case the VM.
 
Anyway, with the Dummy plugged in and booting, it would switch to the RTX as the default GPU and kill the IPMI. 
 
Solution: stop using the Dummy adapter and install the custom virtualdisplay driver: https://github.com/VirtualDisplay/Virtual-Display-Driver
 
Now without the dummy plug, IPMI works and the Gaming VM works with the built-in VNC for accessing the VM. 
 
Still annoying it appears by the manual you can get Supermicro boards to not detect PCIE graphics adapters on boot, but on my X12SCA-F board this doesn't appear to do anything as with the dummy plug it still switches to this display for the BIOS, etc even with that option to skip enabled. 
 
Bottom line: if you have a supermicro board with IPMI and use a Gaming VM with a passthrough GPU, you basically can't have anything plugged in if you're using this headless. I am using this VM for Sunshine/Moonlight streaming to my Apple TVs, iPhone, etc. but because I have SUnshine switch to a custom resolution, enable HDR, etc depending on the device connected, I needed the dummy plug.. 
 
What a mess, but figured it out if anyone ever runs into this.