I did set the PCIe ACES override to disabled, multi, and both. It made no difference in stability, but every time the change was made I had to conf a new VM and point it to the existing image, because the old one didn't boot anymore. That's likely normal behavior right?
It can be that I was on the wrong track and VM itself indeed ran fine. The problem likely was a combination of two things. What I now did was:
a. I reinstalled AMD graphics drivers in the VM
b. I reset the GPU settings to factory defaults again and cleared the buffer (even tho they should have been reset at machine reboot).
Now It stress tests and runs fine so far.
Thanks for the new ideas!