I am not using RDP or VNC, I am using GPU passthrough
Edit : I have just done a CPU stress test and it doesn't seem to be caused by the CPU usage, I was playing a game that was minimised at the time, It wasn't a GPU intensive game and I have a GTX 970, Never had the issue before outside of the VM environment.