Strange issue with one of two identical VM's

[Ibanez343]

I'm trying to get a few gaming VM's working on a SuperMicro X8DAH+-F board with two 6 core Xeons.  This project is just for fun / learning (and yes I'm a LMG fanboy).  I feel like I had this setup working before without this issue but it's been several months since I had this setup going.  I have two identical VM's with a GTX 770 each (the first windows 10 VM was setup then copied for the second VM).  The VM with the GPU in the top slot (Slot 6 on this board) works just fine with no lag in games.  The other VM is considerably slower not only in games but just navigating around the OS (also if I VNC into the second VM the mouse is extremely slow).  I've tried both slot 2 and 4 for the second GPU, physically swapping the GPU's and swapping the disk images between the VM's but the issue stays with the second VM.  I've isolated all but one CPU core from each physical CPU and I've even assigned the CPU cores to each VM that would correspond to the PCIe slot with the GPU (slot 6 for CPU 2 and slot 2 for CPU1).

 

I'm just wondering if anyone has any ideas for troubleshooting this issue.  I've been working on it for several days before reaching out for help.  It's just so odd that two virtually identical VM's would operate so differently

[testdasi]

You need to check your numa topology (search for SpaceInvader One vid on that).

Then set your VM appropriately to what numa node the GPU is connected to (need to allocate core + RAM from the same numa node).

 

Reading up on the works that have been done for Threadripper would also be helpful since TR is essentially multi-CPU in a single package.

[Ibanez343]

I've spent a couple more hours troubleshooting the issue, best I can tell the PCIe devices are on the correct NUMA nodes (I followed the block diagram from the motherboard manual when I first set everything up).  One thing I do notice is that the RAM seems to be on opposite NUMA nodes.  I've tried a few different XML commands but it never changes the NUMA node.

 

<numatune>

<memory mode='strict' nodeset='1'/>

</numatune>

 

The above option does nothing and the RAM is still allocated from NUMA node 0

Edited February 21, 2020 by Ibanez343

[Ibanez343]

Update, using "preferred" instead of "strict" seemed to assign the RAM correctly.  Unfortunately it had no noticeable effect on performance.