Hi,

 

I have recently built a new system that should be able to handle two gaming VMs. I have a powerful xeon cpu, 32GB of DDR4 memory, SSDs outside the array and two GTX 960 for passthrough. I'm running win 10 on both VMs and everything is working fine until I start the second VM, then the first VM starts to lagg and the performance while playing games are really bad. Have anyone had this issue? I have tried different combinations with BIOS and machine types, but nothing seems to fix the problem.. I haven't really been able to test OMVF because after reboot I end up in Shell> and are not able to boot from disk..

 

Any help are very appreciated

 

Thanks

Have you tried pinning specific cores in the setup of each VM?  Doing so and selecting different cores for the two VMs should help reduce any contention between the VMs for the same CPU resources.

Yes, since the cpu have 8 cores, they each have 4 different cores (8 logical) dedicated. When I'm running the two VMs, the cpu almost nerver goes over 60% so I don't think it should be a cpu problem. Maybe it's a bad combination of motherboard and gpu.. I just thought it would be interesting to see if anyone experienced similar problems. The setup is done like Linustechtips with the exception that I'm currently running the SSDs completly outside the array. I have tried the cache disk also but there was no difference in performance.

Yes, since the cpu have 8 cores, they each have 4 different cores (8 logical) dedicated. When I'm running the two VMs, the cpu almost nerver goes over 60% so I don't think it should be a cpu problem. Maybe it's a bad combination of motherboard and gpu.. I just thought it would be interesting to see if anyone experienced similar problems. The setup is done like Linustechtips with the exception that I'm currently running the SSDs completly outside the array. I have tried the cache disk also but there was no difference in performance.

 

Can you share the XML for your VMs please?  I will propose a change to see if this improves things.

Sure, here they are:

 

VM1

 

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

  <name>Dator1</name>

  <uuid>3a3ff8b2-7e4f-ab96-2b4d-8199cd592784</uuid>

  <metadata>

    <vmtemplate name="Custom" icon="windows.png" os="windows"/>

  </metadata>

  <memory unit='KiB'>12582912</memory>

  <currentMemory unit='KiB'>12582912</currentMemory>

  <memoryBacking>

    <nosharepages/>

    <locked/>

  </memoryBacking>

  <vcpu placement='static'>8</vcpu>

  <cputune>

    <vcpupin vcpu='0' cpuset='0'/>

    <vcpupin vcpu='1' cpuset='1'/>

    <vcpupin vcpu='2' cpuset='2'/>

    <vcpupin vcpu='3' cpuset='3'/>

    <vcpupin vcpu='4' cpuset='4'/>

    <vcpupin vcpu='5' cpuset='5'/>

    <vcpupin vcpu='6' cpuset='6'/>

    <vcpupin vcpu='7' cpuset='7'/>

  </cputune>

  <os>

    <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type>

  </os>

  <features>

    <acpi/>

    <apic/>

  </features>

  <cpu mode='host-passthrough'>

    <topology sockets='1' cores='8' threads='1'/>

  </cpu>

  <clock offset='localtime'>

    <timer name='rtc' tickpolicy='catchup'/>

    <timer name='pit' tickpolicy='delay'/>

    <timer name='hpet' present='no'/>

  </clock>

  <on_poweroff>destroy</on_poweroff>

  <on_reboot>restart</on_reboot>

  <on_crash>restart</on_crash>

  <devices>

    <emulator>/usr/bin/qemu-system-x86_64</emulator>

    <disk type='file' device='disk'>

      <driver name='qemu' type='raw' cache='writeback'/>

      <source file='/mnt/disks/Disk1/Dator1/vdisk1.img'/>

      <target dev='hdc' bus='virtio'/>

      <boot order='1'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>

    </disk>

    <controller type='usb' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>

    </controller>

    <controller type='pci' index='0' model='pci-root'/>

    <controller type='ide' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>

    </controller>

    <controller type='virtio-serial' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

    </controller>

    <interface type='bridge'>

      <mac address='52:54:00:9d:3e:fb'/>

      <source bridge='br0'/>

      <model type='virtio'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>

    </interface>

    <serial type='pty'>

      <target port='0'/>

    </serial>

    <console type='pty'>

      <target type='serial' port='0'/>

    </console>

    <channel type='unix'>

      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/Dator1.org.qemu.guest_agent.0'/>

      <target type='virtio' name='org.qemu.guest_agent.0'/>

      <address type='virtio-serial' controller='0' bus='0' port='1'/>

    </channel>

    <hostdev mode='subsystem' type='usb' managed='yes'>

      <source>

        <vendor id='0x046d'/>

        <product id='0xc52e'/>

      </source>

    </hostdev>

    <memballoon model='virtio'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>

    </memballoon>

  </devices>

  <qemu:commandline>

    <qemu:arg value='-device'/>

    <qemu:arg value='ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/>

    <qemu:arg value='-device'/>

    <qemu:arg value='vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>

    <qemu:arg value='-device'/>

    <qemu:arg value='vfio-pci,host=03:00.1,bus=root.1,addr=00.1'/>

  </qemu:commandline>

</domain>

 

 

VM2

 

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

  <name>Dator2</name>

  <uuid>133f00cd-f4db-88f3-b002-3d4b967b5526</uuid>

  <metadata>

    <vmtemplate name="Custom" icon="windows.png" os="windows"/>

  </metadata>

  <memory unit='KiB'>12582912</memory>

  <currentMemory unit='KiB'>12582912</currentMemory>

  <memoryBacking>

    <nosharepages/>

    <locked/>

  </memoryBacking>

  <vcpu placement='static'>8</vcpu>

  <cputune>

    <vcpupin vcpu='0' cpuset='8'/>

    <vcpupin vcpu='1' cpuset='9'/>

    <vcpupin vcpu='2' cpuset='10'/>

    <vcpupin vcpu='3' cpuset='11'/>

    <vcpupin vcpu='4' cpuset='12'/>

    <vcpupin vcpu='5' cpuset='13'/>

    <vcpupin vcpu='6' cpuset='14'/>

    <vcpupin vcpu='7' cpuset='15'/>

  </cputune>

  <os>

    <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type>

  </os>

  <features>

    <acpi/>

    <apic/>

  </features>

  <cpu mode='host-passthrough'>

    <topology sockets='1' cores='8' threads='1'/>

  </cpu>

  <clock offset='localtime'>

    <timer name='rtc' tickpolicy='catchup'/>

    <timer name='pit' tickpolicy='delay'/>

    <timer name='hpet' present='no'/>

  </clock>

  <on_poweroff>destroy</on_poweroff>

  <on_reboot>restart</on_reboot>

  <on_crash>restart</on_crash>

  <devices>

    <emulator>/usr/bin/qemu-system-x86_64</emulator>

    <disk type='file' device='disk'>

      <driver name='qemu' type='raw' cache='writeback'/>

      <source file='/mnt/disks/Disk2/Dator2/vdisk1.img'/>

      <target dev='hdc' bus='virtio'/>

      <boot order='1'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>

    </disk>

    <controller type='usb' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>

    </controller>

    <controller type='pci' index='0' model='pci-root'/>

    <controller type='ide' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>

    </controller>

    <controller type='virtio-serial' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

    </controller>

    <interface type='bridge'>

      <mac address='52:54:00:07:40:e0'/>

      <source bridge='br0'/>

      <model type='virtio'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>

    </interface>

    <serial type='pty'>

      <target port='0'/>

    </serial>

    <console type='pty'>

      <target type='serial' port='0'/>

    </console>

    <channel type='unix'>

      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/Dator2.org.qemu.guest_agent.0'/>

      <target type='virtio' name='org.qemu.guest_agent.0'/>

      <address type='virtio-serial' controller='0' bus='0' port='1'/>

    </channel>

    <hostdev mode='subsystem' type='usb' managed='yes'>

      <source>

        <vendor id='0x046d'/>

        <product id='0xc01d'/>

      </source>

    </hostdev>

    <hostdev mode='subsystem' type='usb' managed='yes'>

      <source>

        <vendor id='0x045e'/>

        <product id='0x0714'/>

      </source>

    </hostdev>

    <hostdev mode='subsystem' type='usb' managed='yes'>

      <source>

        <vendor id='0x045e'/>

        <product id='0x0717'/>

      </source>

    </hostdev>

    <memballoon model='virtio'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>

    </memballoon>

  </devices>

  <qemu:commandline>

    <qemu:arg value='-device'/>

    <qemu:arg value='ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/>

    <qemu:arg value='-device'/>

    <qemu:arg value='vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>

    <qemu:arg value='-device'/>

    <qemu:arg value='vfio-pci,host=02:00.1,bus=root.1,addr=00.1'/>

  </qemu:commandline>

</domain>

 

 

 

How many cores are you dedicating to each VM? I would dedicate no more then two cores per VM, anymore and it could lead to poor performance, not necessarily what you are describing though.

I'm dedicating 4 cores (8 logical cores) to each VM. I will try to reduce the number of cores assigned to see if it will have any impact to the performance. Why do you think that more cores could cause the problem?

You likely want to keep 1 or 2  cores available for the base OS, unraid.

Its a known problem in virtualization, its called over provisioning, there is a sweet spot for provisioning CPU cores to a VM, too many and it works against you resulting in poor performance, the rule is to start off with one and add more as you need them, but unless the VM is doing CPU intensive stuff, two should be more than enough, one might even be fine. Also the other poster is correct, you have to leave some CPU cores for the base OS UnRaid, in your case you were dedicating all the cores between the two VM's, it kind of makes sense that when you fired up the second VM that was using up the remaining four cores of the CPU that everything slowed down, UnRaid had no CPU cores for itself.

Ok. I just tried to lower the amount of cores from 4 to 2 for each VM, but unfortunately that didn't make any difference. It's a very strange problem. If i boot both VMs at the same time and then start a game on VM1, everything works great. If I then close the game on VM1 and start the same game on VM2, performance is not as good as on VM1. The scenario is the same if i start the game on VM2 and then close it and start it on VM1 so It doesn't seem to be related to a specific VM. If I only boot VM1 and start the game on that one everything works great until I fire up VM2, then during the boot of VM2, the game on VM1 starts to lag and stays that way until i reboot the VMs.

Perhaps I should also tell you that I'm only using unRAID for gaming virtualization. I don't use it for storage or docker at the moment so the cpu utilization is 0% when the gaming VMs are powered off.

I don't think that makes a difference. Unfortunately I am not familiar with setting VM's up in UnRaid however it sounds to me like there is either a setting wrong somewhere or the hardware is misconfigured.

So you have two SSD drives, and each VM has one dedicated to it for storage? Have you brought up task manager within the Windows 10 VM's to see what spikes if anything in the first VM when you bring up the second VM?

Correct, one SSD for each VM. I just tried to do a test and look in the task manager of VM1 while playing a game and VM2 boots up. There was nothing in task manager that showed a problem/spike; in other words I couldn't tell just by looking in the task manager that the other VM was booting up.

 

Thanks for all the help and ideas

I have this set up working for me. I'm using the following

CPU: 5960X at 4.4GHz

MB: Asrock x99 ws-e

Memory: 32GB Corsair LPX

HDD Array: 3 x 4TB WD Reds

 

I had 2 gaming rigs, with the below specs running off this

 

VM1:

7 Cores (1 - 7)

100GB OS image

1TB storage image

12GB Memory

GTX 750TI

 

 

VM2:

7 Cores (8 - 14)

100GB OS image

1TB storage image

16GB Memory

GTX 780

 

These were storing steam games and the OS's on the slow mechanical drives and i was still able to play assassins creed syndicate on on vm2 and fallout 4 on vm1 at the same time.

In my setup i allowed 2 cores for unraid and 4gb of memory left over

 

Since then, i have now installed 3 dockers (nginx, mysql, emby server) and it is still working fine.

 

As both machines are using different cores, they won't shows up the booting of 1 from another in the task manager as they only see what they are allocated.

 

I would honestly mount the SSDs within the cache array, i will go through the exact share setup process with you if needs be as Linus's video was very quick (it's what i used to set up my system). One thing that may work is to do the following on both vm XML's

 

Edit them and where yours says

<driver name='qemu' type='raw' cache='writeback'/>

 

Replace this with the following

<driver name='qemu' type='raw' cache='none' io='threads'/>

 

This may improve your performance

 

Regards,

Jamie

jonp will reply with a detailed & technically correct explanation, but basically assigning CPU cores with the VM manager doesn't prevent unraid or even other VM's or Dockers from using those cores. On my system, with 4 physical cores, 2 assigned to a single VM, I still experienced performance issues. I tried documenting the issues I observed by recording DPC latency - and as soon as I experienced FPS drops and audio problems, DPC latency was very high.

 

There is a syslinux.cfg change called isolcpus that ended up working for me. You absolutely cannot assign every core to VM's... unraid needs something to run on. In my case I used isolcpus to pin 2 cores to the VM and left 2 cores for unraid & docker. Those isolcpus no longer appear in the unraid webui and cannot be interrupted by (or assigned to) other VM's.

Also, regarding OVFM rebooting to shell... type

Fs0:

cd efi

cd boot

Bootx64.efi

 

It's annoying, but it works.

jonp will reply with a detailed & technically correct explanation, but basically assigning CPU cores with the VM manager doesn't prevent unraid or even other VM's or Dockers from using those cores. On my system, with 4 physical cores, 2 assigned to a single VM, I still experienced performance issues. I tried documenting the issues I observed by recording DPC latency - and as soon as I experienced FPS drops and audio problems, DPC latency was very high.

 

There is a syslinux.cfg change called isolcpus that ended up working for me. You absolutely cannot assign every core to VM's... unraid needs something to run on. In my case I used isolcpus to pin 2 cores to the VM and left 2 cores for unraid & docker. Those isolcpus no longer appear in the unraid webui and cannot be interrupted by (or assigned to) other VM's.

 

That's interesting, I assigned all four cores to my Windows VM and haven't noticed much of a performance drop. That said, it's worth noting that while playing, it has otherwise idled (I am alone and it's not performing any other tasks than what I assign to it).

That's interesting, I assigned all four cores to my Windows VM and haven't noticed much of a performance drop. That said, it's worth noting that while playing, it has otherwise idled (I am alone and it's not performing any other tasks than what I assign to it).

Whether I assigned all 4 cores to the VM, or just 2 (0+1 or 2+3) made no difference. Simply playing a video on youtube caused audio issues even though both pcie devices were set to MSI (Nvidia 770 - the hdmi audio is a separate device even though it's on the same card). USB audio still isn't fixed, but I've got a card on order to pass through to fix that.

 

For general use isolcpus isn't recommended. It takes away the flexibility of hardware sharing and the isolated cpus are fixed to a single VM. But in my case, given nothing else worked, it definitely did the trick.

For a short time, I had serious issues as well. Even opening the start menu in Windows would lag horribly, and playing anything was out of the question. This resolved itself when I played around with the settings, it would appear that the issues I had were caused by allocating cores 1-3, and it worked as expected with cores 0-2 or 0-3, but in the meantime I also installed QEMU-drivers, so I'm honestly not sure what did the trick.

 

But obviously, you are stilling having issues despite this.

Thanks for all your input, but unfortunately I still have the same problems.

 

@bigjme: I have tried to use the cache disk pool, this was actually the way I started since I followed the guide from Linustechtips. The problem was exactly the same then so i decided to try the "unassigned devices" plugin. This way, each VM could get one SSD and I thought that suited me better compared to the RAID1 you get from the cache disk pool. I tried changing the XML as you proposed, but it didn't help.

 

@goober07: Your issues with DPC latency sounds interesting. Right now i feel very frustrated and just want to get it working so I'm willing to try the isolcpus change if you tell me how to do it. Also, I tried to boot OVMF with the commands you wrote, but I was not able to boot into Windows.. I ended up in shell> again.

 

@heilage: I have experimented with assigning different cores as you say, but that haven't made any difference.

 

 

@jonp: You said that you were to propose a change to the XMLs, could you please tell me more about this change?

@goober07: Your issues with DPC latency sounds interesting. Right now i feel very frustrated and just want to get it working so I'm willing to try the isolcpus change if you tell me how to do it. Also, I tried to boot OVMF with the commands you wrote, but I was not able to boot into Windows.. I ended up in shell> again.

 

I know the feeling. I still feel like I don't know what I'm doing half of the time.

 

See post #15 from jonp in this thread.

Your core assignments will be different. Start with 2 cores for each of the 2 VM's just to see if it works.

I have just tried the isolcpus change, but it didn't fix my problem. I installed latency monitor and before the change I had high DPC latency just like you goober. I started out with isolating 2 cores to each VM. After reboot of unRAID, I booted up the VMs and now the DPC latency was much lower. I started a game on one of the VMs and left it to see if the DPC latency would stay low. After about 4 minutes gaming the latency went to max and stayed there and the game started to perform really bad. After that I tried isolating more cores to the VMs (always kept at least 2 cores for unRAID) but I still had the same problems. I think the performance was worse (higher DPC) if I isolated 7 cores to the VMs compared to 2..

 

I have tried with both i440fx and Q35 but that doesn't seem to matter..

 

More ideas?

 

 

In my case, isolcpus reduced DPC latency and fixed the video & hdmi audio stutter, but my USB headset was still a no-go.

 

After removing it, DPC improved just a little more. There's some overhead to the virtual usb controller, and rather than troubleshoot it, I've got a pcie USB controller on the way... Going to pass that through to Windows. I might suggest removing any non-essential accessories (like a headset/DAC) or trying different keyboard/mouse, just to see if anything changes. Better yet, if you've got more than one controller available, pass it through to the VM & retest.

 

Also, verify both VM's have each display adapter (and it's audio device) using Messaged Signaled Interrupts. My 770 required a registry edit & guest reboot. By default it was supported but not enabled. Others call it the MSI Audio Fix, but it can't hurt to test here...

I'm sure you have already checked this, but the pci express slots you have on your motherboard they do both run full speed. When 2 cards in together.

On my board only the primary will run at full speed. If I put a gpu in second slot then the second slot is limited to x8

This may explain worse performance on second vm??

From what I'm aware of, dropping down to x8 or x4 doesn't provide more then a few fps so it wouldn't cause massive issues.

 

On thing that might be worth trying is if you get the lag when the second machine boots it could be that the drive running it isn't fast enough for the 2 systems. Try running crystal disk mark on my vdisks and see what speeds they say.

 

Have you also tried running any graphics benchmarks on each vm with only 1 at a time, then with both on to see the difference?

 

Might be worth a try