Hello, forum people! I think I really need your help. Any advices would be greatly appreciated.

10 days ago I first time installed unRAID OS and every its feature works fine, but I experience lots of issues with passing through GPUs.

 

The idea was to create a server that hosts one gaming VM and several non-gaming VMs, all with their own videocards.

 

What I have:

 

Intel Core i7-6850K (not overclocked)

Asus X99-E WS/USB3.1 (lastest BIOS)

32 GB of RAM

A running array with one SSD cache drive

850W Corsair PSU

Noname PCI-e USB controller (works fine)

 

GPUs:

MSI GTX 1050Ti (for gaming VM, tested on another PC)

Asus EN8400GS, MSI GT 610, GIGABYTE 210 (all for non-gaming VMs, all tested)

 

(BTW, I built this server online, streaming it on Twitch, if anyone interested, the video is still there, but it will be deleted in 3-4 days - https://www.twitch.tv/videos/122333373 - there are no any problems covered in that stream, only the build itself and the first unRAID run).

 

The short story: I test only one VM for now, and I have a problem with that VM: GTX 1050i works fine only when it's the primary card (single card config) or the secondary card (the primary card works for POST, BIOS and OS console).

 

What happens? When I install a second/third videocard in a MB slot, the VM I have starts to produce all sorts of video glitches - but only in certain circumstances. FurMark and Unigine tests usually work, but when I start Youtube or Twitch in Chrome browser, or just start Opera, the picture covers with white horizontal lines, and from that point anything could happen. Usually it just hangs after some time, but can produce all sorts of artifacts before going down.

 

The only game I tested was DayZ. It just doesn't start if I add any GPU on top of 1050 Ti.

 

I tested it in one Windows 10 VM (i440fx 2.7, OVMF), and it works fine in "console card + gaming card" or "single gaming card" configurations, but it throws artifacts in "gaming card + second card/third card" or "console card + gaming card + third/fourth card" configurations.

 

1st, 3rd, 5th and 7th PCI-e slots are recommended for videocards by the manufacturer, so I always tried to use them (but one time I used not recommended ports for additional GPU). I tried 1050Ti in 1st and 3rd slots, and I really want to have it in the 1st one, because it still will be at 16x even if I take all other slots for cards (16x-8x-8x-8x-8x-8x-8x configuration, according to the manual).

 

Other devices, passed to the VM, work fine - the PCI-e USB controller and 2nd onboard NIC, plus I tried the onboard audiocard (but disabled it). I've already tried to remove them - it doesn't make any difference.

 

TL;DR: One VM with GTX 1050 Ti in the server works fine. When I add more GPUs to the server (without even using them - neither for the console, nor for a VM), that one VM throws video artifacts (mainly in browsers) and eventually hangs.

 

I tried a lot of things, and I completely out of ideas. Please, if you have any thoughs on the subject, you are very welcome.

 

P.S. The XML file. bus='0x05' slot='0x00' - GTX 1050 Ti, bus='0x00' slot='0x19' - 2nd onboard NIC, bus='0x09' slot='0x00' - PCI-e USB controller card. GTX 1050Ti is in its own IOMMU with its HDMI Audio, the NIC and the USB controller have their own dedicated IOMMU groups too.

<domain type='kvm'>
  <name>Windows 10</name>
  <uuid>cd0b4afc-6c16-4299-a4e9-2b892e810eac</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <vcpupin vcpu='3' cpuset='5'/>
    <vcpupin vcpu='4' cpuset='8'/>
    <vcpupin vcpu='5' cpuset='9'/>
    <vcpupin vcpu='6' cpuset='10'/>
    <vcpupin vcpu='7' cpuset='11'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/cd0b4afc-6c16-4299-a4e9-2b892e810eac_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/cache/domains/Windows 10/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Windows.10.x86-x64.Ver1607+LTSB+-Office2016.24in1.by.SmokieBlahBlah.16.08.16.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.126-2.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:e2:b6:74'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/Temp/vbios.dump'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

Edited March 1, 20179 yr by Rive
solved

OK, a small update.

 

I have been experimenting with the single graphics card in slot 1 + another graphics card installed in slot 3 (not used for anything in unRAID). The VM is the same.

 

First, I found a flaw in my VM configuration: I passed through the GPU, but forgot about its sound device. I fixed it, and the system in the VM became a bit more stable - less white lines and other artifacts, longer functioning time in Chrome and Opera - but still far from being stable, after some time its state degrades beyond any usability.

 

After that I tried enabling ACS Control in BIOS - the thing theoretically might help with hardware issues in virtualization, but nothing changed for me.

 

Changing PCI-e generation to 2 (should be 3 by default) didn't help either.

 

Still, if I remove the second card (and it could be any of three I got), the VM becomes stable as a rock.

Well, looks like I've found a solution: I changed the type of machine to Q35 and now the VM works fine when the second (unactive) graphics card is plugged into MB's slot (I tried only the 3rd one). Everything is stable as rock, but I haven't tested it with a second/third/etc VM running, and I haven't tried to reinstall Windows in the VM.

 

So I'm marking the thread as solved, but there will be updates.