Cannot Get 1/4 Dedicated Passthrough GPUs to Work


Recommended Posts

UnRAID 6.5.1
Asus Rampage V Ed10
Intel I7-6850K

64GB Trident Z 3333Mhz RAM
EVGA 1080ti
EVGA 1080ti

EVGA 1080ti
EVGA 1080ti

 

Ok, So I have a Gaming Rig which I specifically built for the purpose of VR + LAN Parties in my Livingroom. Concept is to pull up 2 folding tables out, plug in 4 Monitors, Keyboards and Mice and have a 4 person LAN Party going.

 

The board (Asus Rampage V Ed 10) has 4x PCI-E16 slots with 4x 1080tis . The board is capable of running all 4 and all 4 was running fine on a previous Windows OS (Before Unraid) so I know hardware compatibility wise is fine.

 

This is how I have it laid out

VM1 (Main VM) <- Does not load

Nvidia 1080ti 02:0:0 > Monitor 1

 

VM2 (Gaming VM) <- Loads Fine

Nvidia 1080ti 04:0:0 > Monitor 2

 

VM3 (Gaming VM) <- Loads Fine

NVidia 1080ti 05:0:0 > Monitor 3


VM4 (Gaming VM) <- Loads Fine

NVidia 1080ti 06:0:0 > Monitor 4

This is what I've observed.
VM 2,3,4 will load without issues, and upon initial boot of UnRAID, Monitor 1 will show UnRAIDs CLI. When VM 1 is started, Monitor 1 conks out then turns on again (as if to get signal), backlight comes on, blue light comes on, then loses signal, and blue light turns orange (to show no signal) again.

I have tried adding VNC to VM1 so that I can also see what is happening, and when attempting to connect to the VM via VNC (and the monitor which is still showing no signal), the initial starting splash screen shows (the logo for the Virtual Driver before windows starts loading) but seems to be frozen there. No spinning circles that windows typical shows before boot.

 

This is my VM Logs (But I have no idea what I'm looking at
2018-05-13 19:07:23.771+0000: starting up libvirt version: 4.0.0, qemu version: 2.11.1, hostname: AN-URHost01
LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/local/sbin/qemu -name guest=AN-03,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-3-AN-03/master-key.aes -machine pc-i440fx-2.11,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off -cpu host -drive file=/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/etc/libvirt/qemu/nvram/9bffe904-dfa7-c20d-526c-f24a2b010059_VARS-pure-efi.fd,if=pflash,format=raw,unit=1 -m 12288 -realtime mlock=off -smp 4,sockets=1,cores=2,threads=2 -uuid 9bffe904-dfa7-c20d-526c-f24a2b010059 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-3-AN-03/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on -device nec-usb-xhci,p2=15,p3=15,id=usb,bus=pci.0,-3-AN-03/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device vfio-pci,host=02:00.0,id=hostdev0,bus=pci.0,addr=0x6 -device vfio-pci,host=02:00.1,id=hostdev1,bus=pci.0,addr=0x8 -device usb-host,hostbus=3,hostaddr=7,id=hostdev2,bus=usb.0,port=1 -device usb-host,hostbus=3,hostaddr=3,id=hostdev3,bus=usb.0,port=2 -device usb-host,hostbus=3,hostaddr=10,id=hostdev4,bus=usb.0,port=3 -device usb-host,hostbus=3,hostaddr=5,id=hostdev5,bus=usb.0,port=4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x9 -msg timestamp=on
2018-05-13 19:07:23.771+0000: Domain id=3 is tainted: high-privileges
2018-05-13 19:07:23.771+0000: Domain id=3 is tainted: host-cpu
2018-05-13T19:07:23.840522Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/2 (label charserial0)
libusb: error [udev_hotplug_event] ignoring udev action bind
libusb: error [udev_hotplug_event] ignoring udev action bind
libusb: error [udev_hotplug_event] ignoring udev action bind
libusb: error [udev_hotplug_event] ignoring udev action bind
libusb: error [udev_hotplug_event] ignoring udev action bind
libusb: error [udev_hotplug_event] ignoring udev action bind
libusb: error [udev_hotplug_event] ignoring udev action bind
2018-05-13T19:15:12.183026Z qemu-system-x86_64: terminating on signal 15 from pid 3897 (/usr/sbin/libvirtd)
2018-05-13 19:15:13.583+0000: shutting down, reason=destroyed
2018-05-13 19:17:25.182+0000: starting up libvirt version: 4.0.0, qemu version: 2.11.1, hostname: AN-URHost01
LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/local/sbin/qemu -name guest=AN-03,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-6-AN-03/master-key.aes -machine pc-i440fx-2.11,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off -cpu host -drive file=/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/etc/libvirt/qemu/nvram/9bffe904-dfa7-c20d-526c-f24a2b010059_VARS-pure-efi.fd,if=pflash,format=raw,unit=1 -m 12288 -realtime mlock=off -smp 4,sockets=1,cores=2,threads=2 -uuid 9bffe904-dfa7-c20d-526c-f24a2b010059 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-6-AN-03/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on -device nec-usb-xhci,p2=15,p3=15,id=usb,bus=pci.0,-6-AN-03/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device vfio-pci,host=02:00.0,id=hostdev0,bus=pci.0,addr=0x6 -device vfio-pci,host=02:00.1,id=hostdev1,bus=pci.0,addr=0x8 -device usb-host,hostbus=3,hostaddr=7,id=hostdev2,bus=usb.0,port=1 -device usb-host,hostbus=3,hostaddr=3,id=hostdev3,bus=usb.0,port=2 -device usb-host,hostbus=3,hostaddr=10,id=hostdev4,bus=usb.0,port=3 -device usb-host,hostbus=3,hostaddr=5,id=hostdev5,bus=usb.0,port=4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x9 -msg timestamp=on
2018-05-13 19:17:25.182+0000: Domain id=6 is tainted: high-privileges
2018-05-13 19:17:25.182+0000: Domain id=6 is tainted: host-cpu
2018-05-13T19:17:25.250534Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/3 (label charserial0)

VM 2,3,4 during all this works fine and independent of this issue.

Sometimes VM 1 will shut down on its own (Maybe its timing out on the boot?)

Board only has 4 slots for GPUs so I cannot add a 5th crappy GPU to put UnRAID's CLI onto it. Long story short. 5th PCI-E GPU is not an option, Everything has already been Waterblocked. Id be OK with getting a USB VGA Card if there was a way to set it as the Default GPU for UnRAID to solve this issue.

Edited by RichardBoelens
Link to comment

Don't you have to download the card's BIOS ROM and pass it through as a file in order to use the primary graphics card in a VM? It's not something I have a need for so I'm no expert but I recall people mentioning it. What are you using as your guide for building this machine? I'd expect it to be discussed there. Does the i7-6850K have enough resources for four gaming VMs? 40 PCIe lanes will support four graphics cards in x8 mode but are six cores enough? How are they divided up between the VMs and unRAID itself? I hope what you're asking of your hardware is realistic. If it turns out that you do need a fifth GPU Zotac does one with a PCIe x1 interface, but I suppose your x1 sockets are blocked by the double width cards, which leads me to think once again that you might be expecting too much from your hardware.

Link to comment

Cores 1/2  + Threads 5/6 = VM 1

Core 3 + Thread 7 = VM 2
Core 4 + Thread 8 = VM 3
Core 5 + Thread 9 = VM 4
Core 6 + Thread 10 = Spare, might give to VM 1 or 2, not sure yet

I don't know about BIOS Roms. I've never heard anything about it. Can you elaborate?

Yes, PCIE slots are all blocked and used up as they are disabled due to me running an NVME Drive
I haven't really used any guides. Just got the idea from watching LinusTechTips once

Edited by RichardBoelens
Link to comment
On 5/14/2018 at 6:16 AM, RichardBoelens said:

Cores 1/2  + Threads 5/6 = VM 1

 

I'm confused by your numbering - are you counting from one or zero? You need to leave some resources for unRAID. It favours CPU 0 (both hyperthreads, 0 and 6) so best to let it have exclusive use and divide up the remaining five cores (CPUs 1 to 5) amongst the VMs, keeping the hyperthreads paired together. I'd rewrite your table like this:

 

unRAID: core 0  (threads 0 + 6)

VM 1:     cores 1 + 2 (threads 1 + 2 + 7 + 8 )
VM 2:     core 3 (threads 3 + 9)
VM 3:     core 4 (threads 4 + 10)
VM 4:     core 5 (threads 5 + 11)

Link to comment

Hi Richard.

 

I was also going to point you to the VBIOS thread.

However. I'm very curious if you got this working without problems?

 

When I started using Unraid, according to all sources, it was impossible to GPU passthrough 2 (or more) of the SAME Gfx cards. In order to achieve this you would have had to have different cards.

 

I can't remember if the above was only true if you used the VBIOS method to passthrough Gfx 1 card.

I've specifically added two different Gfx card to my setup to avoid this problem.

 

So therefore I'm very curious if you got your setup working with 4 of same cards (With 1 being main card passthrouged to VM1).

Edited by cyberdude
Link to comment

@John_M
Thanks, I didn't know that. I will shift the Core Assignments next reboot.

@cyberdude
Yes, it was resolved.
I am running it without issues on 4x EVGA 1080ti FTW3s
I only had to apply the VBIOS to the First card, the other 3 work fine without
I've only tested with 4x VMs with 1x 1080ti going to each VM, I haven't tested 4x 1080ti going to 1x VM
I am currently on a different rig, trying to get 2x 1080tis going to the same VM, but I am struggling with an issue with similar symptoms, but different problem. That Thread is Here.

Excuse the Cable Management (Or Lack Thereof) in the Pic. It's still a work in progress and I've yet to receive the cables I ordered from Cablemod.
 

20180515_143858.jpg

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.