Primary GPU passthrough help [solved]


Recommended Posts

Hey there,

 

yes... its another one of these threads.

Like all the others this is my first time playing around with unraid.. loving it, will i will probably switch my main beast over too soon enough

 

So basically i have a gtx 770 in slot two, this can passthrough fine as its not used by unraid. but im unable to passthrough the gfx i have in slot one a GT 620, which unraid uses on boot. i have followed space invaders vid and dumped the bios, i have tried using both bios from techpowerup, i have tried following all the threads i could find but everytime i seem to be faced with "black screen of death". i was hoping to use the better gfx for gaming ona  windows box and use the passthrough override for a dev vm (tried ubuntu and deepin so far).

I have tried doing the on startup on array script
 

Quote

#!/bin/bash
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

but that seems to throw /tmp/user.scripts/tmpScripts/boot/script: line 3: echo: write error: No such device

 

i have also seen one person saying this is alot harder to pull off with older cards i.e the gt620. Slot 1 may not work, dump of bios may not work

i am running an Xeon build so no grfx output via the mobo

 

Quote

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Deepin</name>
  <uuid>236f5713-8190-4550-e5b6-a2a6066d49d6</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Debian" icon="debian.png" os="debian"/>
  </metadata>
  <memory unit='KiB'>4718592</memory>
  <currentMemory unit='KiB'>4718592</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='3'/>
    <vcpupin vcpu='1' cpuset='7'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-3.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/236f5713-8190-4550-e5b6-a2a6066d49d6_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='1' threads='2'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Deepin/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:78:ff:e1'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='3'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/gt620.dump'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc07c'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x258a'/>
        <product id='0x6a88'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

 

Does anyone have any suggestions as to how to debug why this doesn't seem to work or any additional info needed to help trace the issue further?

or am i just forced to wait until i have another pc to play with? :(

 

Thanks!!

Edited by phyzical
Link to comment

Firstly, you are using Q35 machine type v3.1 (presumably since you are on stable 6.7.2). In which case, you need to add this bit of code before </domain>

  <qemu:commandline>
    <qemu:arg value='-global'/>
    <qemu:arg value='pcie-root-port.speed=8'/>
    <qemu:arg value='-global'/>
    <qemu:arg value='pcie-root-port.width=16'/>
  </qemu:commandline>

Alternatively, update to the latest 6.8.0-rc and choose Q35-4.1.

Note: that's not a fix for your issue. That should be done regardless so that your PCIe slot runs at the right speed.

 

Now I'm pretty sure the GT 610 has reset issues so the GT 620 probably does too. So in your case, perhaps we need to think outside the box. Have you tried flipping the card slot i.e. 770 in primary and 620 in secondary?

 

Even if the above is successful, you are also likely to run into another problem, that is you will lose graphics with the GT 620 if the VM restarts (i.e. the reset issues manifesting in a different way). So probably you are likely to need a new GPU anyway.

Link to comment

thanks for the speedy reply testdasi,

 

* should i do this fix for my windows vm too which is running the gtx 770 atm? (assume so, ill upgrade to the rc some time in the future still mucking about)

 

* i have not tried switching the slots, but due to my mobo design 7558_src.jpg and cpu cooler size , ive just remembered i actually have the gtx 770 in the second x16 and the primary is currently in the first x8, so i dont think i can without sacrificing gfx. would an 8x and an 16x attribute to any issues or am i simply experiencing the gfx reboot bug you have mentioned (which seems so as when i was doing the dump i could unbind but it would not let me bind the gfx again)? when i find some time ill try switching it up to see if itll work but due to the above i just dont think itll be viable long term, maybe ill get a smaller cpu cooler if i dont forsee a new pc anytime soon. :(

 

* one last question for curiosity sake should the unraid gui/console rebind once the vm taking control goes down on a typical setup or is losing the bind until unraid reboot normal (jw if this is the reboot bug in action your mentioning)?

 

Thanks for the input so far!

Edited by phyzical
Link to comment
1 hour ago, phyzical said:

thanks for the speedy reply testdasi,

 

* should i do this fix for my windows vm too which is running the gtx 770 atm? (assume so, ill upgrade to the rc some time in the future still mucking about)

 

* i have not tried switching the slots, but due to my mobo design  and cpu cooler size , ive just remembered i actually have the gtx 770 in the second x16 and the primary is currently in the first x8, so i dont think i can without sacrificing gfx. would an 8x and an 16x attribute to any issues or am i simply experiencing the gfx reboot bug you have mentioned (which seems so as when i was doing the dump i could unbind but it would not let me bind the gfx again)? when i find some time ill try switching it up to see if itll work but due to the above i just dont think itll be viable long term, maybe ill get a smaller cpu cooler if i dont forsee a new pc anytime soon. :(

 

* one last question for curiosity sake should the unraid gui/console rebind once the vm taking control goes down on a typical setup or is losing the bind until unraid reboot normal (jw if this is the reboot bug in action your mentioning)?

 

Thanks for the input so far!

You should do the fix (also known as the root port patch) for all Q35-based VM (and only Q35) under v4.0.

 

You have a Gigabyte motherboard, first check your BIOS to see if you can pick which slot as primary. (My Gigabyte mobo has that option so I reckon it's a Gigabyte thing). Then you can use the 770 as primary without having to swap slot.

 

With regards to x8 vs x16, I don't think your 770 / 620 will ever come close to even requiring x4 so there's no issue with that. Your reset issue is a card-based and not a slot-based issue.

 

Yes Unraid will do it correctly as long as the card doesn't resist. I restart my 1070 VM all the time. I used to have a MacOS VM on the 710 and that also had no problem.

Edited by testdasi
Link to comment

Thanks again,

 

So i found an gtx 560 SE and switched that out for the gt 620. did another bios dump, i did have a weird issue where the terminal would lockup on unbind while dumping the bios.. (no known weird issues with this new card is there maybe?). buut same thing no output on vm spinup with the 560 as passthrough with bios..

 

* i will try switching via the bios tomorow (its getting late) and see if i get the same issue with 770 as primary and 560 as secondary.

 

* Oh one thing i am noticing is that when im using the passthrough im forced to force stop the vm, as opposed to when vncing which  i just stop. This would mean the VM isnt loading up the OS properly right and thus cant sned the kill signal? Looking at the "logs" of the vm from the vm tab doesn't seem to have anything useful... buut just incase

 

This is the VNC

Quote

-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=26,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \
-device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \
-device pcie-root-port,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=0xb,chassis=6,id=pci.6,bus=pcie.0,addr=0x1.0x3 \
-device pcie-pci-bridge,id=pci.7,bus=pci.1,addr=0x0 \
-device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x7.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x7 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x7.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x7.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \
-drive file=/mnt/user/domains/Ubuntu/vdisk1.img,format=raw,if=none,id=drive-sata0-0-2,cache=writeback \
-device ide-hd,bus=ide.2,drive=drive-sata0-0-2,id=sata0-0-2,bootindex=1,write-cache=on \
-netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=30 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:78:2f:e9,bus=pci.3,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=31,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=3 \
-device vfio-pci,host=01:00.0,id=hostdev0,x-vga=on,bus=pci.4,addr=0x0,romfile=/mnt/user/isos/gtx560.rom \
-device usb-host,hostbus=1,hostaddr=5,id=hostdev1,bus=usb.0,port=1 \
-device usb-host,hostbus=1,hostaddr=4,id=hostdev2,bus=usb.0,port=2 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2019-10-27 10:46:43.591+0000: Domain id=6 is tainted: high-privileges
2019-10-27 10:46:43.591+0000: Domain id=6 is tainted: host-cpu
char device redirected to /dev/pts/1 (label charserial0)
2019-10-27T10:47:43.883956Z qemu-system-x86_64: terminating on signal 15 from pid 3304 (/usr/sbin/libvirtd)
2019-10-27 10:47:44.278+0000: shutting down, reason=destroyed

when passthrough

Quote

-nodefaults \
-chardev socket,id=charmonitor,fd=26,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \
-device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \
-device pcie-root-port,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=0xb,chassis=6,id=pci.6,bus=pcie.0,addr=0x1.0x3 \
-device pcie-pci-bridge,id=pci.7,bus=pci.1,addr=0x0 \
-device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x7.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x7 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x7.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x7.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \
-drive file=/mnt/user/domains/Ubuntu/vdisk1.img,format=raw,if=none,id=drive-sata0-0-2,cache=writeback \
-device ide-hd,bus=ide.2,drive=drive-sata0-0-2,id=sata0-0-2,bootindex=1,write-cache=on \
-netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:78:2f:e9,bus=pci.3,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=31,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=3 \
-device vfio-pci,host=01:00.0,id=hostdev0,x-vga=on,bus=pci.4,addr=0x0,romfile=/mnt/user/isos/gtx560.rom \
-device usb-host,hostbus=1,hostaddr=5,id=hostdev1,bus=usb.0,port=1 \
-device usb-host,hostbus=1,hostaddr=4,id=hostdev2,bus=usb.0,port=2 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2019-10-27 10:51:15.041+0000: Domain id=9 is tainted: high-privileges
2019-10-27 10:51:15.041+0000: Domain id=9 is tainted: host-cpu
char device redirected to /dev/pts/1 (label charserial0)
libusb: error [udev_hotplug_event] ignoring udev action bind
2019-10-27T11:00:25.867865Z qemu-system-x86_64: terminating on signal 15 from pid 3304 (/usr/sbin/libvirtd)
2019-10-27 11:00:26.164+0000: shutting down, reason=destroyed

i just feel like im screwing something up at this point if its happening on two completely different generations of cards right?

 

Thanks!

Link to comment

Hey @testdasi,

 

* So i tried switching to legacy boot via bios, no change

* the only thing i could see in regards to gpu "switching" was choosing the primary pcie which seemed to have no effect on unraid unfortunately

 

* now the fun one.. so i tried switching the gpus around (gtx 770 in slot 1 and gtx 560 in slot 2), this moved the unraid boot over to the gtx 770, i tried launching a windows vm this time on the gtx 560. i got picture (to be expected as its no longer primary) proceeded to dump the vbios same as  every other time i could unbind could not not rebind (i wonder if this is a symptom at all? had the same issue with the gt 620). When i tried to test a boot on the  windows vm with this bios i got nothing, so i rebooted and confirmed the windows vm rebooted with the bvios rom i dumped. I then proceeded to switch the cards back to the original order (gtx 770 in slot 2 and gtx 550 in slot 1) but unfortunately i got the same issue...

 

* To confirm if i switch the slots i shouldn't have to re dump the bios? its just in the space invader video there was mention of (be sure to make sure the video location has not changed which ofc it does if i have switch the gfx card back)

 

i did have one weird thing happen though the first time i launched it up after switching back to the original order i had the boot screen on the 770 still even though it was in slot 2, so after it didnt work i had a thought maybe i could replicate this to re dump the bios (but ofc if switching the slots doesn't affect the bios then this is a moot point). i also could not replicate this again anyways (tried relaunching with cords only in 770, also tried a boot with only 770 shutdown then boot again with both cards)

 

* one other thing i could replicate (not that i think it indicates anything) was that if i launched a vm with the primary card and then proceeded to unbind the call would never return, as in it just hung indefinably (this is got my hopes up as maybe i had been doing it all wrong) but unfortunately it did not seem to make a difference in the long run

 

Is it possible i just have some hardware that refuses to play ball? Or to Anyone else is lurking have you happened to have run into a similar issue?

 

Thanks again!!

 

Link to comment

So i had another few hours of researching other threads, rereading the ones i had. i found a few people mentioning they had to choose the audio driver also when providing the bios and i realized i dont think i was providing it during my testing of the various combinations.

 

so i gave it all another go but unfornatly this made no difference buuuut, since i had been switching the cards so much i decided to give the 770 a crack, had something in slot 1 and the gtx in slot 2 dumped the bios and bam the 770 let me passthrough as the primary card when in slot 1.. omfg i guess the gt 620 and gtx 560 just don't allow for it to be the primary and passthrough? is there somewhere i should add these cards to add to the list of no goes? i.e a wiki?

 

the gt 620 doesn't seem to suffer from the reboot bug btw, i am able to boot it on a windows vm multiple times without reboot.

 

Now i have a new issue (which may have been compounding to my testing to begin with) i cant seem to get video output to a vm that isnt windows i.e deepin or ubuntu. ive tried the 560 and 620 as secondary cards. i even tried the 770 as the primary card providing the vbios no dice.

 

I will setup a majove vm tommrow and see if that makes any difference also. Guess its time to start investigating again.

 

Just tested space invaders new container for making mac os vms and got an output on the gt 620 while having an output on the 770 as primary to the windows vm :D guess theres some sort of driver issue im facing with the ubuntu and deepin vms

 

thanks for the hand holding @testdasi, ill mark this one as solved

Edited by phyzical
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.