[Solved] Non windows GPU Passthrough issues


Recommended Posts

Hey there all,

 

So im having issues passing through gpu to anything that is not windows?

 This was my first post where i was having issues with passing through the gpu as the primary card, that has since been working great.

Also if i pass through any of my 3 cards through to windows all loads up as expected.

 

i could never get any gpu output on any linux based os,  i tried  ubuntu and deepin  so i sidelined it and focused on the windows vm.

 

When i first tested two Vms after getting primary gpu through on the 770 working i had catlina (i think... from spaceinvaders new docker container ) running on my gt 620 with output and the primary passthrough via 770 to the windows vm i cheered... but after i booted the pc the next day i had no gpu signal to the mac vm anymore. i tried reinstalling it about 6 times trying various combinations, with and without vbios on the GT 620, gtx 560 and the gtx 770 to the mac with no luck. i eventually gave up on the osx vm as technically the cards probably shouldn't work and the first time i must have just got lucky?... i even tried mojave and high sierra (as this is meant to support nvida more) no luck there either

 

So now ive decided to focus on atleast getting an ubuntu box going as that should work with my other two nvidia cards (gt 620 or gtx 560), and work for my development needs, below is my current ubuntu xml.

 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Ubuntu</name>
  <uuid>6b4c1fc5-3957-abff-23ed-085e92516c27</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Ubuntu" icon="ubuntu.png" os="ubuntu"/>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='3'/>
    <vcpupin vcpu='1' cpuset='7'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-3.1'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='1' threads='2'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Ubuntu/vdisk1.img'/>
      <target dev='hdc' bus='sata'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='7' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:c4:2a:8c'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

i have tried the 620, 560 and even the 770 nothing no signal output. no errors logged in diagnostics as far as i can see. the libvrt says

Quote

Nov  8 22:08:21 Tower kernel: vfio-pci 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none

so i think its trying...

 

ive spent a few hours each night trying to get various os's that are not windows to give me an output, hair pulling stuff haha.. especially since i had output the the mac vm just the way i wanted once before... i just want a non vnc output for a dev machine ;(

 

Does anyone have ANY suggestions..

 

Thanks in advance!!

Edited by phyzical
Link to comment

Hi I struggled a bit with the Linux machines and nvidia drivers, so this is my personal experience.

The only distro that was able to automatically pull the drivers and work out of the box was Mint. For all the others I tried (Ubuntu, Kubuntu, Manjaro, Arch) I had to install the latest driver (via noVNC) and then pass it.

Link to comment

Hey @juan11perez,

 

Thanks for the input!

 

to confirm in installing the correct apt. i tried nvidia-driver-435. but same issue. (the nvdia server-x thing from store was also installed with the install install)

 

The weird thing this its getting stuck in a reboot loop i think... as i cant ssh onto the box any more and when i hook up an rgb keyboard the keys flicker (as they do on first boot) every few seconds having a look in the syslog i see the usb keyboard is 

Quote

Nov 9 19:40:03 Tower kernel: usb 2-1.7: reset full-speed USB device number 4 using ehci-pci

over and over until i kill the box. chaging back to vnc restores the box and ssh works again.

 

it does this no matter the gpu primary with bios or secondary gpus.

 

I will try mint tonight just so i can try something linux based that "should work"...

 

another question incase its a factor do you use VirtIO for the vdisk. i am always forced to change it to SATA after removing the iso or it complains no bootable media.

Edited by phyzical
Link to comment

holy crap.... i was about to post another massssive speel about it not working.... and as i was rerunning all the vms to get logs the mint OS passed through the blood GPU OFC only thir time lucky apparently....

 

Thanks Juan for the suggestion. atleast i have something to dev on :D.

But would appreicate answers in regards to how you "update" the drivers correctly on ubuntu incase im doing something wrong with that.

 

Now for me to try and get a mac box passing through again.

 

PROGRESSSS!

 

edit:

spoke too soon.. so i spun down the vm and when i went to spin it back up. no bootable device again, changed to sata like i had todo with the ubutnu box and then i couldnt passthrough the GPU anymore.. but i could ssh onto the box so it was spinning up correctly.

just noticed i can do the whole install via the GPU passthrough with mint so ill try again like that instead

 

okay looks like this might be hitting the reboot bug on the gt 620. unfortanly google suggests the 560 suffers from this too ahwell. atleast i have something working as annoying as it will be for now

Edited by phyzical
Link to comment

@phyzical these are the commands for the ubuntu driver along with the related source:

 

Nvidia driver installation on kubuntu

https://www.cyberciti.biz/faq/ubuntu-linux-install-nvidia-driver-latest-proprietary-driver/

sudo apt search nvidia-driver

sudo apt-cache search nvidia-driver

sudo apt install nvidia-driver-435 

 

I also got it to work with debian with these:

nvidia driver installation on debian plasma

https://www.linuxbabe.com/debian/install-nvidia-driver-debian-9-stretch

sudo apt install lshw

sudo lshw -c display

sudo apt install software-properties-common

sudo add-apt-repository contrib

sudo add-apt-repository non-free

sudo apt update

sudo apt install nvidia-driver 

 

The whole nvidia ordeal is painful at beast; so patience. You'll get there.

 

I use an nvidia gigabyte 1660. 

 

 

Edited by juan11perez
Link to comment

GPU passthrough is essentially broken in the latest "stable" version. It works well on 6.6.6, though. Use that if you want to do Linux with a GPU. But Macinabox only works on 6.7.2, ditto the VMs it creates. So if you're looking to run OSX with a graphics card, you may have to wait a while for bugs to be fixed.

Edited by cyberspectre
Link to comment
GPU passthrough is essentially broken in the latest "stable" version. It works well on 6.6.6, though. Use that if you want to do Linux with a GPU. But Macinabox only works on 6.7.2, ditto the VMs it creates. So if you're looking to run OSX with a graphics card, you may have to wait a while for bugs to be fixed.

What do you mean ? Myself and dozens of people here run 6.7.2 perfectly fine since day one with GpU's passed though to OSX (nvidia/high sierra or AMD/mojave/cataline).
If you have issues it does not mean it does not work for anyone.
Link to comment

@juan11perez

Thanks for that, sounds like i was trying to do it right though i didnt try changing the proprietary settings. i will try with kubuntu just for the sake of it, worse case ive got the mint install. my ubuntu installs are defiantly doing something weird and getting a a reboot bug a soon as i passthrough the GPU

 

@glennv by the sounds of it cyber was just giving his 2 cents on what hes been experiencing in regards to linux not macosx,

In regards to OSX i agree i shouldn't have had an output using catlina and the nvidia card (maybe the macinabox didnt install catlina but high seirra the first time?). so i will try high sierra again.

Should i only need to install the webdrivers? or do i have to make clover config changes also i.e. nvda_enable=1. I am currently trying to use the macinabox docker so i am forced to a prebuilt clover image and i haven't been able to find a way to easily modify this cqow2 image. Im just trying to rule out that im running into the reboot bug with my gfx when the clover bootscreen pops up and thats why im not getting a signal on OSX

 

Thanks

Link to comment

Regarding getting an nvidia to work on OSX (assuming its a suported model on the specific os) the order is typicaly

1 make sure you have a fully working vm first without any passthru.

2 backup the vm file(s) and clover partition file so you can always go back if your next steps fail.

3. boot into the vm , install the correct nvidia web drivers and set the nvidia web flag in clover.

4 shutdown the vm, stub the nvidia card and if applicable its accompanied sound card (and for newer models even an accompanied usb i believe) Reboot the unraid server .

5 backup and then modify the xml manualy (in xml mode), remove the vnc / video device and put the nvidia device(s) there instead. (if you dont know how, you could do it in gui mode and then reapply the code that the gui mode wipes away at the end as well as re-correct network device)

6 Now boot the vm with the stubbed nvidia and fingers crossed.

Once it works install the cuda drivers .

 

If it does not work , recreate/reset to your original xml and point it to your backup clover and backup os vm and boot from there without the stubbed card.

There are lots of posts/threads online describing this process so check them out for details/troubleshooting etc if it does not work from the get go as there are a million way this can go wrong. Spaceinvader of course has lots of videos on this as well.

 

 

Link to comment

@glennv thanks for all that,

 

2 quick questions

 

* "stubbing" this is just assigning the gpu for passthrough correct?

* i tried adding the boot flag in clover via the boot menu but after a restart it seemed it was gone do you have any resources on the correct way to "edit the clover settings". most of the stuff ive been finding over the past few days suggests it should just be visible as a mounted folder? i tried checking all the /dev/disksXX and /Volumes but i could not find it

 

if i can figure out how to edit the clover then i should be able to test skipping the boot loader to rule out the reboot bug

 

Thanks!

Link to comment

1.Stubbing means telling unraid not to touch/see the stubbed devices so they can be used by a VM exclusively. Do a search on this forum on how to do this. 

2. Install clover configurator.

That allows you to mount the efi partition containing clover. Then load the clover config file and you can change anything you feel like changing ;-)

Sounds like you need to first brush up on some basic hackintosh skills, otherwise you will likely  get yourself into all kinds of problems. But of course trying and falling is a perfectly valid menthod to learn as well so go for it. ✌️🤪

ps recommend you watch a few SpaceInvaders videos on how to passthru a GPU and some how to in general setup an OSX VM. You will learn a lot from it and most of us have grown up with them , plus of course lots of trial and errors and frustration that comes with Hackintosh

Edited by glennv
Link to comment

@glennv

 

as a note. I have the primary card (GTX 770 primary passing through fine with the vbios ) the card im trying to use is the secondary card so unraid doesnt boot up on it, do i still need to "stub" it? i will search a bit harder on how to stub it and try it anyway juust incase.

 

i will also have a crack at the clover configurator i almost downloaded it but thought it was about editing the clover BEFORE getting it all up and running.

 

i haven't watched his older videos in regards to osx as i assumed they were not applicable given his latest docker (as in its a total different path) but i will have a watch of those too thanks. i might need to try doing an osx from scratch instead of his ezpz mode solution also haha

 

you are correct i am a tooootal noob in regards to Hackintosh.

if only IOS didnt have such a large market share haha...
 

as is the way with anything learn by fucking up :D

 

thanks again!

Link to comment

 

@phyzical Stubbing ,  isolating any hardware you want to pass thru to a VM, you can do 2 ways. I use it the old way which is adding the vfio-pci.ids parameter to your syslinux config (click main > boot device > flash > syslinux config) with the id's of your gpu + its audio device (if applicable). This makes it that (after a boot) unraid does not touch the card anymore and you will not have conflicts when using this card in your VM.

 

e.g. for my amd and its audio device i added the following 

 

    kernel /bzimage

    append vfio-pci.ids=1002:687f,1002:aaf8 initrd=/bzroot blablabla.......

 

You can find the right id's of your card via Tools > system devices

 

Regarding editing the clover plist file , you can only do this from an exisiting mac using the clover configurator (i am leaving out direct plist editing as for experienced hackintoshers).  Thats why you need to first boot into it, mount the efo partition , make the changes that you want to be in effect the next boots etc.

If any of these changes kills your system, you will see the importance of having an alternative way of accessing the efi partition. You could have for example a second VM to which you can attach the clover efi. I keep the clover EFI in a separate qcow2 file (as is also done with the Macinthebox) so you can back that up before making changes , so you can roll it back very simply and try again.

 

Regarding older spacenvader videos. You dont need to start building a new vm from scratch but looking at the older procedures you will learn other stuff like how to use clover configurator, how to properly manualy modify the xml file, things about kexts and patches etc etc.

So general hackintosh on VM knowledge, which is not that different from hackintosh on bare metal (its actualy a bit simpler in most areas)

 

 

Edited by glennv
Link to comment
On 11/11/2019 at 12:08 AM, glennv said:


What do you mean ? Myself and dozens of people here run 6.7.2 perfectly fine since day one with GpU's passed though to OSX (nvidia/high sierra or AMD/mojave/cataline).
If you have issues it does not mean it does not work for anyone.

Apologies, perhaps I should not have postulated as such. All I can say is that I had issues passing an nVidia GPU to any Linux VM in 6.7.2. When I rolled back to 6.6.6, it worked as expected again. Certainly, I could have attempted to troubleshoot and devise a fix myself. But avoiding that kind of tedium is the reason I bought UnRaid. It's also why I don't use the RC releases. When I upgraded to the latest stable release, I didn't expect features that were already working to break. And, considering the increasing number of threads raising similar concerns, I assumed I wasn't the only one.

Link to comment

No worries. No appologies needed. GPU passthru is a difficult hit and miss subject on any VM OS. There are so many factors that influence your success rate. Motherboard, bios, iommu, target VM os and version, drivers, clover settings etc etc.
I was purely referencing your mac osx related remark. Linux passthru never did or needed.
Also keep in mind that the problem with any tech forum is that its biased towards people with problems. People without issues tend to be extremely underrepresented compared to people with issues.
So the only thing you can say is that you are not alone

Link to comment

Thanks to all that provided input

 

@glennv Thanks for all that, was able to play around with clover, tried stubbing the gt 620, tried providing custom kexts and core display ect from what i could find online alas nothing seemed to help haha.. But there is a silver lining in a sense, i canabilised my main server's GPU GTX 760 which works with catalina OOB hazzah! so got it working by cheating still counts :P

 

@juan11perez Thanks for all the suggestions, alas again the gt620 refused to work with anything other than mint. But again once i started using the gtx 760 as the secondary card i was able to atleast get kubuntu booting. for some reason though ubuntu still refused to play ball, ahwell kubuntu will do :)

 

now my main server is just running 4k off dodgy IGPU until i upgrade my secondary pc some time near the new year. itll do!

 

Marking this as solved even though i feel whatever issues i was running into i never truly solved.. haha ohwell wins a win right...? :P

 

Thanks again all!

 

time for my docker adventure to begin, and then my main server will  be ready to come to unraid :)

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.