[GPU passthrough with only one card?]

I tried the card in another vacant slot and this time unbind worked. I extracted the ROM and added romfile to hostdev in my xml. The vbios ROM is detected and the VM posts with the Gfx card in Slot 1. Thanks for your assistance.

 

OK no problem! Good to see it works for other cards too.

 

[GPU passthrough with only one card?]

No I don't think the rom you have is valid if you read it from the card while it is in the primary slot and not working for passthrough. I tried that, I did not get the same file as you get when the card is not used for booting. You probably get the rom that is used when you don't specify a rom file, which results in a black screen.

 

One thing worth trying, do you have any VM that is configured to use the device as GPU? What happens if you remove all references to the card in any VM and reboot, is it still using the vfio-pci kernel driver? If not try reading the rom, also don't forget the "echo 1 > rom" to activate it.

 

If the vfio-pci unbind trick doesn't work for you, one thing you might try is downloading the bios from http://www.techpowerup.com/vgabios/ or reading it with GPU-Z. Try it, if it doesn't work then I think you have a 'hybrid' bios like I think I had. I determined with a hex-editor that the file I read from my rom was also located somewhere in the middle of the file I had read using GPU-Z. Maybe you can find a way to extract it.

 

[GPU passthrough with only one card?]

Ok that's about the same as I had. If you do:

cd /sys/bus/pci/drivers/vfio-pci/
ls -al

Do you see "0000:03:00.0" listed somewhere, or something else ending with "03:00.0"?

I can't test right now on my unRAID box, but as I remember there were symlinks in that directory for the cards bound to vfio-pci.

If there is a link I would expect that this would work:

echo "0000:03:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind"

 

I'm not sure why you actually need to unbind the card to read the rom. But for the GTX950 this step was required.

 

[GPU passthrough with only one card?]

The only way i could unbind the card and read the rom file was if the gpu was in the 1st slot and no other gpu was installed. If i tried it in the second slot, unbind would not work and cat > rom would also fail. The GPU works fine in passthrough in the 2nd slot at the moment. As soon as I swap it to the 1st slot and specify the ROM file, the VM does not even show OVMF BIOS and is stuck on black screen.

 

Ah but then if the card was used to boot, you're probably reading the shadowed copy which may be the cause of all problems.

When you boot off another card and do 'lspci -v', what does it say for 'Kernel driver in use' for the 980ti?

 

[GPU passthrough with only one card?]

My devices including the 980ti GPU are all passed through using the hostdev method. I tried qemu method but this did not work either.

In fact, when I specify the ROM file, OVMF BIOS does not even appear on the screen so I know the vbios rom is being read.

 

So normally when the 980ti is in the second slot you can pass it through without problems, but when you add the rom file option to the configuration it breaks? I'd say it's a bad romfile then. You're sure you've read it from the card while the card is in the secondary slot and able to pass through? If you read it while the card is not working for passthrough, you probably get the file that doesn't work in the first place. For my GTX950 this file was significantly smaller.

 

I have no experience with using romfiles with the hostdev method, maybe someone can step in here.

 

[GPU passthrough with only one card?]

OK, so I managed to extract the vbios rom, however, I still get the black sreen after powering on the VM. Under hostdev in my xml i use the command like this...

 

</source>

<rom file='/boot/vbios.rom'/>

 

I tried with Seabios and ovmf and both exhibit a black screen.

 

Ok but is that in your VM configuration XML file? It should be ",romfile=/boot/vbios.rom" (no spaces), added to an existing line like in my post. I don't know about other VM configuration file formats, you'd have to look that up.

 

A good way to verify the bios file is to try it while the GPU is still placed in the second slot. If it then works well for passthrough, it should still do so after you add the "romfile=" part to the VM configuration. You could also try specifying a bad file to use as rom to see if the option is actually used (it should not boot).

 

For me using the Vbios file solved the black screen issue, but there may be other issues on you system. The page I linked to mentions: "But even that may fail, either due to problems of the host chipset or BIOS (host kernel complains about unmappable ROM BAR)."

 

[GPU passthrough with only one card?]

I get the following error when trying to unbind... I check my 980ti is 02:00.0.

 

root@MOUNRAID01:/sys/bus/pci/devices/0000:02:00.0# echo "0000:02:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind

 

bash: echo: write error: No such device

 

What am i doing wrong?

 

If the card is not assigned for passthrough to a VM then it's not required to unbind it from vfio-pci.

Maybe that's the case, does it work to enable and cat the rom file?

 

[Almost got it! One last problem]

If the primary card is NVIDIA it may still work. For me a workaround helped me to pass it through as primary graphics card:

http://lime-technology.com/forum/index.php?topic=43644.msg452464#msg452464

 

[GPU passthrough with only one card?]

I solved this on my system: Asus Rampage IV Formula / Intel Core i7-4930k / 4 x NVIDIA Gigabyte GTX 950 Windforce, with all graphics cards passed through to Windows 10 VMs. The problem I was having was that the 3 cards in slots 2, 3 and 4 pass though fine, but passing through the card in slot 1, which is being used to boot unRAID, freezes the connected display.

 

I explored the option to add another graphics card. A USB card won't be recognized by the system BIOS to use for POST. The only other card I could add would be connected by a PCIe 1x to PCIe 16x riser card (which did work by the way for passthrough, but a need to pass through a x16 slot), but it would require modding the mainboard BIOS to get it use it as primary. So I looked for another solution.

 

The problem was caused by the VBIOS on the video card, as mentioned on http://www.linux-kvm.org/page/VGA_device_assignment:

To re-run the POST procedures of the assigned adapter inside the guest, the proper VBIOS ROM image has to be used. However, when passing through the primary adapter of the host, Linux provides only access to the shadowed version of the VBIOS which may differ from the pre-POST version (due to modification applied during POST). This has be been observed with NVDIA Quadro adapters. A workaround is to retrieve the VBIOS from the adapter while it is in secondary mode and use this saved image (-device pci-assign,...,romfile=...). But even that may fail, either due to problems of the host chipset or BIOS (host kernel complains about unmappable ROM BAR).

 

In my case I could not use the VBIOS from http://www.techpowerup.com/vgabios/. The file I got from there, and also the ones read using GPU-Z is probably a Hybrid BIOS, it includes the legacy one as well as the UEFI one. It's probably possible to extract the required part from the file, but it's pretty simple to read it from the card using the following steps:

 

1) Place the NVIDIA card in the second PCIe slot, using another card as primary graphics card to boot the system.

2) Stop any running VMs and open a SSH connection

3) Type "lspci -v" to get the pci id for the NVIDIA card. It is assumed to be 02:00.0 here, otherwise change numbers below accordingly.

4) If the card is configured for passthrough, the above command will show "Kernel driver in use: vfio-pci". To retrieve the VBIOS in my case I had to unbind it from vfio-pci:

echo "0000:02:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind

5) Readout the VBIOS:

cd /sys/bus/pci/devices/0000:02:00.0/
echo 1 > rom
cat rom > /boot/vbios.rom
echo 0 > rom

6) Bind it back to vfio-pci if required:

echo "0000:02:00.0" > /sys/bus/pci/drivers/vfio-pci/bind

 

The card can now be placed back as primary, and a small modification must be made to the VM that will use it, to use the VBIOS file read in the above steps. In the XML for the VM, change the following line:

<qemu:arg value='vfio-pci,host=01:00.0,bus=pcie.0,multifunction=on,x-vga=on'/>

To:

<qemu:arg value='vfio-pci,host=01:00.0,bus=pcie.0,multifunction=on,x-vga=on,romfile=/boot/vbios.rom'/>

 

After this modification, the card is passed through without any problems on my system. This may be the case for more NVIDIA cards used as primary adapters!