hupster

OK no problem! Good to see it works for other cards too.

No I don't think the rom you have is valid if you read it from the card while it is in the primary slot and not working for passthrough. I tried that, I did not get the same file as you get when the card is not used for booting. You probably get the rom that is used when you don't specify a rom file, which results in a black screen. One thing worth trying, do you have any VM that is configured to use the device as GPU? What happens if you remove all references to the card in any VM and reboot, is it still using the vfio-pci kernel driver? If not try reading the rom, also don't forget the "echo 1 > rom" to activate it. If the vfio-pci unbind trick doesn't work for you, one thing you might try is downloading the bios from http://www.techpowerup.com/vgabios/ or reading it with GPU-Z. Try it, if it doesn't work then I think you have a 'hybrid' bios like I think I had. I determined with a hex-editor that the file I read from my rom was also located somewhere in the middle of the file I had read using GPU-Z. Maybe you can find a way to extract it.

Ok that's about the same as I had. If you do: cd /sys/bus/pci/drivers/vfio-pci/ ls -al Do you see "0000:03:00.0" listed somewhere, or something else ending with "03:00.0"? I can't test right now on my unRAID box, but as I remember there were symlinks in that directory for the cards bound to vfio-pci. If there is a link I would expect that this would work: echo "0000:03:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind" I'm not sure why you actually need to unbind the card to read the rom. But for the GTX950 this step was required.

Ah but then if the card was used to boot, you're probably reading the shadowed copy which may be the cause of all problems. When you boot off another card and do 'lspci -v', what does it say for 'Kernel driver in use' for the 980ti?

So normally when the 980ti is in the second slot you can pass it through without problems, but when you add the rom file option to the configuration it breaks? I'd say it's a bad romfile then. You're sure you've read it from the card while the card is in the secondary slot and able to pass through? If you read it while the card is not working for passthrough, you probably get the file that doesn't work in the first place. For my GTX950 this file was significantly smaller. I have no experience with using romfiles with the hostdev method, maybe someone can step in here.

Ok but is that in your VM configuration XML file? It should be ",romfile=/boot/vbios.rom" (no spaces), added to an existing line like in my post. I don't know about other VM configuration file formats, you'd have to look that up. A good way to verify the bios file is to try it while the GPU is still placed in the second slot. If it then works well for passthrough, it should still do so after you add the "romfile=" part to the VM configuration. You could also try specifying a bad file to use as rom to see if the option is actually used (it should not boot). For me using the Vbios file solved the black screen issue, but there may be other issues on you system. The page I linked to mentions: "But even that may fail, either due to problems of the host chipset or BIOS (host kernel complains about unmappable ROM BAR)."

If the card is not assigned for passthrough to a VM then it's not required to unbind it from vfio-pci. Maybe that's the case, does it work to enable and cat the rom file?

If the primary card is NVIDIA it may still work. For me a workaround helped me to pass it through as primary graphics card: http://lime-technology.com/forum/index.php?topic=43644.msg452464#msg452464

I solved this on my system: Asus Rampage IV Formula / Intel Core i7-4930k / 4 x NVIDIA Gigabyte GTX 950 Windforce, with all graphics cards passed through to Windows 10 VMs. The problem I was having was that the 3 cards in slots 2, 3 and 4 pass though fine, but passing through the card in slot 1, which is being used to boot unRAID, freezes the connected display. I explored the option to add another graphics card. A USB card won't be recognized by the system BIOS to use for POST. The only other card I could add would be connected by a PCIe 1x to PCIe 16x riser card (which did work by the way for passthrough, but a need to pass through a x16 slot), but it would require modding the mainboard BIOS to get it use it as primary. So I looked for another solution. The problem was caused by the VBIOS on the video card, as mentioned on http://www.linux-kvm.org/page/VGA_device_assignment: To re-run the POST procedures of the assigned adapter inside the guest, the proper VBIOS ROM image has to be used. However, when passing through the primary adapter of the host, Linux provides only access to the shadowed version of the VBIOS which may differ from the pre-POST version (due to modification applied during POST). This has be been observed with NVDIA Quadro adapters. A workaround is to retrieve the VBIOS from the adapter while it is in secondary mode and use this saved image (-device pci-assign,...,romfile=...). But even that may fail, either due to problems of the host chipset or BIOS (host kernel complains about unmappable ROM BAR). In my case I could not use the VBIOS from http://www.techpowerup.com/vgabios/. The file I got from there, and also the ones read using GPU-Z is probably a Hybrid BIOS, it includes the legacy one as well as the UEFI one. It's probably possible to extract the required part from the file, but it's pretty simple to read it from the card using the following steps: 1) Place the NVIDIA card in the second PCIe slot, using another card as primary graphics card to boot the system. 2) Stop any running VMs and open a SSH connection 3) Type "lspci -v" to get the pci id for the NVIDIA card. It is assumed to be 02:00.0 here, otherwise change numbers below accordingly. 4) If the card is configured for passthrough, the above command will show "Kernel driver in use: vfio-pci". To retrieve the VBIOS in my case I had to unbind it from vfio-pci: echo "0000:02:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind 5) Readout the VBIOS: cd /sys/bus/pci/devices/0000:02:00.0/ echo 1 > rom cat rom > /boot/vbios.rom echo 0 > rom 6) Bind it back to vfio-pci if required: echo "0000:02:00.0" > /sys/bus/pci/drivers/vfio-pci/bind The card can now be placed back as primary, and a small modification must be made to the VM that will use it, to use the VBIOS file read in the above steps. In the XML for the VM, change the following line: <qemu:arg value='vfio-pci,host=01:00.0,bus=pcie.0,multifunction=on,x-vga=on'/> To: <qemu:arg value='vfio-pci,host=01:00.0,bus=pcie.0,multifunction=on,x-vga=on,romfile=/boot/vbios.rom'/> After this modification, the card is passed through without any problems on my system. This may be the case for more NVIDIA cards used as primary adapters!