(SOLVED + HOWTO) Help pls - Trouble with GPU passthrough GTX1080TI


Recommended Posts

Hi all,

 

I'm hoping someone can help. I'm having trouble with passing through a GTX1080TI GPU.

I am using model EVGA 11G-P4-6696-KR, known as "EVGA GeForce GTX 1080 Ti FTW3 GAMING".

 

I had trouble finding a compatible BIOS on techpowerup, because there is no direct comparison.

I saw that there are "FTW3 Elite Gaming" and "FTW3". I've tried the FTW3 BIOS, but not the FTW3 Elite, I'm concerned that it is not appropriate.

 

So I dumped the bios in a bare metal Windows 10 install using nvflash, and have saved it to my array.

 

Even still, I cannot seem to get my VM to boot with it using any configuration.

The screen flashes and then no signal, every time.

I need to reboot to get anything back.

I was able to boot into Windows 10 using the bare metal install, so I think the card is working.

 

I've watched the videos by SpaceInvader and I've passed through cards before using this system (An R9-290 and recently a RX-5600XT that I am returning). Both worked without much issue at all.

 

Can anyone help with advice?

 

I also dumped the version using nvflash:

NVIDIA Firmware Update Utility (Version 5.590.0)
Copyright (C) 1993-2019, NVIDIA Corporation. All rights reserved.


Sign-On Message       : GP102 PG611 SKU 50 VGA BIOS
Build GUID            : 402D5B72B8A34DFBBEB8A9C5CD99EE31
IFR Subsystem ID      : 3842-6696
Subsystem Vendor ID   : 0x3842
Subsystem ID          : 0x6696
Version               : 86.02.39.00.90
Image Hash            : AAA37529FD718F13599221F885E83C25
Product Name          : GP102 Board
Device Name(s)        : GeForce GTX 1080 Ti
Board ID              : 0xEE17
Vendor ID             : 0x10DE
Device ID             : 0x1B06
Hierarchy ID          : Normal Board
Chip SKU              : 350-0
Project               : G611-0050
Build Date            : 01/18/17
Modification Date     : 04/17/17
UEFI Version          : 0x30006
UEFI Variant ID       : 0x0000000000000007 ( GP1xx )
UEFI Signer(s)        : Microsoft Corporation UEFI CA 2011
XUSB-FW Version ID    : N/A
XUSB-FW Build Time    : N/A
InfoROM Version       : G001.0000.01.04
InfoROM Backup        : Not Present
License Placeholder   : Not Present
GPU Mode              : N/A

 

Edited by KptnKMan
Link to comment

I've dunped my BIOS out directly in unRAID, and trying to follow instructions to fix.

I'm unsure if this means something but I've noticed when running "lspci v" is that there is no kernel driver listed next to my card:

 

0d:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: eVga.com. Corp. GP102 [GeForce GTX 1080 Ti]
        Flags: bus master, fast devsel, latency 0, IRQ 5
        Memory at f4000000 (32-bit, non-prefetchable) [size=16M]
        Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Memory at f0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at f000 [size=128]
        Expansion ROM at 000c0000 [disabled] [size=128K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [250] Latency Tolerance Reporting
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] Secondary PCI Express <?>

0d:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)
        Subsystem: eVga.com. Corp. GP102 HDMI Audio Controller
        Flags: bus master, fast devsel, latency 0, IRQ 4
        Memory at f5080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting

Also, I'm unable to either unbind or bind the device:

# echo "0000:0d:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
-bash: echo: write error: No such device

# echo "0000:0d:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
-bash: echo: write error: No such device

 

Edited by KptnKMan
Link to comment

Well, I figured it out.

I spent many hours trying many different things, but this turned out to be it.

 

Really wished that someone could have pointed me in the right direction, especially with so many current thread posts about the same thing on this forum.

 

As such, I hope that I might be able to help someone else by summarising my journey and how I got there.

A full solution is below, but first, I highly recommend watching these 2 excellent videos from @SpaceInvaderOne. Really saved my bacon: 👍

I'm basically summarising much of what he put in the video.

 

 

 

My Summary:

I was able to fix an already working VM, with my newly added GTX1080Ti using these methods.

I didn't need to disable Hyper-V as others have said, and I'm using the latest nvidia drivers.

 

In the end, the "code 43" error that is prevalent, seems to be due to a couple of things:

- Modified BIOS, removing application headers from file, using Hex editor.

- Specifying GPU pci device as "multifunction" in VM XML.

 

Modifying BIOS:

- Download the Hex Editor, install.

- Download your compatible BIOS from techpowerup.

- Open the bios file in hex editor. I'm using HxD like in the video.

- Look for line that looks like "UªyëK7400éLwÌVIDEO", mine was prefaced with a series of "ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ" characters.

image.png.963cb2a0efb86d168b11345227fd4201.png

- Next, delete everything above line "UªyëK7400éLwÌVIDEO", so that the matching offset reads "00000000" (8 zeros).

image.png.85db335dae2b2cfe6e5d263e70b8c6b8.png

- "Save As..." the file with a new filename. Save it somewhere on your unRAID system, like one of the shares.

- I saved mine to "isos" share. You can save it anywhere on unRAID.

- You're done.

 

Modifying your VMs XML:

- Before Starting, make sure your GPU is in its own IOMMU Group (Tools->System Devices).

- If your GPU is not in its own IOMMU Group (Its fine if the GPU shares that group with the GPU-Audio).

image.png.224dcbc252a678b7d37f51049a9d612b.png

- Setup your VM in unRAID as normal.

- I have had success using "Q35-4.2" Machine type, and "OVMF" BIOS. I left Hyper-V "on".

- I'm using VirtIO Drivers "virtio-win-0.1.173-2" if that is helpful.

- Specify your GPU and GPU-Audio Sound Card of the GPU. Always add your GPU-Audio, dont try to avoid it. You need it.

- If you want to use your motherboard sound card, you should add it by clicking the green plus-sign on the left.

- Either "CREATE" (Do not "Start VM after creation") or "UPDATE" the VM at the bottom, to set a baseline config.

- Go back into the VM config. On the right change "FORM VIEW" to "XML VIEW".

- Find the lines for the GPU and GPU-Audio, the line should start with "<hostdev" and a line later will end the section with "</hostdev".

- You should see 2 hostdev devices, there may be more. The first should specify your GPU BIOS you specified earlier.

- You need to change the the lines that specify the "<address" within the VM, not the "<address" outside the VM (in the host).

image.png.07ddb3038d9c4ace54e2f82f1f8eda38.png

- On the first line, change nothing, but add "multifunction='on'" to the end before the "/>" close.

- On the second line, change the "bus" and "function" to match the first line, but add 1 to the function.

image.png.f7b438996ca2f71e985d476a5823cc43.png

- When done editing, I recommend copying the whole xml and keeping a backup somewhere.

- Save the config by clicking "UPDATE" at the bottom.

- You're done.

 

Something to note is that if you change ANYTHING using the GUI, it will overwrite these pci XML modifications.

So, again I'd recommend keeping a backup somewhere in your unRAID.

 

With this, I was able to use my single nvidia card without issues.

 

Thanks again to @SpaceInvaderOne for figuring this out.

 

I hope this can help someone else. ☺️

Edited by KptnKMan
  • Like 1
Link to comment

Thanks @SpaceInvaderOne I appreciate the compliment.

Just hoping that this helps someone else.

 

I really hope that we could get a VM advanced UI for configuring the PCIe settings when using graphics card or sound card:

- multifunction on/off

- bus

- slot

- function

 

These are PCIe devices, so it seems to be a common setting, that would translate to the XML like the other UI settings, overwriting the default. This would go a long way to making this easier to fix.

Edited by KptnKMan
  • Like 1
Link to comment
  • 3 months later...
  • 6 months later...

An update for this thread.

 

I upgraded my system from the "EVGA 1080Ti FTW3 Gaming" to a "Gigabyte RTX3090 Turbo 24G" over this past weekend.

Using these instructions in this thread everything worked fine. I downloaded a new bios from techpowerup and hex-edited it, confirmed all the hostdev addresses and everything worked.

 

I hope this helps other people, post a reply if you found this useful.

Link to comment
  • 6 months later...

@KptnKMan Thanks for the comprehensive write up and share of experience.

 

I have been trying this for ages without luck. I have done the multifunction, bus and slot suggested without any luck... the screen just went blank.

I am on Asus X470 Pro Prime + Ryzen 2700X with Zotac GTX1060 6GB Amped version.

 

The VM works without passthrough via VNC, but with GPU passthrough, I dont think it even booted properly as I can't even RDP in. Not sure how to troubleshoot further.

 

any suggestion?

 

Link to comment
  • 1 month later...

So I'm not using unRAID but Manjaro with KVM. I haven't been able to check if the error 43 is present in the VM yet and I'm actually using a script to unbind and attach the GPU to the VM this actually worked for a while I'll be it slowly but almost directly after disabling my igpu in Manjaro it would boot (i know because my keyboard lights only start when the VM gets past post) but i wouldn't get any output from the card anymore. re-enabling the igpu did not solve anything. by any chance could the code 43 be related? I'll of course try this method as soon as I get the chance but I've already spent weeks on this so it would be nice if i could get a bit more insight

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.