KptnKMan Posted March 19, 2020 Share Posted March 19, 2020 (edited) Hi all, I'm hoping someone can help. I'm having trouble with passing through a GTX1080TI GPU. I am using model EVGA 11G-P4-6696-KR, known as "EVGA GeForce GTX 1080 Ti FTW3 GAMING". I had trouble finding a compatible BIOS on techpowerup, because there is no direct comparison. I saw that there are "FTW3 Elite Gaming" and "FTW3". I've tried the FTW3 BIOS, but not the FTW3 Elite, I'm concerned that it is not appropriate. So I dumped the bios in a bare metal Windows 10 install using nvflash, and have saved it to my array. Even still, I cannot seem to get my VM to boot with it using any configuration. The screen flashes and then no signal, every time. I need to reboot to get anything back. I was able to boot into Windows 10 using the bare metal install, so I think the card is working. I've watched the videos by SpaceInvader and I've passed through cards before using this system (An R9-290 and recently a RX-5600XT that I am returning). Both worked without much issue at all. Can anyone help with advice? I also dumped the version using nvflash: NVIDIA Firmware Update Utility (Version 5.590.0) Copyright (C) 1993-2019, NVIDIA Corporation. All rights reserved. Sign-On Message : GP102 PG611 SKU 50 VGA BIOS Build GUID : 402D5B72B8A34DFBBEB8A9C5CD99EE31 IFR Subsystem ID : 3842-6696 Subsystem Vendor ID : 0x3842 Subsystem ID : 0x6696 Version : 86.02.39.00.90 Image Hash : AAA37529FD718F13599221F885E83C25 Product Name : GP102 Board Device Name(s) : GeForce GTX 1080 Ti Board ID : 0xEE17 Vendor ID : 0x10DE Device ID : 0x1B06 Hierarchy ID : Normal Board Chip SKU : 350-0 Project : G611-0050 Build Date : 01/18/17 Modification Date : 04/17/17 UEFI Version : 0x30006 UEFI Variant ID : 0x0000000000000007 ( GP1xx ) UEFI Signer(s) : Microsoft Corporation UEFI CA 2011 XUSB-FW Version ID : N/A XUSB-FW Build Time : N/A InfoROM Version : G001.0000.01.04 InfoROM Backup : Not Present License Placeholder : Not Present GPU Mode : N/A Edited March 25, 2020 by KptnKMan Quote Link to comment
KptnKMan Posted March 19, 2020 Author Share Posted March 19, 2020 I believe that this should be the correct BIOS for me to use from techpowerup, as it has the same modified date as the build date on my BIOS (2017-04-17) and VBIOS Version (86.02.39.00.90) that I got from the dump. Still doesn't work. I'm not sure what else to do. Quote Link to comment
KptnKMan Posted March 19, 2020 Author Share Posted March 19, 2020 Another strange finding, is that my Windows 10 VM boots up and I can RDP to it. Looking around the system and forums more, it could be the "code 43" error. Quote Link to comment
KptnKMan Posted March 19, 2020 Author Share Posted March 19, 2020 (edited) I've dunped my BIOS out directly in unRAID, and trying to follow instructions to fix. I'm unsure if this means something but I've noticed when running "lspci v" is that there is no kernel driver listed next to my card: 0d:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1) (prog-if 00 [VGA controller]) Subsystem: eVga.com. Corp. GP102 [GeForce GTX 1080 Ti] Flags: bus master, fast devsel, latency 0, IRQ 5 Memory at f4000000 (32-bit, non-prefetchable) [size=16M] Memory at e0000000 (64-bit, prefetchable) [size=256M] Memory at f0000000 (64-bit, prefetchable) [size=32M] I/O ports at f000 [size=128] Expansion ROM at 000c0000 [disabled] [size=128K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Legacy Endpoint, MSI 00 Capabilities: [100] Virtual Channel Capabilities: [250] Latency Tolerance Reporting Capabilities: [128] Power Budgeting <?> Capabilities: [420] Advanced Error Reporting Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Capabilities: [900] Secondary PCI Express <?> 0d:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1) Subsystem: eVga.com. Corp. GP102 HDMI Audio Controller Flags: bus master, fast devsel, latency 0, IRQ 4 Memory at f5080000 (32-bit, non-prefetchable) [size=16K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Also, I'm unable to either unbind or bind the device: # echo "0000:0d:00.0" > /sys/bus/pci/drivers/vfio-pci/bind -bash: echo: write error: No such device # echo "0000:0d:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind -bash: echo: write error: No such device Edited March 19, 2020 by KptnKMan Quote Link to comment
KptnKMan Posted March 20, 2020 Author Share Posted March 20, 2020 (edited) Well, I figured it out. I spent many hours trying many different things, but this turned out to be it. Really wished that someone could have pointed me in the right direction, especially with so many current thread posts about the same thing on this forum. As such, I hope that I might be able to help someone else by summarising my journey and how I got there. A full solution is below, but first, I highly recommend watching these 2 excellent videos from @SpaceInvaderOne. Really saved my bacon: 👍 I'm basically summarising much of what he put in the video. My Summary: I was able to fix an already working VM, with my newly added GTX1080Ti using these methods. I didn't need to disable Hyper-V as others have said, and I'm using the latest nvidia drivers. In the end, the "code 43" error that is prevalent, seems to be due to a couple of things: - Modified BIOS, removing application headers from file, using Hex editor. - Specifying GPU pci device as "multifunction" in VM XML. Modifying BIOS: - Download the Hex Editor, install. - Download your compatible BIOS from techpowerup. - Open the bios file in hex editor. I'm using HxD like in the video. - Look for line that looks like "UªyëK7400éLwÌVIDEO", mine was prefaced with a series of "ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ" characters. - Next, delete everything above line "UªyëK7400éLwÌVIDEO", so that the matching offset reads "00000000" (8 zeros). - "Save As..." the file with a new filename. Save it somewhere on your unRAID system, like one of the shares. - I saved mine to "isos" share. You can save it anywhere on unRAID. - You're done. Modifying your VMs XML: - Before Starting, make sure your GPU is in its own IOMMU Group (Tools->System Devices). - If your GPU is not in its own IOMMU Group (Its fine if the GPU shares that group with the GPU-Audio). - Setup your VM in unRAID as normal. - I have had success using "Q35-4.2" Machine type, and "OVMF" BIOS. I left Hyper-V "on". - I'm using VirtIO Drivers "virtio-win-0.1.173-2" if that is helpful. - Specify your GPU and GPU-Audio Sound Card of the GPU. Always add your GPU-Audio, dont try to avoid it. You need it. - If you want to use your motherboard sound card, you should add it by clicking the green plus-sign on the left. - Either "CREATE" (Do not "Start VM after creation") or "UPDATE" the VM at the bottom, to set a baseline config. - Go back into the VM config. On the right change "FORM VIEW" to "XML VIEW". - Find the lines for the GPU and GPU-Audio, the line should start with "<hostdev" and a line later will end the section with "</hostdev". - You should see 2 hostdev devices, there may be more. The first should specify your GPU BIOS you specified earlier. - You need to change the the lines that specify the "<address" within the VM, not the "<address" outside the VM (in the host). - On the first line, change nothing, but add "multifunction='on'" to the end before the "/>" close. - On the second line, change the "bus" and "function" to match the first line, but add 1 to the function. - When done editing, I recommend copying the whole xml and keeping a backup somewhere. - Save the config by clicking "UPDATE" at the bottom. - You're done. Something to note is that if you change ANYTHING using the GUI, it will overwrite these pci XML modifications. So, again I'd recommend keeping a backup somewhere in your unRAID. With this, I was able to use my single nvidia card without issues. Thanks again to @SpaceInvaderOne for figuring this out. I hope this can help someone else. ☺️ Edited March 20, 2020 by KptnKMan 1 Quote Link to comment
SpaceInvaderOne Posted March 21, 2020 Share Posted March 21, 2020 Great write up. Thanks for taking the time for posting what worked for you. 1 Quote Link to comment
KptnKMan Posted March 24, 2020 Author Share Posted March 24, 2020 (edited) Thanks @SpaceInvaderOne I appreciate the compliment. Just hoping that this helps someone else. I really hope that we could get a VM advanced UI for configuring the PCIe settings when using graphics card or sound card: - multifunction on/off - bus - slot - function These are PCIe devices, so it seems to be a common setting, that would translate to the XML like the other UI settings, overwriting the default. This would go a long way to making this easier to fix. Edited March 24, 2020 by KptnKMan 1 Quote Link to comment
Stupifier Posted July 1, 2020 Share Posted July 1, 2020 Thanks @KptnKMan and @SpaceInvaderOne. I will be using this information to overhaul my old unraid build. I too think it would be amazing if Unraid devs decide to add all sorts of advanced VM configuration in the Web UI. 1 Quote Link to comment
KptnKMan Posted January 26, 2021 Author Share Posted January 26, 2021 An update for this thread. I upgraded my system from the "EVGA 1080Ti FTW3 Gaming" to a "Gigabyte RTX3090 Turbo 24G" over this past weekend. Using these instructions in this thread everything worked fine. I downloaded a new bios from techpowerup and hex-edited it, confirmed all the hostdev addresses and everything worked. I hope this helps other people, post a reply if you found this useful. Quote Link to comment
Dexmorgan Posted August 3, 2021 Share Posted August 3, 2021 @KptnKMan Thanks for the comprehensive write up and share of experience. I have been trying this for ages without luck. I have done the multifunction, bus and slot suggested without any luck... the screen just went blank. I am on Asus X470 Pro Prime + Ryzen 2700X with Zotac GTX1060 6GB Amped version. The VM works without passthrough via VNC, but with GPU passthrough, I dont think it even booted properly as I can't even RDP in. Not sure how to troubleshoot further. any suggestion? Quote Link to comment
KptnKMan Posted August 4, 2021 Author Share Posted August 4, 2021 @Dexmorgan You have not mentioned if you have created a compatible BIOS, but that's an important step. You hexed your BIOS like I did? Quote Link to comment
KC king collin Posted September 10, 2021 Share Posted September 10, 2021 So I'm not using unRAID but Manjaro with KVM. I haven't been able to check if the error 43 is present in the VM yet and I'm actually using a script to unbind and attach the GPU to the VM this actually worked for a while I'll be it slowly but almost directly after disabling my igpu in Manjaro it would boot (i know because my keyboard lights only start when the VM gets past post) but i wouldn't get any output from the card anymore. re-enabling the igpu did not solve anything. by any chance could the code 43 be related? I'll of course try this method as soon as I get the chance but I've already spent weeks on this so it would be nice if i could get a bit more insight Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.