SpaceInvaderOne Posted October 21, 2016 Share Posted October 21, 2016 Hi Guys i have made a video tutorial on how to pass through an nvidia gpu as the primary or only gpu in the server. This guide is based off hupsters great work on dumping the bios of the gpu. Hope you find it useful ! How to pass through an NVIDIA GPU as primary or only gpu in unRAID Here are some bios dumps from various Nvidia gpus GTX 5 Series evga 550ti 1GB https://www.dropbox.com/s/bbctm266ujctj4f/EVGA.GTX550ti.1024.110225.dump?dl=0 If using this please read here first http://lime-technology.com/forum/index.php?topic=52960.msg535568#msg535568 GTX 7 Series msi 7 50ti Twinfrozer https://www.dropbox.com/s/6cntzfu54qv2359/msi750titwinfrozer.dump?dl=0 evga GTX 760sc thanks to 1812) https://www.dropbox.com/s/yulm5ijodwko7jg/evgagtx760sc.dump?dl=0 GTX 9 Series gigabyte gtx960_2G https://www.dropbox.com/s/v6kvtdd5zxlmz02/Gigabytegtx960_GVN960IXOC_2GD.dump?dl=0 evga gtx970 SC https://www.dropbox.com/s/e2q89beqxjsehzz/evgagtx970sc.dump?dl=0 gigabyte gtx 970 https://www.dropbox.com/s/2iqoigtayh7pr5d/gigabyteGTX970.dump?dl=0 GTX 10 Series GTX 1050 Ti ZOTAC OC (thanks to darianf ) https://www.dropbox.com/s/8hxvgub2vxdq657/zotac1050tioc4g.dump?dl=0 GTX 1060 mini 3g Zotac(thanks to darianf ) https://www.dropbox.com/s/s6yhwl0kzksghcn/zotac1060mini3g.dump?dl=0 GTX1060_6gb EVGA (thanks to Brandur) https://www.dropbox.com/s/lyjczmwva573gx4/EVGA-GTX1060_6GB.dump?dl=0 GTX 1060 WINDFORCE OC 6g (thanks to @Lavoslav ) https://www.dropbox.com/s/6plqualklw8eptn/GV-N1060WF2OC-6GD.rom?dl=0 GTX1070 MSI founders edition https://www.dropbox.com/s/gtm917g5nnagpy6/msifoundersedition1070.dump?dl=0 GTX 1070 MSI seahawk . https://www.dropbox.com/s/8u4ryfqexh6ag0i/msiseahawk1070.dump?dl=0 GTX 1080 founders (thanks to heratic) https://www.dropbox.com/s/1f9ridkcmdrklv6/GTX1080FE.dump?dl=0 GTX 1080TI EVGA founders edition (thanks to heratic) . https://www.dropbox.com/s/f2xe3yi3c65kad7/1080ti.rom?dl=0 If anyone who has dumped their gpu bios could you please post here so we can get a collection on the site to help others as techpowerup bios doesnt work for us, nor to bios files dumped by gpuz How to edit a vbios from techpowerup using a hex editor @Matoking has written a script which will hex edit the vbios for you automatically for you. Please note he says it hasn't been tested extensively and warns this is used at your own risk https://github.com/Matoking/NVIDIA-vBIOS-VFIO-Patcher 4 3 Quote Link to comment
peter_sm Posted October 22, 2016 Share Posted October 22, 2016 Hi, I'm wonder if we should have rom bar='on' as well ? <rom bar='on' file='/mnt/appdata/vbios-VIDIA-GeForce-GTX-660/vbios.rom'/> //Peter Quote Link to comment
SpaceInvaderOne Posted October 22, 2016 Author Share Posted October 22, 2016 Hi, I'm wonder if we should have rom bar='on' as well ? <rom bar='on' file='/mnt/appdata/vbios-VIDIA-GeForce-GTX-660/vbios.rom'/> //Peter Hi there Peter, I dont think its necessary to specify that. Looking at the below from https://libvirt.org/formatdomain.html by default i think rombar is on anyway. rom The rom element is used to change how a PCI device's ROM is presented to the guest. The optional bar attribute can be set to "on" or "off", and determines whether or not the device's ROM will be visible in the guest's memory map. (In PCI documentation, the "rombar" setting controls the presence of the Base Address Register for the ROM). If no rom bar is specified, the qemu default will be used (older versions of qemu used a default of "off", while newer qemus have a default of "on"). Since 0.9.7 (QEMU and KVM only). The optional file attribute contains an absolute path to a binary file to be presented to the guest as the device's ROM BIOS. This can be useful, for example, to provide a PXE boot ROM for a virtual function of an sr-iov capable ethernet device (which has no boot ROMs for the VFs). Since 0.9.10 (QEMU and KVM only). Quote Link to comment
Aa617 Posted October 22, 2016 Share Posted October 22, 2016 gridrunner, Watched the video (great work on these vids btw, i check them out every friday) and had a couple questions for you. From the video I take it the 6450 is your primary graphics adapter for unraid and you also pass this card to vms? Have you seen any stability issues with this? I did the same with a RX380X however after several hours it would become unstable and text would become garbled. Aslo if I shut down the VM I could not passthrough the card again. Have you seen these issues with the 6450? Also have you tried using the Nvidia card for an extended period of time as the main graphics adapter as well as passing it to the VM? Any issues shutting down and restarting the VM. Thanks again... Quote Link to comment
SpaceInvaderOne Posted October 22, 2016 Author Share Posted October 22, 2016 gridrunner, Watched the video (great work on these vids btw, i check them out every friday) and had a couple questions for you. From the video I take it the 6450 is your primary graphics adapter for unraid and you also pass this card to vms? Have you seen any stability issues with this? I did the same with a RX380X however after several hours it would become unstable and text would become garbled. Aslo if I shut down the VM I could not passthrough the card again. Have you seen these issues with the 6450? Also have you tried using the Nvidia card for an extended period of time as the main graphics adapter as well as passing it to the VM? Any issues shutting down and restarting the VM. Thanks again... Hi glad you like the vids Yes i normally have the 6450 as the primary card for unRAID which i use for osx vms and openelec. I have run them for long periods of time without issue with the 6450 in seabios and ovmf vms. I have only used the 1070 as primary this last week but have had the vm on for a day or 2 just idle without shutting it down and was stable all that time, I gamed for about 1.5 hours during that time but nothing significant. I have never had probs with restarting vms except a few "one off occasions" if there has been a crash on vm then sometimes i have had to restart server. I would say it may be your rx380x having some issues. Maybe dump its rom and pass it through. Even though its not nvidia, passing the rom file can be very helpful with gpu problems. Quote Link to comment
alexciurea Posted October 26, 2016 Share Posted October 26, 2016 hello Is there performace loss when playing the game in the vm with the gtx1070 passthrough? possible to share some benchmark comparison, between native Windows 10 OS and W10 in VM, on same number of CPU cores allocated? thanks Quote Link to comment
SpaceInvaderOne Posted October 26, 2016 Author Share Posted October 26, 2016 hello Is there performace loss when playing the game in the vm with the gtx1070 passthrough? possible to share some benchmark comparison, between native Windows 10 OS and W10 in VM, on same number of CPU cores allocated? thanks There is no loss i can notice whilst playing but i will be able to do some benchmark comparisions at the weekend. I will post the results here. Quote Link to comment
KRSogaard Posted October 26, 2016 Share Posted October 26, 2016 hello Is there performace loss when playing the game in the vm with the gtx1070 passthrough? possible to share some benchmark comparison, between native Windows 10 OS and W10 in VM, on same number of CPU cores allocated? thanks There is no loss i can notice whilst playing but i will be able to do some benchmark comparisions at the weekend. I will post the results here. I am having problems passing though my GT 610, so i am going to try this when i get home. But will it work using the internal graphics? as my motherboard is ITX so it only have 1 pci-e port. Quote Link to comment
alexciurea Posted October 27, 2016 Share Posted October 27, 2016 thanks gridrunner @KRSogaard - my understanding is that you will use the integrated gpu for unraid and gt610 should passthrough regularly, without the steps mentioned by gridrunner. gridrunner's instructions are required when no GPU can be allocated to unraid. Quote Link to comment
kode54 Posted October 27, 2016 Share Posted October 27, 2016 GT 610 is a rebranded 5xx series card, so it will require Seabios, as I pointed out in another topic. 1 Quote Link to comment
ratosaude Posted December 13, 2016 Share Posted December 13, 2016 hi When I try to do this procedure on my GTX550ti I get the error below root@Frank:~# lspci -v | grep VGA 03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Park [Mobility Radeon HD 5430] (prog-if 00 [VG A controller]) 04:00.0 VGA compatible controller: NVIDIA Corporation GF116 [GeForce GTX 550 Ti] (rev a1) (prog-if 00 [VGA controller]) 0d:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 21) (prog-if 00 [VGA controller]) Flags: VGA palette snoop, medium devsel, IRQ 19, NUMA node 0 83:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] (rev c7) (prog-i f 00 [VGA controller]) 84:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper PRO [Radeon HD 6750] (prog-if 00 [VGA controller]) root@Frank:~# echo "0000:04:00.0" /sys/bus/pci/drivers/vfio-pci/unbind 0000:04:00.0 /sys/bus/pci/drivers/vfio-pci/unbind root@Frank:~# cd /sys/bus/pci/devices/0000:04:00.0/ root@Frank:/sys/bus/pci/devices/0000:04:00.0# root@Frank:/sys/bus/pci/devices/0000:04:00.0# cd /sys/bus/pci/devices/0000:04:00.0/ root@Frank:/sys/bus/pci/devices/0000:04:00.0# root@Frank:/sys/bus/pci/devices/0000:04:00.0# echo 1 > rom root@Frank:/sys/bus/pci/devices/0000:04:00.0# cat rom > /mnt/user/isos/gt550ti.dump cat: rom: Input/output error root@Frank:/sys/bus/pci/devices/0000:04:00.0# Quote Link to comment
SpaceInvaderOne Posted December 13, 2016 Author Share Posted December 13, 2016 hi When I try to do this procedure on my GTX550ti I get the error below root@Frank:~# lspci -v | grep VGA 03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Park [Mobility Radeon HD 5430] (prog-if 00 [VG A controller]) 04:00.0 VGA compatible controller: NVIDIA Corporation GF116 [GeForce GTX 550 Ti] (rev a1) (prog-if 00 [VGA controller]) 0d:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 21) (prog-if 00 [VGA controller]) Flags: VGA palette snoop, medium devsel, IRQ 19, NUMA node 0 83:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] (rev c7) (prog-i f 00 [VGA controller]) 84:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper PRO [Radeon HD 6750] (prog-if 00 [VGA controller]) root@Frank:~# echo "0000:04:00.0" /sys/bus/pci/drivers/vfio-pci/unbind 0000:04:00.0 /sys/bus/pci/drivers/vfio-pci/unbind root@Frank:~# cd /sys/bus/pci/devices/0000:04:00.0/ root@Frank:/sys/bus/pci/devices/0000:04:00.0# root@Frank:/sys/bus/pci/devices/0000:04:00.0# cd /sys/bus/pci/devices/0000:04:00.0/ root@Frank:/sys/bus/pci/devices/0000:04:00.0# root@Frank:/sys/bus/pci/devices/0000:04:00.0# echo 1 > rom root@Frank:/sys/bus/pci/devices/0000:04:00.0# cat rom > /mnt/user/isos/gt550ti.dump cat: rom: Input/output error root@Frank:/sys/bus/pci/devices/0000:04:00.0# Ok. Check that when you are dumping the bios it is in the secondary slot. Not primary as in showing the unraid console. If it isnt secondary then make changes and try again. If was already in secondary then assign the card to a vm and start the vm. Dont shut it down. Do the same prodedure as you did above and dump the bios but whilst the vm is running with that card. Quote Link to comment
ratosaude Posted December 13, 2016 Share Posted December 13, 2016 Running on VGA I managed to create a ROM, however when I shutdown to apply on the XML, the VM didn't start until I rebooted the server Quote Link to comment
sumonpathak Posted December 14, 2016 Share Posted December 14, 2016 Hi Guys i have made a video tutorial on how to pass through an nvidia gpu as the primary or only gpu in the server. This guide is based off hupsters great work on dumping the bios of the gpu. Hope you find it useful ! How to pass through an NVIDIA GPU as primary or only gpu in unRAID Hello, can you guide me how to do this on a windows based system? will the same commands apply? Quote Link to comment
JonathanM Posted December 15, 2016 Share Posted December 15, 2016 Hello, can you guide me how to do this on a windows based system? will the same commands apply? He demonstrated it using a Windows 10 VM. What exactly are you asking? Quote Link to comment
sumonpathak Posted December 15, 2016 Share Posted December 15, 2016 Hello, can you guide me how to do this on a windows based system? will the same commands apply? He demonstrated it using a Windows 10 VM. What exactly are you asking? It looked like a mac guess i got it all wrong. Quote Link to comment
SpaceInvaderOne Posted December 15, 2016 Author Share Posted December 15, 2016 Hello, can you guide me how to do this on a windows based system? will the same commands apply? He demonstrated it using a Windows 10 VM. What exactly are you asking? It looked like a mac guess i got it all wrong. Hi Guys yes, you are both right. The VM that I was adding the 1070 GPU to was a windows 10 VM. But the computer that I was accessing the unraid gui and ssh into the server to dump the vbios was a mac. So I guess, you are asking if its the same from a windows computer to dump the bios? Yes exactly the same. Even though I am using a mac to access the server, I am not dumping the bios from the mac. And if you were using a windows computer you would not be using the windows computer to dump the bios. Both computers are using the server (Linux) by use of a remote telnet/ssh connection to get the vbios. So if you are using windows make an ssh/telnet connection using a programme like putty and do the exact same steps/commands Quote Link to comment
darianf Posted January 16, 2017 Share Posted January 16, 2017 hi, this is a very good tutorial, thanks gridrunner for all the steps I was able to extract the rom for my zotac 1060 mini 3g card, no issues with it. But now i'm facing an issue with my evga 550ti 1gb, placed in second PCIe slot (with 1060 in first slot). If i just start the unraid box, without any VM auto running, and try the steps for rom creation (so lspci -v does not show that line with driver in use for vfio-pcie for 550 ti - device 02:00.0), i get: cat: rom: Input/output error Same happens if i start and stop the VM. If i keep the VM started (550ti in second slot passed through) - btw, only OVMF bios works - then lspci -v shows me that line with the driver in use. I try to unbind, but no luck (command hangs). Only if i stop the VM, then the command returns the prompt. I continue with the steps for rom dumping, but again i get cat: rom: Input/output error Somehting to note, after a start/stop attempt, the VM does not start anymore, due to some GPU invalid rom error - it asks to check dmesg for more info and i get "Invalid ROM" or somehting similar. Therefore I have to reboot the whole unraid box to make it start again. In short, I'm out of options for getting the rom for this 550ti, as i tried various options: 1) immidiately after box restart, VM never started since restart of box - input/output error 2) fresh box restart, VM started and then stopped - input/output error 3) fresh box start, while VM is online - unbind hangs. Then i stop the VM, unbind runs (actually not really sure if it runs, because no message is shown as output, iut just returns to the prompt) but the cat rom fails again with input/output error. All these after fresh restarts of the physical box... What to do next, what suggestion should i try? i see some ACS suggestion, but not sure if relevant or what exactly should i change (not familiar with the concept). Or should i buy another card instead of my old 550ti? any success with the low cost 710/720 cards? thanks -d Quote Link to comment
SpaceInvaderOne Posted January 16, 2017 Author Share Posted January 16, 2017 The the rom file is usually read-protected unless '1' is written to it thats why we do the echo 1 But it seems your 550 rom file isnt being read for some reason. Try dumping the rom starting the vm whilst its running. Dont try to unbind it see if you can dump it anyway. On my system i can. If not then for some reason you cant read the bios on that card. If you are using the 1060 in slot one why the need for the rom file for the 550? Quote Link to comment
darianf Posted January 16, 2017 Share Posted January 16, 2017 thanks now that you say it, i'm not confident if i already tried to dump while VM up, without unbind.. Will update quickly, once i try it out (hopefully today evening) Why the need? i have matx with x99, like you. So no IGP, and only 2 PCIe slots (actually 3, but last one is not usable to fit a GPU, since the matx case would not allow). I'm planning to run 2 VM's with both the GPU's passthrough. And, i want to put the 550ti in my first slot, because i plan to work it less (will have less airflow). Putting 1060 on second slot will give it access to fresh aiflow & will run cooler. -d Quote Link to comment
Junnu Posted January 16, 2017 Share Posted January 16, 2017 Hmm this trick did not work for me. I was able follow instructions and create rom, but even adding it screen goes blank when starting up VM. Anyway what I did notice was that ROM created by dump is only 58 880 bytes large, but if downloading same ROM techpowerup.com or using nvflash to get rom file size 256 512 bytes (for MSI GTX 1070 8192 MB Gaming X) Quote Link to comment
darianf Posted January 16, 2017 Share Posted January 16, 2017 junnu, in fact the rom i downloaded from techpowerup is around 58-60 KB for my 550ti My understanding is that it should be around 128KB? And usually the ones from techpowerup are around 256KB, which ppl are saying they are not good, because they are shadowed roms and will not work for passing through the gpu to vm... maybe the age of the card also matters... maybe older cards are smaller, newer are bigger... just my understanding, don't take it for granted. -d Quote Link to comment
SpaceInvaderOne Posted January 16, 2017 Author Share Posted January 16, 2017 junnu, in fact the rom i downloaded from techpowerup is around 58-60 KB for my 550ti My understanding is that it should be around 128KB? And usually the ones from techpowerup are around 256KB, which ppl are saying they are not good, because they are shadowed roms and will not work for passing through the gpu to vm... maybe the age of the card also matters... maybe older cards are smaller, newer are bigger... just my understanding, don't take it for granted. -d I have never been able to get the techpowerup downloaded roms to work for me. I have in the last year tried about 5 differnet gpus and non worked. You dont have to dump the 550ti bios to get it to work how you want. Well I dont think you do! You have an x99 motherboard like me. So in mine in the bios i can set any of my 3 pcie slots as primary. So with the 1060 in your second slot set that to the primary boot pcie slot. As you have the rom for that you can pass that through. Then put your 550ti in the first slot and it will work becuase the first slot will no longer be the primary slot. darianf could you attach the rom for the 1060 and put exactly which card it is. I am trying to collect as many rom files as possible to post here Hmm this trick did not work for me. I was able follow instructions and create rom, but even adding it screen goes blank when starting up VM. Anyway what I did notice was that ROM created by dump is only 58 880 bytes large, but if downloading same ROM techpowerup.com or using nvflash to get rom file size 256 512 bytes (for MSI GTX 1070 8192 MB Gaming X) Junnu I have a 1070 msi try this bios i dumped should work for you. msi 1070 founders edition https://www.dropbox.com/s/gtm917g5nnagpy6/msifoundersedition1070.dump?dl=0 msi 1070 seahawk . https://www.dropbox.com/s/8u4ryfqexh6ag0i/msiseahawk1070.dump?dl=0 Quote Link to comment
Junnu Posted January 16, 2017 Share Posted January 16, 2017 Junnu I have a 1070 msi try this bios i dumped should work for you. msi 1070 founders edition https://www.dropbox.com/s/gtm917g5nnagpy6/msifoundersedition1070.dump?dl=0 msi 1070 seahawk . https://www.dropbox.com/s/8u4ryfqexh6ag0i/msiseahawk1070.dump?dl=0 Hmm, worked straight way those bios dumps so big thanks So question is why bios I dumped does not work? I did not got any errors during dumping it. Quote Link to comment
darianf Posted January 16, 2017 Share Posted January 16, 2017 so i tried after fresh restart, with the VM up, without unbind. i still get the input/output error. i also tried creating a new VM, this time with seabios (previous attempts were with OVMF - so i believe seabios is recommended for the non uefi gpu's like 550ti). Restarted the box, then starting this VM, performing the isntalltion of the Ubuntu properly. Restarted once again the box, started the VM, then tried to dump the rom without unbinding, same issue - i/o error. Tried also to unbind, again cannot unbind while VM up. after start of VM and stopped, any attempts to start the VM fails and i get black screen output - in the VM log i see this: 2017-01-16T22:07:26.847146Z qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:02:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile= in the dmesg, i see this: [ 159.336702] vfio-pci 0000:02:00.0: Invalid ROM contents [ 176.687421] vfio-pci 0000:02:00.0: Relaying device request to user (#0) [ 216.226455] vfio-pci 0000:02:00.0: Device is currently in use, task "bash" (13024) blocked until device is released the above i assume is during unbind. other messages in dmesg are like: [ 297.524992] pci 0000:02:00.0: Invalid ROM contents [ 337.653449] vgaarb: device changed decodes: PCI:0000:02:00.0,olddecodes=io+mem,decodes=io+mem:owns=io+mem which are probably my attempts to restart the VM (and failing, with black screen) will come back with details of the rom for Zotac 1060 mini 3gb... Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.