Jump to content
SpaceInvaderOne

VIDEO GUIDE***How to pass through an NVIDIA GPU as primary or only gpu in unRAID

130 posts in this topic Last Reply

Recommended Posts

Hi Guys i have made a video tutorial on how to pass through an nvidia gpu as the primary or only gpu in the server. This guide is based off hupsters great work on dumping the bios of the gpu.

Hope you find it useful ! :)

 

How to pass through an NVIDIA GPU as primary or only gpu in unRAID

 

 

 

Here are some bios dumps from various Nvidia gpus

 

 

GTX 5 Series

 

evga 550ti 1GB                                            https://www.dropbox.com/s/bbctm266ujctj4f/EVGA.GTX550ti.1024.110225.dump?dl=0

If using this please read here first                http://lime-technology.com/forum/index.php?topic=52960.msg535568#msg535568

 

GTX 7 Series

 

msi 7 50ti Twinfrozer                                    https://www.dropbox.com/s/6cntzfu54qv2359/msi750titwinfrozer.dump?dl=0

 

evga GTX 760sc thanks to 1812)                  https://www.dropbox.com/s/yulm5ijodwko7jg/evgagtx760sc.dump?dl=0

 

GTX 9 Series

 

gigabyte gtx960_2G                                    https://www.dropbox.com/s/v6kvtdd5zxlmz02/Gigabytegtx960_GVN960IXOC_2GD.dump?dl=0

 

evga gtx970 SC                                          https://www.dropbox.com/s/e2q89beqxjsehzz/evgagtx970sc.dump?dl=0

 

gigabyte gtx 970                                          https://www.dropbox.com/s/2iqoigtayh7pr5d/gigabyteGTX970.dump?dl=0

 

 

GTX 10 Series

 

GTX 1050 Ti ZOTAC OC (thanks to darianf )           https://www.dropbox.com/s/8hxvgub2vxdq657/zotac1050tioc4g.dump?dl=0

 

GTX 1060 mini 3g Zotac(thanks to darianf )            https://www.dropbox.com/s/s6yhwl0kzksghcn/zotac1060mini3g.dump?dl=0

 

GTX1060_6gb  EVGA   (thanks to Brandur)            https://www.dropbox.com/s/lyjczmwva573gx4/EVGA-GTX1060_6GB.dump?dl=0

 

GTX 1060 WINDFORCE OC 6g (thanks to @Lavoslav )  https://www.dropbox.com/s/6plqualklw8eptn/GV-N1060WF2OC-6GD.rom?dl=0

 

GTX1070 MSI founders edition                                https://www.dropbox.com/s/gtm917g5nnagpy6/msifoundersedition1070.dump?dl=0

 

GTX 1070 MSI seahawk .                                          https://www.dropbox.com/s/8u4ryfqexh6ag0i/msiseahawk1070.dump?dl=0

 

GTX 1080 founders  (thanks to heratic)                    https://www.dropbox.com/s/1f9ridkcmdrklv6/GTX1080FE.dump?dl=0

 

GTX 1080TI EVGA founders edition  (thanks to heratic) .        https://www.dropbox.com/s/f2xe3yi3c65kad7/1080ti.rom?dl=0

 

 

 

If anyone who has dumped their gpu bios could you please post here so we can get a collection on the site to help others as

techpowerup bios doesnt work for us, nor to bios files dumped by gpuz

 

How to edit a vbios from techpowerup using a hex editor

 

 

@Matoking  has written a script which will hex edit the vbios for you automatically for you. Please note he says it hasn't been tested extensively and warns this is used at your own risk

https://github.com/Matoking/NVIDIA-vBIOS-VFIO-Patcher

  • Like 3
  • Upvote 3

Share this post


Link to post

Hi,

 

I'm wonder if we should have rom bar='on' as well ?

 

<rom bar='on' file='/mnt/appdata/vbios-VIDIA-GeForce-GTX-660/vbios.rom'/>

 

//Peter

Share this post


Link to post

Hi,

 

I'm wonder if we should have rom bar='on' as well ?

 

<rom bar='on' file='/mnt/appdata/vbios-VIDIA-GeForce-GTX-660/vbios.rom'/>

 

//Peter

 

Hi there Peter, I dont think its necessary to specify that. Looking at the below from https://libvirt.org/formatdomain.html by default i think rombar is on anyway.

 

rom

The rom element is used to change how a PCI device's ROM is presented to the guest. The optional bar attribute can be set to "on" or "off", and determines whether or not the device's ROM will be visible in the guest's memory map. (In PCI documentation, the "rombar" setting controls the presence of the Base Address Register for the ROM). If no rom bar is specified, the qemu default will be used (older versions of qemu used a default of "off", while newer qemus have a default of "on"). Since 0.9.7 (QEMU and KVM only). The optional file attribute contains an absolute path to a binary file to be presented to the guest as the device's ROM BIOS. This can be useful, for example, to provide a PXE boot ROM for a virtual function of an sr-iov capable ethernet device (which has no boot ROMs for the VFs). Since 0.9.10 (QEMU and KVM only).

 

 

 

 

Share this post


Link to post

gridrunner,

 

Watched the video (great work on these vids btw, i check them out every friday) and had a couple questions for you. From the video I take it the 6450 is your primary graphics adapter for unraid and you also pass this card to vms? Have you seen any stability issues with this? I did the same with a RX380X however after several hours it would become unstable and text would become garbled. Aslo if I shut down the VM I could not passthrough the card again. Have you seen these issues with the 6450? Also have you tried using the Nvidia card for an extended period of time as the main graphics adapter as well as passing it to the VM? Any issues shutting down and restarting the VM. Thanks again...

Share this post


Link to post

gridrunner,

 

Watched the video (great work on these vids btw, i check them out every friday) and had a couple questions for you. From the video I take it the 6450 is your primary graphics adapter for unraid and you also pass this card to vms? Have you seen any stability issues with this? I did the same with a RX380X however after several hours it would become unstable and text would become garbled. Aslo if I shut down the VM I could not passthrough the card again. Have you seen these issues with the 6450? Also have you tried using the Nvidia card for an extended period of time as the main graphics adapter as well as passing it to the VM? Any issues shutting down and restarting the VM. Thanks again...

 

Hi glad you like the vids  :)  Yes i normally have the 6450 as the  primary card for unRAID which i use for osx vms and openelec. I have run them for long periods of time without issue with the 6450 in seabios and ovmf vms.

I have only used the  1070 as primary this last week but have had the vm on for a day or 2 just idle without shutting it down and was stable all that time, I gamed for about 1.5 hours during that time but nothing significant.

I have never had probs with restarting vms except a few "one off occasions" if there has been a crash on vm then sometimes i have had to restart server.

I would say it may be your rx380x having some issues. Maybe dump its rom and pass it through. Even though its not nvidia, passing the rom file can be very helpful with gpu problems.

Share this post


Link to post

hello

 

Is there performace loss when playing the game in the vm with the gtx1070 passthrough?

 

possible to share some benchmark comparison, between native Windows 10 OS and W10 in VM, on same number of CPU cores allocated?

 

thanks

Share this post


Link to post

hello

 

Is there performace loss when playing the game in the vm with the gtx1070 passthrough?

 

possible to share some benchmark comparison, between native Windows 10 OS and W10 in VM, on same number of CPU cores allocated?

 

thanks

 

There is no loss i can notice whilst playing but i will be able to do some benchmark comparisions at the weekend. I will post the results here.

Share this post


Link to post

hello

 

Is there performace loss when playing the game in the vm with the gtx1070 passthrough?

 

possible to share some benchmark comparison, between native Windows 10 OS and W10 in VM, on same number of CPU cores allocated?

 

thanks

 

There is no loss i can notice whilst playing but i will be able to do some benchmark comparisions at the weekend. I will post the results here.

 

I am having problems passing though my GT 610, so i am going to try this when i get home. But will it work using the internal graphics? as my motherboard is ITX so it only have 1 pci-e port.

Share this post


Link to post

thanks gridrunner

@KRSogaard - my understanding is that you will use the integrated gpu for unraid and gt610 should passthrough regularly, without the steps mentioned by gridrunner.

gridrunner's instructions are required when no GPU can be allocated to unraid.

Share this post


Link to post

GT 610 is a rebranded 5xx series card, so it will require Seabios, as I pointed out in another topic.

Share this post


Link to post

hi

 

When I try to do this procedure on my GTX550ti I get the error below

 

root@Frank:~# lspci -v | grep VGA
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Park [Mobility Radeon HD 5430] (prog-if 00 [VG
A controller])
04:00.0 VGA compatible controller: NVIDIA Corporation GF116 [GeForce GTX 550 Ti] (rev a1) (prog-if 00 [VGA controller])
0d:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 21) (prog-if 00 [VGA controller])
        Flags: VGA palette snoop, medium devsel, IRQ 19, NUMA node 0
83:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] (rev c7) (prog-i
f 00 [VGA controller])
84:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper PRO [Radeon HD 6750] (prog-if 00 [VGA
controller])
root@Frank:~# echo "0000:04:00.0"  /sys/bus/pci/drivers/vfio-pci/unbind
0000:04:00.0 /sys/bus/pci/drivers/vfio-pci/unbind
root@Frank:~#  cd /sys/bus/pci/devices/0000:04:00.0/
root@Frank:/sys/bus/pci/devices/0000:04:00.0#
root@Frank:/sys/bus/pci/devices/0000:04:00.0# cd /sys/bus/pci/devices/0000:04:00.0/
root@Frank:/sys/bus/pci/devices/0000:04:00.0#
root@Frank:/sys/bus/pci/devices/0000:04:00.0# echo 1 > rom
root@Frank:/sys/bus/pci/devices/0000:04:00.0# cat rom > /mnt/user/isos/gt550ti.dump

cat: rom: Input/output error
root@Frank:/sys/bus/pci/devices/0000:04:00.0#

Share this post


Link to post

hi

 

When I try to do this procedure on my GTX550ti I get the error below

 

root@Frank:~# lspci -v | grep VGA
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Park [Mobility Radeon HD 5430] (prog-if 00 [VG
A controller])
04:00.0 VGA compatible controller: NVIDIA Corporation GF116 [GeForce GTX 550 Ti] (rev a1) (prog-if 00 [VGA controller])
0d:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 21) (prog-if 00 [VGA controller])
        Flags: VGA palette snoop, medium devsel, IRQ 19, NUMA node 0
83:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] (rev c7) (prog-i
f 00 [VGA controller])
84:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper PRO [Radeon HD 6750] (prog-if 00 [VGA
controller])
root@Frank:~# echo "0000:04:00.0"  /sys/bus/pci/drivers/vfio-pci/unbind
0000:04:00.0 /sys/bus/pci/drivers/vfio-pci/unbind
root@Frank:~#  cd /sys/bus/pci/devices/0000:04:00.0/
root@Frank:/sys/bus/pci/devices/0000:04:00.0#
root@Frank:/sys/bus/pci/devices/0000:04:00.0# cd /sys/bus/pci/devices/0000:04:00.0/
root@Frank:/sys/bus/pci/devices/0000:04:00.0#
root@Frank:/sys/bus/pci/devices/0000:04:00.0# echo 1 > rom
root@Frank:/sys/bus/pci/devices/0000:04:00.0# cat rom > /mnt/user/isos/gt550ti.dump

cat: rom: Input/output error
root@Frank:/sys/bus/pci/devices/0000:04:00.0#

 

Ok. Check that when you are dumping the bios it is in the secondary slot. Not primary as in showing the unraid console.

If it isnt secondary then make changes and try again.

 

If was already in secondary then assign the card to a vm and start the vm. Dont shut it down.

Do the same prodedure as you did above and dump the bios but whilst the vm is running with that card.

Share this post


Link to post

Running on VGA I managed to create a ROM, however when I shutdown to apply on the XML, the VM didn't start until I rebooted the server

Share this post


Link to post

Hi Guys i have made a video tutorial on how to pass through an nvidia gpu as the primary or only gpu in the server. This guide is based off hupsters great work on dumping the bios of the gpu.

Hope you find it useful ! :)

 

How to pass through an NVIDIA GPU as primary or only gpu in unRAID

 

 

 

 

Hello,

can you guide me how to do this on a windows based system?

will the same commands apply?

Share this post


Link to post

 

Hello,

can you guide me how to do this on a windows based system?

will the same commands apply?

He demonstrated it using a Windows 10 VM. What exactly are you asking?

Share this post


Link to post

 

Hello,

can you guide me how to do this on a windows based system?

will the same commands apply?

He demonstrated it using a Windows 10 VM. What exactly are you asking?

 

 

It looked like a mac :(

 

guess i got it all wrong.

Share this post


Link to post

 

Hello,

can you guide me how to do this on a windows based system?

will the same commands apply?

He demonstrated it using a Windows 10 VM. What exactly are you asking?

 

 

It looked like a mac :(

 

guess i got it all wrong.

 

Hi Guys yes, you are both right. The VM that I was adding the 1070 GPU to was a windows 10 VM.

But the computer that I was accessing the unraid gui and ssh into the server to dump the vbios was a mac.

 

So I guess, you are asking if its the same from a windows computer to dump the bios?

Yes exactly the same.

Even though I am using a mac to access the server, I am not dumping the bios from the mac.

And if you were using a windows computer you would not be using the windows computer to dump the bios.

Both computers are using the server (Linux) by use of a remote telnet/ssh connection to get the vbios.

So if you are using windows make an ssh/telnet connection using a programme like putty and do the exact same steps/commands  :)

Share this post


Link to post

hi, this is a very good tutorial, thanks gridrunner for all the steps :)

 

I was able to extract the rom for my zotac 1060 mini 3g card, no issues with it.

 

But now i'm facing an issue with my evga 550ti 1gb, placed in second PCIe slot (with 1060 in first slot).

 

If i just start the unraid box, without any VM auto running, and try the steps for rom creation (so lspci -v does not show that line with driver in use for vfio-pcie for 550 ti - device 02:00.0), i get:

 

cat: rom: Input/output error

 

 

Same happens if i start and stop the VM.

 

If i keep the VM started (550ti in second slot passed through) - btw, only OVMF bios works - then lspci -v shows me that line with the driver in use.

I try to unbind, but no luck (command hangs). Only if i stop the VM, then the command returns the prompt.

I continue with the steps for rom dumping, but again i get

 

cat: rom: Input/output error

 

Somehting to note, after a start/stop attempt, the VM does not start anymore, due to some GPU invalid rom error - it asks to check dmesg for more info and i get "Invalid ROM" or somehting similar. Therefore I have to reboot the whole unraid box to make it start again.

 

In short, I'm out of options for getting the rom for this 550ti, as i tried various options:

1) immidiately after box restart, VM never started since restart of box - input/output error

2) fresh box restart, VM started and then stopped - input/output error

3) fresh box start, while VM is online - unbind hangs. Then i stop the VM, unbind runs (actually not really sure if it runs, because no message is shown as output, iut just returns to the prompt) but the cat rom fails again with input/output error.

All these after fresh restarts of the physical box...

 

What to do next, what suggestion should i try? i see some ACS suggestion, but not sure if relevant or what exactly should i change (not familiar with the concept).

Or should i buy another card instead of my old 550ti? any success with the low cost 710/720 cards?

 

thanks

-d

 

 

Share this post


Link to post

The the rom file is usually read-protected unless '1' is written to it thats why we do the echo 1

But it seems your 550 rom file isnt being read for some reason.

 

Try dumping the rom starting the vm whilst its running. Dont try to unbind it see if you can dump it anyway. On my system i can.

If not then for some reason you cant read the bios on that card.

 

If you are using the 1060 in slot one why the need for the rom file for the 550?

 

Share this post


Link to post

thanks :)

 

now that you say it, i'm not confident if i already tried to dump while VM up, without unbind..

Will update quickly, once i try it out (hopefully today evening)

 

Why the need? i have matx with x99, like you. So no IGP, and only 2 PCIe slots (actually 3, but last one is not usable to fit a GPU, since the matx case would not allow).

I'm planning to run 2 VM's with both the GPU's passthrough.

And, i want to put the 550ti in my first slot, because i plan to work it less (will have less airflow). Putting 1060 on second slot will give it access to fresh aiflow & will run cooler.

-d

Share this post


Link to post

Hmm this trick did not work for me. I was able follow instructions and create rom, but even adding it screen goes blank when starting up VM. Anyway what I did notice was that ROM created by dump is only 58 880 bytes large, but if downloading same ROM techpowerup.com or using nvflash to get rom file size 256 512 bytes  (for MSI GTX 1070 8192 MB

Gaming X)

 

 

Share this post


Link to post

junnu, in fact the rom i downloaded from techpowerup is around 58-60 KB for my 550ti :)

My understanding is that it should be around 128KB?

 

And usually the ones from techpowerup are around 256KB, which ppl are saying they are not good, because they are shadowed roms and will not work for passing through the gpu to vm...

 

maybe the age of the card also matters... maybe older cards are smaller, newer are bigger...

 

just my understanding, don't take it for granted.

-d

 

Share this post


Link to post

junnu, in fact the rom i downloaded from techpowerup is around 58-60 KB for my 550ti :)

My understanding is that it should be around 128KB?

 

And usually the ones from techpowerup are around 256KB, which ppl are saying they are not good, because they are shadowed roms and will not work for passing through the gpu to vm...

 

maybe the age of the card also matters... maybe older cards are smaller, newer are bigger...

 

just my understanding, don't take it for granted.

-d

 

I have never been able to get the techpowerup downloaded roms to work for me. I have in the last year tried about 5 differnet gpus and non worked.

 

You dont have to dump the 550ti bios to get it to work how you want.

Well I dont think you do!

You have an x99 motherboard like me. So in mine in the bios i can set any of my 3 pcie slots as primary. So with the 1060 in your second slot set that to the primary boot pcie slot. As you have the rom for that you can pass that through. Then put your 550ti in the first slot and it will work becuase the first slot will no longer be the primary slot.

 

darianf could you attach the rom for the 1060 and put exactly which card it is. I am trying to collect as many rom files as possible to

post here :)

 

Hmm this trick did not work for me. I was able follow instructions and create rom, but even adding it screen goes blank when starting up VM. Anyway what I did notice was that ROM created by dump is only 58 880 bytes large, but if downloading same ROM techpowerup.com or using nvflash to get rom file size 256 512 bytes  (for MSI GTX 1070 8192 MB

Gaming X)

 

Junnu I have a 1070 msi try this bios i dumped should work for you.

msi 1070 founders edition    https://www.dropbox.com/s/gtm917g5nnagpy6/msifoundersedition1070.dump?dl=0

 

msi 1070 seahawk .              https://www.dropbox.com/s/8u4ryfqexh6ag0i/msiseahawk1070.dump?dl=0

 

 

 

 

 

Share this post


Link to post

 

Junnu I have a 1070 msi try this bios i dumped should work for you.

msi 1070 founders edition    https://www.dropbox.com/s/gtm917g5nnagpy6/msifoundersedition1070.dump?dl=0

 

msi 1070 seahawk .              https://www.dropbox.com/s/8u4ryfqexh6ag0i/msiseahawk1070.dump?dl=0

 

Hmm, worked straight way those bios dumps so big thanks :)  So question is why bios I dumped does not work? I did not got any errors during dumping it.

Share this post


Link to post

so i tried after fresh restart, with the VM up, without unbind.

i still get the input/output error.

 

i also tried creating a new VM, this time with seabios (previous attempts were with OVMF - so i believe seabios is recommended for the non uefi gpu's like 550ti).

Restarted the box, then starting this VM, performing the isntalltion of the Ubuntu properly.

Restarted once again the box, started the VM, then tried to dump the rom without unbinding, same issue - i/o error.

 

Tried also to unbind, again cannot unbind while VM up.

 

after start of VM and stopped, any attempts to start the VM fails and i get black screen output - in the VM log i see this:

 

2017-01-16T22:07:26.847146Z qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:02:00.0
Device option ROM contents are probably invalid (check dmesg).
Skip option ROM probe with rombar=0, or load from file with romfile=

 

in the dmesg, i see this:

 

[  159.336702] vfio-pci 0000:02:00.0: Invalid ROM contents
[  176.687421] vfio-pci 0000:02:00.0: Relaying device request to user (#0)
[  216.226455] vfio-pci 0000:02:00.0: Device is currently in use, task "bash" (13024) blocked until device is released

 

the above i assume is during unbind.

 

other messages in dmesg are like:

 

[  297.524992] pci 0000:02:00.0: Invalid ROM contents
[  337.653449] vgaarb: device changed decodes: PCI:0000:02:00.0,olddecodes=io+mem,decodes=io+mem:owns=io+mem

which are probably my attempts to restart the VM (and failing, with black screen)

 

will come back with details of the rom for Zotac 1060 mini 3gb...

 

 

 

 

 

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.