VIDEO GUIDE***How to pass through an NVIDIA GPU as primary or only gpu in unRAID


SpaceInvaderOne

Recommended Posts

@gridrunner - thank you for the awesome tutorials :)

 

@rahool - i assume your friend has unraid already (or similar meanings to dump), so yes... 

The rom is specific to the GPU model/vendor, so it's independent of your/your friend's PC.

 

No harm, it will either not work to pass through the GPU, or will not display the video output [properly], or worst case will freeze the unraid box and you'll probably need to powercycle/reset it (e.g. power your PC down, then power it up).

 

My suggestion is to try first what i dumped, before attempting to do it yourself...

Just so that you know, my card is the mini version (exact item model number ZT-P10610A-10L) and based on your SFF intention i assume this is the one you want.

https://www.newegg.com/Product/Product.aspx?Item=N82E16814500405&cm_re=zotac_1060_mini_3g-_-14-500-405-_-Product

Saying this, because there are other 3gb versions available, like AMP (at least i see this model in europe market), but who knows, probably they have same rom...

-d

 

Link to comment

Is it possible to change primary card of Unraid6 if you have two graphic cards? I have a watercooled setup and it is hard for me to change their positions to get the second card in the upper slot... The whole thing should be still possible when i manage to use the second card for unraid first?!

 

Thanks for help!

Link to comment

@hblockx - some motherboards allow selection of primary GPU, from a bios option. Some add this option only in newer versions of the bios.

Mine does not (asus x99-m ws) yet and as a result, the slot 1 card is used to output the boot vga, and this is the card where unraid console is displayed.

Once i start a VM using this primary card, the console output is taken for displaying the video output of that VM.

And it's never returned back to unraid, even if VM stopped, until the unraid gets restarted.

 

Link to comment

I do not think i am going to have the option, too. Is there a way to do this with a different linux? Maybe a linux which does not even need a graphics card for terminal (just SSH)? Without any way i cannot get this done :'(

Edited by hblockx
Link to comment

@hblockx I don't know of a way of booting the system with a graphics card present in the machine and for the card not to be shadowed.

 

If you cant move the cards I see you have two options.

1. Post what GPU you have here and hopefully someone with the same GPU as you would dump the rom for you to use.

 

2 Un seat both GPUs. Use a PCIe riser cable to be able to plug the primary card into the secondary slot. Then use another cable to plug any other card into the first slot.

This way you wouldn't have to move the cards much at all. No idea if it would work but maybe?

 

mbpb6gpDTIsUQcqOcGGfhXw.jpg

Link to comment

I also thought about this. I guess i am going to drain my system, getting the card in the second slot, with a borrowed small card in the primary. Then afterwards draining again and changing to old system. Hopefully it works then :D Thank you anyways and wish me luck, draining and changing a custom-watercooling two times in a row sucks :'D

Link to comment
4 hours ago, hblockx said:

I also thought about this. I guess i am going to drain my system, getting the card in the second slot, with a borrowed small card in the primary. Then afterwards draining again and changing to old system. Hopefully it works then :D Thank you anyways and wish me luck, draining and changing a custom-watercooling two times in a row sucks :'D

 

Ah best of luck. I would hate to have to do that! What GPU do you use. Maybe someone else here has the same who has already dumped the rom? Would save a lot of hastle?

Link to comment

Good news: adding "append vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot" to the syslinux.cfg file worked for getting the VM to boot with the card!

 

Another snag, though. I get the following message whenever I run the unbind/bind command: "-bash: echo: write error: No such device". The BIOS dump only produces a 126KB file, so I don't think it's happening properly.

 

So, two questions:

1) remind me what we need the bios file for again if we can get the card to boot using the UnRaid GUI? (I'm sure we DO, I'm just unclear what the difference/advantage is between bios dumped and added to XML vs not)

2) Is there a way to verify the validity of my bios dump other than filesize? (And if filesize is the only way to check, how do I know I have the right filesize)

 

Almost there! I'm so grateful for your great videos that help me get the most out of unraid. I feel I'm close, I just want to understand what's going on. (Teach me to "fish")

 

Thank you, again!

Link to comment
  • 4 weeks later...
  • 3 months later...
  • 2 weeks later...

I'm having no luck getting my gtx 980 passed through. Im struggling to locate a second card perform a vbios dump like you demonstrated in the video. I've even put together an old rig only to find out it doesn't support IOMMU. So i've ended up trying multiple different vbios from the techpowerup.com to no avail.

 

I'm really at a loss here and pulling my hair out, my free trial runs out in a few days and i'd really like to get my windows vm with gpu passthrough working so i can continue with my project. I've posted my machine spec and xml below in case I've done something glaringly obviously wrong, any help appreciated, thanks.

 

Machine Spec:

Model: Custom
M/B: ASUSTeK COMPUTER INC. - X99-DELUXE II
CPU: Intel® Core™ i7-6800K CPU @ 3.40GHz
HVM: Enabled
IOMMU: Enabled
Cache: 384 kB, 1536 kB, 15360 kB
Memory: 32 GB (max. installable capacity 512 GB)
Network: bond0: fault-tolerance (active-backup), mtu 1500 
 eth0: 1000 Mb/s, full duplex, mtu 1500 
 eth1: not connected
Kernel: Linux 4.9.30-unRAID x86_64

 

XML

<domain type='kvm'>
  <name>Windows 10</name>
  <uuid>943beee4-6867-a88e-a068-e5aed337923e</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>22020096</memory>
  <currentMemory unit='KiB'>22020096</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='7'/>
    <vcpupin vcpu='6' cpuset='8'/>
    <vcpupin vcpu='7' cpuset='9'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/943beee4-6867-a88e-a068-e5aed337923e_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Win10_1607_English_x64.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.126-2.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10/vdisk1.img'/>
      <target dev='hdc' bus='sata'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <controller type='usb' index='0' model='nec-xhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:ea:e7:f3'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/Drivers/gtx980-141103.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc52e'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x0b05'/>
        <product id='0x180a'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x2433'/>
        <product id='0xb200'/>
      </source>
      <address type='usb' bus='0' port='3'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

Link to comment

Hi @CloseTurkey

When you used the techpowerup vbioses did you remove the Nvidia header as in my video I linked to you before?

Please let me know what errors you have. Or does the vm just start and you have a black screen?

Anyway, I have edited an Evga 980 4 gig vbios for you to try linked here. Give this a try. Your XML looks ok

EVGA.GTX980.4096.rom

Edited by gridrunner
Link to comment

Hi @gridrunner

 

I feel so stupid, i watched it on my iPhone when you posted it and inadvertently watched the posts original video again. Come to think of it i did think it was strange to post the same video twice ??

 

Thanks for your prompt response, i will try the rom you attached tomorrow and let you know how i get on.

Link to comment

No luck so far, I'm getting the error log below.

VM starts up and stuck with black screen

 

LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/local/sbin/qemu -name 'guest=Windows 10,debug-threads=on' -S -object 'secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-Windows 10/master-key.aes' -machine pc-i440fx-2.7,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off -cpu host -drive file=/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/etc/libvirt/qemu/nvram/943beee4-6867-a88e-a068-e5aed337923e_VARS-pure-efi.fd,if=pflash,format=raw,unit=1 -m 21504 -realtime mlock=off -smp 8,sockets=1,cores=4,threads=2 -uuid 943beee4-6867-a88e-a068-e5aed337923e -display none -no-user-config -nodefaults -chardev 'socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-Windows 10/monitor.sock,server,nowait' -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device nec-usb-xhci,i0,server,nowait' -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x5,romfile=/mnt/user/drivers/EVGA.GTX980.4096.rom -device vfio-pci,host=01:00.1,id=hostdev1,bus=pci.0,addr=0x6 -device usb-host,hostbus=1,hostaddr=6,id=hostdev2,bus=usb.0,port=2 -device usb-host,hostbus=1,hostaddr=7,id=hostdev3,bus=usb.0,port=3 -device usb-host,hostbus=1,hostaddr=2,id=hostdev4,bus=usb.0,port=4 -device usb-host,hostbus=1,hostaddr=5,id=hostdev5,bus=usb.0,port=1.1 -device usb-host,hostbus=1,hostaddr=3,id=hostdev6,bus=usb.0,port=1.2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
Domain id=1 is tainted: high-privileges
Domain id=1 is tainted: host-cpu
char device redirected to /dev/pts/0 (label charserial0)

Edited by CloseTurkey
correcting logs
Link to comment

Um @CloseTurkeyThis is strange. You have a similar server to me with an x99 board and Nvidia GPU.

If it's a black screen then it is going to be the vbios of the card. Something is wrong with the file. Maybe I edited it wrong as I was tired

when I did it. Try editing a vbios yourself from techpowerup.

What is the exact model of the gpu you are using. Also please post your iommu groups and device here.

Another thing to try. Please make a new windows 10 vm with seabios as bios type and passthrough the romfile.

I expect you will get the same result. But on the off chance the vbios you are using doesn't support UEFI boot that would cause a black screen when using ovmf.

Also, your card is definitely a 980 not a 980ti as the 980 has 4 gigs of ram and 980 ti 6 gigs , the bios is different?

 

Anyway heres another 980 bios. 980vbios

 

Ps Contact Limetech and they will give you an extension on your trial

 

 

 

Edited by gridrunner
Link to comment

Hi gridrunner,

 

Managed to get passthrough working tonight, the frustration got too much last night and i purchased a GT 710 on prime now to arrive today ?

 

Still not sure what was causing the passthrough to fail with a single gpu.

Tried editing several vbios from techpowerup with no success and I'm wondering if it was the vbios or the actual passthrough?

I'm not really sure where to go with diagnosing this issue but so far

I've taken a dump straight from the GTX 980 so I may take out the GT 710  - I jumped the gun here, dump size is 4 bytes

and try passthrough with a single card again, it would be nice to just have the one in the rig

 

Quote

What is the exact model of the gpu you are using. Also please post your iommu groups and device here.

 

EVGA GeForce GTX 980 SC ACX2.0

Not sure how to get my IOMMU groups and device, I'm still pretty new to all of this

 

Quote

Ps Contact Limetech and they will give you an extension on your trial

Thanks, i'll contact them in the next couple days.

 

Now the real work starts now I'm partially over that hurdle

Edited by CloseTurkey
update
Link to comment
16 hours ago, CloseTurkey said:

Hi gridrunner,

 

Managed to get passthrough working tonight, the frustration got too much last night and i purchased a GT 710 on prime now to arrive today ?

 

Still not sure what was causing the passthrough to fail with a single gpu.

Tried editing several vbios from techpowerup with no success and I'm wondering if it was the vbios or the actual passthrough?

I'm not really sure where to go with diagnosing this issue but so far

I've taken a dump straight from the GTX 980 so I may take out the GT 710  - I jumped the gun here, dump size is 4 bytes

and try passthrough with a single card again, it would be nice to just have the one in the rig

 

 

EVGA GeForce GTX 980 SC ACX2.0

Not sure how to get my IOMMU groups and device, I'm still pretty new to all of this

 

Thanks, i'll contact them in the next couple days.

 

Now the real work starts now I'm partially over that hurdle

 

To check the iommu groups is really easy. Just goto tools system devices.

Here i have 2 GPUs in my rig a 1080 (group 32) and a 750 ti (group 30). You will see each is in its own iommu group with its sound counterpart.

If you have more than this in an iommu that your GPU is in passthrough will not work correctly.

That's why I wanted to see yours when you have just the one GPU installed.5970a6348ec43_ScreenShot2017-07-20at13_43_23.png.64e398507615078689e18b8d0596705a.png

Link to comment

ok so here is what I've found with one gpu

lspci - v  gives me this

04:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 980] (rev a1) (prog-if 00 [VGA controller])

    Subsystem: eVga.com. Corp. GM204 [GeForce GTX 980]

    Flags: fast devsel, IRQ 11, NUMA node 0

    Memory at f8000000 (32-bit, non-prefetchable) [disabled]

    Memory at b0000000 (64-bit, prefetchable) [disabled]

    Memory at c0000000 (64-bit, prefetchable) [disabled]

    I/O ports at d000 [disabled]

    Expansion ROM at f9000000 [disabled by cmd]

    Capabilities: [60] Power Management version 3

    Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+

    Capabilities: [78] Express Legacy Endpoint, MSI 00

    Capabilities: [100] Virtual Channel

    Capabilities: [250] Latency Tolerance Reporting

    Capabilities: [258] L1 PM Substates

    Capabilities: [128] Power Budgeting <?>

    Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>

    Capabilities: [900] #19

 

04:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)

    Subsystem: eVga.com. Corp. GM204 High Definition Audio Controller

    Flags: bus master, fast devsel, latency 0, IRQ 10, NUMA node 0

    Memory at f9080000 (32-bit, non-prefetchable)

    Capabilities: [60] Power Management version 3

    Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+

    Capabilities: [78] Express Endpoint, MSI 00

 

and IOMMU groups are this

IOMMU group 26

    [8086:8d47] 00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC Controller (rev 05)

    [8086:2822] 00:1f.2 RAID bus controller: Intel Corporation SATA Controller [RAID mode] (rev 05)

    [8086:8d22] 00:1f.3 SMBus: Intel Corporation C610/X99 series chipset SMBus Controller (rev 05)

IOMMU group 27

    [10de:13c0] 04:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 980] (rev a1)

    [10de:0fbb] 04:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)

IOMMU group 28

    [1b21:1242] 05:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller

IOMMU group 29

    [1b21:1242] 06:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller

Logs SeaBIOS

Jul 21 11:44:29 Tower kernel: input: Logitech USB Optical Mouse as /devices/pci0000:00/0000:00:14.0/usb1/1-9/1-9.3/1-9.3:1.0/0003:046D:C077.000A/input/input14
Jul 21 11:44:29 Tower kernel: hid-generic 0003:046D:C077.000A: input,hidraw3: USB HID v1.11 Mouse [Logitech USB Optical Mouse] on usb-0000:00:14.0-9.3/input0
Jul 21 11:44:41 Tower kernel: vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=io+mem:owns=io+mem
Jul 21 11:44:41 Tower kernel: br0: port 2(vnet0) entered blocking state
Jul 21 11:44:41 Tower kernel: br0: port 2(vnet0) entered disabled state
Jul 21 11:44:41 Tower kernel: device vnet0 entered promiscuous mode
Jul 21 11:44:41 Tower kernel: br0: port 2(vnet0) entered blocking state
Jul 21 11:44:41 Tower kernel: br0: port 2(vnet0) entered forwarding state
Jul 21 11:44:42 Tower kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e@0x258
Jul 21 11:44:42 Tower kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
Jul 21 11:44:42 Tower kernel: vfio-pci 0000:00:1b.0: enabling device (0000 -> 0002)
Jul 21 11:44:42 Tower acpid: input device has been disconnected, fd 6
Jul 21 11:44:42 Tower acpid: input device has been disconnected, fd 7
Jul 21 11:44:44 Tower kernel: kvm: zapping shadow pages for mmio generation wraparound
Jul 21 11:44:49 Tower kernel: usb 1-9.3: reset low-speed USB device number 11 using xhci_hcd
Jul 21 11:44:50 Tower kernel: usb 1-9.3: reset low-speed USB device number 11 using xhci_hcd
Jul 21 11:44:50 Tower kernel: usb 1-9.2: reset low-speed USB device number 6 using xhci_hcd
Jul 21 11:44:51 Tower kernel: usb 1-9.2: reset low-speed USB device number 6 using xhci_hcd
Jul 21 11:44:51 Tower kernel: kvm: zapping shadow pages for mmio generation wraparound

VM Logs

2017-07-21 18:44:41.766+0000: starting up libvirt version: 2.4.0, qemu version: 2.7.1, hostname: Tower
LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/local/sbin/qemu -name 'guest=Windows 10 Gaming,debug-threads=on' -S -object 'secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-2-Windows 10 Gaming/master-key.aes' -machine pc-i440fx-2.7,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off -cpu host -m 8192 -realtime mlock=off -smp 4,sockets=1,cores=2,threads=2 -uuid 3e3a2e1e-bfa3-59e0-2bde-cab34c96cd5f -display none -no-user-config -nodefaults -chardev 'socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-2-Windows 10 Gaming/monitor.sock,server,nowait' -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device nec-usb-xhci,id=usb,bus=pci.0,addr=0x7 -device ahci,id=sata0,bus=pci.0,addr=0x3 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive 'file=/mnt/user/domains/Windows 10 Gaming/vdisk1.img,format=r=0x6,romfile=/mnt/user/drivers/test.dump -device vfio-pci,host=00:1b.0,id=hostdev1,bus=pci.0,addr=0x8 -device usb-host,hostbus=1,hostaddr=11,id=hostdev2,bus=usb.0,port=1 -device usb-host,hostbus=1,hostaddr=6,id=hostdev3,bus=usb.0,port=2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x9 -msg timestamp=on
Domain id=2 is tainted: high-privileges
Domain id=2 is tainted: host-cpu
char device redirected to /dev/pts/0 (label charserial0)

 

LOGS OVMF

Jul 21 11:52:33 Tower kernel: usb 1-9.3: USB disconnect, device number 13
Jul 21 11:52:34 Tower kernel: usb 1-9.3: new low-speed USB device number 14 using xhci_hcd
Jul 21 11:52:34 Tower kernel: input: Logitech USB Optical Mouse as /devices/pci0000:00/0000:00:14.0/usb1/1-9/1-9.3/1-9.3:1.0/0003:046D:C077.0010/input/input20
Jul 21 11:52:34 Tower kernel: hid-generic 0003:046D:C077.0010: input,hidraw3: USB HID v1.11 Mouse [Logitech USB Optical Mouse] on usb-0000:00:14.0-9.3/input0
Jul 21 11:52:57 Tower kernel: vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=io+mem:owns=io+mem
Jul 21 11:52:57 Tower kernel: br0: port 2(vnet0) entered blocking state
Jul 21 11:52:57 Tower kernel: br0: port 2(vnet0) entered disabled state
Jul 21 11:52:57 Tower kernel: device vnet0 entered promiscuous mode
Jul 21 11:52:57 Tower kernel: br0: port 2(vnet0) entered blocking state
Jul 21 11:52:57 Tower kernel: br0: port 2(vnet0) entered forwarding state
Jul 21 11:52:59 Tower kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e@0x258
Jul 21 11:52:59 Tower kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
Jul 21 11:52:59 Tower kernel: vfio-pci 0000:00:1b.0: enabling device (0000 -> 0002)
Jul 21 11:52:59 Tower acpid: input device has been disconnected, fd 6
Jul 21 11:52:59 Tower acpid: input device has been disconnected, fd 7
Jul 21 11:53:06 Tower kernel: usb 1-9.3: reset low-speed USB device number 14 using xhci_hcd
Jul 21 11:53:07 Tower kernel: usb 1-9.3: reset low-speed USB device number 14 using xhci_hcd
Jul 21 11:53:08 Tower kernel: usb 1-9.2: reset low-speed USB device number 6 using xhci_hcd
Jul 21 11:53:08 Tower kernel: usb 1-9.2: reset low-speed USB device number 6 using xhci_hcd
Jul 21 11:53:10 Tower kernel: kvm: zapping shadow pages for mmio generation wraparound
Jul 21 11:53:10 Tower kernel: kvm: zapping shadow pages for mmio generation wraparound

VM Logs

2017-07-21 18:52:57.121+0000: starting up libvirt version: 2.4.0, qemu version: 2.7.1, hostname: Tower
LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/local/sbin/qemu -name 'guest=Windows 10,debug-threads=on' -S -object 'secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-3-Windows 10/master-key.aes' -machine pc-i440fx-2.7,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off -cpu host -drive file=/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/etc/libvirt/qemu/nvram/988a76c3-d21b-e56b-8e8c-a0f7c78f1c24_VARS-pure-efi.fd,if=pflash,format=raw,unit=1 -m 20480 -realtime mlock=off -smp 8,sockets=1,cores=4,threads=2 -uuid 988a76c3-d21b-e56b-8e8c-a0f7c78f1c24 -display none -no-user-config -nodefaults -chardev 'socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-3-Windows 10/monitor.sock,server,nowait' -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device nec-usb-xhci,i,nowait' -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x6,romfile=/mnt/user/drivers/test.dump -device vfio-pci,host=00:1b.0,id=hostdev1,bus=pci.0,addr=0x8 -device usb-host,hostbus=1,hostaddr=14,id=hostdev2,bus=usb.0,port=1 -device usb-host,hostbus=1,hostaddr=6,id=hostdev3,bus=usb.0,port=2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x9 -msg timestamp=on
Domain id=3 is tainted: high-privileges
Domain id=3 is tainted: host-cpu
char device redirected to /dev/pts/0 (label charserial0)
2017-07-21T18:54:12.473689Z qemu-system-x86_64: libusb_release_interface: -4 [NO_DEVICE]

 

I performed a cat of the rom with the GTX980 in the second pci slot which worked successfully (attached). I then tested that rom by editing the xml on 2 seperate vms. one with SeaBIOS and the other OVMF, both of which gave me a black screen. Ive tried this with the GTX 980 in the primary PCI slot and the secondary slot with the same results.

 

The vms tested above do work with passthrough when there are 2 GFX cards in the rig, when GT 710 is in the primary PCI slot and theGTX 980 in the secondary .

 

Sorry for the lengthy post.

TL;DR

Cannot get to work with 1 GPU and rom dump. Will work with 2 GPU's

 

GTX980test.dump

Link to comment

I was pointed towards this thread when I had trouble isolating my 1070 for PCI passthrough.

 

Long story short, I tried dumping my vBIOS like instructed in the video, but couldn't do so (the `cat` command printed I/O errors instead). Instead, I resorted to dumping the full vBIOS under Windows and using a hex editor to splice the relevant part of the ROM into a new file, using some of the partial vBIOS files uploaded here as samples. This finally allowed me to pass the GPU to the Windows VM!

 

---

 

Anyway, I wrote a Python script that should automate this process (you give it a full ROM from techPowerUp or one you dumped using nvflash under Windows), and it should create a patched ROM that you can use to make GPU passthrough work.

 

I passed a few ROMs I downloaded from techPowerUp through the script and compared them to what you guys uploaded here, and so far the Pascal vBIOS files appeared to match, bit by bit. Still, I can't stress it enough that this script is based on guesswork, so it may end up bricking your GPU if you're unlucky. It does a few rudimentary sanity checks, but I would recommend dumping the partial ROM yourself if you can. Still, for those who are pulling your hair out over not being able to do that, this may be a lifesaver.

 

https://github.com/Matoking/NVIDIA-vBIOS-VFIO-Patcher

 

Edited by Matoking
  • Upvote 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.