Trouble with EVGA GTX 970 Passthrough


Recommended Posts

Hi Everyone,

 

I have been struggling to try to get my EVGA GTX 970 to successfully passthrough.

 

My setup:

 

Motherboard: Asrock X99M Killer (has VT-D support, enabled in BIOS)

GPU: EVGA GTX 970 (VGA EVGA GTX 970 04G-P4-3978-KR RTL)

Processor: Xeon E51650 V3

 

I am able to successfully create VMs with VNC as the graphical support. Anytime I attempt to passthrough the GPU, I don't see any image on my display. I believe the GPU is working fine, as prior to booting the VM I am able to see unRAID boot and the console output (and am able to enter my motherboard BIOS, etc). After starting the VM, the display goes black, then enters power saving mode. I do not see the BIOS output for the VM on the display.

 

I've tried:

- 2 different displays, 2 different ports no the GPU

- Both Q35 and i440fx based VMs

- Both OVMF and SeaBios based VMs (utilizing jonp's workaround for the OVMF issue)

- Windows 7 based and Windows 8.1 based VMs

 

To further test my build, I created a native Windows 7 installation on a spare HDD and booted the system to that. The GPU appeared to work fine.

 

Full XML for current Windows 8.1 VM:

 

<domain type='kvm'>
  <name>Windows 8</name>
  <uuid>4675f57c-4328-6654-f757-fa18a9be70bd</uuid>
  <description>Windows 8 Gaming PC</description>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static'>6</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='5'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-2.2'>hvm</type>
    <loader type='pflash'>/usr/share/qemu/ovmf-x64/OVMF-pure-efi.fd</loader>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='6' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/VM/Images/Win81.iso'/>
      <target dev='hda' bus='virtio'/>
      <readonly/>
      <boot order='2'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/VM/Images/virtio-win-0.1-100.iso'/>
      <target dev='hdb' bus='virtio'/>
      <readonly/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/cache/VM/Windows 8/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:4c:4e:0b'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

Any thoughts or direction on next steps would be greatly appreciated, I am really hoping that somehow I'll be able to get this to work!

 

Thanks in advance,

Greg

Link to comment

Hi jon,

 

It doesn't look like there are any firmware updates available for my GPU. There was a firmware update for my motherboard, which I have applied, but the same issue persists.

 

Here's a video of what it looks like.

 

Ok, let's try this:

 

1 - e-mail [email protected] and ask if they have a BIOS update for UEFI support for your card (provide them the make, model, and if you can obtain it, the serial #).  EVGA support is AWESOME and I would be shocked if they didn't get back to you ASAP.

 

2 - While you're waiting, you can try this for me:

 

Goto this URL:  http://www.techpowerup.com/vgabios/index.php?architecture=NVIDIA&manufacturer=EVGA&model=GTX+970&interface=&memType=&memSize=

 

Download the latest vbios for your card (look for the specific model) and place this file somewhere on your cache drive or array.

 

Edit the XML for your VM.

 

If SeaBIOS:

 

Edit the QEMU argument at the bottom for the GPU.  Look for this:

 

    <qemu:arg value='vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>

 

The host= may be different, that's ok.  Just add to it after x-vga=on:  ,romfile=/mnt/user/path/to/romfile

 

Example:

 

    <qemu:arg value='vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/mnt/user/path/to/romfile'/>

 

For OVMF:

 

Edit the <hostdev> tag for your video card.  Example:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x0' function='0x0'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>

 

Add the following after the </source> tag:

 

<rom file='/mnt/user/path/to/file'/>

 

Report back how it goes.

Link to comment

Hi Jon,

 

Thanks for all of your help and support so far. I reached out to EVGA and will report back when they respond.

 

I downloaded EVGA.GTX970.4096.140930.rom and installed it into /mnt/VM/Images/

 

I tried on both a SeaBios Q-35 system:

 

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>Windows 7</name>
  <uuid>d46c5469-e9f2-edbe-01b2-63da5d44202c</uuid>
  <description>Windows 7 Gaming Image</description>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static'>6</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='5'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-2.2'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='6' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/cache/VM/Windows 7/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/VM/Images/GSP1RMCHPXFREO_EN_DVD.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/VM/Images/virtio-win-0.1-100.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:a2:d0:6d'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=02:00.0,bus=pcie.0,multifunction=on,x-vga=on,romfile=/mnt/user/VM/Images/EVGA.GTX970.4096.140930.rom'/>
  </qemu:commandline>
</domain>

 

And a Q35 OVMF system:

 

<domain type='kvm'>
  <name>Windows 8</name>
  <uuid>b7e4f491-cacd-37e1-9237-81862cb8dab0</uuid>
  <description>Windows 8 - Gaming PC</description>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static'>6</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='5'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-2.2'>hvm</type>
    <loader type='pflash'>/usr/share/qemu/ovmf-x64/OVMF-pure-efi.fd</loader>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='6' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/VM/Images/Win81AIO-20in1-x64-en-US-Nov2013-v2.iso'/>
      <target dev='hda' bus='virtio'/>
      <readonly/>
      <boot order='2'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/VM/Images/virtio-win-0.1-100.iso'/>
      <target dev='hdb' bus='virtio'/>
      <readonly/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/cache/VM/Windows 8/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:ea:02:1f'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/VM/Images/EVGA.GTX970.4096.140930.rom'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

Unfortunately, both reproduced the original symptom.

 

2 asides:

 

1) Would the port used on the card make any difference, or would the passthrough apply to each?

2) Do I need to reboot the array between attempts? I ask because when I stop the VM, I don't get the console back on my display unless I reboot the sever.

Link to comment

Darn.  Was hoping that might work.  Ok, next steps...

 

lspci -n

 

Find the vendor / product ID for your GPU.  It's the last column.  The first column shows the PCI device IDs.  you need to find this for both the GPU and it's audio device.  If your GPU is 01:00.0 your GPU's audio device will be 01:00.1.

 

Now go to the unRAID webGui, main tab, click flash, then edit your syslinux configuration.  Add this after the "append" line:  append pci-stub.ids= followed by your vendor/product ids for each device, seperated by a comma.  My syslinux as an example:

 

  append pci-stub.ids=1002:68f9,1002:aa68,10de:1004,10de:0e1a initrd=/bzroot

 

Reboot the server with that setting.

 

When you pass through a GPU (any GPU) using SeaBIOS, the local VGA console (regardless of which device that is) is lost.  With OVMF, this does not occur, but if the GPU that gets your unRAID console is also the GPU you assign to the VM, then obviously you do lose that console.

Link to comment

Thanks Jon,

 

Will try and update next.

 

As an aside, when I started the VM I took a look at the system log. Does this indicate anything that helps us?

 

Apr 25 17:09:13 Canteen kernel: vfio_ecap_init: 0000:02:00.0 hiding ecap 0x1e@0x258
Apr 25 17:09:13 Canteen kernel: vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x900

Link to comment

Hi Jon,

 

From

lspci -n

 

I located my GPU,

 

02:00.0 0300: 10de:13c2 (rev a1)
02:00.1 0403: 10de:0fbb (rev a1)

 

I edited my Syslinux configuration:

default /syslinux/menu.c32
menu title Lime Technology
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
  append pci-stub.ids=10de:13c2,10de:0fbb initrd=/bzroot
label unRAID OS Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label Memtest86+
  kernel /memtest
label Xen/unRAID OS
  kernel /syslinux/mboot.c32
  append /xen --- /bzimage --- /bzroot
label Xen/unRAID OS Safe Mode (no plugins)
  kernel /syslinux/mboot.c32
  append /xen --- /bzimage --- /bzroot unraidsafemode

 

I rebooted the server, and tried each VM and the same issue persists.

Link to comment

Here's the reply from EVGA:

 

Hello,

 

The newer cards come already with UEFI support and do not require a bios update. However, currently there are no bios updates for this video card.

 

Regards,

EVGA

 

Ok, that is very shocking that the card has UEFI and is having issues.

Link to comment

Hi Jon,

 

It is the Asrock X99M Killer

 

The board supports VT-D (and has it enabled in the BIOS), and I am using it with the latest BIOS version 1.8. I'm using it with a Xeon E5 1650v3.

 

This may be a case of an oddball motherboard issue.  I hate to ask but you don't happen to have a secondary motherboard you could test with, do you?  I am 99% sure if you mailed me that card and I put it in my system, it'd work just fine.

Link to comment

This may be a case of an oddball motherboard issue.  I hate to ask but you don't happen to have a secondary motherboard you could test with, do you?  I am 99% sure if you mailed me that card and I put it in my system, it'd work just fine.

 

Hi Jon,

 

I also have the EVGA X99 Micro, but wasn't able to get that to work in my system (I believe it ended up being incompatible with my ECC RAM). I may be able to borrow some RAM, or even a motherboard / processor combo (I'd rather not fully disassemble my system at this point). Are there any other typical requirements for GPU pass through support from the motherboards other than VT-D support?

Link to comment

This may be a case of an oddball motherboard issue.  I hate to ask but you don't happen to have a secondary motherboard you could test with, do you?  I am 99% sure if you mailed me that card and I put it in my system, it'd work just fine.

 

Hi Jon,

 

I also have the EVGA X99 Micro, but wasn't able to get that to work in my system (I believe it ended up being incompatible with my ECC RAM). I may be able to borrow some RAM, or even a motherboard / processor combo (I'd rather not fully disassemble my system at this point). Are there any other typical requirements for GPU pass through support from the motherboards other than VT-D support?

Nope.  I just triple checked your hardware list and it seems solid.  The only questionable element at this point looks like mobo because I can't find info from anyone else that has tested vtd on that specific board..

Link to comment

Hi Jon,

 

Still checking with friends to see what hardware might be available for me. If all else fails, could I actually mail the card to you for testing purposes? During my research, I've noticed that folks are enabling:

 

pcie_acs_override=downstream

 

Is this setting controlled in VM manager settings (PCIe ACS Override)? Currently, I have it set to "no". Should I have it set to "yes" instead? Would it affect my PCI sata card?

 

Thanks,

Greg

Link to comment

Hi Jon,

 

Still checking with friends to see what hardware might be available for me. If all else fails, could I actually mail the card to you for testing purposes? During my research, I've noticed that folks are enabling:

 

pcie_acs_override=downstream

 

Is this setting controlled in VM manager settings (PCIe ACS Override)? Currently, I have it set to "no". Should I have it set to "yes" instead? Would it affect my PCI sata card?

 

Thanks,

Greg

You don't need that setting. If the vm starts and the graphics disappear, pcie acs won't affect you.

 

You absolutely can mail the card to me for testing if you'd prefer or can't find anyone else to loan you a card.  Send me a pm if you want to do that and I'll give you the address.

Link to comment

Hi Jon et al,

 

It looks like it is the card.

 

On a whim, I ordered another card for testing purposes: EVGA GT 740 (02G-P4-2740-KR). I configured a new VM (Windows 8, using OVFM, Q35) and attempted to passthrough the new card (with the GTX 970 still installed). The passthrough immediately worked and I was able to see the UEFI interface. I reconfigured the same VM to attempt passthrough of the GTX 970 and my original issue returned.

 

I then moved the GTX 970 to the second PCI slot (where I was successfully able to passthrough the GT 740), and my original issue persisted. I'll reach out to EVGA and see if they have any thoughts. Maybe this particular card is too "new" and the drivers don't support it yet?

 

Currently, I think my options are:

1) Get a different make of the 970 chipset that has been verified to work. A ~$350 mistake, but not the end of the world. Because I modified my original 970 (soldered a pigtail for the auxiliary power connections so I could fit it in the case) I doubt I'll be able to resell it.

2) Wait for updates and hope that either an update to KVM or a firmware update for the card fix the issue.

3) Say "to hell with it!" and build a standalone gaming PC.

 

Any other ideas?

 

As an aside, the fan on the GT 740 is LOUD, and the thing runs full blast. I'll need to check and see if there's a firmware fix for it, but for now i've unplugged it because it was driving me crazy.

 

Best,

Greg

Link to comment

 

 

Hi Jon et al,

 

It looks like it is the card.

 

On a whim, I ordered another card for testing purposes: EVGA GT 740 (02G-P4-2740-KR). I configured a new VM (Windows 8, using OVFM, Q35) and attempted to passthrough the new card (with the GTX 970 still installed). The passthrough immediately worked and I was able to see the UEFI interface. I reconfigured the same VM to attempt passthrough of the GTX 970 and my original issue returned.

 

I then moved the GTX 970 to the second PCI slot (where I was successfully able to passthrough the GT 740), and my original issue persisted. I'll reach out to EVGA and see if they have any thoughts. Maybe this particular card is too "new" and the drivers don't support it yet?

 

Currently, I think my options are:

1) Get a different make of the 970 chipset that has been verified to work. A ~$350 mistake, but not the end of the world. Because I modified my original 970 (soldered a pigtail for the auxiliary power connections so I could fit it in the case) I doubt I'll be able to resell it.

2) Wait for updates and hope that either an update to KVM or a firmware update for the card fix the issue.

3) Say "to hell with it!" and build a standalone gaming PC.

 

Any other ideas?

 

As an aside, the fan on the GT 740 is LOUD, and the thing runs full blast. I'll need to check and see if there's a firmware fix for it, but for now i've unplugged it because it was driving me crazy.

 

Best,

Greg

 

Or you can send me the 970 to test with. I'm still beyond shocked that you are having an issue. I'd like to test it both with different mobos and some internal test builds to see if that could fix it. I realize its a tall ask, but if you were willing, it'd be certainly worth our time to test.

Link to comment

Or you can send me the 970 to test with. I'm still beyond shocked that you are having an issue. I'd like to test it both with different mobos and some internal test builds to see if that could fix it. I realize its a tall ask, but if you were willing, it'd be certainly worth our time to test.

 

Thanks Jon. We might as well pursue this avenue if you're up for it. I sent you a PM.

Link to comment
  • 2 months later...

Did you guys ever figure this out? I just picked up unRAID again as I got a new card.

 

I've just upgraded to a KFA2 GTX 970 EXOC and I'm getting exactly this behaviour.

 

lspci -n gives me the exact same vendor codes as expressexcess reported.

 

Even tried stubbing out too with same result (used - append pci-stub.ids=10de:13c2,10de:0fbb initrd=/bzroot)

 

I do have another data point which is that I was getting the same behaviour with my older GTX 580 in the same box - as described in this thread http://lime-technology.com/forum/index.php?topic=40155.0

 

My motherboard is this one:

http://www.gigabyte.com/products/product-page.aspx?pid=4405#ov

My processor is this one (note that there is no IGP on these Xeons):

http://ark.intel.com/products/65730/Intel-Xeon-Processor-E3-1240-v2-8M-Cache-3_40-GHz

 

Any ideas? About to give up after another day spent trying to get unRAID to work. Guess I've been unlucky twice now as many seem to have had great success first time around!

 

P.S. Thanks for jon for updating the docs after my http://lime-technology.com/forum/index.php?topic=40155.0 post, makes things clearer and I've also tried all those steps with the new 970 card to no avail. Sadly I didn't see you'd updated the docs in response to my thread until today and so am now unable to test it out with the old GTX 580 :-(

Link to comment

Did you guys ever figure this out? I just picked up unRAID again as I got a new card.

 

I've just upgraded to a KFA2 GTX 970 EXOC and I'm getting exactly this behaviour.

 

lspci -n gives me the exact same vendor codes as expressexcess reported.

 

Even tried stubbing out too with same result (used - append pci-stub.ids=10de:13c2,10de:0fbb initrd=/bzroot)

 

I do have another data point which is that I was getting the same behaviour with my older GTX 580 in the same box - as described in this thread http://lime-technology.com/forum/index.php?topic=40155.0

 

My motherboard is this one:

http://www.gigabyte.com/products/product-page.aspx?pid=4405#ov

My processor is this one (note that there is no IGP on these Xeons):

http://ark.intel.com/products/65730/Intel-Xeon-Processor-E3-1240-v2-8M-Cache-3_40-GHz

 

Any ideas? About to give up after another day spent trying to get unRAID to work. Guess I've been unlucky twice now as many seem to have had great success first time around!

 

P.S. Thanks for jon for updating the docs after my http://lime-technology.com/forum/index.php?topic=40155.0 post, makes things clearer and I've also tried all those steps with the new 970 card to no avail. Sadly I didn't see you'd updated the docs in response to my thread until today and so am now unable to test it out with the old GTX 580 :-(

 

So I have a different gigabyte motherboard than you, but here's what I can tell you.  I checked the manual for your motherboard.  I only see one setting for supporting virtualization, and no mention of VT-d or IOMMU support at all.  Some motherboards do not seem to have good IOMMU implementation compared to others.  I have had no issues with this board:  http://download.gigabyte.us/FileList/Manual/mb_manual_ga-z87mx-d3h_e.pdf

 

If you search for vt-d in that manual, you will find a specific setting for it.  But in your motherboard manual, I do not see reference to that.

 

It's possible that they half implemented the feature (which is maybe why the GPU is showing up as assignable from the webGui and IOMMU showing enabled under the info button in the webGui), but because there is no toggle for it, I wonder if this was just not fully implemented.

 

Your CPU, on the other hand, is perfectly capable of VT-d.  Your GPU should be as well.  I was able to utilize the 970 from express here just fine with multiple test systems.  It's sitting in one of them right now as I use it for testing until express demands it back ;-).  So far it's kept on ticking throughout all the tests so far without challenge (including on 6.1-beta3).

 

You could always try contacting gigabyte support and just state you are having difficulties with VT-d support on your motherboard and see what they say.

Link to comment

Thanks Jon.

 

I did a bit more research and I think you've hit the nail on the head.

 

S4yD112l.png

 

Thanks for the tips! Lesson learned - make sure every single thing supports VT-d not just the proccessor. Good to know that you have what seems to be the exact same 970 (i.e. vendor id exactly the same), working with passthrough.

 

In anger I actually gave up on the B75M machine before your reply and bought unRAID with the plan to set it up on my other machine which has this motherboard:

 

http://www.asrock.com/mb/intel/h77m/

 

The good news is that I'm now starting to see how unRAID could really work for me if I can just get a few things sorted out.

 

Sadly I realised almost immediately that my other 2nd machine has this processor which does not support VT-d and so no passthrough!

 

http://ark.intel.com/products/65527/Intel-Pentium-Processor-G2120-3M-Cache-3_10-GHz

 

For those following along, from my other thread too, I think this now means that there is good chance that my GTX 580 might work if I pair it with a compatible motherboard and processor.

 

This page lists other H77M from Gigabyte as being compatible with VT-d (i.e. Looks like the H77M chipset supports it unlike the B75M). Slightly worried that the Intel comparison page for H77M does not say 'Yes' like for the B75M but it does not say 'No' either. I guess blank space could mean that no news is good news. I'm hopeful based on the report of the H77M from Gigabyte working here:

 

http://www.overclock.net/t/1338063/vt-d-compatible-motherboards

 

When I go in to the bios on my H77M Asrock board and go in to the CPU setting it only says VT-x supported. But I guess it's just listing what the CPU can do - and I know the G2120 is a no go for VT-d. When I go in to the North Bridge part of the bios I get a line that says 'VT-d Capability'  on the far left and then on the far right it say 'Not Supported'

 

So I'm guessing if I swap processors and put my Xeon in that box I'll be in business and the 'VT-d Capability' flag will switch to 'Supported'.

 

Only one way to find out I guess! But it'll have to wait until after an important meeting I have next week. My Xeon is in my portable VR demo rig.

 

Thanks again, will report back when I get round to trying it out.

 

Anyway now I've got to figure out why xfce4-panel works fine and loads through the native noVNC implementation for unRAID but fails to load when I try and access the guest VM directly using turbovncserver. Thus leaving me with a borked half working XFCE4 desktop. See separate post on this issue, anyone reading who has lots of experience working with VNC I'd love some help!

 

http://lime-technology.com/forum/index.php?topic=41521.0

 

For those wondering why I don't just use the default VNC implementation provided via unRAID. Well the answer is I need more speed for 3D and turbovnc is the only way I know how to get this done so it feels like I'm sitting in front of the actuall workstation. Turbovncserver  should allow me to get super fast VNC with realtime 3d via unRAID once I manage to pass through a GPU. I already have this working on AWS and it rocks on their GPU instances. It really feels like you're sitting in front of a beefy GPU machine providing you can keep your pings to the data centre below 70ms and you have at least 5MB download via DSL. AWS is expensive though and hence my foray into the world of unRAID!

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.