Hello everyone,

 

I’m a new Unraid enthusiast, just created Proof of Concept, planning to scale up once I feel comfortable.

 

Currently Running on dual Xeon X5670 CPU on HP ML350 and 48gb ECC REG Ram.

Running on version 6.3.2

 

After creating a VM and successfully installed Windows 10, I decided to try GPU (nvidia geforce 9500) Passthrough, this is what I have done so far without any success:

 

Enable PCIe ACS

 

Moved SAS controller to a different PCIe slot in order to avoid having two devices on the same IOMMU group:

IOMMU group 18

[10de:0640] 0b:00.0 VGA compatible controller: NVIDIA Corporation G96 [GeForce 9500 GT] (rev a1)

 

Confirmed:

HVM: Enabled

IOMMU: Enabled

 

Here is my vm xml

<domain type='kvm'>

 <name>Windows 10</name>

 <uuid>669a92be-4797-3c2e-575f-4e7375ac56d9</uuid>

 <metadata>

   <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>

 </metadata>

 <memory unit='KiB'>12582912</memory>

 <currentMemory unit='KiB'>12582912</currentMemory>

 <memoryBacking>

   <nosharepages/>

 </memoryBacking>

 <vcpu placement='static'>6</vcpu>

 <cputune>

   <vcpupin vcpu='0' cpuset='8'/>

   <vcpupin vcpu='1' cpuset='10'/>

   <vcpupin vcpu='2' cpuset='16'/>

   <vcpupin vcpu='3' cpuset='18'/>

   <vcpupin vcpu='4' cpuset='20'/>

   <vcpupin vcpu='5' cpuset='22'/>

 </cputune>

 <os>

   <type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type>

   <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>

   <nvram>/etc/libvirt/qemu/nvram/669a92be-4797-3c2e-575f-4e7375ac56d9_VARS-pure-efi.fd</nvram>

 </os>

 <features>

   <acpi/>

   <apic/>

   <hyperv>

     <relaxed state='on'/>

     <vapic state='on'/>

     <spinlocks state='on' retries='8191'/>

     <vendor_id state='on' value='none'/>

   </hyperv>

 </features>

 <cpu mode='host-passthrough'>

   <topology sockets='1' cores='3' threads='2'/>

 </cpu>

 <clock offset='localtime'>

   <timer name='hypervclock' present='yes'/>

   <timer name='hpet' present='no'/>

 </clock>

 <on_poweroff>destroy</on_poweroff>

 <on_reboot>restart</on_reboot>

 <on_crash>restart</on_crash>

 <devices>

   <emulator>/usr/local/sbin/qemu</emulator>

   <disk type='file' device='disk'>

     <driver name='qemu' type='raw' cache='writeback'/>

     <source file='/mnt/user/domains/Windows 10/vdisk4.img'/>

     <target dev='hdc' bus='sata'/>

     <boot order='1'/>

     <address type='drive' controller='0' bus='0' target='0' unit='2'/>

   </disk>

   <disk type='file' device='cdrom'>

     <driver name='qemu' type='raw'/>

     <source file='/mnt/user/isos/Windows10.iso'/>

     <target dev='hda' bus='ide'/>

     <readonly/>

     <boot order='2'/>

     <address type='drive' controller='0' bus='0' target='0' unit='0'/>

   </disk>

   <disk type='file' device='cdrom'>

     <driver name='qemu' type='raw'/>

     <source file='/mnt/user/isos/virtio-win-0.1.126-2.iso'/>

     <target dev='hdb' bus='ide'/>

     <readonly/>

     <address type='drive' controller='0' bus='0' target='0' unit='1'/>

   </disk>

   <controller type='usb' index='0' model='ich9-ehci1'>

     <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>

   </controller>

   <controller type='usb' index='0' model='ich9-uhci1'>

     <master startport='0'/>

     <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>

   </controller>

   <controller type='usb' index='0' model='ich9-uhci2'>

     <master startport='2'/>

     <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>

   </controller>

   <controller type='usb' index='0' model='ich9-uhci3'>

     <master startport='4'/>

     <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>

   </controller>

   <controller type='pci' index='0' model='pci-root'/>

   <controller type='ide' index='0'>

     <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>

   </controller>

   <controller type='sata' index='0'>

     <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

   </controller>

   <controller type='virtio-serial' index='0'>

     <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>

   </controller>

   <interface type='bridge'>

     <mac address='52:54:00:7e:7c:a6'/>

     <source bridge='br0'/>

     <model type='virtio'/>

     <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>

   </interface>

   <serial type='pty'>

     <target port='0'/>

   </serial>

   <console type='pty'>

     <target type='serial' port='0'/>

   </console>

   <channel type='unix'>

     <target type='virtio' name='org.qemu.guest_agent.0'/>

     <address type='virtio-serial' controller='0' bus='0' port='1'/>

   </channel>

   <input type='mouse' bus='ps2'/>

   <input type='keyboard' bus='ps2'/>

   <hostdev mode='subsystem' type='pci' managed='yes'>

     <driver name='vfio'/>

     <source>

       <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>

     </source>

     <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>

   </hostdev>

   <memballoon model='virtio'>

     <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

   </memballoon>

 </devices>

</domain>

 

I have also been searching information on the forum and in the web in general, without any positive results.

Please let me know if you need any other info.

 

Best regards

what gen ML350? G6?

Also, please let me know what your thread pairings look like. My guess is jumping by even numbers? (this is for another error that pops up on unRaid 6.3.x versions)

You will probably have to modify your syslinux.cfg to the following

 

default /syslinux/menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
  append vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot
label unRAID OS GUI Mode
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label Memtest86+
  kernel /memtest

 

 

the key line being:

 

  append vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot

 

 

That has onboard video, right? also make sure bios is set for onboard as primary.

 

 

ML350 G6,

Thank you for the quick reply I will try that immediately 

Still got the same error after adding the suggested line to syslinux

 

Execution error

internal error: qemu unexpectedly closed the monitor: 2017-04-07T00:11:59.695584Z qemu-system-x86_64: -device vfio-pci,host=0b:00.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to set iommu for container: Operation not permitted
2017-04-07T00:11:59.695618Z qemu-system-x86_64: -device vfio-pci,host=0b:00.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to setup container for group 18
2017-04-07T00:11:59.695624Z qemu-system-x86_64: -device vfio-pci,host=0b:00.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to get group 18
2017-04-07T00:11:59.695641Z qemu-system-x86_64: -device vfio-pci,host=0b:00.0,id=hostdev0,bus=pci.0,addr=0x5: Device initialization failed

did you reboot?

Made some progress, rebooted and now the VM is able to see the Nvidia 9500 GT, I was also able to install drivers, but when I connect a monitor there is no output.

 

Appreciate your help.

did you make sure onboard video is set as primary in your bios?

 

what did you use to be able to install drivers- vnc, rdp? iirc, you can't use vnc with a cpu on windows 10 but I could be incorrect.

Good morning,

 

Yes I went to the BIOS and made sure the on-board gpu is the primary and the rest is secondary.

I had to RDP into the machine in order to access, and you are right, there is no VNC access at the moment.

 

Thanks.

 

3 hours ago, nrosario said:

Good morning,

 

Yes I went to the BIOS and made sure the on-board gpu is the primary and the rest is secondary.

I had to RDP into the machine in order to access, and you are right, there is no VNC access at the moment.

 

Thanks.

 

 

Because you are using the default template to setup the windows VM. Your VM  is using OVMF

Your GPU is unlikely to have a UEFI compatible vbios so you will need to install a new windows VM but using seabios.

To select seabios you need to toggle the advanced options on the VM template top right.

It should work if you use seabios.

53 minutes ago, gridrunner said:

Because you are using the default template to setup the windows VM. Your VM  is using OVMF

Your GPU is unlikely to have a UEFI compatible vbios so you will need to install a new windows VM but using seabios.

To select seabios you need to toggle the advanced options on the VM template top right.

It should work if you use seabios.

 

gridrunner, thank you for pitching in.
Just to be sure, my solution will be to create a brand New Windows 10 VM and make sure it uses Seabios instead of OVMF, and I can't just modify my current Windows 10 VM.

 

Thanks

 

that's right. needs a newly setup vm.. old GPU... I overlooked that part... You'll still need the syslinux.cfg change on you board though.

If that card has HDMI, make sure to passthough the audio from the videocard as well. VMs often don't seem to like not having the HDMI audio passed through (even if you don't need it).

 

If it doesn't have HDMI (It looks like that card came out almost 10 years ago), I'm not sure it would have audio to passthrough though... (Maybe some fancier DVI cards?).

Edited April 7, 20179 yr by DoeBoye

Ok guys, tried last suggestions, VM recognize the card but getting the following error 

 

     "Windows has stopped this device because it has reported problems. (Code 43)"

 

Already made sure Hyper V was disabled.

 

This is how the new VM xml looks

 

 

 

 

 

<domain type='kvm'>
  <name>Windows 10</name>
  <uuid>b81687c1-7f84-bd12-371d-e8b28c8e569a</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>6</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='8'/>
    <vcpupin vcpu='1' cpuset='9'/>
    <vcpupin vcpu='2' cpuset='10'/>
    <vcpupin vcpu='3' cpuset='11'/>
    <vcpupin vcpu='4' cpuset='16'/>
    <vcpupin vcpu='5' cpuset='17'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='off'/>
      <vapic state='off'/>
      <spinlocks state='off'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='3' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Windows10.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.126-2.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:d1:30:68'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc52b'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>

You could try passing through the vbios.

The newest vbios i can see is this one so i would try that (if your card is 1024meg) . https://www.techpowerup.com/vgabios/142386/evga-9500gt-1024-121012

 

I dont think you will need to edit the vbios of this card but if you do then please see my guide here

 

You will need to  add the file to your xml you will need to do a manual edit. So after make sure you dont edit in the gui or else that custom edit will go.

You can see how in the guide here. just ignore the parts about dumping your own vbios

 

Excellent news people,

 

Passing through the vbios did the trick.
I'm also able to pass-through input devices such as keyboard and mouse.

Because this is an old video card the bios dump doesn't have any special heather so I didn't have to do the "Heather removal process" using a hex editor.

 

Thank you very much for the great support.

I will continue my test phase and then move to the final hardware.

 

Best regards and thanks again.