Code 43 with nvidia gtx960


Recommended Posts

It seems that no matter what I do, I can't get the code 43 error to go away with passthrough GPUs.  I have 4 GTX960s and am trying to get a 4 gamers 1 cpu build going with my threadripper build.  Here is what I have:

1920x threadripper

X399 Asrock TaiChi (bios 3.20)

1kw EVGA PSU

64 Gb vengeance corsair RAM

4x EVGA GTX960 SSC

 

I have dumped the bios from those cards and am selecting the .rom file within the machine setup.  It does recognize the card- but will not load the driver with a code 43.  It obviously has something to do with not loading because it sees the virtual machine.  I need some help getting things going- and am willing to pay if needed.  This is driving me a little crazy since I went through the guide:

https://forums.unraid.net/topic/69670-guide-fix-nvidia-code-43-issue-on-nvidia-gpu/

and even got an older image of windows 10- but am still having problems. 

 

I ran through the process in the guide with versions 388 and 411 but am having no luck installing- code 43 every time.

 

System info:

IOMMU group 0:[1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge

IOMMU group 1:[1022:1453] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge

IOMMU group 2:[1022:1453] 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge

IOMMU group 3:[1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge

IOMMU group 4:[1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge

IOMMU group 5:[1022:1453] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge

IOMMU group 6:[1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge

IOMMU group 7:[1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge

IOMMU group 8:[1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B

IOMMU group 9:[1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge

IOMMU group 10:[1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B

IOMMU group 11:[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)

[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)

IOMMU group 12:[1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0

[1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1

[1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2

[1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3

[1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4

[1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5

[1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6

[1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7

IOMMU group 13:[1022:1460] 00:19.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0

[1022:1461] 00:19.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1

[1022:1462] 00:19.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2

[1022:1463] 00:19.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3

[1022:1464] 00:19.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4

[1022:1465] 00:19.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5

[1022:1466] 00:19.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6

[1022:1467] 00:19.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7

IOMMU group 14:[1022:43ba] 01:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset USB 3.1 xHCI Controller (rev 02)

IOMMU group 15:[1022:43b6] 01:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset SATA Controller (rev 02)

IOMMU group 16:[1022:43b1] 01:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset PCIe Bridge (rev 02)

IOMMU group 17:[1022:43b4] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)

IOMMU group 18:[1022:43b4] 02:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)

IOMMU group 19:[1022:43b4] 02:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)

IOMMU group 20:[1022:43b4] 02:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)

IOMMU group 21:[1022:43b4] 02:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)

IOMMU group 22:[8086:1539] 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)

IOMMU group 23:[8086:24fb] 05:00.0 Network controller: Intel Corporation Dual Band Wireless-AC 3168NGW [Stone Peak] (rev 10)

IOMMU group 24:[8086:1539] 06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)

IOMMU group 25:[10de:1401] 08:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1)

IOMMU group 26:[10de:0fba] 08:00.1 Audio device: NVIDIA Corporation GM206 High Definition Audio Controller (rev a1)

IOMMU group 27:[10de:1401] 09:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1)

IOMMU group 28:[10de:0fba] 09:00.1 Audio device: NVIDIA Corporation GM206 High Definition Audio Controller (rev a1)

IOMMU group 29:[1022:145a] 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a

IOMMU group 30:[1022:1456] 0a:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor

IOMMU group 31:[1022:145c] 0a:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller

IOMMU group 32:[1022:1455] 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455

IOMMU group 33:[1022:7901] 0b:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

IOMMU group 34:[1022:1457] 0b:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller

IOMMU group 35:[1022:1452] 40:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge

IOMMU group 36:[1022:1453] 40:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge

IOMMU group 37:[1022:1452] 40:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge

IOMMU group 38:[1022:1452] 40:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge

IOMMU group 39:[1022:1453] 40:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge

IOMMU group 40:[1022:1452] 40:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge

IOMMU group 41:[1022:1452] 40:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge

IOMMU group 42:[1022:1454] 40:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B

IOMMU group 43:[1022:1452] 40:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge

IOMMU group 44:[1022:1454] 40:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B

IOMMU group 45:[10de:1401] 41:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1)

IOMMU group 46:[10de:0fba] 41:00.1 Audio device: NVIDIA Corporation GM206 High Definition Audio Controller (rev a1)

IOMMU group 47:[10de:1401] 42:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1)

IOMMU group 48:[10de:0fba] 42:00.1 Audio device: NVIDIA Corporation GM206 High Definition Audio Controller (rev a1)

IOMMU group 49:[1022:145a] 43:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a

IOMMU group 50:[1022:1456] 43:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor

IOMMU group 51:[1022:145c] 43:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller

IOMMU group 52:[1022:1455] 44:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455

IOMMU group 53:[1022:7901] 44:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

 

CPU Thread Pairings

Pair 1:cpu 0 / cpu 12

Pair 2:cpu 1 / cpu 13

Pair 3:cpu 2 / cpu 14

Pair 4:cpu 3 / cpu 15

Pair 5:cpu 4 / cpu 16

Pair 6:cpu 5 / cpu 17

Pair 7:cpu 6 / cpu 18

Pair 8:cpu 7 / cpu 19

Pair 9:cpu 8 / cpu 20

Pair 10:cpu 9 / cpu 21

Pair 11:cpu 10 / cpu 22

Pair 12:cpu 11 / cpu 23

 

USB Devices

Bus 001 Device 001:ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 001 Device 002:ID 8087:0aa7 Intel Corp.

Bus 001 Device 003:ID 2433:b200

Bus 002 Device 001:ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 003 Device 001:ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 003 Device 002:ID 046d:c52b Logitech, Inc. Unifying Receiver

Bus 004 Device 001:ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 005 Device 001:ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 005 Device 002:ID 045e:0745 Microsoft Corp. Nano Transceiver v1.0 for Bluetooth

Bus 006 Device 001:ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 006 Device 002:ID 0781:5583 SanDisk Corp. Ultra Fit

 

SCSI Devices

[0:0:0:0]disk SanDisk Ultra Fit 1.00 /dev/sda 30.7GB

[1:0:0:0]disk ATA WDC WD30PURZ-85G 0A80 /dev/sdb 3.00TB

[2:0:0:0]disk ATA WDC WD30EZRX-00M 0A80 /dev/sdc 3.00TB

[5:0:0:0]disk ATA PLEXTOR PX-512S2 1.04 /dev/sdd 512GB

[7:0:0:0]disk ATA WDC WD30EZRX-00M 0A80 /dev/sde 3.00TB

 

 

VM XML:

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 10-Lance</name>
  <uuid>d6c004a2-4090-d2c2-eeda-2ef31aae118c</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>10485760</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='6'/>
    <vcpupin vcpu='1' cpuset='18'/>
    <vcpupin vcpu='2' cpuset='7'/>
    <vcpupin vcpu='3' cpuset='19'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-3.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/d6c004a2-4090-d2c2-eeda-2ef31aae118c_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='1234567890ab'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10-Lance/vdisk1.img'/>
      <target dev='hdc' bus='scsi'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Win10_1607_English_x64.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.141.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:48:96:de'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/disk1/isos/GM2061304.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>
 

System Log:

ct 2 11:51:50 Tower avahi-daemon[2580]: Leaving mDNS multicast group on interface vnet0.IPv6 with address fe80::fc54:ff:fe48:96de.
Oct 2 11:51:50 Tower kernel: br0: port 3(vnet0) entered disabled state
Oct 2 11:51:50 Tower kernel: device vnet0 left promiscuous mode
Oct 2 11:51:50 Tower kernel: br0: port 3(vnet0) entered disabled state
Oct 2 11:51:50 Tower avahi-daemon[2580]: Withdrawing address record for fe80::fc54:ff:fe48:96de on vnet0.
Oct 2 11:51:51 Tower kernel: vfio-pci 0000:08:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Oct 2 11:52:29 Tower kernel: vfio-pci 0000:08:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Oct 2 11:52:29 Tower kernel: br0: port 3(vnet0) entered blocking state
Oct 2 11:52:29 Tower kernel: br0: port 3(vnet0) entered disabled state
Oct 2 11:52:29 Tower kernel: device vnet0 entered promiscuous mode
Oct 2 11:52:29 Tower kernel: br0: port 3(vnet0) entered blocking state
Oct 2 11:52:29 Tower kernel: br0: port 3(vnet0) entered listening state
Oct 2 11:52:31 Tower kernel: vfio_ecap_init: 0000:08:00.0 hiding ecap 0x1e@0x258
Oct 2 11:52:31 Tower kernel: vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x900
Oct 2 11:52:31 Tower avahi-daemon[2580]: Joining mDNS multicast group on interface vnet0.IPv6 with address fe80::fc54:ff:fe48:96de.
Oct 2 11:52:31 Tower avahi-daemon[2580]: New relevant interface vnet0.IPv6 for mDNS.
Oct 2 11:52:31 Tower avahi-daemon[2580]: Registering new address record for fe80::fc54:ff:fe48:96de on vnet0.*.
Oct 2 11:52:31 Tower kernel: br0: port 3(vnet0) entered learning state
Oct 2 11:52:33 Tower kernel: br0: port 3(vnet0) entered forwarding state
Oct 2 11:52:33 Tower kernel: br0: topology change detected, propagating
Oct 2 13:45:06 Tower avahi-daemon[2580]: Interface vnet0.IPv6 no longer relevant for mDNS.
Oct 2 13:45:06 Tower avahi-daemon[2580]: Leaving mDNS multicast group on interface vnet0.IPv6 with address fe80::fc54:ff:fe48:96de.
Oct 2 13:45:06 Tower kernel: br0: port 3(vnet0) entered disabled state
Oct 2 13:45:06 Tower kernel: device vnet0 left promiscuous mode
Oct 2 13:45:06 Tower kernel: br0: port 3(vnet0) entered disabled state
Oct 2 13:45:06 Tower avahi-daemon[2580]: Withdrawing address record for fe80::fc54:ff:fe48:96de on vnet0.
Oct 2 13:45:07 Tower kernel: vfio-pci 0000:08:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Oct 2 13:46:16 Tower nginx: 2018/10/02 13:46:16 [error] 2634#2634: *87587 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 10.0.0.102, server: , request: "GET /plugins/dynamix.plugin.manager/scripts/plugin&arg1=install&arg2=http://lime-technology.com/forum/index.php?topic=36543.0 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.0.147", referrer: "http://10.0.0.147/Plugins"
Oct 2 13:48:25 Tower emhttpd: req (3): cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=install&arg2=https://raw.github.com/bergware/dynamix/master/unRAIDv6/dynamix.system.info.plg&csrf_token=****************
Oct 2 13:48:25 Tower emhttpd: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin install https://raw.github.com/bergware/dynamix/master/unRAIDv6/dynamix.system.info.plg
Oct 2 13:48:54 Tower root: plugin: running: anonymous
Oct 2 13:48:54 Tower root: plugin: creating: /boot/config/plugins/dynamix.system.info/dynamix.system.info.txz - downloading from URL https://raw.githubusercontent.com/bergware/dynamix/master/archive/dynamix.system.info.txz
Oct 2 13:48:55 Tower root: plugin: checking: /boot/config/plugins/dynamix.system.info/dynamix.system.info.txz - MD5
Oct 2 13:48:55 Tower root: plugin: running: /boot/config/plugins/dynamix.system.info/dynamix.system.info.txz
Oct 2 13:48:55 Tower root: plugin: running: anonymous
Oct 2 14:12:18 Tower kernel: kvm: already loaded the other module

 

I did add some hyper-v and kvm edits to no avail.

 

Any help would be very much appreciated.

 

tower-diagnostics-20181002-1344.zip

Edited by jordanmw
  • Upvote 1
Link to comment

A few things to try.  First, in the motherboard BIOS, do you have an option to enable 4G decoding?  If so, please enable that.  Another thing to try would be to turn on VFIO allow unsafe interrupts.  Also, are you booting using a OVMF or SeaBIOS VM right now?  Try changing that and reinstall windows.

Link to comment

Did you modified the dumped VBIOS of your cards or do you pass them trough without manual edit them first removing the NVIDIA header? @jordanmw I already posted the video from SpaceInvader in the other forum. A dumped VBIOS without the small modifications never worked for me. No matter if I dumped it on my own or used a VBIOS from TechPowerUP. And i only have to do this for the card in the first slot. Get your VM up and running with VNC and install some sort of remote management software like Teamviewer, Anydesk or enable RDP. Make sure you install the VIRTIO drivers especially for the network nic. Check what IP you get assigned to the VM. Test if you can connect and than remove VNC. It might look on the first boot that windows hangs. Watch the CPU load on the Unraid main page and ping the VM to check if the network comes up. 

 

 

  • Upvote 1
Link to comment

No, I haven't edited the bios first.  I am doing as you suggested though- I install teamviewer and can connect to the machine until I add the gpu and reboot.  Then the machine just never boots.  If I add vnc back in with the video card too- it goes to windows setup repair.  I am adding the virtio drivers after installing the os and give it a static ip.  It never responds to ping after a reboot with any of my gpus attached.  Maybe I should hex edit my vbios.  I did dump them from the actual cards- but didn't think I had to edit if I did that.

Link to comment

Followed spaceinvader's guide exactly to setup a new machine- after removing vnc and adding passthrough gpu with rom that was dumped and hexedited- windows does not boot.  Teamviewer and rdp enabled and machine never comes back on the network- checked with ping to IP also.  Anything else that I can post, to help figure this out>?

I also tried one with seabios and it did the same thing.

Machine XML:

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 10</name>
  <uuid>6d351a55-5f4c-d492-618e-48d0dd42f29e</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>10485760</memory>
  <currentMemory unit='KiB'>10485760</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='10'/>
    <vcpupin vcpu='1' cpuset='22'/>
    <vcpupin vcpu='2' cpuset='11'/>
    <vcpupin vcpu='3' cpuset='23'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-3.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/6d351a55-5f4c-d492-618e-48d0dd42f29e_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Win10_1607_English_x64.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.160-1.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:02:39:d8'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x41' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/disk1/isos/GM2061279.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x41' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc52b'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Edited by jordanmw
Link to comment

Did you tried to passthrough another card? In some situations the card in the first PCIE slot can be a bit picky. And don't expect if you have 4 similiar cards that the bios of all of them are the same. Also i would recomend to test one of the BIOS from TechPowerUp. The BIOS i used is from this site. Choose the manufacturer of your card, the specific model and download maybe 2 or 3 versions of the BIOS available. For me it never worked if i dump it on my own and modify it for the EVGA 1050ti. The second one from TechPowerUp i modified and passed through worked.

 

3 hours ago, jordanmw said:

I am also seeing an unidentified device in device manager

check this: 

 

 

Link to comment

The device manager hid issue fix is great, thanks.  I have grabbed the techpowerup bios for my card and tried that- still won't boot windows after I add one card.  Since I have 4 gpus- I have tried each of them and know which one unraid has- based on my screen going black if I assign and start with it attached to a vm. 

 

This is driving me a little crazy- maybe have installed windows 100+ times and just can understand where it is failing.  I literally just started using unraid and bought the pro license thinking this was an easier system to work with than some of the other distros with the same components.... starting to doubt making that call.  I have worked with hyper-v, esxi, and messed around with tons of Linux virtualization- just not with pass through GPUs-- this is starting to drive me a little nuts.

  • Like 1
Link to comment
On 10/2/2018 at 8:37 PM, jonp said:

A few things to try.  First, in the motherboard BIOS, do you have an option to enable 4G decoding?  If so, please enable that.  Another thing to try would be to turn on VFIO allow unsafe interrupts.  Also, are you booting using a OVMF or SeaBIOS VM right now?  Try changing that and reinstall windows.

So, when I enable 4g decoding in the boot menu, the computer will not post.  I ended up resetting the bios multiple times and even with no other bios tweaks, it won't post with it enabled.  I did turn on vfio- what acs setting should I be using?  Here are my current cards:

image.png.8923eb54616eefb94e72e5e6dc2309a4.pngimage.png.6fd6dda7871d07445c3eb7d8a1424296.pngimage.png.3bf79c173a767113fac8caac13a397c6.pngimage.png.05010f613ad6df757eb746bfea279d4e.png

I have dumped each of their bios and hex edited the header.  I am removing all but 2 of the cards, the one that has always been in the top slot and unraid will choose, and the one faster card of the 4- ftw edition.  I can load windows without unraid and put full loads on all cards and p95 the cpu for hours on end with no thermal throttling- so this thing should be rock solid if I can just get unraid working.

 

Still getting the same result for 2 cards installed. VM Windows won't boot without the vnc card added.  It will still load if I add a card, and usually let me install drivers, but on reboot, it won't boot.

 

Edited by jordanmw
Link to comment

Ok, a few more comments to put you on the right path:

 

1)  If enabling 4G Decoding causes the system to not post, you may have defective hardware.  I would open a case with ASRock as that is fairly basic motherboard functionality.

 

2)  What happens if you only have 2 GPUs in the system?  Can the one that Unraid isn't using work correctly?

 

3)  Have you tried any alternate GPUs?  There is no reason the EVGA GeForce 960 shouldn't work because I have that exact same card and it worked fine on my system.

 

4)  Previously I had asked a question and provided a suggestion that I don't see a response to:

On 10/2/2018 at 9:37 PM, jonp said:

Also, are you booting using a OVMF or SeaBIOS VM right now?  Try changing that and reinstall windows.

SeaBIOS is definitely worth trying as for many, it resolves the issue.

 

Edit:  One last thing I may have missed here, but have you updated your motherboard BIOS to latest?

Link to comment

Nope, having some success now- no bad hardware- thank god.  Here is what I have found...

If I install vnc for graphics, it will not install the graphics card drivers, and nvidia detecting virtual systems is for real.  I'll get a code 43 if I install the regular drivers but if I get wdk and use the powershell script for hacking the drivers with version 388.0, I can install those- then upgrade to the 411 branch with no issues.  I just have to do the full install on hardware without ever adding the vnc video.  Never had to pass a rom doing that- just install on physical monitor and thnings go better.  Bios is latest, OVMF for bios- may try some additional testing to see if I can pin down the issues.

Link to comment
29 minutes ago, jordanmw said:

Yeah, I did that registry edit on all 4 machines but it seems like it didn't help.  Do I need to re-install the hd audio driver after I do the edit? 

Are you sure that you enabled MSI on ALL the passed through bits? I seem to remember looking at a list when I did mine and there was an item hiding not sorted with the rest.

Link to comment

We are just talking about a single edit on each machine, right?  Everything that I have followed just suggested finding the one device with the by type view.  I find it- shows HD audio- grab the vendor code, then edit the registry with the corresponding hklm\ccs key to a 1, from a 0.  Am I missing something there?

Link to comment
13 hours ago, jordanmw said:

Nope, having some success now- no bad hardware- thank god.  Here is what I have found...

How did you rule out bad hardware?  Were you able to get the above 4G decoding to work?  That is pretty important for larger system setups.  We needed that in the video where we did 7 gamers on one machine.

13 hours ago, jordanmw said:

If I install vnc for graphics, it will not install the graphics card drivers, and nvidia detecting virtual systems is for real. 

This is actually not your issue.  While NVIDIA makes attempts to detect for virtualization, we have implemented patches that negate that completely.  There are literally TONS of users in our community using NVIDIA GPUs with Unraid VMs and have no problems.  When you get a Code 43 error, it is because of one of a few things.  The most obvious thing is a problem with the hardware, but that's not always the reason.

 

That said, Code 43 will almost always happen if you try to do GPU pass through where VNC is the primary display and the GPU is a secondary.  The big question here is what happens when you try to install windows without VNC (just the GPU).  This should work with both OVMF and SeaBIOS, but if it doesn't work with OVMF, you might want to try SeaBIOS.  Another thing to check is to contact EVGA and see if there are any firmware updates for your card.  EVGA doesn't just have them available for download from their website, you actually need to contact them to check if there are real firmware updates available and they will ask you to send them the mode/serial number for your device(s).  They are VERY responsive to these requests, so I would definitely do it.

 

13 hours ago, jordanmw said:

I'll get a code 43 if I install the regular drivers but if I get wdk and use the powershell script for hacking the drivers with version 388.0, I can install those- then upgrade to the 411 branch with no issues.  I just have to do the full install on hardware without ever adding the vnc video.  Never had to pass a rom doing that- just install on physical monitor and thnings go better.  Bios is latest, OVMF for bios- may try some additional testing to see if I can pin down the issues.

The amount of jumping through hoops you are having to go through to make this work is extremely unusual.  We have a threadripper system in our labs with a GTX 1080 Ti that Eric uses in a Windows VM and it works flawlessly.  Not sure why you're having problems here, but I STRONGLY recommend you attempt to build your VMs using SeaBIOS instead of OVMF.  OVMF should work with any GPUs that support UEFI (which the EVGA 960s do), but just in case, try SeaBIOS to see if that resolves your issue.  It may also solve other issues you're experiencing.

  • Upvote 1
Link to comment

Thanks jonp.  It did feel like I was jumping through way more hoops than I should have to.  I didn't enable the 4G setting and was still able to get everything going if I just pretended that each machine was a separate bare metal install.  I think you are right about getting bios updates for my cards- I will try to get that done this weekend and maybe play around with some of the other things you suggested.  But right now, I have 4 windows 10 installs with 4 cores each and 10Gb of RAM each running p95 and heaven to test things out.  Has been stable for the last 8 hours and no thermal throttling. 

Link to comment

Didn't get as much time as I would like to playing around with settings and rebuilding vms this weekend but did get some quality gaming time on it with the wife.  Things are running pretty well just using ovmf without passing gpu bios roms and have the demon audio issues worked out- just passing the audio from the graphics cards works with the msi fix- didn't apply it to any devices but that.  I think I mentioned that I had a windows 10 install that I can boot from to test overall stability- so I know things are pretty rock solid with the hardware.  It does seem like there is some weirdness with re-assigning usb devices but without dedicated controllers to pass through- I can't complain too much.  I have booted a couple times and seen an error 43 but usually remove and re-install the driver- then it starts working again.  Still experimenting on settings, will post back here with my experiences.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.