Black screen when installing nvidia drivers - UNRAID


Recommended Posts

I am running a windows 10 vm on top of unraid 6.5.3.

GPU and USB passthrough work just fine, but when I try to install the nvidida graphics drivers, my screen goes dark and the vm becomes unresponsive.

This is what I get from the logs:

 

2018-08-28 23:29:51.279+0000: starting up libvirt version: 4.0.0, qemu version: 2.11.1, hostname: Tower
LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/local/sbin/qemu -name 'guest=Windows 10,debug-threads=on' -S -object 'secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-3-Windows 10/master-key.aes' -machine pc-i440fx-2.11,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=none -drive file=/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/etc/libvirt/qemu/nvram/14766b46-90ec-6d64-3057-606a07f63bf3_VARS-pure-efi.fd,if=pflash,format=raw,unit=1 -m 4096 -realtime mlock=off -smp 4,sockets=1,cores=4,threads=1 -uuid 14766b46-90ec-6d64-3057-606a07f63bf3 -display none -no-user-config -nodefaults -chardev 'socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-3-Windows 10/monitor.sock,server,nowait' -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-hpet -no-shutdown -boot strict=on -device ichy,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev 'socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-3-Windows 10/org.qemu.guest_agent.0,server,nowait' -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device vfio-pci,host=07:00.0,id=hostdev0,bus=pci.0,addr=0x5,romfile=/mnt/user/isos/asusgtx1050ti.rom -device vfio-pci,host=07:00.1,id=hostdev1,bus=pci.0,addr=0x6 -device vfio-pci,host=08:00.0,id=hostdev2,bus=pci.0,addr=0x8 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x9 -msg timestamp=on
2018-08-28 23:29:51.279+0000: Domain id=3 is tainted: high-privileges
2018-08-28 23:29:51.279+0000: Domain id=3 is tainted: host-cpu
2018-08-28T23:29:51.330124Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/0 (label charserial0)
2018-08-28T23:29:52.055274Z qemu-system-x86_64: -device vfio-pci,host=07:00.0,id=hostdev0,bus=pci.0,addr=0x5,romfile=/mnt/disk1/isos/asusgtx1050ti.rom: Failed to mmap 0000:07:00.0 BAR 3. Performance may be slow
2018-08-28T23:53:34.843401Z qemu-system-x86_64: vfio_region_write(0000:07:00.0:region3+0x40a8, 0xfffe101,8) failed: Device or resource busy
KVM internal error. Suberror: 1
emulation failure
RAX=ffffa37ff2215000 RBX=ffffa37ff2215000 RCX=ffffa37ff2215000 RDX=0000000000000000
RSI=ffffe486bef7f010 RDI=ffffe486bb6ad000 RBP=ffffe486bb48a000 RSP=fffff486efd950c8
R8 =0000000000001000 R9 =0101010101010101 R10=fffff802e39ecd08 R11=fffff486efd94e80
R12=ffffe486bce4e920 R13=ffffe486bb157360 R14=0000000000000000 R15=0000000000100000
RIP=fffff802e3d10978 RFL=00000216 [----AP-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS [-WA]
CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0018 0000000000000000 00000000 00409300 DPL=0 DS [-WA]
DS =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS [-WA]
FS =0053 0000000000000000 0000bc00 0040f300 DPL=3 DS [-WA]
GS =002b ffff938021fc6000 ffffffff 00c0f300 DPL=3 DS [-WA]
LDT=0000 0000000000000000 0000ffff 00000000
TR =0040 ffff938021fd6000 00000067 00008b00 DPL=0 TSS64-busy
GDT= ffff938021fd7fb0 00000057
IDT= ffff938021fd5000 00000fff
CR0=80050033 CR2=0000000001f81044 CR3=00000000001ad000 CR4=001506f8
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000004d01
Code=66 66 66 66 0f 1f 84 00 00 00 00 00 66 48 0f 6e c2 0f 16 c0 <0f> 11 01 4c 03 c1 48 83 c1 10 48 83 e1 f0 4c 2b c1 4d 8b c8 49 c1 e9 07 74 2f 0f 29 01 0f

This is my XML config for my VM:

<domain type='kvm'>
  <name>Windows 10</name>
  <uuid>14766b46-90ec-6d64-3057-606a07f63bf3</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.11'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/14766b46-90ec-6d64-3057-606a07f63bf3_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Win10_1803_English_x64.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.141-1.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:c5:60:d7'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/asusgtx1050ti.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </memballoon>
  </devices>
</domain>

I am a newbie to unraid,

can anybody help?

Link to comment

Is your 1050Ti the primary GPU? If so turn off VM. Try these commands from ssh and then start VM and see how it goes.

Quote

echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

Also:

  • Post screenshot of your VM set up
  • How did u do your vbios file?
Link to comment
8 hours ago, testdasi said:

Is your 1050Ti the primary GPU? If so turn off VM. Try these commands from ssh and then start VM and see how it goes.

Also:

  • Post screenshot of your VM set up
  • How did u do your vbios file?
<domain type='kvm'>
  <name>Windows 10 GamingVM</name>
  <uuid>2516c62f-d484-1394-3200-6a2b52a69d00</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>6</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='5'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-2.11'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/2516c62f-d484-1394-3200-6a2b52a69d00_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='6' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10 GamingVM/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Win10_1803_English_x64.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.141-1.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0xe'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0xf'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:8c:e6:3b'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/Asus.GTX1050Ti.4096.161020.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

Screen Shot 2018-09-01 at 11.59.40 AM.png

Asus.GTX1050Ti.4096.161020.rom

Edited by sg14_96
Link to comment
On 9/1/2018 at 3:44 AM, testdasi said:

Is your 1050Ti the primary GPU? If so turn off VM. Try these commands from ssh and then start VM and see how it goes.

Also:

  • Post screenshot of your VM set up
  • How did u do your vbios file?

Ok so now I have the infamous code 43. The nvidia driver installer says it was able to  install the drivers. but I still get the error and my resolution is still maxed at 800X600. Any ideas.

Link to comment
3 minutes ago, sg14_96 said:

I have a gtx1050ti.

I downloaded the bios from techpowerup.

I tried running it with the modified bios according to space invader one's tutorial for single gpu servers.

The GPU and audio are both in the same iommu group separate from other devices.

Which 1050Ti?

 

Downloading bios from techpowerup can get tricky to pick the right version. The file you downloaded looks to be this one:

Quote

Asus    GTX 1050 Ti 4 GB STRIX OC 2016-10-20 00:00:00    86.07.22.00.6A    PCI    1380 / 1752    GDDR5

Why did you pick that version?

 

 

Link to comment

Asus strix oc edition. And I picked the bios at random, without a specific reason.

I know the file is available from the site. WHY did you pick that particular one? Randomly? First one on the list? Last one on the list? How?
 
Also, you are not answering my question: WHICH 1050Ti?
 
 


Sent from my PH-1 using Tapatalk

Link to comment
11 hours ago, sg14_96 said:

Asus strix oc edition. And I picked the bios at random, without a specific reason.

 


Sent from my PH-1 using Tapatalk
 

Ok that's sort of good news. Using the wrong bios is known to cause weird issues - been there, done that 😓

 

Ideally, it's better to dump your own bios in unRAID so if you have a spare GPU (or spare £30 for a cheapo GPU off eBay) then that's the best option to make sure you have the right BIOS.

 

Failing that, it's still better to dump bios using GPUz in a Windows envi. So if you can somehow boot barebone to Windows, dump bios and edit it as per Ed's instructions.

 

As a last resort, try to use a different Techpowerup bios. Website listed 2 for your card, the one you used was older with "PCI" interface so might not be the right one. Try the one below instead perhaps (remember to edit it as per Ed's instructions).

https://www.techpowerup.com/vgabios/192938/asus-gtx1050ti-4096-170122-1

 

Then start a fresh Windows template (in case old edits cause problems) and uninstall the Nvidia driver first before retrying.

Now if things still fail then try the below:

  • Check IOMMU group
  • Try a different PCIe slot - don't forget to recheck IOMMU group
  • Check your CPU mounting - screwing it too tightly can cause strange issues with PCIe interface
  • Create a new template using SeaBios instead of OVMF - sometimes that works (but unlikely since the 1050Ti is relatively new)
  • Boot unRAID in legacy mode instead of UEFI - I have seen posts reporting UEFI occupying conflicting memory with the GPU that is resolved by booting in legacy mode
  • Turn on / off Hyper V
  • Re-dump bios using the first 2 methods (i.e. don't rely on a downloaded file)

If all of the above fail then there's really nothing else I can think of - it's at advanced level from that point on.

Link to comment
  • 2 weeks later...
On 9/10/2018 at 4:49 PM, testdasi said:

Glad that I was able to help. Enjoy.

Hi again,

I have been using the VM and has been working fine.

But from one day to the next it no longer works.

Device manager on windows does not show my GPU anymore.

Here is the warning I get from the Logs:

2018-09-22T15:42:19.362479Z qemu-system-x86_64: -device vfio-pci,host=07:00.0,id=hostdev0,bus=pci.0,addr=0x5,romfile=/mnt/disk1/isos/Santi-GPU-Bios-Dump.rom: Failed to mmap 0000:07:00.0 BAR 3. Performance may be slow

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.