I know ugly patch is dead, but I'm guessing a number of Threadripper owners, or "want to's," are watching this thread. @ryoko227 found an interesting issue with secondary GPU vm's so I thought I would re-post here for exposure.

On 12/20/2017 at 10:36 AM, Rhynri said:

Thank you so much for this!  I've been trying to do this on my own for a while.  It works just fine on my system, 1950x, Zenith Extreme, 1080 GTX, 1060 GTX

 

Update:

I had efifb attach to some of the 1080's memory, which was throwing a lot of vfio_region_write errors into the log and not allowing the Nvidia drivers to detect the card. I'm not really sure why that would be, but in case it happens to you, the solution (from here) is this:

 

echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

 

 

 

Does this work on Threadripper? If so, how do you apply this? 

 

I am running a X399 GAMING PRO CARBON AC board with the 1920x processor. I am able to passthrough my GTX1060 but I get the Code 43 error and I am unable to apply any drivers or see output to any attached monitor. Please help I'd really love to get access to my VM. 

Try adding/editing this into your VMs XML:

 

(TOP)

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> 

 

(BOTTOM BEFORE </domain>)

<qemu:commandline>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='host,hv_time,kvm=off,hv_vendor_id=null'/>
</qemu:commandline>

 

Edited March 1, 20188 yr by Rhynri

1 hour ago, Rhynri said:

Try adding/editing this into your VMs XML:

 

(TOP)

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> 

 

(BOTTOM BEFORE </domain>)

<qemu:commandline>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='host,hv_time,kvm=off,hv_vendor_id=null'/>
</qemu:commandline>

 

 

I added both just as you mentioned and I am still receiving the Code 43 error after I reboot the VM. Any other suggestions? I am on Unraid 6.4.1 currently. Latest BIOS on my board. 

On 2/28/2018 at 4:44 PM, ronni3 said:

I am able to passthrough my GTX1060 but I get the Code 43 error and I am unable to apply any drivers or see output to any attached monitor. Please help I'd really love to get access to my VM. 

Out of curiosity @ronni3 are you using GUI-boot, or regular/text boot?  I noticed some seriously funky ^%$# when I tried GUI-boot with my VM. In GUI-boot, on a brand new VM, it would be stable BUT the Xorg graphical environment would "bleed" into the screen of my vm -- it was like watch a laggy VNC desktop update/draw over the mouse or fields would change; once I installed the nvidia driver for guest (win10) the bleeding stopped but 20-30 seconds and *bang* started getting errors/crashes in Windows.  I still haven't been able to get a suitable second graphics card to play, and with the market how it is I may be a while.  If I boot into normal mode, graphics don't seem to be an issue.

9 minutes ago, Jcloud said:

Out of curiosity @ronni3 are you using GUI-boot, or regular/text boot?  I noticed some seriously funky ^%$# when I tried GUI-boot with my VM. In GUI-boot, on a brand new VM, it would be stable BUT the Xorg graphical environment would "bleed" into the screen of my vm -- it was like watch a laggy VNC desktop update/draw over the mouse or fields would change; once I installed the nvidia driver for guest (win10) the bleeding stopped but 20-30 seconds and *bang* started getting errors/crashes in Windows.  I still haven't been able to get a suitable second graphics card to play, and with the market how it is I may be a while.  If I boot into normal mode, graphics don't seem to be an issue.

 

I assume by GUI-boot, you mean the standard way Windows boots. If so, then yes, I am booting in GUI-boot. Unsure exactly what text-boot is. 

 

So far I've tried moving the video card (Asus model Dual-GTX1060-O6G) from the PCIe 1 slot to PCIe 4 & 5 with no difference. I have two Intel SAS cards that I've alternated between different slots as I've moved the video card. Still no difference. I then attempted to use Space Invader One's recommended Rom File fix and so far I've had no luck using two different Rom dumps. Then I tried the Rom Dump alongside the suggestion above from Rhynri and that actually shows that it is failing in the log. 

 

Here's what I get when I add the above QEMU and Domain change. This happens with that XML change with and without adding the Rom file. 

 

2018-03-03 01:51:17.366+0000: starting up libvirt version: 4.0.0, qemu version: 2.11.1, hostname: CheshireCat
LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/local/sbin/qemu -name 'guest=Windows 10,debug-threads=on' -S -object 'secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-14-Windows 10/master-key.aes' -machine pc-i440fx-2.11,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off -cpu host -drive file=/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/etc/libvirt/qemu/nvram/ea3a2052-7f49-4c54-ff5a-9ad2c5ac2a91_VARS-pure-efi.fd,if=pflash,format=raw,unit=1 -m 4096 -realtime mlock=off -smp 2,sockets=1,cores=2,threads=1 -uuid ea3a2052-7f49-4c54-ff5a-9ad2c5ac2a91 -display none -no-user-config -nodefaults -chardev 'socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-14-Windows 10/monitor.sock,server,nowait' -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on -device nec-usb-xhci,iver,nowait' -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device vfio-pci,host=41:00.0,id=hostdev0,bus=pci.0,addr=0x5,romfile=/mnt/user/domains/Asus.GTX1060.6144.160706.modified.dump -device vfio-pci,host=41:00.1,id=hostdev1,bus=pci.0,addr=0x6 -device usb-host,hostbus=5,hostaddr=2,id=hostdev2,bus=usb.0,port=1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -cpu host,hv_time,kvm=off,hv_vendor_id=null -msg timestamp=on
2018-03-03 01:51:17.366+0000: Domain id=14 is tainted: high-privileges
2018-03-03 01:51:17.366+0000: Domain id=14 is tainted: custom-argv
2018-03-03 01:51:17.366+0000: Domain id=14 is tainted: host-cpu
2018-03-03T01:51:17.409748Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/0 (label charserial0)

 

I've even tried changing the CPU Mode from passthrough to QEMU and testing everything above again with no luck. Only way it boots if I leave XML untouched but then I am still where I left off with the card in Device Manager reading Code 43. It recognizes the card but it is unable to link to any drivers and no output to a monitor via any of the display ports. 

 

Any other suggestions?

Actually let me clarify one additional thing. 

 

The only time the VM will boot is if I have VNC enabled as the first video card. I can have the GTX1060 attached or not as a second card. Otherwise the VM freezes and will never boot. 

33 minutes ago, ronni3 said:

I assume by GUI-boot, you mean the standard way Windows boots. If so, then yes, I am booting in GUI-boot. Unsure exactly what text-boot is. 

Sorry no, I meant unRAID itself, not the guestOS. Where it has like, normal mode, Safe mode (no plugins), GUI mode, GUI mode (no plugins), memtest86+ options.  (My wording might be slightly off on the unRAID menu options, trying to recall from brain.)

Edited March 3, 20188 yr by Jcloud

2 minutes ago, Jcloud said:

Sorry no, I meant unRAID itself, not the guestOS. Where it has like, normal mode, Safe mode (no plugins), GUI mode, GUI mode (no plugins), memtest86+ options.  (My wording might be slightly off on the unRAID menu options, trying to recall from brain.)

 

I understand the question now. I usually always boot in GUI mode. Not sure I've ever tried any of the other methods. I will give the others a shot right now and see if that makes any difference. Wasn't really aware it mattered if you booted any other way, but definitely worth trying out. 

8 minutes ago, ronni3 said:

. I usually always boot in GUI mode

Try normal mode (plugins should be OK for VM sake and troubleshooting -- wouldn't worry about that). But I really think my hypothesis is the Linux GUI you were in and VM with code 43 are butting heads. You'll want to make your stable VM auto run, or get a laptop/phone browser ready to work with your box when it powers up.

Edited March 3, 20188 yr by Jcloud

2 minutes ago, Jcloud said:

Try normal mode 

Okay so after taking a quick look... seems I've always chose the first option (default) which is the text boot. I just tried kicking off GUI mode and I've definitely never used that option before. 

Just now, ronni3 said:

Okay so after taking a quick look... seems I've always chose the first option (default) which is the text boot. I just tried kicking off GUI mode and I've definitely never used that option before. 

Ah, Okay, sorry sir. As my co-worker would say beautify theory destroyed by ugly facts.

4 minutes ago, Jcloud said:

As my co-worker would say beautify theory destroyed by ugly facts.

 

It's a very beautiful quote. lol  Oh well, back to the drawing board. 

 

One or two more questions before I lose you... I know I could virtualize Unraid by using Vmware ESXi. Would virtualizing Windows 10 in VMware work better and would I still be able to pass through my graphics card if I went that route? Would things work better or worse if I did that?

34 minutes ago, ronni3 said:

I know I could virtualize Unraid by using Vmware ESXi. Would virtualizing Windows 10 in VMware work better and would I still be able to pass through my graphics card if I went that route? Would things work better or worse if I did that?

Very good questions. I honestly don't know. Hyper-visor is going to be different so yeah I can see potential for things working differently, but I don't know if it that difference would be better or worse. I have never used ESXi so I must plead ignorant-ignoramus.

Edited March 3, 20188 yr by Jcloud
One word missing changes everything.

5 minutes ago, Jcloud said:

Very good questions. I honestly don't know. Hyper-visor is going to be different so yeah I can see potential for things working differently, but I don't know if it that difference would be better or worse. I have never used ESXi so I must plead ignorant-ignoramus.

Speaking of hyper visor, make sure you have it off. 

3 minutes ago, Rhynri said:

Speaking of hyper visor, make sure you have it off. 

 

After reading several threads I was well aware that this needed to be off but I thank you for bringing it up. The issue could very well be with this particular board. Most others seem to have the Asus or Asrock boards. Maybe this one has its set of problems??? 

33 minutes ago, ronni3 said:

The issue could very well be with this particular board. Most others seem to have the Asus or Asrock boards. Maybe this one has its set of problems??? 

Another good question. For bios Asus and Asrock are going to be similar as I hoping firmware will have another revision, hopefully less than six months.  Also still hopeful for micro-code tweaking in long run (1-2yr). Another consideration, TR have TWO PCIe controllers, one on each CPU die. It might be that KVM is messing up on this allocation of the correct PCIe controller. This would be BIOS settings and perhaps be of something to try... ? I looked I couldn't find what I was looking for, so if you do look, I'm sorry I don't think it is in the bios, well mine. 

Edited March 3, 20188 yr by Jcloud
more info