Help with Pass-through, tried everything I can find on google

cgi2099

By cgi2099
in VM Engine (KVM)

Recommended Posts

cgi2099 Rookie

cgi2099

Posted
Posted

I cannot seem to get this to work, my set up is a ryzen 2700x, Crosshair VII, 1060 6GB, 970 evo, x6 2tb hdd, 128ssd cache, 32gb unraid flash drive

 

I have followed the video on dumping and editing my vbios for the 1060GB, plus I have the one from techpowerup

 

What I have tried:

SeaBios & OVMF in multiple configurations with and without hyperV

Different vBios for GPU

Made sure IOMMU was on and that the card was in its own group

 

Idk if it matters but I have passed through my NVME for the install drive

 

Here is my log:

-object 'secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-2-Windows 10 Main/master-key.aes' \
-machine pc-q35-4.1,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off \
-cpu host,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vendor-id=none \
-drive file=/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on \
-drive file=/etc/libvirt/qemu/nvram/b38791be-3bb3-0aba-cde9-6114524512c5_VARS-pure-efi.fd,if=pflash,format=raw,unit=1 \
-m 16384 \
-overcommit mem-lock=off \
-smp 14,sockets=1,cores=14,threads=1 \
-uuid b38791be-3bb3-0aba-cde9-6114524512c5 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=32,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x8,chassis=5,id=pci.5,bus=pcie.0,addr=0x1 \
-device pcie-pci-bridge,id=pci.6,bus=pci.1,addr=0x0 \
-device nec-usb-xhci,p2=15,p3=15,id=usb,bus=pcie.0,addr=0x7 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \
-netdev tap,fd=35,id=hostnet0,vhost=on,vhostfd=36 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:da:8e:4a,bus=pci.3,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=37,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-device 'vfio-pci,host=0000:0a:00.0,id=hostdev0,bus=pci.6,addr=0x1,romfile=/mnt/user/VM Stuff/Graphics Bios/EVGA FTW+ GTX 1060 6G(unraid).rom' \
-device vfio-pci,host=0000:01:00.0,id=hostdev1,bus=pci.4,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2019-12-21 03:54:32.206+0000: Domain id=2 is tainted: high-privileges
2019-12-21 03:54:32.206+0000: Domain id=2 is tainted: host-cpu
char device redirected to /dev/pts/0 (label charserial0)

 

Here is my XML:

 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 10 Main</name>
  <uuid>b38791be-3bb3-0aba-cde9-6114524512c5</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>14</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='9'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='10'/>
    <vcpupin vcpu='4' cpuset='3'/>
    <vcpupin vcpu='5' cpuset='11'/>
    <vcpupin vcpu='6' cpuset='4'/>
    <vcpupin vcpu='7' cpuset='12'/>
    <vcpupin vcpu='8' cpuset='5'/>
    <vcpupin vcpu='9' cpuset='13'/>
    <vcpupin vcpu='10' cpuset='6'/>
    <vcpupin vcpu='11' cpuset='14'/>
    <vcpupin vcpu='12' cpuset='7'/>
    <vcpupin vcpu='13' cpuset='15'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-4.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/b38791be-3bb3-0aba-cde9-6114524512c5_VARS-pure-efi.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='14' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <controller type='usb' index='0' model='nec-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </controller>
    <controller type='pci' index='6' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:da:8e:4a'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/VM Stuff/Graphics Bios/EVGA FTW+ GTX 1060 6G(unraid).rom'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x01' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>
 

Link to comment
cgi2099 Rookie

cgi2099

Posted
  • Author
Posted
14 hours ago, perhansen said:

Have you tried moving the GPU to another slot?
I know for sure, that i will get a blank screen if i dont use the vbios from techpowerup.


Sent from my iPhone using Tapatalk

I will try in a different slot, do you get a black screen still if you dump your own vbios and use it?

Link to comment
perhansen Apprentice

perhansen

Posted
Posted
7 hours ago, cgi2099 said:

I will try in a different slot, do you get a black screen still if you dump your own vbios and use it?

I never did get my own vbios to work, because you need to manually remove the header from the file. Maybe you could try that. SpaceInvaderOne has some excellent guides on pass-through and editing vbios.

I use SeaBios and Q35-4.0.1. I can see you using the newest q35-4.1, i don't know if that makes a different. I also had a lot of problems when i pass-through the USB-controller and Serial bus controller from the Vvidia GPU. Do you pass-through the Nvidia audio controller? If not, you should, that has also been giving people problems.

  • Thanks 1
Link to comment
cgi2099 Rookie

cgi2099

Posted
  • Author
Posted

I have watched all his videos on this and other stuff, I started with editing my header out of my vbios. I then edited the header out of the techpowerup vbios as well. I've also tried using both unedited. I have not tried q35-4.0.1, I will give that a shot. Also I did not pass the GPU audio, I will try that too. I am only passing my nvme drive and gpu at the moment

 

Thank you so much for helping 😁

Link to comment
SpaceInvaderOne Community Regular

SpaceInvaderOne

Posted
Posted

Hi @cgi2099   thanks for the PM that you sent me. I thought that i would reply here in the thread so people with similar probs may be able to be helped by this??

 

So you have a Ryzen 2700x cpu in an ASUS X470 ROG CROSSHAIR VII motherboard. You say you are on the latest bios yes?

So is that  version 2901 released on the 2019/11/04 ?

So passthrough isn't working I am guessing that you are getting no error messages and just a black screen when the vm starts with the GPU passed through.

I have been in the exact same position as you are now using a 2700 and an Asrock board.

It is basically because you are on the latest bios. Looking through the bios releases on the Asus page i think the best version for your board for the 2700x would be Version 1201 from early 2019.

When the support for Ryzen gen 3 was added it seemed to break passthrough. Check my video here. https://www.youtube.com/watch?v=l-WHCCwgyDg

 

I had to downgrade my bios to get it working again. Now this can be risky downgrading and I haven't tried on a Asus motherboard before. But if you want to try I have a video here  https://www.youtube.com/watch?v=ZzqwjVDKAnU

 

 

Now I have changed my cpu in the server since to 3900x and upgraded the bios onto the version  that I had the problem with the 2700 on the Asrock x470 Tiachi. So it seems that the 3rd gen doesn't have the problem with the newer bios with passthrough, well it didn't have for me with the Asrock board.

 

There is one thing that you can try which may help you. Do you have any old GPU lying around or one that you could borrow from a friend to test with.

If so put that GPU in as the primary GPU. (Unraid console should display out of that) then passthrough the 1060 as a second GPU. Also you shouldn't need vbios passed through when it is the second GPU so try without first. (Although i personally prefer to always passthrough the vbios for any GPU )

 

If that still doesn't work swap the GPUs over. So the 1060 is primary and displays the Unraid console then passthrough 1060 as primary. (this time you must have the vbios passed through. And if vbios is downloaded from techpower up it also must be correctly edited - removing header)

 

Hopefully this might work. A friend upgraded the bios on his motherboard to the latest and also had second gen Ryzen and wasn't affected by passthough breaking. He had 2 GPUs in his server and I only had one. So its worth a try.

 

Now if you do decide to attempt to downgrade your bios on your motherboard please, please be careful. There is a risk of bricking the board should anything go wrong.

Whilst this maybe ok if you still have warranty on the motherboard it will mean your server wont work at all

  • Like 1
  • Thanks 1
Link to comment
cgi2099 Rookie

cgi2099

Posted
  • Author
Posted

@SpaceInvaderOne Thank you so much for replying 😁 also your videos have been so helpful and I definitely couldn't have set my server up without them, pretty much everything I have setup was from one lol!

 

I have ordered a cheap Quadro 600 off eBay and it will be here Friday. I have two bios slots on my motherboard so I will flashback to 1201 on one of them and see if that does it. Lastnight I downgraded unraid back to 6.8.0-rc7 hoping the newer Linux kernel would help but was still getting the black screens with no errors. Do you think I should stay on this version or go back to 6.8.0 stable with the older kernel?

 

Hopefully I can try the flashback today and report back 

 

Again thank you and everyone else so much for helping me and going on this trouble shooting adventure with me!

Link to comment
juan11perez Explorer

juan11perez

Posted
Posted

I want to share that I also had the 2700x on the Asus x470-F initially and I did have to downgrade the board bios to 4207 from Dec 2018 to be able to pass-through.

 

Before changing the cpu  to the 3900x, I upgraded to the latest bios 5406 Nov 25, 2019 and I was able to passthrough while still using the 2700x. Meaning that bios fixed whatever problem was preventing pass-through. I immediately swapped the cpu for the 3900x and it also worked without issue.

 

I have 2 cards and I can pass vms on both cards (at the same time) without problem.

Link to comment
mattz Rookie

mattz

Posted
Posted
On 12/22/2019 at 7:32 PM, cgi2099 said:

It is a no go on putting the GPU in a different slot :( Wonder if I am just unlucky and this is my problem: 

 

@cgi2099 - It sounds like this was the issue causing your errors.  Downgrading your motherboard BIOS fixed it, there is a new patch from AMD that is having success so you can update your BIOS to this version:  AGESA AMD BIOS version 1.0.0.4 Patch B (1004 B).  It started rolling out in BIOSes after late November 2019.

Link to comment
cgi2099 Rookie

cgi2099

Posted
  • Author
Posted

@mattz thank you, it seems Asus just released that bios for my board on 12/27/19 (yesterday lol), I am a little iffy about updating since I have everything working now on this old bios from the start of 2019, but I am willing to give it a go if there has been a lot of success updating to the agesa 1.0.0.4B

 

My boards just released Bios: https://www.asus.com/us/Motherboards/ROG-CROSSHAIR-VII-HERO-WI-FI/HelpDesk_BIOS/

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing