cgi2099 Posted December 21, 2019 Share Posted December 21, 2019 I cannot seem to get this to work, my set up is a ryzen 2700x, Crosshair VII, 1060 6GB, 970 evo, x6 2tb hdd, 128ssd cache, 32gb unraid flash drive I have followed the video on dumping and editing my vbios for the 1060GB, plus I have the one from techpowerup What I have tried: SeaBios & OVMF in multiple configurations with and without hyperV Different vBios for GPU Made sure IOMMU was on and that the card was in its own group Idk if it matters but I have passed through my NVME for the install drive Here is my log: -object 'secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-2-Windows 10 Main/master-key.aes' \ -machine pc-q35-4.1,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off \ -cpu host,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vendor-id=none \ -drive file=/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on \ -drive file=/etc/libvirt/qemu/nvram/b38791be-3bb3-0aba-cde9-6114524512c5_VARS-pure-efi.fd,if=pflash,format=raw,unit=1 \ -m 16384 \ -overcommit mem-lock=off \ -smp 14,sockets=1,cores=14,threads=1 \ -uuid b38791be-3bb3-0aba-cde9-6114524512c5 \ -display none \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=32,server,nowait \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=localtime \ -no-hpet \ -no-shutdown \ -boot strict=on \ -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \ -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \ -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \ -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \ -device pcie-root-port,port=0x8,chassis=5,id=pci.5,bus=pcie.0,addr=0x1 \ -device pcie-pci-bridge,id=pci.6,bus=pci.1,addr=0x0 \ -device nec-usb-xhci,p2=15,p3=15,id=usb,bus=pcie.0,addr=0x7 \ -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \ -netdev tap,fd=35,id=hostnet0,vhost=on,vhostfd=36 \ -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:da:8e:4a,bus=pci.3,addr=0x0 \ -chardev pty,id=charserial0 \ -device isa-serial,chardev=charserial0,id=serial0 \ -chardev socket,id=charchannel0,fd=37,server,nowait \ -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \ -device usb-tablet,id=input0,bus=usb.0,port=1 \ -device 'vfio-pci,host=0000:0a:00.0,id=hostdev0,bus=pci.6,addr=0x1,romfile=/mnt/user/VM Stuff/Graphics Bios/EVGA FTW+ GTX 1060 6G(unraid).rom' \ -device vfio-pci,host=0000:01:00.0,id=hostdev1,bus=pci.4,addr=0x0 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on 2019-12-21 03:54:32.206+0000: Domain id=2 is tainted: high-privileges 2019-12-21 03:54:32.206+0000: Domain id=2 is tainted: host-cpu char device redirected to /dev/pts/0 (label charserial0) Here is my XML: <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>Windows 10 Main</name> <uuid>b38791be-3bb3-0aba-cde9-6114524512c5</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>16777216</memory> <currentMemory unit='KiB'>2097152</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>14</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='9'/> <vcpupin vcpu='2' cpuset='2'/> <vcpupin vcpu='3' cpuset='10'/> <vcpupin vcpu='4' cpuset='3'/> <vcpupin vcpu='5' cpuset='11'/> <vcpupin vcpu='6' cpuset='4'/> <vcpupin vcpu='7' cpuset='12'/> <vcpupin vcpu='8' cpuset='5'/> <vcpupin vcpu='9' cpuset='13'/> <vcpupin vcpu='10' cpuset='6'/> <vcpupin vcpu='11' cpuset='14'/> <vcpupin vcpu='12' cpuset='7'/> <vcpupin vcpu='13' cpuset='15'/> </cputune> <os> <type arch='x86_64' machine='pc-q35-4.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/b38791be-3bb3-0aba-cde9-6114524512c5_VARS-pure-efi.fd</nvram> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='14' threads='1'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <controller type='usb' index='0' model='nec-xhci' ports='15'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0x8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </controller> <controller type='pci' index='6' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:da:8e:4a'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/VM Stuff/Graphics Bios/EVGA FTW+ GTX 1060 6G(unraid).rom'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x01' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> </domain> Quote Link to comment
juan11perez Posted December 21, 2019 Share Posted December 21, 2019 (edited) Two things you need to recheck. latest motherboard bios and Start unraid in legacy mode Edited December 21, 2019 by juan11perez Quote Link to comment
cgi2099 Posted December 21, 2019 Author Share Posted December 21, 2019 I will update bios asap, to start in legacy mode do I just boot the flash drive that doesn't say uefi, or is there a setting on the web gui? Quote Link to comment
juan11perez Posted December 21, 2019 Share Posted December 21, 2019 For legacy main > flash under the syslinux config section uncheck the box that says “Permit UEFI boot mode” Quote Link to comment
cgi2099 Posted December 21, 2019 Author Share Posted December 21, 2019 I updated the bios and changed to legacy mode for unraid boot and repeated all the different combos of SeaBios & OVMF. Still all I get is a black screen, does unraid still need two graphics cards, one for unraid and one for passthrough? Thought I read somewhere that was no longer needed Quote Link to comment
juan11perez Posted December 21, 2019 Share Posted December 21, 2019 I have the setup I suggested on asus x470-f and ryzen 3900x. i have 2 cards , slot 1 and 2. i can pass both of them my win10 vm is on SeaBios Quote Link to comment
cgi2099 Posted December 21, 2019 Author Share Posted December 21, 2019 I am not sure what I am doing wrong, here are my bios, one edited one untouched, maybe I messed up on them: https://drive.google.com/open?id=1594ZvBHLVEt0scp47D1t3hzbVrDT4LAh Are you running SeaBios Q35-4.1 or i440fx-4.1? Quote Link to comment
juan11perez Posted December 21, 2019 Share Posted December 21, 2019 SeaBios i440fx-4.1 perhaps you can try another vm. when I started I found Linux Mint to be capable of pulling any required driver out of the box. I just installed it, passed the card and came up without any effort. 1 Quote Link to comment
cgi2099 Posted December 21, 2019 Author Share Posted December 21, 2019 If I get another VM working will that allow for a win10 vm to start working? Does it install a driver to unraid? Quote Link to comment
cgi2099 Posted December 22, 2019 Author Share Posted December 22, 2019 So I have tired a bunch of different Linux distros and still no dice, when I select the 1060 to pass through I just get a black screen like on the win10 vm. Can you remember what all you had to enable on the crosshair vii in the bios, maybe I am missing something there? Quote Link to comment
perhansen Posted December 22, 2019 Share Posted December 22, 2019 Have you tried moving the GPU to another slot?I know for sure, that i will get a blank screen if i dont use the vbios from techpowerup. Sent from my iPhone using Tapatalk Quote Link to comment
cgi2099 Posted December 23, 2019 Author Share Posted December 23, 2019 14 hours ago, perhansen said: Have you tried moving the GPU to another slot? I know for sure, that i will get a blank screen if i dont use the vbios from techpowerup. Sent from my iPhone using Tapatalk I will try in a different slot, do you get a black screen still if you dump your own vbios and use it? Quote Link to comment
cgi2099 Posted December 23, 2019 Author Share Posted December 23, 2019 It is a no go on putting the GPU in a different slot Quote Link to comment
cgi2099 Posted December 23, 2019 Author Share Posted December 23, 2019 It is a no go on putting the GPU in a different slot Wonder if I am just unlucky and this is my problem: Quote Link to comment
perhansen Posted December 23, 2019 Share Posted December 23, 2019 7 hours ago, cgi2099 said: I will try in a different slot, do you get a black screen still if you dump your own vbios and use it? I never did get my own vbios to work, because you need to manually remove the header from the file. Maybe you could try that. SpaceInvaderOne has some excellent guides on pass-through and editing vbios. I use SeaBios and Q35-4.0.1. I can see you using the newest q35-4.1, i don't know if that makes a different. I also had a lot of problems when i pass-through the USB-controller and Serial bus controller from the Vvidia GPU. Do you pass-through the Nvidia audio controller? If not, you should, that has also been giving people problems. 1 Quote Link to comment
cgi2099 Posted December 23, 2019 Author Share Posted December 23, 2019 I have watched all his videos on this and other stuff, I started with editing my header out of my vbios. I then edited the header out of the techpowerup vbios as well. I've also tried using both unedited. I have not tried q35-4.0.1, I will give that a shot. Also I did not pass the GPU audio, I will try that too. I am only passing my nvme drive and gpu at the moment Thank you so much for helping 😁 Quote Link to comment
SpaceInvaderOne Posted December 24, 2019 Share Posted December 24, 2019 Hi @cgi2099 thanks for the PM that you sent me. I thought that i would reply here in the thread so people with similar probs may be able to be helped by this?? So you have a Ryzen 2700x cpu in an ASUS X470 ROG CROSSHAIR VII motherboard. You say you are on the latest bios yes? So is that version 2901 released on the 2019/11/04 ? So passthrough isn't working I am guessing that you are getting no error messages and just a black screen when the vm starts with the GPU passed through. I have been in the exact same position as you are now using a 2700 and an Asrock board. It is basically because you are on the latest bios. Looking through the bios releases on the Asus page i think the best version for your board for the 2700x would be Version 1201 from early 2019. When the support for Ryzen gen 3 was added it seemed to break passthrough. Check my video here. https://www.youtube.com/watch?v=l-WHCCwgyDg I had to downgrade my bios to get it working again. Now this can be risky downgrading and I haven't tried on a Asus motherboard before. But if you want to try I have a video here https://www.youtube.com/watch?v=ZzqwjVDKAnU Now I have changed my cpu in the server since to 3900x and upgraded the bios onto the version that I had the problem with the 2700 on the Asrock x470 Tiachi. So it seems that the 3rd gen doesn't have the problem with the newer bios with passthrough, well it didn't have for me with the Asrock board. There is one thing that you can try which may help you. Do you have any old GPU lying around or one that you could borrow from a friend to test with. If so put that GPU in as the primary GPU. (Unraid console should display out of that) then passthrough the 1060 as a second GPU. Also you shouldn't need vbios passed through when it is the second GPU so try without first. (Although i personally prefer to always passthrough the vbios for any GPU ) If that still doesn't work swap the GPUs over. So the 1060 is primary and displays the Unraid console then passthrough 1060 as primary. (this time you must have the vbios passed through. And if vbios is downloaded from techpower up it also must be correctly edited - removing header) Hopefully this might work. A friend upgraded the bios on his motherboard to the latest and also had second gen Ryzen and wasn't affected by passthough breaking. He had 2 GPUs in his server and I only had one. So its worth a try. Now if you do decide to attempt to downgrade your bios on your motherboard please, please be careful. There is a risk of bricking the board should anything go wrong. Whilst this maybe ok if you still have warranty on the motherboard it will mean your server wont work at all 1 1 Quote Link to comment
cgi2099 Posted December 24, 2019 Author Share Posted December 24, 2019 @SpaceInvaderOne Thank you so much for replying 😁 also your videos have been so helpful and I definitely couldn't have set my server up without them, pretty much everything I have setup was from one lol! I have ordered a cheap Quadro 600 off eBay and it will be here Friday. I have two bios slots on my motherboard so I will flashback to 1201 on one of them and see if that does it. Lastnight I downgraded unraid back to 6.8.0-rc7 hoping the newer Linux kernel would help but was still getting the black screens with no errors. Do you think I should stay on this version or go back to 6.8.0 stable with the older kernel? Hopefully I can try the flashback today and report back Again thank you and everyone else so much for helping me and going on this trouble shooting adventure with me! Quote Link to comment
cgi2099 Posted December 24, 2019 Author Share Posted December 24, 2019 Reporting back from a working Win10 VM with a GPU successfully passed through :). Downgraded my crosshair vii bios to 1201 like @SpaceInvaderOne said and everything is working great!! Thanks so much for everything!! 1 Quote Link to comment
SpaceInvaderOne Posted December 24, 2019 Share Posted December 24, 2019 2 hours ago, cgi2099 said: Reporting back from a working Win10 VM with a GPU successfully passed through :). Downgraded my crosshair vii bios to 1201 like @SpaceInvaderOne said and everything is working great!! Thanks so much for everything!! Ah that's great glad you have it working now Quote Link to comment
juan11perez Posted December 25, 2019 Share Posted December 25, 2019 I want to share that I also had the 2700x on the Asus x470-F initially and I did have to downgrade the board bios to 4207 from Dec 2018 to be able to pass-through. Before changing the cpu to the 3900x, I upgraded to the latest bios 5406 Nov 25, 2019 and I was able to passthrough while still using the 2700x. Meaning that bios fixed whatever problem was preventing pass-through. I immediately swapped the cpu for the 3900x and it also worked without issue. I have 2 cards and I can pass vms on both cards (at the same time) without problem. Quote Link to comment
mattz Posted December 28, 2019 Share Posted December 28, 2019 On 12/22/2019 at 7:32 PM, cgi2099 said: It is a no go on putting the GPU in a different slot Wonder if I am just unlucky and this is my problem: @cgi2099 - It sounds like this was the issue causing your errors. Downgrading your motherboard BIOS fixed it, there is a new patch from AMD that is having success so you can update your BIOS to this version: AGESA AMD BIOS version 1.0.0.4 Patch B (1004 B). It started rolling out in BIOSes after late November 2019. Quote Link to comment
cgi2099 Posted December 29, 2019 Author Share Posted December 29, 2019 @mattz thank you, it seems Asus just released that bios for my board on 12/27/19 (yesterday lol), I am a little iffy about updating since I have everything working now on this old bios from the start of 2019, but I am willing to give it a go if there has been a lot of success updating to the agesa 1.0.0.4B My boards just released Bios: https://www.asus.com/us/Motherboards/ROG-CROSSHAIR-VII-HERO-WI-FI/HelpDesk_BIOS/ Quote Link to comment
cgi2099 Posted December 29, 2019 Author Share Posted December 29, 2019 @mattz @SpaceInvaderOne I can confirm the new bios (3004) for the Crosshair VII with the agesa 1.0.0.4B update is working with pass-through with a single GPU Thanks so much! Quote Link to comment
mattz Posted December 30, 2019 Share Posted December 30, 2019 11 hours ago, cgi2099 said: @mattz @SpaceInvaderOne I can confirm the new bios (3004) for the Crosshair VII with the agesa 1.0.0.4B update is working with pass-through with a single GPU Woo hoo! Glad it worked for you. I understand being hesitant to update the BIOS, but I think having an up-to-date BIOS, if not the latest, is super important. Cheers. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.