February 4, 20233 yr Completely new here! Honestly wouldnt even even be close to where I am with this server without Spaceinvader Ones videos so, theres that... anyway... Im at a complete impasse with this GPU passthrough for the VM so Im finally making a forum post. To start off with here are my system specs and Ive included the anonymised diagnostics for the server (if that helps) CPU: AMD Ryzen 5 5600 6C/12T Unlocked 4.4GHz AM4 65W GPU: Asus GeForce RTX 3060 Ti Dual Mini OC 8GB V2 LHR Graphics Card Motherboard: Gigabyte B550 AORUS PRO V2 AM4 ATX Motherboard Ram: Samsung (4X 16GB) DDR4-2666 PC4-21300 1.2V DR X8 ECC 288-PIN EUDIMM RAM IVe done just about everything in all of Spaceinvader Ones videos as well as a few things Ive read on the forums when setting up the Windows including not using the vertio display driver update (apparently they hate each other? or so some forum post said), pairing the IOMMU group in Tools->Devices for the GPU, editing the XML file of the GPU (and leaving it in XML so it doesnt reset) in order to allow the GPU to passthrough and think its the same card by using multifunction='yes' and changing the bust/ports (at least I think Ive done it right...). Ive also gone and hex edited the VBIOS for my graphics card that I got off techpowerup (included below original and my version to check if I hex edited it right). I opened the VM with the normal VMC boot window thing, booted up fine, got into windows, updated all the drivers EXCEPT the display driver, installed the guest client etc. etc. if its in Spaceinvaders tutorial, Ive done it. Ill post my edited XML below so you guys can proof read it to see if I fucked it up or anything, and then Installed Splashtop Personal so I could use that to view the VM. I confirmed it worked (Ive been remoting through my laptop this whole time) before I changed the display to the Nvidia GPU with hex edited bios, and the Nvidia sound with mobo sound as a + addition underneath, and checked all the usb devices to passthrough for keyboard, mouse, and other usb devices that I didnt recognise but might be part of the GPU so I ticked it anyway. But no matter what I do, I cannot remote into the VM after Ive changed it to the NVIDIA GPU. I will post the XML below. My GPU reads I believe 6.00.0 and sound 6.00.1 (I dont really want to go back to the GUI and wipe my XML changes, it took me 2 hrs to find where I think Im supposed to be there in the XML) <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='1'> <name>Remote Gaming VM</name> <uuid>926f366a-eb38-787d-a18a-a01482a0a70e</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>33030144</memory> <currentMemory unit='KiB'>33030144</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>8</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='8'/> <vcpupin vcpu='2' cpuset='3'/> <vcpupin vcpu='3' cpuset='9'/> <vcpupin vcpu='4' cpuset='4'/> <vcpupin vcpu='5' cpuset='10'/> <vcpupin vcpu='6' cpuset='5'/> <vcpupin vcpu='7' cpuset='11'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-7.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/926f366a-eb38-787d-a18a-a01482a0a70e_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv mode='custom'> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='1' dies='1' cores='4' threads='2'/> <cache mode='passthrough'/> <feature policy='require' name='topoext'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/user/domains/Remote Gaming VM/vdisk1.img' index='3'/> <backingStore/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <alias name='virtio-disk2'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/Windows 10 ISO.iso' index='2'/> <backingStore/> <target dev='hda' bus='sata'/> <readonly/> <boot order='2'/> <alias name='sata0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.229-1.iso' index='1'/> <backingStore/> <target dev='hdb' bus='sata'/> <readonly/> <alias name='sata0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x12'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x13'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0x14'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:5a:5d:0c'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio-net'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-Remote Gaming VM/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'> <alias name='input1'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <audio id='1' type='none'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <rom file='/mnt/user/isos/ASUS RTX 3060Ti Hex Edited VBIOS.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x046d'/> <product id='0x085c'/> <address bus='1' device='5'/> </source> <alias name='hostdev2'/> <address type='usb' bus='0' port='3'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x048d'/> <product id='0x5702'/> <address bus='1' device='3'/> </source> <alias name='hostdev3'/> <address type='usb' bus='0' port='4'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x0d8c'/> <product id='0x0004'/> <address bus='3' device='2'/> </source> <alias name='hostdev4'/> <address type='usb' bus='0' port='5'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x1532'/> <product id='0x007e'/> <address bus='1' device='9'/> </source> <alias name='hostdev5'/> <address type='usb' bus='0' port='6'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x1532'/> <product id='0x0088'/> <address bus='1' device='10'/> </source> <alias name='hostdev6'/> <address type='usb' bus='0' port='2.1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x31e3'/> <product id='0x1220'/> <address bus='1' device='4'/> </source> <alias name='hostdev7'/> <address type='usb' bus='0' port='2.2'/> </hostdev> <hub type='usb'> <alias name='hub0'/> <address type='usb' bus='0' port='2'/> </hub> <memballoon model='none'/> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain> I highlighted in orange the lines I "think" Im supposed to have changed and did what I think was right... but I could be wrong. Does anyone have any other input or things I can do to get this VM to post? Splashtop hasnt recognised anything it can connect to since the swap off VMC to NVIDIA, and I could remote in and login to windows beforehand. After 4 days of trying to figure it out myself and asking friends and watching tutorials and googling the issue, Im at my total wits end... Here I was thinking I was good with computers... smh. I do have IOMMU and SVM modes enabled in bios btw so it is capable of doing it. Thankyou! akashic-records-diagnostics-20230205-0718.zip Asus.RTX3060Ti.8192.210524_1.rom ASUS RTX 3060Ti Hex Edited VBIOS.rom Edited February 5, 20233 yr by headshot_g
February 5, 20233 yr Solution xml is ok, make sure that it wont change the gpu part you highlighted after making modifications. As you can see in your logs: Feb 4 10:44:34 Akashic-Records kernel: pci 0000:06:00.0: BAR 1: assigned to efifb ... ... Feb 4 10:59:41 Akashic-Records kernel: vfio-pci 0000:06:00.0: BAR 1: can't reserve [mem 0xd0000000-0xdfffffff 64bit pref] you need to modify your syslinux config (main > flash > syslinux configuration) and add in the append line video=efifb:off, like in this picture taken from the web: Note that since this gpu is marked as boot gpu the screen on boot will seem frozen, because the gpu will attach to vfio, but unraid will boot, so connect to unraid from another device on the lan. Reboot the server. It seems also that you are using a wrong vbios. From what I found the vbios you attached is for a 3060 Ti, device/vendor id 10de:2489, subsystem 1043:8829. From your lspci your device is 10de:2489, subsystem 1043:884f. It is always better to dump the vbios directly from YOUR gpu, in extreme cases you could download a vbios from techpowerup: this will prevent you from using a wrong vbios, as it seems the case. So, dump the vbios from your gpu and use it (check if hex editing is needed, i.e. if dumped from windows with gpu-z), or if you have no alternatives and it doesn't work with your vbios try the attached one (already hex edited). 241689.rom Edited February 5, 20233 yr by ghost82
February 6, 20233 yr Author Ah so thats where that syslinux solution was meant to go, I read that on the forum somewhere but I had absolutely ZERO clue where to include it before now without that image. So I modified the syslinux line exactly as shown, Im not sure how Im supposed to use gpu-z to pull the bios from the card itself (I dont have a working PC I can slot the GPU into to do so, the only working PC is the server the GPU is currently plugged into). SO I just went ahead and used your correct VBIOS you kindly provided instead. I used the Unraid GUI to link this, so ofc I went back to the XML and re-edited the file to exactly the same as the ^ Orange section above in the original post and left it in XML view. So now with the syslinux line and the GPU with the new bios I rebooted the server. On normal selection boot up I got a strange screen Ive never seen before which Ive included below. Re-tried and booted to GUI mode, which then did as you noted which is look like its frozen, but as Im always remoting in, my screen came up to login to as was mentioned it should on the laptop. On a side note, will this mean that I will have to boot to GUI mode and get the "frozen screen" everytime I want to use this WIn 10 VM with GPU passthrough? AAAAAAAAAAAAND the VM boots and says its started but STILL doesn't do anything. Splashtop Desktop to remote into the VM still acts like the VM isnt actually there/on, which is weird, because I dont have to login to windows for it to work, it works at the sign-in screen for windows. Ive included another diagnostic snapshot incase something else has happened that Im absolutely not qualified enough to tell... akashic-records-diagnostics-20230206-1756 (new).zip
February 6, 20233 yr 1 hour ago, headshot_g said: Re-tried and booted to GUI mode Do not boot the gui mode, but unraid os, as said gpu is isolated from unraid and it cannot be used by the host (unraid), but it will be reserved for the vm. If you look at what you modified in the syslinux config you can see that you modified the "unraid os", not the "unraid os gui mode", so the efifb:off is applied when you boot "unraid os" (command line). Boot unraid os, then connect from the laptop on the lan, then start the vm. Your screenshot shows a kernel panic, system wasn't able to find root partition, this may depend on the unraid usb, try to boot again. Edited February 6, 20233 yr by ghost82
February 6, 20233 yr Author Ok so, having retried booting unraid normally I get the same "kernel panic" as the previous post. Having only happened after I edited that section, Im assuming it has something to do with it. Having luckily not sent this post before I double checked (in gui) I actually didnt type 'append' in front so no wonder it didnt give the right result..... (Im an idiot). I entered the CORRECT syslinux line and now it booted into unraid fine and now I know what you mean by "it looks like its frozen", yeah I see that now, it really does.... Anyway now we're in Unraid, I checked the GPU XML again just to be sure and it stayed the same, no issues, so we should be good. The only issue is again, the same I started with. Splashtop says this VM isnt up, its offline and theres nothing to connect to. The VM doesn't want to display. Its been like 10 minutes so if it were gonna take forever on 1st new boot up It would be up by now. Should I try make a fresh one? now that technically the groundwork has been done? I dunno the syslinux solution was the only other solution I had seen (that I didnt understand until now) for this issue. Is it a VBIOS issue? Im using the one you provided, the one I had hex edited was taken directly off techpowerup off the right page (doubt my box art is wrong ahaha) and hex edited (maybe thats why it said it was wrong? Im not sure im spitballing). I dont have a way to rip its own bios with GPU-Z unless I go to my uncles house and borrow his PC for a bit. Im open to anything cos I sure as heck have nothing. Included a fresh diagnostic copy of the system as it stands rn post edits. akashic-records-diagnostics-20230206-2048 (new 2).zip Edited February 6, 20233 yr by headshot_g
February 6, 20233 yr It seems all is ok now. Can you try the following and report back: 1. Let the syslinux config as it is 2. modify your vm to not pass through the gpu (video and audio), replace it with vnc: check if it boots 3. boot the vm and connect with novnc (directly in unraid) (from your laptop) 4. enable remote desktop inside the vm or install any other vnc server inside windows 5. disable vnc in your vm and passthrough the gpu, check multifunction for gpu is applied 6. start the vm and try to connect to the vm ip from your laptop, so you are connecting directly to the vm 7. if it boots check device manager to see if the gpu is there 8. if it's there, download nvidia drivers and install them Edited February 6, 20233 yr by ghost82
February 6, 20233 yr Author Im at step 4 and I thought you might want to know, THE SECOND I changed back to VNC ( and connected with No VNC no issues) Splashtop that is already installed instantly saw it as a connection thats live and connectable and I could remote into it. This is before windows even logged in past the user screen. I figure that might be a tidbit that might help you narrow down whats happening. Doing the rest now. Edited February 6, 20233 yr by headshot_g
February 6, 20233 yr Not of much help unfortunately, we already know that the vm should work without the passed through gpu..Try to proceed with other steps, but I think you will stop at step 6.. I have a doubt that it could be the vbios..if this is the case you need to find a way to dump it from your gpu..
February 6, 20233 yr Author Ok so I even took the 1st image of my XML in anticipation for this essentially simple "Turn it off turn it back on" trick to not work cos its so simple. But by God it worked!!! It did say it was experiencing slow performance for some reason, and was chugging like ass.... Then I heard the windows usb insert noise and the display changed all by itself (Photo 2) So I went into device manager and THE GPU IS FINALLY THERE. Bruh give me your bitcoin address in a dm I'll buy you a pint. The biggest headache of my week has been solved! Updating my GPU drivers now and Ill mark this as solved. Even though the last post about going back and forth from VNC made it work, I dont think it would have worked without the syslinux part (and maybe your VBIOS) so I chose that to mark as the solution. Edited February 6, 20233 yr by headshot_g
February 6, 20233 yr 5 minutes ago, headshot_g said: Bruh give me your bitcoin address in a dm I'll buy you a pint. ahah no need to send anything!I'm happy you got it working. Just do a good action to another person and maybe the world will be better
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.