November 16, 20214 yr Supermicro X10DRL-i , UNraid 6.10.0-rc1, MSI GTX 1650 . No luck passing through GPU to a vm was working no problem before not sure if I made a mistake or a update. When I start a vm without gpu it works fine but as soon as i try passing though gpu i get blank screen would like to get a working windows VM. Ive just tested a windows baremetal install worked no problem but I also tried a Ubuntu baremetal with no luck. I tried different machine types and dumping the vbios but still no luck any help would be appreciated but try to keep it simple I am just a newbie. ErrorWarningSystem Loading config from /boot/config/vfio-pci.cfg BIND=0000:07:00.0|10de:1f82 0000:07:00.1|10de:10fa 0000:0d:00.0|1106:3483 --- Processing 0000:07:00.0 10de:1f82 Error: Device 0000:07:00.0 does not exist, unable to bind device --- Processing 0000:07:00.1 10de:10fa Error: Device 0000:07:00.1 does not exist, unable to bind device --- Processing 0000:0d:00.0 1106:3483 Vendor:Device 1106:3483 found at 0000:0d:00.0 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:0d:00.0/iommu_group/devices/0000:0d:00.0 Binding... Unbound 0000:0d:00.0 from xhci_hcd chown: cannot access '/dev/vfio/68': No such file or directory Error: unable to adjust group ownership of /dev/vfio/68 --- vfio-pci binding complete Devices listed in /sys/bus/pci/drivers/vfio-pci: lrwxrwxrwx 1 root root 0 Nov 16 17:05 0000:0d:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.4/0000:0d:00.0 ls -l /dev/vfio/ ls: cannot access '/dev/vfio/': No such file or directory tower-diagnostics-20211116-1047.zip
November 17, 20214 yr 12 hours ago, Fastfred said: Processing 0000:07:00.0 10de:1f82 There's no device seen by unraid at bus 7 slot 0 function 0/1. You gpu is at bus 6, slot 0, function 0/1. I don't know which vm you are using, if "Windows 10" or "Windows", however: For the "Windows 10" vm change the xml from this: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/vbios/GTX 1650.rom'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x08' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> to this: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/vbios/GTX 1650.rom'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/> </hostdev> For the "Windows" vm change from this: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/vbios/GTX 1650.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> To this: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/vbios/GTX 1650.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </hostdev> In unraid terminal: nano /boot/config/vfio-pci.cfg Change this line, from this: BIND=0000:07:00.0|10de:1f82 0000:07:00.1|10de:10fa 0000:0d:00.0|1106:3483 to this: BIND=0000:06:00.0|10de:1f82 0000:06:00.1|10de:10fa 0000:0d:00.0|1106:3483 CTRL+o to save in nano CTRL+x to exit nano Reboot unraid and try. Maybe you changed the slot of some device (not necessary the gpu), or you added some device, so the addresses changed, or you did a bios upgrade? Edited November 17, 20214 yr by ghost82
November 17, 20214 yr Author I applied the changes you recommended still had no luck created a new vm (windows 10) still no luck. Thanks for your help by the way. tower-diagnostics-20211117-1022.zip
November 17, 20214 yr did you reboot? because the vfio config is correct, but the log it's not... Moreover in the vm in the diagnostics you didn't apply the multifunction on gpu passthrough that I indicated above. Edited November 17, 20214 yr by ghost82
November 18, 20214 yr Author Looks like I am still doing something wrong . tower-diagnostics-20211118-1033.zip
November 18, 20214 yr Author Ok finally got it to work double checked everything you told me to change , I think what did it was when i completely shut down sever and disconnected power for a min worked after that . Thanks for your help...
April 10, 20224 yr Author Just added a Quadro M2000 GPU When I had windows 10 vm working I didnt have any issues with the new gpu but when I shut down the vm I couldn't restart. Ive tried different PCIe ACS override: settings but no luck... any help would be helpful. -no-shutdown \ -boot strict=on \ -device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.0","addr":"0x7"}' \ -device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.0","addr":"0x3"}' \ -blockdev '{"driver":"file","filename":"/mnt/vm-dockers/domains/Windows 10 d/vdisk1.img","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \ -device '{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x4","drive":"libvirt-2-format","id":"virtio-disk2","bootindex":1,"write-cache":"on"}' \ -blockdev '{"driver":"file","filename":"/mnt/user/isos/virtio-win-0.1.215-2.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \ -device '{"driver":"ide-cd","bus":"ide.0","unit":1,"drive":"libvirt-1-format","id":"ide0-0-1"}' \ -netdev tap,fd=38,id=hostnet0 \ -device '{"driver":"virtio-net","netdev":"hostnet0","id":"net0","mac":"52:54:00:15:dc:05","bus":"pci.0","addr":"0x2"}' \ -chardev pty,id=charserial0 \ -device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0"}' \ -chardev socket,id=charchannel0,fd=39,server=on,wait=off \ -device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \ -audiodev '{"id":"audio1","driver":"none"}' \ -device '{"driver":"vfio-pci","host":"0000:07:00.0","id":"hostdev0","bus":"pci.0","multifunction":true,"addr":"0x5","romfile":"/mnt/user/isos/vbios/M2000.rom"}' \ -device '{"driver":"vfio-pci","host":"0000:07:00.1","id":"hostdev1","bus":"pci.0","addr":"0x5.0x1"}' \ -device '{"driver":"vfio-pci","host":"0000:05:00.0","id":"hostdev2","bus":"pci.0","addr":"0x6"}' \ -device '{"driver":"vfio-pci","host":"0000:0e:00.0","id":"hostdev3","bus":"pci.0","addr":"0x8"}' \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on char device redirected to /dev/pts/1 (label charserial0) 2022-04-10T20:03:56.345874Z qemu-system-x86_64: terminating on signal 15 from pid 5393 (/usr/sbin/libvirtd) 2022-04-10 20:03:57.946+0000: shutting down, reason=destroyed 2022-04-10 20:12:39.805+0000: starting up libvirt version: 7.10.0, qemu version: 6.2.0, kernel: 5.15.30-Unraid, hostname: Tower LC_ALL=C \ PATH=/bin:/sbin:/usr/bin:/usr/sbin \ HOME='/var/lib/libvirt/qemu/domain-3-Windows 10' \ XDG_DATA_HOME='/var/lib/libvirt/qemu/domain-3-Windows 10/.local/share' \ XDG_CACHE_HOME='/var/lib/libvirt/qemu/domain-3-Windows 10/.cache' \ XDG_CONFIG_HOME='/var/lib/libvirt/qemu/domain-3-Windows 10/.config' \ /usr/local/sbin/qemu \ -name 'guest=Windows 10,debug-threads=on' \ -S \ -object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-3-Windows 10/master-key.aes"}' \ -blockdev '{"driver":"file","filename":"/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \ -blockdev '{"driver":"file","filename":"/etc/libvirt/qemu/nvram/05247dcb-91ab-5e1e-a8f9-06991d24e1b9_VARS-pure-efi.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \ -machine pc-i440fx-6.2,usb=off,dump-guest-core=off,mem-merge=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \ -accel kvm \ -cpu host,migratable=on,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vendor-id=none,host-cache-info=on,l3-cache=off \ -m 8704 \ -object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":9126805504}' \ -overcommit mem-lock=off \ -smp 12,sockets=1,dies=1,cores=6,threads=2 \ -uuid 05247dcb-91ab-5e1e-a8f9-06991d24e1b9 \ -display none \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=36,server=on,wait=off \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=localtime \ -no-hpet \ -no-shutdown \ -boot strict=on \ -device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.0","addr":"0x7"}' \ -device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.0","addr":"0x3"}' \ -blockdev '{"driver":"file","filename":"/mnt/vm-dockers/domains/Windows 10 d/vdisk1.img","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \ -device '{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x4","drive":"libvirt-2-format","id":"virtio-disk2","bootindex":1,"write-cache":"on"}' \ -blockdev '{"driver":"file","filename":"/mnt/user/isos/virtio-win-0.1.215-2.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \ -device '{"driver":"ide-cd","bus":"ide.0","unit":1,"drive":"libvirt-1-format","id":"ide0-0-1"}' \ -netdev tap,fd=38,id=hostnet0 \ -device '{"driver":"virtio-net","netdev":"hostnet0","id":"net0","mac":"52:54:00:15:dc:05","bus":"pci.0","addr":"0x2"}' \ -chardev pty,id=charserial0 \ -device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0"}' \ -chardev socket,id=charchannel0,fd=39,server=on,wait=off \ -device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \ -audiodev '{"id":"audio1","driver":"none"}' \ -device '{"driver":"vfio-pci","host":"0000:07:00.0","id":"hostdev0","bus":"pci.0","multifunction":true,"addr":"0x5","romfile":"/mnt/user/isos/vbios/M2000.rom"}' \ -device '{"driver":"vfio-pci","host":"0000:07:00.1","id":"hostdev1","bus":"pci.0","addr":"0x5.0x1"}' \ -device '{"driver":"vfio-pci","host":"0000:05:00.0","id":"hostdev2","bus":"pci.0","addr":"0x6"}' \ -device '{"driver":"vfio-pci","host":"0000:0e:00.0","id":"hostdev3","bus":"pci.0","addr":"0x8"}' \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on char device redirected to /dev/pts/9 (label charserial0)
April 11, 20224 yr Author After starting Windows VM and removing the gpu using VNC , when starting up I get Device 0000:06:00.0 not found: could not access /sys/bus/pci/devices/0000:06:00.0/config: No such file or directory sometimes when trying different settings , PCIe ACS override: , UEFI/Legacy ,Machine type.. I get a blue Screen in vnc for the windows vm.tower-syslog-20220411-1743.zip
April 11, 20224 yr Can you post your entire diagnostics. Far more useful than a simple syslog and your previous diagnostics may now be out of date
April 12, 20224 yr When you add new GPUs to your system, the PCI device addresses may change. You need to edit each VM with pass through devices assigned, reselect those devices, and save each VM. This is definitely something we need to address in the future because it isn't obvious to the user on what to do in these scenarios so that's on us for not making it easier to understand.
April 12, 20224 yr Author I have been adjusting the VM for the gpu passthrough. I've watched the videos to make sure I've done that. I'll post the diagnostic later when I get home.
April 12, 20224 yr Author tower-diagnostics-20220412-1009.zip <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/vbios/M2000.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
August 24, 20223 yr On 4/11/2022 at 9:44 PM, jonp said: When you add new GPUs to your system, the PCI device addresses may change. You need to edit each VM with pass through devices assigned, reselect those devices, and save each VM. Hope you’re all well. Do I need to/should I pass through/BIND the GPU&audio? I current am not and VM is running fine. I don’t know if it usually should be Binded. Thank you
August 31, 20223 yr On 8/24/2022 at 12:20 AM, xlucero1 said: Hope you’re all well. Do I need to/should I pass through/BIND the GPU&audio? I current am not and VM is running fine. I don’t know if it usually should be Binded. Thank you No, if things are working fine, it is not vital to pass through the audio and the GPU together. The audio device is solely if you are using HDMI/Displayport for transmitting audio, but most users have a separate audio device (PCI or USB) that they utilize for that purpose. In addition, passing through audio via HDMI/Displayport can cause weird issues if you don't apply the MSI interrupts fix.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.