Fastfred Posted November 16, 2021 Share Posted November 16, 2021 Supermicro X10DRL-i , UNraid 6.10.0-rc1, MSI GTX 1650 . No luck passing through GPU to a vm was working no problem before not sure if I made a mistake or a update. When I start a vm without gpu it works fine but as soon as i try passing though gpu i get blank screen would like to get a working windows VM. Ive just tested a windows baremetal install worked no problem but I also tried a Ubuntu baremetal with no luck. I tried different machine types and dumping the vbios but still no luck any help would be appreciated but try to keep it simple I am just a newbie. ErrorWarningSystem Loading config from /boot/config/vfio-pci.cfg BIND=0000:07:00.0|10de:1f82 0000:07:00.1|10de:10fa 0000:0d:00.0|1106:3483 --- Processing 0000:07:00.0 10de:1f82 Error: Device 0000:07:00.0 does not exist, unable to bind device --- Processing 0000:07:00.1 10de:10fa Error: Device 0000:07:00.1 does not exist, unable to bind device --- Processing 0000:0d:00.0 1106:3483 Vendor:Device 1106:3483 found at 0000:0d:00.0 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:0d:00.0/iommu_group/devices/0000:0d:00.0 Binding... Unbound 0000:0d:00.0 from xhci_hcd chown: cannot access '/dev/vfio/68': No such file or directory Error: unable to adjust group ownership of /dev/vfio/68 --- vfio-pci binding complete Devices listed in /sys/bus/pci/drivers/vfio-pci: lrwxrwxrwx 1 root root 0 Nov 16 17:05 0000:0d:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.4/0000:0d:00.0 ls -l /dev/vfio/ ls: cannot access '/dev/vfio/': No such file or directory tower-diagnostics-20211116-1047.zip Quote Link to comment
ghost82 Posted November 17, 2021 Share Posted November 17, 2021 (edited) 12 hours ago, Fastfred said: Processing 0000:07:00.0 10de:1f82 There's no device seen by unraid at bus 7 slot 0 function 0/1. You gpu is at bus 6, slot 0, function 0/1. I don't know which vm you are using, if "Windows 10" or "Windows", however: For the "Windows 10" vm change the xml from this: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/vbios/GTX 1650.rom'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x08' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> to this: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/vbios/GTX 1650.rom'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/> </hostdev> For the "Windows" vm change from this: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/vbios/GTX 1650.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> To this: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/vbios/GTX 1650.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </hostdev> In unraid terminal: nano /boot/config/vfio-pci.cfg Change this line, from this: BIND=0000:07:00.0|10de:1f82 0000:07:00.1|10de:10fa 0000:0d:00.0|1106:3483 to this: BIND=0000:06:00.0|10de:1f82 0000:06:00.1|10de:10fa 0000:0d:00.0|1106:3483 CTRL+o to save in nano CTRL+x to exit nano Reboot unraid and try. Maybe you changed the slot of some device (not necessary the gpu), or you added some device, so the addresses changed, or you did a bios upgrade? Edited November 17, 2021 by ghost82 Quote Link to comment
Fastfred Posted November 17, 2021 Author Share Posted November 17, 2021 I applied the changes you recommended still had no luck created a new vm (windows 10) still no luck. Thanks for your help by the way. tower-diagnostics-20211117-1022.zip Quote Link to comment
ghost82 Posted November 17, 2021 Share Posted November 17, 2021 (edited) did you reboot? because the vfio config is correct, but the log it's not... Moreover in the vm in the diagnostics you didn't apply the multifunction on gpu passthrough that I indicated above. Edited November 17, 2021 by ghost82 Quote Link to comment
Fastfred Posted November 17, 2021 Author Share Posted November 17, 2021 Oops ok will try when I get back , thanks Quote Link to comment
Fastfred Posted November 18, 2021 Author Share Posted November 18, 2021 Looks like I am still doing something wrong . tower-diagnostics-20211118-1033.zip Quote Link to comment
Fastfred Posted November 18, 2021 Author Share Posted November 18, 2021 Ok finally got it to work double checked everything you told me to change , I think what did it was when i completely shut down sever and disconnected power for a min worked after that . Thanks for your help... 1 Quote Link to comment
Fastfred Posted April 10, 2022 Author Share Posted April 10, 2022 Just added a Quadro M2000 GPU When I had windows 10 vm working I didnt have any issues with the new gpu but when I shut down the vm I couldn't restart. Ive tried different PCIe ACS override: settings but no luck... any help would be helpful. -no-shutdown \ -boot strict=on \ -device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.0","addr":"0x7"}' \ -device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.0","addr":"0x3"}' \ -blockdev '{"driver":"file","filename":"/mnt/vm-dockers/domains/Windows 10 d/vdisk1.img","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \ -device '{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x4","drive":"libvirt-2-format","id":"virtio-disk2","bootindex":1,"write-cache":"on"}' \ -blockdev '{"driver":"file","filename":"/mnt/user/isos/virtio-win-0.1.215-2.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \ -device '{"driver":"ide-cd","bus":"ide.0","unit":1,"drive":"libvirt-1-format","id":"ide0-0-1"}' \ -netdev tap,fd=38,id=hostnet0 \ -device '{"driver":"virtio-net","netdev":"hostnet0","id":"net0","mac":"52:54:00:15:dc:05","bus":"pci.0","addr":"0x2"}' \ -chardev pty,id=charserial0 \ -device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0"}' \ -chardev socket,id=charchannel0,fd=39,server=on,wait=off \ -device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \ -audiodev '{"id":"audio1","driver":"none"}' \ -device '{"driver":"vfio-pci","host":"0000:07:00.0","id":"hostdev0","bus":"pci.0","multifunction":true,"addr":"0x5","romfile":"/mnt/user/isos/vbios/M2000.rom"}' \ -device '{"driver":"vfio-pci","host":"0000:07:00.1","id":"hostdev1","bus":"pci.0","addr":"0x5.0x1"}' \ -device '{"driver":"vfio-pci","host":"0000:05:00.0","id":"hostdev2","bus":"pci.0","addr":"0x6"}' \ -device '{"driver":"vfio-pci","host":"0000:0e:00.0","id":"hostdev3","bus":"pci.0","addr":"0x8"}' \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on char device redirected to /dev/pts/1 (label charserial0) 2022-04-10T20:03:56.345874Z qemu-system-x86_64: terminating on signal 15 from pid 5393 (/usr/sbin/libvirtd) 2022-04-10 20:03:57.946+0000: shutting down, reason=destroyed 2022-04-10 20:12:39.805+0000: starting up libvirt version: 7.10.0, qemu version: 6.2.0, kernel: 5.15.30-Unraid, hostname: Tower LC_ALL=C \ PATH=/bin:/sbin:/usr/bin:/usr/sbin \ HOME='/var/lib/libvirt/qemu/domain-3-Windows 10' \ XDG_DATA_HOME='/var/lib/libvirt/qemu/domain-3-Windows 10/.local/share' \ XDG_CACHE_HOME='/var/lib/libvirt/qemu/domain-3-Windows 10/.cache' \ XDG_CONFIG_HOME='/var/lib/libvirt/qemu/domain-3-Windows 10/.config' \ /usr/local/sbin/qemu \ -name 'guest=Windows 10,debug-threads=on' \ -S \ -object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-3-Windows 10/master-key.aes"}' \ -blockdev '{"driver":"file","filename":"/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \ -blockdev '{"driver":"file","filename":"/etc/libvirt/qemu/nvram/05247dcb-91ab-5e1e-a8f9-06991d24e1b9_VARS-pure-efi.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \ -machine pc-i440fx-6.2,usb=off,dump-guest-core=off,mem-merge=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \ -accel kvm \ -cpu host,migratable=on,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vendor-id=none,host-cache-info=on,l3-cache=off \ -m 8704 \ -object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":9126805504}' \ -overcommit mem-lock=off \ -smp 12,sockets=1,dies=1,cores=6,threads=2 \ -uuid 05247dcb-91ab-5e1e-a8f9-06991d24e1b9 \ -display none \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=36,server=on,wait=off \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=localtime \ -no-hpet \ -no-shutdown \ -boot strict=on \ -device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.0","addr":"0x7"}' \ -device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.0","addr":"0x3"}' \ -blockdev '{"driver":"file","filename":"/mnt/vm-dockers/domains/Windows 10 d/vdisk1.img","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \ -device '{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x4","drive":"libvirt-2-format","id":"virtio-disk2","bootindex":1,"write-cache":"on"}' \ -blockdev '{"driver":"file","filename":"/mnt/user/isos/virtio-win-0.1.215-2.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \ -device '{"driver":"ide-cd","bus":"ide.0","unit":1,"drive":"libvirt-1-format","id":"ide0-0-1"}' \ -netdev tap,fd=38,id=hostnet0 \ -device '{"driver":"virtio-net","netdev":"hostnet0","id":"net0","mac":"52:54:00:15:dc:05","bus":"pci.0","addr":"0x2"}' \ -chardev pty,id=charserial0 \ -device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0"}' \ -chardev socket,id=charchannel0,fd=39,server=on,wait=off \ -device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \ -audiodev '{"id":"audio1","driver":"none"}' \ -device '{"driver":"vfio-pci","host":"0000:07:00.0","id":"hostdev0","bus":"pci.0","multifunction":true,"addr":"0x5","romfile":"/mnt/user/isos/vbios/M2000.rom"}' \ -device '{"driver":"vfio-pci","host":"0000:07:00.1","id":"hostdev1","bus":"pci.0","addr":"0x5.0x1"}' \ -device '{"driver":"vfio-pci","host":"0000:05:00.0","id":"hostdev2","bus":"pci.0","addr":"0x6"}' \ -device '{"driver":"vfio-pci","host":"0000:0e:00.0","id":"hostdev3","bus":"pci.0","addr":"0x8"}' \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on char device redirected to /dev/pts/9 (label charserial0) Quote Link to comment
Fastfred Posted April 11, 2022 Author Share Posted April 11, 2022 After starting Windows VM and removing the gpu using VNC , when starting up I get Device 0000:06:00.0 not found: could not access /sys/bus/pci/devices/0000:06:00.0/config: No such file or directory sometimes when trying different settings , PCIe ACS override: , UEFI/Legacy ,Machine type.. I get a blue Screen in vnc for the windows vm.tower-syslog-20220411-1743.zip Quote Link to comment
Squid Posted April 11, 2022 Share Posted April 11, 2022 Can you post your entire diagnostics. Far more useful than a simple syslog and your previous diagnostics may now be out of date Quote Link to comment
jonp Posted April 12, 2022 Share Posted April 12, 2022 When you add new GPUs to your system, the PCI device addresses may change. You need to edit each VM with pass through devices assigned, reselect those devices, and save each VM. This is definitely something we need to address in the future because it isn't obvious to the user on what to do in these scenarios so that's on us for not making it easier to understand. Quote Link to comment
Fastfred Posted April 12, 2022 Author Share Posted April 12, 2022 I have been adjusting the VM for the gpu passthrough. I've watched the videos to make sure I've done that. I'll post the diagnostic later when I get home. Quote Link to comment
Fastfred Posted April 12, 2022 Author Share Posted April 12, 2022 tower-diagnostics-20220412-1009.zip <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/vbios/M2000.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/> Quote Link to comment
xlucero1 Posted August 24, 2022 Share Posted August 24, 2022 On 4/11/2022 at 9:44 PM, jonp said: When you add new GPUs to your system, the PCI device addresses may change. You need to edit each VM with pass through devices assigned, reselect those devices, and save each VM. Hope you’re all well. Do I need to/should I pass through/BIND the GPU&audio? I current am not and VM is running fine. I don’t know if it usually should be Binded. Thank you Quote Link to comment
jonp Posted August 31, 2022 Share Posted August 31, 2022 On 8/24/2022 at 12:20 AM, xlucero1 said: Hope you’re all well. Do I need to/should I pass through/BIND the GPU&audio? I current am not and VM is running fine. I don’t know if it usually should be Binded. Thank you No, if things are working fine, it is not vital to pass through the audio and the GPU together. The audio device is solely if you are using HDMI/Displayport for transmitting audio, but most users have a separate audio device (PCI or USB) that they utilize for that purpose. In addition, passing through audio via HDMI/Displayport can cause weird issues if you don't apply the MSI interrupts fix. 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.