Help passthrough GPU windows vm


Fastfred

Recommended Posts

 

Supermicro X10DRL-i , UNraid 6.10.0-rc1, MSI GTX 1650 .

 

No luck passing through GPU to a vm was working no problem before not sure if I made a mistake or a update. When I start a vm without gpu it works fine but as soon as i try passing though gpu i get blank screen would like to get a working windows VM. Ive just tested a windows baremetal install worked no problem but I also tried a Ubuntu baremetal with no luck. I tried different machine types and dumping the vbios but still no luck any help would be appreciated but try to keep it simple I am just a newbie.

 

 

ErrorWarningSystem


Loading config from /boot/config/vfio-pci.cfg
BIND=0000:07:00.0|10de:1f82 0000:07:00.1|10de:10fa 0000:0d:00.0|1106:3483
---
Processing 0000:07:00.0 10de:1f82
Error: Device 0000:07:00.0 does not exist, unable to bind device
---
Processing 0000:07:00.1 10de:10fa
Error: Device 0000:07:00.1 does not exist, unable to bind device
---
Processing 0000:0d:00.0 1106:3483
Vendor:Device 1106:3483 found at 0000:0d:00.0

IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:0d:00.0/iommu_group/devices/0000:0d:00.0

Binding...
Unbound 0000:0d:00.0 from xhci_hcd
chown: cannot access '/dev/vfio/68': No such file or directory
Error: unable to adjust group ownership of /dev/vfio/68
---
vfio-pci binding complete

Devices listed in /sys/bus/pci/drivers/vfio-pci:
lrwxrwxrwx 1 root root 0 Nov 16 17:05 0000:0d:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.4/0000:0d:00.0

ls -l /dev/vfio/
ls: cannot access '/dev/vfio/': No such file or directory

tower-diagnostics-20211116-1047.zip

Link to comment
12 hours ago, Fastfred said:

Processing 0000:07:00.0 10de:1f82

There's no device seen by unraid at bus 7 slot 0 function 0/1.

You gpu is at bus 6, slot 0, function 0/1.

 

I don't know which vm you are using, if "Windows 10" or "Windows", however:

For the "Windows 10" vm change the xml from this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/GTX 1650.rom'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>

 

to this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/GTX 1650.rom'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
    </hostdev>

 

 

For the "Windows" vm change from this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/GTX 1650.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>

 

To this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/GTX 1650.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </hostdev>

 

In unraid terminal:

nano /boot/config/vfio-pci.cfg

 

Change this line, from this:

BIND=0000:07:00.0|10de:1f82 0000:07:00.1|10de:10fa 0000:0d:00.0|1106:3483

 

to this:

BIND=0000:06:00.0|10de:1f82 0000:06:00.1|10de:10fa 0000:0d:00.0|1106:3483

 

CTRL+o to save in nano

CTRL+x to exit nano

 

Reboot unraid and try.

 

Maybe you changed the slot of some device (not necessary the gpu), or you added some device, so the addresses changed, or you did a bios upgrade?

Edited by ghost82
Link to comment
  • 4 months later...

Just added a Quadro M2000 GPU When I had windows 10 vm working I didnt have any issues with the new gpu but when I shut down the vm I couldn't restart. Ive tried different PCIe ACS override: settings but no luck...     any help would be helpful.

 

 

 

 

 

-no-shutdown \
-boot strict=on \
-device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.0","addr":"0x7"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.0","addr":"0x3"}' \
-blockdev '{"driver":"file","filename":"/mnt/vm-dockers/domains/Windows 10 d/vdisk1.img","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x4","drive":"libvirt-2-format","id":"virtio-disk2","bootindex":1,"write-cache":"on"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/virtio-win-0.1.215-2.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"ide-cd","bus":"ide.0","unit":1,"drive":"libvirt-1-format","id":"ide0-0-1"}' \
-netdev tap,fd=38,id=hostnet0 \
-device '{"driver":"virtio-net","netdev":"hostnet0","id":"net0","mac":"52:54:00:15:dc:05","bus":"pci.0","addr":"0x2"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0"}' \
-chardev socket,id=charchannel0,fd=39,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-device '{"driver":"vfio-pci","host":"0000:07:00.0","id":"hostdev0","bus":"pci.0","multifunction":true,"addr":"0x5","romfile":"/mnt/user/isos/vbios/M2000.rom"}' \
-device '{"driver":"vfio-pci","host":"0000:07:00.1","id":"hostdev1","bus":"pci.0","addr":"0x5.0x1"}' \
-device '{"driver":"vfio-pci","host":"0000:05:00.0","id":"hostdev2","bus":"pci.0","addr":"0x6"}' \
-device '{"driver":"vfio-pci","host":"0000:0e:00.0","id":"hostdev3","bus":"pci.0","addr":"0x8"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/1 (label charserial0)
2022-04-10T20:03:56.345874Z qemu-system-x86_64: terminating on signal 15 from pid 5393 (/usr/sbin/libvirtd)
2022-04-10 20:03:57.946+0000: shutting down, reason=destroyed
2022-04-10 20:12:39.805+0000: starting up libvirt version: 7.10.0, qemu version: 6.2.0, kernel: 5.15.30-Unraid, hostname: Tower
LC_ALL=C \
PATH=/bin:/sbin:/usr/bin:/usr/sbin \
HOME='/var/lib/libvirt/qemu/domain-3-Windows 10' \
XDG_DATA_HOME='/var/lib/libvirt/qemu/domain-3-Windows 10/.local/share' \
XDG_CACHE_HOME='/var/lib/libvirt/qemu/domain-3-Windows 10/.cache' \
XDG_CONFIG_HOME='/var/lib/libvirt/qemu/domain-3-Windows 10/.config' \
/usr/local/sbin/qemu \
-name 'guest=Windows 10,debug-threads=on' \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-3-Windows 10/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/etc/libvirt/qemu/nvram/05247dcb-91ab-5e1e-a8f9-06991d24e1b9_VARS-pure-efi.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-i440fx-6.2,usb=off,dump-guest-core=off,mem-merge=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \
-accel kvm \
-cpu host,migratable=on,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vendor-id=none,host-cache-info=on,l3-cache=off \
-m 8704 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":9126805504}' \
-overcommit mem-lock=off \
-smp 12,sockets=1,dies=1,cores=6,threads=2 \
-uuid 05247dcb-91ab-5e1e-a8f9-06991d24e1b9 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=36,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.0","addr":"0x7"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.0","addr":"0x3"}' \
-blockdev '{"driver":"file","filename":"/mnt/vm-dockers/domains/Windows 10 d/vdisk1.img","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x4","drive":"libvirt-2-format","id":"virtio-disk2","bootindex":1,"write-cache":"on"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/virtio-win-0.1.215-2.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"ide-cd","bus":"ide.0","unit":1,"drive":"libvirt-1-format","id":"ide0-0-1"}' \
-netdev tap,fd=38,id=hostnet0 \
-device '{"driver":"virtio-net","netdev":"hostnet0","id":"net0","mac":"52:54:00:15:dc:05","bus":"pci.0","addr":"0x2"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0"}' \
-chardev socket,id=charchannel0,fd=39,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-device '{"driver":"vfio-pci","host":"0000:07:00.0","id":"hostdev0","bus":"pci.0","multifunction":true,"addr":"0x5","romfile":"/mnt/user/isos/vbios/M2000.rom"}' \
-device '{"driver":"vfio-pci","host":"0000:07:00.1","id":"hostdev1","bus":"pci.0","addr":"0x5.0x1"}' \
-device '{"driver":"vfio-pci","host":"0000:05:00.0","id":"hostdev2","bus":"pci.0","addr":"0x6"}' \
-device '{"driver":"vfio-pci","host":"0000:0e:00.0","id":"hostdev3","bus":"pci.0","addr":"0x8"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/9 (label charserial0)
 

 

Link to comment

When you add new GPUs to your system, the PCI device addresses may change.  You need to edit each VM with pass through devices assigned, reselect those devices, and save each VM.

 

This is definitely something we need to address in the future because it isn't obvious to the user on what to do in these scenarios so that's on us for not making it easier to understand.

Link to comment
  • 4 months later...
On 4/11/2022 at 9:44 PM, jonp said:

When you add new GPUs to your system, the PCI device addresses may change.  You need to edit each VM with pass through devices assigned, reselect those devices, and save each VM.

Hope you’re all well. Do I need to/should I pass through/BIND the GPU&audio? I current am not and VM is running fine. I don’t know if it usually should be Binded. Thank you 

Link to comment
On 8/24/2022 at 12:20 AM, xlucero1 said:

Hope you’re all well. Do I need to/should I pass through/BIND the GPU&audio? I current am not and VM is running fine. I don’t know if it usually should be Binded. Thank you 

 

No, if things are working fine, it is not vital to pass through the audio and the GPU together.  The audio device is solely if you are using HDMI/Displayport for transmitting audio, but most users have a separate audio device (PCI or USB) that they utilize for that purpose.  In addition, passing through audio via HDMI/Displayport can cause weird issues if you don't apply the MSI interrupts fix.

  • Thanks 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.