[SOLVED] 3090 passthrough woes


Recommended Posts

Hi, 

 

I've tried all day to get my 3090 playing nicely with UNRAID. So far I feel like I'm on the home stretch, but just need a little more help. 

 

When I try and run my Windows VM I get the execution error at the bottom of this post. 

 

For reference, my XML is;

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 10</name>
  <uuid>XXXRedactedXXX</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>33030144</memory>
  <currentMemory unit='KiB'>33030144</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>10</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='5'/>
    <vcpupin vcpu='1' cpuset='15'/>
    <vcpupin vcpu='2' cpuset='6'/>
    <vcpupin vcpu='3' cpuset='16'/>
    <vcpupin vcpu='4' cpuset='7'/>
    <vcpupin vcpu='5' cpuset='17'/>
    <vcpupin vcpu='6' cpuset='8'/>
    <vcpupin vcpu='7' cpuset='18'/>
    <vcpupin vcpu='8' cpuset='9'/>
    <vcpupin vcpu='9' cpuset='19'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-5.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/d453c6c4-7f4b-0b7c-2bf4-a0569f800301_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='5' threads='2'/>
    <cache mode='passthrough'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Win10_20H2_v2_English_x64.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.190-1.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:10:74:ca'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/rtx3090.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

 

 

20210118_185459.jpg

 

 

Lastly, here is my Syslinux Configuration;

 

20210118_185905.jpg

 

Any help would be greatly appreciated.

 

***Edit***

I have tried Q35-5.1 however I get the following error instead;

 

20210118_190144.jpg

Edited by LoneTraveler
Link to comment

machine='pc-i440fx-5.1 /// switch to Q35 by making a whole new VM with the same settings and selecting Q35 prior to ever starting it; You get that PCI controller error when there is still PC-i440Fx setup in the XML.

 

I'm not sure what motherboard you have but did you enable IMMOU in your bios?

 

Something in your XML is off(missing) can you take a screenshot of your VM page 

 

 

Edited by runamuk
Link to comment
10 hours ago, runamuk said:

machine='pc-i440fx-5.1 /// switch to Q35 by making a whole new VM with the same settings and selecting Q35 prior to ever starting it; You get that PCI controller error when there is still PC-i440Fx setup in the XML.

 

I'm not sure what motherboard you have but did you enable IMMOU in your bios?

 

Something in your XML is off(missing) can you take a screenshot of your VM page 

 

 

 

Hi, 

 

Thank you for taking the time to reply. Below are two screenshots of my VM Manager page and System Information;

 

 

20210119_094104.jpg

 

20210119_094136.jpg

 

I have an MSI Z490 Godlike motherboard. I have scoured the bios and enabled everything I can find relating to virtualisation. 

 

***EDIT***

I have tried all four options under; "PCIe ACS override". 

Edited by LoneTraveler
Link to comment
2 hours ago, LoneTraveler said:

find relating to virtualisation. 

Hi,

 

Have you bound all parts of the card, for example my rtx2060 has a usb controller even thou there is not a physical port.

 

This is on 6.8.3 with vfio plugin.

 

image.thumb.png.398c812b60e8bd669835736f3c459ef5.png

 

Can you show the output from the system devices in tools? Looks like your are on 6.9.x should look similar to the below. I dont have a GPU on my test system but this is the bmc graphics card.

 

image.thumb.png.f7967b302d81fa64c3804d057934ea5d.png

 

 

Link to comment
36 minutes ago, SimonF said:

Hi,

 

Have you bound all parts of the card, for example my rtx2060 has a usb controller even thou there is not a physical port.

 

This is on 6.8.3 with vfio plugin.

 

image.thumb.png.398c812b60e8bd669835736f3c459ef5.png

 

Can you show the output from the system devices in tools? Looks like your are on 6.9.x should look similar to the below. I dont have a GPU on my test system but this is the bmc graphics card.

 

image.thumb.png.f7967b302d81fa64c3804d057934ea5d.png

 

 

 

Hi, 

 

Thank you for replying. 

 

I could only see two bindings for the 3090, 15 & 16. I'm not sure if the 760 card (17 & 18) are causing some sort of conflict. 

 

 

20210119_121345.jpg

20210119_121405.jpg

 

Edited by LoneTraveler
Link to comment

@LoneTraveler

I cannot see anything obvious.

 

Are you able to see in the bios which device is being used as the primary display?

 

I don't use a rom for my rtx2060 or gt720 and both are connected to two windows VMs, but again I do have graphics from onboard BMC.

 

Guessing you have already tried without a rom, can you passthru gt760 does that work?

Link to comment
On 3/13/2017 at 3:27 PM, 1812 said:

It's is probably complaining because you're trying to only pass through 1 device from the group when allowing unsafe interrupts (which i won't allow,) and conversely not allowing unsafe interrupts when only passing the single devices which isn't working because you are not using the correct device  id's from your specific cards.

 

Try this in your syslinux.cfg: 

 

append vfio-pci.ids=XXXX:XXXX,YYYY:YYYY vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot

 

XXXX:XXXX is your video card id

YYYY:YYYY is your sound card id

 

you can obtain your video/sound card id's by going to tools>system devices, and look under pci device for 09:00.0 and 09.00.1 

 

the device id will be at the end of he description in the brackets. replace the X's and Y's above with those numbers, reboot, and try again.

 

If it doesn't work, then post your server diagnostics and xml for the vm.

 

After a bit more searching, the above worked for me. I edited my syslinux.cfg file and added the following line;

 

vfio_iommu_type1.allow_unsafe_interrupts=1

 

 

20210119_135727.jpg

 

Thank you @SimonF @1812 @runamuk @Maor for your assistance. 

Edited by LoneTraveler
  • Like 2
Link to comment
  • ytddewqf changed the title to [SOLVED] 3090 passthrough woes
7 minutes ago, LoneTraveler said:

Thank you

No worries, you may not need the ACS overide, unless you need that for other reasons.

 

Glad you got it working, I didnt know about the unsafe_ints as don't need it on my system but will remember in case I need in future, thanks for posting your find/resolution.

  • Thanks 1
Link to comment

I thought I hade spoke too soon.

 

Upon restart of the server, I received a; "can't reserve error". The solution however was to run the following;

 

echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

 

This needs to be run upon each restart of UNRAID and so a quick script sorts that out.

Link to comment

Hey @LoneTraveler I just upgraded my passthrough 1080Ti to a 3090 this past weekend.

I used the method by SpaceInvaderOne but I made a guide here:

 

I made my 3090 work using the exact same method, and I'm using it right now, rock solid, multiple restarts, everything working.

 

I'd advise taking a look at the HOWTO I wrote above, try the instructions precisely, and let me know if it helped you.

 

  • Thanks 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.