Threadripper & AMD GPU Passthrough


iJumbo

Recommended Posts

Hi,

 

Is my first time posting here so hello!

 

I own a Threadripper 1950X for my workstation and i decided to test out if it will passthrough a gpu (R9 290).

 

I tested my Windows VM with SeaBIOS and OVMF but both give the same result:

(Also i dumped the rom file from SSH)

 

2017-12-13T12:21:00.277343Z qemu-system-x86_64: -device vfio-pci,host=42:00.0,id=hostdev0,x-vga=on,bus=pci.0,addr=0x5,romfile=/mnt/disk1/domains/R9290.dump: Failed to mmap 0000:42:00.0 BAR 0. Performance may be slow
2017-12-13T12:21:01.612110Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3
2017-12-13T12:21:01.612192Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3

 

Is there any way to remove this D3 state from the cpu?

 

 

If i force stop my VM after D3 state and try to turn it on again it gives this error:

 

internal error: Unknown PCI header type '127'

 

I've read many things about this errors but i never found a solution.
 

Thanks.


Settings:

 

<domain type='kvm'>
  <name>Principale SeaBIOS</name>
  <uuid>94458a68-da77-8396-b2d7-833bef533863</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='18'/>
    <vcpupin vcpu='2' cpuset='3'/>
    <vcpupin vcpu='3' cpuset='19'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='20'/>
    <vcpupin vcpu='6' cpuset='5'/>
    <vcpupin vcpu='7' cpuset='21'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.10'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='8' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Principale/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.126-2.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:c1:f0:f6'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x42' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/domains/R9290.dump'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x42' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc31f'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x04f2'/>
        <product id='0x0976'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

Link to comment

I edited the kernel and recompile it using the reddit post i linked before, still crashing some times but i'm trying to make it work.

Also using this:

 

append video=efifb:off amd_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream,multifunction rd.driver.pre=vfio-pci initrd=/bzroot

 

This combination made it work

Link to comment
  • 1 month later...
  • 1 month later...
  • 2 weeks later...
On 3/7/2018 at 6:55 PM, Jcloud said:

@ronni3 did you see this nvidial code 43 guide? I haven't tried it, but I know you're looking for anything that may help.  Also, I have an ASUS Prime X399-A board and a 1950X; I bought an GTX 1050 (not a great gamer card I know) for testing and it just worked. :\  Perhaps Ronnie3 you're reaching into the realm of a different mobo may be the trick?

 

@jcloud Hey I appreciate you looking out for any other ways to help me out. I tried that and it didn't work for me. I sold some 'coins' and went 'full-retard' (Tropic Thunder quote) and splurged on a Intel i9-7980x using another MSI board and I can confirm I am in working order now, but I now have a Threadripper and board I no longer need. I'm sure everyone will think purchasing this processor is overkill but I work in IT and want to build a number of VM's for testing Domain-related and network policy changes so the additional horsepower works for me. 

 

Know anyone that wants a Threadripper? :D

Edited by ronni3
Link to comment
1 hour ago, ronni3 said:

Know anyone that wants a Threadripper? :D

My co-worker does, but he's also saving, and wants to go to gen-con. I'll point him at your for sale post see if he, or another co-worker, will pull the trigger.  I honestly doubt they will, but I'm willing to solicit them for you; they'd be interested, just a matter of spare cash. 

 

1 hour ago, ronni3 said:

I'm sure everyone will think purchasing this processor is overkill but I work in IT and want to build a number of VM's for testing Domain-related and network policy changes so the additional horsepower works for me. 

I'd bet the threadripper would still be great for this if you stuck with the VNC video output instead of passing GPUs around. But, you don't have to justify your spending of shinny bits to me, do what you need/want and works best for you dude.  I've bought plenty of tech to work around bits in my unRAID before, I won't judge. 

 

I hope your new board and cpu work out for you.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.