Jump to content
joecool169

GPU Passthrough

10 posts in this topic Last Reply

Recommended Posts

Windows 10 Vm

Newest version of Unraid

ASUSTeK COMPUTER INC. PRIME X470-PRO Version Rev X.0x

AMD Ryzen 5 2600X Six-Core @ 3600 MHz

Radeon HD 6850

 

"qemu-system-x86_64: vfio: Unable to power on device, stuck in D3"

 

What do I need to do to get this working? I've been trying off and on to pass through a gpu to any vm for weeks.

 

Do I need to post more info here?

I've watched all the spaceinvader videos and read a lot of stuff, but still can't get it to work.

 

Share this post


Link to post
12 hours ago, joecool169 said:

device, stuck in D3

This indicates the device is stuck in a lower power setting and isn't fully reset. The only way to recover from it is to reboot the server. After reboot the passthrough might only work once and a restart or power down and power up of the VM won't brings back the card and you have to restart the server again. This is usually an issue on newer AMD cards starting with the Navi series. Your card is a really old model and I'am not sure if someone managed it to get it working somehow. I guess not.

Share this post


Link to post

It gives that error the very first time you try to start the vm after a server reboot. As a test I disables auto start on all dockers and vm's and even the array. After reboot I brought the array online and then that vm and that vm only. Same error.

 

What else can I try?

Share this post


Link to post

So I have tried off and on for months to pass-through my Radeon HD 6850 to any vm without success. Kinda frustrating. I tried 3 windows 10 vm's, mac osx, and my latest attempt is Arch Linux.

 

 I pass through the card and I get an error : no suitable video mode found. As soon as I attempt to pass the card through to any vm the monitor attached to the card loses signal and signal never comes back unless I reboot. So my only way of trying to login to the vm is through my Fedora install with the vm manager.

 

Any idea how I can get this to work or what info I need to provide to troubleshoot?

 

I login to the vm with nomachine because I am trying to learn more about Linux, but the graphics are very laggy. I assume the lag might be resolved if I could pass this video card through.

Edited by joecool169
spelling mistake

Share this post


Link to post

Since the last post I installed an NVIDIA 8800GTX in place of the Radeon card, but when I try Arch I get the same results. 

 

I tried a fresh vm of Arch, but all Arch Vm's refuse to boot the install media if the video card is passed through

 

UPDATE: I plugged a dedicated monitor into the server. As it turns out the screen does not go blank when I attempt the passthrough, I was switching inputs on my monitor and seeing a blank screen not realizing that the Unraid screen goes blank after a few minutes, spacebar quickly brings the command prompt back. The monitor attached to the server never changes, so I assume the passthrough is never working at all. But the passthrough attempt does make any vm unusable.

 

SECOND UPDATE: I have passed the card through to a windows 10 VM but now I'm getting code 43 in windows. :(

Edited by joecool169
Added more information on my problem

Share this post


Link to post

Well after spending most of the weekend working on this I can pass the card through and everything works until I need to restart the Windows 10 vm. After selecting restart the monitor attached to the server goes black, if I login with spashtop desktop I can see that it goes back to code 43. :(

 

I could use some help, feel like I'm just talking to myself mostly.

 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>UR Gamer</name>
  <uuid>a20b3e5d-85af-0d96-37c2-d8b226341b42</uuid>
  <description>Unraid Server</description>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='10'/>
    <vcpupin vcpu='2' cpuset='5'/>
    <vcpupin vcpu='3' cpuset='11'/>
    <emulatorpin cpuset='0,6'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/a20b3e5d-85af-0d96-37c2-d8b226341b42_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='1234567890ab'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <ioapic driver='kvm'/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='2' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Windows.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.173-2.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/UR Gamer/vdisk1.img'/>
      <target dev='hdc' bus='sata'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:01:93:0e'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='2'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/Downloads/rom/evga.rom'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x0e8f'/>
        <product id='0x00a8'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

 

Edited by joecool169
added xml code

Share this post


Link to post

We may need to consider closing this subforum and directing all support inquiries directly to general support.  Sorry for the lack of attention this thread received.  Had I seen it sooner, I would have told you that your issue really stems down to your hardware.  There is one key problem with your build:  AMD.

 

AMD hardware just doesn't perform as well for this use-case as does Intel/NVIDIA.  We've seen many AMD-based systems just fail to support IOMMU properly.  We've also seen plenty of AMD GPUs that just don't support function-level resets which are required to pass through a GPU to a VM.  And you've already seen the third problem which is that even if you can get the GPU to pass through, rebooting the guest VM can cause the whole system to crash.

 

It is for these reasons we never recommend the use of AMD hardware for any setups that involve GPU pass through.  There are some users that have managed to make this work, but for most folks, these kinds of problems are common.  We've contacted AMD about this in the past but unfortunately our notices to them have fallen on deaf ears.

 

The best advice I can give you would be to A) contact AMD about your issue or B) switch to an Intel + NVIDIA setup.

Share this post


Link to post

A bit of a long shot.

 

if you have enough PCI-E power plugs, can you try the AMD card in x16 slot 1, and the Nvidia card in x16 slot 2.

Unraid should take the AMD card, then you can pass through the Nivida card without a VGA Bios.

 

It may also be worth trying CSM - compatibility support module which can be enabled in BIOS

 

A GT710 or even and elderly GT210 can be picked up for £10 / $15, I have a few as they make a good primary card as they are powered from the slot.

You do need a dummy dongle to fake a monitor and make sure it powers up though ~ $5

 

 

Share this post


Link to post

Well I happen to have an Intel I7 8700K, EVGA Z370FTW with a GTX 1070 sitting on my desk that is my main PC. Part of the reason for all of this passthrough stuff was an attempt to see if I could be happy with my main machine being Unraid and a VM.

 

If I were to put all my hard drives into the Intel machine can I expect that things will run more smoothly? I really do not want to be working on it all the time.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.