GPU pass through failure

[Jetjockey]

Hello.

ive setup a Win 10 VM which works fine until I try to assign a GPU (Gforce GT7600 GS). Upon VM restart I get an Error message about QEMU shutting down the VM due to the vid card IOMMU group not being viable (see attached picture)

Is this due to the admittedly old graphics card not supporting UEFI and IOMMU? I'm thinking to buy a newer GT730. Would that do the job? I'm having difficulty finding out what graphics cards do support UEFI.

Thanks.

Edited February 21, 2017 by Jetjockey
Extra info

[1812]

First, make sure you're also giving the vm the sound component of your video card.

Second, check your iommu groupings under tools tab>system devices and see if your GPU/Sound component are the only 2 items listed in group 1.

If they aren't alone, you either need to physically move the gpu to another slot to try and get it to an isolated group, or consider enabling acs override to try and split the card out from the rest of the group.

 

If that doesn't work, then post your vm xml and system diagnostics.

on a windows vm, i think seabios is the one that works with non-UEFI cards( don't remember). But regardless, that doesn't matter if you don't have the iommu groupings correct.

[Jetjockey]

Hi.

Thanks very much for that, very helpful. I'll check this out before investing in the new GT730 I've reserved.

Regards Seabios, I gave this a try under a new VM as I believe this setting cannot be changed in an existing VM. Anyway, the results were exactly the same so I'm sure your correct re groupings. I'll check re the sound side of things now but I don't think the GT7600GS that I'm using has any sound capabilities?

Regards using another slot, I'm stuffed. When I built the server I used a Supermicro X10SL7-F-O. It appears to be a great quality board but I didn't consider VM's back then and it only has 1 PCIe slot. Dohh! I'm also using an E3 1231v3 CPU which is an excellent CPU but has no integrated GPU.

In hindsight, considering VM's, another Mobo and CPU combo might have been better. Maybe next time!

I'll report back shortly but am I correct in the assumption that:- With only 1 GPU available for VM's, only 1 VM (Win10, Linux etc) can be up and running at any one time?

Edit:...... Just checked the IOMMU groups. See info below. Cant see any sign of Audio hardware on Graphics card.

IOMMU group 0
	[8086:0c08] 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06)
IOMMU group 1
	[8086:0c01] 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
	[8086:0c05] 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06)
	[10de:0392] 01:00.0 VGA compatible controller: NVIDIA Corporation G73 [GeForce 7600 GS] (rev a1)
	[1000:0086] 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
IOMMU group 2
	[8086:8c31] 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
IOMMU group 3
	[8086:8c2d] 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
IOMMU group 4
	[8086:8c10] 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
IOMMU group 5
	[8086:8c14] 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5)
IOMMU group 6
	[8086:8c16] 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)
IOMMU group 7
	[8086:8c26] 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
IOMMU group 8
	[8086:8c52] 00:1f.0 ISA bridge: Intel Corporation C222 Series Chipset Family Server Essential SKU LPC Controller (rev 05)
	[8086:8c02] 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
	[8086:8c22] 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
	[8086:8c24] 00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05)
IOMMU group 9
	[1a03:1150] 03:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
	[1a03:2000] 04:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
IOMMU group 10
	[8086:1533] 05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
IOMMU group 11
	[8086:1533] 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

Thanks. 

Edited February 21, 2017 by Jetjockey
Add Log Filee

[Jetjockey]

Okay. Here are the groupings after setting ACS override to ON.

However.... With ACS overide set the VM appears to now run as there is no error generated, but there is no out put from the video card. The monitor screen is black and the monitor goes to sleep.
This also happens in a Linux VM I have setup. Is a modern video card going to be the only way out here?


IOMMU group 0
	[8086:0c08] 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06)
IOMMU group 1
	[8086:0c01] 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
IOMMU group 2
	[8086:0c05] 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06)
IOMMU group 3
	[8086:8c31] 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
IOMMU group 4
	[8086:8c2d] 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
IOMMU group 5
	[8086:8c10] 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
IOMMU group 6
	[8086:8c14] 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5)
IOMMU group 7
	[8086:8c16] 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)
IOMMU group 8
	[8086:8c26] 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
IOMMU group 9
	[8086:8c52] 00:1f.0 ISA bridge: Intel Corporation C222 Series Chipset Family Server Essential SKU LPC Controller (rev 05)
	[8086:8c02] 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
	[8086:8c22] 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
	[8086:8c24] 00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05)
IOMMU group 10
	[10de:0392] 01:00.0 VGA compatible controller: NVIDIA Corporation G73 [GeForce 7600 GS] (rev a1)
IOMMU group 11
	[1000:0086] 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
IOMMU group 12
	[1a03:1150] 03:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
	[1a03:2000] 04:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
IOMMU group 13
	[8086:1533] 05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
IOMMU group 14
	[8086:1533] 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

[Jetjockey]

Just tried another Graphics card Geforce 7300 and still no video output. XML file below.

<domain type='kvm'>
  <name>Windows 10</name>
  <uuid>28d671f1-bc0d-4a36-770c-37cfd4787876</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='6'/>
    <vcpupin vcpu='1' cpuset='7'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/28d671f1-bc0d-4a36-770c-37cfd4787876_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='1' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/Vdisks/Windows 10/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/Vdisks Images/ESD_ISO.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/Vdisks Images/virtio-win-0.1.130.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:84:8e:d9'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x062a'/>
        <product id='0x4101'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

[JonathanM]

1 hour ago, Jetjockey said:

I'm also using an E3 1231v3 CPU which is an excellent CPU but has no integrated GPU.

That motherboard has IPMI, which means the motherboard has onboard graphics, the ASPEED Graphics that is listed in your devices.

Not that it solves your current issue, but at least you have another graphics card in the system for the unraid console and don't have to worry about that whole mess.

[Jetjockey]

Yup, I suppose so but it's driving me nuts. I've tried almost everything. I even been into the mobo bios and set the offboard graphics for priority. This just gets the console up on both screens!! When I then start the Win10VM I still get no win desktop except for a split second flicker on whichever monitor (GPU) I've selected for the VM.

 

The mobo bios is listing the PCIe slot as "Legacy" so I'll try that on EFI. If no luck I'm going to try turning ACS override back off and see if that works. Finally I could try a newer card (Asus GT 730). If none of those work then I give up short of trying a really modern and expensive card.

 

This shouldn't be that difficult. Maybe it's the hardware I'm using, somebody must have this running!

 

Cheers.

[jonp]

Legacy GPUs are less likely to work for GPU pass through.  We don't have the GPUs you've used to test with ourselves, but it wouldn't shock me if they didn't work.

[Jetjockey]

Hello Jonp.

 

Many thanks for your input. Right I've got it working after a fiddle about. I tried all the steps I mentioned above such as enabling EFI for the PCIe slot on the cpu bus, disabling ACS override, unchecking the assigned USB peripherals (mouse and keyboard) etc, but nothing worked.

I decided to fit the new Graphics card purchased earlier from PC World, it's an Asus GT730 with 1Gb DDR3 ram and a 64 bit bus costing £52. Anyway....... still no luck when trying to start the VM, I was again getting the above message about IOMMU group 1 not being viable etc.

So, once again into the valley of death....... I changed the VM Global setting back to ACS override on. Bingo!

It now runs OK. Flushed with success, I assigned the USB keyboard and mouse to the VM and it still works great.

 

So it appears the older GPU's won't work and that's been the trouble all along.

 

But..... a couple of questions. Does setting ACS override to "on" cause any other problems? Is there another way to solve this IOMMU problem?

 

The Asus GT 730 is not a full size PCIe card, I didn't realise when I purchased it. This means it will fit in the remaining PCIe slot which is half length. The slot is labelled PCH PCIe. Perhaps if I try the card in that slot it will be assigned it's own IOMMU group without using ACS override?

 

Thanks.

Edited February 22, 2017 by Jetjockey

[Guest]

Hey guys!

 

I have the similar issues as above. But instead I already do own a recent graphics card (gtx1070). I'll try again tomorrow and post some results here. But maybe someone does have a solution beforehand

 

Thanks in advance!

[Jetjockey]

Hello Wessel145.

 

The answer to my pass through problems were two fold. First of all my video card was too old, I had to buy a newer one.

Next was the fact that the video card and its built in sound hardware MUST have their own IOMMU group.

You can check the IOMMU groupings on the unRaid webgui by selecting "Tools" and then "System Devices".

If the video card and sound component are sharing an IOMMU group with any other hardware in the computer (eg:- Ethernet controllers, PCI Bridges, etc) then you cant pass through the Graphics card to a VM.

 

Fixes appear to be:- Either enable "PCIe ACS Override" in the "VM Settings" advanced menu which seems to allow the GPU's to have their own IOMMU group.

 

Or...... Move the Graphics card to another slot and see if that allows it to have its own IOMMU group.

 

Both the above solutions worked for me so I eventually moved the Graphics card to another slot and then disabled the ACS override setting.

 

In hindsight (which is always 20/20 of course), I would have purchased a different Motherboard and processor had i known i was going to be using the server to run VM's

I would have picked a CPU with on board GPU, and a Motherboard with plenty of PCIe slots with plenty of room between them.

in fact, i'm going to start trawling the forum to see what others are using.

[Guest]

Thanks for the response

 

With ACS override off my system device group looks like this::

 

IOMMU group 1
	[8086:1901] 00:01.0 PCI bridge: Intel Corporation Skylake PCIe Controller (x16) (rev 07)
	[10de:1b81] 01:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
	[10de:10f0] 01:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)

So here it doesn't have its own IOMMU group. But the VM is turning on, Except i'm left with screen that turns off and goes to sleep (as you mentioned before)

 

When i turn ACS override on, the group looks like this:

 

IOMMU group 13
	[10de:1b81] 01:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
	[10de:10f0] 01:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)

With these settings i get almost the same error as you specified before.

 

I think there isn't a possibility to kick the audio device out of the group right?

 

Will try putting the graphics card in another PCIe slot in a moment

 

Edit: when i remove the sound card from the config i will get a black screen with ACS on AND ACS off

Edited February 23, 2017 by Guest

[1812]

you should be sending the GPU sound to the vm.

 

but if you want to try to separate them into different groups and omit it, you can try replacing he acs override currently in your syslinux.cfg with  the following:

 

 

append pcie_acs_override=id:10de:1b81,10de:10f0 initrd=/bzroot

 

[Jetjockey]

Sorry for the delayed reply. I'm no expert on this and am getting advice from others, but my setup, either with ACS override set to on, or with it off and the GT730 moved to a different slot, gives me both the Video card and its sound hardware in the same IOMMU group. I then allocate both of those items to the Win10 VM and it works.

 

Some suggestions might be...... Have you actually got the video card enabled in the Bios on your motherboard? Some Bios setups by default have the on board GPU or on CPU GPU set as active and any other Graphics device set to off. It may sound a silly idea but I'm grasping at straws here.

Which output are you trying on your GTX 1070? some cards by default only enable the VGA or DVI output until you set the Windows graphics driver to turn on the other output. Again, I'm grasping at straws.

 

My setup gave me exactly your situation when I was using an old GT7600GS card, the VM would start but I just got a black screen and the monitor went to sleep. The newer GT730 fixed that instantly. I am assuming that if the VM starts without complaint then it must be running but i'm sure that a forum expert may shoot me down in flames here.

If I think of anything else i'll post but at the moment you seem to have everything about right. Are you using the default Template settings for your VM?

[Guest]

I fixed it, bought a cheap ass gpu to run the Unraid console at, since then I'm able to succesfully boot into a VM on my GTX1070. Still can't assign my mobo sound to a VM (gives an error), trying to fix that

[Jetjockey]

Well, chase me up a tree with a kipper!

There you go, well done, that was going to be my next suggestion of course.

 

Its interesting that the VM ran in the first scenario you posted above. The consensus appears to be that unless the Graphics card and its Sound hardware have their own IOMMU group, then they cannot be passed through (assigned to the VM) so it should not run. Maybe that does not apply if the CPU itself is the other group member?

 

Edited February 23, 2017 by Jetjockey

[Kudjo]

I'm getting this error too!

 

I was hoping for some help. I have tried this with 5 video cards on two different systems (and three different versions of unraid) and have gotten the same result each time so it has to be something I'm missing.

 

Whenever I go to start a VM with a GPU assigned as the passthrough, I am met with the following:

 

Execution error

 

internal error: process exited while connecting to monitor: 2017-02-24T16:31:08.340691Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to set iommu for container: Operation not permitted
2017-02-24T16:31:08.340724Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to setup container for group 11
2017-02-24T16:31:08.340729Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to get group 11
2017-02-24T16:31:08.340744Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,id=hostdev0,bus=pci.0,addr=0x5: Device initialization failed

 

No amount of googling or searching on unraid forums has explained why this might be happening or how I might fix it. Any ideas?

 

vt-x and vt-d enabled in BIOS, UNRAID 6.3.2, using OVMF and "i440fx-2.7", Windows 10 VM, currently trying to passthrough an Nvidia 1070 Founder's Edition and also have an EVGA GTX 1060 SC 3GB (and several other older cards, 550ti, 8600, etc.), GPU is all alone in IOMMU group 11, Enable PCIe ACS Override set to "No"

 

What am I missing?