(Solved) HP gen8 GPU Passthrough

dominicmck · February 2, 2017

Hi All,

I recently installed a Xeon E3-1265L V2 into my gen8 microserver hoping to get GPU passthrough working for a Win10 casual gaming VM. I am able to select the GPU (and associated sound card) in the VM settings via the drop down, but when I start the VM I get an error (detailes below). I've looked around the forum and found a lot of old and conflicting reports on getting this fixed. Some user have given up, other say its working with brief mentions of fixes & hacks but no details on how to enable them. I've tried 'Enable PCIe ACS Override', and making a change to syslinuk.cfg.

I've googled a lot about what I found in the "KVM VGA Passthrough" spreadsheet (https://docs.google.com/spreadsheets/d/1LnGpTrXalwGVNy0PWJDURhyxa3sgqkGXmvNCIvIMenk/edit#gid=0) there is one reference to a working gen8 with the notes "with libvirt, bus=0x00(meaning use pcie), add kvm hidden into features" but I cant track down the username associated with the record to learn what this means or how to do it.

I'm leaning towards giving up and getting a refund on the CPU and putting my effort into a second machine build in a few months.

Thanks for any help or advice.

Dom

Technical details:

internal error: early end of file from monitor, possible problem: 2017-02-02T14:05:02.861779Z qemu-system-x86_64: -device vfio-pci,host=07:00.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to set iommu for container: Operation not permitted

2017-02-02T14:05:02.861806Z qemu-system-x86_64: -device vfio-pci,host=07:00.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to setup container for group 1

2017-02-02T14:05:02.861812Z qemu-system-x86_64: -device vfio-pci,host=07:00.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to get group 1

2017-02-02T14:05:02.861821Z qemu-system-x86_64: -device vfio-pci,host=07:00.0,id=hostdev0,bus=pci.0,addr=0x5: Device initialization failed

Info from my Tools > System Devices:

PCI Devices

00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller [8086:0158] (rev 09)

00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)

00:06.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:015d] (rev 09)

00:1a.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 [8086:1c2d] (rev 05)

00:1c.0 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 [8086:1c10] (rev b5)

00:1c.4 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 [8086:1c18] (rev b5)

00:1c.6 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 [8086:1c1c] (rev b5)

00:1c.7 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 [8086:1c1e] (rev b5)

00:1d.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 [8086:1c26] (rev 05)

00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev a5)

00:1f.0 ISA bridge [0601]: Intel Corporation C204 Chipset Family LPC Controller [8086:1c54] (rev 05)

00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller [8086:1c02] (rev 05)

01:00.0 System peripheral [0880]: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support [103c:3306] (rev 05)

01:00.1 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. MGA G200EH [102b:0533]

01:00.2 System peripheral [0880]: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging [103c:3307] (rev 05)

01:00.4 USB controller [0c03]: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller [103c:3300] (rev 02)

03:00.0 Ethernet controller [0200]: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe [14e4:165f]

03:00.1 Ethernet controller [0200]: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe [14e4:165f]

04:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] (rev 03)

07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GT218 [GeForce 8400 GS Rev. 3] [10de:10c3] (rev a2)

07:00.1 Audio device [0403]: NVIDIA Corporation High Definition Audio Controller [10de:0be3] (rev a1)

IOMMU Groups

/sys/kernel/iommu_groups/0/devices/0000:00:00.0

/sys/kernel/iommu_groups/1/devices/0000:00:01.0

/sys/kernel/iommu_groups/1/devices/0000:07:00.0

/sys/kernel/iommu_groups/1/devices/0000:07:00.1

/sys/kernel/iommu_groups/2/devices/0000:00:06.0

/sys/kernel/iommu_groups/3/devices/0000:00:1a.0

/sys/kernel/iommu_groups/4/devices/0000:00:1c.0

/sys/kernel/iommu_groups/5/devices/0000:00:1c.4

/sys/kernel/iommu_groups/6/devices/0000:00:1c.6

/sys/kernel/iommu_groups/7/devices/0000:00:1c.7

/sys/kernel/iommu_groups/8/devices/0000:00:1d.0

/sys/kernel/iommu_groups/9/devices/0000:00:1e.0

/sys/kernel/iommu_groups/10/devices/0000:00:1f.0

/sys/kernel/iommu_groups/10/devices/0000:00:1f.2

/sys/kernel/iommu_groups/11/devices/0000:03:00.0

/sys/kernel/iommu_groups/11/devices/0000:03:00.1

/sys/kernel/iommu_groups/12/devices/0000:04:00.0

/sys/kernel/iommu_groups/13/devices/0000:01:00.0

/sys/kernel/iommu_groups/13/devices/0000:01:00.1

/sys/kernel/iommu_groups/13/devices/0000:01:00.2

/sys/kernel/iommu_groups/13/devices/0000:01:00.4

2.6k · February 2, 2017

on my g6 servers, I had a similar, if not the same, iommu issue.

It was resolved by adding the following to the syslinux.cfg to allow unsafe interrupts.

append vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot

I know you've said you've done it, but can you post your complete syslinux.cfg file?

on second look, you also have an iLo usb controller in the same iommu group (13). If you con't pass that through as well, or find a way to split it out, it'll never work. Does ACS override not split the group up better? If not, let me know, I had another issue that I somehow managed to split out a video portion of a card away from an audio portion for passthrough, which might work for this?

just woke up and realized I was looking at the onboard video....

dominicmck · February 2, 2017

Hi, thanks for the quick reply. I've added the code you mentioned to my /boot/syslinux/syslinux.cfg using MC. but no luck, same error. Full content is:

default /syslinux/menu.c32

menu title Lime Technology, Inc.

prompt 0

timeout 50

label unRAID OS

menu default

kernel /bzimage

append vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot

label unRAID OS GUI Mode

kernel /bzimage

append initrd=/bzroot,/bzroot-gui

label unRAID OS Safe Mode (no plugins, no GUI)

kernel /bzimage

append initrd=/bzroot unraidsafemode

label Memtest86+

kernel /memtest

My Win10 VM XML is:

<name>Windows 10</name>

</metadata>

</memoryBacking>

</cputune>

<os>

<loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>

<nvram>/etc/libvirt/qemu/nvram/71292d03-255e-9f6a-9454-c7eb2a22793b_VARS-pure-efi.fd</nvram>

</os>

<acpi/>

<apic/>

</hyperv>

</features>

</cpu>

</clock>

<on_poweroff>destroy</on_poweroff>

<on_reboot>restart</on_reboot>

<on_crash>restart</on_crash>

<emulator>/usr/local/sbin/qemu</emulator>

</disk>

</controller>

</controller>

</controller>

</controller>

</controller>

</interface>

</serial>

</console>

</channel>

</source>

</hostdev>

</source>

</hostdev>

</memballoon>

</devices>

</domain>

2.6k · February 3, 2017

OK, less sleepy answer:

Your problem is that this

00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)

is in the same iommu group with your video card

/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:07:00.0
/sys/kernel/iommu_groups/1/devices/0000:07:00.1

You can't pass through only part of an iommu group. It's all or nothing.

You said that ACS Override did not do anything. So there are 2 options that I can think of (others may have more or better info)

1. Can you move the card to another slot and see if it changes the iommu group to just only contain the video and audio components ( currently 07:00.0 & 07:00.1 but will possibly change)?

2. I had a different issue with iommu groups and a possibly faulty nvidia card https://lime-technology.com/forum/index.php?topic=54786.msg523314#msg523314

If you attach your logs, it will help diagnose the situation better, I believe the issue I had is what you have, but the fix for it might work as it achieves the desired result, separating devices out of an iommu group.

So to try it, add the following to your syslinux.cfg

append vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=id:10de:10c3,10de:0be3 initrd=/bzroot

This will attempt to move the card components into individual iommu groups.

You don't have to use MC, you can do this in the web gui under Main tab > click on the text "Flash" under boot device, scroll down and edit there, apply, then reboot.

After reboot, check iommu groups and see if the devices are now separated (may be 2 different groups.) If so, try to boot vm. --> Since I was not using the audio portion of the gpu in the link I posted, I do not know what the effects may be if the video and audio are in separate iommu groups. If the vm won't boot, then remove audio and try again.

But really, logs would help a great deal.(Tools>diagnostics)

--edit

there is also this, while not directly related (or might be, but can't tell without logs: http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c04781229&sp4ts.oid=5249566

dominicmck · February 3, 2017

Thanks for the detailed answer. Lots for me to test and check so might talk a day or two to get through (I only have remote access most of the time as the server is offsite at the moment). Making the syslinux.cfg change (thx for the tip on easier editing) didn’t help, and since it’s a Gen8 Microserver, there are no more pcie slots to try. I’m going to have another go with a different GPU this weekend, and will let you know how I get on.

The link to the HP article looks like it was relevant, but I already have the bios it suggest as the resolution. I will double check though.

I think unless I can track down someone with the same microserver who has resolved this (I’ve seen a couple of others on here who couldn’t and got new hardware), I might just have to chalk it up to hardware incompatibility and move on. While I’d love to get to a definitive answer about what the issue is, sometime you just need to accept the joys of tech and focus on more productive projects. This was only a long shot to save some money anyway. I should really focus on getting a proper solution in place, with dedicated fit for purpose hardware.

dominicmck · February 3, 2017

Logs attached if you're still interested in taking a look.

Cheers,

Dom

tower-diagnostics-20170203-1219.zip

John_M · February 3, 2017

FWIW I have a Gen 8 and it works very well with unRAID as a NAS and a couple of dockers but I have the stock Celeron processor and don't do any virtualisation on it.

2.6k · February 3, 2017

Ok, so here is a problem in the system log:

Tower kernel: vfio-pci 0000:07:00.1: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor

.

and that's the issue I had. I would say try booting the vm without audio, but the video card is still in group 1 with another device and most likely won't boot... but.. it costs nothing to try.

This also pops up in the logs

Feb  3 12:01:08 Tower kernel: [Firmware Bug]: the BIOS has corrupted hw-PMU resources (MSR 38d is 330)

This actually shows up in my logs as well, but can be ignored from what I read online.

dominicmck · July 16, 2020

For anyone that comes across it. I have managed to get this working. Lots of old info on the forums that aren't current. Newer info that helped was:

Latest Bios J06 04/04/2019
Patched unRaid to get around RMRR.
The usual "append vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=id:XXXX:XXXX,YYYY:YYYY initrd=/bzroot"
DVI to HDMI cable (the hdmi port on my GT 1030 wont output for a VM)
No Bios specified in the xml for this GPU. Windows and Ubuntu, both running fine with no gpu bios. Nvidia drivers installed, no code 43 issue. Adding a Bios actually stopped the VM from working at all.
In the VM XML add the multifunction = on and set sound and gpu to be same slot, different functions.
In the VM XML I had to shuffle the bus and slots for some reason. Slot 5 was the default, but making it slot 2 seemed to be the final thing to make it work.
Hyper-V = no for Win10 VM.

Note, this was with a different CPU to the original post but hopefully doesn't matter. Now using a Intel Xeon E31-260L

Edited July 17, 2020 by dominicmck

Umizoomi · February 26, 2021

On 7/16/2020 at 5:58 PM, dominicmck said:

For anyone that comes across it. I have managed to get this working. Lots of old info on the forums that aren't current. Newer info that helped was:

Latest Bios J06 04/04/2019

Patched unRaid to get around RMRR.

The usual "append vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=id:XXXX:XXXX,YYYY:YYYY initrd=/bzroot"

DVI to HDMI cable (the hdmi port on my GT 1030 wont output for a VM)

No Bios specified in the xml for this GPU. Windows and Ubuntu, both running fine with no gpu bios. Nvidia drivers installed, no code 43 issue. Adding a Bios actually stopped the VM from working at all.

In the VM XML add the multifunction = on and set sound and gpu to be same slot, different functions.

In the VM XML I had to shuffle the bus and slots for some reason. Slot 5 was the default, but making it slot 2 seemed to be the final thing to make it work.

Hyper-V = no for Win10 VM.

Note, this was with a different CPU to the original post but hopefully doesn't matter. Now using a Intel Xeon E31-260L

How are things running now?

I'm trying to passthrough a gt710 to a Ubuntu VM, I tried all the things in the old threads, here's where I am now:

✓ Latest Bios J06 04/04/2019
✓ Patched unRaid to get around RMRR.
✓ The usual "append vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=id:XXXX:XXXX,YYYY:YYYY initrd=/bzroot"
✓ DVI to HDMI cable (the hdmi port on my GT 1030 wont output for a VM)
✓ No Bios specified in the xml for this GPU. Windows and Ubuntu, both running fine with no gpu bios. Nvidia drivers installed, no code 43 issue. Adding a Bios actually stopped the VM from working at all.
✓ In the VM XML add the multifunction = on and set sound and gpu to be same slot, different functions.
? In the VM XML I had to shuffle the bus and slots for some reason. Slot 5 was the default, but making it slot 2 seemed to be the final thing to make it work.
- (i'm using Ubuntu) Hyper-V = no for Win10 VM.

Can you collaborate a bit on what you mean with shuffle the bus and slots? Below is the specs of my machine, VM settings, syslinux configuration and device info

syslinux config

default menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label Unraid OS
  menu default
  kernel /bzimage
  append intel_iommu=relax_rmrr pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot
label Unraid OS GUI Mode
  kernel /bzimage
  append intel_iommu=relax_rmrr pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot,/bzroot-gui
label Unraid OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label Unraid OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest

device info:

IOMMU group 0:				[8086:0158] 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller (rev 09)
IOMMU group 1:				[8086:0151] 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
IOMMU group 2:				[8086:015d] 00:06.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
IOMMU group 3:			 	[8086:1c2d] 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 003: ID 046d:c52b Logitech, Inc. Unifying Receiver
Bus 001 Device 004: ID 0781:556b SanDisk Corp. Cruzer Edge
IOMMU group 4:				[8086:1c10] 00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
IOMMU group 5:				[8086:1c18] 00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
IOMMU group 6:				[8086:1c1c] 00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)
IOMMU group 7:				[8086:1c1e] 00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5)
IOMMU group 8:			 	[8086:1c26] 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 003: ID 0424:2660 Standard Microsystems Corp. Hub
Bus 002 Device 004: ID 0424:4030 Standard Microsystems Corp. Ultra Fast Media Reader
IOMMU group 9:				[8086:244e] 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
IOMMU group 10:			 	[8086:1c54] 00:1f.0 ISA bridge: Intel Corporation C204 Chipset LPC Controller (rev 05)
 	[8086:1c02] 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller (rev 05)
[2:0:0:0]    disk    ATA      WDC WD10EZEX-00M 1A01  /dev/sdc   1.00TB
[3:0:0:0]    disk    ATA      WDC WD10EZEX-08W 1A02  /dev/sdd   1.00TB
[4:0:0:0]    disk    ATA      WDC WD10EZEX-08W 1A02  /dev/sde   1.00TB
[6:0:0:0]    disk    ATA      KINGSTON SUV400S 96R9  /dev/sdf    240GB
IOMMU group 11:			 	[10de:128b] 07:00.0 VGA compatible controller: NVIDIA Corporation GK208B [GeForce GT 710] (rev a1)
IOMMU group 12:			 	[10de:0e0f] 07:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
IOMMU group 13:			 	[14e4:165f] 03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
IOMMU group 14:			 	[14e4:165f] 03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
IOMMU group 15:			 	[1

Ubuntu VM

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='1'>
  <name>Ubuntu</name>
  <uuid>b77061c4-89f4-bc55-ad5a-a874e4ba9112</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Ubuntu" icon="ubuntu.png" os="ubuntu"/>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='2'/>
    <vcpupin vcpu='2' cpuset='1'/>
    <vcpupin vcpu='3' cpuset='3'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/b77061c4-89f4-bc55-ad5a-a874e4ba9112_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='2' threads='2'/>
    <cache mode='passthrough'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Ubuntu/vdisk1.img' index='1'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <filesystem type='mount' accessmode='passthrough'>
      <source dir='/mnt/user/userdata/'/>
      <target dir='userdata'/>
      <alias name='fs0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </filesystem>
    <interface type='bridge'>
      <mac address='52:54:00:7b:6f:70'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-Ubuntu/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc52b'/>
        <address bus='1' device='3'/>
      </source>
      <alias name='hostdev2'/>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>

Machine specs:
HP MicroServer Gen8
E3-1220L V2 CPU
gt710
8gb eec

dominicmck · February 28, 2021

Hi, been a while so I can't find the article that helped me. But as remember it, the issue was the GPU video and audio need to be on the same bus & 'top' slot which it isn't by default. The article I read suggested to think about the virtual motherboard you are trying to create, and the layout of the items on it. You wouldn't put your primary GPU in your bottom PICE slot in a physical MB, so you dont on a virtual one either. My VM thought 2 was the 'top' pcie slot. Your virtual top slot might be different. There was already something defined on slot 2, so I had to shuffle that to another slot before I could put the GPU there.

Umizoomi · February 28, 2021

6 hours ago, dominicmck said:

Hi, been a while so I can't find the article that helped me. But as remember it, the issue was the GPU video and audio need to be on the same bus & 'top' slot which it isn't by default. The article I read suggested to think about the virtual motherboard you are trying to create, and the layout of the items on it. You wouldn't put your primary GPU in your bottom PICE slot in a physical MB, so you dont on a virtual one either. My VM thought 2 was the 'top' pcie slot. Your virtual top slot might be different. There was already something defined on slot 2, so I had to shuffle that to another slot before I could put the GPU there.

Thanks for your reaction. Can you maybe share your VM XML? we have the same board, just a different CPU, and probably GPU. so i'm wondering what's different in your configuration which makes it work.

I managed to get the VM booting now without errors. the screen turns off though after booting up (by default i see the unraid console)

dominicmck · February 28, 2021

I moved my GPU to another PC, so don't have a working XML at the moment, sorry. The attached are from around the time I got it working. Could be a clue in them.

Win10GPUWorking.txt UbuntuGPUWorking.txt

(Solved) HP gen8 GPU Passthrough

Recommended Posts

dominicmck

Link to comment

1812

Link to comment

dominicmck

Link to comment

1812

Link to comment

dominicmck

Link to comment

dominicmck

Link to comment

John_M

Link to comment

1812

Link to comment

dominicmck

Link to comment

Umizoomi

Link to comment

dominicmck

Link to comment

Umizoomi

Link to comment

dominicmck

Link to comment

Join the conversation