HP Proliant / Workstation & unRaid Information Thread


1812

Recommended Posts

On 8/12/2017 at 11:32 AM, 1812 said:

append pcie_acs_override=id:XXXX:XXXX,YYYY:YYYY initrd=/bzroot

 

 

14 hours ago, burningstarIV said:

Picked up a dl360e gen8 for some pfsense passthrough, but having some issues with its iommu groups and not working. Have tried the stub and vfio-pci.ids, with vfio_iommu_type1.allow_unsafe_interrupts=1 - neither has worked. I think its a 2014 bios and didn't see a newer one that I could get in the 2017 updates .... passing through a hp NC364T intel 4 port. Do you have any experience with it, or would a dell branded Broadcom work any differently ? 

 

Was trying on my 360 g6 but was running into rmrr issues even with a pre 2012 bios .. gave up and got this gen 8, which is great, but would be better if the network card worked ... 

 

IOMMU group 25
    [111d:8018] 09:02.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
    [8086:10bc] 0a:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)
    [8086:10bc] 0a:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)
IOMMU group 26
    [111d:8018] 09:04.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
    [8086:10bc] 0b:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)
    [8086:10bc] 0b:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)
 

 

 

 

 

 

The bridge is causing the problem on passthrough. Try this to see if you can pry it away from the ethernet ports:


 

append pcie_acs_override=id:8086:10bc vfio_iommu_type1.allow_unsafe_interrupts=1  vfio-pci.ids=,8086:10bc initrd=/bzroot

or


 

append pcie_acs_override=id:111d:8018,8086:10bc vfio_iommu_type1.allow_unsafe_interrupts=1  vfio-pci.ids=111d:8018,8086:10bc initrd=/bzroot

if that doesn't work and the bridge is still with the 2 ports in the groupings, then you can try manually adding the bridge for the group. The xml would look like this (per your posted grouping for 26 ):

 


 

 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x04' function='0x0'/>
      </source>
    </hostdev>
 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
      </source>
    </hostdev>
 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/>
      </source>
    </hostdev>

 

 

I removed the vm address lines as they will be populated on vm start.

 

I'm not sure if that will work, but it could. 

 

On my machines, I use a PCIe card out of a semi-irrational belief that it's better to keep the firewall traffic slightly more isolated off my server that way, than using the internal 

 

 

For reference, I use 

append pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 vfio-pci.ids=1b73:1100,8086:10e8 initrd=/bzroot

 

 

and my card splits to:

 

IOMMU group 14:	[111d:8018] 02:02.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
IOMMU group 15:	[111d:8018] 02:04.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
IOMMU group 16:	[8086:10e8] 03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
IOMMU group 17:	[8086:10e8] 03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
IOMMU group 18:	[8086:10e8] 04:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
IOMMU group 19:	[8086:10e8] 04:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

 

 

 

 

Link to comment

added that all in but got a new and improved error - looking up on search just seeing acs enabling - let me know what you think 

 

 

default /syslinux/menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
 append pcie_acs_override=id:111d:8018,8086:10bc vfio_iommu_type1.allow_unsafe_interrupts=1  vfio-pci.ids=111d:8018,8086:10bc initrd=/bzroot
label unRAID OS GUI Mode
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label unRAID OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest

 

IOMMU group 25: [111d:8018] 09:02.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
 
IOMMU group 26: [111d:8018] 09:04.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
 
IOMMU group 27: [8086:10bc] 0a:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)
  [8086:10bc] 0a:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)
 
IOMMU group 28: [8086:10bc] 0b:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)
  [8086:10bc] 0b:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)

 

 

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x02' function='0x0'/>
      </source>
    </hostdev>
 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
      </source>
    </hostdev>
 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x1'/>
      </source>
    </hostdev>
 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x04' function='0x0'/>
      </source>
    </hostdev>
 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
      </source>
    </hostdev>
 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/>
      </source>
    </hostdev>

 

 

 

 

Screen Shot 2018-02-09 at 11.43.08 PM.png

Link to comment

Hello,

 

i am also trying to passthrough a HD 6450 from my Microserver gen8 running the latest stable unRaid version. Having the same problem.

 

Here is my config:

default menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
append pcie_acs_override=id:8086:0151,1002:6779,1002:aa98 vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot
label unRAID OS GUI Mode
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label unRAID OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest


 

IOMMU group 0:	[8086:0158] 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller (rev 09)
IOMMU group 1:	[8086:0151] 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
IOMMU group 2:	[8086:015d] 00:06.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
IOMMU group 3:	[8086:1c2d] 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
IOMMU group 4:	[8086:1c10] 00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
IOMMU group 5:	[8086:1c18] 00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
IOMMU group 6:	[8086:1c1c] 00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)
IOMMU group 7:	[8086:1c1e] 00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5)
IOMMU group 8:	[8086:1c26] 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
IOMMU group 9:	[8086:244e] 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
IOMMU group 10:	[8086:1c54] 00:1f.0 ISA bridge: Intel Corporation C204 Chipset Family LPC Controller (rev 05)
[8086:1c02] 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller (rev 05)
IOMMU group 11:	[1002:6779] 07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM]
[1002:aa98] 07:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM]
IOMMU group 12:	[14e4:165f] 03:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe
[14e4:165f] 03:00.1 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe
IOMMU group 13:	[1912:0014] 04:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
IOMMU group 14:	[103c:3306] 01:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 05)
[102b:0533] 01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH
[103c:3307] 01:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 05)
[103c:3300] 01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 02)

Here are the logs:

Feb 10 10:53:14 Server kernel: vfio-pci 0000:07:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Feb 10 10:53:14 Server kernel: br0: port 2(vnet0) entered blocking state
Feb 10 10:53:14 Server kernel: br0: port 2(vnet0) entered disabled state
Feb 10 10:53:14 Server kernel: device vnet0 entered promiscuous mode
Feb 10 10:53:14 Server kernel: br0: port 2(vnet0) entered blocking state
Feb 10 10:53:14 Server kernel: br0: port 2(vnet0) entered forwarding state
Feb 10 10:53:15 Server kernel: pci 0000:07:00.1: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.
Feb 10 10:53:15 Server kernel: br0: port 2(vnet0) entered disabled state
Feb 10 10:53:15 Server kernel: device vnet0 left promiscuous mode
Feb 10 10:53:15 Server kernel: br0: port 2(vnet0) entered disabled state
Feb 10 10:53:15 Server kernel: vfio-pci 0000:07:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none

 

Link to comment
2 hours ago, D874as said:

Hello,

 

i am also trying to passthrough a HD 6450 from my Microserver gen8 running the latest stable unRaid version. Having the same problem.

 

Here is my config:


default menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
append pcie_acs_override=id:8086:0151,1002:6779,1002:aa98 vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot
label unRAID OS GUI Mode
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label unRAID OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest


 


IOMMU group 0:	[8086:0158] 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller (rev 09)
IOMMU group 1:	[8086:0151] 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
IOMMU group 2:	[8086:015d] 00:06.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
IOMMU group 3:	[8086:1c2d] 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
IOMMU group 4:	[8086:1c10] 00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
IOMMU group 5:	[8086:1c18] 00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
IOMMU group 6:	[8086:1c1c] 00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)
IOMMU group 7:	[8086:1c1e] 00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5)
IOMMU group 8:	[8086:1c26] 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
IOMMU group 9:	[8086:244e] 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
IOMMU group 10:	[8086:1c54] 00:1f.0 ISA bridge: Intel Corporation C204 Chipset Family LPC Controller (rev 05)
[8086:1c02] 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller (rev 05)
IOMMU group 11:	[1002:6779] 07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM]
[1002:aa98] 07:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM]
IOMMU group 12:	[14e4:165f] 03:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe
[14e4:165f] 03:00.1 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe
IOMMU group 13:	[1912:0014] 04:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
IOMMU group 14:	[103c:3306] 01:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 05)
[102b:0533] 01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH
[103c:3307] 01:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 05)
[103c:3300] 01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 02)

Here are the logs:


Feb 10 10:53:14 Server kernel: vfio-pci 0000:07:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Feb 10 10:53:14 Server kernel: br0: port 2(vnet0) entered blocking state
Feb 10 10:53:14 Server kernel: br0: port 2(vnet0) entered disabled state
Feb 10 10:53:14 Server kernel: device vnet0 entered promiscuous mode
Feb 10 10:53:14 Server kernel: br0: port 2(vnet0) entered blocking state
Feb 10 10:53:14 Server kernel: br0: port 2(vnet0) entered forwarding state
Feb 10 10:53:15 Server kernel: pci 0000:07:00.1: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.
Feb 10 10:53:15 Server kernel: br0: port 2(vnet0) entered disabled state
Feb 10 10:53:15 Server kernel: device vnet0 left promiscuous mode
Feb 10 10:53:15 Server kernel: br0: port 2(vnet0) entered disabled state
Feb 10 10:53:15 Server kernel: vfio-pci 0000:07:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none

 

 

If you read the section above regarding RMRR problems, you'll see it relates to your audio portion of the gpu. Different problem than previous posters. Try removing 8086:0151 from the override line and see if that allows the video and audio to be split into different IOUMMU groups, and if so, don't pass through the audio to the vm.

Link to comment
16 hours ago, burningstarIV said:

added that all in but got a new and improved error - looking up on search just seeing acs enabling - let me know what you think 

 

 

default /syslinux/menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
 append pcie_acs_override=id:111d:8018,8086:10bc vfio_iommu_type1.allow_unsafe_interrupts=1  vfio-pci.ids=111d:8018,8086:10bc initrd=/bzroot
label unRAID OS GUI Mode
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label unRAID OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest

 

IOMMU group 25: [111d:8018] 09:02.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
 
IOMMU group 26: [111d:8018] 09:04.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
 
IOMMU group 27: [8086:10bc] 0a:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)
  [8086:10bc] 0a:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)
 
IOMMU group 28: [8086:10bc] 0b:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)
  [8086:10bc] 0b:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)

 

 

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x02' function='0x0'/>
      </source>
    </hostdev>
 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
      </source>
    </hostdev>
 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x1'/>
      </source>
    </hostdev>
 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x04' function='0x0'/>
      </source>
    </hostdev>
 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
      </source>
    </hostdev>
 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/>
      </source>
    </hostdev>

 

 

 

 

Screen Shot 2018-02-09 at 11.43.08 PM.png

 

WELL, at least we got them divided into groups now.

 

This is a new error one for me. Try unstubbing the bridge id and reboot. If same error occurs, try unstubbing the card id and reboot (confirming iommu groupings along the way each time.) If that doesn't work, then it may be time for research. 

Link to comment

After editing config file to:

 

default menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
append pcie_acs_override=id:1002:6779,1002:aa98 vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot
label unRAID OS GUI Mode
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label unRAID OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest

 

I got following output:

 

IOMMU group 0:	[8086:0158] 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller (rev 09)
IOMMU group 1:	[8086:0151] 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
	[1002:6779] 07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM]
	[1002:aa98] 07:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM]
IOMMU group 2:	[8086:015d] 00:06.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
IOMMU group 3:	[8086:1c2d] 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
IOMMU group 4:	[8086:1c10] 00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
IOMMU group 5:	[8086:1c18] 00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
IOMMU group 6:	[8086:1c1c] 00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)
IOMMU group 7:	[8086:1c1e] 00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5)
IOMMU group 8:	[8086:1c26] 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
IOMMU group 9:	[8086:244e] 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
IOMMU group 10:	[8086:1c54] 00:1f.0 ISA bridge: Intel Corporation C204 Chipset Family LPC Controller (rev 05)
	[8086:1c02] 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller (rev 05)
IOMMU group 11:	[14e4:165f] 03:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe
	[14e4:165f] 03:00.1 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe
IOMMU group 12:	[1912:0014] 04:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
IOMMU group 13:	[103c:3306] 01:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 05)
	[102b:0533] 01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH
	[103c:3307] 01:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 05)
	[103c:3300] 01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 02)

trying to create a vm with only the video passthrough didn't help.
I Still get the same log:

 

Feb 10 13:28:14 Server kernel: vfio-pci 0000:07:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Feb 10 13:28:14 Server kernel: br0: port 2(vnet0) entered blocking state
Feb 10 13:28:14 Server kernel: br0: port 2(vnet0) entered disabled state
Feb 10 13:28:14 Server kernel: device vnet0 entered promiscuous mode
Feb 10 13:28:14 Server kernel: br0: port 2(vnet0) entered blocking state
Feb 10 13:28:14 Server kernel: br0: port 2(vnet0) entered forwarding state
Feb 10 13:28:14 Server kernel: pci 0000:07:00.1: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.
Feb 10 13:28:14 Server kernel: br0: port 2(vnet0) entered disabled state
Feb 10 13:28:14 Server kernel: device vnet0 left promiscuous mode
Feb 10 13:28:14 Server kernel: br0: port 2(vnet0) entered disabled state
Feb 10 13:28:14 Server kernel: vfio-pci 0000:07:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Feb 10 13:29:13 Server kernel: usb 3-1: USB disconnect, device number 2
Feb 10 13:29:13 Server acpid: input device has been disconnected, fd 4

 

Edited by D874as
Link to comment
20 minutes ago, D874as said:

After editing config file to:

 


default menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
append pcie_acs_override=id:1002:6779,1002:aa98 vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot
label unRAID OS GUI Mode
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label unRAID OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest

 

I got following output:

 


IOMMU group 0:	[8086:0158] 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller (rev 09)
IOMMU group 1:	[8086:0151] 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
	[1002:6779] 07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM]
	[1002:aa98] 07:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM]
IOMMU group 2:	[8086:015d] 00:06.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
IOMMU group 3:	[8086:1c2d] 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
IOMMU group 4:	[8086:1c10] 00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
IOMMU group 5:	[8086:1c18] 00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
IOMMU group 6:	[8086:1c1c] 00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)
IOMMU group 7:	[8086:1c1e] 00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5)
IOMMU group 8:	[8086:1c26] 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
IOMMU group 9:	[8086:244e] 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
IOMMU group 10:	[8086:1c54] 00:1f.0 ISA bridge: Intel Corporation C204 Chipset Family LPC Controller (rev 05)
	[8086:1c02] 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller (rev 05)
IOMMU group 11:	[14e4:165f] 03:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe
	[14e4:165f] 03:00.1 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe
IOMMU group 12:	[1912:0014] 04:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
IOMMU group 13:	[103c:3306] 01:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 05)
	[102b:0533] 01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH
	[103c:3307] 01:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 05)
	[103c:3300] 01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 02)

trying to create a vm with only the video passthrough didn't help.
I Still get the same log:

 


Feb 10 13:28:14 Server kernel: vfio-pci 0000:07:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Feb 10 13:28:14 Server kernel: br0: port 2(vnet0) entered blocking state
Feb 10 13:28:14 Server kernel: br0: port 2(vnet0) entered disabled state
Feb 10 13:28:14 Server kernel: device vnet0 entered promiscuous mode
Feb 10 13:28:14 Server kernel: br0: port 2(vnet0) entered blocking state
Feb 10 13:28:14 Server kernel: br0: port 2(vnet0) entered forwarding state
Feb 10 13:28:14 Server kernel: pci 0000:07:00.1: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.
Feb 10 13:28:14 Server kernel: br0: port 2(vnet0) entered disabled state
Feb 10 13:28:14 Server kernel: device vnet0 left promiscuous mode
Feb 10 13:28:14 Server kernel: br0: port 2(vnet0) entered disabled state
Feb 10 13:28:14 Server kernel: vfio-pci 0000:07:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Feb 10 13:29:13 Server kernel: usb 3-1: USB disconnect, device number 2
Feb 10 13:29:13 Server acpid: input device has been disconnected, fd 4

 

 

 

It put the bridge back in with the GPU...... fun.

 

is there a different slot you can try? my guess is no since it's a micro server....

 

to be clear, this is an HP issue, not unRaid issue. HP has been a pain about actually fixing this. And their "fix" is a workaround that is not a simple as "click this to make it work."

Link to comment
3 minutes ago, 1812 said:

 

 

It put the bridge back in with the GPU...... fun.

 

is there a different slot you can try? my guess is no since it's a micro server....

 

to be clear, this is an HP issue, not unRaid issue. HP has been a pain about actually fixing this. And their "fix" is a workaround that is not a simple as "click this to make it work."

 yes I am using a Microserver ..

how much of a workaround is that? I am still new to this topic, also doing some reading on IOMMU etc.

Link to comment

From reading back and forth on all this stuff it seems the micro servers have a bunch of problems. On my 580 I had to get an older 2012 firmware to get around the rmrr. Try to look into your bios and see if running the latest version. From there you may be able to change the network over to non shared memory to get past this. Not sure what the success rate is but its a problem with how they use the intel cpus 

Link to comment
15 hours ago, D874as said:

 yes I am using a Microserver ..

how much of a workaround is that? I am still new to this topic, also doing some reading on IOMMU etc.

 

This is their fix for the gen 8/9 servers: https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04781229&sp4ts.oid=5249566

 

Not sure how applicable it is for micro servers.

 

Here is also a redhat white paper on it : https://access.redhat.com/sites/default/files/attachments/rmrr-wp1.pdf

 

Additionally, here is a proxmox post (in German) about passthrough problems on a gen 8 microserver. https://forum.proxmox.com/threads/hp-proliant-microserver-gen8-raidcontroller-hp-p410-passthrough-probleme.30547/

 

They also describe changing the boot to include "iommu=on" or "intel_iommu=on" but I have not tried those. (there are also other options intended for AMD processors which include "iommu=pt" and "iommu=1" .   

 

I also remember reading somewhere that in some newer bios for newer servers that HP had an option (don't remember what it was called) that enables the exclusion of cards for passthrough (but this was in passing about 6 months ago)

 

Link to comment
44 minutes ago, 1812 said:

 

This is their fix for the gen 8/9 servers: https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04781229&sp4ts.oid=5249566

 

Not sure how applicable it is for micro servers.

 

Here is also a redhat white paper on it : https://access.redhat.com/sites/default/files/attachments/rmrr-wp1.pdf

 

Additionally, here is a proxmox post (in German) about passthrough problems on a gen 8 microserver. https://forum.proxmox.com/threads/hp-proliant-microserver-gen8-raidcontroller-hp-p410-passthrough-probleme.30547/

 

They also describe changing the boot to include "iommu=on" or "intel_iommu=on" but I have not tried those. (there are also other options intended for AMD processors which include "iommu=pt" and "iommu=1" .   

 

I also remember reading somewhere that in some newer bios for newer servers that HP had an option (don't remember what it was called) that enables the exclusion of cards for passthrough (but this was in passing about 6 months ago)

 

thanks @1812 seems like I need to compile my own kernel. Maybe it also works for a custom made unRaid kernel? First of all, I need to educate myself about kernel building xD

Link to comment
32 minutes ago, D874as said:

thanks @1812 seems like I need to compile my own kernel. Maybe it also works for a custom made unRaid kernel? First of all, I need to educate myself about kernel building xD

 

Also, maybe look into "iommu_inclusive_mapping=1"

 

I've seen a few instances online where it has been used to help with rmrr issues but really know very little about it.

Link to comment

Hi, i have a microserver gen 8 and i'm getting absolutely hammered with the ACPI errors.
 

Quote

Jan 13 16:00:52 UNRAID kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20170728/exfield-427)
Jan 13 16:00:52 UNRAID kernel: ACPI Error: Method parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT (20170728/psparse-550)
Jan 13 16:00:52 UNRAID kernel: ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20170728/power_meter-338)
Jan 13 16:00:53 UNRAID kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20170728/exfield-427)
Jan 13 16:00:53 UNRAID kernel: ACPI Error: Method parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT (20170728/psparse-550)
Jan 13 16:00:53 UNRAID kernel: ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20170728/power_meter-338)

....

Jan 13 16:01:01 UNRAID kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20170728/exfield-427)
Jan 13 16:01:01 UNRAID kernel: ACPI Error: Method parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT (20170728/psparse-550)
Jan 13 16:01:01 UNRAID kernel: ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20170728/power_meter-338)
Jan 13 16:01:02 UNRAID kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20170728/exfield-427)
Jan 13 16:01:02 UNRAID kernel: ACPI Error: Method parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT (20170728/psparse-550)
Jan 13 16:01:02 UNRAID kernel: ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20170728/power_meter-338)


I've tried the suggested fix of modifying the /etc/sensors3.conf file and added the following lines at the bottom of the file:
 

Quote

chip "power_meter-*"

    ignore power1

That however does nothing. I also tried to make a new file under /etc/sensors.d/sensor_quirk with the lines above added but that also does nothing.
Does anyone know how to fix this?

Link to comment
13 hours ago, peeqi said:

Hi, i have a microserver gen 8 and i'm getting absolutely hammered with the ACPI errors.
 


I've tried the suggested fix of modifying the /etc/sensors3.conf file and added the following lines at the bottom of the file:
 

That however does nothing. I also tried to make a new file under /etc/sensors.d/sensor_quirk with the lines above added but that also does nothing.
Does anyone know how to fix this?

 

Hi, I also have a Microserver Gen 8 and was having the same ACPI Error message spamming issue as well. 

 

I found THIS POST and went ahead and unload the "acpi_power_meter" module from the kernel and it solved the problem.  You can add this line to the /boot/config/go file found inside the flash drive hosting unRAID system to have it unload the module during every boot:

rmmod acpi_power_meter

Hope this helps!

 

 

p.s. I found another article from Novell which also sort of explains this situation.

 

Edited by perfecblue
  • Like 1
Link to comment
On 8/13/2017 at 2:57 AM, IrishBiker said:

Hi

 

My current setup has the following:

 

IOMMU group 1
    
[8086:0151] 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
[10de:104a] 07:00.0 VGA compatible controller: NVIDIA Corporation GF119 [GeForce GT 610] (rev a1)
[10de:0e08] 07:00.1 Audio device: NVIDIA Corporation GF119 HDMI Audio Controller (rev a1)

 

So using Step 2.a I added the following line to my syslinux.cfg file

 

append pcie_acs_override=id:10de:104a,10de:0e08 initrd=/bzroot

 

But when I rebooted the box its still the same, any ideas? Oh I also have Enable PCIe ACS Override  switched on.

 

 

 

add this instead append pcie_acs_override=8086:0151

 

it will separate PCI bridge and leave the other two in their own group

Link to comment
  • 3 weeks later...
On 12/02/2018 at 1:46 AM, D874as said:

thanks @1812 seems like I need to compile my own kernel. Maybe it also works for a custom made unRaid kernel? First of all, I need to educate myself about kernel building xD

 

I have a Microserver Gen8 too and want to passthrough my GPU to my VM.

 

I ended up compiling my own kernel for UnRAID with the RMRR check patched out, as what the Proxmox guys did. Attached is kernel for "Linux 4.14.16-unRAID." Copy 'bzimage-new' into '/boot' and modify your syslinux to use the new kernel. I think this kernel will only work on the specific UnRAID version I'm running right now (v6.4.1).

 

My config as follows (I didn't need to do PCS override or 'vfio_iommu_type1.allow_unsafe_interrupts=1' for me):

label unRAID OS (PCIe passthrough - No RMRR chk)
  menu default
  kernel /bzimage-new
  append isolcpus=2,3,6,7 initrd=/bzroot

The ACS override separated my GPU from the PCI bridge (IOMMU group 1) into its own IOMMU group 11.

IOMMU group 0:	[8086:0108] 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 Processor Family DRAM Controller (rev 09)
IOMMU group 1:	[8086:0101] 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
...
[snip]
...
IOMMU group 11:	[10de:104a] 07:00.0 VGA compatible controller: NVIDIA Corporation GF119 [GeForce GT 610] (rev a1)
[10de:0e08] 07:00.1 Audio device: NVIDIA Corporation GF119 HDMI Audio Controller (rev a1)
...
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>

I then installed the NVIDIA drivers in my VM... though after a restart Windows says the driver for my GPU can't be loaded because of a problem. I'll need to look into this some more, but the custom kernel seems to bypass the "Device is ineligible for IOMMU domain attach due to platform RMRR requirement".

 

If you want to compile your own next time (a.k.a. every time UnRAID gets updated, I think), I used CHBMB's kernel compile script here and the Proxmox kernel patch here. I've also attached my modified script and patch. I compile on the UnRAID machine itself. My modified script just copies the Proxmox patch to '/usr/src/linux-*' and comments out fetching of CHBMB's DVB '.config' on line 44. Thanks CHBMB!

 

bzimage-new

kernel-compile-module.sh

remove_rmrr_check.patch

 

Update 14/03/18: After overcoming the RMRR hurdle, I found out the GPU driver could not be loaded (code 43) because of NVIDIA deciding to intentionally block the driver if it detects it's being loaded in a VM to get you to buy their workstation Quattro cards. This is widely documented and can be bypassed with:

  <features>
    ...
    <hyperv>
      ...
      <vendor_id state='on' value='blahblahblah'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>

However, this still didn't work; I still got the code 43 problem. After more googling and dumping my GPU BIOS and whatnot, I think it's because my GPU BIOS doesn't have EFI support. When I substituted my GPU BIOS in the VM config with a similar variant with UEFI support (I couldn't find one for my exact model), the driver loaded correctly and I got output through the HDMI! This was great until my whole host system crashed, probably due to using an incompatible GPU BIOS.

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <rom file='/mnt/user/vm-iso/Gigabyte.GT610.1024.130107-UEFI.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>

Update 16/03/18: I got my Gigabyte GT 610 Silent passthrough partially working. As suspected, the GPU BIOS did not contain UEFI support. I dumped my vBIOS (I did set the GPU to "secondary optional" in the HP RBSU) and confirmed the ROM was not corrupt (hopefully—the md5 happened to match Techpowerup's database and rom-parser didn't complain). I then used the GOPUpd tool to inject the UEFI bit of firmware into my BIOS image (the GOP was included in the tool for supported GPUs—luckily mine, a GF119, was. With the new ROM file, I specified it in my VM config, as above and the code 43 error was gone. Everything appears to work, but when the VM is rebooted, the whole host crashes. I wanted to actually try and flash my patched vBIOS to my GPU, but the EEPROM size on the GPU appears to be too small to fit the patched vBIOS.

 

I did come across a QVM bug which has similar symptoms to mine, except mine crashes regardless of how long I have the VM running. I am running a Windows Server 2016 host. I did try enabling MSI to see if it will solve my host crashing problem to no avail...

 

Update 25/03/18: I believe my GPU is inherently not compatible because of it lacking the feature to reset properly. This in turn causes my host to crash and reboot if a GPU is detached from the VM and re-attached again.

 

Update 10/04/18: Late update, but I tried using SeaBIOS on Ubuntu with the GT610 but there was some unrelated bug and I got a login loop with the Linux Nvidia drivers installed. In the end, I swapped the GPU with a GT520 (not fanless unfortunately), followed all the same steps above with dumping the vBIOS, and it no longer crashes the host under UEFI Windows Server 2016.

Edited by kabloomy
  • Like 1
Link to comment
56 minutes ago, kabloomy said:

 

I have a Microserver Gen8 too and want to passthrough my GPU to my VM.

 

I ended up compiling my own kernel for UnRAID with the RMRR check patched out, as what the Proxmox guys did. Attached is kernel for "Linux 4.14.16-unRAID." Copy 'bzimage-new' into '/boot' and modify your syslinux to use the new kernel. I think this kernel will only work on the specific UnRAID version I'm running right now (v6.4.1).

 

My config as follows:


label unRAID OS (PCIe passthrough - No RMRR chk)
  menu default
  kernel /bzimage-new
  append isolcpus=2,3,6,7 pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot

The ACS override separated my GPU from the PCI bridge (IOMMU group 1) into its own IOMMU group 11.


IOMMU group 0:	[8086:0108] 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 Processor Family DRAM Controller (rev 09)
IOMMU group 1:	[8086:0101] 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
...
[snip]
...
IOMMU group 11:	[10de:104a] 07:00.0 VGA compatible controller: NVIDIA Corporation GF119 [GeForce GT 610] (rev a1)
[10de:0e08] 07:00.1 Audio device: NVIDIA Corporation GF119 HDMI Audio Controller (rev a1)
...

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>

I then installed the NVIDIA drivers in my VM... though after a restart Windows says the driver for my GPU can't be loaded because of a problem. I'll need to look into this some more, but the custom kernel seems to bypass the "Device is ineligible for IOMMU domain attach due to platform RMRR requirement".

 

If you want to compile your own next time (a.k.a. every time UnRAID gets updated, I think), I used CHBMB's kernel compile script here and the Proxmox kernel patch here. I've also attached my modified script and patch. I compile on the UnRAID machine itself. My modified script just copies the Proxmox patch to '/usr/src/linux-*' and comments out fetching of CHBMB's DVB '.config' on line 44. Thanks CHBMB!

bzimage-new

kernel-compile-module.sh

remove_rmrr_check.patch

 

 

This is very cool! If the vm is using the GPU, then regardless of drivers loading or not, that means it's allowing passthrough that was blocked due to RMRR requirement. But it doesn't mean it's not necessarily playing a part of the driver issue I think....

 

Hey @limetech, any chance we can have an option (like the acs override selection) to enable the patch via web gui?  Or is it going to be too much of a pita to maintain every time a new version of unraid comes out? Would also be nice to add the "allow unsafe interrupts" as well that way.

Link to comment
On 2/10/2018 at 7:24 PM, burningstarIV said:

From reading back and forth on all this stuff it seems the micro servers have a bunch of problems. On my 580 I had to get an older 2012 firmware to get around the rmrr. Try to look into your bios and see if running the latest version. From there you may be able to change the network over to non shared memory to get past this. Not sure what the success rate is but its a problem with how they use the intel cpus 

Any chance you could link the firmware for the 580 you used? Im trying to get mine up and running and it has a newer one. Thanks!

Link to comment

hey - yeah it took me probably 10 hours of searching sites, directories and old ftps to get this little thing . You will need a hp proliant ssp installed to usb to inject it. Use the hp usb tool to install spp to a usb - and then navigate to the packages directory ( if I remember all right) and put this file in there. Boot sever from it, and use interactive. Then force select the older bios to install. After that you should be clear of RMRR issues. 

 

 

CP016412.scexe

Link to comment
  • 2 weeks later...

Hi guys, great topic and good reading.  This is all pretty new to me,  but unRAID seems to run like a champ.  I have several HP Proliant series servers and this is all very helpful.  I'm hoping to not to deter the topic that's going on here, but I think it does follow the original topic of "to do's" with Proliant servers.

 

I currently have a DL360 G7 with a PCI Creative SB Audigy FX PCIe 5.1.  I cannot get this card to show in the VM assignment PCI device list, but that's ok.  I'd really like to use this card as a "Sound Card" option, which now shows as a check-off option since installing it.  That tells me that unRAID sees enough of it to acknowledge it.  When I check the SC box, unRAID will not allow me to boot the VM due to a conflict.  I'm currently not running any modified scripts or add-ons/modules.  The system is a currently running VM's with zero issues, Win 2012 Term Server, Zentyal, & 2x Windows 10 Pro.  I'd like to add sound support to the 2 Windows 10 Pro VM's, but so far no luck with anything.  I could add video cards with audio for each VM, but that's not particularly practical seeing all I need is audio and not graphic accelerated workstations and I only have 2 PCI slots in total.  I've searched and searched for answers, but keep coming across the answer of people need to assign the sound card and the corresponding video card to the same VM, which I understand, but it doesn't apply to my situation.

 

I'll be happy to jump to another post or create a new one, but this seemed most relevant.  I'll upload exact verbiage of errors and config data, I'm just not sure of the level of the issue here.  Not sure if this is a simple adjustment that I'm overlooking or if need to write a custom script for this situation.  Basically, I'm not new to VM environments or systems, but I am new to unRAID and it's ability to do so much with so little.  Thank you for any help and efforts.

Link to comment
2 hours ago, ReidS said:

I'll be happy to jump to another post or create a new one, but this seemed most relevant.  I'll upload exact verbiage of errors and config data, I'm just not sure of the level of the issue here.  Not sure if this is a simple adjustment that I'm overlooking or if need to write a custom script for this situation.  Basically, I'm not new to VM environments or systems, but I am new to unRAID and it's ability to do so much with so little.  Thank you for any help and efforts.

 

This is the best place to do it, so that if anyone else has the same problem on a proliant, then they can find the answer here. 

 

Let us know the error and upload your full diagnostics zip as well. (tools>diagnostics)

Link to comment

The attached image is what I'm seeing when I launch the Windows 10 system with the Sound Card option enabled for this VM.  I tried a few different BIOS options in unRAID, but didn't see to have any further luck.  Some time back I saw someone talk about making a change to the system config file and I tried adding a line (I forget what line exactly now) to unlock a feature, but this resulted in preventing the system from booting in general.  I know that little info doesn't help, but I'm a one man IT team doing lots of work and I'm not always quick to make notes on negative results just in case I might have did something wrong.  I like to leave my doors open to re-explore failed attempts.  There's way more to be learned from failures then successes.

diagnostics-20180406-1252.zip

unraid SC error.jpg

Link to comment
1 hour ago, ReidS said:

The attached image is what I'm seeing when I launch the Windows 10 system with the Sound Card option enabled for this VM.  I tried a few different BIOS options in unRAID, but didn't see to have any further luck.  Some time back I saw someone talk about making a change to the system config file and I tried adding a line (I forget what line exactly now) to unlock a feature, but this resulted in preventing the system from booting in general.  I know that little info doesn't help, but I'm a one man IT team doing lots of work and I'm not always quick to make notes on negative results just in case I might have did something wrong.  I like to leave my doors open to re-explore failed attempts.  There's way more to be learned from failures then successes.

diagnostics-20180406-1252.zip

unraid SC error.jpg

 

You need to start with VM passthrough problems on the previous page dealing with unsafe interrupts. Without allowing unsafe interrupts (append vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot) , you will not get any closer to your goal of a working vm.

Link to comment
  • 2 weeks later...

Sorry for my delay is response to this.  I had to prioritize some other things over this project last week.

 

So I added the line above to allow unsafe interrupts, reselected the audio config for the Win10 system and I'm getting the same exact error above still.  I went into the BIOS and tried to manually reassign the audio card to another IRQ and memory address, but the system keeps moving the devices on the same interrupts along with it, like the RAID controller.  I read earlier that the RAID controller doesn't play well and am I possible a victim of circumstances here?  I have 6 disks in this system running off the RAID controller, but not using the hardware RAID.  So using the onboard SATA won't work with only 2 ports.  I also have an PCIe external SATA for an external array, but the array isn't plugged in at the moment and it's just a PNP, no BIOS card.  I'm just tossing out some thoughts.  I'm going to be trying to get this box working 100% by the end of week So I can start the next leg of projects related to this system.  If I need to order another audio card, I'm open to suggestions.

 

Thank you and everybody in advance for any help!

Edited by ReidS
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.