Hi guys/gals, it's my first time here, and I've run into some issues with my first unRAID build.

 

Heres the hardware configuration:

 

Motherboard: EVGA SR-X (Intel C606/X79 Platform)

CPU's: Intel E5-2687W (v1) x2

RAM: Kingston DDR3 8GB *12 (96GB Total)

SSD: Kingston 480GB + Intel 512GB

HDD Mass Storage: 2TB

GPU's: GTX 650 — PCI-E Port 1 (For Unraid GUI/Console)

GPU's: GTX1070 x2 — PCI-E Ports 3+5 (One per VM)

I'm new to virtual machines, and I'm trying to make my own version of Linus's <2 gamers 1 CPU> using old hardware. After rolling back not 1/2, but 3 BIOS versions, I finally got IOMMU and HVM enabled (EVGA took away the VT-d setting for no reason).

 

However, the problem arises with both GTX1070's being in the same group (Group 3), even with ACS Override enabled. 

IOMMU group 0:	[8086:3c00] 00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI2 (rev 05)
IOMMU group 1:	[8086:3c02] 00:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a (rev 05)
IOMMU group 2:	[8086:3c04] 00:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 05)
IOMMU group 3:	[8086:3c08] 00:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode (rev 05)
[10b5:8747] 05:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ba)
[10b5:8747] 06:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ba)
[10b5:8747] 06:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ba)
[10de:1b81] 07:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
[10de:10f0] 07:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
[10de:1b81] 08:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
[10de:10f0] 08:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)

Starting the second VM gives an error saying that one of the GPU's is being used by the other VM.

 

I have tried jonp's Guide on Passing-Through NIC's, but to no avail — I've tried the following syslinux.cfg's (I honestly have no idea what I'm doing here):

default menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
  append pcie_acs_override=downstream pci-stub.ids=10de:1b81 pci-stub.ids=10de:10f0 initrd=/bzroot
label unRAID OS GUI Mode
  kernel /bzimage
  append pcie_acs_override=downstream pci-stub.ids=10de:1b81 pci-stub.ids=10de:10f0 initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append pcie_acs_override=downstream initrd=/bzroot unraidsafemode
label unRAID OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append pcie_acs_override=downstream initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest

default menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
  append pcie_acs_override=downstream pci-stub.ids=10de:1b81 initrd=/bzroot
label unRAID OS GUI Mode
  kernel /bzimage
  append pcie_acs_override=downstream pci-stub.ids=10de:1b81 initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append pcie_acs_override=downstream initrd=/bzroot unraidsafemode
label unRAID OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append pcie_acs_override=downstream initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest

(The second one has the GPU soundcard section (pci-stub.ides=10de:10f0) removed.)

 

However, neither worked. At this point, I don't know what to do. If unRAID is able to change IOMMU groups using "ACS Override," is it possible to manually assign groups? Any help would be appreciated,

 

Thanks!

Have you tried acs downstream, multifunction?

Sent from my SM-G955U using Tapatalk

1 hour ago, david279 said:

Have you tried acs downstream, multifunction?

Sent from my SM-G955U using Tapatalk
 

Nope. If I may, how would I implement that?

 

Thanks!

1 minute ago, hihihi8 said:

Nope. If I may, how would I implement that?

 

Thanks!

 

pcie_acs_override=downstream,multifunction add that to the syslinux config. Click on your flash drive to get to it and edit the config.

8 hours ago, david279 said:

pcie_acs_override=downstream,multifunction add that to the syslinux config. Click on your flash drive to get to it and edit the config.

Hi, just tried that, didn't work

try: pcie_acs_override=id:10de:1b81,10de:10f0

5 hours ago, 1812 said:

try: pcie_acs_override=id:10de:1b81,10de:10f0

Hi, just tried that, still doesn't work

Since the motherboard uses a PLX chip, I've also tried passing through the PLX Chip ID in syslinux.cfg, but it hasn't had any noticeable effects.

 

One thing probably worth noting is that the problem likely has to do with how the PLX chip inherently works:

The SR-X's poor design means that one CPU's entire 40 PCI-E Lanes go to waste, and the rest are split/fabricated by a PEX8747 PLX chip, but isn't done perfectly either. According to mosie in his rant on the EVGA forums:

The PLX chip controls both GTX1070s' PCI-E slots (slots 3+5), and is probably why they are grouped together with the PLX chip.

My current plan is to swap the top card (currently the dual-slot GTX650) with a cheap single slot 9500GT, and then move GTX1070#1 up to slot 2, which is linked to the CPU directly. I'll see how that works out in a few days when the single-slot card arrives.

 

If that doesn't work, I'll resort to getting rid of the GTX650, moving GTX1070#1 up to slot one, and use it as the console card, before dumping the ROM and passing it through to one of the VM's. Something I've read/watched, but haven't fully understood or attempted yet.

 

Cheers and thanks!

41 minutes ago, hihihi8 said:

The PLX chip controls both GTX1070s' PCI-E slots (slots 3+5), and is probably why they are grouped together with the PLX chip.

1

Just noticed something... Is that showing up as 3 devices? It seems as if the Intel CPU's PCI bridge is grouped with the PLX chip (as it is sometimes, when grouped with a GPU), and then the PLX chip itself has 3 lines:

#1: Grouped with PCI Bridge

#2: Grouped with GTX1070#1

#3: Grouped with GTX1070#2

 

So essentially, the PLX chip is acting as a second PCI bridge?? Just my observations, to be taken with a grain of salt.

 

However, hopefully that's helpful!

Edited March 25, 20188 yr by hihihi8

Bump

man , check over here - was sorting the same , need to add "multifunction" into flash config file, then u get more IOMMU groups

label unRAID OS-X
  kernel /bzimage
  append pcie_acs_override=downstream,multifunction initrd=/bzroot

 

On 5/1/2018 at 4:37 PM, killeriq said:

man , check over here - was sorting the same , need to add "multifunction" into flash config file, then u get more IOMMU groups

label unRAID OS-X
  kernel /bzimage
  append pcie_acs_override=downstream,multifunction initrd=/bzroot

 

Hi 

I have 2 nvidia card with same iommu group and I applied  acs override as you suggested. It works , Now cards listed in separate groups but when I try to run vm’s one starts normally but 2nd  gets error saying can’t allocate memory and doesn’t start.

I have 16 gb ram and assigned 4gb each vm so there is enough ram.

any idea ?

if u add 2 cards into 1x vm it works? 

 

Im running:

1x VM linux - 2x 1080ti - like 2gb ram added

1x VM Win10 - 1x r7 290 - 16gb ram

 

 

I succeded to run 2 VM with Win10. (each VM has gtx 1060)

Problem was in my memory slots.  unplug/plug solved.

Thanks.