same PCI ID in multiple IOMMU groups [Solved]


Recommended Posts

I don't know if this issue has been asked before, but I did not seem to find anything on either unraid forum nor google.

Here goes:

I have 4 NICs in 4 IOMMU groups, BUT they all have the same ID [8086:1521]???

IOMMU group 17: [8086:1521] 06:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
 
IOMMU group 18: [8086:1521] 06:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
 
IOMMU group 19: [8086:1521] 06:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
 
IOMMU group 20: [8086:1521] 06:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

 

How do I reserve 2 for VMs? I normally use the vfio-pci.ids in the syslinux config to reserve resources for VMs.

 

Any help is much appreciated.

 

/Alphahelix

 

***********************************************

The summary of this thread:

In order to passthrough PCIe devices that have the same PCIe ID, but are located in different IOMMU groups, Saarg was kind to provide us with a solution. I have tried to make a step by step guide here:

 

  • Find the device(s) you need passed through to a VM. (Tools -> System Devices)
  • Note the device(s) PCI-number you want to be passed through. (in my case 06:00.2 & 06:00.3)
  • In the syslinux config file between "append" and "initrd=/bzroot"  add "xen-pciback.hide=(PCI-number)"  (in my case 06:00.2 & 06:00.3)
append xen-pciback.hide=(06:00.2)(06:00.3)initrd=/bzroot
  • In the VM you want to use your selected PCIe device(s), edit the XML file and add: (in this example I have added the PCI device with PCI-number: 06:00.2)
<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x2'/>
      </source>
    </hostdev>
  • A restart is needed to activate the changes made in syslinux config file.

 

/Alphahelix

 

Edited by Alphahelix
  • Like 1
Link to comment
27 minutes ago, 1812 said:

you use vfio in syslinux one time (the single id) and it covers them all. then pass through which one you want by address (the 06 number)

But if i pass all through, what will unraid use? (They are onboard)

 

My goal is to pass 2 through to VMs and 2 for unraid. Sorry I should have been more clear. My bad.

 

/Alphahelix

Link to comment
1 hour ago, Alphahelix said:

But if i pass all through, what will unraid use? (They are onboard)

 

My goal is to pass 2 through to VMs and 2 for unraid. Sorry I should have been more clear. My bad.

 

/Alphahelix

 

You use xen-pciback.hide and instead of the ID use the PCI number. The downside is that you have to add the devices to the xml manually. 

 

So should look like this:

xen-pciback.hide=(06:00.2)(06:00.3)

Link to comment
2 hours ago, Alphahelix said:

But if i pass all through, what will unraid use? (They are onboard)

 

My goal is to pass 2 through to VMs and 2 for unraid. Sorry I should have been more clear. My bad.

 

/Alphahelix

 

Didn't realize it was your only ones.

 

Try the advice from saarg above. I've never done it, but if he says it should work, then it should work.

Link to comment
14 hours ago, saarg said:

 

You use xen-pciback.hide and instead of the ID use the PCI number. The downside is that you have to add the devices to the xml manually. 

 

So should look like this:

xen-pciback.hide=(06:00.2)(06:00.3)

 

First thank you Saarg!

 

So to get it straight, I:

* do not use: vfio-pci.ids=8086:1521 i syslinux config file

* do use: xen-pciback.hide=(06:00.2)(06:00.3) still in the syslinux config file

* in the XML file I add this for each port: (tried my best to google a solution)

<hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

But where is: domain / bus / slot / function is resolved from?

 

I am no Linux expert, but I am always ready to learn. :)

 

/Alphahelix

Link to comment
59 minutes ago, Alphahelix said:

 

First thank you Saarg!

 

So to get it straight, I:

* do not use: vfio-pci.ids=8086:1521 i syslinux config file

* do use: xen-pciback.hide=(06:00.2)(06:00.3) still in the syslinux config file

* in the XML file I add this for each port: (tried my best to google a solution)


<hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

But where is: domain / bus / slot / function is resolved from?

 

I am no Linux expert, but I am always ready to learn. :)

 

/Alphahelix

 

You are correct about the syslinux.cfg.

For the xml, the below is the correct form.

Bus is the first two numbers (06), slot is the numbers after : (00) and function is the last number (2). Below is the example for 06:00.2.

Add this just above the     <memballoon model='virtio'> line.

 

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x2'/>
      </source>
    </hostdev>

 

Link to comment
  • 2 weeks later...
17 hours ago, WingsGB said:

20180810_173203.jpg

 

I am having the same issue, I have tried the above suggested method but upon reboot after adding xen-pciback.hide=(01:00.0)(01:00.1)(01:00.2)(01:00.3), my network ports are still showing in unraids network settings? the command doesn't seem to do anything.

 

 

Then you have done something wrong. Post your syslinux.cfg here. 

 

Edit: Looks like you didn't read the first number correctly. It's 81, not 01.

Edited by saarg
Link to comment
4 hours ago, saarg said:

 

Then you have done something wrong. Post your syslinux.cfg here. 

 

Edit: Looks like you didn't read the first number correctly. It's 81, not 01.

 

 

IOMMU 



IOMMU group 17:	[8086:1521] 03:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 18:	[8086:1521] 03:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 19:	[8086:1521] 03:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 20:	[8086:1521] 03:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 21:	[8086:1521] 07:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 22:	[8086:1521] 07:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

 

 

syslinux.cfg

default menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  kernel /bzimage
  append initrd=/bzroot
label unRAID OS GUI Mode
  menu default
  kernel /bzimage
  append xen-pciback.hide=(03:00.0)(03:00.1)(03:00.2)(03:00.3)initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label unRAID OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest

I have since tried the card in different slots.

Edited by WingsGB
Link to comment
2 hours ago, WingsGB said:

 

 

IOMMU 


IOMMU group 17:	[8086:1521] 03:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 18:	[8086:1521] 03:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 19:	[8086:1521] 03:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 20:	[8086:1521] 03:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 21:	[8086:1521] 07:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 22:	[8086:1521] 07:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

 

 

syslinux.cfg


default menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  kernel /bzimage
  append initrd=/bzroot
label unRAID OS GUI Mode
  menu default
  kernel /bzimage
  append xen-pciback.hide=(03:00.0)(03:00.1)(03:00.2)(03:00.3)initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label unRAID OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest

I have since tried the card in different slots.

 

You need a space after the last bracket.

Link to comment
9 hours ago, WingsGB said:

 

I have tried with and without a space at the end, it makes no difference.

 

Any other suggestions.

 

If everything is on one line and there is a space after the last bracket, it should not show up in unraids network settings. 

It will show up in the list of all devices on unraid though. 

You need to post your diagnostics. 

Link to comment
34 minutes ago, saarg said:

Looks like pciback is not grabbing the card. Can you try to add it to the non gui mode and see if it works there?

It's working here, but I'm on 6.5.2 still.

 

I moved the "pciback" into "Label unRAID OS" and booted up in none gui.

Made no change?

Link to comment

I tested on 6.5.3 and it works as expected here. I have no idea what is going on on your side, but there is something strange going on.

You shouldn't get anything in the other devices list using pciback. Or was that screenshot taken after trying to pass it through to a VM?

 

Link to comment
17 hours ago, saarg said:

I tested on 6.5.3 and it works as expected here. I have no idea what is going on on your side, but there is something strange going on.

You shouldn't get anything in the other devices list using pciback. Or was that screenshot taken after trying to pass it through to a VM?

 

 

Yes the screenshot was taken after i added the <hostdev> to the xml, i didnt do this untill after editing syslinux and rebooting. I have not tried running the VM yet as the ports are showing in the "network settings" so decided not to go any further

 

 

Link to comment
2 hours ago, WingsGB said:

 

Yes the screenshot was taken after i added the <hostdev> to the xml, i didnt do this untill after editing syslinux and rebooting. I have not tried running the VM yet as the ports are showing in the "network settings" so decided not to go any further

 

 

 

How do you edit the syslinux.cfg? 

Link to comment
36 minutes ago, WingsGB said:

 

I edit it via the webui but after you saying that i thought i would open up the actual file on the flash drive, /boot/syslinux/syslinux.cfg

the changes i had made was in the file.

 

I was thinking that there might have been some weird characters in the file. I have seen others had that problem lately.

Maybe try to edit the file in a linux file editor (or notepad++ on windows) and write it again. Do not copy/paste it. 

 

It looks like the xen-pciback.hide= is not recognized from the boot command by the kernel, as it loads the igb module instead of pciback.

Link to comment
17 hours ago, saarg said:

 

I was thinking that there might have been some weird characters in the file. I have seen others had that problem lately.

Maybe try to edit the file in a linux file editor (or notepad++ on windows) and write it again. Do not copy/paste it. 

 

It looks like the xen-pciback.hide= is not recognized from the boot command by the kernel, as it loads the igb module instead of pciback.

 

? I opened up the file in different editors first to compare, i noticed some margin alignment being out but thats it. To be safe though i copied the default cfg- and started again without copy/ paste.

rebooted and same issue?

is there anything else i could try disabling with the command to test it?

Link to comment
  • 5 months later...
  • 4 weeks later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.