Pass-through 3 ports of a quad port nic?

[choppyesq]

I got an i350 for my server and would like to pass-through 3 of the 4 ports, rather than all 4.

 

They are all seperated by IOMMU groups and I can easily pass the entire card with "vfio-pci.ids=8086:1521" in my syslinux config, but is there a way to take back one of the ports for unraid to use?

 

IOMMU group 8:	[8086:1521] 2e:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 9:	[8086:1521] 2e:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 10:	[8086:1521] 2e:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 11:	[8086:1521] 2e:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

[saarg]

You can use xen-pciback.hide instead of vfio-pci.ids. Then you use the PCI number instead. This way you have to add the cards to the xml manually as they don't show up in the other devices list. 

 

So it should look like this:

xen-pciback.hide=(2e:00.1) (2e:00.2)(2e:00.3)

[choppyesq]

Thanks, I tried that and it didn't show in other devices, so I thought it didn't work.

 

I will add the xml manually.

Edited July 24, 2018 by choppyesq

[choppyesq]

I tried this method and it didn't passthrough, and now I can't passthrough the whole card using vfio-pci.ids.

 

I just saw this post: 

did a recent update break passthrough?

[saarg]

11 hours ago, choppyesq said:

I tried this method and it didn't passthrough, and now I can't passthrough the whole card using vfio-pci.ids.

 

I just saw this post: 

did a recent update break passthrough?

 

Please describe what's happening a little bit more detailed than it doesn't work. Impossible to help you without any info. 

Pass through is not broken or else you would have seen a lot of posts about it. The thread you linked to is a user upgrading from a very old version to latest and some things have changed since. 

 

Exactly what did you add to your xml (post the whole file also) and what is the error message? 

[choppyesq]

Sorry, I will try to be more clear.

 

I modified my syslinux config to use the xen-pciback.hide command.  When I rebooted my server, all the interfaces were showing up in unraid for the quad port nic in the networking settings of unraid.

 

I then removed the xen-pciback.hide command from the syslinux config and reinserted the vfio-pci.ids command and rebooted and the interfaces were still available for unraid to use.

 

I looked through my system log and could not see any error message related to the commands.  I am at work now, but can post the whole diagnostic zip when I get home if it helps.

 

I am really perplexed cause I was able to pass the entire card previously and now it won't. 

 

edit: I attached the diagnostics.

fattony-diagnostics-20180724-2103.zip

Edited July 25, 2018 by choppyesq
attachment

[saarg]

I don't know what you edit the Syslinux.cfg file with, but you have some weird characters in the vfio-pci.ids part. 

Use the webgui to edit the syslinux.cfg file (click on flash on main page and scroll down to the editor) and do not paste it, but write it manually. 

The below is what you have now. 

BOOT_IMAGE=/bzimage vfio-pci.ids=8086:1521 initrd=/bzroot pcie_aspm=off

 

If unraid uses the card, it's always something that is typed wrong. 

[choppyesq]

I use the webgui to edit it, and that garbage didn't show up in it.  I manually typed it instead of copying and pasting and that fixed it though.  Thanks!

 

I think I am just going to stick with passing through all 4 ports, so far I haven't had an issue with unraid itself using the realtek ethernet on the mobo, and i am want to go ahead and get pfsense going.

 

Thanks again!

[snailbrain]

I'm really sorry for bumping this old thread... but this one came up and it's related.

I have a quad intel GB card.
I have stubbed it and i can see all 4 ports in vm configs to passthrough (they look like individual cards:

 

Passing through all 4 ports to pfsense works great and it's been working like that for a while.

 

I'm fine with the motherboard lan for unraid - but, i want to only use 2 ports for pfsense (lan/wan). Passing through all 4 ports works perfect - however i want to use 1 port for a windows VM.
If I try to pass through just 2 ports, it doesn't work and pfsense locks up on bootup or during config.
Passing through one port to windows does seem to work.

 

Do you think:

xen-pciback.hide=

 

is the way forward or is there some issue with what i'm trying to do.

 

hope you can help @saarg

[saarg]

17 hours ago, snailbrain said:

I'm really sorry for bumping this old thread... but this one came up and it's related.

I have a quad intel GB card.
I have stubbed it and i can see all 4 ports in vm configs to passthrough (they look like individual cards:

 

Passing through all 4 ports to pfsense works great and it's been working like that for a while.

 

I'm fine with the motherboard lan for unraid - but, i want to only use 2 ports for pfsense (lan/wan). Passing through all 4 ports works perfect - however i want to use 1 port for a windows VM.
If I try to pass through just 2 ports, it doesn't work and pfsense locks up on bootup or during config.
Passing through one port to windows does seem to work.

 

Do you think:

xen-pciback.hide=

 

is the way forward or is there some issue with what i'm trying to do.

 

hope you can help @saarg

How does your iommu groups look like? 

Which 2 are you passing through to pfsense and which one to the other VM? 

You have already stubbed the card I guess, since you have it available in other PCI devices, so it won't make any difference to use xen-pciback.hide. 

[snailbrain]

4 hours ago, saarg said:

How does your iommu groups look like? 

Which 2 are you passing through to pfsense and which one to the other VM? 

You have already stubbed the card I guess, since you have it available in other PCI devices, so it won't make any difference to use xen-pciback.hide. 

 

thanks for the response

(using acs) - when not using acs, i think they were in pairs

IOMMU group 29:	[8086:10e8] 1d:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
IOMMU group 30:	[8086:10e8] 1d:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
IOMMU group 31:	[8086:10e8] 1e:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
IOMMU group 32:	[8086:10e8] 1e:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

looks like this - but pfsense only likes all 4 going to it.

I've tried the first 2, and also the first and last.
installs fine.. but then seems to freezeup
as soon as you bootup again it sticks on "igb0 is up" or words to that effect

same thing with my working pfsense of which all 4 are passed through - if i change to pass through only 2 ports, it will freeze on bootup too.

 

 

Edited March 2, 2019 by snailbrain

[saarg]

11 hours ago, snailbrain said:

 

thanks for the response

(using acs) - when not using acs, i think they were in pairs

IOMMU group 29:	[8086:10e8] 1d:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
IOMMU group 30:	[8086:10e8] 1d:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
IOMMU group 31:	[8086:10e8] 1e:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
IOMMU group 32:	[8086:10e8] 1e:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

looks like this - but pfsense only likes all 4 going to it.

I've tried the first 2, and also the first and last.
installs fine.. but then seems to freezeup
as soon as you bootup again it sticks on "igb0 is up" or words to that effect

same thing with my working pfsense of which all 4 are passed through - if i change to pass through only 2 ports, it will freeze on bootup too.

 

 

Have you tried installing from scratch with only the 2 first or 2 last passed through? 

[snailbrain]

7 hours ago, saarg said:

Have you tried installing from scratch with only the 2 first or 2 last passed through? 

yeh i did.

it gets through the installation, but locks up, and if you reboot it locks up on "igb1 is coming up or whatever"

it's as if it knows it's the quad card and is looking for the rest of the ports (psychic guess)

 

[alitech]

Sorry for yet again bumping this old thread, I have a slightly different issue. 

My quad port card appears in an iommu that is shared by a lot of things on the motherboard. Its group zero. When I use the vfio command, I am able to see the ports available for VMs however I cannot use them as they conflict with the rest of the iommu group. I have tried ACS etc etc 

 

I just need to know how I can use the 4 ports exclusively for VMs. 

 

Here is what my group 0 looks like. I have highlighted the 4 ports I want to isolate and use just for VMs.

 

IOMMU group 0:[1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

[1022:1483] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

[1022:1483] 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

[144d:a808] 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983

[1022:43d0] 03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43d0 (rev 01)

[1022:43c8] 03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01)

[1022:43c6] 03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge (rev 01)

[1022:43c7] 20:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

[1022:43c7] 20:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

[1022:43c7] 20:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

[1022:43c7] 20:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

[1022:43c7] 20:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

[1022:43c7] 20:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

[10ec:8168] 22:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

[12d8:2304] 23:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)

[12d8:2304] 24:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)

[12d8:2304] 24:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)

[8086:10c9] 25:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

[8086:10c9] 25:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

[8086:10c9] 26:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

[8086:10c9] 26:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

[15b3:1003] 28:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

[1b21:1242] 29:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller

 

Any help much appreciated.