IOMMU group splitting for PCIe passthrough - no success, please help!

MisterSwiffer

By MisterSwiffer
in VM Engine (KVM)

Recommended Posts

MisterSwiffer Noob

MisterSwiffer

Posted
Posted (edited)

hi all,

 

so far I am using unRAID since more than a year and I am very happy! But now I am facing the first problem that I am not able to solve by myself.

 

My board is an Asrock J3455 with one PCIe slot. I added a dual NIC on this slot that I would like to pass to a VM. But the dual NIC is inside a big IOMMU group:

  

IOMMU group 0:	[8086:5af0] 00:00.0 Host bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Host Bridge (rev 0b)
IOMMU group 1:	[8086:5a85] 00:02.0 VGA compatible controller: Intel Corporation Device 5a85 (rev 0b)
IOMMU group 2:	[8086:5a98] 00:0e.0 Audio device: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Audio Cluster (rev 0b)
IOMMU group 3:	[8086:5a9a] 00:0f.0 Communication controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Trusted Execution Engine (rev 0b)
IOMMU group 4:	[8086:5ae3] 00:12.0 SATA controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SATA AHCI Controller (rev 0b)
IOMMU group 5:	[8086:5ad8] 00:13.0 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #1 (rev fb)
[8086:5ad9] 00:13.1 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #2 (rev fb)
[8086:5ada] 00:13.2 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #3 (rev fb)
[8086:5adb] 00:13.3 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #4 (rev fb)
[10ec:8168] 01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 11)
[1b21:0612] 03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
[1b21:1182] 04:00.0 PCI bridge: ASMedia Technology Inc. Device 1182
[1b21:1182] 05:03.0 PCI bridge: ASMedia Technology Inc. Device 1182
[1b21:1182] 05:07.0 PCI bridge: ASMedia Technology Inc. Device 1182
[10ec:8168] 06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
[10ec:8168] 07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
IOMMU group 6:	[8086:5aa8] 00:15.0 USB controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series USB xHCI (rev 0b)
IOMMU group 7:	[8086:5ae8] 00:1f.0 ISA bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Low Pin Count Interface (rev 0b)
[8086:5ad4] 00:1f.1 SMBus: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SMBus Controller (rev 0b)

dual NIC is in group 5 (06:00.0 and 07:00.0) together with the onboard NIC (01:00.0). 

Trying to passthrough the hardware using hostdev will not work 

group 5 is not viable. Please ensure all devices within the IOMMU group are bound to their vfio bus driver

So the goal is to split IOMMU groups a little more up.

I already tried  

pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1

But the groups remain the same.

Then I did a little research and found these threads:

https://www.reddit.com/r/VFIO/comments/61q2k9/asrock_j3455_iommu_groups/

https://www.reddit.com/r/VFIO/comments/63j1p7/any_ideas_why_all_pcie_ports_are_assigned_to_the/

 

A possible solution seems to modify the kernel a bit. Removing the part  

pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS``)

seems to be a solution for the OP. I tried myself by adding this to the go-file 

rmmod pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS)

But after that the server wont boot. At this moment I am running headless so I first need to get another monitor to see the output while booting.

 

Help would really be appreciated. Thanks.

 

(OT: sorry for the bad english, I am from germany)

Edited by MisterSwiffer
Link to comment
  • 2 months later...
  • 2 years later...
alitech Apprentice

alitech

Posted
Posted
On 6/4/2018 at 8:38 PM, MisterSwiffer said:

hi all,

 

so far I am using unRAID since more than a year and I am very happy! But now I am facing the first problem that I am not able to solve by myself.

 

My board is an Asrock J3455 with one PCIe slot. I added a dual NIC on this slot that I would like to pass to a VM. But the dual NIC is inside a big IOMMU group:

  


IOMMU group 0:	[8086:5af0] 00:00.0 Host bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Host Bridge (rev 0b)
IOMMU group 1:	[8086:5a85] 00:02.0 VGA compatible controller: Intel Corporation Device 5a85 (rev 0b)
IOMMU group 2:	[8086:5a98] 00:0e.0 Audio device: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Audio Cluster (rev 0b)
IOMMU group 3:	[8086:5a9a] 00:0f.0 Communication controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Trusted Execution Engine (rev 0b)
IOMMU group 4:	[8086:5ae3] 00:12.0 SATA controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SATA AHCI Controller (rev 0b)
IOMMU group 5:	[8086:5ad8] 00:13.0 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #1 (rev fb)
[8086:5ad9] 00:13.1 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #2 (rev fb)
[8086:5ada] 00:13.2 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #3 (rev fb)
[8086:5adb] 00:13.3 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #4 (rev fb)
[10ec:8168] 01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 11)
[1b21:0612] 03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
[1b21:1182] 04:00.0 PCI bridge: ASMedia Technology Inc. Device 1182
[1b21:1182] 05:03.0 PCI bridge: ASMedia Technology Inc. Device 1182
[1b21:1182] 05:07.0 PCI bridge: ASMedia Technology Inc. Device 1182
[10ec:8168] 06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
[10ec:8168] 07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
IOMMU group 6:	[8086:5aa8] 00:15.0 USB controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series USB xHCI (rev 0b)
IOMMU group 7:	[8086:5ae8] 00:1f.0 ISA bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Low Pin Count Interface (rev 0b)
[8086:5ad4] 00:1f.1 SMBus: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SMBus Controller (rev 0b)

dual NIC is in group 5 (06:00.0 and 07:00.0) together with the onboard NIC (01:00.0). 

Trying to passthrough the hardware using hostdev will not work 


group 5 is not viable. Please ensure all devices within the IOMMU group are bound to their vfio bus driver

So the goal is to split IOMMU groups a little more up.

I already tried  


pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1

But the groups remain the same.

Then I did a little research and found these threads:

https://www.reddit.com/r/VFIO/comments/61q2k9/asrock_j3455_iommu_groups/

https://www.reddit.com/r/VFIO/comments/63j1p7/any_ideas_why_all_pcie_ports_are_assigned_to_the/

 

A possible solution seems to modify the kernel a bit. Removing the part  


pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS``)

seems to be a solution for the OP. I tried myself by adding this to the go-file 


rmmod pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS)

But after that the server wont boot. At this moment I am running headless so I first need to get another monitor to see the output while booting.

 

Help would really be appreciated. Thanks.

 

(OT: sorry for the bad english, I am from germany)

How did you split it? Did it work in the end?

 

I have a similar issue, I have 4 ports and I want to see them available to VMs but nothing I have tried has worked so far. What was your method?

This is what my card looks like in system devices.

 

[8086:10c9] 25:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

[8086:10c9] 25:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

[8086:10c9] 26:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

[8086:10c9] 26:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

 

 

Link to comment
  • 5 months later...
eltonk Noob

eltonk

Posted
Posted (edited)

Hey guys!
 

I'm facing similar issue, but in my case I have an onboard dual NIC that UNRaid is using and I also have a PCIe Dual NIC that I'd like to passthrough to the KVM.
The issue is that both NICS (onboard and PCIe) are of the same brand, and by consequence they have the same ID, as you can see in the picture.

 

How could I separate the NICS from the IOMMU group 1?

- [14e4:165f] 02:00.0

- [14e4:165f] 02:00.1

 

Screen-Shot-2021-08-11-at-2-43-32-PM.png

 

Following the video, after the first reboot none of the networks cards work anymore, and the unraid does not have access to internet anymore.

 

Thanks!

Edited by eltonk
Link to comment
ghost82 Proficient

ghost82

Posted
Posted (edited)
23 minutes ago, eltonk said:

How could I separate the NICS from the IOMMU group 1?

- [14e4:165f] 02:00.0

- [14e4:165f] 02:00.1

Use acs override set to "both" and see if they split.

If they split you can passthrough one to the vm.

If they don't split you can't do nothing :(

23 minutes ago, eltonk said:

The issue is that both NICS (onboard and PCIe) are of the same brand, and by consequence they have the same ID

Why is this an issue?

Even if you passthrough 2 identical devices with the same vendor/device id, libvirt takes into account the addresses (domain:bus:slot.function), so this is a non issue.

For example, if both nics had to be passed through, which is not what you want to do if I understood well, the first one will be at bus 2, slot 0, function 0, the second one will be at bus 2, slot 0, function 1, so 2 devices with the same vendor/device id, but at 2 different sources.

Edited by ghost82
Link to comment
gray squirrel Rookie

gray squirrel

Posted
Posted (edited)
On 8/11/2021 at 2:14 PM, ghost82 said:

Use acs override set to "both" and see if they split.

If they split you can passthrough one to the vm.

If they don't split you can't do nothing :(

 


sometimes in the motherbord bios there are multiple settings for IOMMU. Mine had auto, disabled and enabled. Auto and Enabled where different.

 

an old sever board I was using for my first server had even more setting.

 

mix and match to see if you get the results you want 

Edited by gray squirrel
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing