MisterSwiffer Posted June 4, 2018 Share Posted June 4, 2018 (edited) hi all, so far I am using unRAID since more than a year and I am very happy! But now I am facing the first problem that I am not able to solve by myself. My board is an Asrock J3455 with one PCIe slot. I added a dual NIC on this slot that I would like to pass to a VM. But the dual NIC is inside a big IOMMU group: IOMMU group 0: [8086:5af0] 00:00.0 Host bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Host Bridge (rev 0b) IOMMU group 1: [8086:5a85] 00:02.0 VGA compatible controller: Intel Corporation Device 5a85 (rev 0b) IOMMU group 2: [8086:5a98] 00:0e.0 Audio device: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Audio Cluster (rev 0b) IOMMU group 3: [8086:5a9a] 00:0f.0 Communication controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Trusted Execution Engine (rev 0b) IOMMU group 4: [8086:5ae3] 00:12.0 SATA controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SATA AHCI Controller (rev 0b) IOMMU group 5: [8086:5ad8] 00:13.0 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #1 (rev fb) [8086:5ad9] 00:13.1 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #2 (rev fb) [8086:5ada] 00:13.2 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #3 (rev fb) [8086:5adb] 00:13.3 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #4 (rev fb) [10ec:8168] 01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 11) [1b21:0612] 03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) [1b21:1182] 04:00.0 PCI bridge: ASMedia Technology Inc. Device 1182 [1b21:1182] 05:03.0 PCI bridge: ASMedia Technology Inc. Device 1182 [1b21:1182] 05:07.0 PCI bridge: ASMedia Technology Inc. Device 1182 [10ec:8168] 06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) [10ec:8168] 07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) IOMMU group 6: [8086:5aa8] 00:15.0 USB controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series USB xHCI (rev 0b) IOMMU group 7: [8086:5ae8] 00:1f.0 ISA bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Low Pin Count Interface (rev 0b) [8086:5ad4] 00:1f.1 SMBus: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SMBus Controller (rev 0b) dual NIC is in group 5 (06:00.0 and 07:00.0) together with the onboard NIC (01:00.0). Trying to passthrough the hardware using hostdev will not work group 5 is not viable. Please ensure all devices within the IOMMU group are bound to their vfio bus driver So the goal is to split IOMMU groups a little more up. I already tried pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 But the groups remain the same. Then I did a little research and found these threads: https://www.reddit.com/r/VFIO/comments/61q2k9/asrock_j3455_iommu_groups/ https://www.reddit.com/r/VFIO/comments/63j1p7/any_ideas_why_all_pcie_ports_are_assigned_to_the/ A possible solution seems to modify the kernel a bit. Removing the part pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS``) seems to be a solution for the OP. I tried myself by adding this to the go-file rmmod pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS) But after that the server wont boot. At this moment I am running headless so I first need to get another monitor to see the output while booting. Help would really be appreciated. Thanks. (OT: sorry for the bad english, I am from germany) Edited June 4, 2018 by MisterSwiffer Quote Link to comment
1812 Posted June 5, 2018 Share Posted June 5, 2018 you can try following this (some you have done) as gridrunner talks about breaking up the bridge Quote Link to comment
jbonnett Posted August 25, 2018 Share Posted August 25, 2018 I added pcie_acs_override=id:1b21.1242,id:8086.8d26,id:8086.8d2d,id:8086.8d31 Although only one of the devices shows in the "Other PCIe devices" list. Any ideas? Quote Link to comment
1812 Posted August 25, 2018 Share Posted August 25, 2018 4 hours ago, jbonnett said: I added pcie_acs_override=id:1b21.1242,id:8086.8d26,id:8086.8d2d,id:8086.8d31 Although only one of the devices shows in the "Other PCIe devices" list. Any ideas? probably because you didn't do it right. Quote Link to comment
jbonnett Posted August 25, 2018 Share Posted August 25, 2018 (edited) 1 minute ago, 1812 said: probably because you didn't do it right. I did a straight copy paste of the id. Edited August 25, 2018 by jbonnett Quote Link to comment
1812 Posted August 25, 2018 Share Posted August 25, 2018 53 minutes ago, jbonnett said: I did a straight copy paste of the id. watch the video above with attention to detail. your acs override using devices in the syslinux should look similar to this pcie_acs_override=id:8086:3a40,1b73:1100 Quote Link to comment
jbonnett Posted August 25, 2018 Share Posted August 25, 2018 Just now, 1812 said: watch the video above with attention to detail. your acs override using devices in the syslinux should look similar to this pcie_acs_override=id:8086:3a40,1b73:1100 I also tried that with no luck. Quote Link to comment
John_M Posted August 25, 2018 Share Posted August 25, 2018 6 hours ago, jbonnett said: I also tried that with no luck. First of all, you should have started your own thread as your situation is different from that of the OP. Have you tried the various other suggestions in the video? Quote Link to comment
alitech Posted February 28, 2021 Share Posted February 28, 2021 On 6/4/2018 at 8:38 PM, MisterSwiffer said: hi all, so far I am using unRAID since more than a year and I am very happy! But now I am facing the first problem that I am not able to solve by myself. My board is an Asrock J3455 with one PCIe slot. I added a dual NIC on this slot that I would like to pass to a VM. But the dual NIC is inside a big IOMMU group: IOMMU group 0: [8086:5af0] 00:00.0 Host bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Host Bridge (rev 0b) IOMMU group 1: [8086:5a85] 00:02.0 VGA compatible controller: Intel Corporation Device 5a85 (rev 0b) IOMMU group 2: [8086:5a98] 00:0e.0 Audio device: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Audio Cluster (rev 0b) IOMMU group 3: [8086:5a9a] 00:0f.0 Communication controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Trusted Execution Engine (rev 0b) IOMMU group 4: [8086:5ae3] 00:12.0 SATA controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SATA AHCI Controller (rev 0b) IOMMU group 5: [8086:5ad8] 00:13.0 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #1 (rev fb) [8086:5ad9] 00:13.1 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #2 (rev fb) [8086:5ada] 00:13.2 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #3 (rev fb) [8086:5adb] 00:13.3 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #4 (rev fb) [10ec:8168] 01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 11) [1b21:0612] 03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) [1b21:1182] 04:00.0 PCI bridge: ASMedia Technology Inc. Device 1182 [1b21:1182] 05:03.0 PCI bridge: ASMedia Technology Inc. Device 1182 [1b21:1182] 05:07.0 PCI bridge: ASMedia Technology Inc. Device 1182 [10ec:8168] 06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) [10ec:8168] 07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) IOMMU group 6: [8086:5aa8] 00:15.0 USB controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series USB xHCI (rev 0b) IOMMU group 7: [8086:5ae8] 00:1f.0 ISA bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Low Pin Count Interface (rev 0b) [8086:5ad4] 00:1f.1 SMBus: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SMBus Controller (rev 0b) dual NIC is in group 5 (06:00.0 and 07:00.0) together with the onboard NIC (01:00.0). Trying to passthrough the hardware using hostdev will not work group 5 is not viable. Please ensure all devices within the IOMMU group are bound to their vfio bus driver So the goal is to split IOMMU groups a little more up. I already tried pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 But the groups remain the same. Then I did a little research and found these threads: https://www.reddit.com/r/VFIO/comments/61q2k9/asrock_j3455_iommu_groups/ https://www.reddit.com/r/VFIO/comments/63j1p7/any_ideas_why_all_pcie_ports_are_assigned_to_the/ A possible solution seems to modify the kernel a bit. Removing the part pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS``) seems to be a solution for the OP. I tried myself by adding this to the go-file rmmod pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS) But after that the server wont boot. At this moment I am running headless so I first need to get another monitor to see the output while booting. Help would really be appreciated. Thanks. (OT: sorry for the bad english, I am from germany) How did you split it? Did it work in the end? I have a similar issue, I have 4 ports and I want to see them available to VMs but nothing I have tried has worked so far. What was your method? This is what my card looks like in system devices. [8086:10c9] 25:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) [8086:10c9] 25:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) [8086:10c9] 26:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) [8086:10c9] 26:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) Quote Link to comment
eltonk Posted August 11, 2021 Share Posted August 11, 2021 (edited) Hey guys! I'm facing similar issue, but in my case I have an onboard dual NIC that UNRaid is using and I also have a PCIe Dual NIC that I'd like to passthrough to the KVM. The issue is that both NICS (onboard and PCIe) are of the same brand, and by consequence they have the same ID, as you can see in the picture. How could I separate the NICS from the IOMMU group 1? - [14e4:165f] 02:00.0 - [14e4:165f] 02:00.1 Following the video, after the first reboot none of the networks cards work anymore, and the unraid does not have access to internet anymore. Thanks! Edited August 11, 2021 by eltonk Quote Link to comment
Squid Posted August 11, 2021 Share Posted August 11, 2021 That video is old. Remove the lines you've been adding to syslinux and check off what you want to isolate. You'd isolate group 8 to pass that through to the VM Quote Link to comment
ghost82 Posted August 11, 2021 Share Posted August 11, 2021 (edited) 23 minutes ago, eltonk said: How could I separate the NICS from the IOMMU group 1? - [14e4:165f] 02:00.0 - [14e4:165f] 02:00.1 Use acs override set to "both" and see if they split. If they split you can passthrough one to the vm. If they don't split you can't do nothing 23 minutes ago, eltonk said: The issue is that both NICS (onboard and PCIe) are of the same brand, and by consequence they have the same ID Why is this an issue? Even if you passthrough 2 identical devices with the same vendor/device id, libvirt takes into account the addresses (domain:bus:slot.function), so this is a non issue. For example, if both nics had to be passed through, which is not what you want to do if I understood well, the first one will be at bus 2, slot 0, function 0, the second one will be at bus 2, slot 0, function 1, so 2 devices with the same vendor/device id, but at 2 different sources. Edited August 11, 2021 by ghost82 Quote Link to comment
gray squirrel Posted August 12, 2021 Share Posted August 12, 2021 (edited) On 8/11/2021 at 2:14 PM, ghost82 said: Use acs override set to "both" and see if they split. If they split you can passthrough one to the vm. If they don't split you can't do nothing sometimes in the motherbord bios there are multiple settings for IOMMU. Mine had auto, disabled and enabled. Auto and Enabled where different. an old sever board I was using for my first server had even more setting. mix and match to see if you get the results you want Edited August 12, 2021 by gray squirrel Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.