IOMMU Passthrough


Recommended Posts

I am trying to pass my 4 port GB NIC to pass through to my pfSense VM.
IOMMU shows enabled on Unraid but I do not have a IOMMU setting in my bios.

My NIC is in IOMMU Group 15 with this ID 8086:10bc

[1022:43d5] 02:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller (rev 01)
[1022:43c8] 02:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01)
[1022:43c6] 02:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge (rev 01)
[1022:43c7] 03:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
[1022:43c7] 03:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
[1022:43c7] 03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
[8086:1539] 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
[111d:8018] 06:00.0 PCI bridge: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch (rev 0e)
[111d:8018] 07:02.0 PCI bridge: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch (rev 0e)
[111d:8018] 07:04.0 PCI bridge: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch (rev 0e)
[8086:10bc] 08:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
[8086:10bc] 08:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
[8086:10bc] 09:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
[8086:10bc] 09:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)

I tried adding this under flash vfio-pci.ids=8086:10bc
And the NIC still wont show under other PCI devices

I changed it to the following:
vfio-pci.ids=8086:10bc,1022:43d5,1022:43c8,8086:1539

When I do this I cannot ping my unraid server nor access the web gui remotely.

I tried enabling ACS override and even VFIO allow unsafe interrupts

I cannot get it to work and hoping someone could help.

Link to comment

What Unraid version are you using? There is a new vfio-bind method in Unraid 6.7 and upwards. Also check out the VFIO-PCI Config Plugin in this context. Please use the code tags for configuration snippets next.

 

 

Edited by T0a
Link to comment
5 minutes ago, Armed Ferret said:

I do not seem to have the Vfio-pci.cfg file on my flash drive

 

Is there a way to get it there.  Im on version 6.8.3

The easiest way is to install the VFIO-Config plugin from Community Applications, mentioned by @T0a, above

 

This provides a visual listing of devices, allows you click checkboxes to add them an generates the .cfg file for you.

 

 

Link to comment

So  I added the plugin.  Then I checked the devices I wanted to use for passthrough, rebooted and now I cannot access the Unraid Web GUI.  I tried turning on ACS override to break up the IOMMU group and still cannot access the Web GUI or Ping the server from my PC.

 

I unchecked the devices from the plug in and turned off ACS Override and rebooted again.  Still cant ping or access.

 

Went into my flash drive and deleted the config file.  Still cant ping or access.

 

What do I need to do next?

Link to comment
21 minutes ago, Armed Ferret said:

So  I added the plugin.  Then I checked the devices I wanted to use for passthrough, rebooted and now I cannot access the Unraid Web GUI.  I tried turning on ACS override to break up the IOMMU group and still cannot access the Web GUI or Ping the server from my PC.

 

I unchecked the devices from the plug in and turned off ACS Override and rebooted again.  Still cant ping or access.

 

Went into my flash drive and deleted the config file.  Still cant ping or access.

 

What do I need to do next?

 

Which device did you exactly checked (8086:1539 or 8086:10bc)? Note, when ticking the wrong one, Unraid has no access to this device such as the eth controller, which could be the cause for your ping issues. The "ACS Override" option shouldn't be required as the device you want to pass through resides in its own IOMMU group, right?

 

In case you haven't changed something else, you should be fine with only removing the pcie-vfio config file from the flash drive as you did. Can you attach a monitor to your server? Then pls look for any printed issues. Also try to boot in "save mode" (with disabled plugins and a virgin sys config file). Does this work?

 

I'm so thankful that limetech put their OS on a flash drive with easy access in case of such a problem.

Edited by T0a
Link to comment

I figured it out. 

 

But to answer your question, I specifically checked the 4 port nic 8086:10bc.

 

Somehow it seems my interface rules got changed when doing the vfio config. 

I saw one of the 4 port nics was now assigned to eth0 and my on board mobo nic was assigned to eth2.  I switched them back and rebooted and bam...I have internet.  So let me try this tool again.

 

And no...The device I am passing through does not reside in its own IOMMU group.  So I will need to enable ACS override, correct?

 

 


IOMMU group 15:	[1022:43d5] 02:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller (rev 01)
[1022:43c8] 02:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01)
[1022:43c6] 02:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge (rev 01)
[1022:43c7] 03:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
[1022:43c7] 03:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
[1022:43c7] 03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
[8086:1539] 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
[111d:8018] 06:00.0 PCI bridge: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch (rev 0e)
[111d:8018] 07:02.0 PCI bridge: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch (rev 0e)
[111d:8018] 07:04.0 PCI bridge: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch (rev 0e)
[8086:10bc] 08:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
[8086:10bc] 08:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
[8086:10bc] 09:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
[8086:10bc] 09:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)

8086:10bc is my 4 port nic im trying to passthrough

 

Should I enable ACS override then?

Edited by Armed Ferret
more info
Link to comment

Ok.. I have them all in their IOMMU Groups and enabled ACS override to separate them into their IOMMU groups

 

However, only two of my 4 ports are showing. 

 

I looked in the vfio-cfg file and I see all 4 ids added

 

BIND=08:00.0 08:00.1 09:00.0 09:00.1

And here is my IOMMU groups


IOMMU group 25:	[8086:10bc] 08:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
IOMMU group 26:	[8086:10bc] 08:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
IOMMU group 27:	[8086:10bc] 09:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
IOMMU group 28:	[8086:10bc] 09:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)

 

In my VM the two that are showing:

image.png.7e4cb52ecd6123e52c791aa50c51b6e6.png

 

In my network settings these two are not being passed:

image.png.0a88c8801de0dbba2d8002138b6ab943.png

Is there a reason I cannot get all 4 to pass through?

Edited by Armed Ferret
screenshots
Link to comment
On 5/13/2020 at 4:47 AM, T0a said:

Does maybe this thread help:

Symptoms are the same

 

So it helps some.  However, to me, its not as clear.  

 

This is what is in my XML for the two ports that are currently being passed

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>

 

When I look up the other two I want to pass here are their info:

 

08:00.0 0200: 8086:10bc (rev 06)
08:00.1 0200: 8086:10bc (rev 06)
09:00.0 0200: 8086:10bc (rev 06)
09:00.1 0200: 8086:10bc (rev 06)

 

Now I get I should be changing the bus to 0x09 for the other two.  But I am not sure what else I need to change?  Im guessing I would need to set one to function 0x01.  But on the second source like there is bus 0x03 and 0x04.  Do I need to change those?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.