VM Passthrough


Recommended Posts

I installed a 4 port 1gb Nic and I am trying to pass it through to a FreeBsd VM. However, when I create the VM I do not have any interfaces under "Other PCI Devices".

 

I went through spaceinvader Ones IOMMU tutorial and I cant get it to show in this vm.   Appreciate any help to get this working. 

 

This is my IOMMU group and I want to pass through "8086:10bc"

 

IOMMU group 15:[1022:43d5] 02:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller (rev 01)

[1022:43c8] 02:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01)

[1022:43c6] 02:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge (rev 01)

[1022:43c7] 03:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

[1022:43c7] 03:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

[1022:43c7] 03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

[8086:1539] 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)

[111d:8018] 06:00.0 PCI bridge: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch (rev 0e)

[111d:8018] 07:02.0 PCI bridge: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch (rev 0e)

[111d:8018] 07:04.0 PCI bridge: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch (rev 0e)

[8086:10bc] 08:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)

[8086:10bc] 08:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)

[8086:10bc] 09:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)

[8086:10bc] 09:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)

 

IOMMU shows enabled on Unraid but I do not have a setting in my bios.  

I tried editing the flash config with just the NIC 8086:10bc.  I tried enabling ACS override.  I even tried doing the acs override manual edit that spaceinvader one talks about in his "Breaking up IOMMU groups".  I can get them to break up but the NIC still won't show under "Other PCI Devices". 

 

Any help would be appreciated. 

Link to comment

Have you stubbed the network card, either in your syslinux file, or using vfio-config plugin? (Second option is easiest)

 

for a device to show up for passthrough at vm creation, you. Must tell unraid at boot to ignore it and make it available.

 

That’s assuming you’ve got it to show up in its own iommu group, of course.

Edited by meep
Link to comment

So I got the card to show under other PCI devices but I get this error:

 

internal error: qemu unexpectedly closed the monitor: 2020-05-11T22:05:11.723146Z qemu-system-x86_64: -device vfio-pci,host=0000:08:00.0,id=hostdev0,bus=pci.3,addr=0x0: vfio 0000:08:00.0: group 15 is not viable Please ensure all devices within the iommu_group are bound to their vfio bus driver.

 

I tried adding 

vfio-pci.ids=8086:10bc


I also did vfio-pci.ids=8086:10bc,1022:43d5,1022:43c8,8086:1539

 

I did this both under flash just like spaceinvaders IOMMU group video. 

 

I also enabled ACS override.  

 

Nothing worked. 

 

IOMMU is enabled on my unraid server and no I do not have a bios setting to change it from auto to on.  

Link to comment

So, the basic principle is that if one device in an IOMMU group is stubbed, that all devices in that group will be impacted.

 

Therefore, if the device you want to passthrough is in a group with other devices, all must be stubbed. It sounds like you've got an IOMMU group from which you've stubbed a device that also includes your system network card. Therefore, unRAID OS has no access to this card and therefore no network. (look at the last couple of lines on your boot screen to see the IP addresses unRAID is assigned. It's likely you'll see a self-assigned address there rather than one recognisable from your LAN address range.

 

Take great care with this - its very easy to stub something thats in the same group as your USB controller, and then unRAID won't boot at all as it loses access to the USB thumb-drive!

 

So your first step is to try to get your passthrough device in its own IOMMU group. There's no point trying to stub it until you get this far. You're looking for a neat arrangement something like this;

 

image.thumb.png.a0e7422b2d08c478a8529eb6cdbbcb8d.png

 

You need to play with ACS settings, Bios settings etc. until you achieve this.  Only then can you start stubbing.

Of course, it's entirely possible that your MB does not support this and then you're stuck.

If you do manage to get your device in its own group, please post a fresh listing of your IOMMU groupings and we can take it from there.

 

 

Edited by meep
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.