Cannot get NIC to show when setting up a VM

[partyx]

G'day Unraid Community!

 

I'm very new to servers / unraid so please excuse my ignorance!

 

I'm having trouble getting my NIC to display as an "Other PCI Devices" to allow passthrough when creating a freebsd VM - I'm trying to get the VM to host pfSense.

 

I'm following SpaceInvader One's guide on pfSense install on Unraid:

 

I'm using a Dell r730 server, I have enabled IOMMU within the BIOS. I have also checked that my Broadcom BCM5720 Gigabit Ethernet PCIe is compatible with freebsd.

 

I have no idea what I've done wrong - could anyone please help me out??

 

Please find below pictures to show my config, etc.

 

My Syslinux Config where I performed the ACS overide:

 

My IOMMU groups did look like this prior to the ACS overide:

 

Now what my IOMMU groups look like after the ACS overide:

 

My Network Settings:

 

The VM I'm trying to create:

 

What I'm expecting to see from @SpaceInvaderOne video:

 

Thanks guys!

Edited December 15, 2022 by partyx
Deleted extra screen shot

[ghost82]

Syslinux config looks good to me, but I would set it to:

pcie_acs_override=downstream,multifunction

Reboot the server.

Go to your iommu groups through the unraid gui and see if you can put checkmarks on ehternet controllers that you want to passthrough, save and reboot the server.

Now go to your vm and see if something shows.

Note that you can't passthrough ethernet controllers that are in use by unraid (which seems the case, since you are setting in unraid eth0 -->eth3), boxes will show greyed out in iommu groups.

--

The above instructions may not work..

Be sure to have another device to attach the unraid usb in case something goes wrong and restore a backup

Not sure but you may need to set the config manually since all the controllers have the same vendor/device id.

1. backup the unraid usb, so to restore if something goes wrong

2. open config/vfio-pci.cfg with a text editor in the unraid usb stick

3. Add BIND=0000:01:00.1|14e4:165f 0000:02:00.0|14e4:165f 0000:02:00.1|14e4:165f

Note that I didn't write 01:00.0 which should be eth0, needed and reserved for unraid

 

Save and reboot, and your eth1-->eth3 (3 ports) should now be isolated, without loosing eth0 connectivity for unraid.

Edited December 15, 2022 by ghost82

[partyx]

Thanks for the reply!

 

Ok so I've changed the acs overide to what you suggested:

 

The IOMMU groups for the NIC is already greyed out:

 

VM still doesn't show the card to passthrough:

[ghost82]

Look my edited post above and go/try the manual way

[partyx]

So I left the pcie_acs_override=downstream,multifunction config in the syslinux.

 

I went to the config folder, there was no vfio-pci.cfg file so I manually created it.

 

In that file, I only have the following line written:

 

BIND=14e4:165f 0000:01:00.1|14e4:165f 0000:02:00.0|14e4:165f 0000:02:00.1

 

 

Other PCI Devices within the VM configuration screen, still shows "None available".

 

[partyx]

Ok so I misunderstood in my previous post...

I had to create a vfio-pci.cfg file via the terminal and within the file put:

 

BIND=14e4:165f 0000:01:00.1|14e4:165f 0000:02:00.0|14e4:165f 0000:02:00.1

 

After reboot, nothing has changed...

 

I double checked that the vfio-pci.cfg file is still within the config:

 

There is no button for vfio-pci within the tools: 

 

It appears that the vfio-pci.cfg file isnt being read?

 

Cheers

[partyx]

Thanks for your assistance @ghost82 - I got it going now!

 

 

[ghost82]

Nica that it works now.

Just for completeness, before making it going I can see following errors:

8 hours ago, partyx said:

BIND=14e4:165f 0000:01:00.1|14e4:165f 0000:02:00.0|14e4:165f 0000:02:00.1

 

it's wrong, should be:

BIND=0000:01:00.1|14e4:165f 0000:02:00.0|14e4:165f 0000:02:00.1|14e4:165f

 

5 hours ago, partyx said:

 

Name of the file vfio.pci.cfg is wrong, should be vfio-pci.cfg

[partyx]

Ok cool - I will change the binding now.

 

Yeah i picked up the issue with the vfio.pci.cfg...

 

 

I've selected all 3 for me freebsd VM.

 

The issue I'm having now however, is pfSense is only picking up one interface during the install - bge0.

 

 

[ghost82]

If the vm is setup correctly for all the controllers passed it could be an issue with configuration (?).

I know @Ford Prefect is a sort of guru for network related things, maybe he knows what's wrong.

[partyx]

So after some testing with a mate, we found that only selecting the 2 interfaces on BUS 2 works, you just cannot get split the other BUS and have one interface work.

 

I proved this by creating another VM and just selecting 0000:01:00.1 - during pfSense booting up, it says not ethernet found and wont allow you to go any further with the configuration.

[ghost82]

11 minutes ago, partyx said:

we found that only selecting the 2 interfaces on BUS 2 works

Thanks for reporting. It can be quite tricky even to pass the controllers so consider yourself lucky to have 2 nics available in the vm.

It could be an issue with initialization for the driver, since you have 4 controllers, coupled 2 by 2 in a multifunction device; having one of the multifunction controller missing may cause issue for the driver or for the firmware.

 

So, having only one nic available for the vm (in your last screenshot) was probably due to the error in the vfio-pci.cfg:

BIND=14e4:165f 0000:01:00.1|14e4:165f 0000:02:00.0|14e4:165f 0000:02:00.1 

Most probably only the controller on 02:00.0 was available, but changing the sintax into vfio-pci.cfg made both available.

[Ford Prefect]

12 hours ago, ghost82 said:

If the vm is setup correctly for all the controllers passed it could be an issue with configuration (?).

I know @Ford Prefect is a sort of guru for network related things, maybe he knows what's wrong.

 

10 hours ago, partyx said:

So after some testing with a mate, we found that only selecting the 2 interfaces on BUS 2 works, you just cannot get split the other BUS and have one interface work.

 

I proved this by creating another VM and just selecting 0000:01:00.1 - during pfSense booting up, it says not ethernet found and wont allow you to go any further with the configuration.

...I only had a brief look into this thread....with pfsense/freebsd another issue I remember with NICs on passthrough not showing up in the OS, had something to do with the machine model and BIOS selected in the VM config.

What helped was to select a less advanced machine, like an i440fx-??? and Seabios instead of OVMF.