Need Help with Pfsense on Unraid.

 

I have 4 Ethernet port on my Server. 2 Intel onboard NIC & one dual Intel PCI Ethernet.

 

Currently I connected my router to one of the port in PCI NIC for management. I am trying to install pfsence by passing 2 Onboard NIC.

 

 

 

However after creating the VM and while starting i am getting below error. Can someone help me in fixing this?

 

On 3/12/2021 at 3:53 PM, reubenjack said:

Need Help with Pfsense on Unraid.

 

I have 4 Ethernet port on my Server. 2 Intel onboard NIC & one dual Intel PCI Ethernet.

 

Currently I connected my router to one of the port in PCI NIC for management. I am trying to install pfsence by passing 2 Onboard NIC.

 

Sorry, I am not sure. The screenshots show that the devices are bound to vfio-pci, so Unraid isn't holding onto them. But I don't know why the VM fails to start. I'd suggest posting a new topic over here: https://forums.unraid.net/forum/51-vm-engine-kvm/  Be sure to include your diagnostics ( Tools -> Diagnostics )

Before binding Phison controller (NVME SSD) to VFIO, I first unmounted my NVME SSD from Unassigned devices.  After binding, and reboot, Unassigned devices doesn't "see" NVME SSD.  Neither does making my Primary vDisk location /dev/nvme0n1 work, as it doesn't "see" /dev/...

 

Any advise?

On 4/8/2021 at 4:01 AM, jang430 said:

Before binding Phison controller (NVME SSD) to VFIO, I first unmounted my NVME SSD from Unassigned devices.  After binding, and reboot, Unassigned devices doesn't "see" NVME SSD.  Neither does making my Primary vDisk location /dev/nvme0n1 work, as it doesn't "see" /dev/...

 

Any advise?

 

Binding a device to vfio-pci means that Unraid won't be able to use it, the purpose is to pass the device directly to a VM. So once you bind the device to vfio-pci it will be hidden from Unassigned devices and won't have a /dev/ path.

My purpose is to passthrough my NVME SSD.  I believe I should be binding the Phison controller (says NVME).  After binding it, restarting... When I go to the VM, I can see the controller in Other PCI devices, with an unchecked box (what should I do here). I don't know how to point my Primary vDisk to the said SSD when it's no longer presented on the choices.

Unraid Version: 6.9.1

PCIe ACS override: set to BOTH

 

internal error: 
qemu unexpectedly closed the monitor: 2021-04-10T09:19:09.126415Z qemu-system-x86_64: 
-device vfio-pci,host=0000:08:03.0,id=hostdev0,bus=pci.7,addr=0x1: vfio 0000:08:03.0: 
Failed to set up TRIGGER eventfd signaling for interrupt INTX-0: VFIO_DEVICE_SET_IRQS failure: 
Device or resource busy

 

16 hours ago, jang430 said:

My purpose is to passthrough my NVME SSD.  I believe I should be binding the Phison controller (says NVME).  After binding it, restarting... When I go to the VM, I can see the controller in Other PCI devices, with an unchecked box (what should I do here). I don't know how to point my Primary vDisk to the said SSD when it's no longer presented on the choices.

 

If you want to pass through the entire controller, then bind it to vfio-pci like you did and check the box when configuring the VM. That will give the VM full access to the controller. I don't have direct experience with this myself, but I believe it will show up in the VM the same way as if the VM's OS was running on a bare metal system i.e. you probably don't need to set the Primary vDisk as you aren't using one.

8 hours ago, oldfart said:

Unraid Version: 6.9.1

PCIe ACS override: set to BOTH

 

internal error: 
qemu unexpectedly closed the monitor: 2021-04-10T09:19:09.126415Z qemu-system-x86_64: 
-device vfio-pci,host=0000:08:03.0,id=hostdev0,bus=pci.7,addr=0x1: vfio 0000:08:03.0: 
Failed to set up TRIGGER eventfd signaling for interrupt INTX-0: VFIO_DEVICE_SET_IRQS failure: 
Device or resource busy

 

 

That is a bit beyond this guide, I'd recommend starting your own thread here: https://forums.unraid.net/forum/51-vm-engine-kvm/ 

 

Be sure to upload your diagnostics (from Tools -> Diagnostics). Screenshots / more details would also be helpful

My VM, when applying Phison passthrough, stays stuck in "Updating".

 

Hi All,

 

1. Just wondering if it is possible to use the bind function (BIND=XXXX:XX:XX.X|XXXX:XXX) to force unraid not to use a specific bus line so that it can be bound to a chosen VM.

 

2. The IOMMU group for the USB ports is Group 4. I'm trying to forcefully bind the Oculus Quest Link USB to a VM since whenever I use the checkbox method it works for a brief time, then gets disabled and/or deselected from the VM. My motherboard is a Gigabyte Aorus Z390 which only has one USB controller. I'm not sure how it's even possible that I can divide multiple USB mice/keyboards across 2 or more unique VMs but it does work for whatever reason even though they use the same IOMMU group. Unfortunately my goal, which is to connect multiple VR headsets like the oculus quest cannot be done via this method from what I have tried so far. The specific bus line is below (note I also have a Oculus Rift S on the system):

 

Bus 001 Device 024: ID 2833:0182 Oculus Quest 2

 

The VM xml does show the quest attached:

    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x2833'/>
        <product id='0x0182'/>
        <address bus='1' device='20'/>
      </source>
      <alias name='hostdev7'/>
      <address type='usb' bus='0' port='4'/>
    </hostdev>

 

 

3. I'm trying this since the new Sonnet USB controllers (USB3-PRO-4P10-E or USB3C-4PM-E) with Type C or Type A connections doesn't work in Unraid 6.9. I was hoping the hardware/software links of the new card would resemble the old Sonnet USB3-PRO-4PM-E with 4 unique FL1100 controllers which many have used and does work, however the new one just doesn't work the same. Maybe something to do with the Asmedia controllers and Pericom Bridge configuration, but I'm leaning towards the programming end from Unraid. If anyone knows why I am able to passthrough and bind these successfully yet not have the USB ports recognise any and all types of devices plugged in please let me know. I have manually installed Sonnet's specific drivers on the VM since by default it uses one of Microsofts generic ones for both the Controller and Hub but this still failed to get the ports functioning. You can see the full IOMMU list below with groups 22-26 representing the Sonnet Bridge and Controllers. Out of interest does anyone know what the graphics card controllers are? There are no visible USB-C ports on the RTX 2070 Super cards I'm running.

 

IOMMU group 16:[10de:1ad9] 01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)
IOMMU group 20:[10de:1ad9] 02:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)

 

4. Another frustration I've had with Unraid 6.91 is the lack of update to the VM XML when usb devices are unplugging and/or changed. I shouldn't have to manually enter the XML everytime I unplug a device that is passed through to a given VM to stop VM creation and Vendor issues.

 

IOMMU group 0:[8086:3e30] 00:00.0 Host bridge: Intel Corporation 8th/9th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] (rev 0a)
IOMMU group 1:[8086:1901] 00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 0a)
IOMMU group 2:[8086:1905] 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) (rev 0a)
IOMMU group 3:[8086:a379] 00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)

IOMMU group 4:[8086:a36d] 00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
	Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
	Bus 001 Device 002: ID 2833:2052 Oculus VR Rift S USB Hub
	Bus 001 Device 004: ID 2833:0051 Oculus VR Rift S
	Bus 001 Device 005: ID 05e3:0608 Genesys Logic, Inc. Hub
	Bus 001 Device 006: ID 048d:8297 Integrated Technology Express, Inc. ITE Device(8595)
	Bus 001 Device 007: ID 18a5:0302 Verbatim, Ltd Flash Drive
	Bus 001 Device 008: ID 05e3:0608 Genesys Logic, Inc. Hub
	Bus 001 Device 009: ID 045e:0745 Microsoft Corp. Nano Transceiver v1.0 for Bluetooth
	Bus 001 Device 010: ID 8087:0aaa Intel Corp.
	Bus 001 Device 011: ID 045e:07f8 Microsoft Corp. Wired Keyboard 600 (model 1576)
	Bus 001 Device 012: ID 0781:5591 SanDisk Corp. Ultra Flair
	Bus 001 Device 013: ID 046d:c534 Logitech, Inc. Unifying Receiver
	Bus 001 Device 024: ID 2833:0182 Oculus Quest 2
	Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
	Bus 002 Device 002: ID 2833:2051 Oculus VR Rift S USB Hub

[8086:a36f] 00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
IOMMU group 5:[8086:a370] 00:14.3 Network controller: Intel Corporation Wireless-AC 9560 [Jefferson Peak] (rev 10)
IOMMU group 6:[8086:a360] 00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
IOMMU group 7:[8086:a352] 00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
[2:0:0:0] disk ATA Samsung SSD 860 3B6Q /dev/sdc 1.00TB
IOMMU group 8:[8086:a340] 00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 (rev f0)
IOMMU group 9:[8086:a32c] 00:1b.4 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #21 (rev f0)
IOMMU group 10:[8086:a338] 00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 (rev f0)
IOMMU group 11:[8086:a330] 00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0)
IOMMU group 12:[8086:a305] 00:1f.0 ISA bridge: Intel Corporation Z390 Chipset LPC/eSPI Controller (rev 10)
[8086:a348] 00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10)
[8086:a323] 00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
[8086:a324] 00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
[8086:15bc] 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-V (rev 10)
IOMMU group 13:[10de:1e84] 01:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2070 SUPER] (rev a1)
IOMMU group 14:[10de:10f8] 01:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1)
IOMMU group 15:[10de:1ad8] 01:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1)
	Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
	Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
IOMMU group 16:[10de:1ad9] 01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)
IOMMU group 17:[10de:1e84] 02:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2070 SUPER] (rev a1)
IOMMU group 18:[10de:10f8] 02:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1)
IOMMU group 19:[10de:1ad8] 02:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1)
	Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
	Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
IOMMU group 20:[10de:1ad9] 02:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)
IOMMU group 21:[144d:a808] 03:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
[N:0:4:1] disk Samsung SSD 970 PRO 1TB__1 /dev/nvme0n1 1.02TB


IOMMU group 22:[12d8:2308] 04:00.0 PCI bridge: Pericom Semiconductor Device 2308
IOMMU group 23:[12d8:2308] 05:01.0 PCI bridge: Pericom Semiconductor Device 2308
IOMMU group 24:[12d8:2308] 05:02.0 PCI bridge: Pericom Semiconductor Device 2308
IOMMU group 25:[1b21:2142] 06:00.0 USB controller: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller
This controller is bound to vfio, connected USB devices are not visible.
IOMMU group 26:[1b21:2142] 07:00.0 USB controller: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller
This controller is bound to vfio, connected USB devices are not visible.


IOMMU group 27:[c0a9:540a] 09:00.0 Non-Volatile memory controller: Micron/Crucial Technology Device 540a (rev 01)
[N:1:1:1] disk CT1000P2SSD8__1 /dev/nvme1n1 1.00TB

 

 

Edited April 15, 20215 yr by Clientel

On 4/10/2021 at 6:33 PM, jang430 said:

My VM, when applying Phison passthrough, stays stuck in "Updating".

 

I'd recommend starting your own thread here: https://forums.unraid.net/forum/51-vm-engine-kvm/ 

 

Be sure to upload your diagnostics (from Tools -> Diagnostics)

16 hours ago, Clientel said:

1. Just wondering if it is possible to use the bind function (BIND=XXXX:XX:XX.X|XXXX:XXX) to force unraid not to use a specific bus line so that it can be bound to a chosen VM.

 

You have to bind an entire USB controller along with anything else in that IOMMU group

 

16 hours ago, Clientel said:

I'm not sure how it's even possible that I can divide multiple USB mice/keyboards across 2 or more unique VMs but it does work for whatever reason even though they use the same IOMMU group

 

There are two ways to pass USB devices to a VM. You can bind an entire controller to vfio-pci and pass the entire controller to the VM (this works best, as the VM has full access to the controller and can tell when devices are unplugged/plugged) or you can check the boxes to pass individual USB devices (great if you just want to pass one thing and you don't need to unplug it)

 

16 hours ago, Clientel said:

If anyone knows why I am able to passthrough and bind these successfully yet not have the USB ports recognise any and all types of devices plugged in please let me know.

 

If the devices are successfully bound to vfio-pci (check the log after rebooting) then Unraid should be completely out of the picture. Might be worth starting a new thread with diagnostics over here: https://forums.unraid.net/forum/51-vm-engine-kvm/ 

 

16 hours ago, Clientel said:

Another frustration I've had with Unraid 6.91 is the lack of update to the VM XML when usb devices are unplugging and/or changed. I shouldn't have to manually enter the XML everytime I unplug a device that is passed through to a given VM to stop VM creation and Vendor issues.

 

I believe the solution for this is to pass through the entire USB controller. The KVM board linked above might be able to provide more info

 

On 8/3/2020 at 7:50 PM, Keexrean said:

Hi! Nice. Still on unraid 6.8.1 right now, and I managed to passthrough a quadro card without stubbing with pci-stub.ids= like I used to, or using the VFIO-PCI Config plugin, it handled it gracefully with just pcie_acs_override=downstream and type1.allow_unsafe_interrupts=1.

 

Though I'm facing an issue with onboard NIC and the VFIO-PCI Config plugin.


Dell poweredge R720 here, using 2 other (better) network cards to actually connect the host around to network and back to back stuff, I would have liked to use all 4 'onboard' ports for some pf-sense-VM and routing tests.

 

So I went on and used the VFIO-PCI Config plugin to stub all 4 ports (cause they each appear as their own subdevice) 

but as you can see on that screenshot, UNRAID for some reason keeps grabbing and using 2 of the 4 ports, for NO reason, since, atm, and at boot, no ethernet cables were even plugged in these, and they were all unconfigured and port-down interfaces.

 

 

In network setting, showing in Mac address selection are eth6 and eth7, the two "derping" ports 01:00.0 and 01:00.1, but for some reason only one of the two is actually showing up at all as an available and configurable port (which it shouldn't at all, I don't want them grabbed by unraid)

 

 

And check that sweet iDrac I can see how the onboard management sees the derping, please note they are in reverse order between IOMMU and unraid/idrac:
port one, used to be eth4, good, not grabbed:

port two, used to be eth5, good, not grabbed:

port three, corresponding eth6, half a$$ grabbed, but no difference seen on idrac:

port four, corresponding eth7, fully grabbed, and seen as functional

 

 

I just don't get how it can fail, but since these devices are reset capable, it would be handy to know if there is a way to tell unraid "Bad! Bad hypervisor! Now sit!", and forcefully unload the devices without causing a kernel panic.

If there is, that could be a life-saving option in the up-coming 6.9, to be able to tell unraid to just auto-unload some devices after booting, when they are known to hook themselves up for no reason.

 

Please note, and I repeat, there are no cables plugged in these 4 ports, nothing configured to hook to them, and iDrac has its own dedicated port that isn't linked to the 'onboard' NIC (which in fact is on a mezzanine card.)

 

If you could light my lantern there, I have no explanation of why Unraid is acting stubborn with this NIC, while handling GPU passthrough so blissfully on the other hand.

PLEASSEEEEEEE How did you fix it. I have this problem with fresh installs also

 

Edited November 15, 20214 yr by mojotaker

On 11/13/2021 at 6:15 PM, mojotaker said:

PLEASSEEEEEEE How did you fix it. I have this problem with fresh installs also

 

I'd recommend you start your own thread in the KVM area:
  https://forums.unraid.net/forum/51-vm-engine-kvm/ 

Be sure to include your diagnostics and details about your hardware, along with this screenshot and anything else you think might be relevant.

On 11/14/2021 at 2:15 AM, mojotaker said:

PLEASSEEEEEEE How did you fix it. I have this problem with fresh installs also

 

 

I didn't. I just slapped in a shitty 20bucks 4x1gbps NIC in PCIe, because the Intel Mezzanine card for Dell servers is dogshit, and the broadcom one (which is the brand of most of my PCIe NICs which work flawlessly) for the R720 isn't cheap and an expense I don't want to do.

Edited November 24, 20214 yr by Keexrean

On 3/18/2021 at 12:12 PM, ljm42 said:

 

Sorry, I am not sure. The screenshots show that the devices are bound to vfio-pci, so Unraid isn't holding onto them. But I don't know why the VM fails to start. I'd suggest posting a new topic over here: https://forums.unraid.net/forum/51-vm-engine-kvm/  Be sure to include your diagnostics ( Tools -> Diagnostics )

I see that @reubenjack never made another post in the other form as requested, but I am having the EXACT same issue, and I have an Intel card.

 

@ljm42 I am about to post my issues there.

On 2/16/2021 at 6:58 PM, ljm42 said:

 

At some point in the past you added this to your syslinux file:
  vfio-pci.ids=8086:150e
This is the old way of stubbing a device so that Unraid will not install a driver for it.

 

In 6.9 we do that using the webgui.

 

First go to Main -> Boot Device -> Flash and choose "Flash backup". This will give you a zip file of your settings "just in case"

 

Then go to the Syslinux tab on that page and remove "vfio-pci.ids=8086:150e" from this line:
  append initrd=/bzroot vfio-pci.ids=8086:150e
so it looks like this:

  append initrd=/bzroot 

 

Hit Apply, but don't reboot yet!

 

Then go to Tools -> System Devices and put a checkmark next to device "8086:150e" and click "Bind selected to VFIO at boot"

 

Now reboot. When it comes back up, the System Devices page will be in control of what devices are stubbed to VFIO-PCI. You can press "View VFIO-PCI Log" to see exactly what it did for that device while booting.

Thank you SO MUCH for this. Really helped me solve issue I was having.

Hi all, What is the best way installing VM directly on NVME (baremetal)? Should I use vfio-pci passthrough or by-id?

 

I need 2 partitions/ 2 OS on my 1 TB NVME.

 

Thanks,

wing

Edited October 8, 20241 yr by winglam