I moved PCIE devices around so that my LSI card would receive better cooling. I also repasted it (not sure if this could be the cause) before putting it back in. First boot after moving these cards was fairly normal with the exception of my pfSense VM not booting. I plugged in a second router and navigated to the webUI and the VM was stopped. I hit start and error PCIE device missing. I physically checked the NIC assigned to the VM and the lights were flashing as if it were being used. (I have a second NIC that's assigned to eth0) I powered down the server and checked the NIC I passed through and reseated it. Then rebooted the server.

 

But UNRAID will no longer boot.... fully. 

 

Everything is normal during boot until, during the boot sequence while watching the log, the screen clears and then nothing. No drive activity or anything on the console. LSI card is initialized during boot but since I repasted it, I took it out and rebooted the server. It boots enough to ssh into it but no webUI. 

 

I'm thinking since the NIC being passed through was moved, it's causing issues with other devices but unsure on how to fix it. 

 

 

syslog

Try deleting/renaming /boot/config/vfio-pci.cfg

39 minutes ago, JorgeB said:

Try deleting/renaming /boot/config/vfio-pci.cfg

Thanks! This was what I was thinking I had to do but I didn't know what the file was called. 

 

I ended up moving my LSI card to another spot and it booted without issue. It's booting again after deleting the vfio-pci.cfg | my pfSense NIC wasn't found in system devices so I went ahead and deleted the file you referenced. Will update after the reboot. 

 

11 minutes ago, Pstark said:

my pfSense NIC wasn't found in system devices so I went ahead and deleted the file you referenced. Will update after the reboot. 

After reboot my second NIC it not being seen. 

Post diags please.

3 minutes ago, JorgeB said:

Post diags please.

 

tower-diagnostics-20230407-0056.zip

Unraid is currently detecting 3 NICs, should there be more?

10 hours ago, JorgeB said:

Unraid is currently detecting 3 NICs, should there be more?

Yes there should be an Intel DA2 SFP+ NIC

Isn't that this one:

 

01:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
    Subsystem: Intel Corporation Ethernet Server Adapter X520-1 [8086:0006]
    Kernel driver in use: ixgbe
    Kernel modules: ixgbe

 

And there are the two onboard NICs

 

45:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe [14e4:165f]
    DeviceName: Broadcom 1G Ethernet #1
    Subsystem: Super Micro Computer Inc H12SSL-i [15d9:165f]
    Kernel driver in use: tg3
    Kernel modules: tg3
45:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe [14e4:165f]
    DeviceName: Broadcom 1G Ethernet #2
    Subsystem: Super Micro Computer Inc H12SSL-i [15d9:165f]
    Kernel driver in use: tg3
    Kernel modules: tg3

 

17 hours ago, JorgeB said:

01:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
    Subsystem: Intel Corporation Ethernet Server Adapter X520-1 [8086:0006]
    Kernel driver in use: ixgbe
    Kernel modules: ixgbe

This is my eth0. This is good and not a problem.

 

17 hours ago, JorgeB said:

45:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe [14e4:165f]
    DeviceName: Broadcom 1G Ethernet #1
    Subsystem: Super Micro Computer Inc H12SSL-i [15d9:165f]
    Kernel driver in use: tg3
    Kernel modules: tg3
45:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe [14e4:165f]
    DeviceName: Broadcom 1G Ethernet #2
    Subsystem: Super Micro Computer Inc H12SSL-i [15d9:165f]
    Kernel driver in use: tg3
    Kernel modules: tg3

These are the onboard NIC's from the Supermicro board that I don't want to use.

 

There should be another X520 2 port sfp 10G NIC but I cant get it to show up after moving it. I don't know if the vfio-pcie is stored somewhere else and it's still being stubbed. I cant boot the VM without this NIC available.

tower-diagnostics-20230408-1515.zip

UPDATE;

 

I moved the NIC that wasn't populating in UNRAID for a 3rd or 4th and then it showed up. I'm going to move pfSense off the server and use an old hardware I have laying around to make a dedicated pfSense box. I'm sure my family will appreciate it since the network won't go down if I reboot my server anymore. 

Now I cant get the same NIC to populate in the new setup. I shutdown the Unraid server pulled the NIC put it in the new one and pfSense doesn't recognize the interfaces. Do I need add a driver? I know this isn't a pfSense forum but it related to this thread and I know a lot of Unraid users use pfSense.

I have tried everything I can think of but I cant get this NIC to be recognized.