Hardware issue ? Or config ? Lost 2 ports of a 4 port NIC......


Recommended Posts

Long story short - configured a new pfsense VM for the new 2.5 release.  When it booted, it only offered two of the usual 4 ports for config.

I checked all the passthrough stuff, checked they are not showing in unraid, even did an unraid restart - each time only two showed.

So I rolled back to my previous VM, and it wanted to run the interface setup again - as it had lost two ports.

as I have WAN, LAT and OPT1 networks I need at least one extra port.....

I have rolled back to my laptop built as an emergency pfsense device to get the internet working again - but that means I dont have a spare.....

Any ideas what I can do to try and find out where these went to ?

 

Diagnostics attached.

 

As the original VM was never touched, I am not expecting a config issue here.

tower-diagnostics-20210319-1626.zip

Link to comment

I have a bit more info.

On the pfsense startup, It zips off the screen, but I can see this for example :

 

igb2: Setup of shared code failed, error -2

igb2: IFDI_ATTACH_PRE failed 6

device_attach: igb2 attach returned 6

 

The rest of off the screen higher and I dont know how to see the pfsense startup messages to review.

 

Link to comment

I tried an install of opnsense also - same issue.  Maybe card is faulty.  Tomorrow I will shutdown and re-seat it - or move it to another PCI port maybe.

I am considering just biting the bullet and buying a pfsense apppliance (AUD $350 for the cheapest). 

I will be travelling for 10 weeks soon and managing a VM for the internet may be a bit much for the kids.

It has been bullet proof for a long while though.

Link to comment

now I am going crazy - only 1 interface is now showing of the 4 ports - igb0......  1,2 and 3 are no longer available, and they are also not showing available in network interfaces in unraid, so i guess they are passed through to nowhere.  Spaceinvaders vids are great to get cool tech stuff working, but when it stops working, I have no idea where to look to resolve.

 

I only have one VM - pfsense.

 

Link to comment

Well, That was a fun afternoon.

A comedy of errors.

The issues seems to be as I removed the plugin vfio-pci-plugin in prep for the new unraid version as per instructions, and that took the PCI;s out, but not all at the same time, just 2, then a third.  I removed it a while back but I guess the issue never took affect until I restarted the VM's, then did a reboot to work thogh these issues.  I put the plugin back as part of these issues too, but did not think I needed to restuf using it, as I already had stubbing working beofre it, and removing it I did not think it actually pulled config out that would cause this scenario I had today.

 

Oh well - thats half a day gone while I went around in circles.

Sorry to bother everyone.

  • Like 1
Link to comment

Well, I am back again - having some time to re-do this again.

After an Unraid restart, the interfaces are there, pfsense starts and they have to be re-configured again.

The next VM restart, they are gone again - I only have two.

Reboot unraid, and they come back, then they will go again.

 

I cant be rebooting unraid all the time, I had an uptime of many months before I started this pfsense 2.5.0 work.  Its not pfsense causing this - the same is on my original pfsense image.  

 

While you should never upgrade unraid to try and fix issues, I feel I will use the time instead to go to 6.9.1 and have fun and games there instead.  There are vfio enhancements there - maybe better news for me.

 

For now, back to the trusty old laptop, and considerations on muying a proper netgate pfsense device (at $350).

 

 

 

 

 

Link to comment

alright.....  I upgraded to 6.9.1  I am posting this here still i I suspect the issue I describe below may be related to the 4 port NIC still, and not to the upgrade process.

 

So - as usual, saved flash, copied syslinux.cfg to notepad, etc etc.

Upgrade to 6.9.1 - all went in fine and all dockers started so all good.

It has been years since I did the ACS override to BOTH to get my 4 port NIC into its acceptable iommu groups, so I figured I would remove that and see if anything has changed.

So, I removed the ACS override (BOTH) on the GUI only and restarted, selected GUI (pcie_acs_override=downstream,multifunction)

 

I then get boot issues (see scn shot) :

 

SQUASHFS error : Unable to read data cache entry, i/o error's etc etc and it just stops. 

I took some video (too big to attach it seems) and a screenshot (attached).

 

If I add back the syslinux.cfg pcie_acs_override=downstream,multifunction , it all boots as normal.

 

I suspect the NIC is not being stubbed as it does not match, then the OS is doing something with the 4 port NIC ?

 

No diagnostics as it never boots, just hangs (see screenshot attached).

 

I am going to leave it as is for the rest of the day/night now - make sure 6.9.1 works as intended for everything - others have reported crashing etc and I dont want that added to my woes.  Tomorrow, I will try the pfsense VM's again and see if anything has been made better with this OS upgrade.

 

 

IMG_2323.jpg

Link to comment

ok - so 6.9.1 did not fix this issue (it was a long shot).

I booted the pfsense vm and it looked good - saw 4 interfaces.  configured them.

Rebooted it a second time - still there.

Shutdown that one and started up my original one that was never touched - only 2 interfaces shown.

Stopped than and restarted the new 2.5.0 pfsense one - it has lost two and only had two showing.

 

Very sad as this has been bullet proof for years like this and my primary router.

 

Link to comment

No-one offered any help so I have decided to dump pfsense.

So I will be deleting the VM's for pfsense, removing the 4 port nic and free'ing up a laptop that was the backup pfsense device.

I can buy 2 x Ubiquiti USG Security gateways for the price of one netgate pfsense appliance.  I already use Ubiquity wifi so this will give me a nice load of features in the unifi controller.

I have researched the USG devices and figured out what I can and cant do on it - specifically native multi location openvpn VPN's it seems.

I know you can do stuff at the command level on USG to get that working, and I may get around to that one day.

I will either move all my dockers back to the bridge network and then I can use the VPN dockers to route, or I will add a few VM's that have full time VPN turned on and run the stuff I need through them.  

So I will be deleting the VM's for pfsense, removing the 4 port nic and free'ing up a laptop that was the backup pfsense device.

Link to comment

Ubiquiti USG is in and operational.

Prob about 4 hours to move about 2 years of pfsense config/tweaking onto it.

I must say I am loving the interface.

VPN's requirements now being done with windows VM's.

Was initially worried about the IDS/IPS limiting routing to 85Mbps, but I can still saturate my 100Mbps internet plan.

 

  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.