vw-kombi Posted March 19, 2021 Share Posted March 19, 2021 Long story short - configured a new pfsense VM for the new 2.5 release. When it booted, it only offered two of the usual 4 ports for config. I checked all the passthrough stuff, checked they are not showing in unraid, even did an unraid restart - each time only two showed. So I rolled back to my previous VM, and it wanted to run the interface setup again - as it had lost two ports. as I have WAN, LAT and OPT1 networks I need at least one extra port..... I have rolled back to my laptop built as an emergency pfsense device to get the internet working again - but that means I dont have a spare..... Any ideas what I can do to try and find out where these went to ? Diagnostics attached. As the original VM was never touched, I am not expecting a config issue here. tower-diagnostics-20210319-1626.zip Quote Link to comment
vw-kombi Posted March 19, 2021 Author Share Posted March 19, 2021 I have a bit more info. On the pfsense startup, It zips off the screen, but I can see this for example : igb2: Setup of shared code failed, error -2 igb2: IFDI_ATTACH_PRE failed 6 device_attach: igb2 attach returned 6 The rest of off the screen higher and I dont know how to see the pfsense startup messages to review. Quote Link to comment
vw-kombi Posted March 19, 2021 Author Share Posted March 19, 2021 I tried an install of opnsense also - same issue. Maybe card is faulty. Tomorrow I will shutdown and re-seat it - or move it to another PCI port maybe. I am considering just biting the bullet and buying a pfsense apppliance (AUD $350 for the cheapest). I will be travelling for 10 weeks soon and managing a VM for the internet may be a bit much for the kids. It has been bullet proof for a long while though. Quote Link to comment
vw-kombi Posted March 19, 2021 Author Share Posted March 19, 2021 now I am going crazy - only 1 interface is now showing of the 4 ports - igb0...... 1,2 and 3 are no longer available, and they are also not showing available in network interfaces in unraid, so i guess they are passed through to nowhere. Spaceinvaders vids are great to get cool tech stuff working, but when it stops working, I have no idea where to look to resolve. I only have one VM - pfsense. Quote Link to comment
vw-kombi Posted March 19, 2021 Author Share Posted March 19, 2021 Well, That was a fun afternoon. A comedy of errors. The issues seems to be as I removed the plugin vfio-pci-plugin in prep for the new unraid version as per instructions, and that took the PCI;s out, but not all at the same time, just 2, then a third. I removed it a while back but I guess the issue never took affect until I restarted the VM's, then did a reboot to work thogh these issues. I put the plugin back as part of these issues too, but did not think I needed to restuf using it, as I already had stubbing working beofre it, and removing it I did not think it actually pulled config out that would cause this scenario I had today. Oh well - thats half a day gone while I went around in circles. Sorry to bother everyone. 1 Quote Link to comment
ChatNoir Posted March 19, 2021 Share Posted March 19, 2021 No bother, thanks to have shared your findings. It might help others in the future. Quote Link to comment
vw-kombi Posted March 23, 2021 Author Share Posted March 23, 2021 Well, I am back again - having some time to re-do this again. After an Unraid restart, the interfaces are there, pfsense starts and they have to be re-configured again. The next VM restart, they are gone again - I only have two. Reboot unraid, and they come back, then they will go again. I cant be rebooting unraid all the time, I had an uptime of many months before I started this pfsense 2.5.0 work. Its not pfsense causing this - the same is on my original pfsense image. While you should never upgrade unraid to try and fix issues, I feel I will use the time instead to go to 6.9.1 and have fun and games there instead. There are vfio enhancements there - maybe better news for me. For now, back to the trusty old laptop, and considerations on muying a proper netgate pfsense device (at $350). Quote Link to comment
vw-kombi Posted March 23, 2021 Author Share Posted March 23, 2021 alright..... I upgraded to 6.9.1 I am posting this here still i I suspect the issue I describe below may be related to the 4 port NIC still, and not to the upgrade process. So - as usual, saved flash, copied syslinux.cfg to notepad, etc etc. Upgrade to 6.9.1 - all went in fine and all dockers started so all good. It has been years since I did the ACS override to BOTH to get my 4 port NIC into its acceptable iommu groups, so I figured I would remove that and see if anything has changed. So, I removed the ACS override (BOTH) on the GUI only and restarted, selected GUI (pcie_acs_override=downstream,multifunction) I then get boot issues (see scn shot) : SQUASHFS error : Unable to read data cache entry, i/o error's etc etc and it just stops. I took some video (too big to attach it seems) and a screenshot (attached). If I add back the syslinux.cfg pcie_acs_override=downstream,multifunction , it all boots as normal. I suspect the NIC is not being stubbed as it does not match, then the OS is doing something with the 4 port NIC ? No diagnostics as it never boots, just hangs (see screenshot attached). I am going to leave it as is for the rest of the day/night now - make sure 6.9.1 works as intended for everything - others have reported crashing etc and I dont want that added to my woes. Tomorrow, I will try the pfsense VM's again and see if anything has been made better with this OS upgrade. Quote Link to comment
vw-kombi Posted March 24, 2021 Author Share Posted March 24, 2021 ok - so 6.9.1 did not fix this issue (it was a long shot). I booted the pfsense vm and it looked good - saw 4 interfaces. configured them. Rebooted it a second time - still there. Shutdown that one and started up my original one that was never touched - only 2 interfaces shown. Stopped than and restarted the new 2.5.0 pfsense one - it has lost two and only had two showing. Very sad as this has been bullet proof for years like this and my primary router. Quote Link to comment
vw-kombi Posted March 25, 2021 Author Share Posted March 25, 2021 No-one offered any help so I have decided to dump pfsense. So I will be deleting the VM's for pfsense, removing the 4 port nic and free'ing up a laptop that was the backup pfsense device. I can buy 2 x Ubiquiti USG Security gateways for the price of one netgate pfsense appliance. I already use Ubiquity wifi so this will give me a nice load of features in the unifi controller. I have researched the USG devices and figured out what I can and cant do on it - specifically native multi location openvpn VPN's it seems. I know you can do stuff at the command level on USG to get that working, and I may get around to that one day. I will either move all my dockers back to the bridge network and then I can use the VPN dockers to route, or I will add a few VM's that have full time VPN turned on and run the stuff I need through them. So I will be deleting the VM's for pfsense, removing the 4 port nic and free'ing up a laptop that was the backup pfsense device. Quote Link to comment
vw-kombi Posted March 27, 2021 Author Share Posted March 27, 2021 Ubiquiti USG is in and operational. Prob about 4 hours to move about 2 years of pfsense config/tweaking onto it. I must say I am loving the interface. VPN's requirements now being done with windows VM's. Was initially worried about the IDS/IPS limiting routing to 85Mbps, but I can still saturate my 100Mbps internet plan. 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.