*VIDEO GUIDE* A comprehensive guide to pfSense both unRAID VM and physical


SpaceInvaderOne

Recommended Posts

6 hours ago, jowi said:

I've been running pfsense in a vm on a headless UNRAID server for some time now, and it works great... until the server has an issue. And then you can't do anything... you can't reach pfsense, you can't ssh or even IPMI into unraid... you don't have network, you don't have internet. The only thing you can do is turn the server off HARD and pray that your disks are ok...

 

And IF you are rebooting, you can't IPMI to enter bios or even see bios etc because you can reach the server only after it has booted and pfsense is started... it's a great way get into pfsense, but i'm gonna go the dedicated pfsense hardware route as soon as i can.

That's why you keep a basic router programmed to get your basic network back up in a pinch. No port forwards or VPN, just basic NAT with dhcp.

Link to comment

Yeah, i know, and i do. But then you have to rewire everything... and there is not much room in the closet. 

 

Maybe i just wire it up anyway, so when it is needed, i only have to turn it on, and switch the cable modem connection to the emergency router in stead of the unraid server... but then what happens if i reboot the unraid server and the pfsense vm starts up? Then there are 2 routers with ip address 192.168.1.1... or can i just use e.g. 192.168.1.1 for pfsense and 192.168.1.2 for my failover router, and just disable dhcp on the failover router while pfsense is running?

I would like to 'switch' if unraid/pfsense is down for some reason, without a lot of recabling. Just unplug the cablemodem from pfsense, and plug it in the failover router, enable dhcp and get on with it? Is that possible?

Edited by jowi
Link to comment
  • 3 weeks later...

So after burning my brains i managed to get everything working  except the last part:

ssh admin@10.10.20.1 /etc/rc.halt

You will need to generate some ssh key pairs on unRAID and copy the public key to the admin user in pfsense

i have generated my keys pasted the public version into the admin user of pfsense ssh is enabled and works have tested this in puTTY. i created a user script with "ssh admin@192.168.1.1 /etc/rc.halt" no quotes and ip address being the lan IP of pfsense can anyone tell me the last part im missing to make this work

Link to comment

UPDATE @ 3:20pm Mountain: I've finally managed to at least get the Intel dual port 1Gbps NIC passed through to the pfSense VM and for the 1st time I was able to reach the pfSense menu and configure the interfaces. It's not seeing my HP NC523SFP dual port 10Gbps NIC but that's probably a driver/module issue that I can update in the pfSense VM. I've read that others have used the HP (QLogic) 10Gbps adapters so I'm sure I'll eventually figure it out

 

UPDATE #2: did some testing and found that pfSense fails to detect the NICs with QEMU 4.x. QEMU 3.1 is the version that I'm currently using and so far pfSense seems stable.


As for the current solution that enabled the Intel NIC: it was as simple as changing the machine type from Q35-4.2 to Q35-2.11 - saw this in one of the many threads that discuss passthrough. It was the only change made to get it to recognize at least the Intel NIC.

 

Original post:

 

I'm having some issues with getting my NICs passed through to the pfSense VM. I've tried both the vfio-pci.cfg/BIND and the vfio-pci.ids=xxxx.yyyy methods. I have two dual port NICs, one an Intel 82571 dual port 1Gbps and the other a HP NC523SFP dual port 10Gbps. Both methods do seem to bind the cards to vfio as unRAID no longer sees them under Network Settings.

 

The initial creation of the VM as per part 3 of the videos works as expected. When the VM reboots at the end of the install, pfSense loads from the vdisk but fails before getting to the pfSense menu options. The error message reported states that 'Configuration references interfaces that do not exist (em0, em1).

 

pfSenseIssue.thumb.jpg.6a07165d33d0496be4c8c093d809b8db.jpg

 

I never get to the ‘interface assignment’ stage…. I’ve also tried with PCIe ACS override and VFIO allow unsafe interrupts options in VM Settings. When I reboot the VM and hit escape at the loader menu I can issue an 'lsdev' command and it shows the following:

 

pfSenseIssues2.thumb.jpg.4e8f0db7152adfd3bd5e3c9e6211bdb3.jpg

 

Are then net0: and net1: devices 2 of my 4 NIC ports? Where are the other two? I tried with just the Intel dual port card and saw the same result. I then tried with just the HP NC523SFP card by itself, also yielding the same results.

 

As another note, when I try to review the logs of the VM boot, it’s very short and only has the last couple of screens of info prior to the halt. How do I increase the log size for my VM so I can capture all of the messages from start of the boot through to the halt. At least then I can possibly find a cause for my issue.

 

Here's the IOMMU list for my NICs:

 

pfSenseIssues3.thumb.jpg.c9040666a7c9cb045e0942f6363cfe77.jpg

 

The onboard dual 1Gbps NICs have a different PCI ID and they are still visible to unRAID and working in a failover bonded pair. Any suggestions on what I need to do to overcome the halt issue for the pfSense VM? TIA!

 

Edited by AgentXXL
Cleanup formatting and add missing info
Link to comment
  • 4 weeks later...

Hello.  When I launch PFSense for the first time I get stuck on the black screen where it says booting... and nothing else happens. 

 

image.thumb.png.f927f1b8a4f31273998a8c2261fb0ffb.png

 

I have attached my logs from my PFsense Vm.  I see the following errors 

 

2020-05-24 02:05:07.735+0000: Domain id=1 is tainted: high-privileges
2020-05-24 02:05:07.735+0000: Domain id=1 is tainted: host-cpu
char device redirected to /dev/pts/0 (label charserial0)
2020-05-24T02:05:18.252838Z qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:08:00.0
Device option ROM contents are probably invalid (check dmesg).
Skip option ROM probe with rombar=0, or load from file with romfile=
2020-05-24T02:05:18.254745Z qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:09:00.0
Device option ROM contents are probably invalid (check dmesg).
Skip option ROM probe with rombar=0, or load from file with romfile=
2020-05-24T02:05:18.256320Z qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:08:00.1
Device option ROM contents are probably invalid (check dmesg).
Skip option ROM probe with rombar=0, or load from file with romfile=
2020-05-24T02:05:18.257872Z qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:09:00.1
Device option ROM contents are probably invalid (check dmesg).
Skip option ROM probe with rombar=0, or load from file with romfile=

 

 

I did some searches and someone mentioned it has to do with the CPU pinning but I pinned 2 threads on 1 core.  I even tried switching the pinning to a different core.  I had 1 and 7 pinned but now I have 5 and 11.  

 

image.png.ce36f3456ba519c37f5448b86e88e75a.png

 

So Im a little lost here.  I used the vfio plugin to get passthrough working.  I also had to enable ACS override to get the IOMMU groups broken up. 

 

Here is my pfsense VM config

Part 1

image.thumb.png.6497a1f8941346c292aaac92d080de3e.png

 

Part 2 

image.png.279df92e2b975330ef0a16a059ed0593.png

 

I have attached my pfsense logs and my IOMMU group in txt files. 

 

Any help would be appreciated. 

 

 

PFsense logs.txt IOMMU Groups.txt

Link to comment
  • 3 weeks later...

Hey guys, I could really use some help.. I'm not sure what is going on. 

 

I'm trying to get my gbe nic passed through to my VM, but I don't think my syslinux config is working properly

 

This was my original Config

kernel /bzimage
append pcie_acs_override=downstream initrd=/bzroot pci=nomsi,noaer

image.thumb.png.3d64e1d1b23676199042fc8a7733c6e2.png

 

This is my edited config

First I tried 

kernel /bzimage
append vfio-pci.ids=8086:1079 pcie_acs_override=downstream initrd=/bzroot pci=nomsi,noaer

 

This seemed to have 0 effect, the devices were still in the network settings.

 

Then I tried below, as I saw it in a post

 

kernel /bzimage
append pci-stub.ids=8086:1079 pcie_acs_override=downstream initrd=/bzroot pci=nomsi,noaer

 

I can no longer see the controllers in my network settings, but I also cannot see them available to passthrough to the VM

image.png.59bc0a0eae5d5c7ecd6407d22926b055.png

 

Any idea's? 

 

Edit: I have just been editing the Unraid OS syslinux config, I assume this is fine as this is how I boot it

 

Edited by Addy
Link to comment
  • 2 weeks later...

Have followed along and when its time to assign interface ports, pfsense doesnt see that 4port NIC

This is my IOMMU groups

image.thumb.png.a650fcbe544abc422fae41fabaa715e2.png

I am assuming the ticked items in Group 1 is the 4 port intel NIC, but not sure if the remaining item in thatgroup are ascociated?

Do they need to be split?

 

I have tried the overide option on or off, this is what pfsense displays

image.png.bd2a24ee067cc165cd7c0323bb45010b.png

igb0 is the onboard nic?

 

EDIT: In the VM creation it automatically sets the machine type as Q35-5.0

Once i changed it to Q35-2.11 everything worked as per the video

And i did not have to set any pcie overrides. My intel nic in group 1 passed through no problems. pfSense now installed!!

Edited by bdydrp
Link to comment

I'd be super interested in getting more information about failover. Aside from a brief explanation in the first video I haven't found any other references to it.

 

I'm specifically curious about:

  • Are the configurations identical (e.g., did you take a backup from the primary pfsense instance and restore it on the failover device (obviously updating the interface assignments to WAN and LAN)? Have all the extra packages installed?
  • Does it require any different configurations of the network setup? For example, do devices on the network complain when the switchover occurs since DHCP assignments may get wonky? Does it require a different gateway?
  • Does the changeover require physically changing the modem network cable to the physical box, or do you have something going on with the switch to assign it its own VLAN?

If anyone can shed some light, that would be outstanding! Thanks!

Link to comment

OK - Dumb question

My current setup is:

VDSL modem (DHCP) > netgear router > switch

With pfSense, what the correct way?

VDSL > wan port on nic > lan port on nic > switch?

VDSL > switch > wan port on nic > lan port on nic > switch?

 

reason i ask , is that when setting up pfsense, and auto selecting wan/lan ports both these were connected directly to the switch, but pfsense didnt get an external IP from my ISP but instead an IP in my local range (192.168.1.xxx)

 

When i disconnect my current router, cant seem to get pfsense to get an IP from my ISP!!

Edited by bdydrp
Link to comment
2 minutes ago, bdydrp said:

OK - Dumb question

My current setup is:

VDSL modem (DHCP) > netgear router > switch

With pfSense, what the correct way?

VDSL > wan port on nic > lan port on nic > switch?

VDSL > switch > nic wan port > nic lan port > switch?

 

reason i ask , is that when setting up pfsense, and auto selecting wan/lan ports both these were connected directly to the switch, but pfsense didnt get an external IP from my ISP but instead an IP in my local range (192.168.1.xxx)

 

When i disconnect my current router, cant seem to get pfsense to get an IP from my ISP!!

 

I had pfsense prior, it is amazing!! But usually its ISP -> pfsense -> switch

pfsense handles everything else and can be configured for how you want it to be

Link to comment
1 hour ago, bdydrp said:

Sorry, in a VM!

I remember seeing an issue posted somewhere in the forums about this and I've been trying to find the page for this but no luck so far... I haven't fully gotten mines up in a vm yet, I'm mainly done it bare metal

Link to comment

all good

When i was setting up the vm, at the point of auto selecting wan/lan port, i connected a cable from my switch to these ports.

So i might delete and remove vm.

Then that time of auto selecting wan port, unplug current router and connect isp straight to wan port, then lan to switch.

See how that goes.....Nothing to lose!!

Link to comment
  • 3 weeks later...
On 4/20/2018 at 8:20 PM, SpaceInvaderOne said:

I am starting a series of videos on pfSense. Both physical and VM instances will be used. Topics such as using a failover physical pfSense to work with a VM pfSense. Setting up OpenVPN (both an OpenVPN server and OpenVPN multiple clients). Using VLANs. Blocking ads. Setting up squid and squid guard and other topics. T

 

This part is an introduction part gives an overview of the series of videos and talks about pfSense and its advantages.

 

 

 

Second part of is on hardware and network equipment

 

 

Part 3 install and basic config

 

 

 

Part 4 customize backup and aupdate

 

 

 

 

Part 5   DHCP, Interfaces and WIFI

 

 

Part 6  Pfsense and DNS

 

 

Part 7 - Firewall rules, Portforwarding/NAT, Aliases and UPnp

 

 

Part 8  Open NAT for XBOX ONE and PS4

 

 

So, 

 

How do we get the physical machine to turn on as failover when the PFSENSE VM failis?

 

Also, is there a way to sync settings between the two (I dunno... have both load from a network image or sonething?

Link to comment
  • 3 months later...

Hi,

 

Is there any chance to get this working on Ryzen with Unraid 6.9? I've not been able to install pfsense (without or with passthrough of my intel nic) on Unraid 6.8. Tested on Unraid 6.8 with Ryzen 1700/Asus Prime X370-Pro, Ryzen 2700X/Asus Prime X470-Pro and Ryzen 3900X/Aorus X570 Ultra.

I would like to shutdown the two pc I use to run pfsense and keep only Unraid ones.

Link to comment
  • 1 month later...

Hi Spaceinvader One,

 

Thanks for your excellent videos on pfsense and other helpful videos.

 

I'd like to share that pfsense now has a new developer version available that specifically doesn't require AES-NI. I have "new" processors underway that do support AES-NI but I might give that a try first, and go to a stable version, current 2.45 or later a stable version of 2.5.

 

Thanks again.

 

Cheers.

Link to comment
  • 2 weeks later...

Wondering if anyone has any tips around performance. It seems we get a penalty under a VM. I have 1000Mbps/50Mbps but the best I can get via Pfsense in Unraid is 750Mbps, this seems to be common for anyone running in a VM. Anyone had any joy?  (I can plug in directly and get full speed. Its related to the VM)

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.