*VIDEO GUIDE* A comprehensive guide to pfSense both unRAID VM and physical


SpaceInvaderOne

Recommended Posts

Hi have long time think I will give it a try so order a 4 port nic ... just ofcores forgot I don't have any more free pcie so in the video you talk abut if you only have one nic I try  google but I do not even get a ip  .
I have a other pc I can try it with but will love not too and keep all in my unraid  and just with Vlan .
if it work and I have not misunderstand  
my setup are
ISP modem router bridge mode to my netgear r7800  ( it can run Vlan)  > ether a cheep switch  or direct in the unraid box 

 

Link to comment
On 1/24/2019 at 3:33 AM, jetkraus0 said:

Hi, I love the guide, however, I am having an issue with starting the VM. I am on step 3.

This pops up

internal error: process exited while connecting to monitor: 2019-01-24T03:29:45.614726Z qemu-system-x86_64: -device vfio-pci,host=07:00.0,id=hostdev0,bus=pci.3,addr=0x0: vfio error: 0000:07:00.0: failed to setup container for group 15: failed to set iommu for container: Operation not permitted

 

Thanks

It looks like the nic that you are passing through isnt in its own iommu. You will need to break it up or passthrough everything in that group. have a look at my vid here https://www.youtube.com/watch?v=qQiMMeVNw-o&t=791s

 

Link to comment

Hi,

relate to Part 3 install and basic config. I have two networkcards. One in the mainboard and one in the pcie slot. But all Ports have the same ID. What is the solution to continue the tutorial if you can't take the ID like in the video, because they all have the same ID ?

I don't want to buy a new networkcard..

id.PNG

Edited by Abigel
Link to comment
  • 2 weeks later...

Just checking to make sure I'm understanding properly.

 

My current unRAID server has a single NIC with 4 physical ports.  I only use one at the moment for a single connection of unRAID to my LAN.

 

I assume that I still need an additional dedicated NIC for pfsense in order to avoid having to work with VLANs?  There's no practical way to isolate three of the interfaces and use those for pfsense and allow a single interface be used as unRAID's method of connecting to the LAN?  My guess is because they all share the same device # it's an all or nothing proposition, but figured I'd ask.

Link to comment
  • 3 weeks later...

@Living Legend: I think you have it correct.  I have a similar setup in my unraid server.  I pass the entire 4 port NIC to my pfsense VM.  I don't think the 4 ports can be split up very cleanly between VMs and unraid, as all 4 ports are in the same IOMMU group.  I suppose you could try to break that IOMMU group up (as spaceinvader1 mentions in video 3, I think) but that may or may not work.  In my case, my server has another ethernet port on the motherboard, so I use that for unraid.

Link to comment

I am a bit confused about how pfsense, the binhex delugevpn docker, and PIA port forwarding all work together. 

 

The introductory pfsense video discusses a few PIA VPN connections configured in pfsense, 1 of which (Germany, which was a port-forward enabled PIA server at the time the video was made) is used for downloads and such.  However, the slightly older delugevpn docker setup video discusses using the VPN connection built into that docker. 

 

So, if all of the VPN connections are made in pfsense, how does the deluge docker know which port is forwarded from the PIA server?  Does that all happen automatically?  The PIA port forwarding documentation mentions using either the PIA desktop client or an API and scripts to request a forwarded port from the PIA server.  But I'm not sure how that would work in pfsense?

 

I know how pfsense could be configured to pass the delugevpn docker straight thru to WAN, and just use the built in VPN connection in that docker, but I would prefer for pfsense to handle all the VPN connections, if possible.  Just not sure how the port forwarding fits in.

Link to comment

@Living Legend: Hey, I was looking at my server this afternoon and realized I was a bit off in my last post.  For my 4 port NIC, each port is actually in its own IOMMU group.  So that isn't the issue.  However, the method of passing the PCI device to the VM relies on the PCI ID, which is of course the same for all 4 ports of the NIC, since they are all on the same PCI device.

 

So my details were a bit off, but end result is the same: you had it correct, its an all or nothing situation with a multiport PCI NIC.

Link to comment
  • 2 weeks later...
On 2/20/2019 at 4:53 PM, diarnu said:

@Living Legend: Hey, I was looking at my server this afternoon and realized I was a bit off in my last post.  For my 4 port NIC, each port is actually in its own IOMMU group.  So that isn't the issue.  However, the method of passing the PCI device to the VM relies on the PCI ID, which is of course the same for all 4 ports of the NIC, since they are all on the same PCI device.

 

So my details were a bit off, but end result is the same: you had it correct, its an all or nothing situation with a multiport PCI NIC.

 

Interesting, because while I did ultimately buy another quad port NIC for cheap on eBay, I'm now fairy certain that I could have managed without one.  When I installed the additional NIC, I now had 8 ports, all with the same ID, so my problem still existed.  I followed this thread to resolve:

 

 

This method allows the user the select specific PCI numbers  to pass through rather than device IDs.  Then these specific PCI #s can be utilized in the VM by manually editing the VM XML file.

 

Edited by Living Legend
Link to comment
  • 3 weeks later...

My UNRAID server is headless, situated in a utility closet. I monitor it using IPMI... if i need to reboot etc.

My network architecture at the moment is basically as follows:

 

----[MODEM]----[Netgear router + wifi]-----[16 port switch]----[ {UNRAID}, {MAC}, {PC} {etc}]

 

(The ISP modem is in bridge mode)

So what if i remove the Netgear router and setup pfsense on a VM in the UNRAID server? 

 

----[MODEM]----[UNRAID/PFSENSE]-----[16 port switch]----[ {MAC}, {PC} {etc}]

 

(the old netgear wifi router will be added as an access point)

But... how do i keep IPMI working this way? 

If the UNRAID server is turned off (over IPMI!)... where does it get an IP address?

Link to comment

Looks like the BIOS itself is giving IPMI an IP address...? I've disconnected my router, and made sure my Mac Mini and IPMI were connected to the same switch. I've turned OFF the unraid server (over IPMI). So there was no DHCP, no WAN, no internet anywhere on the network. But... i could still connect to the UNRAID server from the MacMini over IPMI and boot it up. So i guess PFSense is still an option :)

Link to comment
  • 2 weeks later...
On 3/20/2019 at 10:18 AM, jowi said:

Yes, i know... but again, if i remove my current router and use PFSense on my UNRAID machine as a router, and i turn my UNRAID machine off... so PfSense is NOT running, then there is nothing connected to the IPMI interface? 

I'm really curious about this. How does it work? I'd like some more technical network advice. If the VM is the only router, how do the different peripherals still communicate between each other when unRAID is turned off? Let's say it's a 4-port PCI card with 1 port set as WAN and the 3 other as LAN. The WAN port has the modem, 2 of the LAN ports have the IPMI interface and unRAID (motherboard NIC) and the last one has the only other computer. How can that computer communicate with the IPMI interface if the tower is shut down, can communication actually go through the PCI NIC even though it doesn't have power? I'm confused.

Link to comment

I think it is mandatory to use a switch to connect all your devices, including unraid, so at least the (static?) devices can 'see' each other if the unraid server + pfsense is turned off. Also connect the IPMI from the unraid server to the switch so you can manage the server from another pc even if the unraid server is turned off. IPMI will give itself a static ip anyway as it seems (see pic, eth0..3 is quad nic, IPMI/LAN is internal supermicro nics, all other devices connect to the switch as well, AP is my old wifi router as an access point, upperleft is the cablemodem)

 

Knipsel.PNG

Edited by jowi
Link to comment
  • 2 weeks later...

I have set up the above topology. Unraid is running pfsense, and my DD-WRT router is now configured as an access point. So far so good. All devices can connect, eiter using the LAN (192.168.1.x) or WIFI (192.168.2.x) subnets. I can also access the unraid GUI on subnet 192.168.1.x on my (wifi) ipad which is in subnet 192.168.2.x. Also good. Me happy.

 

But... if i turn on my vpn on my ipad (goose vpn, in app form on the local device like pia/nordvpn etc) i can surf the internet etc but i can not acces the 192.168.1.x subdomain anymore? So i can not enter unraid's webgui etc. If i turn off the (local) vpn, all is ok... btw, if i use the vpn on my wired mac mini, there is no problem entering the unraid gui. Both mac-mini as unraid are on the same subnet.

 

How do i change this?

Or is this a DD-WRT setting i'm missing?

 

*edit1*

i also can not acces my receiver and apple tv (both are in 192.168.1.x) from apps on the ipad (192.168.2.x)... this must be related?

How do i make both subnets 'see' each other for all devices (not just some?)

 

*edit2*

Fixed the subnet-could-not-see-eachother issue by defining a bridge between them. Don't know if that is the best solution but it works. Also tried rules that would allow traffic from subnet a to b and vice versa but did not work.

 

The (local) vpn issue remains.

 

Edited by jowi
Link to comment

Not much pfsense knowledge here i guess... it's a complex piece of kit. Hard to understand the concepts.

 

I did manage to set up an openvpn client for my vpn provider (GooseVPN in the Netherlands) and it is up and running. But... not the way i want, and not anything like the tutorials on it. Usually once you have the openvpn client set up, you add a new interface for it, and then use that interface as the gateway for your other interfaces, like LAN and LAN2 (where LAN2 in my case is igb2, and has a DD-WRT wifi router attached to it as AP). Also you have to add rules for both LAN/LAN1 that allow the vpn to work...

 

But... if i set this up this way, only my LAN is working over the openvpn, and my WIFI (LAN2) is not. LAN2/WIFI can't connect to anything anymore...

 

So what i did now is just add the openvpn client, and once that is up, i set the outbound NAT rules, and from then on both my LAN and LAN2 are routed through the vpn... which is ok, but it is not working like the tutorials say it should with using the gateways and whatnot.

Link to comment
Not much pfsense knowledge here i guess... it's a complex piece of kit. Hard to understand the concepts.
 
I did manage to set up an openvpn client for my vpn provider (GooseVPN in the Netherlands) and it is up and running. But... not the way i want, and not anything like the tutorials on it. Usually once you have the openvpn client set up, you add a new interface for it, and then use that interface as the gateway for your other interfaces, like LAN and LAN2 (where LAN2 in my case is igb2, and has a DD-WRT wifi router attached to it as AP). Also you have to add rules for both LAN/LAN1 that allow the vpn to work...
 
But... if i set this up this way, only my LAN is working over the openvpn, and my WIFI (LAN2) is not. LAN2/WIFI can't connect to anything anymore...
 
So what i did now is just add the openvpn client, and once that is up, i set the outbound NAT rules, and from then on both my LAN and LAN2 are routed through the vpn... which is ok, but it is not working like the tutorials say it should with using the gateways and whatnot.
Are you referring to spaceinvader tutorial?

Sent from my SM-N960U using Tapatalk

Link to comment

I followed this guide to install pfsense as vm on unraid and the required steps to passthrough the NIC (Intel Pro/1000 dual port), but when I try to boot the pfsense VM I get this error:

 

"internal error: process exited while connecting to monitor: 2019-04-20T15:41:25.512263Z qemu-system-x86_64: -device vfio-pci,host=10:00.0,id=hostdev0,bus=pci.3,addr=0x0: vfio error: 0000:10:00.0: failed to setup container for group 17: failed to set iommu for container: Operation not permitted"

 

I also tried the 'enable pcie acs override" option which did not seem to help.

 

Group 17, 18 is my card in 'system devices':

"IOMMU group 16:[8086:10d3] 0b:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

IOMMU group 17:[8086:105e] 10:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper applications) (rev 06)

IOMMU group 18:[8086:105e] 10:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper applications) (rev 06)

IOMMU group 19:[8086:10d3] 15:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection"

 

Can someone help me get this working?

 

UPDATE: this fixed my issue:

https://forums.unraid.net/topic/37959-guide-passing-through-network-controllers-to-unraid-6-virtual-machines/?do=findComment&comment=592661

Edited by guruleenyc
Link to comment

I could get the quad nic to be in a separate group using the guide, but in the end i just moved the quad nic card to another physical slot, and after that i did not have to configure anything, it was in a separate group natively. Did you try that?

Link to comment
13 hours ago, jowi said:

I could get the quad nic to be in a separate group using the guide, but in the end i just moved the quad nic card to another physical slot, and after that i did not have to configure anything, it was in a separate group natively. Did you try that?

The nic is in separate groups already without override. Would I still need to move my card? 

Link to comment
  • 2 weeks later...

I've been told that my config, with the netgear dd-wrt ap connected to a separate NIC (OPT1, WIFI see above pic), makes no sense, if i use a bridge to connect LAN and WIFI...  It would be better to just add the access point to the switch and make it a part of the LAN network. Does that makes sense?

Edited by jowi
Link to comment

Hey jowi.

 

So from your diagram, it looks to me like the goal was to keep WIFI and LAN clients separate.  If that is not the case, the simplest solution to get the WIFI and LAN clients to talk to each other is probably to move the AP to the switch.  This has the added benefit of only needing NAT and firewall rules to route clients thru the VPN on 1 interface (LAN) rather than 2 (LAN + WIFI).

 

You could keep the AP on a separate interface and use firewall rules to pass traffic between the subnets, but that seems way harder than just moving the AP to the switch.

 

My way of thinking about it: the biggest reason to have different clients on different subnets in a small setup is for them to NOT communicate.  For example, I have different HOME, GUEST, etc. subnets (via VLANs, not different hardware interfaces, but functionally about the same).  Clients connected to the GUEST wifi cannot see my server on the HOME subnet.  And I like it that way :)

 

Hope that helps.

 

P.S. based on your earlier posts, I'm not sure how the iPad on the WIFI subnet ever saw the server on the LAN subnet ...

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.