NEW SERVER BUILD | NEW PFSENSE NEW PROBLEMS

[Pstark]

Built my new server (below) with the intention of running pfSense as a VM. I got everything up and running with minimal issue. The first issue I noticed was I didn't have a br0 network for my dockers. To fix: 'rm /var/lib/docker/network/files/local-kv.db"  then "/etc/rc.d/rc.docker restart" then created my docker network and everything seemed fine. But since doing this, things seem to not work as before the new build. For example I cant connect to photoprism webui. I spent most of yesterday trying to solve it to no avail. I restarted the server in the process and once again br0 is gone. Could pfSense be the cause of these issues?

 

H12SSL-I

EPYC 7302P

96GB EEC 3200MHz

X520-DA1 (was being used but currently waiting for transceiver to arrive)

X520-DA2 (pfSense)

GPU

LSI 16 port

[Pstark]

Diagnostics

tower-diagnostics-20230329-0827.zip

[Pstark]

One of the other problems I was having (pending if fixed) was some of my docker containers would error on starting the container saying the container doesn't exist. I found out that all the containers that were using "br0" network were now getting this error when attempting to start. The containers were still on the docker tab and I just went into the container and the network would be set to "None", I just changed it to any other network available, save and then go back in and change it to "br0". Problem fixed and run without issue.

 

I just don't know if I restart the server again I will loose br0 again?

 

Another thought I'm having: Since I'm running pfSense as a VM could it cause this issue since Unraid wont get an IP until after the pfSense VM boots?

[JonathanM]

1 hour ago, Pstark said:

Unraid wont get an IP until after the pfSense VM boots?

IMHO this is a situation where Unraid must have a static IP assigned, ideally you set a DHCP assignment for the server and also set the IP in Unraid. You can make a note in pfSense to remind yourself to update the static IP in Unraid if you ever change things around.

 

I also script it so other VM's and containers wait until pfSense is responding before they start. It takes a few minutes to get things all spun up after a reboot, but that only happens every few months.

[Pstark]

1 hour ago, JonathanM said:

IMHO this is a situation where Unraid must have a static IP assigned, ideally you set a DHCP assignment for the server and also set the IP in Unraid. You can make a note in pfSense to remind yourself to update the static IP in Unraid if you ever change things around.

So in pfSense it gives Unraid a static IP but in Unraid it gets it IP from DHCP server. So should I set Unraid (in Unraid network settings) to the same static IP? I thought this could cause problems.

 

1 hour ago, JonathanM said:

I also script it so other VM's and containers wait until pfSense is responding before they start. It takes a few minutes to get things all spun up after a reboot, but that only happens every few months.

I will look into this. Is it possible to do the same with Unraid networking?

[JonathanM]

4 minutes ago, Pstark said:

set Unraid (in Unraid network settings) to the same static IP?

That's what I do. I've never had any issues with it.

 

5 minutes ago, Pstark said:

I will look into this. Is it possible to do the same with Unraid networking?

Not sure what you mean, I'm running pfSense as a VM in Unraid.

[Pstark]

10 hours ago, JonathanM said:

I'm running pfSense as a VM in Unraid.

I was running it as a VM as well. It was painful getting it running the first time but today I received a new drive and had to stop the array (loosing pfSense) then I couldn't connect to Unraid anymore. I connected directly to Unraid but the connection very inconsistent. I gave up on that and used my old router, finished the new config but was never able to connect to the pfSense network. My adapter couldn't get an IP from the DHCP server. I think my pfSense vdisk may have been changed but I will investigate tomorrow.

[Pstark]

11 hours ago, Pstark said:

IMHO this is a situation where Unraid must have a static IP assigned, ideally you set a DHCP assignment for the server and also set the IP in Unraid.

I did this and then got DNS errors causing not access to the internet.

[JonathanM]

When setting a static IP on Unraid you must set the DNS manually as well.

 

When the pfSense VM is down, Unraid will only be accessible by IP, and the device trying to access it must still have a valid IP in the same subnet. If the device accessing is restarted, or the DHCP reservation times out you must have a static IP on that device as well as Unraid.

[Pstark]

On 4/1/2023 at 4:28 AM, JonathanM said:

When setting a static IP on Unraid you must set the DNS manually as well.

 

When the pfSense VM is down, Unraid will only be accessible by IP, and the device trying to access it must still have a valid IP in the same subnet. If the device accessing is restarted, or the DHCP reservation times out you must have a static IP on that device as well as Unraid.

I attached a screenshot of the settings I used when attempting to set a static IP. Unraid doesn't like my DNS settings but only with the static IP. pfSense gives the same DNS settings. I loose br0 after every reboot as well.

[JonathanM]

15 minutes ago, Pstark said:

pfSense gives the same DNS settings

Your screenshot shows different settings.

[Pstark]

7 hours ago, JonathanM said:

Your screenshot shows different settings.

pfSense uses cloudflare DNS servers. So Unraid wants the gateway IP for both primary and secondary DNS servers?

[MrGrey]

On 3/29/2023 at 11:21 AM, Pstark said:

Built my new server (below) with the intention of running pfSense as a VM.

Don't do that because it's (fill in). If you're an Unraid expert, VM expert, Network expert, and pfSense expert then it *might* be a cool experiment.

 

Imho, you want your Unraid server (bare metal) behind your pfSense (bare metal).

 

Of course, I could be wrong and you can call me a (fill in).

 

MrGrey.