Twice now my unRAID server starts acting odd. I notice it when all my Dockers show that Unavailable version notice instead. If I try to ping to 1.1.1.1 or anything beyond my firewall, they fail. Curls outbound fail to connect as well. Looking at my firewall logs, everything from the server is being broadcast to 10.15.0.255 over nbdatagram (UDP/138). I can ping anything internally no problem. The only thing I have found to resolve this is to reboot my unraid. I don't want to get into the habit of weekly reboots. Is there anything within the attached diags reports that pinpoint the issue?

 

Diag ending in 904 is *during* the issue.

Diag ending in 913 is *after* the reboot and things are working. And my firewall logs show traffic egressing over all the expected ports.  

unraid-diagnostics-20221208-0913.zip unraid-diagnostics-20221208-0904.zip

Edited December 8, 20223 yr by UncleStu
fix typo

in the logs, where the failure had occurred, I might suggest that you search for:

 

 err

Note:  there is a   <space>   at the beginning of that string....

Then look for the second occurrence and you will find a general protection fault.  This fault is never good news,,,

In the syslog log, I see 

Dec  1 22:36:50 unRAID kernel: traps: PMS GTP[26851] general protection fault ip:153c257137d0 sp:153c1622f348 error:0 in ld-musl-x86_64.so.1[153c25703000+53000]

The " err" before this was in regards to the VM Backup plugin not connecting to my Synology NAS in a timely fashion. But I don't think the blocked traffic is related to this fault. Unless it takes a few days to manifest. Continuing looking at the syslog, the first Fix Common Problems error I see if Dec 7th, where it couldn't communicate with github. So between the 1st and 6th, everything was fine. Somewhere on the 7th, or night of the 6th, things went South

 

The only other "error" in the logs, during and after the egress traffic being blocked is a time sync error.

unRAID ntpd[1424]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized

 

Edited December 9, 20223 yr by UncleStu

I looked more at my firewall logs. The last good traffic was at 12:10 AM (PT) and the first broadcast traffic start at 12:20 AM. Looking at the syslog for this time, the Mover finished at 12:00 AM and then my HDD drives started sleeping at 1:35 AM. Still don't know the root cause.

My server lost its capability to egress out again tonight. I think I've narrowed it down, but I don't know why. I upgraded my firewall software today so the server was without internet access for about 45 minutes. I cannot sign into the My Server plugin. Any plugin or docker update checks fail, but I can successfully access my reverse proxy and I have active external plex streams going on. A simple reboot will resolve this but is there another way that I can/should be restarting the network services? Any idea why the server just stops getting out after my firewall is down for a little bit?

 

Why would a traceroute from the server start and stop with itself?

 

Is the default route listed as a FQDN the issue? It is not defined this way within my settings.

Edited December 15, 20223 yr by UncleStu
fixed a typo

I stopped all my dockers, and the docker service, and outbound connectivity was restored. Any hints as to what to look for within my dockers to pinpoint the issue?

 

Restarting the docker service and images and everything is still working as expected.

Interesting that restarting Docker resolves the problem. Next time it happens, try restarting just the network - at a web terminal type:

inet reload

 

The only odd thing I see in your syslog from the 6th are error messages about IPv6. Do you actually need IPv6? If not, you might disable that just to simplify things.

10 minutes ago, ljm42 said:

error messages about IPv6

That is very odd. I have it set to IPv4 only. Is there somewhere else I need to check/disable it?

 

Interesting. I see IPv6 messages on my system too. We might need to make some changes, but I'd be surprised if that ends up being your issue.

1 hour ago, ljm42 said:

I'd be surprised if that ends up being your issue

Likewise.

 

I reviewed my logs from the time I took my firewall offline for the update to when it returned, and all the way through the end of the logs. Nothing seems to indicate why it went offline and stopped knowing how to get to the gateway. In my last docker service restart I changed my "Host access to custom networks:" from enabled to disabled and it appears to have removed two routes from the table. 10.15.0.0/25 and 10.15.0.128/25 are no longer listed. I don't see how these would affect the server itself from getting to its gateway though. But possibly related since the stopping of the docker service restored the traffic flow?