Hi all, happy holidays.

 

I've noticed in the past week that my server has been more and more unreliable in terms of staying online. I've had this server running for close to 5 years and I've had sporadic issues in the past, but this one is really odd to me.

 

I will start my server, it'll start all its docker containers, and after that's done, it'll run smoothly. Then all of a sudden, it's unreachable, and so are all the containers.

 

Things I've tried so far:

 

- switching from macvlan to ipvlan

- removing unused docker containers

- removing non-essential User Scripts

- waiting ~10 minutes after boot before manually starting the array and then manually starting the docker containers another 10 minutes later (to see if this would have any effect, it did not, still crashed)

- setting up a syslog server on another computer to track any errors or warnings: noticed that after some time a bunch of "pool www child exited on signal 9 started appearing over and over", but unsure if related

 

I have attached the last 2 diagnostics I was able to grab, from two separate times the server crashed today.

 

Thank you in advance for the guidance!

kino-diagnostics-20241225-2330.zip kino-diagnostics-20241225-1825.zip

Edited December 26, 20241 yr by KINO
another thing tried

5 hours ago, KINO said:

setting up a syslog server on another computer

Please post that as well.

8 hours ago, JorgeB said:

Please post that as well.

Thank you for the response, Jorge.

 

I've attached the syslog as well. This is just since I setup the syslog server a few days ago, but the issue was happening before that too.

syslog

Edited December 26, 20241 yr by KINO

14 hours ago, KINO said:

but the issue was happening before that too.

Do you mean it hasn't happened yet since the syslog was enabled?

19 minutes ago, JorgeB said:

Do you mean it hasn't happened yet since the syslog was enabled?

It has happened since the syslog was enabled, and is actively still happening. Stays on for about 2 hours and then I have to do an unclean shutdown to restart it as it hangs. 

 

The only idea I have is that there is some sort of memory leak that makes the OS unstable after some time, but I can't pinpoint why this is happening all of sudden, if that's even the case. 

There appears to be a container constantly restarting, see if you find which by looking at their uptimes.

I am not seeing any inconsistencies with uptimes that could help determine which container is constantly restarting. Would you be able to show which lines in the syslog or diagnostics is showing that? I appreciate your help! 

Disable all containers, then start one by one, and look in the syslog for lines like these, a few are normal after a container starts, but not constantly repeating every few seconds:

 

10.0.0.12    Dec 26 02:04:11    KINO    kern    info    kernel    docker0: port 42(vethfdb778c) entered blocking state
10.0.0.12    Dec 26 02:04:11    KINO    kern    info    kernel    docker0: port 42(vethfdb778c) entered forwarding state
10.0.0.12    Dec 26 02:04:12    KINO    user    notice    rc.docker    scrutiny: started succesfully!
10.0.0.12    Dec 26 02:04:13    KINO    kern    info    kernel    docker0: port 43(veth81ad755) entered blocking state
10.0.0.12    Dec 26 02:04:13    KINO    kern    info    kernel    docker0: port 43(veth81ad755) entered disabled state
10.0.0.12    Dec 26 02:04:13    KINO    kern    info    kernel    device veth81ad755 entered promiscuous mode
10.0.0.12    Dec 26 02:04:16    KINO    kern    info    kernel    eth0: renamed from vethd44f9a6
10.0.0.12    Dec 26 02:04:16    KINO    kern    info    kernel    IPv6: ADDRCONF(NETDEV_CHANGE): veth81ad755: link becomes ready
10.0.0.12    Dec 26 02:04:16    KINO    kern    info    kernel    docker0: port 43(veth81ad755) entered blocking state
10.0.0.12    Dec 26 02:04:16    KINO    kern    info    kernel    docker0: port 43(veth81ad755) entered forwarding state
10.0.0.12    Dec 26 02:04:17    KINO    user    notice    rc.docker    Glances: started succesfully!
10.0.0.12    Dec 26 02:04:21    KINO    user    notice    rc.docker    HDDTemp: started succesfully!
10.0.0.12    Dec 26 02:04:23    KINO    kern    info    kernel    docker0: port 44(vethefc1e51) entered blocking state
10.0.0.12    Dec 26 02:04:23    KINO    kern    info    kernel    docker0: port 44(vethefc1e51) entered disabled state
10.0.0.12    Dec 26 02:04:23    KINO    kern    info    kernel    device vethefc1e51 entered promiscuous mode
10.0.0.12    Dec 26 02:04:27    KINO    kern    info    kernel    eth0: renamed from vethbc3e960
10.0.0.12    Dec 26 02:04:27    KINO    kern    info    kernel    IPv6: ADDRCONF(NETDEV_CHANGE): vethefc1e51: link becomes ready
10.0.0.12    Dec 26 02:04:27    KINO    kern    info    kernel    docker0: port 44(vethefc1e51) entered blocking state
10.0.0.12    Dec 26 02:04:27    KINO    kern    info    kernel    docker0: port 44(vethefc1e51) entered forwarding state
10.0.0.12    Dec 26 02:04:28    KINO    user    notice    rc.docker    Matchering: started succesfully!
10.0.0.12    Dec 26 02:04:32    KINO    kern    info    kernel    docker0: port 45(veth813c06c) entered blocking state
10.0.0.12    Dec 26 02:04:32    KINO    kern    info    kernel    docker0: port 45(veth813c06c) entered disabled state
10.0.0.12    Dec 26 02:04:32    KINO    kern    info    kernel    device veth813c06c entered promiscuous mode
10.0.0.12    Dec 26 02:04:38    KINO    kern    info    kernel    eth0: renamed from vethab3148e
10.0.0.12    Dec 26 02:04:38    KINO    kern    info    kernel    IPv6: ADDRCONF(NETDEV_CHANGE): veth813c06c: link becomes ready
10.0.0.12    Dec 26 02:04:38    KINO    kern    info    kernel    docker0: port 45(veth813c06c) entered blocking state
10.0.0.12    Dec 26 02:04:38    KINO    kern    info    kernel    docker0: port 45(veth813c06c) entered forwarding state
10.0.0.12    Dec 26 02:04:39    KINO    user    notice    rc.docker    syncthing: started succesfully!
10.0.0.12    Dec 26 02:04:41    KINO    kern    info    kernel    docker0: port 46(veth5644e5a) entered blocking state
10.0.0.12    Dec 26 02:04:41    KINO    kern    info    kernel    docker0: port 46(veth5644e5a) entered disabled state
10.0.0.12    Dec 26 02:04:41    KINO    kern    info    kernel    device veth5644e5a entered promiscuous mode
10.0.0.12    Dec 26 02:04:44    KINO    kern    info    kernel    eth0: renamed from veth87d1720
10.0.0.12    Dec 26 02:04:44    KINO    kern    info    kernel    IPv6: ADDRCONF(NETDEV_CHANGE): veth5644e5a: link becomes ready
10.0.0.12    Dec 26 02:04:44    KINO    kern    info    kernel    docker0: port 46(veth5644e5a) entered blocking state
10.0.0.12    Dec 26 02:04:44    KINO    kern    info    kernel    docker0: port 46(veth5644e5a) entered forwarding state
10.0.0.12    Dec 26 02:04:45    KINO    user    notice    rc.docker    paperless-ngx: started succesfully!
10.0.0.12    Dec 26 02:04:46    KINO    kern    info    kernel    docker0: port 47(vethbeea4b5) entered blocking state
10.0.0.12    Dec 26 02:04:46    KINO    kern    info    kernel    docker0: port 47(vethbeea4b5) entered disabled state
10.0.0.12    Dec 26 02:04:46    KINO    kern    info    kernel    device vethbeea4b5 entered promiscuous mode
10.0.0.12    Dec 26 02:04:58    KINO    kern    info    kernel    eth0: renamed from veth052ac22

 

 

 

I found out that there was a Python script I wrote that was inadvertently eating up all of my resources, rendering the OS unusable when it was running.

 

Thank you for the help, Jorge!