Hello there,

I do use this server since 2 years and could troubleshoot most of the problems by myself, until now.
I might have traced back the error to a memory leak, but there ends it. My server does crash, that I have to A) Do a cold reboot by flicking the power switch B) Wait and hope that the server does get back up on its feet, so that I can access the WebUI, but by then the Docker App crashed and I have to reboot.

I can not access the Shares via SMB in the local network, and even the Web UI Page is no longer accessible if the server crashes.

 

I do have my diagnostics attached, and would be glad if someone could help me with identifying the problem.

Kind Regards
Max

tower-diagnostics-20240423-2145.zip

You have this in your syslog:

Apr 23 09:32:36 Tower root: Fix Common Problems: Error: Macvlan and Bridging found

 

so it appears you are using macvlan for docker networking.    If using macvlan then for stability you need to disable bridging on eth0.   The alternative is to use ipvlan for docker networking.

Did change it to ipvlan, i guess it worked - if otherwise i will mention it here
@itimpi thank you very much!

Here I am again
I did fix the error you mentionend, thanks! But I do run out of memory relatively fast after a reboot.

Server Crashed is the situation
1. Cold Reboot (wait 1 min)
2. Open Web GUI - Have to reboot, because Docker Service does fail to start

3. Restart

4. Wait 1 min

5. Server is back online, Docker Service's start up

6. RAM (32 GB) is at 30-40 %

7. All Services started up (minimal start up, unnecessary services were stopped and cut of autostart, for the time until a fix is found)

8. wait 5 min

9. RAM starts to go up to 80,87,95 %

9.1. meanwhile - I do monitor my Container and each of them operates in a good amount of RAM (all together less than 10 GB, viewed in the Docker Page)
9.2. but under the tab for the processes, it is shown that almost all of the RAM that is used is allocated to the dockerd service (around 77% of all memory, odd because auf 9.2.)

10. Nevertheless the WEB GUI portraits 99% RAM Allocation, and at this point the server is unresponsive

I plan to create a swap file (Plugin) on an extra SSD which will not be in a RAID, like my Cache Pool - for that I have ordered an extra cable (run out of enough power cables for SATA Connection (lol)) - maybe this will help, but if you would like to look into it, I couldn't find a solution in the diagnostics (Snapshot at rougly 85% RAM Allocation) be my guest.
I do think that this swap file might be not enough (32 GB of RAM where only 10 GB are in active use by the Docker Services, the remaining 20 are usually enough for the overhead - I do not have any VM's in use)

This issue was reproducable arround 3 times, after which I am writing this

Thank you again!

tower-diagnostics-20240425-0409.zip

RAM usage growing so quickly won't be fixed by swap, you need to find out what's causing it, boot with all services disable then start enabling one by one, including starting containers also one by one.

will try this, thank you! but propably not before the weekend, will come back and report

Edit: I dont know if this is the solution, but maybe my last paragraph here has the possible solution (I didn't find the error, just updated all my outstanding containers)


Okay i had some time and tried my best, but this are the results (cannot find the cause)

1. Started the server

2. Straight the Settings and stopped the autostart of the general Docker Service

3. Waited some time, but Ram was idling at 5-10%

4. Started up the Docker Service, and immideatily disabled the autostart for most of the apps

4.1. Said apps went on and server was running at 20 %, until suddenly the RAM got up to 50,60,70,80
4.2. Started to power down some apps - still at 80 % (seemed to only slow down the process of the RAM 'growing')

4.3. Powered down all Docker Container by Stop All in the Docker View

4.4 RAM still at 97 % which does mean 30 GB of RAM (with no active apps (triple checked))

5. Waited some time ~5min RAM was +-1% at 96-98 %
5.1. captured the following line in the process overview in the settings:

root 21802 171 91.0 33754864 29626460 ? Sl 21:50 58:13 /usr/bin/dockerd -p /var/run/dockerd.pid --log-opt max-size=50m --log-opt max-file=1 --log-level=fatal --storage-driver=btrfs
the 91.0 is the RAM value for this service

6. Turned of the general Docker Service
6.1. RAM dropped immediatly to 4-5 %

Okay so next thing to try:

I am notorius to just leave my containers on the current version and forget about updating them. Yes I know that has to change. Just updated everything and now tried to start everything one on one.



Start the Docker Service and start up one container after each other with time breaks of around 5min between each!
--- author's note: will send this posting when I am done doing this process - all in all it took around 3 hours----
Docker Service is started but all containers are set to do not auto start
Waited 10 min and set everything one after one other to start up.
Now all my Docker Containers are started up, each operating wth a reasanable amount of RAM, if something will break again I will report it back here.

Possible Solution: Didn't update my containers and something there did break... Or I did something accidental that fixed the problem. Will mark the comment from JorgeB to be the solution, because impatience was not helping, and the slow but steady process was.

Thank you!

Edited April 26, 20242 yr by Maximilian-Tek