docker engine keeps freezes - common checks done !

[VinceB]

Hi everyone !

I wish i could find a solution or even a clue that could lead me to get rid of frequent docker daemon freezes.

I am running a dell poweredge t140, with sufficient resources scaled for what I am doing.

I’ve gone through all the resources in this forum that relate to docker. I’ve already ensured that :

 

My docker image was not completely filled, any of the containers were writing inside the docker image. But no clue in that direction.

 

What I noticed, precisely in that sequence:

 

When Docker crashes

 

1.     Plex server becomes unresponsive, cannot connect the client…

2.     While other containers works

3.     I edit the pms container in order to restart it

4.     Then all the docker container crash when trying to restart pms

5.     Via the settings I try to stop docker daemon : stuck…

6.     Then docker daemon failed to start (the famous one)

 

 

I’ve read a lot that the docker image file could be corrupted: ok, but what is the root cause ?

How can we / the developers explain that this image is that sensible that it gets corrupted every couple of days ?

 

Can we really rely on such system to run a productive environment to run apps ?

 

 

 

I will be glad to get your help, thoughts or guidance.

 

Cheers !

 

Vincent

 

 

 

themis-diagnostics-20201111-1920.zip

[trurl]

Why have you given 40G to docker.img? I suspect you have been filling it since 20G is usually much more than enough. I run 17 dockers and they take less than half of 20G.

 

Also, your appdata, domains, and system shares have files on the array, so your dockers / VMs will have performance affected by slower parity, and dockers / VMs will keep array disks spinning since these files are always open.

 

18 minutes ago, VinceB said:

I’ve already ensured that :

 

My docker image was not completely filled, any of the containers were writing inside the docker image. But no clue in that direction.

Not quite sure what to make of these statements. You start out saying you "ensured that" and end by saying "no clue".

 

 

[VinceB]

Hi Trurl ! 

 

Thanks for your feedback !

you're entirely right, i'll set the appdata - iso - and domains share to cache only. That's said, i still don't know what causes docker engine to crash, as my docker image file was about 40% filled.

Probably there's one app in a containers that fill docker image file to a certain extend, but can one says that it's the culprit for docker engine to crash ? How to troubleshoot to now exactly what happened at time of the crash ?

 

thanks again.

 

best regards

 

Vincent

[VinceB]

another concern is that if my configs of my docker app go to the appdata share that is set to "only" as you suggest (and best practice too), in case of hardware failure that damage the cache disk, how can i recover my app config ?

[trurl]

6 hours ago, VinceB said:

i'll set the appdata - iso - and domains share to cache only.

Don't know why you mention iso share, I never did. I did mention system share, and that is the most important one to fix.

 

Cache-only is ignored by mover, so setting those shares to cache-only will not move them to cache. Cache-prefer is the only setting that can move them to cache. The reason they weren't moved to cache when you had them at prefer is because mover can't move open files, so in addition to setting them to prefer, you have to go to Settings and disable both Docker and VMs then run mover.

 

7 hours ago, VinceB said:

Probably there's one app in a containers that fill docker image file to a certain extend, but can one says that it's the culprit for docker engine to crash ?

If you corrupt docker.img by filling it then of course docker can't work.

 

6 hours ago, VinceB said:

if my configs of my docker app go to the appdata share that is set to "only" as you suggest (and best practice too), in case of hardware failure that damage the cache disk, how can i recover my app config ?

I never suggested setting those to "only" as noted above.

 

appdata contains the "working storage" of each of your containers. Perhaps you have even more than that in mind when you ask about "configs". The settings for each container that you make in the Add / Edit Container page are stored on flash. Both appdata and flash (and libvirt) can have scheduled backups using the CA Backup plugin.

 

Go to the Docker page and click the Container Size button, then post a screenshot of the result.