Jump to content
Sign in to follow this  
alael

Docker container freezing and unable to reboot

10 posts in this topic Last Reply

Recommended Posts

Posted (edited)

Hello

 

I am having sporadic problem where some docker container stop working peroperly and then when i try to restart them they wont restart not using docker kill <id> yeld any result

 

Stopping the array in a clean way also become impossible only an hard reset is left to this point 

 

Things i tried:

 

stopping the container via gui and  via docket stop

tried to kill the container with docker kill

Stop the array

reboot the server

 

The only things that work is pressing the reset button.

 

It does not happen often but should not happen at all its not normal that one has to hard reset the server like this

 

Any idea?

 

(ignore the error on disk4 it has been fixed already :p)

 

 

 

Edited by alael

Share this post


Link to post

Configuration issues would be my first guess.  Check your settings on all plugins and Dockers to make sure that they are configured to write any data or 'skratch' files to a physical device.  i.e., /mnt/diskX/     /mnt/cache/   /mnt/user/ or   /mnt/user0/    Any other location will be to the RAM disk that Unraid uses to store itself in!  Filling up that RAM disk will virtually guarantee a crash! 

 

Be very careful in using a HD that is to be mounted with Unassigned Devices.  If the device is not mounted, the writes will be to RAM. 

Share this post


Link to post
3 minutes ago, Frank1940 said:

Configuration issues would be my first guess.  Check your settings on all plugins and Dockers to make sure that they are configured to write any data or 'skratch' files to a physical device.  i.e., /mnt/diskX/     /mnt/cache/   /mnt/user/ or   /mnt/user0/    Any other location will be to the RAM disk that Unraid uses to store itself in!  Filling up that RAM disk will virtually guarantee a crash! 

 

Be very careful in using a HD that is to be mounted with Unassigned Devices.  If the device is not mounted, the writes will be to RAM. 

I made sure to change ll what could be changed to /mnt/disk and only kept the rest to /mnt/user/ for the data that is split in various disk,  i also moved  libvrt and docker.img to /disk/cache instead of the default /disk/user/ i also do monitor often if the docker img size and its stable since months now

 

Today some docker container became unresponsive for no apparent reason and all my effort to stop them failed this is not the first time that this did happen and i cant seems to track down the issue.

 

Either there is some  bug with docker itself which is what i do think or something within unraid. all docker run and are mounted directly on the cache disk, 

 

Will try to run more memtest when possible last run i did 3 months ago i left the system 24hour and resulted in 0 error therefore i dont see where the problem could be

 

thanks.

Share this post


Link to post
6 minutes ago, alael said:

Will try to run more memtest when possible last run i did 3 months ago i left the system 24hour and resulted in 0 error therefore i dont see where the problem could be

The built-in memtst will not work properly if you have ECC memory...

 

8 minutes ago, alael said:

I made sure to change ll what could be changed to /mnt/disk and only kept the rest to /mnt/user/ for the data that is split in various disk,  i also moved  libvrt and docker.img to /disk/cache instead of the default /disk/user/ i also do monitor often if the docker img size and its stable since months now

 

Both of the file system locations that I have highlighted will be written on the RAM disk!!!   All physical disks on mounted in the /mnt directory  (except for the flash drive which is at /boot ).   Any file location whose path does not begin with   /mnt/   should be considered suspect and carefully investigated! 

 

If you are having problems getting your head around this, run the following commands in a terminal window:

ls -al /

ls -al /mnt/

ls -al /mnt/user

The first command shows all of the directories/folders for the entire system.  One thing that I found hard to get straight in my head (Some forty year ago now) was that UNIX (and, thus,  Linux) mounts each physical storage device as a individual file directory.  In the case of Unraid, these directories are inside of another folder call  mnt  

 

Unraid OS  is unpacked and installed on a RAM disk every time the system is started.  This is unlike the procedure used by most other OS's where the OS is installed once on hard disk and the required modules are loaded when the computer is turned. 

Share this post


Link to post
2 minutes ago, Frank1940 said:

The built-in memtst will not work properly if you have ECC memory...

 

Both of the file system locations that I have highlighted will be written on the RAM disk!!!   All physical disks on mounted in the /mnt directory  (except for the flash drive which is at /boot ).   Any file location whose path does not begin with   /mnt/   should be considered suspect and carefully investigated! 

 

If you are having problems getting your head around this, run the following commands in a terminal window:


ls -al /

ls -al /mnt/

ls -al /mnt/user

The first command shows all of the directories/folders for the entire system.  One thing that I found hard to get straight in my head (Some forty year ago now) was that UNIX (and, thus,  Linux) mounts each physical storage device as a individual file directory.  In the case of Unraid, these directories are inside of another folder call  mnt  

 

Unraid OS  is unpacked and installed on a RAM disk every time the system is started.  This is unlike the procedure used by most other OS's where the OS is installed once on hard disk and the required modules are loaded when the computer is turned. 

By /mnt/user i meant /mnt/user/sharename i was just simplifying my writing, I am aware how unraid operate, this is not  the cause of the as stated already the docker.img size is stable and does not move all whats inside are docker own image.

 

Additionally the symptom i am having have nothing to do with wrong path is more like zombie process created by docker become unkillable even by init, Which i suspect 90% its caused by either some conflict or bugs. 

Share this post


Link to post

OK.  If the paths are all correct then there is another problem.   First thing.  Go back to your first post and consider editing it extensively.  Use Capitalization and punctuation.  For each thing that you tried, give the outcome of that action-- don't assume that the reader knows.  The Diagnostics file is quite old.  Why is this?  Is it from a time when you were actually experiencing the problem?      Does the system continue to function (serving files, GUI, etc.) except for the Dockers?  If not, what are the issues?  What Docker(s)  is/are giving the problem?  After you have done this you might flag LimeTech by typing an '@' and continue with "Limetech" until it appears on the drop list.  This will flag this thread for someone there to have a look at it.  Example:   @alael

 

By the way, I believe the convention that any Linux path that begins with  "/" is starting from the root.  If there is no '/', then it is from the present working directory--  pwd.  Don't use a shortcut in notation in an Unraid post unless you alert the reader as to what you are doing.  

Share this post


Link to post

Can you post a screenshot of your Docker page on the webGui so we can see what containers you have running and what their configuration settings are?

Share this post


Link to post
Posted (edited)
19 hours ago, jonp said:

Can you post a screenshot of your Docker page on the webGui so we can see what containers you have running and what their configuration settings are?

Is this okay? psting the full page would result in a enormous screenshot, Maybe there is a more compact text version i can grab somewhere?

 

 

 

Edited by alael

Share this post


Link to post

Specifically looking to see the volume mappings part.

Share this post


Link to post
13 hours ago, jonp said:

Specifically looking to see the volume mappings part.

sent via pm

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this