frequent docker execution errors with "nginx error - upstream timed out" in logs

[enmesh-parisian-latest]

Hey, since upgrading to 6.6 (now 6.6.3) I've been having a lot of issues with docker. Frequently containers won't start or stop resulting in "execution error" message in gui and the following output in logs:

Oct 24 12:44:54 tobor-server nginx: 2018/10/24 12:44:54 [error] 9878#9878: *36131 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.10.128, server: , request: "POST /webGui/include/Notify.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.10.200", referrer: "http://192.168.10.200/Docker"
Oct 24 12:44:55 tobor-server nginx: 2018/10/24 12:44:55 [error] 9878#9878: *36347 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.10.128, server: , request: "POST /plugins/dynamix.system.temp/include/SystemTemp.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.10.200", referrer: "http://192.168.10.200/Docker"
Oct 24 12:45:16 tobor-server nginx: 2018/10/24 12:45:16 [error] 9878#9878: *36389 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.10.128, server: , request: "POST /plugins/preclear.disk/Preclear.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.10.200", referrer: "http://192.168.10.200/Docker"
Oct 24 12:45:16 tobor-server nginx: 2018/10/24 12:45:16 [error] 9878#9878: *36391 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.10.128, server: , request: "POST /plugins/preclear.disk/Preclear.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.10.200", referrer: "http://192.168.10.200/Docker"
Oct 24 12:45:18 tobor-server nginx: 2018/10/24 12:45:18 [error] 9878#9878: *36404 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.10.128, server: , request: "GET /Docker/AddContainer HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.10.200", referrer: "http://192.168.10.200/Docker"

Usually the server gui is unresponsive for about a minute before resuming function. I've tried deleting my docker img and recreating all containers. No luck so far.
Any ideas?

tobor-server-diagnostics-20181024-1253.zip

[enmesh-parisian-latest]

My server is now giving "504 Gateway Time-out - nginx" errors when trying to access the gui, ssh not responding. I forced a reboot and was able to start uninstalling plugins before the same problem occurred. Doing another forced reboot now but I'm not optimistic. I'll try and generate another diagnostics file if I can access the gui again.

Any help would be greatly appreciated.

EDIT: I managed to grab the server diagnostics.
I'm getting errors like this when removing plugins:

tobor-server nginx: 2018/10/28 11:41:35 [error] 4596#4596: *381 FastCGI sent in stderr: "Unable to open primary script: /usr/local/emhttp/plugins/dynamix.system.temp/include/SystemTemp.php (No such file or directory)" while reading response header from upstream, client: 127.0.0.1, server: , request: "POST /plugins/dynamix.system.temp/include/SystemTemp.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "localhost", referrer: "http://localhost/Plugins"

tobor-server ntpd[1922]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized

tobor-server-diagnostics-20181028-1144.zip

Edited October 28, 2018 by enmesh-parisian-latest

[Squid]

tobor-server nginx: 2018/10/28 11:41:35 [error] 4596#4596: *381 FastCGI sent in stderr: "Unable to open primary script: /usr/local/emhttp/plugins/dynamix.system.temp/include/SystemTemp.php (No such file or directory)" while reading response header from upstream, client: 127.0.0.1, server: , request: "POST /plugins/dynamix.system.temp/include/SystemTemp.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "localhost", referrer: "http://localhost/Plugins"

This one is simple.  You've removed the system temp plugin, but until you refresh the plugins page you were sitting on to remove it, the page still is looking for that script to update the temperatures, hence the error.

[enmesh-parisian-latest]

Thanks for the reply. Any idea about the other issues? System has been ok for past few hours after removing a bunch of plugins and a few containers. I'd like to get to the bottom of the cause however.

 

EDIT: scratch that, the server died overnight. I guess I'll hard reboot again and remove all plugins this time?

Edited October 28, 2018 by enmesh-parisian-latest

[enmesh-parisian-latest]

Bump

[scloutier]

I am also experiencing similar issues.  Any operation that involves updating/starting/stopping docker container takes forever now.  Log also shows lots of upstream timed out error.  Reboot didn't fix, disabled most of my docker to see if I could isolate to a culprit, removed some plug-ins (that have been there for a long time).  nothing seems to help.  this started somewhat recently (couldn't say when exactly).

[enmesh-parisian-latest]

Try dumping your docker.img and starting fresh. 6 Months later my docker still isn't perfect and nginx is a bitch but everything mostly works.