6.5.0 Docker Tab Unresponsive


Recommended Posts

My dockers tab has become unresponsive. The dockers are running, I just cannot load the dockers or dashboard tab of my GUI.

 

Mar 18 11:39:04 BigBang nginx: 2018/03/18 11:39:04 [error] 8777#8777: *408549 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.23, server: , request: "POST /plugins/dynamix.docker.manager/include/DockerUpdate.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "bigbang", referrer: "http://bigbang/Docker"
Mar 18 11:42:17 BigBang nginx: 2018/03/18 11:42:17 [error] 8777#8777: *408549 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.23, server: , request: "POST /webGui/include/DashboardApps.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "bigbang", referrer: "http://bigbang/Dashboard"

 

Diags also took quite some time to generate.

 

Any assistance would be appreciated.

bigbang-diagnostics-20180318-1146.zip

Link to comment
32 minutes ago, trurl said:

Not likely the cause of your issue, but why do you have parity2 but no parity disk?

 

Are some of your disks connected USB instead of SATA?

 

Upgraded parity disk to a larger one and.... somehow ended up with parity2... I believe I added the second, rebuilt. removed the first, then never did anything else....

 

And none are USB.

Edited by NotYetRated
Link to comment

I also have this issue. Rebooting it fixes it for a couple hours then it comes back. The issue also affects the docker display on the dashboard for me as well.

 

For me, the Fix Common Problems also becomes unusable, when you load the tab it scans forever and never finishes.

Edited by yippy3000
Link to comment

Same here,

i7 5930k stock clock cpu and 32GB Ram.

Only Docker was running and never used more than 20% Ram.

After Reboot it worked for like 2 hours and stopped responding.

Couldn't even reboot it (Button nor reboot command worked, nothing happened) and had to force shutdown with power button.

Will stay on 6.4.1 until this is fixed.

Link to comment
On 3/19/2018 at 11:34 PM, MadeOfCard said:

I had the same issue. I had to downgrade to remedy the problem. A restart was only a temporary fix.

 

21 hours ago, Uncledome said:

I am currently facing the same issue and will rollback to 6.4.1 until this is fixed.

 

You both see nginx communication errors in the system log, like the OP?

Link to comment
4 minutes ago, MadeOfCard said:

I did look at the JS Console in Chrome, the call was timing out.

 

As for nginx, I can look later tonight. Will the previous 6.5.0' logs still be accessible to me while I stay on 6.4.1?

 

Check your logs folder on your flash device, it may hold the necessary diagnostics files (check their creation date).

Link to comment

I've posted these over on the main release thread, but here's my diagnostics for the same issue

 

https://lime-technology.com/applications/core/interface/file/attachment.php?id=39269

 

I wouldn't know where to start with sifting through them but I notice we all have the same error in our docker logs, specifically 

 

level=error msg="stream copy error: reading from a closed fifo"

I don;t know what a log looks like with out this issue so that could be a red herring. I couldn't see anything at the same time in my sys log

Link to comment

So, I am currently back at home and checked the log I saved before downgrading to 6.4.1

 

This is the last error I had:

Mar 20 16:23:13 Tower nginx: 2018/03/20 16:23:13 [error] 26217#26217: *455803 upstream timed out (110: Connection timed out) while reading upstream, client: xxx.xxx.xxx, server: , request: "GET /plugins/dynamix.docker.manager/include/CreateDocker.php?updateContainer=true&ct[]=PlexMediaServer HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.xxx.xxx", referrer: "http://192.168.xxx.xxx/Docker"

 

as soon as this error popped, docker was not responsible anymore.

Couldn't even reboot the system with: "reboot // shutdown -hP now // powerdown // powerdown -r"

Nothing happened, had to force shutdown the server with keeping power button pressed.

Link to comment

Yes, this is the common denominator between users with the problem.

 

I tend to say that Docker hangs at a certain point, and when the GUI needs to query some status (it calls docker for that) it subsequently hangs the GUI interaction too.

 

Don't know if it related but there is a TCP bug in the kernel, which can cause strange behavior on applications doing TCP communication, such as a docker container doing file downloads.

 

Link to comment
1 hour ago, Uncledome said:

So, I am currently back at home and checked the log I saved before downgrading to 6.4.1

 

This is the last error I had:


Mar 20 16:23:13 Tower nginx: 2018/03/20 16:23:13 [error] 26217#26217: *455803 upstream timed out (110: Connection timed out) while reading upstream, client: xxx.xxx.xxx, server: , request: "GET /plugins/dynamix.docker.manager/include/CreateDocker.php?updateContainer=true&ct[]=PlexMediaServer HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.xxx.xxx", referrer: "http://192.168.xxx.xxx/Docker"

 

as soon as this error popped, docker was not responsible anymore.

Couldn't even reboot the system with: "reboot // shutdown -hP now // powerdown // powerdown -r"

Nothing happened, had to force shutdown the server with keeping power button pressed.

Not that it's any better than holding down the power button but if you can still get a ssh session, you can do the following to force a reboot

 

echo 1 > /proc/sys/kernel/sysrq 
echo b > /proc/sysrq-trigger

https://major.io/2009/01/29/linux-emergency-reboot-or-shutdown-with-magic-commands/

Link to comment
Just now, michaelmcq said:

It stopped and started without issue but didn't seem to make any difference to the docker tab unfortunately

 

Ok, thanks for testing. It confirms the diagnostics info that the php-fpm process is not killed, but rather becomes unresponsive due to "something" hanging.

 

Link to comment

Just to throw this into the thread:

 

Mar 23 17:24:07 ninja nginx: 2018/03/23 17:24:07 [error] 22572#22572: *278287 upstream timed out (110: Connection timed out) while reading response header from upstream, client: X.X.X.X, server: , request: "POST /webGui/include/DashboardApps.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXX.unraid.net", referrer: "https://XXX.unraid.net/Dashboard"

 

After upgrading to 6.5.0 I had to fix my own app (running in docker): https://github.com/rix1337/RSScrawler/commit/4a34e5d0f6be7ed4af193b51297b1f89370c8978

 

When I sent to many simulatenous requests to this app (built on flask) it crashed with the https://stackoverflow.com/questions/12591760/flask-broken-pipe-with-requests broken pipe error. This also resulted in the Dashboard losing access to docker/vm.

 

My Dashboard still locks up (though less frequently so), so I guess some tcp-related error from any of my running docker containers is "able" to crash the whole thing.

This likely stems from the new (and awesome) socket implementation coupled with the tcp bug..

 

but I dont know

Edited by rix
Link to comment

The thing is, the whole Docker Section is missing

dash.thumb.JPG.fa688711cef43e8af49c401cd546093e.JPG

(notice also the last Parity check after unclean shutdown #3)

 

On the docker page itself, I get only the spinner

dockerpage.JPG.5799291cc6c168d9d776784b68fe033f.JPG

 

So basically after a while the complete docker section of the webUI "dissappears" even though all Containers are still happily running.

Even my automatic appdata Backup at 4:00 am including the container restart/update does work.

 

Its really all about not being displayed in the frontend.

 

EDiT: Stopping the array will not work. Unraid just hangs, if I try. Apart from that it happily chugs on as usual.

Edited by rix
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.