NotYetRated Posted March 18, 2018 Share Posted March 18, 2018 My dockers tab has become unresponsive. The dockers are running, I just cannot load the dockers or dashboard tab of my GUI. Mar 18 11:39:04 BigBang nginx: 2018/03/18 11:39:04 [error] 8777#8777: *408549 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.23, server: , request: "POST /plugins/dynamix.docker.manager/include/DockerUpdate.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "bigbang", referrer: "http://bigbang/Docker" Mar 18 11:42:17 BigBang nginx: 2018/03/18 11:42:17 [error] 8777#8777: *408549 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.23, server: , request: "POST /webGui/include/DashboardApps.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "bigbang", referrer: "http://bigbang/Dashboard" Diags also took quite some time to generate. Any assistance would be appreciated. bigbang-diagnostics-20180318-1146.zip Quote Link to comment
trurl Posted March 18, 2018 Share Posted March 18, 2018 Not likely the cause of your issue, but why do you have parity2 but no parity disk? Are some of your disks connected USB instead of SATA? Quote Link to comment
NotYetRated Posted March 18, 2018 Author Share Posted March 18, 2018 (edited) 32 minutes ago, trurl said: Not likely the cause of your issue, but why do you have parity2 but no parity disk? Are some of your disks connected USB instead of SATA? Upgraded parity disk to a larger one and.... somehow ended up with parity2... I believe I added the second, rebuilt. removed the first, then never did anything else.... And none are USB. Edited March 18, 2018 by NotYetRated Quote Link to comment
MadeOfCard Posted March 19, 2018 Share Posted March 19, 2018 I had the same issue. I had to downgrade to remedy the problem. A restart was only a temporary fix. 1 Quote Link to comment
Uncledome Posted March 21, 2018 Share Posted March 21, 2018 I am currently facing the same issue and will rollback to 6.4.1 until this is fixed. Quote Link to comment
yippy3000 Posted March 22, 2018 Share Posted March 22, 2018 (edited) I also have this issue. Rebooting it fixes it for a couple hours then it comes back. The issue also affects the docker display on the dashboard for me as well. For me, the Fix Common Problems also becomes unusable, when you load the tab it scans forever and never finishes. Edited March 22, 2018 by yippy3000 Quote Link to comment
bonienl Posted March 22, 2018 Share Posted March 22, 2018 How much memory do you have? Are you close to the limit? Quote Link to comment
MadeOfCard Posted March 22, 2018 Share Posted March 22, 2018 I've never ran anything that would bring it to a limit. I have 32gb and the highest utilization Ive seen it at was 60% while running a Windows 10 VM. One wild card could be that I'm running a Ryzen 1800x. IDK about the other people though. Quote Link to comment
Uncledome Posted March 22, 2018 Share Posted March 22, 2018 Same here, i7 5930k stock clock cpu and 32GB Ram. Only Docker was running and never used more than 20% Ram. After Reboot it worked for like 2 hours and stopped responding. Couldn't even reboot it (Button nor reboot command worked, nothing happened) and had to force shutdown with power button. Will stay on 6.4.1 until this is fixed. Quote Link to comment
bonienl Posted March 22, 2018 Share Posted March 22, 2018 On 3/19/2018 at 11:34 PM, MadeOfCard said: I had the same issue. I had to downgrade to remedy the problem. A restart was only a temporary fix. 21 hours ago, Uncledome said: I am currently facing the same issue and will rollback to 6.4.1 until this is fixed. You both see nginx communication errors in the system log, like the OP? Quote Link to comment
MadeOfCard Posted March 22, 2018 Share Posted March 22, 2018 I did look at the JS Console in Chrome, the call was timing out. As for nginx, I can look later tonight. Will the previous 6.5.0' logs still be accessible to me while I stay on 6.4.1? Quote Link to comment
bonienl Posted March 22, 2018 Share Posted March 22, 2018 4 minutes ago, MadeOfCard said: I did look at the JS Console in Chrome, the call was timing out. As for nginx, I can look later tonight. Will the previous 6.5.0' logs still be accessible to me while I stay on 6.4.1? Check your logs folder on your flash device, it may hold the necessary diagnostics files (check their creation date). Quote Link to comment
Uncledome Posted March 22, 2018 Share Posted March 22, 2018 Yep. I cannot currentlly paste you the log but it was the exact same error with dynamix.docker.manager and nginx. Quote Link to comment
yippy3000 Posted March 22, 2018 Share Posted March 22, 2018 My issue as well is the same nginx error and I have 32Gb ram with 44% usage. Quote Link to comment
rix Posted March 23, 2018 Share Posted March 23, 2018 I have the same issue, will post my diagnostics later. Only an unclean shutdown helps (for a few hours) Quote Link to comment
michaelmcq Posted March 23, 2018 Share Posted March 23, 2018 I've posted these over on the main release thread, but here's my diagnostics for the same issue https://lime-technology.com/applications/core/interface/file/attachment.php?id=39269 I wouldn't know where to start with sifting through them but I notice we all have the same error in our docker logs, specifically level=error msg="stream copy error: reading from a closed fifo" I don;t know what a log looks like with out this issue so that could be a red herring. I couldn't see anything at the same time in my sys log Quote Link to comment
bonienl Posted March 23, 2018 Share Posted March 23, 2018 Difficult to replicate, never happens on my systems. A next time when this happens can you try the following: /etc/rc.d/rc.php-fpm restart Quote Link to comment
Uncledome Posted March 23, 2018 Share Posted March 23, 2018 So, I am currently back at home and checked the log I saved before downgrading to 6.4.1 This is the last error I had: Mar 20 16:23:13 Tower nginx: 2018/03/20 16:23:13 [error] 26217#26217: *455803 upstream timed out (110: Connection timed out) while reading upstream, client: xxx.xxx.xxx, server: , request: "GET /plugins/dynamix.docker.manager/include/CreateDocker.php?updateContainer=true&ct[]=PlexMediaServer HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.xxx.xxx", referrer: "http://192.168.xxx.xxx/Docker" as soon as this error popped, docker was not responsible anymore. Couldn't even reboot the system with: "reboot // shutdown -hP now // powerdown // powerdown -r" Nothing happened, had to force shutdown the server with keeping power button pressed. Quote Link to comment
bonienl Posted March 23, 2018 Share Posted March 23, 2018 Yes, this is the common denominator between users with the problem. I tend to say that Docker hangs at a certain point, and when the GUI needs to query some status (it calls docker for that) it subsequently hangs the GUI interaction too. Don't know if it related but there is a TCP bug in the kernel, which can cause strange behavior on applications doing TCP communication, such as a docker container doing file downloads. Quote Link to comment
michaelmcq Posted March 23, 2018 Share Posted March 23, 2018 3 hours ago, bonienl said: Difficult to replicate, never happens on my systems. A next time when this happens can you try the following: /etc/rc.d/rc.php-fpm restart It stopped and started without issue but didn't seem to make any difference to the docker tab unfortunately Quote Link to comment
michaelmcq Posted March 23, 2018 Share Posted March 23, 2018 1 hour ago, Uncledome said: So, I am currently back at home and checked the log I saved before downgrading to 6.4.1 This is the last error I had: Mar 20 16:23:13 Tower nginx: 2018/03/20 16:23:13 [error] 26217#26217: *455803 upstream timed out (110: Connection timed out) while reading upstream, client: xxx.xxx.xxx, server: , request: "GET /plugins/dynamix.docker.manager/include/CreateDocker.php?updateContainer=true&ct[]=PlexMediaServer HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.xxx.xxx", referrer: "http://192.168.xxx.xxx/Docker" as soon as this error popped, docker was not responsible anymore. Couldn't even reboot the system with: "reboot // shutdown -hP now // powerdown // powerdown -r" Nothing happened, had to force shutdown the server with keeping power button pressed. Not that it's any better than holding down the power button but if you can still get a ssh session, you can do the following to force a reboot echo 1 > /proc/sys/kernel/sysrq echo b > /proc/sysrq-trigger https://major.io/2009/01/29/linux-emergency-reboot-or-shutdown-with-magic-commands/ Quote Link to comment
bonienl Posted March 23, 2018 Share Posted March 23, 2018 Just now, michaelmcq said: It stopped and started without issue but didn't seem to make any difference to the docker tab unfortunately Ok, thanks for testing. It confirms the diagnostics info that the php-fpm process is not killed, but rather becomes unresponsive due to "something" hanging. Quote Link to comment
rix Posted March 23, 2018 Share Posted March 23, 2018 (edited) Just to throw this into the thread: Mar 23 17:24:07 ninja nginx: 2018/03/23 17:24:07 [error] 22572#22572: *278287 upstream timed out (110: Connection timed out) while reading response header from upstream, client: X.X.X.X, server: , request: "POST /webGui/include/DashboardApps.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXX.unraid.net", referrer: "https://XXX.unraid.net/Dashboard" After upgrading to 6.5.0 I had to fix my own app (running in docker): https://github.com/rix1337/RSScrawler/commit/4a34e5d0f6be7ed4af193b51297b1f89370c8978 When I sent to many simulatenous requests to this app (built on flask) it crashed with the https://stackoverflow.com/questions/12591760/flask-broken-pipe-with-requests broken pipe error. This also resulted in the Dashboard losing access to docker/vm. My Dashboard still locks up (though less frequently so), so I guess some tcp-related error from any of my running docker containers is "able" to crash the whole thing. This likely stems from the new (and awesome) socket implementation coupled with the tcp bug.. but I dont know Edited March 23, 2018 by rix Quote Link to comment
bonienl Posted March 23, 2018 Share Posted March 23, 2018 9 minutes ago, rix said: My Dashboard still locks up (though less frequently so), so I guess some tcp-related error from any of my running docker containers is "able" to crash the whole thing. When you stop all containers. Does the Dashboard still hangs? Quote Link to comment
rix Posted March 23, 2018 Share Posted March 23, 2018 (edited) The thing is, the whole Docker Section is missing (notice also the last Parity check after unclean shutdown #3) On the docker page itself, I get only the spinner So basically after a while the complete docker section of the webUI "dissappears" even though all Containers are still happily running. Even my automatic appdata Backup at 4:00 am including the container restart/update does work. Its really all about not being displayed in the frontend. EDiT: Stopping the array will not work. Unraid just hangs, if I try. Apart from that it happily chugs on as usual. Edited March 23, 2018 by rix Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.