March 18, 20188 yr My dockers tab has become unresponsive. The dockers are running, I just cannot load the dockers or dashboard tab of my GUI. Mar 18 11:39:04 BigBang nginx: 2018/03/18 11:39:04 [error] 8777#8777: *408549 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.23, server: , request: "POST /plugins/dynamix.docker.manager/include/DockerUpdate.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "bigbang", referrer: "http://bigbang/Docker" Mar 18 11:42:17 BigBang nginx: 2018/03/18 11:42:17 [error] 8777#8777: *408549 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.23, server: , request: "POST /webGui/include/DashboardApps.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "bigbang", referrer: "http://bigbang/Dashboard" Diags also took quite some time to generate. Any assistance would be appreciated. bigbang-diagnostics-20180318-1146.zip
March 18, 20188 yr Community Expert Not likely the cause of your issue, but why do you have parity2 but no parity disk? Are some of your disks connected USB instead of SATA?
March 18, 20188 yr Author 32 minutes ago, trurl said: Not likely the cause of your issue, but why do you have parity2 but no parity disk? Are some of your disks connected USB instead of SATA? Upgraded parity disk to a larger one and.... somehow ended up with parity2... I believe I added the second, rebuilt. removed the first, then never did anything else.... And none are USB. Edited March 18, 20188 yr by NotYetRated
March 19, 20188 yr I had the same issue. I had to downgrade to remedy the problem. A restart was only a temporary fix.
March 21, 20188 yr I am currently facing the same issue and will rollback to 6.4.1 until this is fixed.
March 22, 20188 yr I also have this issue. Rebooting it fixes it for a couple hours then it comes back. The issue also affects the docker display on the dashboard for me as well. For me, the Fix Common Problems also becomes unusable, when you load the tab it scans forever and never finishes. Edited March 22, 20188 yr by yippy3000
March 22, 20188 yr I've never ran anything that would bring it to a limit. I have 32gb and the highest utilization Ive seen it at was 60% while running a Windows 10 VM. One wild card could be that I'm running a Ryzen 1800x. IDK about the other people though.
March 22, 20188 yr Same here, i7 5930k stock clock cpu and 32GB Ram. Only Docker was running and never used more than 20% Ram. After Reboot it worked for like 2 hours and stopped responding. Couldn't even reboot it (Button nor reboot command worked, nothing happened) and had to force shutdown with power button. Will stay on 6.4.1 until this is fixed.
March 22, 20188 yr On 3/19/2018 at 11:34 PM, MadeOfCard said: I had the same issue. I had to downgrade to remedy the problem. A restart was only a temporary fix. 21 hours ago, Uncledome said: I am currently facing the same issue and will rollback to 6.4.1 until this is fixed. You both see nginx communication errors in the system log, like the OP?
March 22, 20188 yr I did look at the JS Console in Chrome, the call was timing out. As for nginx, I can look later tonight. Will the previous 6.5.0' logs still be accessible to me while I stay on 6.4.1?
March 22, 20188 yr 4 minutes ago, MadeOfCard said: I did look at the JS Console in Chrome, the call was timing out. As for nginx, I can look later tonight. Will the previous 6.5.0' logs still be accessible to me while I stay on 6.4.1? Check your logs folder on your flash device, it may hold the necessary diagnostics files (check their creation date).
March 22, 20188 yr Yep. I cannot currentlly paste you the log but it was the exact same error with dynamix.docker.manager and nginx.
March 23, 20188 yr I have the same issue, will post my diagnostics later. Only an unclean shutdown helps (for a few hours)
March 23, 20188 yr I've posted these over on the main release thread, but here's my diagnostics for the same issue https://lime-technology.com/applications/core/interface/file/attachment.php?id=39269 I wouldn't know where to start with sifting through them but I notice we all have the same error in our docker logs, specifically level=error msg="stream copy error: reading from a closed fifo" I don;t know what a log looks like with out this issue so that could be a red herring. I couldn't see anything at the same time in my sys log
March 23, 20188 yr Difficult to replicate, never happens on my systems. A next time when this happens can you try the following: /etc/rc.d/rc.php-fpm restart
March 23, 20188 yr So, I am currently back at home and checked the log I saved before downgrading to 6.4.1 This is the last error I had: Mar 20 16:23:13 Tower nginx: 2018/03/20 16:23:13 [error] 26217#26217: *455803 upstream timed out (110: Connection timed out) while reading upstream, client: xxx.xxx.xxx, server: , request: "GET /plugins/dynamix.docker.manager/include/CreateDocker.php?updateContainer=true&ct[]=PlexMediaServer HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.xxx.xxx", referrer: "http://192.168.xxx.xxx/Docker" as soon as this error popped, docker was not responsible anymore. Couldn't even reboot the system with: "reboot // shutdown -hP now // powerdown // powerdown -r" Nothing happened, had to force shutdown the server with keeping power button pressed.
March 23, 20188 yr Yes, this is the common denominator between users with the problem. I tend to say that Docker hangs at a certain point, and when the GUI needs to query some status (it calls docker for that) it subsequently hangs the GUI interaction too. Don't know if it related but there is a TCP bug in the kernel, which can cause strange behavior on applications doing TCP communication, such as a docker container doing file downloads.
March 23, 20188 yr 3 hours ago, bonienl said: Difficult to replicate, never happens on my systems. A next time when this happens can you try the following: /etc/rc.d/rc.php-fpm restart It stopped and started without issue but didn't seem to make any difference to the docker tab unfortunately
March 23, 20188 yr 1 hour ago, Uncledome said: So, I am currently back at home and checked the log I saved before downgrading to 6.4.1 This is the last error I had: Mar 20 16:23:13 Tower nginx: 2018/03/20 16:23:13 [error] 26217#26217: *455803 upstream timed out (110: Connection timed out) while reading upstream, client: xxx.xxx.xxx, server: , request: "GET /plugins/dynamix.docker.manager/include/CreateDocker.php?updateContainer=true&ct[]=PlexMediaServer HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.xxx.xxx", referrer: "http://192.168.xxx.xxx/Docker" as soon as this error popped, docker was not responsible anymore. Couldn't even reboot the system with: "reboot // shutdown -hP now // powerdown // powerdown -r" Nothing happened, had to force shutdown the server with keeping power button pressed. Not that it's any better than holding down the power button but if you can still get a ssh session, you can do the following to force a reboot echo 1 > /proc/sys/kernel/sysrq echo b > /proc/sysrq-trigger https://major.io/2009/01/29/linux-emergency-reboot-or-shutdown-with-magic-commands/
March 23, 20188 yr Just now, michaelmcq said: It stopped and started without issue but didn't seem to make any difference to the docker tab unfortunately Ok, thanks for testing. It confirms the diagnostics info that the php-fpm process is not killed, but rather becomes unresponsive due to "something" hanging.
March 23, 20188 yr Just to throw this into the thread: Mar 23 17:24:07 ninja nginx: 2018/03/23 17:24:07 [error] 22572#22572: *278287 upstream timed out (110: Connection timed out) while reading response header from upstream, client: X.X.X.X, server: , request: "POST /webGui/include/DashboardApps.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXX.unraid.net", referrer: "https://XXX.unraid.net/Dashboard" After upgrading to 6.5.0 I had to fix my own app (running in docker): https://github.com/rix1337/RSScrawler/commit/4a34e5d0f6be7ed4af193b51297b1f89370c8978 When I sent to many simulatenous requests to this app (built on flask) it crashed with the https://stackoverflow.com/questions/12591760/flask-broken-pipe-with-requests broken pipe error. This also resulted in the Dashboard losing access to docker/vm. My Dashboard still locks up (though less frequently so), so I guess some tcp-related error from any of my running docker containers is "able" to crash the whole thing. This likely stems from the new (and awesome) socket implementation coupled with the tcp bug.. but I dont know Edited March 23, 20188 yr by rix
March 23, 20188 yr 9 minutes ago, rix said: My Dashboard still locks up (though less frequently so), so I guess some tcp-related error from any of my running docker containers is "able" to crash the whole thing. When you stop all containers. Does the Dashboard still hangs?
March 23, 20188 yr The thing is, the whole Docker Section is missing (notice also the last Parity check after unclean shutdown #3) On the docker page itself, I get only the spinner So basically after a while the complete docker section of the webUI "dissappears" even though all Containers are still happily running. Even my automatic appdata Backup at 4:00 am including the container restart/update does work. Its really all about not being displayed in the frontend. EDiT: Stopping the array will not work. Unraid just hangs, if I try. Apart from that it happily chugs on as usual. Edited March 23, 20188 yr by rix
Archived
This topic is now archived and is closed to further replies.