bonienl Posted March 24, 2018 Share Posted March 24, 2018 Your container is declared "unhealthy", which is the only difference I see with my container. "Health": { "Status": "unhealthy", I am not sure how to get it "healthy", perhaps deleting the container+image and re-install may help? Quote Link to comment
bonienl Posted March 24, 2018 Share Posted March 24, 2018 May I ask everybody who had the Docker tab hanging to execute the below command (just copy + paste) and see if this succeeds. docker inspect --format='{{.State.Running}}#{{range $p,$c := .HostConfig.PortBindings}}{{$p}}:{{(index $c 0).HostPort}}|{{end}}#{{range $p,$c := .Config.ExposedPorts}}{{$p}}|{{end}}#{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}#{{range $c := .HostConfig.Binds}}{{$c}}|{{end}}' $(docker ps -a --format='{{.Names}}') Thanks Quote Link to comment
Uncledome Posted March 24, 2018 Share Posted March 24, 2018 Just upgraded again to 6.5.0 to test this, but will have to downgrade before sunday evening because I am a week away and cannot restart the server. Currently it looks to be running and your command just showed me all my containers on true except two, which had false. Quote Link to comment
bonienl Posted March 24, 2018 Share Posted March 24, 2018 1 minute ago, Uncledome said: Just upgraded again to 6.5.0 to test this, but will have to downgrade before sunday evening because I am a week away and cannot restart the server. Currently it looks to be running and your command just showed me all my containers on true except two, which had false. Ok, thanks for testing. Please try this command again when you have the "failing" page situation. Quote Link to comment
Uncledome Posted March 24, 2018 Share Posted March 24, 2018 Just now, bonienl said: Ok, thanks for testing. Please try this command again when you have the "failing" page situation. Will do and will let you know when and what happens. Quote Link to comment
cschanot Posted March 25, 2018 Share Posted March 25, 2018 I am having this issue as well, I am attempting to figure out what is causing the hanging, only recent system change besides upgrading to 6.5.0 was adding another nic and changing to active-backup. Same symptoms as everyone else. Quote Link to comment
cschanot Posted March 25, 2018 Share Posted March 25, 2018 4 hours ago, bonienl said: docker inspect --format='{{.State.Running}}#{{range $p,$c := .HostConfig.PortBindings}}{{$p}}:{{(index $c 0).HostPort}}|{{end}}#{{range $p,$c := .Config.ExposedPorts}}{{$p}}|{{end}}#{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}#{{range $c := .HostConfig.Binds}}{{$c}}|{{end}}' $(docker ps -a --format='{{.Names}}') This command and docker stats hang for me as well. Nothing unusually memory or cpu usage wise either. I will report back if I figure anything out. Quote Link to comment
cschanot Posted March 25, 2018 Share Posted March 25, 2018 (edited) After running docker stats --no-stream on each container separately it only would hang on pi-hole. docker stop and docker kill would not work on the pi-hole container either so I had to kill the process. Once that was done everything was responsive again. I will try to figure out what the issue is with pi-hole tomorrow if I have time as I would like to continue running it. Thanks to bonienl and everyone else in this thread for helping to point me in the right direction. So not sure if this will work long term but I will report back..for 2 days pi-hole has been running with no issues. I changed br0 to bond0 and that was it. I thought it might have something to do with networking after reading about a more general docker issue with hanging because of network settings in the docker. Bond0 works for me as I am running with two network cards bonded as active-backup. This was for the interface variable in pi-hole. Good luck to anyone else having this issue but everything that has been outlined will at least help you identify the docker causing your issue! Edited March 26, 2018 by cschanot Quote Link to comment
michaelmcq Posted March 25, 2018 Share Posted March 25, 2018 Just an update from me, with my troublesome docker turned off everything is working fine, it's not Pi-Hole for me but rather this : https://github.com/chrisns/docker-node-sonos-http-api perhaps there are similarilties, although not obviously to me! I've not had a chance to update to RC2 yet. Quote Link to comment
yippy3000 Posted March 28, 2018 Share Posted March 28, 2018 For me after several rounds with reboots it seems to have fixed itself. The problem hasn't come back in the last 5 days. Quote Link to comment
eubbenhadd Posted March 29, 2018 Share Posted March 29, 2018 FWIW I have the same timeout issue starting with 6.4.1. Didn't have it on 6.3.x and was hoping 6.5.0 would fix problem. Quote Link to comment
local.bin Posted March 30, 2018 Share Posted March 30, 2018 On 24/03/2018 at 9:05 PM, bonienl said: May I ask everybody who had the Docker tab hanging to execute the below command (just copy + paste) and see if this succeeds. docker inspect --format='{{.State.Running}}#{{range $p,$c := .HostConfig.PortBindings}}{{$p}}:{{(index $c 0).HostPort}}|{{end}}#{{range $p,$c := .Config.ExposedPorts}}{{$p}}|{{end}}#{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}#{{range $c := .HostConfig.Binds}}{{$c}}|{{end}}' $(docker ps -a --format='{{.Names}}') Thanks Just hangs for me on 6.5 and I confirm the other reported observations too. Should I update to rc2 or thats confirmed to still have the issue? Thanks Quote Link to comment
bonienl Posted March 30, 2018 Share Posted March 30, 2018 (edited) The issue is with docker itself. 6.5.1-rc2 uses the same docker version as 6.5.0. The temporary workaround is to find out which container is causing docker to hang and stop using this container. Btw it is worthwhile to update to rc2 because it does have several corrections. Edited March 30, 2018 by bonienl Quote Link to comment
local.bin Posted March 30, 2018 Share Posted March 30, 2018 (edited) Ok I see, thanks, so is there a shortened command of yours above to see which are running and being able to shut them down individually? Mmh ok, I am using the dvb kernel for 6.5 and they don't do rc versions so would prefer to stay on 6.5 unless the updates in rc2 are critical? Thanks in advance Edit: I will just use docker stop <container> && docker ps if thats all thats needed... Edit: I am not able to stop dockers individually, the stop command just hangs there. So will stop the array and see if I can isolate the docker with the issue. Edit: Seems to have all hung, unclean shutdown .... Edit: Going back to 6.4.1 again, as its too laggy to resolve quickly and its a production system, so will keep an eye on further updates, resolving this issue Edited March 30, 2018 by local.bin Quote Link to comment
Uncledome Posted March 30, 2018 Share Posted March 30, 2018 On 3/24/2018 at 10:15 PM, bonienl said: Ok, thanks for testing. Please try this command again when you have the "failing" page situation. So, after a week away from home and letting the server just run, I have the same issue again. I am currently running 6.5.0 since a week, here is the error I am receiving in /var/log/syslog Quote Mar 30 12:00:11 Tower nginx: 2018/03/30 12:00:11 [error] 6328#6328: *817760 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.XXX.XXX, server: , request: "POST /plugins/dynamix.docker.manager/include/ContainerManager.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.XXX.XXX", referrer: "http://192.168.XXX.XXX/Docker" When I run the command you have stated, On 3/24/2018 at 10:05 PM, bonienl said: May I ask everybody who had the Docker tab hanging to execute the below command (just copy + paste) and see if this succeeds. docker inspect --format='{{.State.Running}}#{{range $p,$c := .HostConfig.PortBindings}}{{$p}}:{{(index $c 0).HostPort}}|{{end}}#{{range $p,$c := .Config.ExposedPorts}}{{$p}}|{{end}}#{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}#{{range $c := .HostConfig.Binds}}{{$c}}|{{end}}' $(docker ps -a --format='{{.Names}}') Thanks I get a true on every line of my docker containers. Quote Link to comment
bonienl Posted March 30, 2018 Share Posted March 30, 2018 14 minutes ago, Uncledome said: I get a true on every line of my docker containers. Can you show the output? I am not sure what your statement means. # docker inspect --format='... true##19999/tcp|#10.0.105.194#/proc:/host/proc:ro|/sys:/host/sys:ro|/mnt/cache/appdata/netdata:/etc/netdata:rw| true#443/tcp:8443|80/tcp:8080|#443/tcp|80/tcp|#172.17.0.2#/mnt/cache/appdata/heimdall:/config:rw| true##8080/tcp|#10.0.105.192#/var/lib/docker/:/var/lib/docker/:ro|/:/rootfs:ro|/var/run:/var/run:ro|/sys:/sys:ro| true##443/tcp|80/tcp|#10.0.101.80#/mnt/cache/appdata/Nginx:/config:rw| true##53/tcp|53/udp|80/tcp|#10.0.101.99#/mnt/cache/appdata/pihole/pihole/:/etc/pihole/:rw|/mnt/cache/appdata/pihole/dnsmasq.d/:/etc/dnsmasq.d/:rw| true##8080/tcp|8081/tcp|8443/tcp|8843/tcp|8880/tcp|#10.0.101.7#/mnt/cache/appdata/unifi/:/config:rw| true##8181/tcp|#10.0.101.192#/mnt/cache/appdata/tautulli:/logs:rw|/mnt/cache/appdata/tautulli:/config:rw| true##10001/tcp|#10.0.101.193#/mnt/user/:/media:ro|/mnt/cache/appdata/tonido:/config:rw| true###10.0.105.193#/mnt/user/films/:/Movies:ro|/mnt/cache/handbrake/:/Output:rw|/mnt/cache/appdata/handbrake:/nobody/.config/ghb:rw| true##1900/udp|7359/udp|8096/tcp|8920/tcp|#10.0.101.82#/mnt/user:/media:ro|/mnt/cache/appdata/Emby:/config:rw| true##1900/udp|32400/tcp|32400/udp|32469/tcp|32469/udp|5353/udp|#10.0.101.81#/mnt/cache/appdata/plex/:/config:rw|/tmp:/transcode:rw|/mnt/cache/appdata/plex/Backup/:/backup:rw|/mnt/user/:/media:ro| The first "true" means the container is in a running state. The remainder are the settings & status read for each container. Quote Link to comment
Uncledome Posted March 30, 2018 Share Posted March 30, 2018 (edited) 31 minutes ago, bonienl said: Can you show the output? I am not sure what your statement means. # docker inspect --format='... true##19999/tcp|#10.0.105.194#/proc:/host/proc:ro|/sys:/host/sys:ro|/mnt/cache/appdata/netdata:/etc/netdata:rw| true#443/tcp:8443|80/tcp:8080|#443/tcp|80/tcp|#172.17.0.2#/mnt/cache/appdata/heimdall:/config:rw| true##8080/tcp|#10.0.105.192#/var/lib/docker/:/var/lib/docker/:ro|/:/rootfs:ro|/var/run:/var/run:ro|/sys:/sys:ro| true##443/tcp|80/tcp|#10.0.101.80#/mnt/cache/appdata/Nginx:/config:rw| true##53/tcp|53/udp|80/tcp|#10.0.101.99#/mnt/cache/appdata/pihole/pihole/:/etc/pihole/:rw|/mnt/cache/appdata/pihole/dnsmasq.d/:/etc/dnsmasq.d/:rw| true##8080/tcp|8081/tcp|8443/tcp|8843/tcp|8880/tcp|#10.0.101.7#/mnt/cache/appdata/unifi/:/config:rw| true##8181/tcp|#10.0.101.192#/mnt/cache/appdata/tautulli:/logs:rw|/mnt/cache/appdata/tautulli:/config:rw| true##10001/tcp|#10.0.101.193#/mnt/user/:/media:ro|/mnt/cache/appdata/tonido:/config:rw| true###10.0.105.193#/mnt/user/films/:/Movies:ro|/mnt/cache/handbrake/:/Output:rw|/mnt/cache/appdata/handbrake:/nobody/.config/ghb:rw| true##1900/udp|7359/udp|8096/tcp|8920/tcp|#10.0.101.82#/mnt/user:/media:ro|/mnt/cache/appdata/Emby:/config:rw| true##1900/udp|32400/tcp|32400/udp|32469/tcp|32469/udp|5353/udp|#10.0.101.81#/mnt/cache/appdata/plex/:/config:rw|/tmp:/transcode:rw|/mnt/cache/appdata/plex/Backup/:/backup:rw|/mnt/user/:/media:ro| The first "true" means the container is in a running state. The remainder are the settings & status read for each container. Sure, this is the output of the command: true#8181/tcp:8181|#8181/tcp|#172.17.0.10#/mnt/user/appdata/tautulli:/config:rw|/mnt/user/appdata/PlexMediaServer/Library/Application Support/Plex Media Server/Logs/:/plex_logs:rw| true##1900/udp|3005/tcp|32400/tcp|32410/udp|32412/udp|32413/udp|32414/udp|32469/tcp|8324/tcp|##/mnt/user/appdata/PlexMediaServer/transcode:/transcode:rw|/mnt/disks/DS22_Media:/data:ro,slave|/mnt/user/appdata/PlexMediaServer:/config:rw| true#8989/tcp:8989|#8989/tcp|#172.17.0.8#/mnt/disks/DS22_Media//Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/user/appdata/sonarr:/config:rw|/dev/rtc:/dev/rtc:ro|/mnt/disks/DS22_Media/:/tv:rw,slave| true#7878/tcp:7878|#7878/tcp|#172.17.0.9#/mnt/disks/DS22_Media//Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/disks/DS22_Media/Movies:/movies:rw,slave|/mnt/user/appdata/radarr:/config:rw| true#443/tcp:444|80/tcp:8383|#443/tcp|80/tcp|#172.17.0.13#/mnt/user/appdata/heimdall:/config:rw| true#8090/tcp:8090|#8090/tcp|8091/tcp|#172.17.0.14#/mnt/user/appdata/confluence:/var/atlassian/application-data/confluence:rw,slave| true#80/tcp:8282|#443/tcp|80/tcp|#172.17.0.4#/mnt/user/appdata/organizr:/config:rw| true#5076/tcp:5076|#5075/tcp|5076/tcp|#172.17.0.6#/mnt/user/appdata/hydra2:/config:rw|/mnt/disks/DS22_Media/Downloads/NZBs/Complete:/downloads:ro,slave| true##58846/tcp|58946/tcp|58946/udp|8112/tcp|##/mnt/user/Torrent Downloads/:/downloads:rw|/mnt/user/appdata/deluge:/config:rw| true##10011/tcp|30033/tcp|41144/tcp|9987/udp|##/mnt/user/appdata/teamspeak3:/config:rw| true#8080/tcp:8080|9090/tcp:9090|#8080/tcp|9090/tcp|#172.17.0.7#/mnt/disks/DS22_Media/Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/disks/DS22_Media/Downloads/NZBs/Incomplete:/incomplete-downloads:rw,slave|/mnt/user/appdata/sabnzbd:/config:rw| true#8181/tcp:8182|#8181/tcp|#172.17.0.2#/mnt/disks/DS22_Media/Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/disks/DS22_Media/Music:/music:rw,slave|/mnt/user/appdata/headphones:/config:rw| true#5800/tcp:7803|5900/tcp:7903|#5800/tcp|5900/tcp|#172.17.0.11#/mnt/user/Handbrake/:/storage:ro|/mnt/user/Handbrake/watch:/watch:rw|/mnt/user/Handbrake/output:/output:rw|/mnt/user/appdata/HandBrake:/config:rw| true#8080/tcp:8082|#3389/tcp|8080/tcp|#172.17.0.12#/mnt/user/JDownloader Downloads/:/downloads:rw|/mnt/user/appdata/JDownloader2:/config:rw| true#5432/tcp:5432|#5432/tcp|#172.17.0.5#/mnt/cache/appdata/postgresql:/var/lib/postgresql:rw| true#8080/tcp:8081|#8080/tcp|#172.17.0.3#/:/rootfs:ro|/var/run:/var/run:rw|/sys:/sys:ro|/var/lib/docker/:/var/lib/docker/:ro| is there a way to restart the server without forcing it down through the power button? Web UI is not accessible at the moment, only SSH. reboot & powerdown -r as well as shutdown -hP now does not work. Edited March 30, 2018 by Uncledome Quote Link to comment
bonienl Posted March 30, 2018 Share Posted March 30, 2018 (edited) To restart the GUI try /etc/rc.d/rc.nginx restart /etc/rc.d/rc.php-fpm restart If you don't manage to bring the GUI back, can you SSH in and do diagnostics before you reboot? Edited March 30, 2018 by bonienl Quote Link to comment
Uncledome Posted March 30, 2018 Share Posted March 30, 2018 4 minutes ago, bonienl said: To restart the GUI try /etc/rc.d/rc.nginx restart /etc/rc.d/rc.php-fpm restart Tried both, first command told me nginx was not running so I used the start parameter. Afterwards it told me that the daemon is starting, status parameter confirmed this. the php-fpm could not be restarted, even with a force-quit, it just failed. Quote Link to comment
bonienl Posted March 30, 2018 Share Posted March 30, 2018 14 minutes ago, Uncledome said: Tried both, first command told me nginx was not running so I used the start parameter. Afterwards it told me that the daemon is starting, status parameter confirmed this. the php-fpm could not be restarted, even with a force-quit, it just failed. Need diagnostics to investigate further, but not certain the cause can be revealed. There is a kernel patch included in unRAID 6.5.1-rc2 which solves a TCP bug. This bug caused some inexplainable application hangs. It is worthwhile to upgrrade to rc2. Quote Link to comment
Uncledome Posted March 30, 2018 Share Posted March 30, 2018 12 minutes ago, bonienl said: Need diagnostics to investigate further, but not certain the cause can be revealed. There is a kernel patch included in unRAID 6.5.1-rc2 which solves a TCP bug. This bug caused some inexplainable application hangs. It is worthwhile to upgrrade to rc2. How can I send you the diagnostic without having access to WebUI? I still can access with SSH & WinSCP. Quote Link to comment
MyKroFt Posted March 30, 2018 Share Posted March 30, 2018 I am not home atm to post diags, but when this happens to me, if I manually kill the docker daemon, everything comes back, as I get this hang on community apps looking for updates and the log complains about php-fpm as well as the docker page Myk Quote Link to comment
bonienl Posted March 30, 2018 Share Posted March 30, 2018 18 minutes ago, Uncledome said: How can I send you the diagnostic without having access to WebUI? I still can access with SSH & WinSCP. When you SSH in type: diagnostics This will create a zip file in the /logs folder on you USB device. Normally the USB device is remotely accessible with SMB, e.g. //tower/flash Quote Link to comment
Uncledome Posted March 30, 2018 Share Posted March 30, 2018 (edited) 50 minutes ago, bonienl said: When you SSH in type: diagnostics This will create a zip file in the /logs folder on you USB device. Normally the USB device is remotely accessible with SMB, e.g. //tower/flash Okay, I started the diagnostics a few minutes ago when you posted how to do it. Looks like it us currently still running so I am guessing this might take a while. (at the moment: Starting diagnostics collection...) I'll edit this post once it is finished and I got the zip folder. //EDIT1: Okay, I have waited a few minutes and still nothing is happening. Is there a possibility to force the server to restart without losing the validity of my parity? Otherwise I'll have to force it down using the power button. //EDIT2: Okay, I've went with the force reboot. Had to reboot it afterwards again because the Passphrase Box was missing... The Diagnostic should be attached. (Made the Diag after reboot) tower-diagnostics-20180330-0552.zip Edited March 30, 2018 by Uncledome Quote Link to comment
Uncledome Posted March 30, 2018 Share Posted March 30, 2018 (edited) Okay, Server has issues again. Guess I'll be downgrading to 6.4.1 as I had no issues there. Just for your info: Quote Mar 30 08:40:12 Tower nginx: 2018/03/30 08:40:12 [error] 6209#6209: *28369 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.XXX.XXX, server: , request: "POST /webGui/include/DashUpdate.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.XXX.XXX", referrer: "http://192.168.XXX.XXX/Dashboard" Diagnostic won't work, wether trying through WebUI nor directly from CLI. Just hangs right after "Starting diagnostic collection". As for the Docker command: true#8181/tcp:8181|#8181/tcp|#172.17.0.9#/mnt/user/appdata/tautulli:/config:rw|/mnt/user/appdata/PlexMediaServer/Library/Application Support/Plex Media Server/Logs/:/plex_logs:rw| true##1900/udp|3005/tcp|32400/tcp|32410/udp|32412/udp|32413/udp|32414/udp|32469/tcp|8324/tcp|##/mnt/user/appdata/PlexMediaServer/transcode:/transcode:rw|/mnt/disks/DS22_Media:/data:ro,slave|/mnt/user/appdata/PlexMediaServer:/config:rw| true#8989/tcp:8989|#8989/tcp|#172.17.0.7#/mnt/disks/DS22_Media//Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/user/appdata/sonarr:/config:rw|/dev/rtc:/dev/rtc:ro|/mnt/disks/DS22_Media/:/tv:rw,slave| true#7878/tcp:7878|#7878/tcp|#172.17.0.8#/mnt/disks/DS22_Media//Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/disks/DS22_Media/Movies:/movies:rw,slave|/mnt/user/appdata/radarr:/config:rw| true#443/tcp:444|80/tcp:8383|#443/tcp|80/tcp|#172.17.0.12#/mnt/user/appdata/heimdall:/config:rw| true#8090/tcp:8090|#8090/tcp|8091/tcp|#172.17.0.13#/mnt/user/appdata/confluence:/var/atlassian/application-data/confluence:rw,slave| true#80/tcp:8282|#443/tcp|80/tcp|#172.17.0.3#/mnt/user/appdata/organizr:/config:rw| true#5076/tcp:5076|#5075/tcp|5076/tcp|#172.17.0.5#/mnt/user/appdata/hydra2:/config:rw|/mnt/disks/DS22_Media/Downloads/NZBs/Complete:/downloads:ro,slave| false##58846/tcp|58946/tcp|58946/udp|8112/tcp|##/mnt/user/Torrent Downloads/:/downloads:rw|/mnt/user/appdata/deluge:/config:rw| true##10011/tcp|30033/tcp|41144/tcp|9987/udp|##/mnt/user/appdata/teamspeak3:/config:rw| true#8080/tcp:8080|9090/tcp:9090|#8080/tcp|9090/tcp|#172.17.0.6#/mnt/disks/DS22_Media/Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/disks/DS22_Media/Downloads/NZBs/Incomplete:/incomplete-downloads:rw,slave|/mnt/user/appdata/sabnzbd:/config:rw| false#8181/tcp:8182|#8181/tcp|#172.17.0.2#/mnt/disks/DS22_Media/Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/disks/DS22_Media/Music:/music:rw,slave|/mnt/user/appdata/headphones:/config:rw| true#5800/tcp:7803|5900/tcp:7903|#5800/tcp|5900/tcp|#172.17.0.10#/mnt/user/Handbrake/:/storage:ro|/mnt/user/Handbrake/watch:/watch:rw|/mnt/user/Handbrake/output:/output:rw|/mnt/user/appdata/HandBrake:/config:rw| true#8080/tcp:8082|#3389/tcp|8080/tcp|#172.17.0.11#/mnt/user/JDownloader Downloads/:/downloads:rw|/mnt/user/appdata/JDownloader2:/config:rw| true#5432/tcp:5432|#5432/tcp|#172.17.0.4#/mnt/cache/appdata/postgresql:/var/lib/postgresql:rw| true#8080/tcp:8081|#8080/tcp|#172.17.0.2#/:/rootfs:ro|/var/run:/var/run:rw|/sys:/sys:ro|/var/lib/docker/:/var/lib/docker/:ro| Deluge and Headphones were offline from the start as they do not automatically start with server startup (i disabled them). Edited March 30, 2018 by Uncledome Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.