March 23, 20188 yr 8 minutes ago, rix said: Its really all about not being displayed in the frontend. The frontend calls the docker service to obtain status information. It seems the docker service itself gets stuck. I have no clue why, because I can't recreate this on any of my systems. Hence my question if you stop all containers from running, do you experience still the same issue?
March 23, 20188 yr Is there a docker command to stop them all? I can’t do that at the minute, but will try over the weekend at some point. As an aside I ran “docker stats” from ssh and it doesn’t return anything, have to ctrl c to get a prompt back “docker ps” does return successfully
March 23, 20188 yr There isn't a single command to stop containers at once, but some trickery can do: docker stop $(docker ps -a --filter status='running' --format='{{.Names}}')
March 23, 20188 yr For the last 3 hours I have been watching the episodes of Planet Earth 2, streamed to my tv using Plex. At the same time another Plex video stream goes to my PC and a third Emby video stream goes to yet another PC. Meanwhile 10 container web GUIs are opened to serve requests. And of course the unRAID GUI itself is open. No crashes, no hangups, all works perfect when clicking thru the GUI and obtaining all kinds of page information
March 23, 20188 yr Can’t stop them all, one of them, a node docker for the Sonos api that I added wouldn’t stop. The others all have. I’ve subsequently tried to restart the docker service with /etc/rc.d/rc.docker restart but that’s not working, looks like I might be doing a forced reboot in a bit and then disabling that particular docker
March 23, 20188 yr Just to chime in... I have the same issues/symptoms as the other folks who have posted here.
March 23, 20188 yr 7 minutes ago, bdillahu said: Just to chime in... I have the same issues/symptoms as the other folks who have posted here. Please check if you are able to stop your containers.
March 23, 20188 yr 5 minutes ago, bonienl said: Please check if you are able to stop your containers. All but one of them.. "minio-unraid" won't stop... I'm not docker expert, so not quite sure how to debug, but willing to try if you can tell me things to look for.
March 23, 20188 yr 14 minutes ago, bonienl said: Try to read the details from that container docker inspect minio-unraid Just hangs, doesn't respond...
March 24, 20188 yr So, I killed every process with "docker" that I could find and restarted rc.docker (cd /etc/rc.d; rc.docker start) and things seem to be working for now...
March 24, 20188 yr 9 hours ago, bdillahu said: All but one of them.. "minio-unraid" won't stop... I'm not docker expert, so not quite sure how to debug, but willing to try if you can tell me things to look for. For me it is diginc/pi-hole Last login: Fri Mar 23 07:25:31 2018 Linux 4.14.26-unRAID. root@ninja:~# docker stop $(docker ps -a --filter status='running' --format='{{.Names}}') Ubooquity Sonarr-Processing SABnzbd-VPN Plex Organizr Jackett Headphones Duplicati HandBrake Deluge-VPN HomeAssistant Tautulli RSScrawler MKVToolNix JDownloader Avidemux DNScrypt Verarbeitung SyncLounge Ripper ^C root@ninja:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 80680360fa8f diginc/pi-hole "/s6-init" 3 weeks ago Up 2 days (healthy) Pi-Hole root@ninja:~# docker inspect Pi-Hole ^C root@ninja:~# pkill -f docker Has worked, now updated / rebooting to the latest rc Edited March 24, 20188 yr by rix
March 24, 20188 yr Can you test again with unRAID-6.5.1-rc2. This version has the correction for the TCP bug and addresses the issue when docker "hangs" on reading container information.
March 24, 20188 yr Yep, am updating right now. Will not deactivate the faulty container. Thanks for the kill all docker processes hint. No unclean shutdown while updating to rc2! Edited March 24, 20188 yr by rix
March 24, 20188 yr Thanks @bonienl the update seems to have done the trick. Will post back here, if the issue should arise again.
March 24, 20188 yr Just as I posted it happened again. Mar 24 11:46:26 ninja nginx: 2018/03/24 11:46:26 [error] 6302#6302: *15078 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: , request: "POST /webGui/include/DashboardApps.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXYYZZ.unraid.net", referrer: "https://XXYYZZ.unraid.net/Dashboard" This time running the 6.5.1 RC2
March 24, 20188 yr 5 minutes ago, rix said: Just as I posted it happened again. Mar 24 11:46:26 ninja nginx: 2018/03/24 11:46:26 [error] 6302#6302: *15078 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: , request: "POST /webGui/include/DashboardApps.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXYYZZ.unraid.net", referrer: "https://XXYYZZ.unraid.net/Dashboard" This time running the 6.5.1 RC2 If you wait long enough, does the list of Docker containers eventually appear or you keep a spinner on that page? Edited March 24, 20188 yr by bonienl
March 24, 20188 yr 13 minutes ago, bonienl said: If you wait long enough, does the list of Docker containers eventually appear or you keep a spinner on that page? Infinite spinner across Firefox/Edge,.. also checked using private tabs. The same error (see above) appeared four times now. To me it seems that after a minute of timeout it fails to connect again: Mar 24 12:00:19 ninja nginx: 2018/03/24 12:00:19 [error] 6302#6302: *17385 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: , request: "POST /plugins/dynamix.docker.manager/include/DockerUpdate.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXYYZZ.unraid.net", referrer: "https://XXYYZZ.unraid.net/Docker" Mar 24 12:00:19 ninja nginx: 2018/03/24 12:00:19 [error] 6302#6302: *17385 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: , request: "POST /plugins/ca.docker.autostart/include/exec.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXYYZZ.unraid.net", referrer: "https://XXYYZZ.unraid.net/Docker" Mar 24 12:02:20 ninja nginx: 2018/03/24 12:02:20 [error] 6302#6302: *18397 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: , request: "POST /plugins/dynamix.docker.manager/include/DockerUpdate.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXYYZZ.unraid.net", referrer: "https://XXYYZZ.unraid.net/Docker" Mar 24 12:02:20 ninja nginx: 2018/03/24 12:02:20 [error] 6302#6302: *18397 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: , request: "POST /plugins/ca.docker.autostart/include/exec.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXYYZZ.unraid.net", referrer: "https://XXYYZZ.unraid.net/Docker" Mar 24 12:03:02 ninja nginx: 2018/03/24 12:03:02 [error] 6302#6302: *18528 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: , request: "POST /plugins/dynamix.docker.manager/include/DockerUpdate.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXYYZZ.unraid.net", referrer: "https://XXYYZZ.unraid.net/Docker" Mar 24 12:03:02 ninja nginx: 2018/03/24 12:03:02 [error] 6302#6302: *18528 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: , request: "POST /plugins/ca.docker.autostart/include/exec.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXYYZZ.unraid.net", referrer: "https://XXYYZZ.unraid.net/Docker" Should I try restarting the php5 process as described above? EDiT1: Currently running 2 transcodes inside my containers so I wont reboot immediately, but I now know how to reboot cleanly (killing all docker-related processes, then rebooting). EDiT2: Related JS-Error: The connection to https://XXYYZZ.unraid.net/sub/var was interrupted while the page was loading. (dynamix.js:34:10380) Edited March 24, 20188 yr by rix
March 24, 20188 yr No the php-fpm process is not the problem. If you don't mind some 'hacking'. In the file /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php is the following statement at the near top. return exec("timeout 20 /usr/bin/docker $cmd 2>/dev/null",$var); Can you change the timeout value of 20 to 5. After the change simply reload the Docker page in your browser. Edited March 24, 20188 yr by bonienl
March 24, 20188 yr 24 minutes ago, bonienl said: No the php-fpm process is not the problem. If you don't mind some 'hacking'. In the file /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php is the following statement at the near top. return exec("timeout 20 /usr/bin/docker $cmd 2>/dev/null",$var); Can you change the timeout value of 20 to 5. After the change simply reload the Docker page in your browser. I tweaked the timeout and saved the file: The spinner persists. Further: The php5-fpm.sock error does not pop up on loading the Docker page, even though it seems overall related. It appeared ~3 minutes after loading the Docker page. Here are the latest entries in my log: Mar 24 12:30:29 ninja kernel: TCP: request_sock_TCP: Possible SYN flooding on port 8181. Sending cookies. Check SNMP counters. Mar 24 12:32:47 ninja kernel: mdcmd (53): spindown 4 Mar 24 12:35:07 ninja sshd[9358]: Accepted password for root from XX.XX.XX.XX port 52543 ssh2 Mar 24 12:41:02 ninja nginx: 2018/03/24 12:41:02 [error] 6302#6302: *23512 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: , request: "POST /plugins/dynamix.docker.manager/include/DockerUpdate.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXYYZZ.unraid.net", referrer: "https://XXYYZZ.unraid.net/Docker" Mar 24 12:41:02 ninja nginx: 2018/03/24 12:41:02 [error] 6302#6302: *23512 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: , request: "POST /plugins/ca.docker.autostart/include/exec.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "XXYYZZ.unraid.net", referrer: "https://XXYYZZ.unraid.net/Docker" The SYN flooding one is new to me.
March 24, 20188 yr 3 hours ago, rix said: The SYN flooding one is new to me. Harmless, this comes from Tautulli, I see this as well. I have installed & configured pi-hole, but can not recreate the issue. Pi-hole is happily running (and blocking ads), I can start/stop it without issue. Docker page stays accessible all the time.
March 24, 20188 yr 4 hours ago, bonienl said: Harmless, this comes from Tautulli, I see this as well. I have installed & configured pi-hole, but can not recreate the issue. Pi-hole is happily running (and blocking ads), I can start/stop it without issue. Docker page stays accessible all the time. OK, I have kept pi-hole stopped for now. The issue has since been resolved. That doesn't solve its cause but I have no idea how to get to the bottom of this as of now. Thanks for your support!
March 24, 20188 yr Thanks for testing with me. With pi-hole stopped can you get its information: docker inspect pi-hole
March 24, 20188 yr Sure thing: root@ninja:~# docker inspect Pi-Hole [ { "Id": "80680360fa8faa1bc88db1c2734502f98adfd4bea62f261673e24f4bf6533b01", "Created": "2018-02-27T03:08:50.116377106Z", "Path": "/s6-init", "Args": [], "State": { "Status": "exited", "Running": false, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 0, "ExitCode": 0, "Error": "", "StartedAt": "2018-03-24T14:44:41.38719067Z", "FinishedAt": "2018-03-24T14:45:27.583023854Z", "Health": { "Status": "unhealthy", "FailingStreak": 0, "Log": [ { "Start": "2018-03-24T11:42:11.763336777+01:00", "End": "2018-03-24T11:42:22.104227867+01:00", "ExitCode": 0, "Output": "\n; <<>> DiG 9.10.3-P4-Debian <<>> @127.0.0.1 pi.hole\n; (1 server found)\n;; global options: +cmd\n;; Got answer:\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56472\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version: 0, flags:; udp: 4096\n;; QUESTION SECTION:\n;pi.hole.\t\t\tIN\tA\n\n;; ANSWER SECTION:\npi.hole.\t\t300\tIN\tA\t10.10.20.40\n\n;; Query time: 1 msec\n;; SERVER: 127.0.0.1#53(127.0.0.1)\n;; WHEN: Sat Mar 24 11:42:22 CET 2018\n;; MSG SIZE rcvd: 52\n\n" }, { "Start": "2018-03-24T11:42:52.531040003+01:00", "End": "2018-03-24T11:42:52.625293238+01:00", "ExitCode": 0, "Output": "\n; <<>> DiG 9.10.3-P4-Debian <<>> @127.0.0.1 pi.hole\n; (1 server found)\n;; global options: +cmd\n;; Got answer:\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29373\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version: 0, flags:; udp: 4096\n;; QUESTION SECTION:\n;pi.hole.\t\t\tIN\tA\n\n;; ANSWER SECTION:\npi.hole.\t\t300\tIN\tA\t10.10.20.40\n\n;; Query time: 1 msec\n;; SERVER: 127.0.0.1#53(127.0.0.1)\n;; WHEN: Sat Mar 24 11:42:52 CET 2018\n;; MSG SIZE rcvd: 52\n\n" }, { "Start": "2018-03-24T11:43:28.891350036+01:00", "End": "2018-03-24T11:43:29.018972396+01:00", "ExitCode": 0, "Output": "\n; <<>> DiG 9.10.3-P4-Debian <<>> @127.0.0.1 pi.hole\n; (1 server found)\n;; global options: +cmd\n;; Got answer:\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7335\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version: 0, flags:; udp: 4096\n;; QUESTION SECTION:\n;pi.hole.\t\t\tIN\tA\n\n;; ANSWER SECTION:\npi.hole.\t\t300\tIN\tA\t10.10.20.40\n\n;; Query time: 3 msec\n;; SERVER: 127.0.0.1#53(127.0.0.1)\n;; WHEN: Sat Mar 24 11:43:29 CET 2018\n;; MSG SIZE rcvd: 52\n\n" }, { "Start": "2018-03-24T11:44:00.160101955+01:00", "End": "2018-03-24T11:44:30.1610434+01:00", "ExitCode": -1, "Output": "Health check exceeded timeout (30s)" }, { "Start": "2018-03-24T15:45:11.387398931+01:00", "End": "2018-03-24T15:45:11.434955272+01:00", "ExitCode": 0, "Output": "\n; <<>> DiG 9.10.3-P4-Debian <<>> @127.0.0.1 pi.hole\n; (1 server found)\n;; global options: +cmd\n;; Got answer:\n;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14818\n;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version: 0, flags:; udp: 4096\n;; QUESTION SECTION:\n;pi.hole.\t\t\tIN\tA\n\n;; ANSWER SECTION:\npi.hole.\t\t300\tIN\tA\t10.10.20.40\n\n;; Query time: 0 msec\n;; SERVER: 127.0.0.1#53(127.0.0.1)\n;; WHEN: Sat Mar 24 15:45:11 CET 2018\n;; MSG SIZE rcvd: 52\n\n" } ] } }, "Image": "sha256:bb640716ae33ec36f3c2d359443e4afa40461dc5981c1716987655d8b3987628", "ResolvConfPath": "/var/lib/docker/containers/80680360fa8faa1bc88db1c2734502f98adfd4bea62f261673e24f4bf6533b01/resolv.conf", "HostnamePath": "/var/lib/docker/containers/80680360fa8faa1bc88db1c2734502f98adfd4bea62f261673e24f4bf6533b01/hostname", "HostsPath": "/var/lib/docker/containers/80680360fa8faa1bc88db1c2734502f98adfd4bea62f261673e24f4bf6533b01/hosts", "LogPath": "/var/lib/docker/containers/80680360fa8faa1bc88db1c2734502f98adfd4bea62f261673e24f4bf6533b01/80680360fa8faa1bc88db1c2734502f98adfd4bea62f261673e24f4bf6533b01-json.log", "Name": "/Pi-Hole", "RestartCount": 0, "Driver": "btrfs", "Platform": "linux", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "", "ExecIDs": null, "HostConfig": { "Binds": [ "/mnt/user/appdata/pihole/pihole/:/etc/pihole/:rw", "/mnt/user/appdata/pihole/dnsmasq.d/:/etc/dnsmasq.d/:rw" ], "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "br0", "PortBindings": {}, "RestartPolicy": { "Name": "no", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": [ "NET_ADMIN" ], "CapDrop": null, "Dns": [], "DnsOptions": [], "DnsSearch": [], "ExtraHosts": null, "GroupAdd": null, "IpcMode": "shareable", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "runc", "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": [], "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [], "DeviceCgroupRules": null, "DiskQuota": 0, "KernelMemory": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": false, "PidsLimit": 0, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0 }, "GraphDriver": { "Data": null, "Name": "btrfs" }, "Mounts": [ { "Type": "bind", "Source": "/mnt/user/appdata/pihole/dnsmasq.d", "Destination": "/etc/dnsmasq.d", "Mode": "rw", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/mnt/user/appdata/pihole/pihole", "Destination": "/etc/pihole", "Mode": "rw", "RW": true, "Propagation": "rprivate" } ], "Config": { "Hostname": "80680360fa8f", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": true, "AttachStderr": true, "ExposedPorts": { "53/tcp": {}, "53/udp": {}, "80/tcp": {} }, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "TZ=Europe/Berlin", "HOST_OS=unRAID", "TCP_PORT_53=53", "UDP_PORT_53=53", "TCP_PORT_80=80", "ServerIP=10.10.20.40", "DNS1=10.10.20.30", "DNS2=10.10.20.30", "IPv6=False", "TZ=Europe/Berlin", "WEBPASSWORD=m2ayAKZntvzyDR5nRR23A8JvGS85Bp", "PATH=/opt/pihole:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "TAG=debian", "ARCH=amd64", "setupVars=/etc/pihole/setupVars.conf", "PIHOLE_INSTALL=/tmp/ph_install.sh", "S6OVERLAY_RELEASE=https://github.com/just-containers/s6-overlay/releases/download/v1.21.2.2/s6-overlay-amd64.tar.gz", "PHP_ENV_CONFIG=/etc/lighttpd/conf-enabled/15-fastcgi-php.conf", "PHP_ERROR_LOG=/var/log/lighttpd/error.log", "S6_LOGGING=0", "S6_KEEP_ENV=1", "S6_BEHAVIOUR_IF_STAGE2_FAILS=2" ], "Cmd": null, "Healthcheck": { "Test": [ "CMD-SHELL", "dig @127.0.0.1 pi.hole || exit 1" ] }, "ArgsEscaped": true, "Image": "diginc/pi-hole", "Volumes": null, "WorkingDir": "", "Entrypoint": [ "/s6-init" ], "OnBuild": null, "Labels": { "image": "diginc/pi-hole:debian_amd64", "maintainer": "[email protected]", "url": "https://www.github.com/diginc/docker-pi-hole" } }, "NetworkSettings": { "Bridge": "", "SandboxID": "e5b92ca209a817a17c2d91687b361ca7449de6e3d037748e6eea93dd6de57528", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": {}, "SandboxKey": "/var/run/docker/netns/e5b92ca209a8", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": { "br0": { "IPAMConfig": { "IPv4Address": "10.10.20.40" }, "Links": null, "Aliases": [ "80680360fa8f" ], "NetworkID": "3d68d516eab65c46e225eb8150b3d4a93cc55dd540038c845d149e13d086e602", "EndpointID": "", "Gateway": "", "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "", "DriverOpts": null } } } } ]
Archived
This topic is now archived and is closed to further replies.