March 30, 20188 yr On 24/03/2018 at 9:05 PM, bonienl said: May I ask everybody who had the Docker tab hanging to execute the below command (just copy + paste) and see if this succeeds. docker inspect --format='{{.State.Running}}#{{range $p,$c := .HostConfig.PortBindings}}{{$p}}:{{(index $c 0).HostPort}}|{{end}}#{{range $p,$c := .Config.ExposedPorts}}{{$p}}|{{end}}#{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}#{{range $c := .HostConfig.Binds}}{{$c}}|{{end}}' $(docker ps -a --format='{{.Names}}') Thanks Just hangs for me on 6.5 and I confirm the other reported observations too. Should I update to rc2 or thats confirmed to still have the issue? Thanks
March 30, 20188 yr The issue is with docker itself. 6.5.1-rc2 uses the same docker version as 6.5.0. The temporary workaround is to find out which container is causing docker to hang and stop using this container. Btw it is worthwhile to update to rc2 because it does have several corrections. Edited March 30, 20188 yr by bonienl
March 30, 20188 yr Ok I see, thanks, so is there a shortened command of yours above to see which are running and being able to shut them down individually? Mmh ok, I am using the dvb kernel for 6.5 and they don't do rc versions so would prefer to stay on 6.5 unless the updates in rc2 are critical? Thanks in advance Edit: I will just use docker stop <container> && docker ps if thats all thats needed... Edit: I am not able to stop dockers individually, the stop command just hangs there. So will stop the array and see if I can isolate the docker with the issue. Edit: Seems to have all hung, unclean shutdown .... Edit: Going back to 6.4.1 again, as its too laggy to resolve quickly and its a production system, so will keep an eye on further updates, resolving this issue Edited March 30, 20188 yr by local.bin
March 30, 20188 yr On 3/24/2018 at 10:15 PM, bonienl said: Ok, thanks for testing. Please try this command again when you have the "failing" page situation. So, after a week away from home and letting the server just run, I have the same issue again. I am currently running 6.5.0 since a week, here is the error I am receiving in /var/log/syslog Quote Mar 30 12:00:11 Tower nginx: 2018/03/30 12:00:11 [error] 6328#6328: *817760 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.XXX.XXX, server: , request: "POST /plugins/dynamix.docker.manager/include/ContainerManager.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.XXX.XXX", referrer: "http://192.168.XXX.XXX/Docker" When I run the command you have stated, On 3/24/2018 at 10:05 PM, bonienl said: May I ask everybody who had the Docker tab hanging to execute the below command (just copy + paste) and see if this succeeds. docker inspect --format='{{.State.Running}}#{{range $p,$c := .HostConfig.PortBindings}}{{$p}}:{{(index $c 0).HostPort}}|{{end}}#{{range $p,$c := .Config.ExposedPorts}}{{$p}}|{{end}}#{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}#{{range $c := .HostConfig.Binds}}{{$c}}|{{end}}' $(docker ps -a --format='{{.Names}}') Thanks I get a true on every line of my docker containers.
March 30, 20188 yr 14 minutes ago, Uncledome said: I get a true on every line of my docker containers. Can you show the output? I am not sure what your statement means. # docker inspect --format='... true##19999/tcp|#10.0.105.194#/proc:/host/proc:ro|/sys:/host/sys:ro|/mnt/cache/appdata/netdata:/etc/netdata:rw| true#443/tcp:8443|80/tcp:8080|#443/tcp|80/tcp|#172.17.0.2#/mnt/cache/appdata/heimdall:/config:rw| true##8080/tcp|#10.0.105.192#/var/lib/docker/:/var/lib/docker/:ro|/:/rootfs:ro|/var/run:/var/run:ro|/sys:/sys:ro| true##443/tcp|80/tcp|#10.0.101.80#/mnt/cache/appdata/Nginx:/config:rw| true##53/tcp|53/udp|80/tcp|#10.0.101.99#/mnt/cache/appdata/pihole/pihole/:/etc/pihole/:rw|/mnt/cache/appdata/pihole/dnsmasq.d/:/etc/dnsmasq.d/:rw| true##8080/tcp|8081/tcp|8443/tcp|8843/tcp|8880/tcp|#10.0.101.7#/mnt/cache/appdata/unifi/:/config:rw| true##8181/tcp|#10.0.101.192#/mnt/cache/appdata/tautulli:/logs:rw|/mnt/cache/appdata/tautulli:/config:rw| true##10001/tcp|#10.0.101.193#/mnt/user/:/media:ro|/mnt/cache/appdata/tonido:/config:rw| true###10.0.105.193#/mnt/user/films/:/Movies:ro|/mnt/cache/handbrake/:/Output:rw|/mnt/cache/appdata/handbrake:/nobody/.config/ghb:rw| true##1900/udp|7359/udp|8096/tcp|8920/tcp|#10.0.101.82#/mnt/user:/media:ro|/mnt/cache/appdata/Emby:/config:rw| true##1900/udp|32400/tcp|32400/udp|32469/tcp|32469/udp|5353/udp|#10.0.101.81#/mnt/cache/appdata/plex/:/config:rw|/tmp:/transcode:rw|/mnt/cache/appdata/plex/Backup/:/backup:rw|/mnt/user/:/media:ro| The first "true" means the container is in a running state. The remainder are the settings & status read for each container.
March 30, 20188 yr 31 minutes ago, bonienl said: Can you show the output? I am not sure what your statement means. # docker inspect --format='... true##19999/tcp|#10.0.105.194#/proc:/host/proc:ro|/sys:/host/sys:ro|/mnt/cache/appdata/netdata:/etc/netdata:rw| true#443/tcp:8443|80/tcp:8080|#443/tcp|80/tcp|#172.17.0.2#/mnt/cache/appdata/heimdall:/config:rw| true##8080/tcp|#10.0.105.192#/var/lib/docker/:/var/lib/docker/:ro|/:/rootfs:ro|/var/run:/var/run:ro|/sys:/sys:ro| true##443/tcp|80/tcp|#10.0.101.80#/mnt/cache/appdata/Nginx:/config:rw| true##53/tcp|53/udp|80/tcp|#10.0.101.99#/mnt/cache/appdata/pihole/pihole/:/etc/pihole/:rw|/mnt/cache/appdata/pihole/dnsmasq.d/:/etc/dnsmasq.d/:rw| true##8080/tcp|8081/tcp|8443/tcp|8843/tcp|8880/tcp|#10.0.101.7#/mnt/cache/appdata/unifi/:/config:rw| true##8181/tcp|#10.0.101.192#/mnt/cache/appdata/tautulli:/logs:rw|/mnt/cache/appdata/tautulli:/config:rw| true##10001/tcp|#10.0.101.193#/mnt/user/:/media:ro|/mnt/cache/appdata/tonido:/config:rw| true###10.0.105.193#/mnt/user/films/:/Movies:ro|/mnt/cache/handbrake/:/Output:rw|/mnt/cache/appdata/handbrake:/nobody/.config/ghb:rw| true##1900/udp|7359/udp|8096/tcp|8920/tcp|#10.0.101.82#/mnt/user:/media:ro|/mnt/cache/appdata/Emby:/config:rw| true##1900/udp|32400/tcp|32400/udp|32469/tcp|32469/udp|5353/udp|#10.0.101.81#/mnt/cache/appdata/plex/:/config:rw|/tmp:/transcode:rw|/mnt/cache/appdata/plex/Backup/:/backup:rw|/mnt/user/:/media:ro| The first "true" means the container is in a running state. The remainder are the settings & status read for each container. Sure, this is the output of the command: true#8181/tcp:8181|#8181/tcp|#172.17.0.10#/mnt/user/appdata/tautulli:/config:rw|/mnt/user/appdata/PlexMediaServer/Library/Application Support/Plex Media Server/Logs/:/plex_logs:rw| true##1900/udp|3005/tcp|32400/tcp|32410/udp|32412/udp|32413/udp|32414/udp|32469/tcp|8324/tcp|##/mnt/user/appdata/PlexMediaServer/transcode:/transcode:rw|/mnt/disks/DS22_Media:/data:ro,slave|/mnt/user/appdata/PlexMediaServer:/config:rw| true#8989/tcp:8989|#8989/tcp|#172.17.0.8#/mnt/disks/DS22_Media//Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/user/appdata/sonarr:/config:rw|/dev/rtc:/dev/rtc:ro|/mnt/disks/DS22_Media/:/tv:rw,slave| true#7878/tcp:7878|#7878/tcp|#172.17.0.9#/mnt/disks/DS22_Media//Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/disks/DS22_Media/Movies:/movies:rw,slave|/mnt/user/appdata/radarr:/config:rw| true#443/tcp:444|80/tcp:8383|#443/tcp|80/tcp|#172.17.0.13#/mnt/user/appdata/heimdall:/config:rw| true#8090/tcp:8090|#8090/tcp|8091/tcp|#172.17.0.14#/mnt/user/appdata/confluence:/var/atlassian/application-data/confluence:rw,slave| true#80/tcp:8282|#443/tcp|80/tcp|#172.17.0.4#/mnt/user/appdata/organizr:/config:rw| true#5076/tcp:5076|#5075/tcp|5076/tcp|#172.17.0.6#/mnt/user/appdata/hydra2:/config:rw|/mnt/disks/DS22_Media/Downloads/NZBs/Complete:/downloads:ro,slave| true##58846/tcp|58946/tcp|58946/udp|8112/tcp|##/mnt/user/Torrent Downloads/:/downloads:rw|/mnt/user/appdata/deluge:/config:rw| true##10011/tcp|30033/tcp|41144/tcp|9987/udp|##/mnt/user/appdata/teamspeak3:/config:rw| true#8080/tcp:8080|9090/tcp:9090|#8080/tcp|9090/tcp|#172.17.0.7#/mnt/disks/DS22_Media/Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/disks/DS22_Media/Downloads/NZBs/Incomplete:/incomplete-downloads:rw,slave|/mnt/user/appdata/sabnzbd:/config:rw| true#8181/tcp:8182|#8181/tcp|#172.17.0.2#/mnt/disks/DS22_Media/Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/disks/DS22_Media/Music:/music:rw,slave|/mnt/user/appdata/headphones:/config:rw| true#5800/tcp:7803|5900/tcp:7903|#5800/tcp|5900/tcp|#172.17.0.11#/mnt/user/Handbrake/:/storage:ro|/mnt/user/Handbrake/watch:/watch:rw|/mnt/user/Handbrake/output:/output:rw|/mnt/user/appdata/HandBrake:/config:rw| true#8080/tcp:8082|#3389/tcp|8080/tcp|#172.17.0.12#/mnt/user/JDownloader Downloads/:/downloads:rw|/mnt/user/appdata/JDownloader2:/config:rw| true#5432/tcp:5432|#5432/tcp|#172.17.0.5#/mnt/cache/appdata/postgresql:/var/lib/postgresql:rw| true#8080/tcp:8081|#8080/tcp|#172.17.0.3#/:/rootfs:ro|/var/run:/var/run:rw|/sys:/sys:ro|/var/lib/docker/:/var/lib/docker/:ro| is there a way to restart the server without forcing it down through the power button? Web UI is not accessible at the moment, only SSH. reboot & powerdown -r as well as shutdown -hP now does not work. Edited March 30, 20188 yr by Uncledome
March 30, 20188 yr To restart the GUI try /etc/rc.d/rc.nginx restart /etc/rc.d/rc.php-fpm restart If you don't manage to bring the GUI back, can you SSH in and do diagnostics before you reboot? Edited March 30, 20188 yr by bonienl
March 30, 20188 yr 4 minutes ago, bonienl said: To restart the GUI try /etc/rc.d/rc.nginx restart /etc/rc.d/rc.php-fpm restart Tried both, first command told me nginx was not running so I used the start parameter. Afterwards it told me that the daemon is starting, status parameter confirmed this. the php-fpm could not be restarted, even with a force-quit, it just failed.
March 30, 20188 yr 14 minutes ago, Uncledome said: Tried both, first command told me nginx was not running so I used the start parameter. Afterwards it told me that the daemon is starting, status parameter confirmed this. the php-fpm could not be restarted, even with a force-quit, it just failed. Need diagnostics to investigate further, but not certain the cause can be revealed. There is a kernel patch included in unRAID 6.5.1-rc2 which solves a TCP bug. This bug caused some inexplainable application hangs. It is worthwhile to upgrrade to rc2.
March 30, 20188 yr 12 minutes ago, bonienl said: Need diagnostics to investigate further, but not certain the cause can be revealed. There is a kernel patch included in unRAID 6.5.1-rc2 which solves a TCP bug. This bug caused some inexplainable application hangs. It is worthwhile to upgrrade to rc2. How can I send you the diagnostic without having access to WebUI? I still can access with SSH & WinSCP.
March 30, 20188 yr I am not home atm to post diags, but when this happens to me, if I manually kill the docker daemon, everything comes back, as I get this hang on community apps looking for updates and the log complains about php-fpm as well as the docker page Myk
March 30, 20188 yr 18 minutes ago, Uncledome said: How can I send you the diagnostic without having access to WebUI? I still can access with SSH & WinSCP. When you SSH in type: diagnostics This will create a zip file in the /logs folder on you USB device. Normally the USB device is remotely accessible with SMB, e.g. //tower/flash
March 30, 20188 yr 50 minutes ago, bonienl said: When you SSH in type: diagnostics This will create a zip file in the /logs folder on you USB device. Normally the USB device is remotely accessible with SMB, e.g. //tower/flash Okay, I started the diagnostics a few minutes ago when you posted how to do it. Looks like it us currently still running so I am guessing this might take a while. (at the moment: Starting diagnostics collection...) I'll edit this post once it is finished and I got the zip folder. //EDIT1: Okay, I have waited a few minutes and still nothing is happening. Is there a possibility to force the server to restart without losing the validity of my parity? Otherwise I'll have to force it down using the power button. //EDIT2: Okay, I've went with the force reboot. Had to reboot it afterwards again because the Passphrase Box was missing... The Diagnostic should be attached. (Made the Diag after reboot) tower-diagnostics-20180330-0552.zip Edited March 30, 20188 yr by Uncledome
March 30, 20188 yr Okay, Server has issues again. Guess I'll be downgrading to 6.4.1 as I had no issues there. Just for your info: Quote Mar 30 08:40:12 Tower nginx: 2018/03/30 08:40:12 [error] 6209#6209: *28369 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.XXX.XXX, server: , request: "POST /webGui/include/DashUpdate.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.XXX.XXX", referrer: "http://192.168.XXX.XXX/Dashboard" Diagnostic won't work, wether trying through WebUI nor directly from CLI. Just hangs right after "Starting diagnostic collection". As for the Docker command: true#8181/tcp:8181|#8181/tcp|#172.17.0.9#/mnt/user/appdata/tautulli:/config:rw|/mnt/user/appdata/PlexMediaServer/Library/Application Support/Plex Media Server/Logs/:/plex_logs:rw| true##1900/udp|3005/tcp|32400/tcp|32410/udp|32412/udp|32413/udp|32414/udp|32469/tcp|8324/tcp|##/mnt/user/appdata/PlexMediaServer/transcode:/transcode:rw|/mnt/disks/DS22_Media:/data:ro,slave|/mnt/user/appdata/PlexMediaServer:/config:rw| true#8989/tcp:8989|#8989/tcp|#172.17.0.7#/mnt/disks/DS22_Media//Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/user/appdata/sonarr:/config:rw|/dev/rtc:/dev/rtc:ro|/mnt/disks/DS22_Media/:/tv:rw,slave| true#7878/tcp:7878|#7878/tcp|#172.17.0.8#/mnt/disks/DS22_Media//Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/disks/DS22_Media/Movies:/movies:rw,slave|/mnt/user/appdata/radarr:/config:rw| true#443/tcp:444|80/tcp:8383|#443/tcp|80/tcp|#172.17.0.12#/mnt/user/appdata/heimdall:/config:rw| true#8090/tcp:8090|#8090/tcp|8091/tcp|#172.17.0.13#/mnt/user/appdata/confluence:/var/atlassian/application-data/confluence:rw,slave| true#80/tcp:8282|#443/tcp|80/tcp|#172.17.0.3#/mnt/user/appdata/organizr:/config:rw| true#5076/tcp:5076|#5075/tcp|5076/tcp|#172.17.0.5#/mnt/user/appdata/hydra2:/config:rw|/mnt/disks/DS22_Media/Downloads/NZBs/Complete:/downloads:ro,slave| false##58846/tcp|58946/tcp|58946/udp|8112/tcp|##/mnt/user/Torrent Downloads/:/downloads:rw|/mnt/user/appdata/deluge:/config:rw| true##10011/tcp|30033/tcp|41144/tcp|9987/udp|##/mnt/user/appdata/teamspeak3:/config:rw| true#8080/tcp:8080|9090/tcp:9090|#8080/tcp|9090/tcp|#172.17.0.6#/mnt/disks/DS22_Media/Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/disks/DS22_Media/Downloads/NZBs/Incomplete:/incomplete-downloads:rw,slave|/mnt/user/appdata/sabnzbd:/config:rw| false#8181/tcp:8182|#8181/tcp|#172.17.0.2#/mnt/disks/DS22_Media/Downloads/NZBs/Complete:/downloads:rw,slave|/mnt/disks/DS22_Media/Music:/music:rw,slave|/mnt/user/appdata/headphones:/config:rw| true#5800/tcp:7803|5900/tcp:7903|#5800/tcp|5900/tcp|#172.17.0.10#/mnt/user/Handbrake/:/storage:ro|/mnt/user/Handbrake/watch:/watch:rw|/mnt/user/Handbrake/output:/output:rw|/mnt/user/appdata/HandBrake:/config:rw| true#8080/tcp:8082|#3389/tcp|8080/tcp|#172.17.0.11#/mnt/user/JDownloader Downloads/:/downloads:rw|/mnt/user/appdata/JDownloader2:/config:rw| true#5432/tcp:5432|#5432/tcp|#172.17.0.4#/mnt/cache/appdata/postgresql:/var/lib/postgresql:rw| true#8080/tcp:8081|#8080/tcp|#172.17.0.2#/:/rootfs:ro|/var/run:/var/run:rw|/sys:/sys:ro|/var/lib/docker/:/var/lib/docker/:ro| Deluge and Headphones were offline from the start as they do not automatically start with server startup (i disabled them). Edited March 30, 20188 yr by Uncledome
March 30, 20188 yr 42 minutes ago, Uncledome said: Just hangs right after "Starting diagnostic collection". Curious about something if you haven't already rebooted. Can you open another terminal, then code cd / ls There's going to be a temporary folder in there named (in your case, Tower-diagnostics-Today'sDate&Time) Can you then ls THATFOLDERNAME/system And post the output
March 30, 20188 yr 9 minutes ago, Squid said: Curious about something if you haven't already rebooted. Can you open another terminal, then code cd / ls There's going to be a temporary folder in there named (in your case, Tower-diagnostics-Today'sDate&Time) Can you then ls THATFOLDERNAME/system And post the output Sure, Output of cd / and ls: root@Tower:~# cd / root@Tower:/# ls bin/ boot/ dev/ etc/ home/ init@ lib/ lib64/ mnt/ proc/ root/ run/ sbin/ sys/ tmp/ tower-diagnostics-20180330-0840/ tower-diagnostics-20180330-0841/ tower-diagnostics-20180330-0848/ usr/ var/ and output of ls Diag Folder: root@Tower:/# ls tower-diagnostics-20180330-0848/ config/ logs/ qemu/ shares/ smart/ system/ unRAID-6.5.0.txt root@Tower:/# I am currently still waiting for your answer before forcing the server down again.
March 30, 20188 yr 1 minute ago, bonienl said: Can you do tree /tower-diagnostics-20180330-0848 Sure root@Tower:/# tree tower-diagnostics-20180330-0848/ tower-diagnostics-20180330-0848/ ├── config ├── logs ├── qemu ├── shares ├── smart ├── system │ ├── df.txt │ ├── lscpu.txt │ ├── lsmod.txt │ ├── lsof.txt │ ├── lspci.txt │ ├── lsscsi.txt │ ├── lsusb.txt │ ├── memory.txt │ ├── ps.txt │ ├── top.txt │ └── vars.txt └── unRAID-6.5.0.txt 6 directories, 12 files root@Tower:/#
March 30, 20188 yr What does this do? df -h and ifconfig -a -s both commands should complete instantaneously. Where diagnostics is hanging up might prove important on bonienl's investigation
March 30, 20188 yr Just now, Squid said: What does this do? df -h and ifconfig -a -s both commands should complete instantaneously. Where diagnostics is hanging up might prove important on bonienl's investigation df -h does nothing, just shows a empty line without text which cannot be exited with CTRL+C. So i have to restart my terminal. ifconfig -a -s shows: root@Tower:~# ifconfig -a -s Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg bond0 1500 23636415 0 38 0 21457397 0 0 0 BMPmRU br0 1500 5307429 0 998 0 4803019 0 0 0 BMRU docker0 1500 1528385 0 0 0 1729053 0 0 0 BMRU erspan0 1450 0 0 0 0 0 0 0 0 BM eth0 1500 23579910 0 38 0 21457397 0 0 0 BMsRU eth1 1500 56505 0 0 0 0 0 0 0 BMsRU gre0 1476 0 0 0 0 0 0 0 0 O gretap0 1462 0 0 0 0 0 0 0 0 BM ip_vti0 1364 0 0 0 0 0 0 0 0 O lo 65536 158225 0 0 0 158225 0 0 0 LRU sit0 1480 0 0 0 0 0 0 0 0 O tunl0 1480 0 0 0 0 0 0 0 0 O veth3121 1500 1190196 0 0 0 1389978 0 0 0 BMRU veth4e02 1500 0 0 0 0 4767 0 0 0 BMRU veth6f3e 1500 3166 0 0 0 7943 0 0 0 BMRU veth78c8 1500 11491 0 0 0 16615 0 0 0 BMRU vethb6b7 1500 0 0 0 0 4722 0 0 0 BMRU vethcf79 1500 3373 0 0 0 8134 0 0 0 BMRU vethdad5 1500 4939 0 0 0 10547 0 0 0 BMRU vethec2c 1500 0 0 0 0 4785 0 0 0 BMRU
March 30, 20188 yr 1 minute ago, Squid said: Does this also hang the terminal? df /var/lib/docker No, this works: root@Tower:~# df /var/lib/docker Filesystem 1K-blocks Used Available Use% Mounted on /dev/loop3 20971520 6844000 12667136 36% /var/lib/docker
March 30, 20188 yr 1 minute ago, Squid said: it'll be loop3 Yea it is and I've posted the results right before bonienl
Archived
This topic is now archived and is closed to further replies.