Jason Schell

I am monitoring my Unraid server WebGUI with Uptime Kuma. Every minute, Uptime Kuma sends a HTTP request to the URL over the Cloudflare Tunnel. So it's browser independent!

I use cloudflare-tunnel in Docker, to access the GUI when I am not at home. Cloudflare Tunnel seems to keep connections alive every minute and that's why there where so many log entries. But nevertheless, Cloudflare Tunnel has a valid usecase - so how should I implement CF Tunnel in a correct way that such entries don't occur anymore?

I think it's not normal, as it shows up every minute (see syslog). violet-syslog-20230909-1827.zip

I made the update to 6.12.4 and this error did not appear again. But the server is still hanging up completely with 100% CPU and not responding to anything anymore, until I restart. This happens once a day.

Syslog and diagnostics attached. I rebooted the server now. I can't create complete diagnostics as the diagnostics process hangs up during log creation (copied from Webinterface): top -bn1 -o%CPU 2>/dev/null|todos >'/violet-diagnostics-20230829-1230/system/top.txt' tail /boot/bz*.sha256 >> '/violet-diagnostics-20230829-1230/unraid-6.12.3.txt' uptime nproc lscpu 2>/dev/null|todos >'/violet-diagnostics-20230829-1230/system/lscpu.txt' lsscsi -vgl 2>/dev/null|todos >'/violet-diagnostics-20230829-1230/system/lsscsi.txt' lspci -knn 2>/dev/null|todos >'/violet-diagnostics-20230829-1230/system/lspci.txt' lsusb 2>/dev/null|todos >'/violet-diagnostics-20230829-1230/system/lsusb.txt' free -mth 2>/dev/null|todos >'/violet-diagnostics-20230829-1230/system/memory.txt' ps -auxf --sort=-pcpu 2>/dev/null|todos >'/violet-diagnostics-20230829-1230/system/ps.txt' lsof -Pni 2>/dev/null|todos >'/violet-diagnostics-20230829-1230/system/lsof.txt' lsmod|sort 2>/dev/null|todos >'/violet-diagnostics-20230829-1230/system/lsmod.txt' df -h 2>/dev/null|todos >'/violet-diagnostics-20230829-1230/system/df.txt' ifconfig -a -s 2>/dev/null|grep -Po '^(eth|bond)[0-9]+' dmidecode -qt2|awk -F: '/^ Manufacturer:/{m=$2};/^ Product Name:/{p=$2} END{print m" -"p}' 2>/dev/null|todos >'/violet-diagnostics-20230829-1230/system/motherboard.txt' dmidecode -qt0 2>/dev/null|todos >>'/violet-diagnostics-20230829-1230/system/motherboard.txt' cat /proc/meminfo 2>/dev/null|todos >'/violet-diagnostics-20230829-1230/system/meminfo.txt' dmidecode --type 17 2>/dev/null|todos >>'/violet-diagnostics-20230829-1230/system/meminfo.txt' ethtool 'bond0' 2>/dev/null|todos >>'/violet-diagnostics-20230829-1230/system/ethtool.txt' ethtool -i 'bond0' 2>/dev/null|todos >>'/violet-diagnostics-20230829-1230/system/ethtool.txt' ethtool 'eth0' 2>/dev/null|todos >>'/violet-diagnostics-20230829-1230/system/ethtool.txt' ethtool -i 'eth0' 2>/dev/null|todos >>'/violet-diagnostics-20230829-1230/system/ethtool.txt' ifconfig -a 2>/dev/null|todos >'/violet-diagnostics-20230829-1230/system/ifconfig.txt' find /sys/kernel/iommu_groups/ -type l 2>/dev/null|sort -V|todos >'/violet-diagnostics-20230829-1230/system/iommu_groups.txt' todos '/violet-diagnostics-20230829-1230/system/cmdline.txt' echo -ne ' /boot ' >>'/violet-diagnostics-20230829-1230/system/folders.txt';ls -l '/boot'|todos >>'/violet-diagnostics-20230829-1230/system/folders.txt' echo -ne ' /boot/config ' >>'/violet-diagnostics-20230829-1230/system/folders.txt';ls -l '/boot/config'|todos >>'/violet-diagnostics-20230829-1230/system/folders.txt' echo -ne ' /boot/config/plugins ' >>'/violet-diagnostics-20230829-1230/system/folders.txt';ls -l '/boot/config/plugins'|todos >>'/violet-diagnostics-20230829-1230/system/folders.txt' echo -ne ' /boot/syslinux ' >>'/violet-diagnostics-20230829-1230/system/folders.txt';ls -l '/boot/syslinux'|todos >>'/violet-diagnostics-20230829-1230/system/folders.txt' echo -ne ' /var/log ' >>'/violet-diagnostics-20230829-1230/system/folders.txt';ls -l '/var/log'|todos >>'/violet-diagnostics-20230829-1230/system/folders.txt' echo -ne ' /var/log/plugins ' >>'/violet-diagnostics-20230829-1230/system/folders.txt';ls -l '/var/log/plugins'|todos >>'/violet-diagnostics-20230829-1230/system/folders.txt' echo -ne ' /boot/extra folder does not exist ' >>'/violet-diagnostics-20230829-1230/system/folders.txt' echo -ne ' /var/log/packages ' >>'/violet-diagnostics-20230829-1230/system/folders.txt';ls -l '/var/log/packages'|todos >>'/violet-diagnostics-20230829-1230/system/folders.txt' echo -ne ' /var/lib/pkgtools/packages ' >>'/violet-diagnostics-20230829-1230/system/folders.txt';ls -l '/var/lib/pkgtools/packages'|todos >>'/violet-diagnostics-20230829-1230/system/folders.txt' echo -ne ' /tmp ' >>'/violet-diagnostics-20230829-1230/system/folders.txt';ls -l '/tmp'|todos >>'/violet-diagnostics-20230829-1230/system/folders.txt' cp /boot/config/*.{cfg,conf,dat} '/violet-diagnostics-20230829-1230/config' 2>/dev/null cp /boot/config/go '/violet-diagnostics-20230829-1230/config/go.txt' 2>/dev/null sed -i -e '/password/c ***line removed***' -e '/user/c ***line removed***' -e '/pass/c ***line removed***' '/violet-diagnostics-20230829-1230/config/go.txt' sed -ri 's/^((disk|flash)(Read|Write)List.*=")[^"]+/\1.../' '/violet-diagnostics-20230829-1230/config/*.cfg' 2>/dev/null /usr/local/emhttp/webGui/scripts/show_interfaces ip|tr -d ' '|tr '#' ' '|tr ',' ' ' >'/violet-diagnostics-20230829-1230/config/listen.txt' /usr/local/emhttp/webGui/scripts/error_interfaces|sed 's///' >>'/violet-diagnostics-20230829-1230/config/listen.txt' sed -ri 's/^(share(Comment|ReadList|WriteList)=")[^"]+/\1.../' '/violet-diagnostics-20230829-1230/shares/appdata.cfg' 2>/dev/null sed -ri 's/^(share(Comment|ReadList|WriteList)=")[^"]+/\1.../' '/violet-diagnostics-20230829-1230/shares/D---------------x.cfg' 2>/dev/null sed -ri 's/^(share(Comment|ReadList|WriteList)=")[^"]+/\1.../' '/violet-diagnostics-20230829-1230/shares/D---------------n.cfg' 2>/dev/null sed -ri 's/^(share(Comment|ReadList|WriteList)=")[^"]+/\1.../' '/violet-diagnostics-20230829-1230/shares/domains.cfg' 2>/dev/null sed -ri 's/^(share(Comment|ReadList|WriteList)=")[^"]+/\1.../' '/violet-diagnostics-20230829-1230/shares/isos.cfg' 2>/dev/null sed -ri 's/^(share(Comment|ReadList|WriteList)=")[^"]+/\1.../' '/violet-diagnostics-20230829-1230/shares/M---a.cfg' 2>/dev/null sed -ri 's/^(share(Comment|ReadList|WriteList)=")[^"]+/\1.../' '/violet-diagnostics-20230829-1230/shares/system.cfg' 2>/dev/null sed -ri 's/^(share(Comment|ReadList|WriteList)=")[^"]+/\1.../' '/violet-diagnostics-20230829-1230/shares/t---------e.cfg' 2>/dev/null todos '/violet-diagnostics-20230829-1230/logs/docker.txt' ls -l /dev/disk/by-id/[asun]* 2>/dev/null|sed '/-part/d;s|^.*/by-id/[^-]*-||;s|-> ../../||;s|:|-|' smartctl -x '/dev/sdf' 2>/dev/null|todos >'/violet-diagnostics-20230829-1230/smart/ADATA_SP900_7E5020002361-20230829-1230 cache (sdf).txt' smartctl -x '/dev/sde' 2>/dev/null|todos >'/violet-diagnostics-20230829-1230/smart/ST1000DM003-1ER162_Z4Y5664R-20230829-1230 parity (sde).txt' smartctl -x '/dev/sdg' 2>/dev/null|todos >'/violet-diagnostics-20230829-1230/smart/ST9750422AS_5WS3KZV9-20230829-1230 disk3 (sdg).txt' smartctl -x '/dev/sdb' 2>/dev/null|todos >'/violet-diagnostics-20230829-1230/smart/Samsung_Portable_SSD_T5_S49XNV0N100714K-20230829-1230 disk2 (sdb).txt' smartctl -x '/dev/sdd' 2>/dev/null|todos >'/violet-diagnostics-20230829-1230/smart/Generic-_Multi-Card_20071114173400000-0-0-20230829-1230 (sdd).txt' smartctl -x '/dev/sdc' 2>/dev/null|todos >'/violet-diagnostics-20230829-1230/smart/SanDisk_Extreme_SSD_323030323744343030363835-0-0-20230829-1230 disk1 (sdc).txt' smartctl -x '/dev/sda' 2>/dev/null|todos >'/violet-diagnostics-20230829-1230/smart/SanDisk_Ultra_Fit_4C531001611217111012-0-0-20230829-1230 flash (sda).txt' echo 'Pool: cache'|todos >>'/violet-diagnostics-20230829-1230/system/btrfs-usage.txt' /sbin/btrfs filesystem usage -T /mnt/cache 2>/dev/null|todos >>'/violet-diagnostics-20230829-1230/system/btrfs-usage.txt' /sbin/btrfs filesystem show /dev/sdfp1 2>/dev/null|todos >>'/violet-diagnostics-20230829-1230/system/btrfs-usage.txt' /usr/sbin/zpool status 2>/dev/null|todos >>'/violet-diagnostics-20230829-1230/system/zfs-info.txt' violet-diagnostics-20230828-0622.zip violet-diagnostics-20230829-1230.zip syslog

I have problems on my Unraid 6.12.3 server. A few hours or days after a fresh restart, nginx hangs and the web interface can no longer be accessed. However, SSH works. I entered nginx via SSH and received the following output: root@Violet:~# nginx nginx: [emerg] bind() to unix:/var/run/nginx.socket failed (98: Address already in use) nginx: [emerg] bind() to 127.0.0.1:80 failed (98: Address already in use) nginx: [emerg] bind() to 192.168.68.57:80 failed (98: Address already in use) nginx: [emerg] bind() to 127.0.0.1:443 failed (98: Address already in use) nginx: [emerg] bind() to [::1]:80 failed (98: Address already in use) nginx: [emerg] bind() to [2a01:598:c834:6377:baae:edff:fe3b:efe3]:80 failed (99: Cannot assign requested address) Message from syslogd@Violet at Aug 28 10:34:42 ... nginx: 2023/08/28 10:34:42 [emerg] 11962#11962: bind() to unix:/var/run/nginx.socket failed (98: Address already in use) Message from syslogd@Violet at Aug 28 10:34:42 ... nginx: 2023/08/28 10:34:42 [emerg] 11962#11962: bind() to 127.0.0.1:80 failed (98: Address already in use) Message from syslogd@Violet at Aug 28 10:34:42 ... nginx: 2023/08/28 10:34:42 [emerg] 11962#11962: bind() to 192.168.68.57:80 failed (98: Address already in use) Message from syslogd@Violet at Aug 28 10:34:42 ... nginx: 2023/08/28 10:34:42 [emerg] 11962#11962: bind() to 127.0.0.1:443 failed (98: Address already in use) Message from syslogd@Violet at Aug 28 10:34:42 ... nginx: 2023/08/28 10:34:42 [emerg] 11962#11962: bind() to [::1]:80 failed (98: Address already in use) Message from syslogd@Violet at Aug 28 10:34:42 ... nginx: 2023/08/28 10:34:42 [emerg] 11962#11962: bind() to [2a01:598:c834:6377:baae:edff:fe3b:efe3]:80 failed (99: Cannot assign requested address) I checked with netstat which processes use those ports already, but came out it's just nginx itself: root@Violet:~# netstat -tulpen | grep :80 tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 0 29063 9008/docker-proxy tcp 5 0 127.0.0.1:80 0.0.0.0:* LISTEN 0 15172 3285/nginx: master tcp 16 0 192.168.68.57:80 0.0.0.0:* LISTEN 0 15173 3285/nginx: master tcp6 0 0 :::8000 :::* LISTEN 0 51604 15985/portainer tcp6 0 0 :::8080 :::* LISTEN 0 29066 9016/docker-proxy tcp6 0 0 2a01:598:c834:6377:b:80 :::* LISTEN 0 15176 3285/nginx: master tcp6 0 0 ::1:80 :::* LISTEN 0 15175 3285/nginx: master root@Violet:~# netstat -tulpen | grep :443 tcp 0 0 127.0.0.1:443 0.0.0.0:* LISTEN 0 15174 3285/nginx: master tcp6 0 0 ::1:443 :::* LISTEN 0 15177 3285/nginx: master I tried to stop all nginx-processes: /etc/rc.d/rc.nginx stop Shutdown Nginx gracefully... Stopping 1 unraid-api process(es)... And kill the 3285 process, but it still remains in netstat. /etc/rc.d/rc.nginx restart does not help. Here is my nginx servers.conf: root@Violet:~# cat /etc/nginx/conf.d/servers.conf # # Listen on local socket for nchan publishers # server { listen unix:/var/run/nginx.socket default_server; location ~ /pub/(.*)$ { nchan_publisher; nchan_channel_id "$1"; nchan_message_buffer_length $arg_buffer_length; } location ~ /nchan_stub_status$ { nchan_stub_status; } } # # Always accept http requests from localhost # ex: http://localhost # ex: http://127.0.0.1 # ex: http://[::1] # server { listen 127.0.0.1:80; # lo listen 127.0.0.1:443; # lo listen [::1]:80; # lo listen [::1]:443; # lo # include /etc/nginx/conf.d/locations.conf; } # # Port settings for http protocol # ex: http://tower (IP address resolved via NetBIOS) # ex: http://tower.local (IP address resolved via mDNS) # ex: http://192.168.1.100 # ex: http://[::ffff:192.168.1.100] # server { listen 192.168.68.57:80 default_server; # br0 listen [2a01:598:c834:6377:baae:edff:fe3b:efe3]:80 default_server; # br0 # location ~ /wsproxy/80/ { return 403; } include /etc/nginx/conf.d/locations.conf; } I am using Docker, but containers have their own IP (br0) or they use only ports other than 80 or 443 (bridge). root@Violet:~# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 56702e2e2033 plexinc/pms-docker "/init" 26 hours ago Up 26 hours (healthy) Plex-Media-Server 0c4b5b08b81e portainer/portainer-ce "/portainer" 26 hours ago Up 26 hours Portainer-CE 09608a486e3c lscr.io/linuxserver/sonarr:latest "/init" 2 days ago Up 26 hours 0.0.0.0:8989->8989/tcp, :::8989->8989/tcp sonarr bc27da92ef39 lscr.io/linuxserver/jackett:latest "/init" 2 days ago Up 26 hours 0.0.0.0:9117->9117/tcp, :::9117->9117/tcp jackett c50c2b279bf0 homeassistant/home-assistant "/init" 2 days ago Up 26 hours Home-Assistant-Container 30eb5b12dcc3 cloudflare/cloudflared "cloudflared --no-au…" 2 days ago Up 26 hours cloudflared 62841a9d1e99 adguard/adguardhome "/opt/adguardhome/Ad…" 7 days ago Up 26 hours AdGuard-Home 4163a5cbbfc7 mbentley/timemachine:smb "/entrypoint.sh s6-s…" 7 days ago Exited (0) 7 days ago timemachine 322929be7215 lscr.io/linuxserver/qbittorrent:latest "/init" 8 days ago Up 26 hours qbittorrent-vpn-torrent-1 3071b80ccf22 ghcr.io/bubuntux/nordvpn:latest "/init /bin/sh -c 'n…" 7 weeks ago Up 26 hours 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp qbittorrent-vpn-vpn-1 909f98aa5a58 containrrr/watchtower "/watchtower --sched…" 7 weeks ago Exited (1) 26 hours ago watchtower 8d5beae99051 oznu/homebridge:ubuntu "/init" 7 weeks ago Up 26 hours homebridge Furthermore the server hangs completely once or twice a week, the fan runs at full speed, no services like Docker containers are working anymore and only a hard reset at the power button helps. After booting up, everything runs as usual again. But I think this is another problem, which is not necessarily part of the nginx problem. Diagnostics are being created at the moment, I'll upload them together with syslog files afterwards. Can somebody help me here with this problem(s)? Thanks a lot!