Can0n Posted April 23 Share Posted April 23 Weird issue tonight login page was fine once i logged in just had a generic chrome page not available (tried incognito to rule out cookies) tried restarting/reloading, stopping and starting on nginx no change. connect said my server was offline. found a handy command to stop the array since i new my docker container were still running. this saved me from having to do a parity check on reboot. CSRF=$(cat /var/local/emhttp/var.ini | grep -oP 'csrf_token="\K[^"]+') curl -k --data "startState=STARTED&file=&csrf_token=${CSRF}&cmdStop=Stop" http://localhost/update.htm diagnostics attached any insight is greatly appreciated thor-diagnostics-20240422-2016.zip Quote Link to comment
Can0n Posted April 24 Author Share Posted April 24 just happened again this time however it recovered on its own after about 5 min thor-diagnostics-20240424-1214.zip Quote Link to comment
bmartino1 Posted April 25 Share Posted April 25 are unraid shares still accessible? ssh still accessible? website error # ? 500 / 404 ? reviewing current diagnostic. Quote Link to comment
bmartino1 Posted April 25 Share Posted April 25 per top in diagnostic you have a zombie process most likely a bad command that killed something but not its parent.. Still reviewing that latter. 32556 root 20 0 0 0 0 Z 0.0 0.0 0:00.00 sh ^-may have been caused by creating diagnostic... per diagnosis, you have a strange go file that remount the log file. Messing with docker system permissions... ^this could also be doing things it shouldn't... #!/bin/bash ##Enable Multi-Gen LRU for better memory management 6.12.x and higher## echo y > /sys/kernel/mm/lru_gen/enabled # Start the Management Utility /usr/local/sbin/emhttp -p 80 & #Bash Profile #Pulseway setup ###cp /boot/pulseway/config.xml /etc/pulseway/config.xml ###cp /boot/pulseway/pulseway.id /var/pulseway/pulseway.id ###Pulse way commands ###/etc/rc.d/rc.pulseway (stop/start/restart) ###/etc/rc.d/rc.pulseway start #Hardware video Transcode Enable and TMP File System #modprobe i915 ***line removed*** #chmod -R 777 /dev/dri #mkdir /tmp/PlexRamScratch #chmod -R 777 /tmp/PlexRamScratch #mount -t tmpfs -o size=32g tmpfs /tmp/PlexRamScratch mount -o remount,size=512m /var/log/ #Docker Shell cp /boot/custom/docker-shell /usr/local/bin chmod 755 /usr/local/bin/docker-shell #SSH Keys Copy and enable #mkdir /root/.ssh #chmod 700 /root/.ssh #cp /boot/config/ssh/authorized_keys /root/.ssh/ #chmod 600 /root/.ssh/authorized_keys Honestly, alot to go on here. Looks like your using ipvlan to your wireguard tunnel. you have disk write issues vias FCP in syslog... with weird docker bridging with dockers on the vpn. *When running Unifi I highly recommend to use macvlan... so turn off bridging. and have dockers custom network set to macvlan... Apr 24 04:10:00 Thor rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="19275" x-info="https://www.rsyslog.com"] rsyslogd was HUPed ### [PREVIOUS LINE REPEATED 4 TIMES] ### Apr 24 08:30:14 Thor root: Fix Common Problems Version 2024.04.14 Apr 24 08:30:19 Thor root: Fix Common Problems: Other Warning: Docker application unifi-controller has moderator comments listed Apr 24 08:30:23 Thor root: Fix Common Problems: Warning: Write Cache is disabled on disk9 ** Ignored if you have a docker set to host and this docker has a web ui it will override unraids web ui ports.. Honestly, with quite a bit going on it will be hard to pinpoint what causing the problems... looks like the system is getting an static ip: Then we have connections. Apr 22 20:31:31 Thor avahi-daemon[20172]: Registering new address record for 10.0.0.87 on br0.IPv4. But your system has a bunch of veth taped for ipv6... lo UNKNOWN 127.0.0.1/8 ::1/128 tunl0@NONE DOWN eth0 UP eth1 DOWN br0 UP 10.0.0.87/24 metric 1 vhost1@eth1 LOWERLAYERDOWN wg0 UNKNOWN 10.0.3.1/32 br-7f0a9fe8c228 UP 172.18.0.1/16 fe80::42:3fff:fee6:d8aa/64 br-d773e863805b DOWN 172.31.200.1/24 docker0 DOWN 172.17.0.1/16 veth3d6534d@if13 UP fe80::c49f:bfff:fe17:160e/64 virbr0 DOWN 192.168.122.1/24 veth71a1162@if48 UP fe80::d435:8ff:fe0a:26dc/64 veth5162665@if56 UP fe80::98e8:c2ff:fec8:643a/64 veth6f8d859@if58 UP fe80::d07d:3fff:fe38:afc7/64 vethefb7d55@if60 UP fe80::cc49:4dff:fe81:e468/64 veth5491d77@if62 UP fe80::58bc:efff:fe42:add0/64 vethf386ea9@if64 UP fe80::583b:e2ff:fec2:ecc5/64 veth53f22ab@if66 UP fe80::dc97:ceff:fee4:c25b/64 vethac7393c@if68 UP fe80::1458:5bff:fee5:8e26/64 vethe8f5bfa@if71 UP fe80::bc18:95ff:fe5a:13c7/64 veth95b8b55@if73 UP fe80::d3:f2ff:fe90:de58/64 veth1213c5f@if75 UP fe80::800b:55ff:feed:b1b6/64 veth467cf9c@if77 UP fe80::3cca:12ff:feb6:ff5d/64 veth9b30110@if79 UP fe80::840a:e2ff:fe75:7450/64 vethc107e26@if81 UP fe80::c4ef:31ff:fecc:4ec4/64 veth153f6d8@if83 UP fe80::70a6:1dff:fe16:3f50/64 vethfa5b6aa@if85 UP fe80::5028:8aff:fe5b:f40/64 Made harder as it apears you are trying to omit somethign in the logs... previous line repeated x times is not normal to see in dmesg... Quote Apr 24 04:01:44 Thor kernel: br-7f0a9fe8c228: port 5(veth819e40e) entered disabled state ### [PREVIOUS LINE REPEATED 1 TIMES] ### Apr 24 04:01:44 Thor kernel: device veth819e40e left promiscuous mode Apr 24 04:01:44 Thor kernel: br-7f0a9fe8c228: port 5(veth819e40e) entered disabled state Apr 24 04:01:44 Thor kernel: br-7f0a9fe8c228: port 5(veth95b8b55) entered blocking state Apr 24 04:01:44 Thor kernel: br-7f0a9fe8c228: port 5(veth95b8b55) entered disabled state Apr 24 04:01:44 Thor kernel: device veth95b8b55 entered promiscuous mode Apr 24 04:01:44 Thor kernel: br-7f0a9fe8c228: port 5(veth95b8b55) entered blocking state Apr 24 04:01:44 Thor kernel: br-7f0a9fe8c228: port 5(veth95b8b55) entered forwarding state Apr 24 04:01:44 Thor kernel: eth0: renamed from vethd2c5885 Apr 24 04:01:44 Thor kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth95b8b55: link becomes ready Apr 24 04:01:51 Thor kernel: veth5a90edc: renamed from eth0 Apr 24 04:01:51 Thor kernel: br-7f0a9fe8c228: port 16(veth3808ffb) entered disabled state ### [PREVIOUS LINE REPEATED 1 TIMES] ### Apr 24 04:01:51 Thor kernel: device veth3808ffb left promiscuous mode Apr 24 04:01:51 Thor kernel: br-7f0a9fe8c228: port 16(veth3808ffb) entered disabled state Apr 24 04:01:51 Thor kernel: br-7f0a9fe8c228: port 16(veth1213c5f) entered blocking state Apr 24 04:01:51 Thor kernel: br-7f0a9fe8c228: port 16(veth1213c5f) entered disabled state Apr 24 04:01:51 Thor kernel: device veth1213c5f entered promiscuous mode Apr 24 04:01:51 Thor kernel: br-7f0a9fe8c228: port 16(veth1213c5f) entered blocking state Apr 24 04:01:51 Thor kernel: br-7f0a9fe8c228: port 16(veth1213c5f) entered forwarding state Apr 24 04:01:51 Thor kernel: eth0: renamed from vethe28a2f2 Apr 24 04:01:51 Thor kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth1213c5f: link becomes ready Apr 24 04:01:57 Thor kernel: br-7f0a9fe8c228: port 14(veth1986b35) entered disabled state Apr 24 04:01:57 Thor kernel: vethd990739: renamed from eth0 Apr 24 04:01:57 Thor kernel: br-7f0a9fe8c228: port 14(veth1986b35) entered disabled state Apr 24 04:01:57 Thor kernel: device veth1986b35 left promiscuous mode Apr 24 04:01:57 Thor kernel: br-7f0a9fe8c228: port 14(veth1986b35) entered disabled state Apr 24 04:01:57 Thor kernel: br-7f0a9fe8c228: port 14(veth467cf9c) entered blocking state Apr 24 04:01:57 Thor kernel: br-7f0a9fe8c228: port 14(veth467cf9c) entered disabled state Apr 24 04:01:57 Thor kernel: device veth467cf9c entered promiscuous mode Apr 24 04:01:57 Thor kernel: br-7f0a9fe8c228: port 14(veth467cf9c) entered blocking state Apr 24 04:01:57 Thor kernel: br-7f0a9fe8c228: port 14(veth467cf9c) entered forwarding state Apr 24 04:01:57 Thor kernel: eth0: renamed from veth64b8cc5 Apr 24 04:01:57 Thor kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth467cf9c: link becomes ready Apr 24 04:02:03 Thor kernel: br-7f0a9fe8c228: port 13(veth4d215bf) entered disabled state Apr 24 04:02:03 Thor kernel: veth6780395: renamed from eth0 Apr 24 04:02:03 Thor kernel: br-7f0a9fe8c228: port 13(veth4d215bf) entered disabled state Apr 24 04:02:03 Thor kernel: device veth4d215bf left promiscuous mode Apr 24 04:02:03 Thor kernel: br-7f0a9fe8c228: port 13(veth4d215bf) entered disabled state Apr 24 04:02:03 Thor kernel: br-7f0a9fe8c228: port 13(veth9b30110) entered blocking state Apr 24 04:02:03 Thor kernel: br-7f0a9fe8c228: port 13(veth9b30110) entered disabled state Apr 24 04:02:03 Thor kernel: device veth9b30110 entered promiscuous mode Apr 24 04:02:03 Thor kernel: br-7f0a9fe8c228: port 13(veth9b30110) entered blocking state Apr 24 04:02:03 Thor kernel: br-7f0a9fe8c228: port 13(veth9b30110) entered forwarding state Apr 24 04:02:03 Thor kernel: eth0: renamed from veth336c65a Apr 24 04:02:03 Thor kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth9b30110: link becomes ready Apr 24 04:02:08 Thor kernel: veth5effc31: renamed from eth0 Apr 24 04:02:08 Thor kernel: br-7f0a9fe8c228: port 10(veth078aaac) entered disabled state ### [PREVIOUS LINE REPEATED 1 TIMES] ### Apr 24 04:02:08 Thor kernel: device veth078aaac left promiscuous mode Apr 24 04:02:08 Thor kernel: br-7f0a9fe8c228: port 10(veth078aaac) entered disabled state Apr 24 04:02:08 Thor kernel: br-7f0a9fe8c228: port 10(vethc107e26) entered blocking state Apr 24 04:02:08 Thor kernel: br-7f0a9fe8c228: port 10(vethc107e26) entered disabled state Apr 24 04:02:08 Thor kernel: device vethc107e26 entered promiscuous mode Apr 24 04:02:08 Thor kernel: br-7f0a9fe8c228: port 10(vethc107e26) entered blocking state Apr 24 04:02:08 Thor kernel: br-7f0a9fe8c228: port 10(vethc107e26) entered forwarding state Apr 24 04:02:09 Thor kernel: eth0: renamed from vethe45b33a Apr 24 04:02:09 Thor kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethc107e26: link becomes ready Apr 24 04:02:14 Thor kernel: br-7f0a9fe8c228: port 2(veth0c9c4d9) entered disabled state Apr 24 04:02:14 Thor kernel: veth83bb629: renamed from eth0 Apr 24 04:02:14 Thor kernel: br-7f0a9fe8c228: port 2(veth0c9c4d9) entered disabled state Apr 24 04:02:14 Thor kernel: device veth0c9c4d9 left promiscuous mode Apr 24 04:02:14 Thor kernel: br-7f0a9fe8c228: port 2(veth0c9c4d9) entered disabled state Apr 24 04:02:14 Thor kernel: br-7f0a9fe8c228: port 2(veth153f6d8) entered blocking state Apr 24 04:02:14 Thor kernel: br-7f0a9fe8c228: port 2(veth153f6d8) entered disabled state Apr 24 04:02:14 Thor kernel: device veth153f6d8 entered promiscuous mode Apr 24 04:02:14 Thor kernel: br-7f0a9fe8c228: port 2(veth153f6d8) entered blocking state Apr 24 04:02:14 Thor kernel: br-7f0a9fe8c228: port 2(veth153f6d8) entered forwarding state Apr 24 04:02:15 Thor kernel: eth0: renamed from veth4ade605 Apr 24 04:02:15 Thor kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth153f6d8: link becomes ready Apr 24 04:02:18 Thor kernel: br-7f0a9fe8c228: port 6(vethf65cb8e) entered disabled state Apr 24 04:02:18 Thor kernel: veth45c4c69: renamed from eth0 Apr 24 04:02:18 Thor kernel: br-7f0a9fe8c228: port 6(vethf65cb8e) entered disabled state Apr 24 04:02:18 Thor kernel: device vethf65cb8e left promiscuous mode Apr 24 04:02:18 Thor kernel: br-7f0a9fe8c228: port 6(vethf65cb8e) entered disabled state Apr 24 04:02:18 Thor kernel: br-7f0a9fe8c228: port 6(vethfa5b6aa) entered blocking state Apr 24 04:02:18 Thor kernel: br-7f0a9fe8c228: port 6(vethfa5b6aa) entered disabled state Apr 24 04:02:18 Thor kernel: device vethfa5b6aa entered promiscuous mode Apr 24 04:02:18 Thor kernel: br-7f0a9fe8c228: port 6(vethfa5b6aa) entered blocking state Apr 24 04:02:18 Thor kernel: br-7f0a9fe8c228: port 6(vethfa5b6aa) entered forwarding state Apr 24 04:02:18 Thor kernel: eth0: renamed from veth5b75e19 So my bet is a misconfigured bind that is caused as it appears the eth1 is the primary adapter due to how vhost taped it and that eth1 is down so is the bind to the web server... -Not sure where nginx web server configs is located, as this may need to be edited to change adapters and restarted to regain web UI. ^BUT! you have a lot going on without any other info so this is a guess... It appears to be an intermittent issue as well... Not sure where to start for you... Quote Link to comment
Can0n Posted April 27 Author Share Posted April 27 That's all very odd...i am aware of the Disk 9 Write caching and I am adding and replacing drives very frequently lately the go file has been like that for years without issues. almost all of it is commented out the docker-shell is just a small script that lets me via via CLI press a number to access any running docker containers cli it was introduced by spaceInvader1 some years ago. the log file was set set larger to accommodate some issues i had a long time ago. I do not have IPv6 enabled on my network or unRAID so that's very odd. I am using IPvlan as MacVLAN had issues a version or two back. eth0 is my 10gbps NIC and eth1 is the on board 1gbps NIC i would think nothing is wrong there since its been working for just over a year in this configuration. either way its something that recently happening and hasn't since the last recovery....very odd 1 Quote Link to comment
bmartino1 Posted Sunday at 12:51 AM Share Posted Sunday at 12:51 AM you may need to ssh / termainl in and kill the nginx service: So lets kill it via terminal command: lsof -i :80 lsof -i :443 example of my output of above command.: root@BMM-Unraid:~# lsof -i :80 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME nginx 12275 root 8u IPv4 14264 0t0 TCP localhost:http (LISTEN) nginx 12275 root 9u IPv4 14265 0t0 TCP BMM-Unraid:http (LISTEN) Then kill all and any apps in the list to do this look at PID # Command: I forget the command... https://docs.oracle.com/cd/E19253-01/817-0403/eoizf/index.html This should work example: kill -9 12275 or kill -SIGKILL 12275 you can also use htop in terminal.... Then try to start nginx: /etc/rc.d/rc.nginx start /etc/rc.d/rc.nginx restart Quote Link to comment
Can0n Posted Sunday at 05:05 PM Author Share Posted Sunday at 05:05 PM 16 hours ago, bmartino1 said: you may need to ssh / termainl in and kill the nginx service: So lets kill it via terminal command: lsof -i :80 lsof -i :443 example of my output of above command.: root@BMM-Unraid:~# lsof -i :80 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME nginx 12275 root 8u IPv4 14264 0t0 TCP localhost:http (LISTEN) nginx 12275 root 9u IPv4 14265 0t0 TCP BMM-Unraid:http (LISTEN) Then kill all and any apps in the list to do this look at PID # Command: I forget the command... https://docs.oracle.com/cd/E19253-01/817-0403/eoizf/index.html This should work example: kill -9 12275 or kill -SIGKILL 12275 you can also use htop in terminal.... Then try to start nginx: /etc/rc.d/rc.nginx start /etc/rc.d/rc.nginx restart thank you never thought to kill the process first i did run: /etc/rc.d/rc.nginx stop got the successly stopped message then ran: /etc/rc.d/rc.nginx start it started fine. its only happened twice so far so really not sure what caused it and why first time required a full reboot to get it back while second time it self recovered in 5min I have set port down on the on board ethernet eth1 and that should hopefully prevent those spamming messages.. i do run syslog too so could look at those logs when time permits thank you for the suggestions i will try and see if they work if i see this happen again Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.