web gui fails to load without safe mode

[bpage]

Not sure exactly what is going on. The only thing I did was remove the libvirtwol plugin and after reboot all went to hell.

 

It looks like nginx is not loading upon boot. When I do manually start it, I get tons of errors in /var/log.nginx/error.log. They are below. When running in safe mode, the gui and the array load fine.

 

Any ideas? I am on 6.8.0-rc7.

 

2019/12/09 11:26:45 [crit] 19929#19929: *1 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:26:45 [error] 19929#19929: *1 auth request unexpected status: 502 while sending to client, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:26:45 [crit] 19929#19929: *3 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/dockerload HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:26:45 [error] 19929#19929: *3 auth request unexpected status: 502 while sending to client, client: 10.0.1.35, server: , request: "GET /sub/dockerload HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:26:48 [crit] 19929#19929: *10 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:26:48 [error] 19929#19929: *10 auth request unexpected status: 502 while sending to client, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:26:50 [crit] 19929#19929: *13 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/dockerload HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:26:50 [error] 19929#19929: *13 auth request unexpected status: 502 while sending to client, client: 10.0.1.35, server: , request: "GET /sub/dockerload HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:26:52 [crit] 19929#19929: *17 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:26:52 [error] 19929#19929: *17 auth request unexpected status: 502 while sending to client, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:26:54 [crit] 19929#19929: *21 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.2.2, server: , request: "GET /Main HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:26:54 [error] 19929#19929: *21 auth request unexpected status: 502 while sending to client, client: 10.0.2.2, server: , request: "GET /Main HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:26:54 [crit] 19929#19929: *23 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.2.2, server: , request: "GET /favicon.ico HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10", referrer: "http://10.0.1.10/Main"
2019/12/09 11:26:54 [error] 19929#19929: *23 auth request unexpected status: 502 while sending to client, client: 10.0.2.2, server: , request: "GET /favicon.ico HTTP/1.1", host: "10.0.1.10", referrer: "http://10.0.1.10/Main"
2019/12/09 11:26:56 [crit] 19929#19929: *27 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/dockerload HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:26:56 [error] 19929#19929: *27 auth request unexpected status: 502 while sending to client, client: 10.0.1.35, server: , request: "GET /sub/dockerload HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:26:59 [crit] 19929#19929: *32 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:26:59 [error] 19929#19929: *32 auth request unexpected status: 502 while sending to client, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:27:00 [crit] 19929#19929: *35 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/dockerload HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:27:00 [error] 19929#19929: *35 auth request unexpected status: 502 while sending to client, client: 10.0.1.35, server: , request: "GET /sub/dockerload HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:27:03 [crit] 19929#19929: *41 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:27:03 [error] 19929#19929: *41 auth request unexpected status: 502 while sending to client, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:27:04 [crit] 19929#19929: *43 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/dockerload HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:27:04 [error] 19929#19929: *43 auth request unexpected status: 502 while sending to client, client: 10.0.1.35, server: , request: "GET /sub/dockerload HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:27:07 [crit] 19929#19929: *51 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"

 

[bpage]

Ok. So after manually starting php-fpm via ssh I can get to the webgui, but, the array is stuck on "Array Starting * Mounting Disks"

[bpage]

Find diagnostics attached.

pageraid-diagnostics-20191209-1637.zip

[trurl]

Why are you running the webUI on a non-standard port. Your go file:

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp -p 2785&

cp /boot/custom/docker-shell /usr/local/bin

 

[bpage]

6 minutes ago, trurl said:

Why are you running the webUI on a non-standard port. Your go file:

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp -p 2785&

cp /boot/custom/docker-shell /usr/local/bin

 

I am not sure why that is in go file, as I typically hit the webgui on port 80....

[trurl]

And what about that cp line? What is the purpose of that?

[bpage]

The cp line is to add an easy way to get into the docker shell. That has been working for a few years at this point. I have removed the -p config. I am trying a reboot now. 

 

 

As seen here: 

 

#!/bin/bash
# easily "docker exec" into a running Docker container
# latest version: https://gist.github.com/ljm42/2b3bfd8ff886015bbce8

# for unRAID, place this script on your flash drive as /boot/custom/docker-shell
# then add this to your go script (without the leading pound sign):
# cp /boot/custom/docker-shell /usr/local/bin

CONTAINERS=`docker ps | awk 'NR==1 {offset=index($0,"NAMES")};NR>1{print substr($0,offset)}' | sort -f | tr "\n" " "`

echo
echo "Choose a Docker container:"
i=1
for container in $CONTAINERS
do
 echo "$((i++)) : $container"
done

read MYCHOICE
CHOSEN=`echo $CONTAINERS | cut -d' ' -f$MYCHOICE 2>/dev/null`

if [ "$CHOSEN" = "" ]; then
  echo Invalid option
  exit
fi

# try running bash, if exit code 126 then try running sh
for SHELL in bash sh
do
  clear
  echo " "
  echo -e '\E[30;42m'"\033[5m    $CHOSEN - $SHELL   \033[0m"
  echo  " "
  tput sgr0
  docker exec -it $CHOSEN $SHELL
  if [ $? -ne 126 ]; then
    break
  fi
done

 

[trurl]

1 minute ago, bpage said:

The cp line is to add an easy way to get into the docker shell.

You can easily get a shell inside any of your containers by simply clicking on its icon on the Dashboard or Docker page and selecting >_ Console.

[bpage]

Just now, trurl said:

You can easily get a shell inside any of your containers by simply clicking on its icon on the Dashboard or Docker page and selecting >_ Console.

That's true and I don't really use that shell script anymore, now that I have gotten better using direct docker commands. But again, that had been there for a long time and my server was just fine. I am happy to remove it, but I do not suspect that is causing the issue. 

[bpage]

rebooting after commenting out the -p flag and the cp command resulted in the same issue. NGINX does not start. 

 

Booting into Safe mode still works fine. 

[trurl]

5 minutes ago, bpage said:

Booting into Safe mode still works fine

That normally indicates a plugin issue.

 

Post a new diagnostic.

[bpage]

attached

 

 

pageraid-diagnostics-20191209-1728.zip

[trurl]

Nothing obvious. Delete all .plg files in config/plugins and see if it boots in normal mode with no plugins.

[bpage]

I see lots of folders as well as .plg files. Should I leave the folders and just delete the file? 

 

 

[trurl]

3 minutes ago, bpage said:

I see lots of folders as well as .plg files. Should I leave the folders and just delete the file? 

 

 

Nothing from the folders will be loaded without the corresponding .plg file

[bpage]

3 minutes ago, trurl said:

Nothing from the folders will be loaded without the corresponding .plg file

Ok. Trying now. I should also mentioned, I have a backup of the flash drive from a couple of days ago. I used Duplicacy to backup and just restored to a folder on my desktop. Not sure if I could just replace the contents and reboot.... It seems the kernel is older though. 

 

5.3.12 vs 4.19.56

[trurl]

Not familiar with that method to backup flash. You can just download a zipped copy of flash from Main - Boot Device - Flash - Flash Backup.

 

Different version of kernel would of course be a different version of Unraid.

 

Usually a new install with the config folder copied from backup should get things going as they were, but let's see what happens without plugins.

[bpage]

Ok. So without the plugins, the server boots correctly and the the array starts. 

 

 

 

 

[bpage]

One other thing to note, I did last week install a 10gbe network card and I wonder if that may have been causing this. I say this because when I booted the server after removing the plugins, I did not have any internet access to the outside world. I believe this may have been because the 10gbe network (different subnet) did not have WAN access. 

 

I am not exactly sure how to have the 10gbe network ONLY work for internal storage sharing, but not for unRAID internet access. 

 

I am not exactly sure I am explaining that correctly...

[bpage]

Regardless, I much appreciate the help @trurl ! I was growing more concerned by the minute.