Not sure exactly what is going on. The only thing I did was remove the libvirtwol plugin and after reboot all went to hell.

 

It looks like nginx is not loading upon boot. When I do manually start it, I get tons of errors in /var/log.nginx/error.log. They are below. When running in safe mode, the gui and the array load fine.

 

Any ideas? I am on 6.8.0-rc7.

 

2019/12/09 11:26:45 [crit] 19929#19929: *1 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:26:45 [error] 19929#19929: *1 auth request unexpected status: 502 while sending to client, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:26:45 [crit] 19929#19929: *3 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/dockerload HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:26:45 [error] 19929#19929: *3 auth request unexpected status: 502 while sending to client, client: 10.0.1.35, server: , request: "GET /sub/dockerload HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:26:48 [crit] 19929#19929: *10 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:26:48 [error] 19929#19929: *10 auth request unexpected status: 502 while sending to client, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:26:50 [crit] 19929#19929: *13 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/dockerload HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:26:50 [error] 19929#19929: *13 auth request unexpected status: 502 while sending to client, client: 10.0.1.35, server: , request: "GET /sub/dockerload HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:26:52 [crit] 19929#19929: *17 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:26:52 [error] 19929#19929: *17 auth request unexpected status: 502 while sending to client, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:26:54 [crit] 19929#19929: *21 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.2.2, server: , request: "GET /Main HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:26:54 [error] 19929#19929: *21 auth request unexpected status: 502 while sending to client, client: 10.0.2.2, server: , request: "GET /Main HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:26:54 [crit] 19929#19929: *23 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.2.2, server: , request: "GET /favicon.ico HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10", referrer: "http://10.0.1.10/Main"
2019/12/09 11:26:54 [error] 19929#19929: *23 auth request unexpected status: 502 while sending to client, client: 10.0.2.2, server: , request: "GET /favicon.ico HTTP/1.1", host: "10.0.1.10", referrer: "http://10.0.1.10/Main"
2019/12/09 11:26:56 [crit] 19929#19929: *27 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/dockerload HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:26:56 [error] 19929#19929: *27 auth request unexpected status: 502 while sending to client, client: 10.0.1.35, server: , request: "GET /sub/dockerload HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:26:59 [crit] 19929#19929: *32 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:26:59 [error] 19929#19929: *32 auth request unexpected status: 502 while sending to client, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:27:00 [crit] 19929#19929: *35 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/dockerload HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:27:00 [error] 19929#19929: *35 auth request unexpected status: 502 while sending to client, client: 10.0.1.35, server: , request: "GET /sub/dockerload HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:27:03 [crit] 19929#19929: *41 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:27:03 [error] 19929#19929: *41 auth request unexpected status: 502 while sending to client, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:27:04 [crit] 19929#19929: *43 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/dockerload HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"
2019/12/09 11:27:04 [error] 19929#19929: *43 auth request unexpected status: 502 while sending to client, client: 10.0.1.35, server: , request: "GET /sub/dockerload HTTP/1.1", host: "10.0.1.10"
2019/12/09 11:27:07 [crit] 19929#19929: *51 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 10.0.1.35, server: , request: "GET /sub/var?last_event_id=1575906911%3A3 HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "10.0.1.10"

 

Ok. So after manually starting php-fpm via ssh I can get to the webgui, but, the array is stuck on "Array Starting * Mounting Disks"

Find diagnostics attached.

pageraid-diagnostics-20191209-1637.zip

Why are you running the webUI on a non-standard port. Your go file:

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp -p 2785&

cp /boot/custom/docker-shell /usr/local/bin

 

6 minutes ago, trurl said:

Why are you running the webUI on a non-standard port. Your go file:

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp -p 2785&

cp /boot/custom/docker-shell /usr/local/bin

 

I am not sure why that is in go file, as I typically hit the webgui on port 80....

And what about that cp line? What is the purpose of that?

The cp line is to add an easy way to get into the docker shell. That has been working for a few years at this point. I have removed the -p config. I am trying a reboot now. 

 

 

As seen here: 

 

#!/bin/bash
# easily "docker exec" into a running Docker container
# latest version: https://gist.github.com/ljm42/2b3bfd8ff886015bbce8

# for unRAID, place this script on your flash drive as /boot/custom/docker-shell
# then add this to your go script (without the leading pound sign):
# cp /boot/custom/docker-shell /usr/local/bin

CONTAINERS=`docker ps | awk 'NR==1 {offset=index($0,"NAMES")};NR>1{print substr($0,offset)}' | sort -f | tr "\n" " "`

echo
echo "Choose a Docker container:"
i=1
for container in $CONTAINERS
do
 echo "$((i++)) : $container"
done

read MYCHOICE
CHOSEN=`echo $CONTAINERS | cut -d' ' -f$MYCHOICE 2>/dev/null`

if [ "$CHOSEN" = "" ]; then
  echo Invalid option
  exit
fi

# try running bash, if exit code 126 then try running sh
for SHELL in bash sh
do
  clear
  echo " "
  echo -e '\E[30;42m'"\033[5m    $CHOSEN - $SHELL   \033[0m"
  echo  " "
  tput sgr0
  docker exec -it $CHOSEN $SHELL
  if [ $? -ne 126 ]; then
    break
  fi
done

 

1 minute ago, bpage said:

The cp line is to add an easy way to get into the docker shell.

You can easily get a shell inside any of your containers by simply clicking on its icon on the Dashboard or Docker page and selecting >_ Console.

Just now, trurl said:

You can easily get a shell inside any of your containers by simply clicking on its icon on the Dashboard or Docker page and selecting >_ Console.

That's true and I don't really use that shell script anymore, now that I have gotten better using direct docker commands. But again, that had been there for a long time and my server was just fine. I am happy to remove it, but I do not suspect that is causing the issue. 

rebooting after commenting out the -p flag and the cp command resulted in the same issue. NGINX does not start. 

 

Booting into Safe mode still works fine. 

5 minutes ago, bpage said:

Booting into Safe mode still works fine

That normally indicates a plugin issue.

 

Post a new diagnostic.

attached

 

 

pageraid-diagnostics-20191209-1728.zip

Nothing obvious. Delete all .plg files in config/plugins and see if it boots in normal mode with no plugins.

I see lots of folders as well as .plg files. Should I leave the folders and just delete the file? 

 

 

3 minutes ago, bpage said:

I see lots of folders as well as .plg files. Should I leave the folders and just delete the file? 

 

 

Nothing from the folders will be loaded without the corresponding .plg file

3 minutes ago, trurl said:

Nothing from the folders will be loaded without the corresponding .plg file

Ok. Trying now. I should also mentioned, I have a backup of the flash drive from a couple of days ago. I used Duplicacy to backup and just restored to a folder on my desktop. Not sure if I could just replace the contents and reboot.... It seems the kernel is older though. 

 

5.3.12 vs 4.19.56

Not familiar with that method to backup flash. You can just download a zipped copy of flash from Main - Boot Device - Flash - Flash Backup.

 

Different version of kernel would of course be a different version of Unraid.

 

Usually a new install with the config folder copied from backup should get things going as they were, but let's see what happens without plugins.

Ok. So without the plugins, the server boots correctly and the the array starts. 

 

 

 

 

One other thing to note, I did last week install a 10gbe network card and I wonder if that may have been causing this. I say this because when I booted the server after removing the plugins, I did not have any internet access to the outside world. I believe this may have been because the 10gbe network (different subnet) did not have WAN access. 

 

I am not exactly sure how to have the 10gbe network ONLY work for internal storage sharing, but not for unRAID internet access. 

 

I am not exactly sure I am explaining that correctly...

Regardless, I much appreciate the help @trurl ! I was growing more concerned by the minute.