Problem upgrading to 6.8.0

[wackydog]

Hi all,

 

I just updated to the latest stable version and I can't connect to the web interface.  I can SSH into the machine and my Plex docker seems to be running fine.  

 

Following are the last (sanitized) entries in the syslog if that is helpful.

 

Thanks for any help or ideas!

 

 

Dec 16 17:36:52 myserv ntpd[1745]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
Dec 16 17:37:01 myserv crond[1765]: exit status 127 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null
Dec 16 17:38:01 myserv crond[1765]: exit status 127 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null
Dec 16 17:39:01 myserv crond[1765]: exit status 127 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null
Dec 16 17:39:03 myserv nginx: 2019/12/16 17:39:03 [crit] 5026#5026: *633 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 192.168.1.218, server: , request: "GET /Main HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.1.44"
Dec 16 17:39:03 myserv nginx: 2019/12/16 17:39:03 [error] 5026#5026: *633 auth request unexpected status: 502 while sending to client, client: 192.168.1.218, server: , request: "GET /Main HTTP/1.1", host: "192.168.1.44"
Dec 16 17:39:03 myserv nginx: 2019/12/16 17:39:03 [crit] 5026#5026: *635 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 192.168.1.218, server: , request: "GET /favicon.ico HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.1.44", referrer: "http://192.168.1.44/Main"
Dec 16 17:39:03 myserv nginx: 2019/12/16 17:39:03 [error] 5026#5026: *635 auth request unexpected status: 502 while sending to client, client: 192.168.1.218, server: , request: "GET /favicon.ico HTTP/1.1", host: "192.168.1.44", referrer: "http://192.168.1.44/Main"
Dec 16 17:39:12 myserv nginx: 2019/12/16 17:39:12 [crit] 5026#5026: *647 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 192.168.1.218, server: , request: "GET /Main HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "myserv"
Dec 16 17:39:12 myserv nginx: 2019/12/16 17:39:12 [error] 5026#5026: *647 auth request unexpected status: 502 while sending to client, client: 192.168.1.218, server: , request: "GET /Main HTTP/1.1", host: "myserv"
Dec 16 17:39:12 myserv nginx: 2019/12/16 17:39:12 [crit] 5026#5026: *649 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 192.168.1.218, server: , request: "GET /favicon.ico HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "myserv", referrer: "http://myserv/Main"
Dec 16 17:39:12 myserv nginx: 2019/12/16 17:39:12 [error] 5026#5026: *649 auth request unexpected status: 502 while sending to client, client: 192.168.1.218, server: , request: "GET /favicon.ico HTTP/1.1", host: "myserv", referrer: "http://myserv/Main"
Dec 16 17:39:17 myserv nginx: 2019/12/16 17:39:17 [crit] 5026#5026: *655 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 192.168.1.218, server: , request: "GET /Main HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "myserv"
Dec 16 17:39:17 myserv nginx: 2019/12/16 17:39:17 [error] 5026#5026: *655 auth request unexpected status: 502 while sending to client, client: 192.168.1.218, server: , request: "GET /Main HTTP/1.1", host: "myserv"
Dec 16 17:39:17 myserv nginx: 2019/12/16 17:39:17 [crit] 5026#5026: *657 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 192.168.1.218, server: , request: "GET /favicon.ico HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "myserv", referrer: "http://myserv/Main"
Dec 16 17:39:17 myserv nginx: 2019/12/16 17:39:17 [error] 5026#5026: *657 auth request unexpected status: 502 while sending to client, client: 192.168.1.218, server: , request: "GET /favicon.ico HTTP/1.1", host: "myserv", referrer: "http://myserv/Main"
Dec 16 17:39:22 myserv nginx: 2019/12/16 17:39:22 [crit] 5026#5026: *664 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 192.168.1.218, server: , request: "GET /favicon.ico HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.1.44", referrer: "http://192.168.1.44/Main"
Dec 16 17:39:22 myserv nginx: 2019/12/16 17:39:22 [error] 5026#5026: *664 auth request unexpected status: 502 while sending to client, client: 192.168.1.218, server: , request: "GET /favicon.ico HTTP/1.1", host: "192.168.1.44", referrer: "http://192.168.1.44/Main"
Dec 16 17:40:01 myserv crond[1765]: exit status 127 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null
Dec 16 17:41:01 myserv crond[1765]: exit status 127 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null
Dec 16 17:42:01 myserv crond[1765]: exit status 127 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null
Dec 16 17:42:39 myserv nginx: 2019/12/16 17:42:39 [crit] 5026#5026: *881 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 192.168.1.218, server: , request: "GET /Docker HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "myserv"
Dec 16 17:42:39 myserv nginx: 2019/12/16 17:42:39 [error] 5026#5026: *881 auth request unexpected status: 502 while sending to client, client: 192.168.1.218, server: , request: "GET /Docker HTTP/1.1", host: "myserv"
Dec 16 17:42:39 myserv nginx: 2019/12/16 17:42:39 [crit] 5026#5026: *883 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 192.168.1.218, server: , request: "GET /favicon.ico HTTP/1.1", subrequest: "/auth_request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "myserv", referrer: "http://myserv/Docker"
Dec 16 17:42:39 myserv nginx: 2019/12/16 17:42:39 [error] 5026#5026: *883 auth request unexpected status: 502 while sending to client, client: 192.168.1.218, server: , request: "GET /favicon.ico HTTP/1.1", host: "myserv", referrer: "http://myserv/Docker"
Dec 16 17:43:01 myserv crond[1765]: exit status 127 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null
Dec 16 17:44:01 myserv crond[1765]: exit status 127 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null
Dec 16 17:45:01 myserv crond[1765]: exit status 127 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null
Dec 16 17:46:01 myserv crond[1765]: exit status 127 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null

[wackydog]

So, looking a few of the other posts, I tried to use the 'diagnostic' command in the console, but I got:

 

/usr/bin/php: error while loading shared libraries: libicui18n.so.64: cannot open shared object file: No such file or directory

 

[itimpi]

Have you tried booting in Safe Mode?    That error makes me wonder if you are running a plugin that is incompatible with the 6.8.0 release and running in Safe Mode will stop any plugins from being loaded.

[wackydog]

Yes, it looks like that might has something to do with it.  In safe mode I can connect to the GUI.  I looked under 'plugins' and it showed no plugins installed and three plugins on the error tab.  I deleted them in the GUI and restarted no longer in safe mode, but I still can't load the GUI.  Could there be other plugins not shown by the GUI that I could remove from the command line console?

[wackydog]

I see several plugins in '/boot/config/plugins'  Is this the directory to remove?  Is it safe just to delete the contents and reboot?

[itimpi]

10 minutes ago, wackydog said:

Could there be other plugins not shown by the GUI that I could remove from the command line console?

There should not be.   You could check by going into the config/plugins folder on the flash drive and seeing if there are any .plg type files as these are what triggers a plugin to be installed.  

 

Do you by any chance have an 'extras' folder on the flash drive?   If you do any packages in there could be getting installed and they might be ones that interfere with standard operation.

[wackydog]

The directory /boot/extra is empty, but as I said, there are many plugins in '/boot/config/plugins'.  Can I just empty this directory?

 

 

[itimpi]

14 minutes ago, wackydog said:

The directory /boot/extra is empty, but as I said, there are many plugins in '/boot/config/plugins'.  Can I just empty this directory?

There are a few folders in their you do not want to lose as they are built into Unraid as standard so I an not sure what the effect would be if you emptied the folder.  You could simply rename the .plg files to have a different extension and reboot.    That way you have a record of what plugins you had installed.

 

Giving a list of those plugins might help someone point out any that might be troublesome.

 

[wackydog]

I removed the folder and rebooted, and that worked, but I did lose my GUI dark mode.  I moved all the files in /boot/config/plugins to another directory, so I'll move the files back and reset it as you suggested.  Time for me to head to bed for now though.  My hardware takes about 10 minutes to reboot each time, so I'll have to wait until tomorrow to play with it again.  

 

Thanks very much for your help! 

 

I'll report my findings hopefully tomorrow once I've narrowed down the exact problem.  Goodnight. 

[itimpi]

15 minutes ago, wackydog said:

I removed the folder and rebooted, and that worked, but I did lose my GUI dark mode.  I moved all the files in /boot/config/plugins to another directory, so I'll move the files back and reset it as you suggested.  Time for me to head to bed for now though.  My hardware takes about 10 minutes to reboot each time, so I'll have to wait until tomorrow to play with it again.  

 

Thanks very much for your help! 

 

I'll report my findings hopefully tomorrow once I've narrowed down the exact problem.  Goodnight. 

You should be able to move back all the folders from within the config/plugins folder which should get back the standard built-in plugins.   It is the .plg files that are at that level that correspond to user installed plugins.

[wackydog]

Hi all,

 

I set all the original files back into directory '/boot/config/plugins'.  Then I removed all .plg files, except for community.applications.plg, and restarted.  Everything is working fine.  

 

That means the problem is likely with one of the following plugins which I removed:

 

fix.common.problems.plg
recycle.bin.plg 
NerdPack.plg
dynamix.ssd.trim.plg
preclear.disk.plg
tips.and.tweaks.plg

 

I don't have too many interesting settings in these plugins, so I think I'll try to re-install them and see if they work with fresh configurations.

 

Edited December 17, 2019 by wackydog

[wackydog]

Hi Maarty,

 

If you only have ssh access, you can set unraid to boot to GUI safe mode.  Edit the file /boot/syslinux/syslinux.cfg file.  Just move the line 'menu default' to beneath the line 'label unRAID OS GUI Safe Mode (no plugins)'.  Then save and reboot.

 

Just remember to set it back when you're finished.

 

label unRAID OS
  menu default
  kernel /bzimage
  append initrd=/bzroot
label unRAID OS GUI Mode
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label unRAID OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest
 

[itimpi]

59 minutes ago, wackydog said:

Hi all,

 

I set all the original files back into directory '/boot/config/plugins'.  Then I removed all .plg files, except for community.applications.plg, and restarted.  Everything is working fine.  

 

That means the problem is likely with one of the following plugins which I removed:

 

fix.common.problems.plg
recycle.bin.plg 
NerdPack.plg
dynamix.ssd.trim.plg
preclear.disk.plg
tips.and.tweaks.plg

 

I don't have too many interesting settings in these plugins, so I think I'll try to re-install them and see if they work with fresh configurations.

 

I would think that the most likely one could (indirectly) end up being NerdPack.     It would not be the plugin itself that is the problem but one of the packages you have selected for it to install from the many that are available.    If you put that plugin back I would recommend keeping careful track of which packages you have told it to install so we can identify which ones might need updating for 6.8.0 compatibility.

[wackydog]

It looks like that was it, though I haven't tested the other plugins.  I tried to just re-add the NerdPack.plg file with the existing plugin folder and the GUI would not start.  I then deleted the NerdPack.plg and its corresponding folder, and then after reboot re-added the plugin from community apps and it worked fine.  I'm not actively dependent on any of the utils, so I'll just add things as I need them.

 

The only thing that doesn't work now is my authorized_keys file.  Years ago I wrote a custom 'startup.sh' file that copied my authorized_keys from the boot drive to '/root/.ssh'.  This startup file no longer runs at startup, so I can't ssh into the machine using my keys.

 

Is there a standard way for setting up authorized ssh keys in Unraid 6.8?

Edited December 17, 2019 by wackydog

[itimpi]

1 hour ago, wackydog said:

The only thing that doesn't work now is my authorized_keys file.  Years ago I wrote a custom 'startup.sh' file that copied my authorized_keys from the boot drive to '/root/.ssh'.  This startup file no longer runs at startup, so I can't ssh into the machine using my keys.

How are you running this script?   The tightening of the security on the flash drive means files located there can no longer have execute permission.  the simplest way to handle this is to use:

bash scriptname

rather than simply the script name

[wackydog]

I guess my problem is that I don't recall what the mechanism was that made it automatically run at startup in the first place. 

 

I just need a way for my authorized_keys file to survive reboots.  The file is on the thumb drive, I just need to make sure it's always in the .ssh folder of the running system.

 

[trurl]

4 minutes ago, wackydog said:

I guess my problem is that I don't recall what the mechanism was that made it automatically run at startup in the first place. 

Likely the go file in the config folder on flash. Either that or the User Scripts plugin. Those are about the only builtin ways to get something to run at boot.

[wackydog]

Thanks!  I'm off to bed, but I'll try to use the Users Script plugin first thing tomorrow.  I'll let you know if it works.

[trurl]

38 minutes ago, wackydog said:

Thanks!  I'm off to bed, but I'll try to use the Users Script plugin first thing tomorrow.  I'll let you know if it works.

If you can post diagnostics we can take a look at that go file I mentioned and see if the code is there.

[wackydog]

It works!  I loaded the User Script plugin and set my startup script there and all is working like before.  I did see the call to my startup script was still in the go file, but it wasn't running.  Oh well, the User Script plugin is working fine.  Thanks to itimpi and trurl for all the help.  I really appreciate it.  Also much thanks to the Unraid team for fixing the annoying update problems in Docker.

[trurl]

5 hours ago, wackydog said:

I did see the call to my startup script was still in the go file, but it wasn't running.  Oh well, the User Script plugin is working fine.

Security was updated so nothing on flash has execute permission as itimpi noted and he suggested a workaround.

 

User Scripts is really the preferred method now though so you are good with that. You should remove that from your go file if you haven't already.

[xavierda]

Hi everyone,

 

After upgrading to 6.8.0, I too encountered the: "error while loading shared libraries: libicui18n.so.64" on boot, which prevented the GUI/Web interface loading. I could SSH into my server though.

 

After reading through this post, I can confirm, deleting the NerdPack.plg and associated folder from /config/plugins, rebooting and reinstalling works. I only needed python3 to run a separate script - all working again now.

 

The Update OS assistant didn't pick this one up; not sure where to let whoever updates that know?

 

Cheers,

 

Xav.