February 8, 20251 yr Editing top post to include a TLDR of the findings from below ------------------------------------ It appears the directory and all the files within /usr/local/emhttp/state/ are gone. Notably these two referenced in the phplog errors /usr/local/emhttp/state/network.ini /usr/local/emhttp/state/var.ini This is causing the dynamix plugin to fail and breaking the Web GUI completely. Apparently these files are recreated on a restart so that explains why it comes good after an unraid reboot. However I'm still unsure what is cauusing them to dissapear. Is there a way to regenerate these withouut a reboot? ------------------------------------ Original First Post Below ------------------------------------ I've recently had the Unraid Web GUI stop responding completely with error 500. This has been happening on versions 6.12.13 about a month ago and is till happening on 7.0.0 (upgraded just yesterday) Searching the forums suggested restarting the web interface which never works # restart Web GUI /etc/rc.d/rc.php-fpm restart The only way to get the web GUI back is to restart the whole server which is a bit of a pain since you can't access the Web GUI. Ideally I'd stop all docker containers before restarting but there doesn't seem to be a way to stop all containers from ssh and they have to be done one by one. # List all contatiners docker ps # Stop or Kill containers docker stop container <container number> docker kill container <container number> I've got all my containers and plugins up to date as I just upgraded to 7.0.0 yesterday. My suspicion is the GPU Statistics Plugin as when GUI error 500 was happening a month ago searching the forums I found a post suggesting GPU Statistics Plugin was the issue. I did remove it a while ago and the problems went away. However yesterday I decided to reinstall it after upgrading to 7.0.0 and I'm back at Web GUI error 500 after about 12 hours so it's the prime suspect. root@Tower:~# ls /boot/config/plugins AAA-UnraidPatch-BootLoader-DO_NOT_DELETE.plg dockerMan/ dynamix.system.buttons.plg intel-gpu-top/ unassigned.devices-plus/ NerdPack/ dynamix/ dynamix.system.info/ intel-gpu-top.plg unassigned.devices-plus.plg NerdTools/ dynamix.active.streams/ dynamix.system.info.plg nvidia-driver/ unassigned.devices.plg appdata.backup/ dynamix.active.streams.plg dynamix.system.stats/ nvidia-driver.plg unassigned.devices.preclear/ appdata.backup.plg dynamix.apcupsd/ dynamix.system.stats.plg open.files/ unassigned.devices.preclear.plg ca.mover.tuning/ dynamix.file.integrity/ dynamix.system.temp/ open.files.plg unbalanced/ ca.mover.tuning.plg dynamix.file.integrity.plg dynamix.system.temp.plg rclone/ unbalanced.plg community.applications/ dynamix.my.servers/ dynamix.unraid.net.plg rclone.plg unraid.patch/ community.applications.plg dynamix.s3.sleep/ dynamix.wireguard/ tailscale/ unraid.patch.plg coral-driver/ dynamix.s3.sleep.plg fix.common.problems/ tailscale.plg user.scripts/ coral-driver.plg dynamix.system.autofan/ fix.common.problems.plg tips.and.tweaks/ user.scripts.plg corefreq/ dynamix.system.autofan.plg gpustat/ tips.and.tweaks.plg corefreq.plg dynamix.system.buttons/ gpustat.plg unassigned.devices/ Ideally I'd like to identify what the problem is so the developer of the GPU Statistics Plugin can try to fix. Any ideas where to start and what logs to look at to show what is causing the web GUI to die? I'd post a diagnostic but running diagnostic from the SSH session seems to produce an unredacted diagnostics file. What's the paramter to redact? Attached is my syslog.txt and phplog.txt from the diagnostic phplog.txt syslog.txt Edited February 9, 20251 yr by phoenixdiigital Adding aTLDR of findings.
February 8, 20251 yr Author I just tried removing GPU Stats plugin via the CLI then restarting the web gui but I'm still getting Web GUI Error 500 root@Tower:~# plugin remove gpustat plugin: removing: gpustat plugin: gpustat removed Executing hook script: post_plugin_checks root@Tower:~# /etc/rc.d/rc.php-fpm restart rc.php-fpm: Restarting PHP-fpm daemon... rc.php-fpm: Stopping PHP-fpm daemon... rc.php-fpm: PHP-fpm daemon... Stopped. rc.php-fpm: Starting PHP-fpm daemon... rc.php-fpm: PHP-fpm daemon... Started. root@Tower:~# plugin remove intel-gpu-top plugin: removing: intel-gpu-top plugin: intel-gpu-top removed Executing hook script: post_plugin_checks root@Tower:~# /etc/rc.d/rc.php-fpm restart rc.php-fpm: Restarting PHP-fpm daemon... rc.php-fpm: Stopping PHP-fpm daemon... rc.php-fpm: PHP-fpm daemon... Stopped. rc.php-fpm: Starting PHP-fpm daemon... rc.php-fpm: PHP-fpm daemon... Started. I'd love to avoid having to restart the whole server if at all possible. Interestingly after running the above commands the plugins are still in /boot/config/plugins directory. I guess they'll go after a reboot? Edited February 8, 20251 yr by phoenixdiigital More information
February 8, 20251 yr Author Also I've tried multiple browsers and OS types (windows, Fedora). Deleting the cookies presents the login page and as soon as you login you get the 500 Error.
February 8, 20251 yr Author There are errors in /var/log/phplog like this which match up to when I try to load the web GUI [09-Feb-2025 08:56:41 Australia/Brisbane] PHP Fatal error: Uncaught TypeError: extract(): Argument #1 ($array) must be of type array, false given in /usr/local/emhttp/plugins/dynamix/template.php:50 Stack trace: #0 /usr/local/emhttp/plugins/dynamix/template.php(50): extract(false) #1 {main} thrown in /usr/local/emhttp/plugins/dynamix/template.php on line 50 That error seems to indicate it might not be GPU statistics plugin. Based on that log I see plugins seem to be here too root@Tower:/usr/local/emhttp/plugins# ls /usr/local/emhttp/plugins/ CoreFreq/ dynamix.active.streams/ dynamix.plugin.manager/ dynamix.system.temp/ nvidia-driver/ unassigned.devices/ appdata.backup/ dynamix.apcupsd/ dynamix.s3.sleep/ dynamix.unraid.net/ open.files/ unassigned.devices-plus/ ca.mover.tuning/ dynamix.docker.manager/ dynamix.system.autofan/ dynamix.vm.manager/ rclone/ unassigned.devices.preclear/ community.applications/ dynamix.file.integrity/ dynamix.system.buttons/ fix.common.problems/ tailscale/ unbalanced/ coral-driver/ dynamix.gui.search/ dynamix.system.info/ gpustat/ tips.and.tweaks/ unraid.patch/ dynamix/ dynamix.my.servers/ dynamix.system.stats/ intel-gpu-top/ unRAIDServer/ user.scripts/ I temporarily moved gpustat and retarted web gui but the error still remains. mv /usr/local/emhttp/plugins/gpustat /root/ /etc/rc.d/rc.php-fpm restart I've since moved it back.
February 8, 20251 yr Author Based on the error I edited /usr/local/emhttp/plugins/dynamix/template.php to address the error casting to an array // Read network settings # extract(parse_ini_file('state/network.ini',true)); extract((array)@parse_ini_file('state/network.ini',true)); phplog then had this error [09-Feb-2025 09:19:39 Australia/Brisbane] PHP Fatal error: Uncaught TypeError: date(): Argument #2 ($timestamp) must be of type ?int, string given in /usr/local/emhttp/plugins/dynamix/template.php:55 Stack trace: #0 /usr/local/emhttp/plugins/dynamix/template.php(55): date('Ymd', '') #1 {main} thrown in /usr/local/emhttp/plugins/dynamix/template.php on line 55 so I changed /usr/local/emhttp/plugins/dynamix/template.php to cast as (int) // Language translations $_SESSION['locale'] = _var($display,'locale'); # $_SESSION['buildDate'] = date('Ymd',_var($var,'regBuildTime')); $_SESSION['buildDate'] = date('Ymd',(int)@_var($var,'regBuildTime')); require_once "$docroot/webGui/include/Translations.php"; Reloading the web GUI worked... kinda. It didn't throw the 500 error but the web GUI loaded a bit and sat there constantly refreshing. I reverted all code and am back to where I was with the 500 error
February 9, 20251 yr Author OK so based on this thread all of my /usr/local/emhttp/state/ files are gone which would explain these errors. root@Tower:/usr/local/emhttp# ls -al /usr/local/emhttp total 47 drwxr-xr-x 1 root root 180 Feb 9 06:08 ./ drwxr-xr-x 1 root root 160 Jan 29 19:05 ../ -rw-r--r-- 1 root root 0 Feb 9 10:28 0 -rw-r--r-- 1 root root 1742 Feb 8 08:25 auth-request.php -rw-rw-rw- 1 root root 648 Feb 8 08:25 auth-request.php.patch lrwxrwxrwx 1 root root 5 Nov 16 2023 boot -> /boot/ -rw-r--r-- 1 root root 15086 Nov 16 2023 favicon.ico -rw-r--r-- 1 root root 940 Feb 8 11:10 gpustat.cfg drwxr-xr-x 1 root root 60 Nov 16 2023 languages/ lrwxrwxrwx 1 root root 8 Nov 16 2023 log -> /var/log/ -rw-r--r-- 1 root root 6757 Nov 16 2023 logging.htm -rw-r--r-- 1 root root 1438 Feb 3 2024 login.php lrwxrwxrwx 1 root root 4 Nov 16 2023 mnt -> /mnt/ drwxr-xr-x 1 root root 700 Feb 9 08:56 plugins/ -rw-r--r-- 1 root root 1649 Jan 10 08:26 redirect.htm -rw-r--r-- 1 root root 26 Nov 16 2023 robots.txt -rw-r--r-- 1 root root 3036 Nov 16 2023 update.htm -rw-r--r-- 1 root root 6006 Aug 28 05:19 update.php lrwxrwxrwx 1 root root 15 Nov 16 2023 webGui -> plugins/dynamix/ Who knows where or why they are gone. From what I've been reading a restart will recreate these files. I'll hold off rebooting in case someone has a good suggestion for tracking down what caused the issue. Edited February 9, 20251 yr by phoenixdiigital
February 9, 20251 yr Author I got impatient and rebooted. I also removed GPU Statistics Plugin (just in case) Web GUI is working again and I've got some more insight into that /usr/local/emhttp/state directory which appears to be a symlink root@Tower:~# ls -al /usr/local/emhttp/ total 43 drwxr-xr-x 1 root root 140 Feb 9 15:52 ./ drwxr-xr-x 1 root root 160 Dec 20 00:15 ../ -rw-r--r-- 1 root root 0 Feb 9 15:53 0 -rw-r--r-- 1 root root 1742 Feb 9 15:51 auth-request.php -rw-rw-rw- 1 root root 648 Feb 9 15:51 auth-request.php.patch lrwxrwxrwx 1 root root 5 Nov 16 2023 boot -> /boot/ -rw-r--r-- 1 root root 15086 Nov 16 2023 favicon.ico drwxr-xr-x 1 root root 60 Nov 16 2023 languages/ lrwxrwxrwx 1 root root 8 Nov 16 2023 log -> /var/log/ -rw-r--r-- 1 root root 6757 Nov 16 2023 logging.htm -rw-r--r-- 1 root root 1438 Feb 3 2024 login.php lrwxrwxrwx 1 root root 4 Nov 16 2023 mnt -> /mnt/ drwxr-xr-x 1 root root 700 Dec 20 00:15 plugins/ -rw-r--r-- 1 root root 1649 Jan 10 08:26 redirect.htm -rw-r--r-- 1 root root 26 Nov 16 2023 robots.txt lrwxrwxrwx 1 root root 17 Nov 16 2023 state -> /var/local/emhttp/ -rw-r--r-- 1 root root 3036 Nov 16 2023 update.htm -rw-r--r-- 1 root root 6006 Aug 28 05:19 update.php lrwxrwxrwx 1 root root 15 Nov 16 2023 webGui -> plugins/dynamix/ With the contents root@Tower:~# ls -al /var/local/emhttp/ total 88 drwxr-xr-x 4 root root 400 Feb 9 15:53 ./ drwxr-xr-x 5 root root 100 Feb 9 15:47 ../ -rw-r--r-- 1 root root 660 Feb 9 15:53 cpuload.ini -rw-r--r-- 1 root root 886 Feb 9 15:53 devs.ini -rw-r--r-- 1 root root 263 Feb 9 15:53 diskload.ini -rw-r--r-- 1 root root 12906 Feb 9 15:53 disks.ini -rw-r--r-- 1 root root 85 Feb 9 15:52 flashbackup.ini -rw------- 1 root root 205 Feb 9 15:48 monitor.ini -rw-rw-rw- 1 root root 837 Feb 9 15:51 myservers.cfg -rw-r--r-- 1 root root 477 Feb 9 15:48 network.ini -rw-r--r-- 1 root root 448 Feb 9 15:52 nginx.ini drwxr-xr-x 5 root root 100 Feb 9 15:52 plugins/ -rwxr-xr-x 1 root root 35 Feb 9 15:48 proxy.ini* -rw-r--r-- 1 root root 3648 Feb 9 15:51 sec.ini -rw-r--r-- 1 root root 1799 Feb 9 15:51 sec_nfs.ini -rw-r--r-- 1 root root 3253 Feb 9 15:53 shares.ini drwxr-xr-x 2 root root 520 Feb 9 15:53 smart/ -rw-r--r-- 1 root root 14445 Feb 9 15:51 unassigned.devices.ini -rw-r--r-- 1 root root 277 Feb 9 15:51 users.ini -rw-r--r-- 1 root root 3455 Feb 9 15:51 var.ini Makes sense why the webgui is completely broken when /usr/local/emhttp/state is missing. Only mystery is what caused it to disappear? I wonder if this happens again all I need to do it recreate that symlink ( /usr/local/emhttp/state ) as /usr appears to be a squashfs root@Tower:~# lsblk -f NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS loop0 squashfs 4.0 29.4G 6% /usr loop1 squashfs 4.0 29.4G 6% /lib loop2 btrfs bc576ead-e78c-xxxx-9ca6-xxxxxd87f268 71.2G 27% /var/lib/docker/btrfs /var/lib/docker loop3 btrfs 2d42084a-379a-xxxx-8656-xxxxxfd1c52f 903.8M 1% /etc/libvirt .... loop0 comes from /boot/bzfirmware root@Tower:~# losetup -a /dev/loop1: [2049]:10 (/boot/bzmodules) /dev/loop2: [66305]:304660166 (/mnt/disks/Dockers/system/docker.img) /dev/loop0: [2049]:12 (/boot/bzfirmware) /dev/loop3: [66307]:2149391489 (/mnt/disks/VMs/system/libvirt/libvirt.img) Edited February 9, 20251 yr by phoenixdiigital More information supplied about loop0
February 9, 20251 yr Author tower-diagnostics-20250209-0639.zip - Diagnostic I took via CLI when the issue was present tower-diagnostics-20250210-0605.zip - Diagnostic I took just now with Web GUI working fine after reboot yesterday. Regarding my suspicion of GPU Statistics Plugin. I've been running it for about 6 months without issue. The WebGUI 500 errors started appearing late Dec 2024 from memory. tower-diagnostics-20250209-0639.zip tower-diagnostics-20250210-0605.zip Edited February 9, 20251 yr by phoenixdiigital
February 10, 20251 yr Community Expert Don't see anything relevant in the logs, would recommend retesting in safe mode.
February 10, 20251 yr Author 50 minutes ago, JorgeB said: Don't see anything relevant in the logs, would recommend retesting in safe mode. Thanks for checking. Retesting what specifically as the issue crops up randomly? The server will be running fine for days/weeks and then one day the Web GUI is broken and returns HTTP Error 500. Shares, SSH, Containers and VMs are still running fine just the web interface for unraid is dead. After my deeper investigation this time it appears that the symlink for /usr/local/emhttp/state -> /var/local/emhttp/ appears to have disappeared. I'm not sure what causes it at all. Rebooting always brings the web interface back to a funuctioning state. So there must be something at startup which creates that symlink. If it happens again I'll try manually recreating the symlink before a reboot. Edited February 10, 20251 yr by phoenixdiigital
February 10, 20251 yr Community Expert Sorry, but without anything logged in the diags difficult to guess what the problem is, do you know the approximate time it last happened?
February 10, 20251 yr Author No apologies needed. I agree there is no obvious cause in the logs. At least I know a possible fix if I see it again (recreate symlink). I'd say it happened in the tower-diagnostics-20250209-0639.zip diagnostic between 10:30pm on the 8th Feb - > 6am on the 9th of Feb when I woke up and noticed the web gui was dead. Overnight it was running a unbalance to move stuff around between disks. The other times it died unbalance was not being used so that's not likely the contributing factor. First signs of that symlink being gone would be this 2025-02-09T06:08:10.707468+10:00 Tower webGUI: error: /plugins/unassigned.devices.preclear/include/Preclear.php - uninitialized csrf_token Which is likely when I tried to load the WebGUI from the browser and the php code behind it tried to load a bunch of config from /usr/local/emhttp/state which was no longer there. It's as if /usr got remounted from /boot/bzfirmware and none of the stuff UnRAID adds to that filesystem during boot was there anymore. Any ideas where this happens during boot and if I could run it manually?
February 10, 20251 yr Community Expert When that happens again, check the output from df -h / Just to make sure rootfs is not getting filled up
February 10, 20251 yr Author 7 hours ago, JorgeB said: When that happens again, check the output from df -h / Just to make sure rootfs is not getting filled up I also did check the output of "df - h" at the time and none of the mounts were full. Even the cache drive had settled down by the time I was troubleshooting.
February 10, 20251 yr I'm hopping the Solution was finding! I'm self Problem with the Webinterface and so. The Temp fix was Hard Reboot too. But i'm on 6.12.15 with AMD. Mysterious! Edited February 10, 20251 yr by Revan335
March 22, 20251 yr I have this exact same issue on 7.0 as well. My server froze and then when I booted it back up I got the same issue. You got a little further than me finding the state folder I was looking more at the fact that the PHP-FPM service wasn't starting, but it sounds like it is a dynamix issue with the update maybe. Did you end up finding the path that the symlink needs to go to? Or do you know if there's a way I could just mainly remove the plug in to stop this?
April 2, 20251 yr Author On 3/22/2025 at 11:12 AM, baldsealion said: I have this exact same issue on 7.0 as well. My server froze and then when I booted it back up I got the same issue. You got a little further than me finding the state folder I was looking more at the fact that the PHP-FPM service wasn't starting, but it sounds like it is a dynamix issue with the update maybe. Did you end up finding the path that the symlink needs to go to? Or do you know if there's a way I could just mainly remove the plug in to stop this? I never got any further. I've removed the GPU Stats plugin and haven't seen the issue again however I'm still not 100% sure it was the cause. As per my findings above the state diretory should be a symlink like this lrwxrwxrwx 1 root root 17 Nov 16 2023 state -> /var/local/emhttp/ It appears to get recreated on startup but no idea what script in the boot sequence creates it or how to trigger it without doing a restart. Edited April 2, 20251 yr by phoenixdiigital Clarification
April 22, 20251 yr On 2/10/2025 at 5:50 AM, phoenixdiigital said: No apologies needed. I agree there is no obvious cause in the logs. At least I know a possible fix if I see it again (recreate symlink). Just wanted to let you know that I tried this possible fix, but I am still receiving the 500 error. I ran this: ln -s /var/local/emhttp /usr/local/emhttp/state and restarted php-fpm and nginx for good measure, but I'm still seeing the 500 error. I also created symlinks for boot, log, mnt, and webGui in /usr/local/emhttp, but the error persists. I'm missing all php and htm files (except for redirect.htm) in that directory as I'm not sure where those are generated from. Just thought I'd share my steps and results in case it helps the next person. I'm getting impatient and need to reboot the server. I also plan on removing the GPU Stats plugin too because I can't keep having the webgui go down. Here is my /usr/local/emhttp directory after following the steps outlined above: drwxr-xr-x 1 root root 320 Apr 22 11:13 ./ drwxr-xr-x 1 root root 140 Dec 19 08:15 ../ lrwxrwxrwx 1 root root 5 Apr 22 11:13 boot -> /boot/ drwxr-xr-x 1 root root 60 Nov 15 2023 languages/ lrwxrwxrwx 1 root root 8 Apr 22 11:13 log -> /var/log/ lrwxrwxrwx 1 root root 4 Apr 22 11:13 mnt -> /mnt/ drwxr-xr-x 1 root root 340 Apr 21 01:00 plugins/ -rw-r--r-- 1 root root 1649 Jan 9 16:26 redirect.htm lrwxrwxrwx 1 root root 17 Apr 22 11:13 state -> /var/local/emhttp/ lrwxrwxrwx 1 root root 17 Apr 22 11:02 webGui -> ./plugins/dynamix/
April 24, 20251 yr Author On 4/23/2025 at 2:16 AM, BigSloppy said: Just wanted to let you know that I tried this possible fix, but I am still receiving the 500 error. Thanks for the update and the testing. That's a shame it didn't resolve it. I haven't seen this issue since my original post on this. Probably because I haven't had GPU Statistics installed since then as well so it's looking like it's the primary culprit.
April 27, 20251 yr Having the same issue now but with the includes directory being wiped out. State is still there. Only thing new I can think of was enabling unbalanced and running it. / is 99% free. I wonder if unbalance is doing something weird when it finishes or maybe there’s a case when something fails?
May 2, 20251 yr I'm fighting this same issue the past week. Been running 7.01 since it released. 500 Internal Server Error on the dashboard. I had the GPU Statistics plugin but removed it as @phoenixdiigital was saying, but still seeing the issue even after two reboots last night. Trying to just take it slow and not mad rush into any attempts at fixes, at least until I need to do something on the dashboard - lol
May 2, 20251 yr I have the same issus guys, you are not alone.. webui randomly gets unresponsible.. only reboot fixes.. all my users are getting pissed of the unstability of everything im providing them. i found basically only this thread but i can not imagine we are the only ones.. syslogs is not helpfull no errors.. same for ngix
May 3, 20251 yr Author The only recommendation to @ryans100 and @Excision I would have would be remove all non-critical plugins you have then try a reboot and see. If it works start adding them back one by one leaving a day or two between adding another one. In my case GPU Statistics was a "nice to have" purely from a nerd perspective to see the GPU usage. It made no difference to the functionality of my UnRAID server. It's definitely an annoying bug and even more annoying that nothing was captured in the logs.
May 10, 20251 yr I just got this issue tonight, out of the blue. I've not touched the OS/plugins/dockers in weeks, but the webgui suddenly stopped working today with this exact same error. Missing the symlink to the state. I'm on 7.0.0 as well. Def seems like there might be some sort of bug or errant plugin code.... May 8 00:00:02 towerstor Plugin Auto Update: Checking for available plugin updates May 8 00:00:06 towerstor Plugin Auto Update: Checking for language updates May 8 00:00:06 towerstor Plugin Auto Update: Community Applications Plugin Auto Update finished May 8 01:13:43 towerstor root: /mnt/cache: 82.5 GiB (88530468864 bytes) trimmed on /dev/nvme0n1p1 May 8 02:56:35 towerstor root: /mnt/vms: 211.2 GiB (226770313216 bytes) trimmed on /dev/sdi1 May 8 02:56:37 towerstor root: /var/lib/docker: 32.1 GiB (34504237056 bytes) trimmed on /dev/loop2 May 8 22:39:16 towerstor crond[1834]: exit status 255 from user root php /usr/local/emhttp/plugins/community.applications/scripts/notices.php > /dev/null 2>&1 May 8 23:39:01 towerstor crond[1834]: exit status 255 from user root php /usr/local/emhttp/plugins/community.applications/scripts/notices.php > /dev/null 2>&1 May 9 00:00:01 towerstor Plugin Auto Update: Checking for available plugin updates May 9 00:00:06 towerstor Plugin Auto Update: Checking for language updates May 9 00:00:06 towerstor Plugin Auto Update: Community Applications Plugin Auto Update finished May 9 00:39:01 towerstor crond[1834]: exit status 255 from user root php /usr/local/emhttp/plugins/community.applications/scripts/notices.php > /dev/null 2>&1 This is the only thing in the syslogs
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.