Hello! I noticed here recently, especially after installing the "My Servers" plugin (But I'm not sure if that's caused it, I've uninstalled it for now and will see if I continue to get this), I've been having issues with my UNRAID server. This has been happening every day, when I go to the Main tab it doesn't really show my disks correctly, and I have the "undefined: undefined" error you see in the screenshot. I attempted to pull the Diagnostics but was unable to as it just wouldn't finish loading. 

 

I've pulled the syslog, and there are errors, but I'm not really sure if any of them are pointing to the issue or not. 
Could anyone take a look and let me know if you see anything specific you think could be causing this behavior? 

Appreciate any time and all the effort of the community

UNRAID Syslog

Looks like the error starts here :

Sep  7 06:29:09 Apollo unraid-api[5054]: ⚠️ Caught exception: Maximum call stack size exceeded
Sep  7 06:29:09 Apollo unraid-api[5054]: ⚠️ UNRAID API crashed with exit code 1
Sep  7 06:40:53 Apollo nginx: 2022/09/07 06:40:53 [crit] 5329#5329: *407337 pwritev() "/var/lib/nginx/client_body/0000049031" has written only 4096 of 9051, client: unix:, server: , request: "POST /pub/disks?buffer_length=1 HTTP/1.1", host: "localhost"
Sep  7 06:43:37 Apollo nmbd[23864]: [2022/09/07 06:43:37.650269,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 06:43:37 Apollo nmbd[23864]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 06:46:29 Apollo kernel: TCP: request_sock_TCP: Possible SYN flooding on port 32611. Sending cookies.  Check SNMP counters.
Sep  7 06:48:02 Apollo unraid-api[32346]: ✔️ UNRAID API started successfully!
Sep  7 06:48:07 Apollo unraid-api[32346]: ⚠️ Caught exception: Maximum call stack size exceeded
Sep  7 06:48:07 Apollo unraid-api[32346]: ⚠️ UNRAID API crashed with exit code 1
Sep  7 06:58:05 Apollo unraid-api[20466]: ✔️ UNRAID API started successfully!
Sep  7 06:58:15 Apollo unraid-api[20466]: ⚠️ Caught exception: Maximum call stack size exceeded
Sep  7 06:58:15 Apollo unraid-api[20466]: ⚠️ UNRAID API crashed with exit code 1
Sep  7 06:58:37 Apollo nmbd[23214]: [2022/09/07 06:58:37.656692,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 06:58:37 Apollo nmbd[23214]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 07:13:37 Apollo nmbd[18929]: [2022/09/07 07:13:37.660920,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 07:13:37 Apollo nmbd[18929]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 07:18:09 Apollo unraid-api[27759]: ✔️ UNRAID API started successfully!
Sep  7 07:18:12 Apollo unraid-api[27759]: ⚠️ Caught exception: Maximum call stack size exceeded
Sep  7 07:18:12 Apollo unraid-api[27759]: ⚠️ UNRAID API crashed with exit code 1
Sep  7 07:28:37 Apollo nmbd[15440]: [2022/09/07 07:28:37.667499,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 07:28:37 Apollo nmbd[15440]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 07:28:42 Apollo smbd[15577]: [2022/09/07 07:28:42.980566,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 07:28:42 Apollo smbd[15577]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 07:43:37 Apollo winbindd[8829]: [2022/09/07 07:43:37.655531,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 07:43:37 Apollo winbindd[8829]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 07:43:37 Apollo nmbd[8830]: [2022/09/07 07:43:37.670856,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 07:43:37 Apollo nmbd[8830]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 07:43:43 Apollo smbd[8972]: [2022/09/07 07:43:43.084651,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 07:43:43 Apollo smbd[8972]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 07:59:35 Apollo nginx: 2022/09/07 07:59:35 [crit] 5329#5329: *450673 pwritev() "/var/lib/nginx/client_body/0000049032" failed (28: No space left on device), client: unix:, server: , request: "POST /pub/disks?buffer_length=1 HTTP/1.1", host: "localhost"
Sep  7 08:09:46 Apollo unraid-api[21794]: ✔️ UNRAID API started successfully!
Sep  7 08:11:45 Apollo unraid-api[21794]: ⚠️ Caught exception: Maximum call stack size exceeded
Sep  7 08:11:45 Apollo unraid-api[21794]: ⚠️ UNRAID API crashed with exit code 1
Sep  7 08:28:37 Apollo winbindd[25498]: [2022/09/07 08:28:37.669425,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 08:28:37 Apollo winbindd[25498]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 08:28:37 Apollo nmbd[25499]: [2022/09/07 08:28:37.700896,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 08:28:37 Apollo nmbd[25499]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 08:43:37 Apollo winbindd[17959]: [2022/09/07 08:43:37.674478,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 08:43:37 Apollo winbindd[17959]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 08:43:37 Apollo nmbd[17960]: [2022/09/07 08:43:37.709682,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 08:43:37 Apollo nmbd[17960]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 08:43:43 Apollo smbd[18115]: [2022/09/07 08:43:43.459556,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 08:43:43 Apollo smbd[18115]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 08:58:37 Apollo winbindd[11312]: [2022/09/07 08:58:37.678150,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 08:58:37 Apollo winbindd[11312]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 08:58:37 Apollo nmbd[11313]: [2022/09/07 08:58:37.721422,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 08:58:37 Apollo nmbd[11313]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 09:10:07 Apollo root: error: /webGui/include/Notify.php: uninitialized csrf_token
Sep  7 09:10:07 Apollo root: error: /plugins/community.applications/scripts/notices.php: uninitialized csrf_token
Sep  7 09:10:07 Apollo root: error: /webGui/include/Report.php: uninitialized csrf_token
Sep  7 09:10:15 Apollo unassigned.devices: PHP Warning: syntax error, unexpected end of file, expecting TC_DOLLAR_CURLY or TC_QUOTED_STRING or ''' in /usr/local/emhttp/state/disks.ini on line 270  in /usr/local/emhttp/plugins/unassigned.devices/include/lib.php on line 49
Sep  7 09:10:15 Apollo unassigned.devices: PHP Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/dynamix/include/DefaultPageLayout.php(528) : eval()'d code on line 21
Sep  7 09:10:15 Apollo root: error: /webGui/include/Notify.php: uninitialized csrf_token
Sep  7 09:10:15 Apollo root: error: /webGui/include/Report.php: uninitialized csrf_token
Sep  7 09:10:15 Apollo root: error: /plugins/community.applications/scripts/notices.php: uninitialized csrf_token
Sep  7 09:10:15 Apollo root: error: /plugins/dynamix.plugin.manager/scripts/PluginAPI.php: uninitialized csrf_token
Sep  7 09:10:15 Apollo root: error: /plugins/unassigned.devices/UnassignedDevices.php: uninitialized csrf_token
Sep  7 09:10:15 Apollo root: error: /plugins/unassigned.devices/UnassignedDevices.php: uninitialized csrf_token
Sep  7 09:10:18 Apollo root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token
Sep  7 09:10:21 Apollo root: error: /webGui/include/Report.php: uninitialized csrf_token
Sep  7 09:10:21 Apollo root: error: /plugins/unassigned.devices/UnassignedDevices.php: uninitialized csrf_token
Sep  7 09:10:22 Apollo unassigned.devices: PHP Warning: file_put_contents(): Only -1 of 114 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/unassigned.devices/include/lib.php on line 92
Sep  7 09:10:22 Apollo unassigned.devices: PHP Warning: file_put_contents(): Only -1 of 2420 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/unassigned.devices/include/lib.php on line 210
Sep  7 09:10:22 Apollo unassigned.devices: PHP Warning: file_put_contents(): Only -1 of 114 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/unassigned.devices/include/lib.php on line 92
Sep  7 09:10:22 Apollo unassigned.devices: PHP Warning: file_put_contents(): Only -1 of 2225 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/unassigned.devices/include/lib.php on line 210
Sep  7 09:10:23 Apollo unassigned.devices: PHP Warning: file_put_contents(): Only -1 of 114 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/unassigned.devices/include/lib.php on line 92

 

and then the php errors from UD continue. They might be the consequence of the previous errors, not the cause ?

1 hour ago, ChatNoir said:

Looks like the error starts here :

Sep  7 06:29:09 Apollo unraid-api[5054]: ⚠️ Caught exception: Maximum call stack size exceeded
Sep  7 06:29:09 Apollo unraid-api[5054]: ⚠️ UNRAID API crashed with exit code 1
Sep  7 06:40:53 Apollo nginx: 2022/09/07 06:40:53 [crit] 5329#5329: *407337 pwritev() "/var/lib/nginx/client_body/0000049031" has written only 4096 of 9051, client: unix:, server: , request: "POST /pub/disks?buffer_length=1 HTTP/1.1", host: "localhost"
Sep  7 06:43:37 Apollo nmbd[23864]: [2022/09/07 06:43:37.650269,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 06:43:37 Apollo nmbd[23864]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 06:46:29 Apollo kernel: TCP: request_sock_TCP: Possible SYN flooding on port 32611. Sending cookies.  Check SNMP counters.
Sep  7 06:48:02 Apollo unraid-api[32346]: ✔️ UNRAID API started successfully!
Sep  7 06:48:07 Apollo unraid-api[32346]: ⚠️ Caught exception: Maximum call stack size exceeded
Sep  7 06:48:07 Apollo unraid-api[32346]: ⚠️ UNRAID API crashed with exit code 1
Sep  7 06:58:05 Apollo unraid-api[20466]: ✔️ UNRAID API started successfully!
Sep  7 06:58:15 Apollo unraid-api[20466]: ⚠️ Caught exception: Maximum call stack size exceeded
Sep  7 06:58:15 Apollo unraid-api[20466]: ⚠️ UNRAID API crashed with exit code 1
Sep  7 06:58:37 Apollo nmbd[23214]: [2022/09/07 06:58:37.656692,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 06:58:37 Apollo nmbd[23214]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 07:13:37 Apollo nmbd[18929]: [2022/09/07 07:13:37.660920,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 07:13:37 Apollo nmbd[18929]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 07:18:09 Apollo unraid-api[27759]: ✔️ UNRAID API started successfully!
Sep  7 07:18:12 Apollo unraid-api[27759]: ⚠️ Caught exception: Maximum call stack size exceeded
Sep  7 07:18:12 Apollo unraid-api[27759]: ⚠️ UNRAID API crashed with exit code 1
Sep  7 07:28:37 Apollo nmbd[15440]: [2022/09/07 07:28:37.667499,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 07:28:37 Apollo nmbd[15440]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 07:28:42 Apollo smbd[15577]: [2022/09/07 07:28:42.980566,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 07:28:42 Apollo smbd[15577]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 07:43:37 Apollo winbindd[8829]: [2022/09/07 07:43:37.655531,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 07:43:37 Apollo winbindd[8829]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 07:43:37 Apollo nmbd[8830]: [2022/09/07 07:43:37.670856,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 07:43:37 Apollo nmbd[8830]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 07:43:43 Apollo smbd[8972]: [2022/09/07 07:43:43.084651,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 07:43:43 Apollo smbd[8972]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 07:59:35 Apollo nginx: 2022/09/07 07:59:35 [crit] 5329#5329: *450673 pwritev() "/var/lib/nginx/client_body/0000049032" failed (28: No space left on device), client: unix:, server: , request: "POST /pub/disks?buffer_length=1 HTTP/1.1", host: "localhost"
Sep  7 08:09:46 Apollo unraid-api[21794]: ✔️ UNRAID API started successfully!
Sep  7 08:11:45 Apollo unraid-api[21794]: ⚠️ Caught exception: Maximum call stack size exceeded
Sep  7 08:11:45 Apollo unraid-api[21794]: ⚠️ UNRAID API crashed with exit code 1
Sep  7 08:28:37 Apollo winbindd[25498]: [2022/09/07 08:28:37.669425,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 08:28:37 Apollo winbindd[25498]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 08:28:37 Apollo nmbd[25499]: [2022/09/07 08:28:37.700896,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 08:28:37 Apollo nmbd[25499]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 08:43:37 Apollo winbindd[17959]: [2022/09/07 08:43:37.674478,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 08:43:37 Apollo winbindd[17959]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 08:43:37 Apollo nmbd[17960]: [2022/09/07 08:43:37.709682,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 08:43:37 Apollo nmbd[17960]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 08:43:43 Apollo smbd[18115]: [2022/09/07 08:43:43.459556,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 08:43:43 Apollo smbd[18115]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 08:58:37 Apollo winbindd[11312]: [2022/09/07 08:58:37.678150,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 08:58:37 Apollo winbindd[11312]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 08:58:37 Apollo nmbd[11313]: [2022/09/07 08:58:37.721422,  0] ../../source3/lib/util.c:482(reinit_after_fork)
Sep  7 08:58:37 Apollo nmbd[11313]:   messaging_reinit() failed: NT_STATUS_DISK_FULL
Sep  7 09:10:07 Apollo root: error: /webGui/include/Notify.php: uninitialized csrf_token
Sep  7 09:10:07 Apollo root: error: /plugins/community.applications/scripts/notices.php: uninitialized csrf_token
Sep  7 09:10:07 Apollo root: error: /webGui/include/Report.php: uninitialized csrf_token
Sep  7 09:10:15 Apollo unassigned.devices: PHP Warning: syntax error, unexpected end of file, expecting TC_DOLLAR_CURLY or TC_QUOTED_STRING or ''' in /usr/local/emhttp/state/disks.ini on line 270  in /usr/local/emhttp/plugins/unassigned.devices/include/lib.php on line 49
Sep  7 09:10:15 Apollo unassigned.devices: PHP Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/dynamix/include/DefaultPageLayout.php(528) : eval()'d code on line 21
Sep  7 09:10:15 Apollo root: error: /webGui/include/Notify.php: uninitialized csrf_token
Sep  7 09:10:15 Apollo root: error: /webGui/include/Report.php: uninitialized csrf_token
Sep  7 09:10:15 Apollo root: error: /plugins/community.applications/scripts/notices.php: uninitialized csrf_token
Sep  7 09:10:15 Apollo root: error: /plugins/dynamix.plugin.manager/scripts/PluginAPI.php: uninitialized csrf_token
Sep  7 09:10:15 Apollo root: error: /plugins/unassigned.devices/UnassignedDevices.php: uninitialized csrf_token
Sep  7 09:10:15 Apollo root: error: /plugins/unassigned.devices/UnassignedDevices.php: uninitialized csrf_token
Sep  7 09:10:18 Apollo root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token
Sep  7 09:10:21 Apollo root: error: /webGui/include/Report.php: uninitialized csrf_token
Sep  7 09:10:21 Apollo root: error: /plugins/unassigned.devices/UnassignedDevices.php: uninitialized csrf_token
Sep  7 09:10:22 Apollo unassigned.devices: PHP Warning: file_put_contents(): Only -1 of 114 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/unassigned.devices/include/lib.php on line 92
Sep  7 09:10:22 Apollo unassigned.devices: PHP Warning: file_put_contents(): Only -1 of 2420 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/unassigned.devices/include/lib.php on line 210
Sep  7 09:10:22 Apollo unassigned.devices: PHP Warning: file_put_contents(): Only -1 of 114 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/unassigned.devices/include/lib.php on line 92
Sep  7 09:10:22 Apollo unassigned.devices: PHP Warning: file_put_contents(): Only -1 of 2225 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/unassigned.devices/include/lib.php on line 210
Sep  7 09:10:23 Apollo unassigned.devices: PHP Warning: file_put_contents(): Only -1 of 114 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/unassigned.devices/include/lib.php on line 92

 

and then the php errors from UD continue. They might be the consequence of the previous errors, not the cause ?

 

Seems you agree it may have been due to the My Servers plugin, would this be more appropriate in the "My Servers Plugin Support", and can you move it there or should I re-create there? 
Thanks!

It's possible indeed, but I am not knowledgeable enough to be sure.

Let's seem what @ljm42 thinks about it. We can move the thread if that's confirmed.

Hmm, unraid-api is definitely not happy. Hard to say at the moment whether it is the cause or just another symptom of the problems.

 

Would you please open a web terminal and run:

cat /var/log/unraid-api/crash.json

and then copy/paste the output here?

6 minutes ago, ljm42 said:

Hmm, unraid-api is definitely not happy. Hard to say at the moment whether it is the cause or just another symptom of the problems.

 

Would you please open a web terminal and run:

cat /var/log/unraid-api/crash.json

and then copy/paste the output here?

 

I appreciate you taking a look ljm42

Unfortunately, I did reboot and uninstall the My Servers plugin so this doens't provide any output

 

root@Apollo:~# cat /var/log/unraid-api/crash.json
cat: /var/log/unraid-api/crash.json: No such file or directory

 

I just reinstalled, verified it was connected before and after sign in, and Activated flash backup

I'm still getting the "Failed to sync flash backup" 
The crash.json is also still empty

I went ahead and rebooted for good measure

 

After the reboot verified still connected, and no change, when hitting the Update button it just reloads the page and loads "Failed to sync flash backup" 
I had previously posted my gitflash & diagnostics on this thread, not sure if that might have what you were looking for. 

 

If not then I'll see if this happens again tomorrow after reinstalling and if so, should the crash.json be there after reboot? Or should I collect that log while it's affected if possible? 
Appreciate your time. 

Flash backup is unrelated to this thread, please see my recent comment here for more details: https://forums.unraid.net/topic/128043-my-servers-will-not-do-flash-backup/#comment-1167192

 

 

If you see something like this in your syslog:
  ⚠️ UNRAID API crashed with exit code 1
then doing this (before rebooting) will provide a few more details:

cat /var/log/unraid-api/crash.json

 

At this point we don't know why the system was having issues, but I'm glad you are back up and running.

1 minute ago, ljm42 said:

Flash backup is unrelated to this thread, please see my recent comment here for more details: https://forums.unraid.net/topic/128043-my-servers-will-not-do-flash-backup/#comment-1167192

 

 

If you see something like this in your syslog:
  ⚠️ UNRAID API crashed with exit code 1
then doing this (before rebooting) will provide a few more details:

cat /var/log/unraid-api/crash.json

 

At this point we don't know why the system was having issues, but I'm glad you are back up and running.

 

I'll keep an eye out for this, just so you're aware, in case this is related to unraid-api, I was able to reboot the server and regain normal functionality until 12-24 hours later it started doing this original issue. 

I haven't experienced this issue after reinstalling so I'm assuming that may have resolved it.