Server Hard Crashing Please HELP

[Aspect]

I'm having an issue with my server crashing i thought it was a dynamix plugin causing it because the last time it happened the syslog had this error - <br />
<b>Fatal error</b>:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 413106712 bytes) in <b>/usr/local/emhttp/plugins/dynamix.file.manager/include/Browse.php</b> on line <b>62</b><br />

 

however that plugin now has an error and is not loaded. last night at about 18:00 EST my server locked up again (when this happens I'm not notified because from the outside it doesn't seem like anything happened) when logging in this morning i got to a the dashboard that has a full white background and looks almost like it was reset to all default settings and i have a bunch of notifications and warnings.

 

I can click restart in the GUI and restart successfully however when the server comes back up it thinks it was a unclean shutdown and starts a parity check.

 

I really need help as I'm worried about data lose and other issues caused by these crash's.

 

I'm on version 6.11.5 and this has only happened to me while on this version. i started a syslog the last 2 times it happened so i have that log if its helpful or can provide the diagnostics log. any info needed to help ill provide i just want to find an answer and resolve this problem.

 

any help would be greatly appreciated!

[JorgeB]

4 minutes ago, Aspect said:

i started a syslog the last 2 times it happened so i have that log if its helpful or can provide the diagnostics log

Post these.

[Aspect]

16 minutes ago, JorgeB said:

Post these.

as requested. see attached.

rocky-diagnostics-20230207-0751.zip syslog-127.0.0.1.zip

[JorgeB]

Start here:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=819173

 

[Aspect]

1 hour ago, JorgeB said:

Start here:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=819173

 

just to confirm your thinking potential C-State issue? that could make sense as i made these changes when you posted about this but i forgot to implement them again when updating my BIOS. when i was looking in the logs myself it looks like the USB boot drive loses connection / unmounts can this be caused by C-State switching and crashing the server?

[JorgeB]

14 minutes ago, Aspect said:

the USB boot drive loses connection / unmounts can this be caused by C-State switching and crashing the server?

Not usually, but try that first.

[Aspect]

1 minute ago, JorgeB said:

Not usually, but try that first.

i will try and get back to you. thanks for looking into this and giving your suggestion

[Aspect]

On 2/7/2023 at 11:56 AM, JorgeB said:

Not usually, but try that first.

@JorgeB Reporting back i have changed "Power Supply Idle Control" to "typical current idle" and disabled global C-states. unfortunately at about 16:49 yesterday (per log events) the issue happened again and my server went into that weird state. all white background all dockers and VM's not shown but still running. a lot of warnings and notifications log showing 100% and the 1 syslog showing the following error again.

 

<br />
<b>Fatal error</b>:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 227812408 bytes) in <b>/usr/local/emhttp/plugins/dynamix.file.manager/include/Browse.php</b> on line <b>62</b><br />

syslog-127.0.0.1.log.zip

[JorgeB]

Don't know what causes those errors but the log is completely spammed, making it very difficult to look for anything else.

[Aspect]

Just now, JorgeB said:

Don't know what causes those errors but the log is completely spammed, making it very difficult to look for anything else.

@JorgeB yeah that's been my issue as well with trying to hunt something down in the log. i notice the USB boot device looks like it drops out and then i get the File manager error then the spamming begins and the file manager error made me think it was something to do with that plugin. im going to uninstall that plugin for now and see what happens. hopefully someone can help me because right now this is happening almost every other day.

[JorgeB]

Make sure you are using a USB 2.0 port.

[Aspect]

Just now, JorgeB said:

Make sure you are using a USB 2.0 port.

@JorgeB I am using a USB 2.0 port on the cases front USB ports but its one that's plugged directly into the motherboard not sure that would cause a problem?

 

do these symptoms look anything like a failing USB drive?

[JorgeB]

Worth trying another port, one on the board, if the same try with a different flash drive.

[Aspect]

Just now, JorgeB said:

Worth trying another port, one on the board, if the same try with a different flash drive.

@JorgeB i just uninstalled file manager since that was specifically in the logs, i installed the flash auto remount plugin and tonight i will swap to a USB2.0 on the motherboard. and i will report back.

[trurl]

If you continue to get emhttp memory size errors, check that you aren't filling rootfs

df -h /

 

[Aspect]

6 minutes ago, trurl said:

If you continue to get emhttp memory size errors, check that you aren't filling rootfs

df -h /

 

@trurl Current output is -

 

root@Rocky:~# df -h /
Filesystem      Size  Used Avail Use% Mounted on
rootfs           32G  2.0G   30G   7% /

 

would i need to just continue to check this or check when i log into the GUI when its in that partial crash state?

Edited February 9, 2023 by Aspect

[trurl]

7 hours ago, Aspect said:

check when i log into the GUI when its in that partial crash state

 

[Aspect]

Digging through my log file i see at Feb  8 16:49:XX its almost like my USB drive got disconnected and the server logs show like the server restarted multiple times within a few minutes?

 

I'm going to look through every line of these logs and try to find something that makes sense. I really want to figure this out because the only way I know when the server has crashed in this odd state is when I log in to check or do something. @JorgeB @trurl Thank you both for your suggestions and looking into this. Hopefully I can report back what the cause was and what resolved the issue.

syslog-127.0.0.1.log.zip

Edited February 9, 2023 by Aspect

[trurl]

Why does your server have a public IP address?

IPADDR[0]="105.55.5.45"

 

https://en.wikipedia.org/wiki/Private_network

 

Looks like that is owned by something in Kenya

https://ipinfo.io/AS33771/105.48.0.0/12-105.55.4.0/22

[Aspect]

@trurl I use PFsense and i have my server ipv4 port set to that IP its internal to my network i use 192.168.2.0 , 10.10.10.0 and 105.55.5.0. Is that something i should change i assumed the ipv4 addresses of my internal network didnt matter? 

Edited February 9, 2023 by Aspect

[Aspect]

I changed it realized my mistake. using 10.10.5.0/24 now.

[Aspect]

@trurl @JorgeB So this morning i logged in to check to make sure the server wasn't in a soft crash state, its not fully in the state i had started this thread for, now it seems normal other than the fact that its stuck saying starting services in the bottom left even though everything is started and has been for 2 days. i checked in a private browser and on chrome and Firefox and still shows the same.

 

output of df -h / is

root@Rocky:~# df -h /
Filesystem      Size  Used Avail Use% Mounted on
rootfs           32G  2.1G   30G   7% /
 

so it went up by .1GB.

 

there are no notifications or errors other than the "starting services" notification.

 

i'm attaching the current diagnostics.

 

since i last posted i have moved the USB to a 2.0 space directly on the motherboard and changed the IP range that my servers are on.

 

is there any chance this is a symptom of a dying USB? the one that's in this server has been there for a few years and it use to be in a 3.0 slot until i read that that can cause it to burn out quicker.

 

I'm at a lose and don't know what to do this is becoming an upsetting issue because no matter what i do i cant seem to find or resolve the problem.

 

 

rocky-diagnostics-20230212-1011.zip

[trurl]

7 minutes ago, Aspect said:

stuck saying starting services

Do you have other browsers or tabs open to your Unraid webUI on any devices?

[Aspect]

@trurl no i do not currently just the 1 tab open for unraid webUI. only 2 devices i log into this server are my computer and phone. phone is logged out and this computer has 1 tab open.

 

can it cause an issue when there is a big difference in shares when running these 2 commands?

 

ls -al /mnt/user

and

ls -al /boot/config/shares

 

[trurl]

/boot/config/shares are just the .cfg files for any shares you have created and made settings for. Some shares may not be in that list if no settings were made for them, and there may be .cfg files for shares that no longer exist.