Out of Memory Problems

[ZipsServer]

Hiya,

 

Unfortunately I have had some reoccurring "out of memory" problems with my server over the past few weeks. This time I was actually able to get some diagnostics output which I have attached.

I had to run `diagnostics` from the command line which also gave errors.
 

Starting diagnostics collection... 
Warning: file_put_contents(): Only 0 of 5 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 80

Warning: file_put_contents(): Only 0 of 15 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 12 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 665 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 13 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 599 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 1034 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 15 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 11 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 15 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 14 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 11 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 82 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 101

Warning: file_put_contents(): Only 0 of 2 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 120

Warning: file_put_contents(): Only 0 of 34 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 122
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device

Warning: file_put_contents(): Only 0 of 29 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

Warning: file_put_contents(): Only 0 of 25 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

 

EDIT: Currently running a MEMTEST. First pass completed without an error.

tower-diagnostics-20200710-1912.zip

Edited July 11, 2020 by ZipsServer

[ZipsServer]

I ran a MEMTEST for 18 hours with no errors. Still unsure what is causing these issues.

[JorgeB]

All files on the diags are empty, reboot and try to get new diags as soon as the issue starts.

[ZipsServer]

This problem has happened 5+ times now. Every time I am unable to get diagnostics.

I am able to get the syslog, but it really only shows rcpbind messages from my NFS shares.

[ZipsServer]

Could it possibly be a disk problem?

 

I have three external hard drives running in a btrfs pool through Unassigned Devices. My system just froze again and I was unable to run diagnostics. I also noticed that my dockers using the Unassigned Devices btrfs pool were not working properly and the Unassigned Devices plugin UI was not displaying any info.

I hard disconnected the external hard drives and everything returned to normal. Unsure if this was the cause of previous problems. Attached is the diags i just ran.

mastertower-diagnostics-20200719-1509.zip

 

EDIT: I want to clarify that I have run these external disks through a preclear without any errors and SMART seemed to be ok. This leads me to believe there may be a problem with using a manually created btrfs pool (no raid) with external disks. Any thoughts here?

Edited July 19, 2020 by ZipsServer

[JorgeB]

13 hours ago, ZipsServer said:

but it really only shows rcpbind messages from my NFS shares.

Those messages end up filling the syslog.

 

12 hours ago, ZipsServer said:

This leads me to believe there may be a problem with using a manually created btrfs pool (no raid) with external disks.

If by external you mean USB disks we don't recommend those used for pools/array devices, but it's not necessarily a problem, impossible to say if there are issues with this without the diags, ones posted just have the NFS spam.

[ZipsServer]

A new diag is attached. A preclear using the Unassigned Devices plugin just failed for some reason.

mastertower-diagnostics-20200726-1307.zip

[JorgeB]

Logs are still being spammed, very difficult to look for anything else.

[ZipsServer]

The rcpbind log spamming with NFS shares has been brought up in other posts such as this

 

Unfortunately, there is no way to stop these log messages that I know of currently. I have sent a message to the support email but have not heard back.

I really don't understand why more people don't complain about the rcpbind spamming.

[JWHIII]

I have two disk shares that are accessed by two devices using NFS. The spamming is so bad I can't see anything else in my logs. Is there still no fix for this?