Hiya,

 

Unfortunately I have had some reoccurring "out of memory" problems with my server over the past few weeks. This time I was actually able to get some diagnostics output which I have attached.

I had to run `diagnostics` from the command line which also gave errors.
 

Starting diagnostics collection... 
Warning: file_put_contents(): Only 0 of 5 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 80

Warning: file_put_contents(): Only 0 of 15 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 12 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 665 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 13 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 599 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 1034 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 15 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 11 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 15 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 14 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 11 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 82 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 101

Warning: file_put_contents(): Only 0 of 2 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 120

Warning: file_put_contents(): Only 0 of 34 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 122
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device

Warning: file_put_contents(): Only 0 of 29 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

Warning: file_put_contents(): Only 0 of 25 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

 

EDIT: Currently running a MEMTEST. First pass completed without an error.

tower-diagnostics-20200710-1912.zip

Edited July 11, 20205 yr by ZipsServer

I ran a MEMTEST for 18 hours with no errors. Still unsure what is causing these issues.

All files on the diags are empty, reboot and try to get new diags as soon as the issue starts.

This problem has happened 5+ times now. Every time I am unable to get diagnostics.

I am able to get the syslog, but it really only shows rcpbind messages from my NFS shares.

Could it possibly be a disk problem?

 

I have three external hard drives running in a btrfs pool through Unassigned Devices. My system just froze again and I was unable to run diagnostics. I also noticed that my dockers using the Unassigned Devices btrfs pool were not working properly and the Unassigned Devices plugin UI was not displaying any info.

I hard disconnected the external hard drives and everything returned to normal. Unsure if this was the cause of previous problems. Attached is the diags i just ran.

mastertower-diagnostics-20200719-1509.zip

 

EDIT: I want to clarify that I have run these external disks through a preclear without any errors and SMART seemed to be ok. This leads me to believe there may be a problem with using a manually created btrfs pool (no raid) with external disks. Any thoughts here?

Edited July 19, 20205 yr by ZipsServer

13 hours ago, ZipsServer said:

but it really only shows rcpbind messages from my NFS shares.

Those messages end up filling the syslog.

 

12 hours ago, ZipsServer said:

This leads me to believe there may be a problem with using a manually created btrfs pool (no raid) with external disks.

If by external you mean USB disks we don't recommend those used for pools/array devices, but it's not necessarily a problem, impossible to say if there are issues with this without the diags, ones posted just have the NFS spam.

A new diag is attached. A preclear using the Unassigned Devices plugin just failed for some reason.

mastertower-diagnostics-20200726-1307.zip

Logs are still being spammed, very difficult to look for anything else.

The rcpbind log spamming with NFS shares has been brought up in other posts such as this

 

Unfortunately, there is no way to stop these log messages that I know of currently. I have sent a message to the support email but have not heard back.

I really don't understand why more people don't complain about the rcpbind spamming.

I have two disk shares that are accessed by two devices using NFS. The spamming is so bad I can't see anything else in my logs. Is there still no fix for this?