Server Lock Up


Recommended Posts

Last few weeks my server has been locking up and sending a mass of error emails with the subject of 

"cron for user root /etc/rc.d/rc.diskinfo --daemon &> /dev/null" and in the body "/bin/sh: fork: retry: Resource temporarily unavailable". I am able to log in to the gui and ssh but it refuses to do a clean shut down. I have to hard power off. This has started happening every couple of days now. I thought it was due to Plex transcoding to RAM which I set up about a month ago but I've since reverted it back to default and this issue continues happening. Diagnostics file is attached, I would be grateful for any assistance I can get with this problem. 

 

tower-diagnostics-20200125-0937.zip

Link to comment

The badness starts at Jan 25 04:01:07. What do you have configured to run at 04:00? It might be something inside one of you containers.

 

Is it always at 04:00?

 

Can you get dmesg output when the system is in a bad state? Also consider a remote syslog facility.

 

Since you say the system is responding over TCP/IP; ssh in, then run dmesg command look for oops (kernel panic) and oom (out of memory) errors.

Link to comment
  • 2 weeks later...

This happened again this morning at 4am. It only happens every few weeks. The only thing I have running is the user script to backup my VMs. I can access the gui and SSH but it hangs when I try to reboot. The console shows "Waiting up to 420 seconds for graceful shutdown". I stopped the docker and VM service before trying to reboot so I don't know what it's hanging on. 
I was able to get the dmesg output but a search for oops and oom turned up nothing. Not sure what is causing this to happen.

 

 

dmesg2.5.20.txt tower-diagnostics-20200205-0821.zip

Link to comment

Log is filled with these:

Feb  5 04:20:02 TOWER rc.diskinfo[12971]: PHP Warning: exec(): Unable to fork [timeout -s 9 60 /bin/lsblk -nbP -o name,type,size,mountpoint,fstype,label 2>/dev/null] in /etc/rc.d/rc.diskinfo on line 361
Feb  5 04:20:03 TOWER rc.diskinfo[20296]: PHP Warning: exec(): Unable to fork [timeout -s 9 60 /bin/lsblk -nbP -o name,type,size,mountpoint,fstype,label 2>/dev/null] in /etc/rc.d/rc.diskinfo on line 361

 

Any idea what you have installed causing this? If not try booting in safe mode.

Link to comment
1 hour ago, johnnie.black said:

Log is filled with these:


Feb  5 04:20:02 TOWER rc.diskinfo[12971]: PHP Warning: exec(): Unable to fork [timeout -s 9 60 /bin/lsblk -nbP -o name,type,size,mountpoint,fstype,label 2>/dev/null] in /etc/rc.d/rc.diskinfo on line 361
Feb  5 04:20:03 TOWER rc.diskinfo[20296]: PHP Warning: exec(): Unable to fork [timeout -s 9 60 /bin/lsblk -nbP -o name,type,size,mountpoint,fstype,label 2>/dev/null] in /etc/rc.d/rc.diskinfo on line 361

 

Any idea what you have installed causing this? If not try booting in safe mode.

Seeing the diskinfo line the only thing that stands out to me would be the disk location plugin. Could that be causing this issue?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.