Services crashing overnight?

[petjek]

Hi there,

 

During the last few days I considered that each and every morning there are the same problems with my unRaid.

I have 5.0RC5 running with the following add ons:

- unMenu

- SimpleFeatures (all but iTunes-Server and S3)

- Plex

- pyLoad (+ dependencies)

- Shell-in-a-box

 

Every morning (if I have a look at the morning) unmenu is no more running. So is Plex and emhttp.

Restarting unMenu and Plex ist easy doing but there is no chance to get emhttp running again except by a complete restart.

Mover script also runs at 3.40h cleaning the cache drive.

 

To restart emhttp I entered  "/usr/local/sbin/emhttp &"

This ist what the syslog says:

Aug 15 07:25:58 hightower unmenu-status: Starting unmenu web-server
Aug 15 07:28:53 hightower emhttp: unRAID System Management Utility version 5.0-rc5 (Lime Tech)
Aug 15 07:28:53 hightower emhttp: Copyright (C) 2005-2012, Lime Technology, LLC (Lime Tech)
Aug 15 07:28:53 hightower emhttp: Plus key detected, GUID: 090C-1000-0011-122200040548 (Other emhttp)
Aug 15 07:28:54 hightower emhttp: rdevName.22 not found (Other emhttp)
Aug 15 07:28:54 hightower emhttp: diskFsStatus.1 not found (Other emhttp)
Aug 15 07:28:54 hightower kernel: emhttp[2471]: segfault at 0 ip b7471760 sp bf90ceb0 error 4 in libc-2.11.1.so[b73f8000+15c000] (Errors)

So I tried "nohup /usr/local/sbin/emhttp &". I have no idea what this nohup command means but the result is almost the same.

Aug 15 07:25:58 hightower unmenu-status: Starting unmenu web-server
Aug 15 07:28:53 hightower emhttp: unRAID System Management Utility version 5.0-rc5 (Lime Tech)
Aug 15 07:28:53 hightower emhttp: Copyright (C) 2005-2012, Lime Technology, LLC (Lime Tech)
Aug 15 07:28:53 hightower emhttp: Plus key detected, GUID: 090C-1000-0011-122200040548 (Other emhttp)
(Errors)

So I think the last line describes the problem:

Aug 15 07:28:54 hightower kernel: emhttp[2471]: segfault at 0 ip b7471760 sp bf90ceb0 error 4 in libc-2.11.1.so[b73f8000+15c000] (Errors)

But not for me unfortunately. Anybody who can translate for a newbee? What can I do? Do I have to reinstall anything? What is this "libc-2.11.1.so"?

Is there a way to see why those services crash overnight?

 

petjek

[Joe L.]

Odds are one of your add-on programs is using all the memory and as a result the kernel out-of-memory process kills off programs that have been idle the longest in an attempt to reclaim memory.

 

According to Tom @lime-tech emhttp was not designed to be re-started, therefore, you attempting to re-start it is failing.  (at one point, on older releases, you could re-start it.  apparently, unless he decides it is a bug with high priority and fixes it, you cannot any longer) 

 

Unfortunately, you cannot shut down the array easily without it.  You'll need to follow the individual steps listed in the wiki. (or just force power off and expect it to start a parity check)  Since you have unmenu installed, and may be able to re-start it, you can try to use its stop array button to stop the array.  It may not work, as if you are out-of-memory,it may be killed off mid-process too.

 

The fix if an add-on is causing the crash is obvious, find the add-on that is using up your memory, and remove it (or replace it with a fixed version)

[Joe L.]

libc .so is the shared library containing the system calls used to access the operating system devices.

Is there a way to see why those services crash overnight?

Yes,  read the sticky at the top of this general support forum.  Then, do as it requests... provide a system log for analysis when you make your next post.

[petjek]

As Plex crashed again today at 9.03am I uninstalled everything I didn't need 'for life'!

As I can see now there ist less than a gigabyte of ram used (used & cached). That was different before as there was a hugh amount (almost 3GB) of ram cached.

Hope things will work fine now.

[petjek]

Well, that didn't help. Emhttp still crashes overnight. I pushed all of the simple feature extensions to a tmp directory and restarted.

I had a look at the syslog but from there I can't see what happened. Where else can I probably see the problem? Are there any mystic console commands that will show up what really happens? So far the only thing I see is, that almost all of the computer memory is in use.

Total: 3735404

Used: 3611440

Free: 123964

Shared: 0

Buffers: 167980

Cached: 3116460

[Influencer]

On my phone so I can't see if you've attached a syslog, if not please do so.

 

The cached ram isn't an issue, it doesn't matter if you have 1 GB or 8 GB linux will use it all for cache unless needed.

 

Do you have cache_dirs enabled? If so try disabling it for a day. I use to get segfaults every day before I played with the settings for cache_dirs.

 

When you say you disabled all add-ons you didn't need, which are still running? It could be one of those causing the issue. From the list you provided, plex seems to be the likely suspect. Where is the install directory for plex located?

[petjek]

On my phone so I can't see if you've attached a syslog, if not please do so.

Here comes syslog: http://drpln.gs/Pe2jbq

 

The cached ram isn't an issue, it doesn't matter if you have 1 GB or 8 GB linux will use it all for cache unless needed.

Are you sure? I did a restart yesterday and for the first 30 mins the ram wasn't cached. See Screenshot.

 

Do you have cache_dirs enabled? If so try disabling it for a day. I use to get segfaults every day before I played with the settings for cache_dirs.

What is cache_dirs?

 

When you say you disabled all add-ons you didn't need, which are still running? It could be one of those causing the issue. From the list you provided, plex seems to be the likely suspect. Where is the install directory for plex located?

Currently there are SimpleFeatures, unMenu, Plex, Shell-in-a-box, pyLoad and dependencies installed. Deactivated iStat, SimpleFeatures iTunesServer, DS_Store cleanup, proftpd.

Plex is installed to... well, I don't know! To start/stop Plex I do use /etc/rc.d/rc.plexmediaserver start/stop so probably this is where it is installed? All I did to install Plex was dropping the file to the packages-folder, restart the server and set up the directories.

 

As I can see today everything is still running well. Management console is available even though I re-activated SimpleFeatures yesterday. But that happened before. Within a few hors it may look different.

 

petjek

 

[dgaschk]

The cached RAM is definitely not an issue. It is used as needed, if available, for caching disk reads. It is made available to process as needed and process release RAM making it free again (hopefully). To understand the graph take a graduate level Operating Systems class and then install a kernel debugger on the system.

 

See my sig for info on removing add-ons.

[lainie]

To understand how Linux uses memory, visit http://www.linuxatemyram.com/.

[petjek]

Uh, ok. I see. Memory is use is ok. Any other ideas? Somebody? Please! 

[JonathanM]

Uh, ok. I see. Memory is use is ok. Any other ideas? Somebody? Please!

Are you still running any add ons, or is it crashing with a bone stock unraid?