So I was running rc8a for the longest time, no issues. Didn't really have many packages/plugins. I had mySql, uumenu, minidlna and that's pretty much it (cache_dirs too).

 

I just upgraded to 5.0 Final and everything seemed to be ok. Then I realized that whenever the server would do a parity check, I'd run out of memory.

 

I disabled all my plugins/packages and it's still doing it.

 

I can see that OOM is being called which means memory is running out and it would start to kill samba, telnet and sometimes uuMenu as well...

 

attached is the last few snippets of syslog when it happens, nothing really obvious:

 

Sep 18 04:47:01 Tower crond[1206]: ignoring /var/spool/cron/crontabs/root- (non-existent user)

Sep 18 04:47:19 Tower sSMTP[9024]: Creating SSL connection to host

Sep 18 04:47:19 Tower sSMTP[9024]: SSL connection using RC4-SHA

Sep 18 04:47:23 Tower sSMTP[9024]: Sent mail for root@[email protected] (221 2.0.0 closing connection y9sm3799500qaj.9 - gsmtp) uid=0 username=root outbytes=1870

Sep 18 04:50:05 Tower kernel: shfs invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0

Sep 18 04:50:06 Tower kernel: Pid: 14331, comm: shfs Tainted: G O 3.9.6p-unRAID #23

 

I do have a swap file still set up, and have now installed the iStat package so able to remotely monitor resources from my iPad and I see a huge amount of pagein requests for the swap file... which is probably keeping the system up, ie. parity check is still running but everything else is pretty much stopped.

 

 

Anyone have any thoughts ?

 

I never had this issue previously with 5.0 rc8a...

 

 

 

Attach a syslog and  the output of "free -lm" when it happens.

I'll try to get a log but it's hard because when it happens the telnet server gets killed and Samba gets killed too... the last few snippets of syslog that I was able to get showed I got a status parity check mail at 447am, I started it around 11... then it started killing processes...

console was still responsive so I copied the syslog and did a free -lm and copied it to flash.

 

The syslog is pretty empty... but i noticed on the console screen that many processes were killed... probably due to OOM...

syslog-20130918-045006.txt

mem.txt

Try with no add-ons.

i pretty much had no add ons except for istat which i installed as a debugging tool.

 

i disabled the swap space and tried running parity check and i didnt encounter the problem. i'll re-enable the swap space and run parity chk to confirm it was related to the swwap space.

so i installed the swapfile plug in for 5.0 and enabled it. And this morning, lo and behold, the server was exhibiting the same problems.

 

Attached is the syslog and below is output of free -lm:

 

            total      used      free    shared    buffers    cached

Mem:          4017      3994        22          0        76      3709

Low:          846        835        11

High:        3170      3159        11

-/+ buffers/cache:        207      3809

Swap:        4095          8      4087

 

syslog.txt

now i'm not so sure what's going on... last night, i wasn't doing anything, swapfile was off...

 

and this morning unraid reported running out of memory so OOM killed a whole bunch of stuff, SAMBA was off, but luckily the server was still running so captured the following logs..

 

what do you think ?

 

results of free -lm :

 

            total      used      free    shared    buffers    cached

Mem:          4017      3993        23          0        64      3733

Low:          846        835        11

High:        3170      3157        12

-/+ buffers/cache:        195      3821

Swap:            0          0          0

 

this is really starting to get annoying, i love unRaid when it works, but its these boundry cases of unstability that I just don't have time to deal with 

syslog.txt

Run memtest for min 10hrs. Maybe a DIMM is dropping offline.

A while back rc-wise, it was suggested to add these lines to one's GO script:

pgrep -f "/usr/local/sbin/emhttp" | while read PID; do echo -1000 > /proc/$PID/oom_score_adj; done
pgrep -f "/usr/local/sbin/smbd" | while read PID; do echo -1000 > /proc/$PID/oom_score_adj; done

They are designed to prevent OOM from killing off emhttp and smb.

 

Now that we're in V5.0-final I thought perhaps it's time to stop using them because SURELY these services should be protected in the final release. But, recently I think I saw a reference to keeping them in; so for better or worse I'm using them. My system has been VERY stable; although not necessarily a result of them. 

 

You might consider adding them to your GO script, /boot/config/go file.

 

BTW, here's my output from "free -lm" command:

root@Tower:~# free -lm

            total      used      free    shared    buffers    cached

Mem:          4048      3594        453          0        85      3166

Low:          844        656        187

High:        3203      2938        265

-/+ buffers/cache:        342      3705

Swap:            0          0          0

 

This caused by an add-on. Try running without add-ons. Then add  back a few at a time until the incompatible one is found.

Study this line in your syslog:

Sep 21 04:41:09 Tower kernel: HighMem free:420kB min:512kB low:4064kB high:7616kB active_anon:40116kB inactive_anon:388kB active_file:0kB inactive_file:60kB unevictable:3194408kB isolated(anon):0kB isolated(file):0kB present:3246792kB managed:3246792kB mlocked:0kB dirty:0kB writeback:0kB mapped:11192kB shmem:636kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:1168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:49 all_unreclaimable? yes

 

You have 3.2GB of unevictable high memory.

unevictable:3194408kB

 

Out of a total of 3.2GB.

present:3246792kB

 

Probably something filling up your RAMdisk filesystem.

In fact, I would rather it kills emhttp and smb first -- at least I'd still be able to telnet into the server, grab a syslog, stop the array, and do some investigating, all of which I won't be able to do if the system just completely crashes.

Good Point.

while digging through my "go" script, I realized that I was manually installing Samba 3.6.8... when I had rc8 it came with 3.6.7 so I must have had a reason to use a newer version.  Now that 5.0 Final is installed, it comes with Samba 3.6.10 so I was actually forcing the use of an older version of Samba...

 

I've removed this, doing a parity check now... see if that solves the problem...

running "du / -ahxd1" gives me an invalid 'd' and invalid '1' option. Running without the d1 gives me 223M, which isn't all that much more than stock unRaid... certainly nothing to overly consume my memory I don't think...

 

running the grep command gives me:

 

unevictable: 229920 kB which matches the above more or less...

 

this is after I removed the 3.6.8 samba that I was manually including... I think that was the problem... to confirm, I'll re-enable the 3.6.8 in my go file and see if my problems come back...

 

 

actually I ran that a couple of hours ago, after the server had been up for at least a day and a half, and after I ran a parity check, and re-enabled my swap file... I'll keep monitoring it, now that I know the commands...

 

I'll check it probably later this week when I have more time and re-enable 3.6.8 and see what happens... appreciate your help!