Weird error showing in my SYSLOG Help.

[kciaccio]

Can someone tell me what this error means. Array seems to work fine. Replaced a drive a few days ago.

 

Dec 25 11:56:00 Tower kernel: Swap cache stats: add 0, delete 0, find 0/0

Dec 25 11:56:00 Tower kernel: Free swap = 0kB

Dec 25 11:56:00 Tower kernel: Total swap = 0kB

Dec 25 11:56:00 Tower kernel: 523952 pages RAM

Dec 25 11:56:00 Tower kernel: 296114 pages HighMem

Dec 25 11:56:00 Tower kernel: 5549 pages reserved

Dec 25 11:56:00 Tower kernel: 41262 pages shared

Dec 25 11:56:00 Tower kernel: 473598 pages non-shared

Dec 25 11:56:00 Tower kernel: Out of memory: kill process 8664 (sh) score 32802 or a child

Dec 25 11:56:00 Tower kernel: Killed process 8664 (sh)

Dec 25 11:56:06 Tower crond[9045]: unable to exec /usr/sbin/sendmail -t, user -oem, output to sink null

Dec 25 12:00:54 Tower kernel: istatd invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0

Dec 25 12:00:54 Tower kernel: Pid: 3791, comm: istatd Not tainted 2.6.32.9-unRAID #2

Dec 25 12:00:54 Tower kernel: Call Trace:

Dec 25 12:00:54 Tower kernel: [] oom_kill_process+0x59/0x1cd

Dec 25 12:00:54 Tower kernel: [] __out_of_memory+0xef/0x102

Dec 25 12:00:54 Tower kernel: [] out_of_memory+0x5e/0x83

Dec 25 12:00:54 Tower kernel: [] __alloc_pages_nodemask+0x375/0x42f

Dec 25 12:00:54 Tower kernel: [] handle_mm_fault+0x254/0x8f1

Dec 25 12:00:54 Tower kernel: [] ? arch_report_meminfo+0x2e/0x36

Dec 25 12:00:54 Tower kernel: [] ? meminfo_proc_show+0x388/0x394

Dec 25 12:00:54 Tower kernel: [] do_page_fault+0x17c/0x1e4

Dec 25 12:00:54 Tower kernel: [] ? do_page_fault+0x0/0x1e4

Dec 25 12:00:54 Tower kernel: [] error_code+0x66/0x6c

Dec 25 12:00:54 Tower kernel: [] ? do_page_fault+0x0/0x1e4

Dec 25 12:00:54 Tower kernel: [] ? __copy_to_user_ll+0xc6/0xd2

Dec 25 12:00:54 Tower kernel: [] copy_to_user+0x33/0x3c

Dec 25 12:00:54 Tower kernel: [] seq_read+0x297/0x30e

Dec 25 12:00:54 Tower kernel: [] ? seq_read+0x0/0x30e

Dec 25 12:00:54 Tower kernel: [] proc_reg_read+0x56/0x6a

Dec 25 12:00:54 Tower kernel: [] ? proc_reg_read+0x0/0x6a

Dec 25 12:00:54 Tower kernel: [] vfs_read+0x8a/0x114

Dec 25 12:00:54 Tower kernel: [] sys_read+0x3b/0x60

Dec 25 12:00:54 Tower kernel: [] syscall_call+0x7/0xb

Dec 25 12:00:54 Tower kernel: Mem-Info:

Dec 25 12:00:54 Tower kernel: DMA per-cpu:

Dec 25 12:00:54 Tower kernel: CPU 0: hi: 0, btch: 1 usd: 0

Dec 25 12:00:54 Tower kernel: Normal per-cpu:

Dec 25 12:00:54 Tower kernel: CPU 0: hi: 186, btch: 31 usd: 163

Dec 25 12:00:54 Tower kernel: HighMem per-cpu:

Dec 25 12:00:54 Tower kernel: CPU 0: hi: 186, btch: 31 usd: 192

Dec 25 12:00:54 Tower kernel: active_anon:298862 inactive_anon:65446 isolated_anon:0

Dec 25 12:00:54 Tower kernel: active_file:13523 inactive_file:13418 isolated_file:0

Dec 25 12:00:54 Tower kernel: unevictable:87046 dirty:0 writeback:0 unstable:0

Dec 25 12:00:54 Tower kernel: free:12275 slab_reclaimable:4271 slab_unreclaimable:2348

Dec 25 12:00:54 Tower kernel: mapped:3513 shmem:44 pagetables:991 bounce:0

Dec 25 12:00:54 Tower kernel: DMA free:8116kB min:64kB low:80kB high:96kB active_anon:3900kB inactive_anon:3896kB active_file:0kB inactive_file:0kB unevictable:12kB isolated(anon):0kB isolated(file):0kB present:15852kB mlocked:0kB dirty:0kB writeback:0kB mapped:8kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:20kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:352 all_unreclaimable? yes

Dec 25 12:00:54 Tower kernel: lowmem_reserve[]: 0 867 2014 2014

Dec 25 12:00:54 Tower kernel: Normal free:40488kB min:3732kB low:4664kB high:5596kB active_anon:575048kB inactive_anon:52324kB active_file:54092kB inactive_file:53656kB unevictable:916kB isolated(anon):0kB isolated(file):0kB present:887976kB mlocked:0kB dirty:0kB writeback:0kB mapped:624kB shmem:36kB slab_reclaimable:17084kB slab_unreclaimable:9392kB kernel_stack:928kB pagetables:1316kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:100577 all_unreclaimable? no

Dec 25 12:00:54 Tower kernel: lowmem_reserve[]: 0 0 9181 9181

Dec 25 12:00:54 Tower kernel: HighMem free:496kB min:512kB low:1744kB high:2980kB active_anon:616500kB inactive_anon:205564kB active_file:0kB inactive_file:16kB unevictable:347256kB isolated(anon):0kB isolated(file):0kB present:1175200kB mlocked:0kB dirty:0kB writeback:0kB mapped:13420kB shmem:140kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:2628kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:4 all_unreclaimable? no

Dec 25 12:00:54 Tower kernel: lowmem_reserve[]: 0 0 0 0

Dec 25 12:00:54 Tower kernel: DMA: 1*4kB 2*8kB 0*16kB 1*32kB 2*64kB 2*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 8116kB

Dec 25 12:00:54 Tower kernel: Normal: 840*4kB 695*8kB 361*16kB 100*32kB 69*64kB 32*128kB 15*256kB 4*512kB 4*1024kB 0*2048kB 1*4096kB = 40488kB

Dec 25 12:00:54 Tower kernel: HighMem: 124*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 496kB

Dec 25 12:00:54 Tower kernel: 114026 total pagecache pages

Dec 25 12:00:54 Tower kernel: 0 pages in swap cache

Dec 25 12:00:54 Tower kernel: Swap cache stats: add 0, delete 0, find 0/0

Dec 25 12:00:54 Tower kernel: Free swap = 0kB

Dec 25 12:00:54 Tower kernel: Total swap = 0kB

Dec 25 12:00:54 Tower kernel: 523952 pages RAM

Dec 25 12:00:54 Tower kernel: 296114 pages HighMem

Dec 25 12:00:54 Tower kernel: 5549 pages reserved

Dec 25 12:00:54 Tower kernel: 41263 pages shared

Dec 25 12:00:54 Tower kernel: 473459 pages non-shared

Dec 25 12:00:54 Tower kernel: Out of memory: kill process 9047 (sh) score 32802 or a child

Dec 25 12:00:54 Tower kernel: Killed process 9047 (sh)

Dec 25 12:00:56 Tower crond[9428]: unable to exec /usr/sbin/sendmail -t, user -oem, output to sink null

[RobJ]

You OOM'd, which means you ran out of memory.  Even if it appears to be running fine, I would reboot.

 

You can see the memory manager reclaiming memory, and ends up with a fair amount by the last memory status dump.  But at least one process was killed, with unknown results.

 

You are running UnRAID v4.5.4, which is really old, and not even the best of that series, v4.5.6 was.  It's time to upgrade to v4.7, then to v5.0.6.

[kciaccio]

Thanks for all the help.

 

I am building a new UNRAID server so I will probably upgrade after the migration. Thanks for letting me know what order to upgrade in. I was not sure. I knew I had to do it in two parts but was not sure which versions.