theone Posted December 13, 2012 Author Share Posted December 13, 2012 Again everything has been running smoothly for the last 8 days. I will now enable WEB SERVER plugin (on port 1234) with a simple index.htm file and let it run for another period of ~7 days. Quote Link to comment
theone Posted December 17, 2012 Author Share Posted December 17, 2012 Been running for another 4 days (since my last post) without issues when I noticed that emhttp invoked oom-killer. The killed process was "VBoxHeadless" and not emhttp itself, so I can restart VBoxHeadless and continue monitoring the server (don't have to restart it because emhttp didn't crash). I don't know why it invoked the oom-killer and why it decided to kill VBoxHeadless. Will keep running to see if it happens again. Below is an excerpt from the syslog showing the problem: Dec 17 09:15:48 Tower kernel: emhttp invoked oom-killer: gfp_mask=0x84d0, order=0, oom_adj=-17, oom_score_adj=-1000 Dec 17 09:15:48 Tower kernel: Pid: 20034, comm: emhttp Tainted: G O 3.4.11-unRAID #1 Dec 17 09:15:48 Tower kernel: Call Trace: Dec 17 09:15:48 Tower kernel: [] T.619+0x56/0x159 Dec 17 09:15:48 Tower kernel: [] ? irq_exit+0x56/0x58 Dec 17 09:15:48 Tower kernel: [] ? ___ratelimit+0xaf/0xc4 Dec 17 09:15:48 Tower kernel: [] T.618+0x45/0x217 Dec 17 09:15:48 Tower kernel: [] ? has_ns_capability_noaudit+0x10/0x18 Dec 17 09:15:48 Tower kernel: [] out_of_memory+0x21c/0x29e Dec 17 09:15:48 Tower kernel: [] __alloc_pages_nodemask+0x40b/0x4a5 Dec 17 09:15:48 Tower kernel: [] __get_free_pages+0xf/0x21 Dec 17 09:15:48 Tower kernel: [] pgd_alloc+0x18/0xd4 Dec 17 09:15:48 Tower kernel: [] mm_init+0xad/0xd1 Dec 17 09:15:48 Tower kernel: [] mm_alloc+0x33/0x37 Dec 17 09:15:48 Tower kernel: [] bprm_mm_init+0x12/0x65 Dec 17 09:15:48 Tower kernel: [] do_execve_common+0x190/0x317 Dec 17 09:15:48 Tower kernel: [] do_execve+0xb/0xe Dec 17 09:15:48 Tower kernel: [] sys_execve+0x2c/0x54 Dec 17 09:15:48 Tower kernel: [] ptregs_execve+0x12/0x18 Dec 17 09:15:48 Tower kernel: [] ? syscall_call+0x7/0xb Dec 17 09:15:48 Tower kernel: [] ? alloc_node_mem_map+0x70/0x8a Dec 17 09:15:48 Tower kernel: Mem-Info: Dec 17 09:15:48 Tower kernel: DMA per-cpu: Dec 17 09:15:48 Tower kernel: CPU 0: hi: 0, btch: 1 usd: 0 Dec 17 09:15:48 Tower kernel: CPU 1: hi: 0, btch: 1 usd: 0 Dec 17 09:15:48 Tower kernel: CPU 2: hi: 0, btch: 1 usd: 0 Dec 17 09:15:48 Tower kernel: CPU 3: hi: 0, btch: 1 usd: 0 Dec 17 09:15:48 Tower kernel: Normal per-cpu: Dec 17 09:15:48 Tower kernel: CPU 0: hi: 186, btch: 31 usd: 0 Dec 17 09:15:48 Tower kernel: CPU 1: hi: 186, btch: 31 usd: 0 Dec 17 09:15:48 Tower kernel: CPU 2: hi: 186, btch: 31 usd: 0 Dec 17 09:15:48 Tower kernel: CPU 3: hi: 186, btch: 31 usd: 0 Dec 17 09:15:48 Tower kernel: HighMem per-cpu: Dec 17 09:15:48 Tower kernel: CPU 0: hi: 186, btch: 31 usd: 0 Dec 17 09:15:48 Tower kernel: CPU 1: hi: 186, btch: 31 usd: 0 Dec 17 09:15:48 Tower kernel: CPU 2: hi: 186, btch: 31 usd: 0 Dec 17 09:15:48 Tower kernel: CPU 3: hi: 186, btch: 31 usd: 0 Dec 17 09:15:48 Tower kernel: active_anon:30970 inactive_anon:6452 isolated_anon:0 Dec 17 09:15:48 Tower kernel: active_file:144728 inactive_file:142520 isolated_file:96 Dec 17 09:15:48 Tower kernel: unevictable:233473 dirty:0 writeback:0 unstable:0 Dec 17 09:15:48 Tower kernel: free:121786 slab_reclaimable:7842 slab_unreclaimable:121203 Dec 17 09:15:48 Tower kernel: mapped:228767 shmem:0 pagetables:1135 bounce:0 Dec 17 09:15:48 Tower kernel: DMA free:5756kB min:2280kB low:2848kB high:3420kB active_anon:0kB inactive_anon:0kB active_file:1444kB inactive_file:1376kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15780kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:476kB slab_unreclaimable:6804kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:5814 all_unreclaimable? yes Dec 17 09:15:48 Tower kernel: lowmem_reserve[]: 0 869 4039 4039 Dec 17 09:15:48 Tower kernel: Normal free:128788kB min:128788kB low:160984kB high:193180kB active_anon:0kB inactive_anon:0kB active_file:14608kB inactive_file:14248kB unevictable:1636kB isolated(anon):0kB isolated(file):128kB present:890008kB mlocked:0kB dirty:0kB writeback:0kB mapped:149548kB shmem:0kB slab_reclaimable:30892kB slab_unreclaimable:478008kB kernel_stack:1944kB pagetables:424kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:69027 all_unreclaimable? yes Dec 17 09:15:48 Tower kernel: lowmem_reserve[]: 0 0 25363 25363 Dec 17 09:15:48 Tower kernel: HighMem free:352600kB min:512kB low:117960kB high:235408kB active_anon:123880kB inactive_anon:25808kB active_file:562860kB inactive_file:554456kB unevictable:932256kB isolated(anon):0kB isolated(file):0kB present:3246564kB mlocked:72kB dirty:0kB writeback:0kB mapped:765520kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:4116kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Dec 17 09:15:48 Tower kernel: lowmem_reserve[]: 0 0 0 0 Dec 17 09:15:48 Tower kernel: DMA: 165*4kB 3*8kB 1*16kB 2*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 5756kB Dec 17 09:15:48 Tower kernel: Normal: 724*4kB 1697*8kB 1153*16kB 599*32kB 404*64kB 209*128kB 55*256kB 0*512kB 0*1024kB 0*2048kB 2*4096kB = 128968kB Dec 17 09:15:48 Tower kernel: HighMem: 133*4kB 45*8kB 35*16kB 29*32kB 1532*64kB 157*128kB 541*256kB 125*512kB 13*1024kB 4*2048kB 2*4096kB = 352716kB Dec 17 09:15:48 Tower kernel: 528881 total pagecache pages Dec 17 09:15:48 Tower kernel: 8142 pages in swap cache Dec 17 09:15:48 Tower kernel: Swap cache stats: add 469055, delete 460913, find 4059368/4088306 Dec 17 09:15:48 Tower kernel: Free swap = 1846116kB Dec 17 09:15:48 Tower kernel: Total swap = 2097148kB Dec 17 09:15:48 Tower kernel: 1310704 pages RAM Dec 17 09:15:48 Tower kernel: 1082370 pages HighMem Dec 17 09:15:48 Tower kernel: 490919 pages reserved Dec 17 09:15:48 Tower kernel: 179396 pages shared Dec 17 09:15:48 Tower kernel: 533736 pages non-shared Dec 17 09:15:48 Tower kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name Dec 17 09:15:48 Tower kernel: [ 701] 0 701 565 124 1 -17 -1000 udevd Dec 17 09:15:48 Tower kernel: [ 986] 0 986 477 131 3 0 0 syslogd Dec 17 09:15:48 Tower kernel: [ 990] 0 990 464 80 3 0 0 klogd Dec 17 09:15:48 Tower kernel: [ 1090] 1 1090 490 100 1 0 0 rpc.portmap Dec 17 09:15:48 Tower kernel: [ 1094] 0 1094 527 165 3 0 0 rpc.statd Dec 17 09:15:48 Tower kernel: [ 1104] 0 1104 475 121 0 0 0 inetd Dec 17 09:15:48 Tower kernel: [ 1111] 0 1111 1123 328 3 0 0 ntpd Dec 17 09:15:48 Tower kernel: [ 1118] 0 1118 466 134 2 0 0 acpid Dec 17 09:15:48 Tower kernel: [ 1128] 81 1128 619 116 3 0 0 dbus-daemon Dec 17 09:15:48 Tower kernel: [ 1133] 0 1133 477 143 1 0 0 crond Dec 17 09:15:48 Tower kernel: [ 1135] 0 1135 475 54 2 0 0 atd Dec 17 09:15:48 Tower kernel: [ 3936] 0 3936 14672 750 1 0 0 hamachid Dec 17 09:15:48 Tower kernel: [16148] 0 16148 3230 366 1 -17 -1000 emhttp Dec 17 09:15:48 Tower kernel: [16180] 0 16180 2134 352 1 0 0 nmbd Dec 17 09:15:48 Tower kernel: [16182] 0 16182 4015 829 0 0 0 smbd Dec 17 09:15:48 Tower kernel: [16197] 0 16197 4016 302 2 0 0 smbd Dec 17 09:15:48 Tower kernel: [16351] 0 16351 2299 554 0 0 0 VBoxXPCOMIPCD Dec 17 09:15:48 Tower kernel: [16362] 0 16362 5523 1317 1 0 0 VBoxSVC Dec 17 09:15:48 Tower kernel: [16403] 0 16403 32971 455 2 -17 -1000 shfs Dec 17 09:15:48 Tower kernel: [16464] 99 16464 26516 8977 3 -17 -1000 slimserver.pl Dec 17 09:15:48 Tower kernel: [16522] 0 16522 7598 990 2 0 0 vboxwebsrv Dec 17 09:15:48 Tower kernel: [16533] 0 16533 918 468 0 0 0 unraid_addon_co Dec 17 09:15:48 Tower kernel: [16579] 0 16579 464 118 1 0 0 agetty Dec 17 09:15:48 Tower kernel: [16580] 0 16580 464 118 1 0 0 agetty Dec 17 09:15:48 Tower kernel: [16581] 0 16581 464 118 1 0 0 agetty Dec 17 09:15:48 Tower kernel: [16582] 0 16582 464 118 3 0 0 agetty Dec 17 09:15:48 Tower kernel: [16583] 0 16583 464 118 2 0 0 agetty Dec 17 09:15:48 Tower kernel: [16584] 0 16584 464 118 0 0 0 agetty Dec 17 09:15:48 Tower kernel: [16586] 0 16586 564 112 3 -17 -1000 udevd Dec 17 09:15:48 Tower kernel: [16587] 0 16587 564 109 0 -17 -1000 udevd Dec 17 09:15:48 Tower kernel: [16596] 0 16596 60888 2990 0 -17 -1000 Plex Media Serv Dec 17 09:15:48 Tower kernel: [16649] 99 16649 108128 16730 2 -17 -1000 java Dec 17 09:15:48 Tower kernel: [16701] 0 16701 238121 221094 1 0 0 VBoxHeadless Dec 17 09:15:48 Tower kernel: [16792] 0 16792 89319 1673 1 -17 -1000 Plex DLNA Serve Dec 17 09:15:48 Tower kernel: [16927] 99 16927 23806 4640 0 -17 -1000 utserver Dec 17 09:15:48 Tower kernel: [21611] 0 21611 688 330 0 -17 -1000 unraid_notify Dec 17 09:15:48 Tower kernel: [ 2628] 0 2628 6125 934 3 0 0 smbd Dec 17 09:15:48 Tower kernel: [ 4799] 0 4799 4964 1142 0 0 0 smbd Dec 17 09:15:48 Tower kernel: [ 2395] 0 2395 48484 5797 0 -17 -1000 python Dec 17 09:15:48 Tower kernel: [ 7960] 0 7960 1128 247 3 -17 -1000 lighttpd Dec 17 09:15:48 Tower kernel: [ 7961] 0 7961 14591 1228 1 -17 -1000 php-cgi Dec 17 09:15:48 Tower kernel: [ 7968] 0 7968 14591 1228 2 -17 -1000 php-cgi Dec 17 09:15:48 Tower kernel: [ 7969] 0 7969 14591 1228 1 -17 -1000 php-cgi Dec 17 09:15:48 Tower kernel: [ 7970] 0 7970 14591 96 1 -17 -1000 php-cgi Dec 17 09:15:48 Tower kernel: [ 7973] 0 7973 14591 1228 0 -17 -1000 php-cgi Dec 17 09:15:48 Tower kernel: [ 7974] 0 7974 14591 95 2 -17 -1000 php-cgi Dec 17 09:15:48 Tower kernel: [ 7975] 0 7975 14591 96 1 -17 -1000 php-cgi Dec 17 09:15:48 Tower kernel: [ 7976] 0 7976 14591 96 1 -17 -1000 php-cgi Dec 17 09:15:48 Tower kernel: [18415] 0 18415 463 61 1 -17 -1000 sleep Dec 17 09:15:48 Tower kernel: [19997] 0 19997 4015 275 3 0 0 smbd Dec 17 09:15:48 Tower kernel: [20026] 0 20026 463 62 1 0 0 sleep Dec 17 09:15:48 Tower kernel: [20033] 0 20033 3230 74 2 -17 -1000 emhttp Dec 17 09:15:48 Tower kernel: [20034] 0 20034 3230 74 1 -17 -1000 emhttp Dec 17 09:15:48 Tower kernel: Out of memory: Kill process 16701 (VBoxHeadless) score 115 or sacrifice child Dec 17 09:15:48 Tower kernel: Killed process 16701 (VBoxHeadless) total-vm:952484kB, anon-rss:10044kB, file-rss:874332kB Dec 17 09:15:48 Tower kernel: VBoxDrvLinuxIOCtl: copy_to_user(0xaf0e81d0,,0x18); uCmd=0xc0305618! Dec 17 09:15:48 Tower kernel: VBoxDrvLinuxIOCtl: copy_to_user(0xaf2681d0,,0x18); uCmd=0xc0305618! Dec 17 09:15:48 Tower kernel: device eth1 left promiscuous mode Dec 17 09:15:48 Tower kernel: vboxnetflt: dropped 0 out of 0 packets Dec 17 09:15:48 Tower kernel: device eth0 left promiscuous mode Dec 17 09:15:48 Tower kernel: vboxnetflt: dropped 13576960 out of 92450222 packets Quote Link to comment
Joe L. Posted December 17, 2012 Share Posted December 17, 2012 Been running for another 4 days (since my last post) without issues when I noticed that emhttp invoked oom-killer. The killed process was "VBoxHeadless" and not emhttp itself, so I can restart VBoxHeadless and continue monitoring the server (don't have to restart it because emhttp didn't crash). I don't know why it invoked the oom-killer and why it decided to kill VBoxHeadless. Will keep running to see if it happens again. Below is an excerpt from the syslog showing the problem: Dec 17 09:15:48 Tower kernel: emhttp invoked oom-killer: gfp_mask=0x84d0, order=0, oom_adj=-17, oom_score_adj=-1000 Dec 17 09:15:48 Tower kernel: Pid: 20034, comm: emhttp Tainted: G O 3.4.11-unRAID #1 Dec 17 09:15:48 Tower kernel: Call Trace: Dec 17 09:15:48 Tower kernel: [] T.619+0x56/0x159 Dec 17 09:15:48 Tower kernel: [] ? irq_exit+0x56/0x58 Dec 17 09:15:48 Tower kernel: [] ? ___ratelimit+0xaf/0xc4 Dec 17 09:15:48 Tower kernel: [] T.618+0x45/0x217 Dec 17 09:15:48 Tower kernel: [] ? has_ns_capability_noaudit+0x10/0x18 Dec 17 09:15:48 Tower kernel: [] out_of_memory+0x21c/0x29e Dec 17 09:15:48 Tower kernel: [] __alloc_pages_nodemask+0x40b/0x4a5 Dec 17 09:15:48 Tower kernel: [] __get_free_pages+0xf/0x21 Dec 17 09:15:48 Tower kernel: [] pgd_alloc+0x18/0xd4 Dec 17 09:15:48 Tower kernel: [] mm_init+0xad/0xd1 Dec 17 09:15:48 Tower kernel: [] mm_alloc+0x33/0x37 Dec 17 09:15:48 Tower kernel: [] bprm_mm_init+0x12/0x65 Dec 17 09:15:48 Tower kernel: [] do_execve_common+0x190/0x317 Dec 17 09:15:48 Tower kernel: [] do_execve+0xb/0xe Dec 17 09:15:48 Tower kernel: [] sys_execve+0x2c/0x54 Dec 17 09:15:48 Tower kernel: [] ptregs_execve+0x12/0x18 Dec 17 09:15:48 Tower kernel: [] ? syscall_call+0x7/0xb Dec 17 09:15:48 Tower kernel: [] ? alloc_node_mem_map+0x70/0x8a Dec 17 09:15:48 Tower kernel: Mem-Info: Dec 17 09:15:48 Tower kernel: DMA per-cpu: Dec 17 09:15:48 Tower kernel: CPU 0: hi: 0, btch: 1 usd: 0 Dec 17 09:15:48 Tower kernel: CPU 1: hi: 0, btch: 1 usd: 0 Dec 17 09:15:48 Tower kernel: CPU 2: hi: 0, btch: 1 usd: 0 Dec 17 09:15:48 Tower kernel: CPU 3: hi: 0, btch: 1 usd: 0 Dec 17 09:15:48 Tower kernel: Normal per-cpu: Dec 17 09:15:48 Tower kernel: CPU 0: hi: 186, btch: 31 usd: 0 Dec 17 09:15:48 Tower kernel: CPU 1: hi: 186, btch: 31 usd: 0 Dec 17 09:15:48 Tower kernel: CPU 2: hi: 186, btch: 31 usd: 0 Dec 17 09:15:48 Tower kernel: CPU 3: hi: 186, btch: 31 usd: 0 Dec 17 09:15:48 Tower kernel: HighMem per-cpu: Dec 17 09:15:48 Tower kernel: CPU 0: hi: 186, btch: 31 usd: 0 Dec 17 09:15:48 Tower kernel: CPU 1: hi: 186, btch: 31 usd: 0 Dec 17 09:15:48 Tower kernel: CPU 2: hi: 186, btch: 31 usd: 0 Dec 17 09:15:48 Tower kernel: CPU 3: hi: 186, btch: 31 usd: 0 Dec 17 09:15:48 Tower kernel: active_anon:30970 inactive_anon:6452 isolated_anon:0 Dec 17 09:15:48 Tower kernel: active_file:144728 inactive_file:142520 isolated_file:96 Dec 17 09:15:48 Tower kernel: unevictable:233473 dirty:0 writeback:0 unstable:0 Dec 17 09:15:48 Tower kernel: free:121786 slab_reclaimable:7842 slab_unreclaimable:121203 Dec 17 09:15:48 Tower kernel: mapped:228767 shmem:0 pagetables:1135 bounce:0 Dec 17 09:15:48 Tower kernel: DMA free:5756kB min:2280kB low:2848kB high:3420kB active_anon:0kB inactive_anon:0kB active_file:1444kB inactive_file:1376kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15780kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:476kB slab_unreclaimable:6804kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:5814 all_unreclaimable? yes Dec 17 09:15:48 Tower kernel: lowmem_reserve[]: 0 869 4039 4039 Dec 17 09:15:48 Tower kernel: Normal free:128788kB min:128788kB low:160984kB high:193180kB active_anon:0kB inactive_anon:0kB active_file:14608kB inactive_file:14248kB unevictable:1636kB isolated(anon):0kB isolated(file):128kB present:890008kB mlocked:0kB dirty:0kB writeback:0kB mapped:149548kB shmem:0kB slab_reclaimable:30892kB slab_unreclaimable:478008kB kernel_stack:1944kB pagetables:424kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:69027 all_unreclaimable? yes Dec 17 09:15:48 Tower kernel: lowmem_reserve[]: 0 0 25363 25363 Dec 17 09:15:48 Tower kernel: HighMem free:352600kB min:512kB low:117960kB high:235408kB active_anon:123880kB inactive_anon:25808kB active_file:562860kB inactive_file:554456kB unevictable:932256kB isolated(anon):0kB isolated(file):0kB present:3246564kB mlocked:72kB dirty:0kB writeback:0kB mapped:765520kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:4116kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Dec 17 09:15:48 Tower kernel: lowmem_reserve[]: 0 0 0 0 Dec 17 09:15:48 Tower kernel: DMA: 165*4kB 3*8kB 1*16kB 2*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 5756kB Dec 17 09:15:48 Tower kernel: Normal: 724*4kB 1697*8kB 1153*16kB 599*32kB 404*64kB 209*128kB 55*256kB 0*512kB 0*1024kB 0*2048kB 2*4096kB = 128968kB Dec 17 09:15:48 Tower kernel: HighMem: 133*4kB 45*8kB 35*16kB 29*32kB 1532*64kB 157*128kB 541*256kB 125*512kB 13*1024kB 4*2048kB 2*4096kB = 352716kB Dec 17 09:15:48 Tower kernel: 528881 total pagecache pages Dec 17 09:15:48 Tower kernel: 8142 pages in swap cache Dec 17 09:15:48 Tower kernel: Swap cache stats: add 469055, delete 460913, find 4059368/4088306 Dec 17 09:15:48 Tower kernel: Free swap = 1846116kB Dec 17 09:15:48 Tower kernel: Total swap = 2097148kB Dec 17 09:15:48 Tower kernel: 1310704 pages RAM Dec 17 09:15:48 Tower kernel: 1082370 pages HighMem Dec 17 09:15:48 Tower kernel: 490919 pages reserved Dec 17 09:15:48 Tower kernel: 179396 pages shared Dec 17 09:15:48 Tower kernel: 533736 pages non-shared Dec 17 09:15:48 Tower kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name Dec 17 09:15:48 Tower kernel: [ 701] 0 701 565 124 1 -17 -1000 udevd Dec 17 09:15:48 Tower kernel: [ 986] 0 986 477 131 3 0 0 syslogd Dec 17 09:15:48 Tower kernel: [ 990] 0 990 464 80 3 0 0 klogd Dec 17 09:15:48 Tower kernel: [ 1090] 1 1090 490 100 1 0 0 rpc.portmap Dec 17 09:15:48 Tower kernel: [ 1094] 0 1094 527 165 3 0 0 rpc.statd Dec 17 09:15:48 Tower kernel: [ 1104] 0 1104 475 121 0 0 0 inetd Dec 17 09:15:48 Tower kernel: [ 1111] 0 1111 1123 328 3 0 0 ntpd Dec 17 09:15:48 Tower kernel: [ 1118] 0 1118 466 134 2 0 0 acpid Dec 17 09:15:48 Tower kernel: [ 1128] 81 1128 619 116 3 0 0 dbus-daemon Dec 17 09:15:48 Tower kernel: [ 1133] 0 1133 477 143 1 0 0 crond Dec 17 09:15:48 Tower kernel: [ 1135] 0 1135 475 54 2 0 0 atd Dec 17 09:15:48 Tower kernel: [ 3936] 0 3936 14672 750 1 0 0 hamachid Dec 17 09:15:48 Tower kernel: [16148] 0 16148 3230 366 1 -17 -1000 emhttp Dec 17 09:15:48 Tower kernel: [16180] 0 16180 2134 352 1 0 0 nmbd Dec 17 09:15:48 Tower kernel: [16182] 0 16182 4015 829 0 0 0 smbd Dec 17 09:15:48 Tower kernel: [16197] 0 16197 4016 302 2 0 0 smbd Dec 17 09:15:48 Tower kernel: [16351] 0 16351 2299 554 0 0 0 VBoxXPCOMIPCD Dec 17 09:15:48 Tower kernel: [16362] 0 16362 5523 1317 1 0 0 VBoxSVC Dec 17 09:15:48 Tower kernel: [16403] 0 16403 32971 455 2 -17 -1000 shfs Dec 17 09:15:48 Tower kernel: [16464] 99 16464 26516 8977 3 -17 -1000 slimserver.pl Dec 17 09:15:48 Tower kernel: [16522] 0 16522 7598 990 2 0 0 vboxwebsrv Dec 17 09:15:48 Tower kernel: [16533] 0 16533 918 468 0 0 0 unraid_addon_co Dec 17 09:15:48 Tower kernel: [16579] 0 16579 464 118 1 0 0 agetty Dec 17 09:15:48 Tower kernel: [16580] 0 16580 464 118 1 0 0 agetty Dec 17 09:15:48 Tower kernel: [16581] 0 16581 464 118 1 0 0 agetty Dec 17 09:15:48 Tower kernel: [16582] 0 16582 464 118 3 0 0 agetty Dec 17 09:15:48 Tower kernel: [16583] 0 16583 464 118 2 0 0 agetty Dec 17 09:15:48 Tower kernel: [16584] 0 16584 464 118 0 0 0 agetty Dec 17 09:15:48 Tower kernel: [16586] 0 16586 564 112 3 -17 -1000 udevd Dec 17 09:15:48 Tower kernel: [16587] 0 16587 564 109 0 -17 -1000 udevd Dec 17 09:15:48 Tower kernel: [16596] 0 16596 60888 2990 0 -17 -1000 Plex Media Serv Dec 17 09:15:48 Tower kernel: [16649] 99 16649 108128 16730 2 -17 -1000 java Dec 17 09:15:48 Tower kernel: [16701] 0 16701 238121 221094 1 0 0 VBoxHeadless Dec 17 09:15:48 Tower kernel: [16792] 0 16792 89319 1673 1 -17 -1000 Plex DLNA Serve Dec 17 09:15:48 Tower kernel: [16927] 99 16927 23806 4640 0 -17 -1000 utserver Dec 17 09:15:48 Tower kernel: [21611] 0 21611 688 330 0 -17 -1000 unraid_notify Dec 17 09:15:48 Tower kernel: [ 2628] 0 2628 6125 934 3 0 0 smbd Dec 17 09:15:48 Tower kernel: [ 4799] 0 4799 4964 1142 0 0 0 smbd Dec 17 09:15:48 Tower kernel: [ 2395] 0 2395 48484 5797 0 -17 -1000 python Dec 17 09:15:48 Tower kernel: [ 7960] 0 7960 1128 247 3 -17 -1000 lighttpd Dec 17 09:15:48 Tower kernel: [ 7961] 0 7961 14591 1228 1 -17 -1000 php-cgi Dec 17 09:15:48 Tower kernel: [ 7968] 0 7968 14591 1228 2 -17 -1000 php-cgi Dec 17 09:15:48 Tower kernel: [ 7969] 0 7969 14591 1228 1 -17 -1000 php-cgi Dec 17 09:15:48 Tower kernel: [ 7970] 0 7970 14591 96 1 -17 -1000 php-cgi Dec 17 09:15:48 Tower kernel: [ 7973] 0 7973 14591 1228 0 -17 -1000 php-cgi Dec 17 09:15:48 Tower kernel: [ 7974] 0 7974 14591 95 2 -17 -1000 php-cgi Dec 17 09:15:48 Tower kernel: [ 7975] 0 7975 14591 96 1 -17 -1000 php-cgi Dec 17 09:15:48 Tower kernel: [ 7976] 0 7976 14591 96 1 -17 -1000 php-cgi Dec 17 09:15:48 Tower kernel: [18415] 0 18415 463 61 1 -17 -1000 sleep Dec 17 09:15:48 Tower kernel: [19997] 0 19997 4015 275 3 0 0 smbd Dec 17 09:15:48 Tower kernel: [20026] 0 20026 463 62 1 0 0 sleep Dec 17 09:15:48 Tower kernel: [20033] 0 20033 3230 74 2 -17 -1000 emhttp Dec 17 09:15:48 Tower kernel: [20034] 0 20034 3230 74 1 -17 -1000 emhttp Dec 17 09:15:48 Tower kernel: Out of memory: Kill process 16701 (VBoxHeadless) score 115 or sacrifice child Dec 17 09:15:48 Tower kernel: Killed process 16701 (VBoxHeadless) total-vm:952484kB, anon-rss:10044kB, file-rss:874332kB Dec 17 09:15:48 Tower kernel: VBoxDrvLinuxIOCtl: copy_to_user(0xaf0e81d0,,0x18); uCmd=0xc0305618! Dec 17 09:15:48 Tower kernel: VBoxDrvLinuxIOCtl: copy_to_user(0xaf2681d0,,0x18); uCmd=0xc0305618! Dec 17 09:15:48 Tower kernel: device eth1 left promiscuous mode Dec 17 09:15:48 Tower kernel: vboxnetflt: dropped 0 out of 0 packets Dec 17 09:15:48 Tower kernel: device eth0 left promiscuous mode Dec 17 09:15:48 Tower kernel: vboxnetflt: dropped 13576960 out of 92450222 packets Actually, you have it backwards. The kernel's Out-Of-Memory task is killing off the processes it thinks have been idle the longest in an attempt to free memory needed by other more active processes. (VboxHeadless and emhttp just happen to be idle most of the time) Basically, you ran out of RAM and it is likely it is your add-ons that have used up all available. Joe L. Quote Link to comment
theone Posted December 17, 2012 Author Share Posted December 17, 2012 This happened in the morning when the server is usually idle - plugins running in the background. When running idle I usually have over 50-75% free memory. No one is accessing the server at this time. I don't see how the server suddenly ran out of memory. Quote Link to comment
Joe L. Posted December 19, 2012 Share Posted December 19, 2012 This happened in the morning when the server is usually idle - plugins running in the background. When running idle I usually have over 50-75% free memory. No one is accessing the server at this time. I don't see how the server suddenly ran out of memory. All it takes is one program to allocate memory and not free it when done to run you out of memory. Your stack trace clearly shows unRAID was attempting to allocate memory and it specifically says: out_of_memory+0x21c/0x29e Based on the rss value, VbocHeadless seems to be the memory hog. Quote Link to comment
theone Posted December 19, 2012 Author Share Posted December 19, 2012 VboxHeadless is indeed the memory hog but it is constant - I allocated a constant 800MB for the virtual machine - It doesn't grow over time. Quote Link to comment
theone Posted December 21, 2012 Author Share Posted December 21, 2012 Again everything has been running smoothly for the last 8 days. I will now enable WEB SERVER plugin (on port 1234) with a simple index.htm file and let it run for another period of ~7 days. Restarted server with WEB SERVER enabled and phpvirtualbox as the index.html as originally used. Waiting to see if emhttp crashes again... Quote Link to comment
theone Posted December 28, 2012 Author Share Posted December 28, 2012 A week has passed and again no access to unRAID WEBUI. I am very frustrated. Quote Link to comment
unevent Posted January 3, 2013 Share Posted January 3, 2013 There are a lot running on that 'lil Atom with 4GB, curious what the sysload is. Could try a swap partition on a cache drive or remove some plugins. Run 'free -l' every day to see what the low mem is doing - not sure if simple features tracks low mem. As others have said there may be a leak in one or more addon programs causing the slow death. Search the forum for posts about keeping the kernel oom task from killing off processes like emhttp and other important processes as first choice and there are also posts about adding a swap partition. Swap can be added to a cache drive, or for temporary use, installed on a USB flash drive (not the unraid flash) as a cheap, albeit limited life swap. If any of the addon programs has a mem leak no amount of bandaids will fix it and will still eventually fail, just takes longer. Unraid 5.0-rc8a - Asus M5A78L-MLX Plus - AMD Athlon II X3 450 Rana 3.2GHz - 8GB DDR3 - Antec NEO ECO 620W - Antec Three Hundred - 1x Rosewill RC-211 - 3.75TB Array, Intel 40GB SSD Cache, 44W idle Quote Link to comment
theone Posted January 3, 2013 Author Share Posted January 3, 2013 There are a lot running on that 'lil Atom with 4GB, curious what the sysload is. Could try a swap partition on a cache drive or remove some plugins. Run 'free -l' every day to see what the low mem is doing - not sure if simple features tracks low mem. As others have said there may be a leak in one or more addon programs causing the slow death. Search the forum for posts about keeping the kernel oom task from killing off processes like emhttp and other important processes as first choice and there are also posts about adding a swap partition. Swap can be added to a cache drive, or for temporary use, installed on a USB flash drive (not the unraid flash) as a cheap, albeit limited life swap. If any of the addon programs has a mem leak no amount of bandaids will fix it and will still eventually fail, just takes longer. I have a swap file running, I see no memory shortage problems, I have changed the kill priority of emhttp and smdb as forum suggests. I get no OOM for emhttp. emhttp is not killed. Even when I cannot reach the WEBUI, emhttp is still running in the process list. Quote Link to comment
theone Posted January 17, 2013 Author Share Posted January 17, 2013 After 7 days again of no problems, webserver installed but disabled, I enabled the web server. After ~8 hours I tried connecting to WEBUI. At first it worked but after that I tried opening the syslog via the LOG link on top right of WEBUI with no response. WEBUI is not reachable at all now. Syslog shows nothing as usual. Quote Link to comment
joelones Posted January 21, 2013 Share Posted January 21, 2013 Same situation here. Webui inaccessible, nothing in log. Using rc10. Running a VM with 4gb, on esxi. Using three plugins (sab/sickbeard) and virtual memory. Quote Link to comment
theone Posted February 5, 2013 Author Share Posted February 5, 2013 Installed RC11 and again after less than 1 day the WEBUI is inaccessible - all else is accessible. Web-server which I thought was the culprit is not running. Nothing suspicious in the syslog. Quote Link to comment
bonienl Posted February 6, 2013 Share Posted February 6, 2013 Installed RC11 and again after less than 1 day the WEBUI is inaccessible - all else is accessible. Web-server which I thought was the culprit is not running. Nothing suspicious in the syslog. The webserver plugin installs a number of packages, including a updated version of PHP. There is a difference between installed/not running and "not installed". This goes for more plugins which install packages in the background. Best test approach would be to *not* install a plugin at all (i.e. remove it from the plugins folder). Quote Link to comment
theone Posted March 20, 2013 Author Share Posted March 20, 2013 Again WEBUI got stuck ~1week after enabling WebServer Plugin. Attached are netstat -nope (unRAID) and netstat -nob (Windows). Please take a look and see if anything is noticable... netstat_unRAID.txt netstat_Windows.txt Quote Link to comment
aptalca Posted March 21, 2013 Share Posted March 21, 2013 For me, the webserver causes webui crashes very frequently. I only use the webserver for phpvirtualbox. Now that my virtualbox is totally set up, I no longer need to access the webserver and so I disabled the webserver. Haven't had a problem since. I guess I'll deal with the crashes if I need to make any changes to my virtualbox config Oh and by the way, when the webui was inaccessible for me (and emhttp still running), killing the webserver would make the webui accessible again. Is that the case for you as well? Quote Link to comment
theone Posted March 21, 2013 Author Share Posted March 21, 2013 how do you kill the webserver? Quote Link to comment
abs0lut.zer0 Posted March 21, 2013 Share Posted March 21, 2013 how do you kill the webserver? +1 i have the same problem Quote Link to comment
aptalca Posted May 13, 2013 Share Posted May 13, 2013 When I said killing the webserver, I meant killing the webserver plugin. But I spoke too soon. Since my message, my gui started crashing even without the webserver plugin running. I didn't install any new plugins, so I am guessing an update to one of them is causing this (I am suspecting plex) I tried killing every process one by one, but did not get the gui back. I also tried to kill emhttp and restart it with the "/usr/local/sbin/emhttp &" but I get the segmentation fault I read somewhere else that in order to restart emhttp that way, you may have to stop the array and unmount all drives, in that case I might as well just do a reboot I wish I had some message in the log about emhttp crashing, maybe that would provide some clue as to what's crashing it. Quote Link to comment
Joe L. Posted May 13, 2013 Share Posted May 13, 2013 I also tried to kill emhttp and restart it with the "/usr/local/sbin/emhttp &" but I get the segmentation fault Correct. Can't do that in the 5.X series. It usually worked in the 4.7 series and prior. I read somewhere else that in order to restart emhttp that way, you may have to stop the array and unmount all drives, in that case I might as well just do a reboot Exactly. Quote Link to comment
axeman Posted May 21, 2013 Share Posted May 21, 2013 i'm wondering if this is related to the Tap-a-talk notifier/plugin. I've noticed when I access the UI via Chrome on a machine that has tapatalk notifier there are messages that are output on the console. For 5 days now, I've been using IE to get to webui, and so far, everything is still accessible. When I started with the 5.0 release, I hadn't gone more than 3 days w.o. losing the UI access. Just a thought from a noob. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.