February 14, 201511 yr I have only these 3 containers installed. Using 6.0 b12 with 32GB RAM on a Supermicro X10SL7-F. Crashplan works ok with less than 100Gb files selected. It crashes with more than this. The gui says "disconnected from backup engine". It already has 1024M in the srv parameters Squeezebox has about 100Gb of music to scan. It crashes every few seconds. Below is a syslog excerpt. I guess both these containers need more RAM? How do I do that? Feb 14 21:33:01 Tower kernel: cpuload invoked oom-killer: gfp_mask=0x2000d0, order=2, oom_score_adj=0 Feb 14 21:33:01 Tower kernel: cpuload cpuset=/ mems_allowed=0 Feb 14 21:33:01 Tower kernel: CPU: 1 PID: 1591 Comm: cpuload Not tainted 3.17.4-unRAID #1 Feb 14 21:33:01 Tower kernel: Hardware name: Supermicro X10SL7-F/X10SL7-F, BIOS 2.00 04/24/2014 Feb 14 21:33:01 Tower kernel: 0000000000000000 ffff880056507b48 ffffffff815e4145 ffff880055e1c890 Feb 14 21:33:01 Tower kernel: ffff880056507bd0 ffffffff815e0bc1 ffffffff8107f1a2 0000000000000000 Feb 14 21:33:01 Tower kernel: ffffffff815e9979 0000000000050bc8 ffff880056507ba8 ffffffff81099a70 Feb 14 21:33:01 Tower kernel: Call Trace: Feb 14 21:33:01 Tower kernel: [] dump_stack+0x45/0x56 Feb 14 21:33:01 Tower kernel: [] dump_header+0x7a/0x1df Feb 14 21:33:01 Tower kernel: [] ? ktime_get+0x49/0x87 Feb 14 21:33:01 Tower kernel: [] ? _raw_spin_unlock_irqrestore+0x50/0x60 Feb 14 21:33:01 Tower kernel: [] ? delayacct_end+0x50/0x59 Feb 14 21:33:01 Tower kernel: [] oom_kill_process+0x6d/0x328 Feb 14 21:33:01 Tower kernel: [] out_of_memory+0x40c/0x425 Feb 14 21:33:01 Tower kernel: [] __alloc_pages_nodemask+0x6eb/0x832 Feb 14 21:33:01 Tower kernel: [] alloc_kmem_pages_node+0x74/0xeb Feb 14 21:33:01 Tower kernel: [] copy_process.part.38+0xeb/0x15f1 Feb 14 21:33:01 Tower kernel: [] ? get_empty_filp+0xa8/0x16d Feb 14 21:33:01 Tower kernel: [] do_fork+0xb4/0x29d Feb 14 21:33:01 Tower kernel: [] ? __set_task_blocked+0x5e/0x65 Feb 14 21:33:01 Tower kernel: [] SyS_clone+0x11/0x13 Feb 14 21:33:01 Tower kernel: [] stub_clone+0x69/0x90 Feb 14 21:33:01 Tower kernel: [] ? system_call_fastpath+0x16/0x1b Feb 14 21:33:01 Tower kernel: Mem-Info: Feb 14 21:33:01 Tower kernel: Node 0 DMA per-cpu: Feb 14 21:33:01 Tower kernel: CPU 0: hi: 0, btch: 1 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 1: hi: 0, btch: 1 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 2: hi: 0, btch: 1 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 3: hi: 0, btch: 1 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 4: hi: 0, btch: 1 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 5: hi: 0, btch: 1 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 6: hi: 0, btch: 1 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 7: hi: 0, btch: 1 usd: 0 Feb 14 21:33:01 Tower kernel: Node 0 DMA32 per-cpu: Feb 14 21:33:01 Tower kernel: CPU 0: hi: 186, btch: 31 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 1: hi: 186, btch: 31 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 2: hi: 186, btch: 31 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 3: hi: 186, btch: 31 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 4: hi: 186, btch: 31 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 5: hi: 186, btch: 31 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 6: hi: 186, btch: 31 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 7: hi: 186, btch: 31 usd: 0 Feb 14 21:33:01 Tower kernel: Node 0 Normal per-cpu: Feb 14 21:33:01 Tower kernel: CPU 0: hi: 0, btch: 1 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 1: hi: 0, btch: 1 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 2: hi: 0, btch: 1 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 3: hi: 0, btch: 1 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 4: hi: 0, btch: 1 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 5: hi: 0, btch: 1 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 6: hi: 0, btch: 1 usd: 0 Feb 14 21:33:01 Tower kernel: CPU 7: hi: 0, btch: 1 usd: 0 Feb 14 21:33:01 Tower kernel: active_anon:179851 inactive_anon:10785 isolated_anon:0 Feb 14 21:33:01 Tower kernel: active_file:6100 inactive_file:17272 isolated_file:5 Feb 14 21:33:01 Tower kernel: unevictable:36 dirty:14187 writeback:2960 unstable:0 Feb 14 21:33:01 Tower kernel: free:68443 slab_reclaimable:54932 slab_unreclaimable:12316 Feb 14 21:33:01 Tower kernel: mapped:13226 shmem:116814 pagetables:3038 bounce:0 Feb 14 21:33:01 Tower kernel: free_cma:0 Feb 14 21:33:01 Tower kernel: Node 0 DMA free:6240kB min:48kB low:60kB high:72kB active_anon:1216kB inactive_anon:220kB active_file:0kB inactive_file:20kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15968kB managed:15884kB mlocked:0kB dirty:0kB writeback:8kB mapped:64kB shmem:412kB slab_reclaimable:4800kB slab_unreclaimable:1540kB kernel_stack:432kB pagetables:28kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:140 all_unreclaimable? yes Feb 14 21:33:01 Tower kernel: lowmem_reserve[]: 0 1553 1553 1553 Feb 14 21:33:01 Tower kernel: Node 0 DMA32 free:267532kB min:5016kB low:6268kB high:7524kB active_anon:718188kB inactive_anon:42920kB active_file:24404kB inactive_file:69068kB unevictable:144kB isolated(anon):0kB isolated(file):20kB present:3612964kB managed:1592052kB mlocked:144kB dirty:56748kB writeback:11832kB mapped:52840kB shmem:466844kB slab_reclaimable:214928kB slab_unreclaimable:47724kB kernel_stack:4784kB pagetables:12124kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:575948 all_unreclaimable? yes Feb 14 21:33:01 Tower kernel: lowmem_reserve[]: 0 0 0 0 Feb 14 21:33:01 Tower kernel: Node 0 Normal free:0kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:19443856kB managed:6640kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Feb 14 21:33:01 Tower kernel: lowmem_reserve[]: 0 0 0 0 Feb 14 21:33:01 Tower kernel: Node 0 DMA: 61*4kB (UEM) 72*8kB (E) 97*16kB (EM) 3*32kB (R) 1*64kB (R) 1*128kB (R) 0*256kB 1*512kB (R) 1*1024kB (R) 1*2048kB (R) 0*4096kB = 6244kB Feb 14 21:33:01 Tower kernel: Node 0 DMA32: 62229*4kB (UEMR) 2190*8kB (UEMR) 66*16kB (UEMR) 1*32kB (R) 1*64kB (R) 2*128kB (R) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 267844kB Feb 14 21:33:01 Tower kernel: Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB Feb 14 21:33:01 Tower kernel: 140175 total pagecache pages Feb 14 21:33:01 Tower kernel: 0 pages in swap cache Feb 14 21:33:01 Tower kernel: Swap cache stats: add 0, delete 0, find 0/0 Feb 14 21:33:01 Tower kernel: Free swap = 0kB Feb 14 21:33:01 Tower kernel: Total swap = 0kB Feb 14 21:33:01 Tower kernel: 5768197 pages RAM Feb 14 21:33:01 Tower kernel: 0 pages HighMem/MovableOnly Feb 14 21:33:01 Tower kernel: 4859304 pages reserved Feb 14 21:33:01 Tower kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Feb 14 21:33:01 Tower kernel: [ 1101] 0 1101 5395 618 16 0 -1000 udevd Feb 14 21:33:01 Tower kernel: [ 1308] 0 1308 58495 891 26 0 0 rsyslogd Feb 14 21:33:01 Tower kernel: [ 1478] 1 1478 1856 370 7 0 0 rpc.portmap Feb 14 21:33:01 Tower kernel: [ 1482] 32 1482 3209 485 11 0 0 rpc.statd Feb 14 21:33:01 Tower kernel: [ 1492] 0 1492 1615 430 8 0 0 inetd Feb 14 21:33:01 Tower kernel: [ 1502] 0 1502 6232 641 15 0 -1000 sshd Feb 14 21:33:01 Tower kernel: [ 1511] 0 1511 48118 2548 97 0 0 ntpd Feb 14 21:33:01 Tower kernel: [ 1521] 0 1521 1096 404 7 0 0 acpid Feb 14 21:33:01 Tower kernel: [ 1533] 81 1533 4359 52 13 0 0 dbus-daemon Feb 14 21:33:01 Tower kernel: [ 1535] 0 1535 1617 417 8 0 0 crond Feb 14 21:33:01 Tower kernel: [ 1537] 0 1537 1615 25 8 0 0 atd Feb 14 21:33:01 Tower kernel: [ 1541] 0 1541 49065 1206 93 0 0 nmbd Feb 14 21:33:01 Tower kernel: [ 1543] 0 1543 68502 3506 133 0 0 smbd Feb 14 21:33:01 Tower kernel: [ 1575] 0 1575 2777 578 11 0 0 xenstored Feb 14 21:33:01 Tower kernel: [ 1582] 0 1582 21173 488 15 0 0 xenconsoled Feb 14 21:33:01 Tower kernel: [ 1586] 0 1586 42174 1749 50 0 0 qemu-system-x86 Feb 14 21:33:01 Tower kernel: [ 1591] 0 1591 2384 592 9 0 0 cpuload Feb 14 21:33:01 Tower kernel: [ 2786] 0 2786 22264 835 17 0 0 emhttp Feb 14 21:33:01 Tower kernel: [ 2787] 0 2787 1618 434 8 0 0 agetty Feb 14 21:33:01 Tower kernel: [ 2788] 0 2788 1618 431 7 0 0 agetty Feb 14 21:33:01 Tower kernel: [ 2789] 0 2789 1618 435 8 0 0 agetty Feb 14 21:33:01 Tower kernel: [ 2790] 0 2790 1618 423 8 0 0 agetty Feb 14 21:33:01 Tower kernel: [ 2791] 0 2791 1618 412 8 0 0 agetty Feb 14 21:33:01 Tower kernel: [ 2792] 0 2792 1618 436 8 0 0 agetty Feb 14 21:33:01 Tower kernel: [ 2862] 61 2862 8613 778 22 0 0 avahi-daemon Feb 14 21:33:01 Tower kernel: [ 2863] 61 2863 8547 60 21 0 0 avahi-daemon Feb 14 21:33:01 Tower kernel: [ 2871] 0 2871 3186 408 11 0 0 avahi-dnsconfd Feb 14 21:33:01 Tower kernel: [ 2941] 0 2941 72056 472 21 0 0 shfs Feb 14 21:33:01 Tower kernel: [ 2951] 0 2951 291894 1821 48 0 0 shfs Feb 14 21:33:01 Tower kernel: [ 3014] 0 3014 206642 6616 67 0 0 docker Feb 14 21:33:01 Tower kernel: [ 3071] 0 3071 3000 628 10 0 0 rc.snap Feb 14 21:33:01 Tower kernel: [ 3072] 0 3072 2998 626 10 0 0 rc.snap Feb 14 21:33:01 Tower kernel: [ 3084] 0 3084 1631 386 8 0 0 inotifywait Feb 14 21:33:01 Tower kernel: [ 3085] 0 3085 1631 365 8 0 0 inotifywait Feb 14 21:33:01 Tower kernel: [ 3086] 0 3086 2998 193 10 0 0 rc.snap Feb 14 21:33:01 Tower kernel: [ 3087] 0 3087 3000 195 10 0 0 rc.snap Feb 14 21:33:01 Tower kernel: [ 3702] 0 3702 28047 3168 28 0 0 xl Feb 14 21:33:01 Tower kernel: [ 3848] 0 3848 28047 3166 27 0 0 xl Feb 14 21:33:01 Tower kernel: [ 4003] 0 4003 28047 3173 27 0 0 xl Feb 14 21:33:01 Tower kernel: [ 4187] 0 4187 28047 3205 27 0 0 xl Feb 14 21:33:01 Tower kernel: [ 4377] 0 4377 44973 3206 29 0 0 xl Feb 14 21:33:01 Tower kernel: [ 6545] 0 6545 2420 651 9 0 0 mount.ntfs-3g Feb 14 21:33:01 Tower kernel: [23346] 0 23346 8437 1279 21 0 0 my_init Feb 14 21:33:01 Tower kernel: [23358] 0 23358 49 4 3 0 0 runsvdir Feb 14 21:33:01 Tower kernel: [23359] 0 23359 44 4 3 0 0 runsv Feb 14 21:33:01 Tower kernel: [23360] 0 23360 44 3 3 0 0 runsv Feb 14 21:33:01 Tower kernel: [23361] 0 23361 44 4 3 0 0 runsv Feb 14 21:33:01 Tower kernel: [23363] 0 23363 16469 412 36 0 0 syslog-ng Feb 14 21:33:01 Tower kernel: [23364] 0 23364 7111 61 17 0 0 cron Feb 14 21:33:01 Tower kernel: [28062] 0 28062 731509 11989 328 0 0 qemu-system-x86 Feb 14 21:33:01 Tower kernel: [29459] 0 29459 669384 11078 237 0 0 qemu-system-x86 Feb 14 21:33:01 Tower kernel: [31221] 0 31221 705257 11875 183 0 0 qemu-system-x86 Feb 14 21:33:01 Tower kernel: [ 537] 1000 537 76189 6555 148 0 0 smbd Feb 14 21:33:01 Tower kernel: [ 4059] 0 4059 76704 4035 145 0 0 smbd Feb 14 21:33:01 Tower kernel: [18430] 99 18430 110518 174 30 0 0 btsync Feb 14 21:33:01 Tower kernel: [12671] 0 12671 20727 476 14 0 0 apcupsd Feb 14 21:33:01 Tower kernel: [ 4064] 0 4064 697096 11146 193 0 0 qemu-system-x86 Feb 14 21:33:01 Tower kernel: [26722] 0 26722 626644 8252 183 0 0 qemu-system-x86 Feb 14 21:33:01 Tower kernel: [10468] 0 10468 601944 9954 169 0 0 qemu-system-x86 Feb 14 21:33:01 Tower kernel: [10574] 0 10574 25565 674 23 0 0 xl Feb 14 21:33:01 Tower kernel: [10880] 99 10880 76734 6115 149 0 0 smbd Feb 14 21:33:01 Tower kernel: [31860] 1000 31860 76990 6855 148 0 0 smbd Feb 14 21:33:01 Tower kernel: [17371] 0 17371 3334 844 11 0 0 smartctl Feb 14 21:33:01 Tower kernel: Out of memory: Kill process 28062 (qemu-system-x86) score 29 or sacrifice child Feb 14 21:33:01 Tower kernel: Killed process 28062 (qemu-system-x86) total-vm:2926036kB, anon-rss:36548kB, file-rss:11408kB Feb 14 21:33:01 Tower avahi-daemon[2862]: Withdrawing workstation service for vif11.0-emu. Feb 14 21:33:01 Tower kernel: xenbr0: port 12(vif11.0-emu) entered disabled state Feb 14 21:33:01 Tower kernel: device vif11.0-emu left promiscuous mode Feb 14 21:33:01 Tower kernel: xenbr0: port 12(vif11.0-emu) entered disabled state Feb 14 21:33:03 Tower php: /usr/bin/docker start LogitechMediaServer Feb 14 21:33:04 Tower kernel: nf_conntrack: falling back to vmalloc. Feb 14 21:33:04 Tower kernel: nf_conntrack: falling back to vmalloc. Feb 14 21:33:04 Tower kernel: device veth90e73bc entered promiscuous mode Feb 14 21:33:04 Tower avahi-daemon[2862]: Withdrawing workstation service for vetha2e80c0. Feb 14 21:33:04 Tower kernel: eth0: renamed from vetha2e80c0 Feb 14 21:33:04 Tower kernel: docker0: port 1(veth90e73bc) entered forwarding state Feb 14 21:33:04 Tower kernel: docker0: port 1(veth90e73bc) entered forwarding state Feb 14 21:33:04 Tower php: LogitechMediaServer Feb 14 21:33:04 Tower php: Feb 14 21:33:05 Tower kernel: kworker/u16:5 invoked oom-killer: gfp_mask=0x2000d0, order=2, oom_score_adj=0 Feb 14 21:33:05 Tower kernel: kworker/u16:5 cpuset=/ mems_allowed=0 Feb 14 21:33:05 Tower kernel: CPU: 4 PID: 17484 Comm: kworker/u16:5 Not tainted 3.17.4-unRAID #1 Feb 14 21:33:05 Tower kernel: Hardware name: Supermicro X10SL7-F/X10SL7-F, BIOS 2.00 04/24/2014 Feb 14 21:33:05 Tower kernel: Workqueue: khelper __call_usermodehelper Feb 14 21:33:05 Tower kernel: 0000000000000000 ffff880000b0ba00 ffffffff815e4145 ffff88000b988810 Feb 14 21:33:05 Tower kernel: ffff880000b0ba88 ffffffff815e0bc1 ffffffff8107f1a2 0000000000000000 Feb 14 21:33:05 Tower kernel: ffffffff815e9979 000000000020073b ffff880000b0ba60 ffffffff81099a70 Feb 14 21:33:05 Tower kernel: Call Trace: Feb 14 21:33:05 Tower kernel: [] dump_stack+0x45/0x56 Feb 14 21:33:05 Tower kernel: [] dump_header+0x7a/0x1df Feb 14 21:33:05 Tower kernel: [] ? ktime_get+0x49/0x87 Feb 14 21:33:05 Tower kernel: [] ? _raw_spin_unlock_irqrestore+0x50/0x60 Feb 14 21:33:05 Tower kernel: [] ? delayacct_end+0x50/0x59 Feb 14 21:33:05 Tower kernel: [] oom_kill_process+0x6d/0x328 Feb 14 21:33:05 Tower kernel: [] ? oom_badness+0xb0/0xf9 Feb 14 21:33:05 Tower kernel: [] out_of_memory+0x40c/0x425
February 15, 201511 yr I'm far from an expert, but how much memory have you allocated to your VM's vs left over for unRaid?
February 15, 201511 yr Author Hi Squid, good catch. I had over provisioned the VMs. I take it that Xen doesn't do thin provisioning. I've reduced them now, so XEN + Dom0 takes around 22GB RAM. I've restarted Crashplan and Squeezeserver and so far still running. Memory usage for Unraid seems high - 80%, should I increase the amount allocated? How do I know how much memory Docker containers are taking? cheers.
February 16, 201511 yr Author Thanks for the reply Squid. Here is the syslog. This is from a after a hard reboot yesterday morning, and before I reduced the VM allocation down yesterday lunch time. syslog.zip
February 18, 201511 yr You're definitely running out of memory. The OS invoked the OOM killer which terminated Java (which presumably in turn toasted CrashPlan) At the time CrashPlan was nuked, you were also running a parity check (probably because of hard shutdowns). This also takes a fair chunk of memory, depending on what your tunables settings are at. Cancelling it will help the situation out a bit, but that's definitely not a solution you really want to use. Try stopping your other Dockers, and see what happens. After that, you're probably going to have to play with the VM settings which is something I really can't help you with.
February 18, 201511 yr Author Thanks for the analysis Squid. My VMs have 20GB Ram allocated, and I have 32GB RAM, so that should be plenty. I suspect that I need to allocate more memory to Docker, but I'm not not sure how to do that. Does Docker RAM come out of the 1,576MB that unRaid itself is allocated? Crashplan is working fine on a small backup set. Anyone any suggestions?
February 18, 201511 yr Create a nice large swapfile to handle these kind of problems. I was just looking at options in docker to limit memory usage, but it appears that Docker itself will kill the container if it goes over the limit. (but at least in that situation it can't potentially crash the entire system like OOM can)
February 19, 201511 yr Author Ah ok, thanks. So should my unRaid memory be only 1.5GB? Can I change this? Should I change it?
February 19, 201511 yr The dashboard allocated shows what unRaid actually has to use. In your case, 1.5 gig which is obviously too little for Crashplan I don't run Xen so I can't really answer that question. Try searching the in the unRaid Xen forum
February 19, 201511 yr Author Sorry for a dumb question, but why do you say this is related to Xen? The Xen VMs are working fine, it is the Docker containers that are running out of memory. Unless I misunderstand this is a docker or unRaid memory config issue, not related to Xen?
February 19, 201511 yr I could be wrong (just going by what I've gathered by reading the posts), but in Xen don't you reserve a certain amount of memory for the host (unRaid)? In your case, 1.5 gig + 22 gig doesn't equal 32 gig. Or you can create a swap file in unRaid to try and solve the problem
February 19, 201511 yr ok... Looks like I'm wrong. Just found this post which says that memory not allocated to Xen is automatically allocated to unRaid. http://lime-technology.com/forum/index.php?topic=36206.msg337362#msg337362 But, in your case you're still looking at unRaid and therefore the dockers only having access to 1.5 Gig which isn't enough for what you're trying to do.
February 19, 201511 yr Author Your post just clicked! Dom0 is set to use 2048Mb in syslinux.cfg. unRaid memory comes out of this! I've changed syslinux to be 4096, and rebooted. Now the dashboard reports 3.5GB free, so hopefully gives the Dockers much more headroom. Now to add a few TB to crashplan! thanks
March 9, 201511 yr Author A quick update - Crashplan seems to be behaving better now, but it still crashes every couple of days with OOM killing Java. The Unraid dashboard is showing about 55% to 60% memory usage. I have 4GB allocated to Unraid. Is there anything else I can check?
Archived
This topic is now archived and is closed to further replies.