June 30, 20188 yr I'm a new user to unRAID, coming over from FreeNAS after an arduous past few years. I've been having some trouble and hope someone may be able to point me in the right direction. Ever since it's all been up and running, I get significant lag and stalling whenever there's anything going on (streaming a video, downloading, copying over LAN, off-site backup via Resilio). I installed the Glances container through CA to help get some visibility into what's going on and notice a great deal of CPU_IOWAIT alerts being thrown. This is an older, but still relevant, server that has ran the same set of applications more or less for about the past six years: Plex, Usenet client, Torrent client, TV episode manager, movie manager, and local file repository, all running on the same hardware in FreeNAS, Windows Server 2012 R2 w/Ubuntu Server VM, & Windows Home Server. Hardware is SuperMicro X9SCM-F w/Xeon E3-1230 v2, 16GB ECC RAM, LSI 9211-8i controller (JBOD), 5x 4TB HDs for main array (mixed manufacturers, but all 7200s), a 1TB & 2TB added for cache. I have never had this kind of system stalling on other OS's. Everything is affected when it happens including every container as well as the unRAID UI itself. If I wait, it will eventually clear up, but Plex may timeout on playback, etc. I have the containers pinned across the cores, but since this seems to be file system lag-related and not a CPU bottleneck, I'm at a bit of a loss as to how to fix this. Glances screenshots and diagnostics .zip attached. Thank you in advance for any assistance! carbon-diagnostics-20180628-1453.zip
June 30, 20188 yr Many, many controllers pause all transfers if a drive needs to spin up to fulfill an I/O request. Does the same thing happen if you disable spin down?Sent via Tapatalk because I'm either at work or enjoying the summer
June 30, 20188 yr neat-o oom errors and call traces... relating to docker. So, probably try limiting the apps memory usage in each docker app: https://lime-technology.com/forums/topic/57181-real-docker-faq/ Jun 28 00:03:39 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 00:03:39 carbon kernel: CPU: 4 PID: 20189 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 00:03:39 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 00:03:39 carbon kernel: Call Trace: Jun 28 00:03:39 carbon kernel: dump_stack+0x5d/0x79 Jun 28 00:03:39 carbon kernel: dump_header+0x8f/0x214 Jun 28 00:03:39 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 00:03:39 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 00:03:39 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 00:03:39 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 00:03:39 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 00:03:39 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 00:03:39 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 00:03:39 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 00:03:39 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 00:03:39 carbon kernel: page_fault+0x45/0x50 Jun 28 00:03:39 carbon kernel: RIP: f244edf0:0xffffffffffffff92 Jun 28 00:03:39 carbon kernel: RSP: f244ee20:000055614c50aeb0 EFLAGS: 000151bf Jun 28 00:03:39 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 00:03:39 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 87308599 Jun 28 00:03:39 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 00:03:39 carbon kernel: kmem: usage 28600kB, limit 9007199254740988kB, failcnt 0 Jun 28 00:03:39 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:1880KB rss:493808KB rss_huge:221184KB shmem:0KB mapped_file:636KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:493808KB inactive_file:744KB active_file:688KB unevictable:0KB Jun 28 00:03:39 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 00:03:39 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 00:03:39 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 00:03:39 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 00:03:39 carbon kernel: [20134] 99 20134 395265 123170 313 4 0 0 rslsync Jun 28 00:03:39 carbon kernel: Memory cgroup out of memory: Kill process 20134 (rslsync) score 942 or sacrifice child Jun 28 00:03:39 carbon kernel: Killed process 20134 (rslsync) total-vm:1581060kB, anon-rss:492680kB, file-rss:0kB, shmem-rss:0kB Jun 28 00:03:39 carbon kernel: oom_reaper: reaped process 20134 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 00:07:54 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 00:07:54 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 00:07:54 carbon kernel: CPU: 4 PID: 22038 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 00:07:54 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 00:07:54 carbon kernel: Call Trace: Jun 28 00:07:54 carbon kernel: dump_stack+0x5d/0x79 Jun 28 00:07:54 carbon kernel: dump_header+0x8f/0x214 Jun 28 00:07:54 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 00:07:54 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 00:07:54 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 00:07:54 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 00:07:54 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 00:07:54 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 00:07:54 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 00:07:54 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 00:07:54 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 00:07:54 carbon kernel: page_fault+0x45/0x50 Jun 28 00:07:54 carbon kernel: RIP: 7f99d010:0x14a6a4342b8c Jun 28 00:07:54 carbon kernel: RSP: 0000:0000000000000044 EFLAGS: 00200000 Jun 28 00:07:54 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 00:07:54 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 87424018 Jun 28 00:07:54 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 00:07:54 carbon kernel: kmem: usage 31504kB, limit 9007199254740988kB, failcnt 0 Jun 28 00:07:54 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:2252KB rss:490532KB rss_huge:12288KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:490532KB inactive_file:1964KB active_file:32KB unevictable:0KB Jun 28 00:07:54 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 00:07:54 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 00:07:54 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 00:07:54 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 00:07:54 carbon kernel: [14929] 99 14929 413935 122075 310 5 0 0 rslsync Jun 28 00:07:54 carbon kernel: Memory cgroup out of memory: Kill process 14929 (rslsync) score 933 or sacrifice child Jun 28 00:07:54 carbon kernel: Killed process 14929 (rslsync) total-vm:1655740kB, anon-rss:488300kB, file-rss:0kB, shmem-rss:0kB Jun 28 00:07:54 carbon kernel: oom_reaper: reaped process 14929 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 00:20:35 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 00:20:35 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 00:20:35 carbon kernel: CPU: 4 PID: 3203 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 00:20:35 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 00:20:35 carbon kernel: Call Trace: Jun 28 00:20:35 carbon kernel: dump_stack+0x5d/0x79 Jun 28 00:20:35 carbon kernel: dump_header+0x8f/0x214 Jun 28 00:20:35 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 00:20:35 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 00:20:35 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 00:20:35 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 00:20:35 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 00:20:35 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 00:20:35 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 00:20:35 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 00:20:35 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 00:20:35 carbon kernel: page_fault+0x45/0x50 Jun 28 00:20:35 carbon kernel: RIP: 9163edf0:0xffffffffffffff92 Jun 28 00:20:35 carbon kernel: RSP: 9163ee20:000055ea679b3210 EFLAGS: 000005ef Jun 28 00:20:35 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 00:20:35 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 87587792 Jun 28 00:20:35 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 00:20:35 carbon kernel: kmem: usage 38640kB, limit 9007199254740988kB, failcnt 0 Jun 28 00:20:35 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:2620KB rss:483028KB rss_huge:4096KB shmem:0KB mapped_file:56KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:483028KB inactive_file:1056KB active_file:924KB unevictable:0KB Jun 28 00:20:35 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 00:20:35 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 00:20:35 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 00:20:35 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 00:20:35 carbon kernel: [ 3151] 99 3151 411827 120598 310 5 0 0 rslsync Jun 28 00:20:35 carbon kernel: Memory cgroup out of memory: Kill process 3151 (rslsync) score 922 or sacrifice child Jun 28 00:20:35 carbon kernel: Killed process 3151 (rslsync) total-vm:1647308kB, anon-rss:482392kB, file-rss:0kB, shmem-rss:0kB Jun 28 00:20:35 carbon kernel: oom_reaper: reaped process 3151 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 00:28:55 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 00:28:55 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 00:28:55 carbon kernel: CPU: 4 PID: 690 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 00:28:55 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 00:28:55 carbon kernel: Call Trace: Jun 28 00:28:55 carbon kernel: dump_stack+0x5d/0x79 Jun 28 00:28:55 carbon kernel: dump_header+0x8f/0x214 Jun 28 00:28:55 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 00:28:55 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 00:28:55 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 00:28:55 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 00:28:55 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 00:28:55 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 00:28:55 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 00:28:55 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 00:28:55 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 00:28:55 carbon kernel: page_fault+0x45/0x50 Jun 28 00:28:55 carbon kernel: RIP: 5f60cdf0:0xffffffffffffff92 Jun 28 00:28:55 carbon kernel: RSP: 5f60ce20:0000555eeaa41eb0 EFLAGS: 000003e5 Jun 28 00:28:55 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 00:28:55 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 87751952 Jun 28 00:28:55 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 00:28:55 carbon kernel: kmem: usage 25752kB, limit 9007199254740988kB, failcnt 0 Jun 28 00:28:55 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:4104KB rss:494432KB rss_huge:2048KB shmem:0KB mapped_file:164KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:494432KB inactive_file:1828KB active_file:1508KB unevictable:0KB Jun 28 00:28:55 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 00:28:55 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 00:28:55 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 00:28:55 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 00:28:55 carbon kernel: [ 641] 99 641 415432 123486 316 5 0 0 rslsync Jun 28 00:28:55 carbon kernel: Memory cgroup out of memory: Kill process 641 (rslsync) score 944 or sacrifice child Jun 28 00:28:55 carbon kernel: Killed process 641 (rslsync) total-vm:1661728kB, anon-rss:493944kB, file-rss:0kB, shmem-rss:0kB Jun 28 00:28:55 carbon kernel: oom_reaper: reaped process 641 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 00:36:36 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 00:36:36 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 00:36:36 carbon kernel: CPU: 4 PID: 9298 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 00:36:36 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 00:36:36 carbon kernel: Call Trace: Jun 28 00:36:36 carbon kernel: dump_stack+0x5d/0x79 Jun 28 00:36:36 carbon kernel: dump_header+0x8f/0x214 Jun 28 00:36:36 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 00:36:36 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 00:36:36 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 00:36:36 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 00:36:36 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 00:36:36 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 00:36:36 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 00:36:36 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 00:36:36 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 00:36:36 carbon kernel: page_fault+0x45/0x50 Jun 28 00:36:36 carbon kernel: RIP: fe3cedf0:0xffffffffffffff92 Jun 28 00:36:36 carbon kernel: RSP: fe3cee20:0000561daa839210 EFLAGS: 00000399 Jun 28 00:36:36 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 00:36:36 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 87927529 Jun 28 00:36:36 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 00:36:36 carbon kernel: kmem: usage 25456kB, limit 9007199254740988kB, failcnt 0 Jun 28 00:36:36 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:2708KB rss:496124KB rss_huge:6144KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:496124KB inactive_file:1408KB active_file:788KB unevictable:0KB Jun 28 00:36:36 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 00:36:36 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 00:36:36 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 00:36:36 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 00:36:36 carbon kernel: [ 9296] 99 9296 414998 123978 319 4 0 0 rslsync Jun 28 00:36:36 carbon kernel: Memory cgroup out of memory: Kill process 9296 (rslsync) score 948 or sacrifice child Jun 28 00:36:36 carbon kernel: Killed process 9296 (rslsync) total-vm:1659992kB, anon-rss:495912kB, file-rss:0kB, shmem-rss:0kB Jun 28 00:36:36 carbon kernel: oom_reaper: reaped process 9296 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 00:45:27 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 00:45:27 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 00:45:27 carbon kernel: CPU: 4 PID: 16537 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 00:45:27 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 00:45:27 carbon kernel: Call Trace: Jun 28 00:45:27 carbon kernel: dump_stack+0x5d/0x79 Jun 28 00:45:27 carbon kernel: dump_header+0x8f/0x214 Jun 28 00:45:27 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 00:45:27 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 00:45:27 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 00:45:27 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 00:45:27 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 00:45:27 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 00:45:27 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 00:45:27 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 00:45:27 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 00:45:27 carbon kernel: page_fault+0x45/0x50 Jun 28 00:45:27 carbon kernel: RIP: d5286df0:0xffffffffffffff92 Jun 28 00:45:27 carbon kernel: RSP: d5286e20:000055c8bfd77210 EFLAGS: 00000401 Jun 28 00:45:27 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 00:45:27 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 88147943 Jun 28 00:45:27 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 00:45:27 carbon kernel: kmem: usage 25448kB, limit 9007199254740988kB, failcnt 0 Jun 28 00:45:27 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:3040KB rss:495800KB rss_huge:4096KB shmem:0KB mapped_file:172KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:495800KB inactive_file:1440KB active_file:1216KB unevictable:0KB Jun 28 00:45:27 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 00:45:27 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 00:45:27 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 00:45:27 carbon kernel: [29093] 0 29093 49 4 3 2 0 0 s6-supervise Jun 28 00:45:27 carbon kernel: [15201] 99 15201 413492 123885 313 5 0 0 rslsync Jun 28 00:45:27 carbon kernel: Memory cgroup out of memory: Kill process 15201 (rslsync) score 947 or sacrifice child Jun 28 00:45:27 carbon kernel: Killed process 15201 (rslsync) total-vm:1653968kB, anon-rss:495540kB, file-rss:0kB, shmem-rss:0kB Jun 28 00:45:27 carbon kernel: oom_reaper: reaped process 15201 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 00:49:49 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 00:49:49 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 00:49:49 carbon kernel: CPU: 4 PID: 26688 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 00:49:49 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 00:49:49 carbon kernel: Call Trace: Jun 28 00:49:49 carbon kernel: dump_stack+0x5d/0x79 Jun 28 00:49:49 carbon kernel: dump_header+0x8f/0x214 Jun 28 00:49:49 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 00:49:49 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 00:49:49 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 00:49:49 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 00:49:49 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 00:49:49 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 00:49:49 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 00:49:49 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 00:49:49 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 00:49:49 carbon kernel: page_fault+0x45/0x50 Jun 28 00:49:49 carbon kernel: RIP: 97addea0:0xffffffffffffff92 Jun 28 00:49:49 carbon kernel: RSP: b8f2d000:00005556b8f2d008 EFLAGS: 000b799d Jun 28 00:49:49 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 00:49:49 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 88323075 Jun 28 00:49:49 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 00:49:49 carbon kernel: kmem: usage 26232kB, limit 9007199254740988kB, failcnt 0 Jun 28 00:49:49 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:2760KB rss:495296KB rss_huge:4096KB shmem:0KB mapped_file:116KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:495296KB inactive_file:1356KB active_file:1020KB unevictable:0KB Jun 28 00:49:49 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 00:49:49 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 00:49:49 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 00:49:49 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 00:49:49 carbon kernel: [26623] 99 26623 415421 123752 314 5 0 0 rslsync Jun 28 00:49:49 carbon kernel: Memory cgroup out of memory: Kill process 26623 (rslsync) score 946 or sacrifice child Jun 28 00:49:49 carbon kernel: Killed process 26623 (rslsync) total-vm:1661684kB, anon-rss:495008kB, file-rss:0kB, shmem-rss:0kB Jun 28 00:49:49 carbon kernel: oom_reaper: reaped process 26623 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 00:54:57 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 00:54:57 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 00:54:57 carbon kernel: CPU: 4 PID: 15864 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 00:54:57 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 00:54:57 carbon kernel: Call Trace: Jun 28 00:54:57 carbon kernel: dump_stack+0x5d/0x79 Jun 28 00:54:57 carbon kernel: dump_header+0x8f/0x214 Jun 28 00:54:57 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 00:54:57 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 00:54:57 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 00:54:57 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 00:54:57 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 00:54:57 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 00:54:57 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 00:54:57 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 00:54:57 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 00:54:57 carbon kernel: page_fault+0x45/0x50 Jun 28 00:54:57 carbon kernel: RIP: 4ebe42b0:0x5ec47 Jun 28 00:54:57 carbon kernel: RSP: 4f78bb78:000056124f78c008 EFLAGS: 56124f78c030 Jun 28 00:54:57 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 00:54:57 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 88537271 Jun 28 00:54:57 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 00:54:57 carbon kernel: kmem: usage 26276kB, limit 9007199254740988kB, failcnt 0 Jun 28 00:54:57 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:2812KB rss:495200KB rss_huge:4096KB shmem:0KB mapped_file:196KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:495200KB inactive_file:1276KB active_file:640KB unevictable:0KB Jun 28 00:54:57 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 00:54:57 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 00:54:57 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 00:54:57 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 00:54:57 carbon kernel: [15828] 99 15828 415671 123695 318 4 0 0 rslsync Jun 28 00:54:57 carbon kernel: Memory cgroup out of memory: Kill process 15828 (rslsync) score 946 or sacrifice child Jun 28 00:54:57 carbon kernel: Killed process 15828 (rslsync) total-vm:1662684kB, anon-rss:494780kB, file-rss:0kB, shmem-rss:0kB Jun 28 00:54:57 carbon kernel: oom_reaper: reaped process 15828 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 01:00:17 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 01:00:17 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 01:00:17 carbon kernel: CPU: 4 PID: 10427 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 01:00:17 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 01:00:17 carbon kernel: Call Trace: Jun 28 01:00:17 carbon kernel: dump_stack+0x5d/0x79 Jun 28 01:00:17 carbon kernel: dump_header+0x8f/0x214 Jun 28 01:00:17 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 01:00:17 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 01:00:17 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 01:00:17 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 01:00:17 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 01:00:17 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 01:00:17 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 01:00:17 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 01:00:17 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 01:00:17 carbon kernel: page_fault+0x45/0x50 Jun 28 01:00:17 carbon kernel: RIP: 70da7ea0:0xffffffffffffff92 Jun 28 01:00:17 carbon kernel: RSP: 12489000:000055f812489008 EFLAGS: 000ba0f7 Jun 28 01:00:17 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 01:00:17 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 88769712 Jun 28 01:00:17 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 01:00:17 carbon kernel: kmem: usage 26188kB, limit 9007199254740988kB, failcnt 0 Jun 28 01:00:17 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:3520KB rss:494580KB rss_huge:4096KB shmem:0KB mapped_file:28KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:494580KB inactive_file:1600KB active_file:1536KB unevictable:0KB Jun 28 01:00:17 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 01:00:17 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 01:00:17 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 01:00:17 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 01:00:17 carbon kernel: [ 8521] 99 8521 414364 123585 314 4 0 0 rslsync Jun 28 01:00:17 carbon kernel: Memory cgroup out of memory: Kill process 8521 (rslsync) score 945 or sacrifice child Jun 28 01:00:17 carbon kernel: Killed process 8521 (rslsync) total-vm:1657456kB, anon-rss:494340kB, file-rss:0kB, shmem-rss:0kB Jun 28 01:00:17 carbon kernel: oom_reaper: reaped process 8521 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 01:05:12 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 01:05:12 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 01:05:12 carbon kernel: CPU: 4 PID: 7341 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 01:05:12 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 01:05:12 carbon kernel: Call Trace: Jun 28 01:05:12 carbon kernel: dump_stack+0x5d/0x79 Jun 28 01:05:12 carbon kernel: dump_header+0x8f/0x214 Jun 28 01:05:12 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 01:05:12 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 01:05:12 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 01:05:12 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 01:05:12 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 01:05:12 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 01:05:12 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 01:05:12 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 01:05:12 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 01:05:12 carbon kernel: page_fault+0x45/0x50 Jun 28 01:05:12 carbon kernel: RIP: 70980ea0:0xffffffffffffff92 Jun 28 01:05:12 carbon kernel: RSP: a2b56b70:00001517a2b43250 EFLAGS: 0013983b Jun 28 01:05:12 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 01:05:12 carbon kernel: memory: usage 524292kB, limit 524288kB, failcnt 89021197 Jun 28 01:05:12 carbon kernel: memory+swap: usage 524292kB, limit 1048576kB, failcnt 0 Jun 28 01:05:12 carbon kernel: kmem: usage 26156kB, limit 9007199254740988kB, failcnt 0 Jun 28 01:05:12 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:2600KB rss:495536KB rss_huge:2048KB shmem:0KB mapped_file:52KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:495536KB inactive_file:1296KB active_file:792KB unevictable:0KB Jun 28 01:05:12 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 01:05:12 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 01:05:12 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 01:05:12 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 01:05:12 carbon kernel: [ 2295] 99 2295 415823 123826 317 4 0 0 rslsync Jun 28 01:05:12 carbon kernel: Memory cgroup out of memory: Kill process 2295 (rslsync) score 947 or sacrifice child Jun 28 01:05:12 carbon kernel: Killed process 2295 (rslsync) total-vm:1663292kB, anon-rss:495304kB, file-rss:0kB, shmem-rss:0kB Jun 28 01:05:12 carbon kernel: oom_reaper: reaped process 2295 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 01:14:41 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 01:14:41 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 01:14:41 carbon kernel: CPU: 4 PID: 26651 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 01:14:41 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 01:14:41 carbon kernel: Call Trace: Jun 28 01:14:41 carbon kernel: dump_stack+0x5d/0x79 Jun 28 01:14:41 carbon kernel: dump_header+0x8f/0x214 Jun 28 01:14:41 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 01:14:41 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 01:14:41 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 01:14:41 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 01:14:41 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 01:14:41 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 01:14:41 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 01:14:41 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 01:14:41 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 01:14:41 carbon kernel: page_fault+0x45/0x50 Jun 28 01:14:41 carbon kernel: RIP: 22862ea0:0xffffffffffffff92 Jun 28 01:14:41 carbon kernel: RSP: 7df12000:000055b17df12008 EFLAGS: 000a5a85 Jun 28 01:14:41 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 01:14:41 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 89329862 Jun 28 01:14:41 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 01:14:41 carbon kernel: kmem: usage 25852kB, limit 9007199254740988kB, failcnt 0 Jun 28 01:14:41 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:4200KB rss:494236KB rss_huge:204800KB shmem:0KB mapped_file:672KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:494236KB inactive_file:2132KB active_file:1684KB unevictable:0KB Jun 28 01:14:41 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 01:14:41 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 01:14:41 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 01:14:41 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 01:14:41 carbon kernel: [26554] 99 26554 398175 123337 314 4 0 0 rslsync Jun 28 01:14:41 carbon kernel: Memory cgroup out of memory: Kill process 26554 (rslsync) score 943 or sacrifice child Jun 28 01:14:41 carbon kernel: Killed process 26554 (rslsync) total-vm:1592700kB, anon-rss:493348kB, file-rss:0kB, shmem-rss:0kB Jun 28 01:14:41 carbon kernel: oom_reaper: reaped process 26554 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 01:31:14 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 01:31:14 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 01:31:14 carbon kernel: CPU: 4 PID: 8304 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 01:31:14 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 01:31:14 carbon kernel: Call Trace: Jun 28 01:31:14 carbon kernel: dump_stack+0x5d/0x79 Jun 28 01:31:14 carbon kernel: dump_header+0x8f/0x214 Jun 28 01:31:14 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 01:31:14 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 01:31:14 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 01:31:14 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 01:31:14 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 01:31:14 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 01:31:14 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 01:31:14 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 01:31:14 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 01:31:14 carbon kernel: page_fault+0x45/0x50 Jun 28 01:31:14 carbon kernel: RIP: 000b:0x80000001 Jun 28 01:31:14 carbon kernel: RSP: 92de210:000055a6092de438 EFLAGS: 147790273ec4 Jun 28 01:31:14 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 01:31:14 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 89802376 Jun 28 01:31:14 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 01:31:14 carbon kernel: kmem: usage 25836kB, limit 9007199254740988kB, failcnt 0 Jun 28 01:31:14 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:4536KB rss:493916KB rss_huge:4096KB shmem:0KB mapped_file:668KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:493916KB inactive_file:2204KB active_file:1820KB unevictable:0KB Jun 28 01:31:14 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 01:31:14 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 01:31:14 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 01:31:14 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 01:31:14 carbon kernel: [ 8248] 99 8248 412743 123356 312 5 0 0 rslsync Jun 28 01:31:14 carbon kernel: Memory cgroup out of memory: Kill process 8248 (rslsync) score 943 or sacrifice child Jun 28 01:31:14 carbon kernel: Killed process 8248 (rslsync) total-vm:1650972kB, anon-rss:493424kB, file-rss:0kB, shmem-rss:0kB Jun 28 01:31:14 carbon kernel: oom_reaper: reaped process 8248 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 01:34:46 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 01:34:46 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 01:34:46 carbon kernel: CPU: 4 PID: 24673 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 01:34:46 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 01:34:46 carbon kernel: Call Trace: Jun 28 01:34:46 carbon kernel: dump_stack+0x5d/0x79 Jun 28 01:34:46 carbon kernel: dump_header+0x8f/0x214 Jun 28 01:34:46 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 01:34:46 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 01:34:46 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 01:34:46 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 01:34:46 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 01:34:46 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 01:34:46 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 01:34:46 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 01:34:46 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 01:34:46 carbon kernel: page_fault+0x45/0x50 Jun 28 01:34:46 carbon kernel: RIP: 000b:0x80000001 Jun 28 01:34:46 carbon kernel: RSP: e77a1210:00005623e77a1438 EFLAGS: 1462934b6ec4 Jun 28 01:34:46 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 01:34:46 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 89908932 Jun 28 01:34:46 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 01:34:46 carbon kernel: kmem: usage 29840kB, limit 9007199254740988kB, failcnt 0 Jun 28 01:34:46 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:2640KB rss:491808KB rss_huge:0KB shmem:0KB mapped_file:564KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:491808KB inactive_file:1188KB active_file:1068KB unevictable:0KB Jun 28 01:34:46 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 01:34:46 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 01:34:46 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 01:34:46 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 01:34:46 carbon kernel: [24660] 99 24660 397862 122866 313 5 0 0 rslsync Jun 28 01:34:46 carbon kernel: Memory cgroup out of memory: Kill process 24660 (rslsync) score 939 or sacrifice child Jun 28 01:34:46 carbon kernel: Killed process 24660 (rslsync) total-vm:1591448kB, anon-rss:491420kB, file-rss:44kB, shmem-rss:0kB Jun 28 01:34:46 carbon kernel: oom_reaper: reaped process 24660 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 01:38:07 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 01:38:07 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 01:38:07 carbon kernel: CPU: 4 PID: 9804 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 01:38:07 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 01:38:07 carbon kernel: Call Trace: Jun 28 01:38:07 carbon kernel: dump_stack+0x5d/0x79 Jun 28 01:38:07 carbon kernel: dump_header+0x8f/0x214 Jun 28 01:38:07 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 01:38:07 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 01:38:07 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 01:38:07 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 01:38:07 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 01:38:07 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 01:38:07 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 01:38:07 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 01:38:07 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 01:38:07 carbon kernel: page_fault+0x45/0x50 Jun 28 01:38:07 carbon kernel: RIP: 8c4fcdf0:0xffffffffffffff92 Jun 28 01:38:07 carbon kernel: RSP: 8c4fce20:000055d927290210 EFLAGS: 0000018f Jun 28 01:38:07 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 01:38:07 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 90042554 Jun 28 01:38:07 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 01:38:07 carbon kernel: kmem: usage 34532kB, limit 9007199254740988kB, failcnt 0 Jun 28 01:38:07 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:1624KB rss:488132KB rss_huge:2048KB shmem:0KB mapped_file:536KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:488132KB inactive_file:800KB active_file:568KB unevictable:0KB Jun 28 01:38:07 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 01:38:07 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 01:38:07 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 01:38:07 carbon kernel: [29093] 0 29093 49 4 3 2 0 0 s6-supervise Jun 28 01:38:07 carbon kernel: [ 9750] 99 9750 413685 121928 311 5 0 0 rslsync Jun 28 01:38:07 carbon kernel: Memory cgroup out of memory: Kill process 9750 (rslsync) score 932 or sacrifice child Jun 28 01:38:07 carbon kernel: Killed process 9750 (rslsync) total-vm:1654740kB, anon-rss:487712kB, file-rss:0kB, shmem-rss:0kB Jun 28 01:38:07 carbon kernel: oom_reaper: reaped process 9750 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 01:48:48 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 01:48:48 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 01:48:48 carbon kernel: CPU: 4 PID: 26513 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 01:48:48 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 01:48:48 carbon kernel: Call Trace: Jun 28 01:48:48 carbon kernel: dump_stack+0x5d/0x79 Jun 28 01:48:48 carbon kernel: dump_header+0x8f/0x214 Jun 28 01:48:48 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 01:48:48 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 01:48:48 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 01:48:48 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 01:48:48 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 01:48:48 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 01:48:48 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 01:48:48 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 01:48:48 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 01:48:48 carbon kernel: page_fault+0x45/0x50 Jun 28 01:48:48 carbon kernel: RIP: e80008d0:0x1493127e0e50 Jun 28 01:48:48 carbon kernel: RSP: 403b5a0:0000000000000020 EFLAGS: 1492e80008c0 Jun 28 01:48:48 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 01:48:48 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 90359794 Jun 28 01:48:48 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 01:48:48 carbon kernel: kmem: usage 31048kB, limit 9007199254740988kB, failcnt 0 Jun 28 01:48:48 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:2264KB rss:490976KB rss_huge:212992KB shmem:0KB mapped_file:4KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:490976KB inactive_file:1308KB active_file:188KB unevictable:0KB Jun 28 01:48:48 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 01:48:48 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 01:48:48 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 01:48:48 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 01:48:48 carbon kernel: [26450] 99 26450 409163 122613 322 5 0 0 rslsync Jun 28 01:48:48 carbon kernel: Memory cgroup out of memory: Kill process 26450 (rslsync) score 937 or sacrifice child Jun 28 01:48:48 carbon kernel: Killed process 26450 (rslsync) total-vm:1636652kB, anon-rss:490452kB, file-rss:0kB, shmem-rss:0kB Jun 28 01:48:48 carbon kernel: oom_reaper: reaped process 26450 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 01:53:33 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 01:53:33 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 01:53:33 carbon kernel: CPU: 4 PID: 16266 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 01:53:33 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 01:53:33 carbon kernel: Call Trace: Jun 28 01:53:33 carbon kernel: dump_stack+0x5d/0x79 Jun 28 01:53:33 carbon kernel: dump_header+0x8f/0x214 Jun 28 01:53:33 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 01:53:33 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 01:53:33 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 01:53:33 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 01:53:33 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 01:53:33 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 01:53:33 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 01:53:33 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 01:53:33 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 01:53:33 carbon kernel: page_fault+0x45/0x50 Jun 28 01:53:33 carbon kernel: RIP: 6c868df0:0xffffffffffffff92 Jun 28 01:53:33 carbon kernel: RSP: 6c868e20:00005595f73e7eb0 EFLAGS: 00000233 Jun 28 01:53:33 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 01:53:33 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 90523735 Jun 28 01:53:33 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 01:53:33 carbon kernel: kmem: usage 25624kB, limit 9007199254740988kB, failcnt 0 Jun 28 01:53:33 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:2012KB rss:496652KB rss_huge:4096KB shmem:0KB mapped_file:124KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:496652KB inactive_file:964KB active_file:416KB unevictable:0KB Jun 28 01:53:33 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 01:53:33 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 01:53:33 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 01:53:33 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 01:53:33 carbon kernel: [16100] 99 16100 415366 124076 316 4 0 0 rslsync Jun 28 01:53:33 carbon kernel: Memory cgroup out of memory: Kill process 16100 (rslsync) score 949 or sacrifice child Jun 28 01:53:33 carbon kernel: Killed process 16100 (rslsync) total-vm:1661464kB, anon-rss:496304kB, file-rss:0kB, shmem-rss:0kB Jun 28 01:53:33 carbon kernel: oom_reaper: reaped process 16100 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 01:57:09 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 01:57:09 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 01:57:09 carbon kernel: CPU: 4 PID: 6931 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 01:57:09 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 01:57:09 carbon kernel: Call Trace: Jun 28 01:57:09 carbon kernel: dump_stack+0x5d/0x79 Jun 28 01:57:09 carbon kernel: dump_header+0x8f/0x214 Jun 28 01:57:09 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 01:57:09 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 01:57:09 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 01:57:09 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 01:57:09 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 01:57:09 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 01:57:09 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 01:57:09 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 01:57:09 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 01:57:09 carbon kernel: page_fault+0x45/0x50 Jun 28 01:57:09 carbon kernel: RIP: 1c0008d0:0x14d7442e59c0 Jun 28 01:57:09 carbon kernel: RSP: 3803b570:0000000000000020 EFLAGS: 14d71c0008c0 Jun 28 01:57:09 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 01:57:09 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 90701520 Jun 28 01:57:09 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 01:57:09 carbon kernel: kmem: usage 25852kB, limit 9007199254740988kB, failcnt 0 Jun 28 01:57:09 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:1368KB rss:497068KB rss_huge:2048KB shmem:0KB mapped_file:136KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:497068KB inactive_file:656KB active_file:488KB unevictable:0KB Jun 28 01:57:09 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 01:57:09 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 01:57:09 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 01:57:09 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 01:57:09 carbon kernel: [ 6861] 99 6861 416221 124176 317 5 0 0 rslsync Jun 28 01:57:09 carbon kernel: Memory cgroup out of memory: Kill process 6861 (rslsync) score 949 or sacrifice child Jun 28 01:57:09 carbon kernel: Killed process 6861 (rslsync) total-vm:1664884kB, anon-rss:496704kB, file-rss:0kB, shmem-rss:0kB Jun 28 01:57:09 carbon kernel: oom_reaper: reaped process 6861 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 02:02:10 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 02:02:10 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 02:02:10 carbon kernel: CPU: 4 PID: 24660 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 02:02:10 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 02:02:10 carbon kernel: Call Trace: Jun 28 02:02:10 carbon kernel: dump_stack+0x5d/0x79 Jun 28 02:02:10 carbon kernel: dump_header+0x8f/0x214 Jun 28 02:02:10 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 02:02:10 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 02:02:10 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 02:02:10 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 02:02:10 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 02:02:10 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 02:02:10 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 02:02:10 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 02:02:10 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 02:02:10 carbon kernel: page_fault+0x45/0x50 Jun 28 02:02:10 carbon kernel: RIP: 22bcddf0:0xffffffffffffff92 Jun 28 02:02:10 carbon kernel: RSP: 22bcde20:0000560f3e4e3eb0 EFLAGS: 00000257 Jun 28 02:02:10 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 02:02:10 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 90882859 Jun 28 02:02:10 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 02:02:10 carbon kernel: kmem: usage 26688kB, limit 9007199254740988kB, failcnt 0 Jun 28 02:02:10 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:5116KB rss:492484KB rss_huge:10240KB shmem:0KB mapped_file:572KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:492484KB inactive_file:2660KB active_file:2200KB unevictable:0KB Jun 28 02:02:10 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 02:02:10 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 02:02:10 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 02:02:10 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 02:02:10 carbon kernel: [24647] 99 24647 414850 123033 313 5 0 0 rslsync Jun 28 02:02:10 carbon kernel: Memory cgroup out of memory: Kill process 24647 (rslsync) score 941 or sacrifice child Jun 28 02:02:10 carbon kernel: Killed process 24647 (rslsync) total-vm:1659400kB, anon-rss:492132kB, file-rss:0kB, shmem-rss:0kB Jun 28 02:02:10 carbon kernel: oom_reaper: reaped process 24647 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 02:06:14 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 02:06:14 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 02:06:14 carbon kernel: CPU: 4 PID: 19803 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 02:06:14 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 02:06:14 carbon kernel: Call Trace: Jun 28 02:06:14 carbon kernel: dump_stack+0x5d/0x79 Jun 28 02:06:14 carbon kernel: dump_header+0x8f/0x214 Jun 28 02:06:14 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 02:06:14 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 02:06:14 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 02:06:14 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 02:06:14 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 02:06:14 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 02:06:14 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 02:06:14 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 02:06:14 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 02:06:14 carbon kernel: page_fault+0x45/0x50 Jun 28 02:06:14 carbon kernel: RIP: 4a196ea0:0xffffffffffffff92 Jun 28 02:06:14 carbon kernel: RSP: 7ab0b170:000014d47a568e10 EFLAGS: 00139891 Jun 28 02:06:14 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 02:06:14 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 91154308 Jun 28 02:06:14 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 02:06:14 carbon kernel: kmem: usage 26356kB, limit 9007199254740988kB, failcnt 0 Jun 28 02:06:14 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:4244KB rss:493688KB rss_huge:2048KB shmem:0KB mapped_file:16KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:493688KB inactive_file:2556KB active_file:1432KB unevictable:0KB Jun 28 02:06:14 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 02:06:14 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 02:06:14 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 02:06:14 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 02:06:14 carbon kernel: [16882] 99 16882 415388 123376 315 5 0 0 rslsync Jun 28 02:06:14 carbon kernel: Memory cgroup out of memory: Kill process 16882 (rslsync) score 943 or sacrifice child Jun 28 02:06:14 carbon kernel: Killed process 16882 (rslsync) total-vm:1661552kB, anon-rss:493504kB, file-rss:0kB, shmem-rss:0kB Jun 28 02:06:14 carbon kernel: oom_reaper: reaped process 16882 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 02:12:53 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 02:12:53 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 02:12:53 carbon kernel: CPU: 4 PID: 4414 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 02:12:53 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 02:12:53 carbon kernel: Call Trace: Jun 28 02:12:53 carbon kernel: dump_stack+0x5d/0x79 Jun 28 02:12:53 carbon kernel: dump_header+0x8f/0x214 Jun 28 02:12:53 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 02:12:53 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 02:12:53 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 02:12:53 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 02:12:53 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 02:12:53 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 02:12:53 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 02:12:53 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 02:12:53 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 02:12:53 carbon kernel: page_fault+0x45/0x50 Jun 28 02:12:53 carbon kernel: RIP: 000b:0x80000001 Jun 28 02:12:53 carbon kernel: RSP: b6631210:000055c0b6631438 EFLAGS: 147ffa71dec4 Jun 28 02:12:53 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 02:12:53 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 91467136 Jun 28 02:12:53 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 02:12:53 carbon kernel: kmem: usage 26412kB, limit 9007199254740988kB, failcnt 0 Jun 28 02:12:53 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:4552KB rss:493324KB rss_huge:2048KB shmem:0KB mapped_file:696KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:493324KB inactive_file:2872KB active_file:1680KB unevictable:0KB Jun 28 02:12:53 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 02:12:53 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 02:12:53 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 02:12:53 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 02:12:53 carbon kernel: [ 4344] 99 4344 398535 123232 314 4 0 0 rslsync Jun 28 02:12:53 carbon kernel: Memory cgroup out of memory: Kill process 4344 (rslsync) score 942 or sacrifice child Jun 28 02:12:53 carbon kernel: Killed process 4344 (rslsync) total-vm:1594140kB, anon-rss:492928kB, file-rss:0kB, shmem-rss:0kB Jun 28 02:12:53 carbon kernel: oom_reaper: reaped process 4344 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 02:16:39 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 02:16:39 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 02:16:39 carbon kernel: CPU: 4 PID: 4723 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 02:16:39 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 02:16:39 carbon kernel: Call Trace: Jun 28 02:16:39 carbon kernel: dump_stack+0x5d/0x79 Jun 28 02:16:39 carbon kernel: dump_header+0x8f/0x214 Jun 28 02:16:39 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 02:16:39 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 02:16:39 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 02:16:39 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 02:16:39 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 02:16:39 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 02:16:39 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 02:16:39 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 02:16:39 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 02:16:39 carbon kernel: page_fault+0x45/0x50 Jun 28 02:16:39 carbon kernel: RIP: 3c13bea0:0xffffffffffffff92 Jun 28 02:16:39 carbon kernel: RSP: 3e676000:0000561c3e676008 EFLAGS: 000b071d Jun 28 02:16:39 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 02:16:39 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 91718911 Jun 28 02:16:39 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 02:16:39 carbon kernel: kmem: usage 26048kB, limit 9007199254740988kB, failcnt 0 Jun 28 02:16:39 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:2600KB rss:495640KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:495640KB inactive_file:1708KB active_file:124KB unevictable:0KB Jun 28 02:16:39 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 02:16:39 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 02:16:39 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 02:16:39 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 02:16:39 carbon kernel: [ 4714] 99 4714 416351 123808 316 4 0 0 rslsync Jun 28 02:16:39 carbon kernel: Memory cgroup out of memory: Kill process 4714 (rslsync) score 947 or sacrifice child Jun 28 02:16:39 carbon kernel: Killed process 4714 (rslsync) total-vm:1665404kB, anon-rss:495232kB, file-rss:0kB, shmem-rss:0kB Jun 28 02:16:39 carbon kernel: oom_reaper: reaped process 4714 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 02:22:54 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 02:22:54 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 02:22:54 carbon kernel: CPU: 4 PID: 23252 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 02:22:54 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 02:22:54 carbon kernel: Call Trace: Jun 28 02:22:54 carbon kernel: dump_stack+0x5d/0x79 Jun 28 02:22:54 carbon kernel: dump_header+0x8f/0x214 Jun 28 02:22:54 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 02:22:54 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 02:22:54 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 02:22:54 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 02:22:54 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 02:22:54 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 02:22:54 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 02:22:54 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 02:22:54 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 02:22:54 carbon kernel: page_fault+0x45/0x50 Jun 28 02:22:54 carbon kernel: RIP: e0dbadf0:0xffffffffffffff92 Jun 28 02:22:54 carbon kernel: RSP: e0dbae20:000055f7ecee4eb0 EFLAGS: 000002ed Jun 28 02:22:54 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 02:22:54 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 91980457 Jun 28 02:22:54 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 02:22:54 carbon kernel: kmem: usage 24736kB, limit 9007199254740988kB, failcnt 0 Jun 28 02:22:54 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:2460KB rss:497092KB rss_huge:65536KB shmem:0KB mapped_file:8KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:497092KB inactive_file:716KB active_file:56KB unevictable:0KB Jun 28 02:22:54 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 02:22:54 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 02:22:54 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 02:22:54 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 02:22:54 carbon kernel: [23248] 99 23248 416203 124162 317 5 0 0 rslsync Jun 28 02:22:54 carbon kernel: Memory cgroup out of memory: Kill process 23248 (rslsync) score 949 or sacrifice child Jun 28 02:22:54 carbon kernel: Killed process 23248 (rslsync) total-vm:1664812kB, anon-rss:496648kB, file-rss:0kB, shmem-rss:0kB Jun 28 02:22:54 carbon kernel: oom_reaper: reaped process 23248 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 02:42:04 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 02:42:04 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 02:42:04 carbon kernel: CPU: 4 PID: 21847 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 02:42:04 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 02:42:04 carbon kernel: Call Trace: Jun 28 02:42:04 carbon kernel: dump_stack+0x5d/0x79 Jun 28 02:42:04 carbon kernel: dump_header+0x8f/0x214 Jun 28 02:42:04 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 02:42:04 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 02:42:04 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 02:42:04 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 02:42:04 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 02:42:04 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 02:42:04 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 02:42:04 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 02:42:04 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 02:42:04 carbon kernel: page_fault+0x45/0x50 Jun 28 02:42:04 carbon kernel: RIP: 17892ea0:0xffffffffffffff92 Jun 28 02:42:04 carbon kernel: RSP: b7400000:000055b7b7400008 EFLAGS: 000acbbf Jun 28 02:42:04 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 02:42:04 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 92232746 Jun 28 02:42:04 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 02:42:04 carbon kernel: kmem: usage 24676kB, limit 9007199254740988kB, failcnt 0 Jun 28 02:42:04 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:4536KB rss:495076KB rss_huge:73728KB shmem:0KB mapped_file:324KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:495076KB inactive_file:2656KB active_file:1496KB unevictable:0KB Jun 28 02:42:04 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 02:42:04 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 02:42:04 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 02:42:04 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 02:42:04 carbon kernel: [21834] 99 21834 386685 123690 320 5 0 0 rslsync Jun 28 02:42:04 carbon kernel: Memory cgroup out of memory: Kill process 21834 (rslsync) score 946 or sacrifice child Jun 28 02:42:04 carbon kernel: Killed process 21834 (rslsync) total-vm:1546740kB, anon-rss:494760kB, file-rss:0kB, shmem-rss:0kB Jun 28 02:42:04 carbon kernel: oom_reaper: reaped process 21834 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 02:51:45 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 02:51:45 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 02:51:45 carbon kernel: CPU: 4 PID: 19823 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 02:51:45 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 02:51:45 carbon kernel: Call Trace: Jun 28 02:51:45 carbon kernel: dump_stack+0x5d/0x79 Jun 28 02:51:45 carbon kernel: dump_header+0x8f/0x214 Jun 28 02:51:45 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 02:51:45 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 02:51:45 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 02:51:45 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 02:51:45 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 02:51:45 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 02:51:45 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 02:51:45 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 02:51:45 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 02:51:45 carbon kernel: page_fault+0x45/0x50 Jun 28 02:51:45 carbon kernel: RIP: 2371ddf0:0xffffffffffffff92 Jun 28 02:51:45 carbon kernel: RSP: 2371de20:000055686e277eb0 EFLAGS: 00000475 Jun 28 02:51:45 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 02:51:45 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 92506715 Jun 28 02:51:45 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 02:51:45 carbon kernel: kmem: usage 24424kB, limit 9007199254740988kB, failcnt 0 Jun 28 02:51:45 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:3176KB rss:496688KB rss_huge:4096KB shmem:0KB mapped_file:4KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:496688KB inactive_file:2188KB active_file:92KB unevictable:0KB Jun 28 02:51:45 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 02:51:45 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 02:51:45 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 02:51:45 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 02:51:45 carbon kernel: [18993] 99 18993 416959 124101 316 5 0 0 rslsync Jun 28 02:51:45 carbon kernel: Memory cgroup out of memory: Kill process 18993 (rslsync) score 949 or sacrifice child Jun 28 02:51:45 carbon kernel: Killed process 18993 (rslsync) total-vm:1667836kB, anon-rss:496404kB, file-rss:0kB, shmem-rss:0kB Jun 28 02:51:45 carbon kernel: oom_reaper: reaped process 18993 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 03:03:49 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 03:03:49 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 03:03:49 carbon kernel: CPU: 4 PID: 1977 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 03:03:49 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 03:03:49 carbon kernel: Call Trace: Jun 28 03:03:49 carbon kernel: dump_stack+0x5d/0x79 Jun 28 03:03:49 carbon kernel: dump_header+0x8f/0x214 Jun 28 03:03:49 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 03:03:49 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 03:03:49 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 03:03:49 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 03:03:49 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 03:03:49 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 03:03:49 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 03:03:49 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 03:03:49 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 03:03:49 carbon kernel: page_fault+0x45/0x50 Jun 28 03:03:49 carbon kernel: RIP: 9496eea0:0xffffffffffffff92 Jun 28 03:03:49 carbon kernel: RSP: 76009000:0000555e76009008 EFLAGS: 00073aad Jun 28 03:03:49 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 03:03:49 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 92816515 Jun 28 03:03:49 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 03:03:49 carbon kernel: kmem: usage 25824kB, limit 9007199254740988kB, failcnt 0 Jun 28 03:03:49 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:2132KB rss:496332KB rss_huge:2048KB shmem:0KB mapped_file:4KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:496332KB inactive_file:948KB active_file:728KB unevictable:0KB Jun 28 03:03:49 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 03:03:49 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 03:03:49 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 03:03:49 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 03:03:49 carbon kernel: [ 1968] 99 1968 416443 124005 321 5 0 0 rslsync Jun 28 03:03:49 carbon kernel: Memory cgroup out of memory: Kill process 1968 (rslsync) score 948 or sacrifice child Jun 28 03:03:49 carbon kernel: Killed process 1968 (rslsync) total-vm:1665772kB, anon-rss:496020kB, file-rss:0kB, shmem-rss:0kB Jun 28 03:03:49 carbon kernel: oom_reaper: reaped process 1968 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 03:12:48 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0 Jun 28 03:12:48 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 03:12:48 carbon kernel: CPU: 4 PID: 29019 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 03:12:48 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 03:12:48 carbon kernel: Call Trace: Jun 28 03:12:48 carbon kernel: dump_stack+0x5d/0x79 Jun 28 03:12:48 carbon kernel: dump_header+0x8f/0x214 Jun 28 03:12:48 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 03:12:48 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 03:12:48 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 03:12:48 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 03:12:48 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 03:12:48 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 03:12:48 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 03:12:48 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 03:12:48 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 03:12:48 carbon kernel: page_fault+0x45/0x50 Jun 28 03:12:48 carbon kernel: RIP: 48000078:0x2710 Jun 28 03:12:48 carbon kernel: RSP: 48000020:0000000000000051 EFLAGS: 154148000078 Jun 28 03:12:48 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 03:12:48 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 93143289 Jun 28 03:12:48 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 03:12:48 carbon kernel: kmem: usage 25908kB, limit 9007199254740988kB, failcnt 0 Jun 28 03:12:48 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:2676KB rss:495704KB rss_huge:2048KB shmem:0KB mapped_file:1212KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:495704KB inactive_file:1432KB active_file:1244KB unevictable:0KB Jun 28 03:12:48 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 03:12:48 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 03:12:48 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 03:12:48 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 03:12:48 carbon kernel: [28940] 99 28940 399873 124040 314 5 0 0 rslsync Jun 28 03:12:48 carbon kernel: Memory cgroup out of memory: Kill process 28940 (rslsync) score 948 or sacrifice child Jun 28 03:12:48 carbon kernel: Killed process 28940 (rslsync) total-vm:1599492kB, anon-rss:495412kB, file-rss:748kB, shmem-rss:0kB Jun 28 03:12:49 carbon kernel: oom_reaper: reaped process 28940 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 03:22:52 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 03:22:52 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 03:22:52 carbon kernel: CPU: 4 PID: 31660 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 03:22:52 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 03:22:52 carbon kernel: Call Trace: Jun 28 03:22:52 carbon kernel: dump_stack+0x5d/0x79 Jun 28 03:22:52 carbon kernel: dump_header+0x8f/0x214 Jun 28 03:22:52 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 03:22:52 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 03:22:52 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 03:22:52 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 03:22:52 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 03:22:52 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 03:22:52 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 03:22:52 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 03:22:52 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 03:22:52 carbon kernel: page_fault+0x45/0x50 Jun 28 03:22:52 carbon kernel: RIP: f99c7010:0x14ba1e6c6b8c Jun 28 03:22:52 carbon kernel: RSP: 0000:000000000000004c EFLAGS: 00100000 Jun 28 03:22:52 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 03:22:52 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 93489453 Jun 28 03:22:52 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 03:22:52 carbon kernel: kmem: usage 26984kB, limit 9007199254740988kB, failcnt 0 Jun 28 03:22:52 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:2080KB rss:495224KB rss_huge:2048KB shmem:0KB mapped_file:144KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:495224KB inactive_file:1040KB active_file:528KB unevictable:0KB Jun 28 03:22:52 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 03:22:52 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 03:22:52 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 03:22:52 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 03:22:52 carbon kernel: [ 8679] 99 8679 416510 123706 326 5 0 0 rslsync Jun 28 03:22:52 carbon kernel: Memory cgroup out of memory: Kill process 8679 (rslsync) score 946 or sacrifice child Jun 28 03:22:52 carbon kernel: Killed process 8679 (rslsync) total-vm:1666040kB, anon-rss:494824kB, file-rss:0kB, shmem-rss:0kB Jun 28 03:22:52 carbon kernel: oom_reaper: reaped process 8679 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 03:34:03 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 03:34:03 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 03:34:03 carbon kernel: CPU: 4 PID: 26870 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 03:34:03 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 03:34:03 carbon kernel: Call Trace: Jun 28 03:34:03 carbon kernel: dump_stack+0x5d/0x79 Jun 28 03:34:03 carbon kernel: dump_header+0x8f/0x214 Jun 28 03:34:03 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 03:34:03 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 03:34:03 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 03:34:03 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 03:34:03 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 03:34:03 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 03:34:03 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 03:34:03 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 03:34:03 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 03:34:03 carbon kernel: page_fault+0x45/0x50 Jun 28 03:34:03 carbon kernel: RIP: 2836df0:0xffffffffffffff92 Jun 28 03:34:03 carbon kernel: RSP: 2836e20:000055d590320eb0 EFLAGS: 00000521 Jun 28 03:34:03 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 03:34:03 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 93848523 Jun 28 03:34:03 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 03:34:03 carbon kernel: kmem: usage 24944kB, limit 9007199254740988kB, failcnt 0 Jun 28 03:34:03 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:3720KB rss:495624KB rss_huge:4096KB shmem:0KB mapped_file:104KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:495624KB inactive_file:2224KB active_file:728KB unevictable:0KB Jun 28 03:34:03 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 03:34:03 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 03:34:03 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 03:34:03 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 03:34:03 carbon kernel: [25792] 99 25792 416564 123795 316 5 0 0 rslsync Jun 28 03:34:03 carbon kernel: Memory cgroup out of memory: Kill process 25792 (rslsync) score 946 or sacrifice child Jun 28 03:34:03 carbon kernel: Killed process 25792 (rslsync) total-vm:1666256kB, anon-rss:495180kB, file-rss:0kB, shmem-rss:0kB Jun 28 03:34:03 carbon kernel: oom_reaper: reaped process 25792 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 03:50:18 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 03:50:18 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 03:50:18 carbon kernel: CPU: 4 PID: 16311 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 03:50:18 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 03:50:18 carbon kernel: Call Trace: Jun 28 03:50:18 carbon kernel: dump_stack+0x5d/0x79 Jun 28 03:50:18 carbon kernel: dump_header+0x8f/0x214 Jun 28 03:50:18 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 03:50:18 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 03:50:18 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 03:50:18 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 03:50:18 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 03:50:18 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 03:50:18 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 03:50:18 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 03:50:18 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 03:50:18 carbon kernel: page_fault+0x45/0x50 Jun 28 03:50:18 carbon kernel: RIP: 5b6b62b0:0x61568 Jun 28 03:50:18 carbon kernel: RSP: 5c25e000:000055cb5c25e008 EFLAGS: 55cb5c25e030 Jun 28 03:50:18 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 03:50:18 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 94198522 Jun 28 03:50:18 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 03:50:18 carbon kernel: kmem: usage 23992kB, limit 9007199254740988kB, failcnt 0 Jun 28 03:50:18 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:3616KB rss:496680KB rss_huge:110592KB shmem:0KB mapped_file:64KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:496680KB inactive_file:2340KB active_file:508KB unevictable:0KB Jun 28 03:50:18 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 03:50:18 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 03:50:18 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 03:50:18 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 03:50:18 carbon kernel: [16148] 99 16148 400294 124098 319 5 0 0 rslsync Jun 28 03:50:18 carbon kernel: Memory cgroup out of memory: Kill process 16148 (rslsync) score 949 or sacrifice child Jun 28 03:50:18 carbon kernel: Killed process 16148 (rslsync) total-vm:1601176kB, anon-rss:496392kB, file-rss:0kB, shmem-rss:0kB Jun 28 03:50:18 carbon kernel: oom_reaper: reaped process 16148 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 04:06:16 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 04:06:16 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 04:06:16 carbon kernel: CPU: 4 PID: 30915 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 04:06:16 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 04:06:16 carbon kernel: Call Trace: Jun 28 04:06:16 carbon kernel: dump_stack+0x5d/0x79 Jun 28 04:06:16 carbon kernel: dump_header+0x8f/0x214 Jun 28 04:06:16 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 04:06:16 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 04:06:16 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 04:06:16 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 04:06:16 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 04:06:16 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f Jun 28 04:06:16 carbon kernel: pagefault_out_of_memory+0x1a/0x47 Jun 28 04:06:16 carbon kernel: __do_page_fault+0x38b/0x3e7 Jun 28 04:06:16 carbon kernel: ? page_fault+0x2f/0x50 Jun 28 04:06:16 carbon kernel: page_fault+0x45/0x50 Jun 28 04:06:16 carbon kernel: RIP: 7d40425b:0x1548deae8000 Jun 28 04:06:16 carbon kernel: RSP: e40f9db0:0000000000000000 EFLAGS: 55b97c36d2f0 Jun 28 04:06:16 carbon kernel: Task in /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 killed as a result of limit of /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 Jun 28 04:06:16 carbon kernel: memory: usage 524288kB, limit 524288kB, failcnt 94524158 Jun 28 04:06:16 carbon kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0 Jun 28 04:06:16 carbon kernel: kmem: usage 29584kB, limit 9007199254740988kB, failcnt 0 Jun 28 04:06:16 carbon kernel: Memory cgroup stats for /docker/b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6: cache:3048KB rss:491656KB rss_huge:2048KB shmem:0KB mapped_file:768KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:491656KB inactive_file:1980KB active_file:812KB unevictable:0KB Jun 28 04:06:16 carbon kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jun 28 04:06:16 carbon kernel: [28708] 0 28708 49 1 3 2 0 0 s6-svscan Jun 28 04:06:16 carbon kernel: [28851] 0 28851 49 1 3 2 0 0 s6-supervise Jun 28 04:06:16 carbon kernel: [29093] 0 29093 49 3 3 2 0 0 s6-supervise Jun 28 04:06:16 carbon kernel: [30789] 99 30789 414401 122834 314 4 0 0 rslsync Jun 28 04:06:16 carbon kernel: Memory cgroup out of memory: Kill process 30789 (rslsync) score 939 or sacrifice child Jun 28 04:06:16 carbon kernel: Killed process 30789 (rslsync) total-vm:1657604kB, anon-rss:491268kB, file-rss:68kB, shmem-rss:0kB Jun 28 04:06:16 carbon kernel: oom_reaper: reaped process 30789 (rslsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 28 04:18:40 carbon crond[1705]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null Jun 28 04:24:06 carbon kernel: rslsync invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Jun 28 04:24:06 carbon kernel: rslsync cpuset=b7c5cbaf24a85980d441b9c894d547a241d60a730b22706084ccd1b725a3f1e6 mems_allowed=0 Jun 28 04:24:06 carbon kernel: CPU: 4 PID: 13918 Comm: rslsync Not tainted 4.14.49-unRAID #1 Jun 28 04:24:06 carbon kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015 Jun 28 04:24:06 carbon kernel: Call Trace: Jun 28 04:24:06 carbon kernel: dump_stack+0x5d/0x79 Jun 28 04:24:06 carbon kernel: dump_header+0x8f/0x214 Jun 28 04:24:06 carbon kernel: ? mem_cgroup_scan_tasks+0x2c/0x94 Jun 28 04:24:06 carbon kernel: oom_kill_process+0x82/0x37d Jun 28 04:24:06 carbon kernel: out_of_memory+0x3a3/0x3c3 Jun 28 04:24:06 carbon kernel: mem_cgroup_out_of_memory+0x38/0x51 Jun 28 04:24:06 carbon kernel: mem_cgroup_oom_synchronize+0x282/0x296 Jun 28 04:24:06 carbon kernel: ? __mem_cgroup_insert_exceeded+0x7f/0x7f
July 1, 20188 yr Author @Squid In Settings > Disk Settings I have Default spin down delay set to Never and Enable spinup groups set to No. Sometimes it does seem to behave like the disks are spun down (long initial delay on file access, etc.) but they always show green in the dashboard. @1812 I do have the containers' RAM limited, but only after I started investigating the stalls. The issue was noticed from the outset after getting everything installed. These are the current settings, but I'll obviously have to expand at least Resilio: glances --cpuset-cpus=0,4 --memory=512M openvpn-as --cpuset-cpus=0,4 --memory=512M resilio-sync --cpuset-cpus=4 --memory=512M nzbget --cpuset-cpus=1,5 --memory=2G binhex-delugevpn --cpuset-cpus=1,5 --memory=1G radarr --cpuset-cpus=1,5 --memory=1G sonarr --cpuset-cpus=1,5 --memory=1G plex --cpuset-cpus=2,3,6,7 --memory=4G handbrake --cpuset-cpus=2,3,6,7 --memory=2G tautulli --cpuset-cpus=7 --memory=512M
July 11, 20187 yr Author Any other thoughts? I have the same sporadic file system delay issues even if I have only a single container running.
Archived
This topic is now archived and is closed to further replies.