August 17, 20178 yr I had one of my unraid dockers "crash" on 8/15 ~12am... My SageTV docker app used to record OTA TV shows effectively crashed and became unresponsive. At the time, it was recording 2 HD shows and thereby writing about 12GB/hour to the cache pool. Looking into it, I noticed that the mover is in fact also running at 12am and was trying to move 20-40GB of data (previously recorded HD shows) from the cache to the array. My mover is set to run every 8 hours (12am, 8am, 4pm). The mover has created all kinds of exceptions in the syslog between 12:00 and 12:03am, and it eventually killed one of my java processes (pid=14906. I believe this would be the SageTV java process). It seems to be using 6GB? I have 32 GB of RAM on the machine... Here are some excerpts of the log below, and I also attached the full log. Has anyone seen out of memory errors like this when the mover is running or anyone know what might be wrong or what I can do about it? Thanks! Aug 15 00:00:01 unraid root: mover started Aug 15 00:00:01 unraid root: moving "sagemedia" to array Aug 15 00:00:01 unraid root: .d..t...... ./ Aug 15 00:00:01 unraid root: .d..t...... sagemedia/ Aug 15 00:00:01 unraid root: .d..t...... sagemedia/tv/ Aug 15 00:00:01 unraid root: >f+++++++++ sagemedia/tv/WildKratts-S03E02-WheretheBisonRoam-7816684-0.mpg Aug 15 00:00:25 unraid kernel: warn_alloc: 1069190 callbacks suppressed Aug 15 00:00:25 unraid kernel: java: page allocation stalls for 10985ms, order:0, mode:0x34200ca(GFP_HIGHUSER_MOVABLE|__GFP_WRITE) Aug 15 00:00:25 unraid kernel: CPU: 4 PID: 19630 Comm: java Not tainted 4.9.30-unRAID #1 Aug 15 00:00:25 unraid kernel: Hardware name: System manufacturer System Product Name/PRIME H270-PRO, BIOS 0323 01/04/2017 Aug 15 00:00:25 unraid kernel: ffffc9000c59ba68 ffffffff813a4a1b 0000000000000001 0000000000000000 Aug 15 00:00:25 unraid kernel: ffffc9000c59baf8 ffffffff810cb5b1 034200ca810c9d8d ffffffff8193d4e2 Aug 15 00:00:25 unraid kernel: ffffc9000c59ba90 0000000000000010 ffffc9000c59bb08 ffffc9000c59baa8 Aug 15 00:00:25 unraid kernel: Call Trace: Aug 15 00:00:25 unraid kernel: [<ffffffff813a4a1b>] dump_stack+0x61/0x7e Aug 15 00:00:25 unraid kernel: [<ffffffff810cb5b1>] warn_alloc+0x102/0x116 Aug 15 00:00:25 unraid kernel: [<ffffffff810cbb67>] __alloc_pages_nodemask+0x541/0xc71 Aug 15 00:00:25 unraid kernel: [<ffffffff8167c00e>] ? __schedule+0x2b1/0x46a Aug 15 00:00:25 unraid kernel: [<ffffffff8107c0fd>] ? wake_up_bit+0x25/0x25 Aug 15 00:00:25 unraid kernel: [<ffffffff81245966>] ? fuse_request_free+0x3b/0x3e Aug 15 00:00:25 unraid kernel: [<ffffffff81102d82>] alloc_pages_current+0xbe/0xe8 Aug 15 00:00:25 unraid kernel: [<ffffffff810c4d78>] __page_cache_alloc+0x89/0x9f Aug 15 00:00:25 unraid kernel: [<ffffffff810c4ecc>] pagecache_get_page+0x13e/0x1e6 Aug 15 00:00:25 unraid kernel: [<ffffffff810c4f8f>] grab_cache_page_write_begin+0x1b/0x32 Aug 15 00:00:25 unraid kernel: [<ffffffff8124d8d7>] fuse_perform_write+0x186/0x484 Aug 15 00:00:25 unraid kernel: [<ffffffff81138461>] ? file_remove_privs+0x42/0x98 Aug 15 00:00:25 unraid kernel: [<ffffffff81153c96>] ? fsnotify_destroy_event+0x5d/0x64 Aug 15 00:00:25 unraid kernel: [<ffffffff811553eb>] ? inotify_handle_event+0xe2/0x100 ... ... Aug 15 00:03:08 unraid kernel: Mem-Info: Aug 15 00:03:08 unraid kernel: active_anon:967739 inactive_anon:16972 isolated_anon:0 Aug 15 00:03:08 unraid kernel: active_file:6219808 inactive_file:634419 isolated_file:2592 Aug 15 00:03:08 unraid kernel: unevictable:0 dirty:634162 writeback:1385 unstable:0 Aug 15 00:03:08 unraid kernel: slab_reclaimable:114589 slab_unreclaimable:140487 Aug 15 00:03:08 unraid kernel: mapped:21252 shmem:135948 pagetables:6557 bounce:0 Aug 15 00:03:08 unraid kernel: free:65981 free_pcp:103 free_cma:0 Aug 15 00:03:08 unraid kernel: Node 0 active_anon:3870956kB inactive_anon:67888kB active_file:24879232kB inactive_file:2537676kB unevictable:0kB isolated(anon):0kB isolated(file):10368kB mapped:85008kB dirty:2536648kB writeback:5540kB shmem:543792kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 2609152kB writeback_tmp:0kB unstable:0kB pages_scanned:475716 all_unreclaimable? no Aug 15 00:03:08 unraid kernel: Node 0 DMA free:15896kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15980kB managed:15896kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Aug 15 00:03:08 unraid kernel: lowmem_reserve[]: 0 2906 31944 31944 Aug 15 00:03:08 unraid kernel: Node 0 DMA32 free:126392kB min:12288kB low:15360kB high:18432kB active_anon:405944kB inactive_anon:0kB active_file:2286824kB inactive_file:235532kB unevictable:0kB writepending:234380kB present:3136448kB managed:3126452kB mlocked:0kB slab_reclaimable:57748kB slab_unreclaimable:5744kB kernel_stack:116kB pagetables:308kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Aug 15 00:03:08 unraid kernel: lowmem_reserve[]: 0 0 29038 29038 Aug 15 00:03:08 unraid kernel: Node 0 Normal free:121636kB min:122808kB low:153508kB high:184208kB active_anon:3465012kB inactive_anon:67888kB active_file:22592408kB inactive_file:2302676kB unevictable:0kB writepending:2307808kB present:30261248kB managed:29735764kB mlocked:0kB slab_reclaimable:400608kB slab_unreclaimable:556204kB kernel_stack:15244kB pagetables:25920kB bounce:0kB free_pcp:412kB local_pcp:0kB free_cma:0kB Aug 15 00:03:08 unraid kernel: lowmem_reserve[]: 0 0 0 0 Aug 15 00:03:08 unraid kernel: Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (U) 3*4096kB (M) = 15896kB Aug 15 00:03:08 unraid kernel: Node 0 DMA32: 136*4kB (UMEH) 163*8kB (UMEH) 70*16kB (UEH) 34*32kB (UMEH) 185*64kB (UMEH) 159*128kB (UMEH) 83*256kB (UMEH) 41*512kB (UMEH) 19*1024kB (UME) 2*2048kB (M) 6*4096kB (M) = 126616kB Aug 15 00:03:08 unraid kernel: Node 0 Normal: 4508*4kB (ME) 11750*8kB (UMEH) 182*16kB (UME) 0*32kB 0*64kB 0*128kB 1*256kB (H) 1*512kB (H) 0*1024kB 1*2048kB (H) 1*4096kB (H) = 121856kB Aug 15 00:03:08 unraid kernel: 6992822 total pagecache pages Aug 15 00:03:08 unraid kernel: 0 pages in swap cache Aug 15 00:03:08 unraid kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 15 00:03:08 unraid kernel: Free swap = 0kB Aug 15 00:03:08 unraid kernel: Total swap = 0kB Aug 15 00:03:08 unraid kernel: 8353419 pages RAM Aug 15 00:03:08 unraid kernel: 0 pages HighMem/MovableOnly Aug 15 00:03:08 unraid kernel: 133891 pages reserved Aug 15 00:03:08 unraid kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Aug 15 00:03:08 unraid kernel: [ 1390] 0 1390 6700 743 14 3 0 -1000 udevd Aug 15 00:03:08 unraid kernel: [ 1564] 0 1564 59436 689 25 4 0 0 rsyslogd Aug 15 00:03:08 unraid kernel: [ 1699] 81 1699 4900 60 14 3 0 0 dbus-daemon Aug 15 00:03:08 unraid kernel: [ 1707] 1 1707 3342 512 11 3 0 0 rpcbind Aug 15 00:03:08 unraid kernel: [ 1713] 32 1713 5352 1467 15 3 0 0 rpc.statd Aug 15 00:03:08 unraid kernel: [ 1723] 0 1723 1619 395 8 3 0 0 inetd Aug 15 00:03:08 unraid kernel: [ 1732] 0 1732 6120 762 17 3 0 -1000 sshd Aug 15 00:03:08 unraid kernel: [ 1746] 44 1746 26121 1281 26 3 0 0 ntpd Aug 15 00:03:08 unraid kernel: [ 1753] 0 1753 1095 29 7 3 0 0 acpid Aug 15 00:03:08 unraid kernel: [ 1762] 0 1762 1621 413 8 3 0 0 crond Aug 15 00:03:08 unraid kernel: [ 1764] 0 1764 1618 25 8 3 0 0 atd Aug 15 00:03:08 unraid kernel: [ 1790] 0 1790 2383 592 9 3 0 0 cpuload Aug 15 00:03:08 unraid kernel: [ 7810] 0 7810 75777 37884 137 3 0 0 php Aug 15 00:03:08 unraid kernel: [ 8701] 0 8701 22477 950 17 3 0 0 emhttp Aug 15 00:03:08 unraid kernel: [ 8702] 0 8702 1627 413 8 3 0 0 agetty Aug 15 00:03:08 unraid kernel: [ 8703] 0 8703 1627 390 8 3 0 0 agetty Aug 15 00:03:08 unraid kernel: [ 8704] 0 8704 1627 410 8 3 0 0 agetty Aug 15 00:03:08 unraid kernel: [ 8705] 0 8705 1627 399 7 3 0 0 agetty Aug 15 00:03:08 unraid kernel: [ 8706] 0 8706 1627 390 8 3 0 0 agetty Aug 15 00:03:08 unraid kernel: [ 8707] 0 8707 1627 391 7 3 0 0 agetty Aug 15 00:03:08 unraid kernel: [12491] 0 12491 55386 1472 106 3 0 0 nmbd Aug 15 00:03:08 unraid kernel: [12493] 0 12493 75119 3820 144 3 0 0 smbd Aug 15 00:03:08 unraid kernel: [12495] 0 12495 73584 1167 137 3 0 0 smbd-notifyd Aug 15 00:03:08 unraid kernel: [12496] 0 12496 73591 1332 137 3 0 0 cleanupd Aug 15 00:03:08 unraid kernel: [12500] 0 12500 68258 1990 130 3 0 0 winbindd Aug 15 00:03:08 unraid kernel: [12501] 0 12501 68150 1698 131 3 0 0 winbindd Aug 15 00:03:08 unraid kernel: [12515] 61 12515 8622 766 21 3 0 0 avahi-daemon Aug 15 00:03:08 unraid kernel: [12516] 61 12516 8557 63 21 3 0 0 avahi-daemon Aug 15 00:03:08 unraid kernel: [12525] 0 12525 3185 27 11 3 0 0 avahi-dnsconfd Aug 15 00:03:08 unraid kernel: [12915] 0 12915 68258 985 129 3 0 0 winbindd Aug 15 00:03:08 unraid kernel: [13255] 0 13255 55287 302 18 3 0 0 shfs Aug 15 00:03:08 unraid kernel: [13265] 0 13265 535312 32529 132 5 0 0 shfs Aug 15 00:03:08 unraid kernel: [13302] 0 13302 2382 606 9 3 0 0 diskload Aug 15 00:03:08 unraid kernel: [13445] 0 13445 1618 26 8 3 0 0 atd Aug 15 00:03:08 unraid kernel: [13446] 0 13446 2906 640 9 3 0 0 sh Aug 15 00:03:08 unraid kernel: [13447] 0 13447 43376 5464 74 3 0 0 startBackground Aug 15 00:03:08 unraid kernel: [13460] 0 13460 2907 644 10 3 0 0 sh Aug 15 00:03:08 unraid kernel: [13461] 0 13461 2907 687 10 3 0 0 script Aug 15 00:03:08 unraid kernel: [13465] 0 13465 1098 194 7 3 0 0 tail Aug 15 00:03:08 unraid kernel: [13532] 0 13532 225961 8847 74 5 0 -500 dockerd Aug 15 00:03:08 unraid kernel: [13547] 0 13547 126958 2699 37 6 0 -500 docker-containe Aug 15 00:03:08 unraid kernel: [13744] 0 13744 105820 784 27 6 0 -500 docker-containe Aug 15 00:03:08 unraid kernel: [13767] 0 13767 8081 1260 19 3 0 0 my_init Aug 15 00:03:08 unraid kernel: [13825] 0 13825 89436 780 25 6 0 -500 docker-containe Aug 15 00:03:08 unraid kernel: [13844] 0 13844 49 1 3 2 0 0 s6-svscan Aug 15 00:03:08 unraid kernel: [13938] 0 13938 49 1 3 2 0 0 s6-supervise Aug 15 00:03:08 unraid kernel: [14070] 0 14070 89724 775 25 6 0 -500 docker-containe Aug 15 00:03:08 unraid kernel: [14123] 0 14123 1126 18 7 3 0 0 sh Aug 15 00:03:08 unraid kernel: [14173] 0 14173 4548 104 13 3 0 0 noip.sh Aug 15 00:03:08 unraid kernel: [14192] 65534 14192 4273 46 13 3 0 0 noip2-x86_64 Aug 15 00:03:08 unraid kernel: [14225] 0 14225 5052 674 16 5 0 -500 docker-proxy Aug 15 00:03:08 unraid kernel: [14236] 0 14236 20923 703 16 5 0 -500 docker-proxy Aug 15 00:03:08 unraid kernel: [14246] 0 14246 4539 684 15 5 0 -500 docker-proxy Aug 15 00:03:08 unraid kernel: [14256] 0 14256 56703 1455 23 5 0 -500 docker-proxy Aug 15 00:03:08 unraid kernel: [14265] 0 14265 73052 775 25 5 0 -500 docker-containe Aug 15 00:03:08 unraid kernel: [14284] 0 14284 8532 1257 18 3 0 0 my_init Aug 15 00:03:08 unraid kernel: [14356] 0 14356 89500 787 25 5 0 -500 docker-containe Aug 15 00:03:08 unraid kernel: [14376] 0 14376 7269 1248 17 3 0 0 my_init Aug 15 00:03:08 unraid kernel: [14466] 0 14466 1098 21 7 3 0 0 runsvdir Aug 15 00:03:08 unraid kernel: [14467] 0 14467 1060 18 6 3 0 0 runsv Aug 15 00:03:08 unraid kernel: [14468] 0 14468 1060 18 7 3 0 0 runsv Aug 15 00:03:08 unraid kernel: [14469] 0 14469 1060 17 6 3 0 0 runsv Aug 15 00:03:08 unraid kernel: [14470] 0 14470 1060 17 7 3 0 0 runsv Aug 15 00:03:08 unraid kernel: [14471] 0 14471 1060 17 7 3 0 0 runsv Aug 15 00:03:08 unraid kernel: [14472] 0 14472 1060 17 7 3 0 0 runsv Aug 15 00:03:08 unraid kernel: [14473] 0 14473 46482 710 77 3 0 0 openbox Aug 15 00:03:08 unraid kernel: [14474] 0 14474 18114 400 39 3 0 0 syslog-ng Aug 15 00:03:08 unraid kernel: [14475] 0 14475 5320 56 14 3 0 0 run Aug 15 00:03:08 unraid kernel: [14476] 0 14476 1904 17 9 3 0 0 tail Aug 15 00:03:08 unraid kernel: [14477] 0 14477 5318 56 15 3 0 0 run Aug 15 00:03:08 unraid kernel: [14478] 0 14478 19076 3300 31 3 0 0 Xvnc Aug 15 00:03:08 unraid kernel: [14480] 0 14480 5329 76 15 3 0 0 bash Aug 15 00:03:08 unraid kernel: [14482] 0 14482 1699563 202924 566 9 0 0 java Aug 15 00:03:08 unraid kernel: [14508] 0 14508 24760 3365 49 3 0 0 python Aug 15 00:03:08 unraid kernel: [14656] 0 14656 5320 67 15 3 0 0 startapp.sh Aug 15 00:03:08 unraid kernel: [14783] 0 14783 49 1 3 2 0 0 s6-supervise Aug 15 00:03:08 unraid kernel: [14785] 0 14785 49 1 3 2 0 0 s6-supervise Aug 15 00:03:08 unraid kernel: [14786] 0 14786 49 1 3 2 0 0 s6-supervise Aug 15 00:03:08 unraid kernel: [14788] 106 14788 11231 128 26 3 0 0 avahi-daemon Aug 15 00:03:08 unraid kernel: [14789] 105 14789 10723 103 25 3 0 0 dbus-daemon Aug 15 00:03:08 unraid kernel: [14792] 99 14792 190996 17926 387 3 0 0 Plex Media Serv Aug 15 00:03:08 unraid kernel: [14829] 99 14829 444237 22233 148 5 0 0 Plex Script Hos Aug 15 00:03:08 unraid kernel: [14860] 99 14860 3206 59 11 3 0 0 startsagecore Aug 15 00:03:08 unraid kernel: [14872] 0 14872 1098 22 7 3 0 0 runsvdir Aug 15 00:03:08 unraid kernel: [14873] 0 14873 1060 17 7 3 0 0 runsv Aug 15 00:03:08 unraid kernel: [14874] 0 14874 1060 18 7 3 0 0 runsv Aug 15 00:03:08 unraid kernel: [14875] 0 14875 1060 17 6 3 0 0 runsv Aug 15 00:03:08 unraid kernel: [14876] 0 14876 1060 17 7 3 0 0 runsv Aug 15 00:03:08 unraid kernel: [14878] 0 14878 1904 18 9 3 0 0 tail Aug 15 00:03:08 unraid kernel: [14879] 0 14879 7318 60 19 3 0 0 cron Aug 15 00:03:08 unraid kernel: [14880] 0 14880 18114 404 38 3 0 0 syslog-ng Aug 15 00:03:08 unraid kernel: [14906] 99 14906 1666125 314068 843 9 0 0 java Aug 15 00:03:08 unraid kernel: [15088] 99 15088 86737 13555 147 3 0 0 Plex DLNA Serve Aug 15 00:03:08 unraid kernel: [15096] 99 15096 175075 579 62 3 0 0 Plex Tuner Serv Aug 15 00:03:08 unraid kernel: [15270] 0 15270 1082950 72118 327 8 0 0 java Aug 15 00:03:08 unraid kernel: [15360] 0 15360 18821 853 38 3 0 0 virtlockd Aug 15 00:03:08 unraid kernel: [15366] 0 15366 35734 983 41 3 0 0 virtlogd Aug 15 00:03:08 unraid kernel: [15382] 0 15382 54005 2967 75 3 0 0 libvirtd Aug 15 00:03:08 unraid kernel: [15510] 99 15510 4378 493 13 3 0 0 dnsmasq Aug 15 00:03:08 unraid kernel: [15511] 0 15511 4345 53 12 3 0 0 dnsmasq Aug 15 00:03:08 unraid kernel: [15651] 0 15651 1099 21 6 3 0 0 runsvdir Aug 15 00:03:08 unraid kernel: [15652] 0 15652 1061 18 7 3 0 0 runsv Aug 15 00:03:08 unraid kernel: [15653] 0 15653 1061 17 7 3 0 0 runsv Aug 15 00:03:08 unraid kernel: [15654] 0 15654 1061 17 7 3 0 0 runsv Aug 15 00:03:08 unraid kernel: [15655] 0 15655 5741 59 14 3 0 0 run Aug 15 00:03:08 unraid kernel: [15656] 0 15656 70062 39114 139 3 0 0 squeezeboxserve Aug 15 00:03:08 unraid kernel: [15658] 0 15658 2331 17 7 3 0 0 tail Aug 15 00:03:08 unraid kernel: [ 839] 1000 839 94791 5590 151 3 0 0 smbd Aug 15 00:03:08 unraid kernel: [13182] 0 13182 104077 6130 177 3 0 0 smbd Aug 15 00:03:08 unraid kernel: [ 7284] 99 7284 223937 8800 101 3 0 0 Plex Script Hos Aug 15 00:03:08 unraid kernel: [ 7376] 99 7376 222014 8872 96 4 0 0 Plex Script Hos Aug 15 00:03:08 unraid kernel: [12717] 99 12717 78857 46008 120 4 0 0 comskip Aug 15 00:03:08 unraid kernel: [ 7879] 99 7879 33345 1507 68 3 0 0 Plex Transcoder Aug 15 00:03:08 unraid kernel: [ 8790] 0 8790 2374 625 9 3 0 0 sh Aug 15 00:03:08 unraid kernel: [ 8793] 0 8793 2397 661 9 3 0 0 mover Aug 15 00:03:08 unraid kernel: [ 8794] 0 8794 1622 419 8 3 0 0 logger Aug 15 00:03:08 unraid kernel: [ 8801] 0 8801 1087 171 7 3 0 0 move Aug 15 00:03:08 unraid kernel: [ 8813] 0 8813 3039 575 10 3 0 0 rsync Aug 15 00:03:08 unraid kernel: [ 8814] 0 8814 2944 396 10 3 0 0 rsync Aug 15 00:03:08 unraid kernel: [ 8815] 0 8815 3009 345 10 3 0 0 rsync Aug 15 00:03:08 unraid kernel: [ 9502] 0 9502 1094 18 7 3 0 0 sleep Aug 15 00:03:08 unraid kernel: [ 9659] 99 9659 190996 18028 363 3 0 0 Plex Media Serv Aug 15 00:03:08 unraid kernel: [ 9770] 0 9770 29108 2130 46 3 0 0 php Aug 15 00:03:08 unraid kernel: [ 9779] 0 9779 2374 630 9 3 0 0 sh Aug 15 00:03:08 unraid kernel: [ 9784] 0 9784 5375 123 14 3 0 0 awk Aug 15 00:03:08 unraid kernel: Out of memory: Kill process 14906 (java) score 38 or sacrifice child Aug 15 00:03:08 unraid kernel: Killed process 14906 (java) total-vm:6664500kB, anon-rss:1256272kB, file-rss:0kB, shmem-rss:0kB Aug 15 00:03:08 unraid kernel: oom_reaper: reaped process 14906 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Aug 15 00:04:35 unraid root: .d..t...... sagemedia/tv/ Aug 15 00:04:35 unraid root: >f+++++++++ sagemedia/tv/WildKratts-S03E02-WheretheBisonRoam-7816684-0.log Aug 15 00:04:36 unraid root: >f+++++++++ sagemedia/tv/WildKratts-S03E02-WheretheBisonRoam-7816684-0.txt Aug 15 00:04:36 unraid root: .d..t...... sagemedia/tv/ syslog-20170811.18.10.04.358914876.txt
August 17, 20178 yr Community Expert Start here and see if this helps: It hasn't 'cured' every case but it has been successful for several folks.
August 22, 20178 yr Author Frank! Thanks a lot for this info.... I installed the "Tips and Tweaks" plugin and set disk cache settings as follows: vm.dirty_background_ratio = 1 % vm.dirty_ratio = 2 % I have not had an OOM situation since then in the last week or so. I'll report back if I see any OOM mover related errors this week, but for now, I think the issue is resolved. Thanks again! -Glen.
Archived
This topic is now archived and is closed to further replies.