Skip to content
View in the app

A better way to browse. Learn more.
Unraid

A full-screen app on your home screen with push notifications, badges and more.

To install this app on iOS and iPadOS
  1. Tap the Share icon in Safari
  2. Scroll the menu and tap Add to Home Screen.
  3. Tap Add in the top-right corner.
To install this app on Android
  1. Tap the 3-dot menu (⋮) in the top-right corner of the browser.
  2. Tap Add to Home screen or Install app.
  3. Confirm by tapping Install.
Unraid Summer Sale in live: 20% off Starter and Unleashed + buy one, get one 50% off

week of server uptime = out of memory

joyless
in User Customizations

Featured Replies

Hello folks,

this is 4th week in a row where unRAID is reporting it's out of memory and is killing python processes (usually SABnzb, sometimes Plex or Sickbeard). It's also 4th time in a row this happens when server uptime is 6 or 7 days, today SAB was killed and uptime is 6 days 1 hour. Restart will fix the issue for another week.

Will increasing RAM help? I'm currently running only 1GB, which I know is kinda low, but I also read somewhere that this is because of "low memory" and amount of RAM sticks doesn't really matter? What can I do about this?

 

Thanks!

 

Nov 22 10:53:56 UNRAID kernel: [ 850] 0 850 464 97 0 0 0 klogd
Nov 22 10:53:56 UNRAID kernel: [ 876] 0 876 511 102 0 0 0 dhcpcd
Nov 22 10:53:56 UNRAID kernel: [ 958] 1 958 490 119 0 0 0 rpc.portmap
Nov 22 10:53:56 UNRAID kernel: [ 962] 0 962 527 197 0 0 0 rpc.statd
Nov 22 10:53:56 UNRAID kernel: [ 972] 0 972 475 139 0 0 0 inetd
Nov 22 10:53:56 UNRAID kernel: [ 979] 0 979 2041 722 0 0 0 ntpd
Nov 22 10:53:56 UNRAID kernel: [ 986] 0 986 466 158 0 0 0 acpid
Nov 22 10:53:56 UNRAID kernel: [ 997] 81 997 619 210 0 0 0 dbus-daemon
Nov 22 10:53:56 UNRAID kernel: [ 1002] 0 1002 477 146 0 0 0 crond
Nov 22 10:53:56 UNRAID kernel: [ 1004] 0 1004 475 73 0 0 0 atd
Nov 22 10:53:56 UNRAID kernel: [11577] 0 11577 2973 376 0 0 0 emhttp
Nov 22 10:53:56 UNRAID kernel: [11596] 0 11596 464 133 0 0 0 agetty
Nov 22 10:53:56 UNRAID kernel: [11597] 0 11597 464 133 0 0 0 agetty
Nov 22 10:53:56 UNRAID kernel: [11598] 0 11598 464 133 0 0 0 agetty
Nov 22 10:53:56 UNRAID kernel: [11599] 0 11599 464 133 0 0 0 agetty
Nov 22 10:53:56 UNRAID kernel: [11600] 0 11600 464 133 0 0 0 agetty
Nov 22 10:53:56 UNRAID kernel: [11601] 0 11601 464 132 0 0 0 agetty
Nov 22 10:53:56 UNRAID kernel: [11633] 0 11633 560 180 0 -17 -1000 udevd
Nov 22 10:53:56 UNRAID kernel: [11634] 0 11634 560 165 0 -17 -1000 udevd
Nov 22 10:53:56 UNRAID kernel: [11662] 0 11662 32433 784 0 0 0 shfs
Nov 22 10:53:56 UNRAID kernel: [11706] 99 11706 22416 9163 0 0 0 python
Nov 22 10:53:56 UNRAID kernel: [11978] 99 11978 84422 32214 0 0 0 python
Nov 22 10:53:56 UNRAID kernel: [12051] 99 12051 56400 12601 0 0 0 python
Nov 22 10:53:56 UNRAID kernel: [16681] 0 16681 1099 279 0 0 0 cnid_metad
Nov 22 10:53:56 UNRAID kernel: [16682] 0 16682 1461 452 0 0 0 afpd
Nov 22 10:53:56 UNRAID kernel: [16692] 61 16692 761 414 0 0 0 avahi-daemon
Nov 22 10:53:56 UNRAID kernel: [16693] 61 16693 728 100 0 0 0 avahi-daemon
Nov 22 10:53:56 UNRAID kernel: [16701] 0 16701 509 130 0 0 0 avahi-dnsconfd
Nov 22 10:53:56 UNRAID kernel: [29168] 999 29168 92084 9625 0 0 0 Plex Media Serv
Nov 22 10:53:56 UNRAID kernel: [29248] 999 29248 46342 3705 0 0 0 Plex DLNA Serve
Nov 22 10:53:56 UNRAID kernel: [ 5469] 999 5469 50820 9171 0 0 0 python
Nov 22 10:53:56 UNRAID kernel: [17353] 0 17353 1709 477 0 0 0 in.telnetd
Nov 22 10:53:56 UNRAID kernel: [17354] 0 17354 1085 447 0 0 0 bash
Nov 22 10:53:56 UNRAID kernel: [20243] 0 20243 2466 587 0 0 0 afpd
Nov 22 10:53:56 UNRAID kernel: [20262] 0 20262 6469 659 0 0 0 cnid_dbd
Nov 22 10:53:56 UNRAID kernel: [21197] 999 21197 24729 14468 0 0 0 Plex Media Scan
Nov 22 10:53:56 UNRAID kernel: [21369] 999 21369 55918 6803 0 0 0 python
Nov 22 10:53:56 UNRAID kernel: [21471] 999 21471 35051 5880 0 0 0 python
Nov 22 10:53:56 UNRAID kernel: Out of memory: Kill process 11978 (python) score 125 or sacrifice child
Nov 22 10:53:56 UNRAID kernel: Killed process 11978 (python) total-vm:337688kB, anon-rss:124748kB, file-rss:4108kB

  • Quote

    Increasing RAM will just make it take longer to fail. Attach the syslog after the next time it happens. Could be the log is filling RAM due to too many error entries or an add-on has a memory leak.

  • Quote

    Did you try to enable swap file?

     

    I had the same problem with YAMJ until yesterday (and had enough "free" memory while this happened) until I enabled a swap file.

     

  • Quote
    • Author

    @dgaschk

    syslog for today below.

    i browsed through past 3 days, but only renew lease logs, spindowns, plex updates and afp restarts, nothing else

     

    @theone

    no, i didn't. i'll look into it, but i can't see anything related to swap file in standard configuration, so i'm guessing this is a addon?

     

    Nov 22 00:31:12 UNRAID kernel: mdcmd (199): spindown 0
Nov 22 00:44:36 UNRAID kernel: mdcmd (200): spindown 1
Nov 22 01:07:03 UNRAID afpd[21659]: dsi_stream_read: len:0, unexpected EOF
Nov 22 01:40:54 UNRAID kernel: mdcmd (201): spindown 0
Nov 22 01:40:55 UNRAID kernel: mdcmd (202): spindown 2
Nov 22 02:33:04 UNRAID kernel: mdcmd (203): spindown 1
Nov 22 09:36:30 UNRAID dhcpcd[876]: eth0: renewing lease of 192.168.1.111
Nov 22 09:36:30 UNRAID dhcpcd[876]: eth0: acknowledged 192.168.1.111 from 192.168.1.1
Nov 22 09:36:30 UNRAID dhcpcd[876]: eth0: leased 192.168.1.111 for 86400 seconds
Nov 22 09:37:25 UNRAID in.telnetd[17353]: connect from 192.168.1.101 (192.168.1.101)
Nov 22 09:37:25 UNRAID login[17354]: ROOT LOGIN  on '/dev/pts/0' from '192.168.1.101'
Nov 22 09:46:44 UNRAID in.telnetd[17918]: connect from 192.168.1.101 (192.168.1.101)
Nov 22 09:46:44 UNRAID login[17919]: ROOT LOGIN  on '/dev/pts/1' from '192.168.1.101'
Nov 22 10:32:15 UNRAID kernel: mdcmd (204): spindown 0
Nov 22 10:32:15 UNRAID kernel: mdcmd (205): spindown 1
Nov 22 10:53:56 UNRAID kernel: shfs invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Nov 22 10:53:56 UNRAID kernel: Pid: 20364, comm: shfs Not tainted 3.4.11-unRAID #1
Nov 22 10:53:56 UNRAID kernel: Call Trace:
Nov 22 10:53:56 UNRAID kernel:  [] T.619+0x56/0x159
Nov 22 10:53:56 UNRAID kernel:  [] ? zone_watermark_ok+0x1e/0x25
Nov 22 10:53:56 UNRAID kernel:  [] ? ___ratelimit+0xaf/0xc4
Nov 22 10:53:56 UNRAID kernel:  [] T.618+0x45/0x217
Nov 22 10:53:56 UNRAID kernel:  [] ? has_ns_capability_noaudit+0x10/0x18
Nov 22 10:53:56 UNRAID kernel:  [] ? has_capability_noaudit+0xf/0x11
Nov 22 10:53:56 UNRAID kernel:  [] out_of_memory+0x21c/0x29e
Nov 22 10:53:56 UNRAID kernel:  [] __alloc_pages_nodemask+0x40b/0x4a5
Nov 22 10:53:56 UNRAID kernel:  [] T.925+0x2ff/0x39d
Nov 22 10:53:56 UNRAID kernel:  [] generic_file_aio_read+0x1bf/0x1ef
Nov 22 10:53:56 UNRAID kernel:  [] do_sync_read+0x94/0xcf
Nov 22 10:53:56 UNRAID kernel:  [] vfs_read+0x88/0xfa
Nov 22 10:53:56 UNRAID kernel:  [] ? do_sync_write+0xcf/0xcf
Nov 22 10:53:56 UNRAID kernel:  [] sys_pread64+0x43/0x5c
Nov 22 10:53:56 UNRAID kernel:  [] syscall_call+0x7/0xb
Nov 22 10:53:56 UNRAID kernel:  [] ? alloc_node_mem_map+0x70/0x8a
Nov 22 10:53:56 UNRAID kernel: Mem-Info:
Nov 22 10:53:56 UNRAID kernel: DMA per-cpu:
Nov 22 10:53:56 UNRAID kernel: CPU    0: hi:    0, btch:   1 usd:   0
Nov 22 10:53:56 UNRAID kernel: Normal per-cpu:
Nov 22 10:53:56 UNRAID kernel: CPU    0: hi:  186, btch:  31 usd:   0
Nov 22 10:53:56 UNRAID kernel: HighMem per-cpu:
Nov 22 10:53:56 UNRAID kernel: CPU    0: hi:   42, btch:   7 usd:   0
Nov 22 10:53:56 UNRAID kernel: active_anon:92666 inactive_anon:19 isolated_anon:0
Nov 22 10:53:56 UNRAID kernel:  active_file:3481 inactive_file:3489 isolated_file:64
Nov 22 10:53:56 UNRAID kernel:  unevictable:139420 dirty:0 writeback:0 unstable:0
Nov 22 10:53:56 UNRAID kernel:  free:2954 slab_reclaimable:5396 slab_unreclaimable:2693
Nov 22 10:53:56 UNRAID kernel:  mapped:7149 shmem:29 pagetables:582 bounce:0
Nov 22 10:53:56 UNRAID kernel: DMA free:4024kB min:64kB low:80kB high:96kB active_anon:9760kB inactive_anon:0kB active_file:36kB inactive_file:72kB unevictable:1920kB isolated(anon):0kB isolated(file):0kB present:15804kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:40kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:44kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:448 all_unreclaimable? yes
Nov 22 10:53:56 UNRAID kernel: lowmem_reserve[]: 0 869 992 992
Nov 22 10:53:56 UNRAID kernel: Normal free:7664kB min:3736kB low:4668kB high:5604kB active_anon:318288kB inactive_anon:0kB active_file:13888kB inactive_file:13884kB unevictable:472824kB isolated(anon):0kB isolated(file):256kB present:890008kB mlocked:0kB dirty:0kB writeback:0kB mapped:19948kB shmem:0kB slab_reclaimable:21544kB slab_unreclaimable:10772kB kernel_stack:1592kB pagetables:1768kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:58220 all_unreclaimable? yes
Nov 22 10:53:56 UNRAID kernel: lowmem_reserve[]: 0 0 983 983
Nov 22 10:53:56 UNRAID kernel: HighMem free:128kB min:128kB low:260kB high:392kB active_anon:42616kB inactive_anon:76kB active_file:0kB inactive_file:0kB unevictable:82936kB isolated(anon):0kB isolated(file):0kB present:125928kB mlocked:0kB dirty:0kB writeback:0kB mapped:8648kB shmem:116kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:516kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov 22 10:53:56 UNRAID kernel: lowmem_reserve[]: 0 0 0 0
Nov 22 10:53:56 UNRAID kernel: DMA: 2*4kB 20*8kB 15*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 4024kB
Nov 22 10:53:56 UNRAID kernel: Normal: 366*4kB 231*8kB 38*16kB 7*32kB 3*64kB 4*128kB 3*256kB 0*512kB 2*1024kB 0*2048kB 0*4096kB = 7664kB
Nov 22 10:53:56 UNRAID kernel: HighMem: 24*4kB 3*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 136kB
Nov 22 10:53:56 UNRAID kernel: 146483 total pagecache pages
Nov 22 10:53:56 UNRAID kernel: 0 pages in swap cache
Nov 22 10:53:56 UNRAID kernel: Swap cache stats: add 0, delete 0, find 0/0
Nov 22 10:53:56 UNRAID kernel: Free swap  = 0kB
Nov 22 10:53:56 UNRAID kernel: Total swap = 0kB
Nov 22 10:53:56 UNRAID kernel: 260064 pages RAM
Nov 22 10:53:56 UNRAID kernel: 31730 pages HighMem
Nov 22 10:53:56 UNRAID kernel: 3536 pages reserved
Nov 22 10:53:56 UNRAID kernel: 26942 pages shared
Nov 22 10:53:56 UNRAID kernel: 239007 pages non-shared
Nov 22 10:53:56 UNRAID kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Nov 22 10:53:56 UNRAID kernel: [  635]     0   635      561      211   0     -17         -1000 udevd
Nov 22 10:53:56 UNRAID kernel: [  846]     0   846      477      152   0       0             0 syslogd
Nov 22 10:53:56 UNRAID kernel: [  850]     0   850      464       97   0       0             0 klogd
Nov 22 10:53:56 UNRAID kernel: [  876]     0   876      511      102   0       0             0 dhcpcd
Nov 22 10:53:56 UNRAID kernel: [  958]     1   958      490      119   0       0             0 rpc.portmap
Nov 22 10:53:56 UNRAID kernel: [  962]     0   962      527      197   0       0             0 rpc.statd
Nov 22 10:53:56 UNRAID kernel: [  972]     0   972      475      139   0       0             0 inetd
Nov 22 10:53:56 UNRAID kernel: [  979]     0   979     2041      722   0       0             0 ntpd
Nov 22 10:53:56 UNRAID kernel: [  986]     0   986      466      158   0       0             0 acpid
Nov 22 10:53:56 UNRAID kernel: [  997]    81   997      619      210   0       0             0 dbus-daemon
Nov 22 10:53:56 UNRAID kernel: [ 1002]     0  1002      477      146   0       0             0 crond
Nov 22 10:53:56 UNRAID kernel: [ 1004]     0  1004      475       73   0       0             0 atd
Nov 22 10:53:56 UNRAID kernel: [11577]     0 11577     2973      376   0       0             0 emhttp
Nov 22 10:53:56 UNRAID kernel: [11596]     0 11596      464      133   0       0             0 agetty
Nov 22 10:53:56 UNRAID kernel: [11597]     0 11597      464      133   0       0             0 agetty
Nov 22 10:53:56 UNRAID kernel: [11598]     0 11598      464      133   0       0             0 agetty
Nov 22 10:53:56 UNRAID kernel: [11599]     0 11599      464      133   0       0             0 agetty
Nov 22 10:53:56 UNRAID kernel: [11600]     0 11600      464      133   0       0             0 agetty
Nov 22 10:53:56 UNRAID kernel: [11601]     0 11601      464      132   0       0             0 agetty
Nov 22 10:53:56 UNRAID kernel: [11633]     0 11633      560      180   0     -17         -1000 udevd
Nov 22 10:53:56 UNRAID kernel: [11634]     0 11634      560      165   0     -17         -1000 udevd
Nov 22 10:53:56 UNRAID kernel: [11662]     0 11662    32433      784   0       0             0 shfs
Nov 22 10:53:56 UNRAID kernel: [11706]    99 11706    22416     9163   0       0             0 python
Nov 22 10:53:56 UNRAID kernel: [11978]    99 11978    84422    32214   0       0             0 python
Nov 22 10:53:56 UNRAID kernel: [12051]    99 12051    56400    12601   0       0             0 python
Nov 22 10:53:56 UNRAID kernel: [16681]     0 16681     1099      279   0       0             0 cnid_metad
Nov 22 10:53:56 UNRAID kernel: [16682]     0 16682     1461      452   0       0             0 afpd
Nov 22 10:53:56 UNRAID kernel: [16692]    61 16692      761      414   0       0             0 avahi-daemon
Nov 22 10:53:56 UNRAID kernel: [16693]    61 16693      728      100   0       0             0 avahi-daemon
Nov 22 10:53:56 UNRAID kernel: [16701]     0 16701      509      130   0       0             0 avahi-dnsconfd
Nov 22 10:53:56 UNRAID kernel: [29168]   999 29168    92084     9625   0       0             0 Plex Media Serv
Nov 22 10:53:56 UNRAID kernel: [29248]   999 29248    46342     3705   0       0             0 Plex DLNA Serve
Nov 22 10:53:56 UNRAID kernel: [ 5469]   999  5469    50820     9171   0       0             0 python
Nov 22 10:53:56 UNRAID kernel: [17353]     0 17353     1709      477   0       0             0 in.telnetd
Nov 22 10:53:56 UNRAID kernel: [17354]     0 17354     1085      447   0       0             0 bash
Nov 22 10:53:56 UNRAID kernel: [20243]     0 20243     2466      587   0       0             0 afpd
Nov 22 10:53:56 UNRAID kernel: [20262]     0 20262     6469      659   0       0             0 cnid_dbd
Nov 22 10:53:56 UNRAID kernel: [21197]   999 21197    24729    14468   0       0             0 Plex Media Scan
Nov 22 10:53:56 UNRAID kernel: [21369]   999 21369    55918     6803   0       0             0 python
Nov 22 10:53:56 UNRAID kernel: [21471]   999 21471    35051     5880   0       0             0 python
Nov 22 10:53:56 UNRAID kernel: Out of memory: Kill process 11978 (python) score 125 or sacrifice child
Nov 22 10:53:56 UNRAID kernel: Killed process 11978 (python) total-vm:337688kB, anon-rss:124748kB, file-rss:4108kB

  • Quote
    • Author

    vmstat and free output, didn't restart the server yet:

     

    root@UNRAID:~# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
0  0      0 107780  53472 635176    0    0    70    29   16   16  3  1 89  6

     

    root@UNRAID:~# free
             total       used       free     shared    buffers     cached
Mem:       1026112     918456     107656          0      53536     635184
-/+ buffers/cache:     229736     796376
Swap:            0          0          0

  • Quote
    • Author

    I'm running latest RC8a, no unmenu.

    I'll give the plugin a try, thanks.

     

     

  • Quote

    vmstat and free output, didn't restart the server yet:

     

    root@UNRAID:~# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
0  0      0 107780  53472 635176    0    0    70    29   16   16  3  1 89  6

     

    root@UNRAID:~# free
             total       used       free     shared    buffers     cached
Mem:       1026112     918456     107656          0      53536     635184
-/+ buffers/cache:     229736     796376
Swap:            0          0          0

    you need to  use the -l option see the "low" memory usage.  It typically is what is exhausted first.

     

    free -l

     

    This issue is caused by your add-ons.

     

    Joe L.

  • Quote
    • Author

    This issue is caused by your add-ons.

    root@UNRAID:~# free -l
             total       used       free     shared    buffers     cached
Mem:       1026112    1006608      19504          0      61316     701848
Low:        899192     880024      19168
High:       126920     126584        336
-/+ buffers/cache:     243444     782668
Swap:            0          0          0

     

    so, is there anything I can do about it?

     

  • Quote

    Keep an eye on the free memory over time.  (You said you had just rebooted)

    You might find one app with a memory leak, allocating memory, but never freeing it.  Update it with a fixed version.

     

    Other than that, add more RAM.  It is the easiest if it is possible.  Then, add a swap file.  (It may not help, but it can't hurt)  Don't create files in /root or /tmp that are huge and use up a lot of RAM.  Running lots of apps with only 1 Gig of ram is going to be hard.

     

    Run fewer apps, or more memory efficient ones, or change options so they are more efficient.  (And add more memory)

     

    Joe L.

  • Quote
    • Author

    Thank you Joe, some good tips here.

    I already changed some settings in SABnzbd, previously it was dying every 3 days ;)

    All my apps are up to date, I'll check out the swap file and add RAM when possible.

  • Quote

    • Archived

    This topic is now archived and is closed to further replies.

    Go to topic listing

    Account

    Navigation

    Search

    Search

    Configure browser push notifications
    Chrome (Android)
    1. Tap the lock icon next to the address bar.
    2. Tap Permissions → Notifications.
    3. Adjust your preference.
    Chrome (Desktop)
    1. Click the padlock icon in the address bar.
    2. Select Site settings.
    3. Find Notifications and adjust your preference.