Hi,

 

i woke up to some services on my server not beeing available (but some are).

So i had a look at the logs and saw this:

 

Nov  7 04:51:13 unraid root: mover: finished
Nov  7 05:26:49 unraid emhttpd: spinning down /dev/sdg
Nov  7 05:31:30 unraid emhttpd: spinning down /dev/sdi
Nov  7 05:53:59 unraid emhttpd: spinning down /dev/sdj
Nov  7 05:53:59 unraid emhttpd: spinning down /dev/sdh
Nov  7 05:55:30 unraid kernel: source_queue:sr invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
Nov  7 05:55:30 unraid kernel: CPU: 0 PID: 2902 Comm: source_queue:sr Tainted: P     U     O       6.1.106-Unraid #1
Nov  7 05:55:30 unraid kernel: Hardware name: Gigabyte Technology Co., Ltd. B760M DS3H DDR4/B760M DS3H DDR4, BIOS F18d 08/07/2024
Nov  7 05:55:30 unraid kernel: Call Trace:
Nov  7 05:55:30 unraid kernel: <TASK>
Nov  7 05:55:30 unraid kernel: dump_stack_lvl+0x44/0x5c
Nov  7 05:55:30 unraid kernel: dump_header+0x4a/0x211
Nov  7 05:55:30 unraid kernel: oom_kill_process+0x80/0x111
Nov  7 05:55:30 unraid kernel: out_of_memory+0x3b3/0x3e5
Nov  7 05:55:30 unraid kernel: mem_cgroup_out_of_memory+0x7c/0xb2
Nov  7 05:55:30 unraid kernel: try_charge_memcg+0x44a/0x5ad
Nov  7 05:55:30 unraid kernel: charge_memcg+0x31/0x79
Nov  7 05:55:30 unraid kernel: __mem_cgroup_charge+0x29/0x41
Nov  7 05:55:30 unraid kernel: __handle_mm_fault+0x881/0xcf9
Nov  7 05:55:30 unraid kernel: handle_mm_fault+0x13d/0x20f
Nov  7 05:55:30 unraid kernel: do_user_addr_fault+0x2c3/0x465
Nov  7 05:55:30 unraid kernel: exc_page_fault+0xfb/0x11d
Nov  7 05:55:30 unraid kernel: asm_exc_page_fault+0x22/0x30
Nov  7 05:55:30 unraid kernel: RIP: 0033:0x1472521852f2
Nov  7 05:55:30 unraid kernel: Code: 48 39 f7 72 17 74 25 4c 8d 0c 16 4c 39 cf 0f 82 db 01 00 00 48 89 f9 48 29 f1 eb 06 48 89 f1 48 29 f9 83 f9 3f 76 76 48 89 d1 <f3> a4 c3 80 fa 10 73 17 80 fa 08 73 2a 80 fa 04 73 36 80 fa 01 77
Nov  7 05:55:30 unraid kernel: RSP: 002b:00001470b79fcb88 EFLAGS: 00010216
Nov  7 05:55:30 unraid kernel: RAX: 0000146f4d4bd5e0 RBX: 0000000000000000 RCX: 000000000001c260
Nov  7 05:55:30 unraid kernel: RDX: 0000000000062c80 RSI: 0000146f45e88ab0 RDI: 0000146f4d504000
Nov  7 05:55:30 unraid kernel: RBP: 0000000000000000 R08: 0000146f45e42000 R09: 0000146f45ea4d10
Nov  7 05:55:30 unraid kernel: R10: 0000000000000001 R11: 00001472527f5168 R12: 0000000000062c80
Nov  7 05:55:30 unraid kernel: R13: 00001470b79fcbb0 R14: 0000146f57e78a20 R15: 0000146f4d4bd5e0
Nov  7 05:55:30 unraid kernel: </TASK>
Nov  7 05:55:30 unraid kernel: memory: usage 2097152kB, limit 2097152kB, failcnt 6801
Nov  7 05:55:30 unraid kernel: swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Nov  7 05:55:30 unraid kernel: Memory cgroup stats for /docker/9e9b5fa0b172e563cbc6b73056779372b18090752a0fc59bfa459e38816ae840:
Nov  7 05:55:30 unraid kernel: anon 2137632768
Nov  7 05:55:30 unraid kernel: file 0
Nov  7 05:55:30 unraid kernel: kernel 9768960
Nov  7 05:55:30 unraid kernel: kernel_stack 2654208
Nov  7 05:55:30 unraid kernel: pagetables 5808128
Nov  7 05:55:30 unraid kernel: sec_pagetables 0
Nov  7 05:55:30 unraid kernel: percpu 792
Nov  7 05:55:30 unraid kernel: sock 0
Nov  7 05:55:30 unraid kernel: vmalloc 8192
Nov  7 05:55:30 unraid kernel: shmem 0
Nov  7 05:55:30 unraid kernel: file_mapped 0
Nov  7 05:55:30 unraid kernel: file_dirty 0
Nov  7 05:55:30 unraid kernel: file_writeback 0
Nov  7 05:55:30 unraid kernel: swapcached 0
Nov  7 05:55:30 unraid kernel: anon_thp 624951296
Nov  7 05:55:30 unraid kernel: file_thp 0
Nov  7 05:55:30 unraid kernel: shmem_thp 0
Nov  7 05:55:30 unraid kernel: inactive_anon 2137600000
Nov  7 05:55:30 unraid kernel: active_anon 8192
Nov  7 05:55:30 unraid kernel: inactive_file 0
Nov  7 05:55:30 unraid kernel: active_file 0
Nov  7 05:55:30 unraid kernel: unevictable 0
Nov  7 05:55:30 unraid kernel: slab_reclaimable 72096
Nov  7 05:55:30 unraid kernel: slab_unreclaimable 1179536
Nov  7 05:55:30 unraid kernel: slab 1251632
Nov  7 05:55:30 unraid kernel: workingset_refault_anon 0
Nov  7 05:55:30 unraid kernel: workingset_refault_file 4590
Nov  7 05:55:30 unraid kernel: workingset_activate_anon 0
Nov  7 05:55:30 unraid kernel: workingset_activate_file 271
Nov  7 05:55:30 unraid kernel: workingset_restore_anon 0
Nov  7 05:55:30 unraid kernel: workingset_restore_file 239
Nov  7 05:55:30 unraid kernel: workingset_nodereclaim 0
Nov  7 05:55:30 unraid kernel: pgscan 7491
Nov  7 05:55:30 unraid kernel: pgsteal 4834
Nov  7 05:55:30 unraid kernel: pgscan_kswapd 2337
Nov  7 05:55:30 unraid kernel: pgscan_direct 5154
Nov  7 05:55:30 unraid kernel: pgsteal_kswapd 1037
Nov  7 05:55:30 unraid kernel: pgsteal_direct 3797
Nov  7 05:55:30 unraid kernel: pgfault 1954245
Nov  7 05:55:30 unraid kernel: pgmajfault 227
Nov  7 05:55:30 unraid kernel: pgrefill 2342
Nov  7 05:55:30 unraid kernel: pgactivate 1953
Nov  7 05:55:30 unraid kernel: pgdeactivate 2224
Nov  7 05:55:30 unraid kernel: pglazyfree 0
Nov  7 05:55:30 unraid kernel: pglazyfreed 0
Nov  7 05:55:30 unraid kernel: thp_fault_alloc 206
Nov  7 05:55:30 unraid kernel: thp_collapse_alloc 190
Nov  7 05:55:30 unraid kernel: Tasks state (memory values in pages):
Nov  7 05:55:30 unraid kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Nov  7 05:55:30 unraid kernel: [   2419]     0  2419      620       17    45056        0             0 entrypoint.sh
Nov  7 05:55:30 unraid kernel: [   2432]     0  2432  3148012   520838  5787648        0             0 neolink
Nov  7 05:55:30 unraid kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=9e9b5fa0b172e563cbc6b73056779372b18090752a0fc59bfa459e38816ae840,mems_allowed=0,oom_memcg=/docker/9e9b5fa0b172e563cbc6b73056779372b18090752a0fc59bfa459e38816ae840,task_memcg=/docker/9e9b5fa0b172e563cbc6b73056779372b18090752a0fc59bfa459e38816ae840,task=neolink,pid=2432,uid=0
Nov  7 05:55:30 unraid kernel: Memory cgroup out of memory: Killed process 2432 (neolink) total-vm:12592048kB, anon-rss:2079364kB, file-rss:4580kB, shmem-rss:0kB, UID:0 pgtables:5652kB oom_score_adj:0

 

Obviously this is some sort of memory error but i have troubles to understand it.

I have plenty of Memory (128GB non-ecc DDR4) and usually the usage is around 40-50% (so 40-60GB).

According to unraid UI, docker is not running. But services like jellyfin are still functioning. others are dead for example.

 

Could someone explain this behaviour? shoud i run memtest to check memory? (Oh no, this will take decades! :D)

I am curious because i had no problem with the server in months!

Looks like neolink is making the sever run out of RAM, is that a container?

Ah, did not see this.

Yep this is a container. Unfortunately i still have old reolink cams without rtsp which why i need this tool. And I know it has memory leaks which is why i am killing the container every 2 hours and also limited the memory consumption via docker extra args -> probably there was bad memory access anyways?

I need to get new cams to get rid of it