Hello all,

This morning my NAS suddenly became unresponsive. After a while it came back to life and in the syslog I can see a bunch of kernel messages related to an "Out Of Memory situation" affecting containerd.

My setup is normally very stable and I have not seen this behavior before. I see that the responsible process is "python3" but I can't quite isolate which container this problem is coming from,.

Is there a way I can investigate further?

Mar 30 10:31:07 Nas3 kernel: containerd-shim invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=1

Mar 30 10:31:08 Nas3 kernel: CPU: 2 UID: 0 PID: 1694636 Comm: containerd-shim Tainted: P O 6.12.54-Unraid #1

Mar 30 10:31:08 Nas3 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE

Mar 30 10:31:08 Nas3 kernel: Hardware name: CWWK CW-NAS-ADLN-K/, BIOS 5.27 06/12/2024

Mar 30 10:31:08 Nas3 kernel: Call Trace:

Mar 30 10:31:08 Nas3 kernel: <TASK>

Mar 30 10:31:08 Nas3 kernel: dump_stack_lvl+0x47/0x70

Mar 30 10:31:08 Nas3 kernel: dump_header+0x44/0x190

Mar 30 10:31:08 Nas3 kernel: oom_kill_process+0x82/0x190

Mar 30 10:31:08 Nas3 kernel: out_of_memory+0x3f6/0x430

Mar 30 10:31:08 Nas3 kernel: __alloc_pages_slowpath.constprop.0+0x7f1/0x960

Mar 30 10:31:08 Nas3 kernel: __alloc_pages_noprof+0x109/0x1c0

Mar 30 10:31:08 Nas3 kernel: alloc_pages_mpol_noprof+0xb2/0x1e0

Mar 30 10:31:08 Nas3 kernel: folio_alloc_noprof+0x10/0x20

Mar 30 10:31:08 Nas3 kernel: __filemap_get_folio+0x193/0x200

Mar 30 10:31:08 Nas3 kernel: filemap_fault+0x3c5/0x5b0

Mar 30 10:31:08 Nas3 kernel: __do_fault+0x2a/0xa0

Mar 30 10:31:08 Nas3 kernel: __handle_mm_fault+0xe21/0x1050

Mar 30 10:31:08 Nas3 kernel: ? preempt_latency_start+0x2b/0x50

Mar 30 10:31:08 Nas3 kernel: ? _raw_spin_lock_irq+0x14/0x20

Mar 30 10:31:08 Nas3 kernel: handle_mm_fault+0x145/0x1f0

Mar 30 10:31:08 Nas3 kernel: do_user_addr_fault+0x27f/0x490

Mar 30 10:31:08 Nas3 kernel: exc_page_fault+0xfb/0x120

Mar 30 10:31:08 Nas3 kernel: asm_exc_page_fault+0x22/0x30

Mar 30 10:31:08 Nas3 kernel: RIP: 0033:0x40708e

Mar 30 10:31:08 Nas3 kernel: Code: Unable to access opcode bytes at 0x407064.

Mar 30 10:31:08 Nas3 kernel: RSP: 002b:000000c00038f6d0 EFLAGS: 00010246

Mar 30 10:31:08 Nas3 kernel: RAX: 0000000000000000 RBX: 0000000000000005 RCX: 000000000040708e

Mar 30 10:31:08 Nas3 kernel: RDX: 0000000000000080 RSI: 000000c00038f774 RDI: 0000000000000005

Mar 30 10:31:08 Nas3 kernel: RBP: 000000c00038f718 R08: 0000000000000000 R09: 0000000000000000

Mar 30 10:31:08 Nas3 kernel: R10: 0000000000000db1 R11: 0000000000000246 R12: 000000c00038f7a8

Mar 30 10:31:08 Nas3 kernel: R13: 0000000000000001 R14: 000000c000102fc0 R15: 0000000000000003

Mar 30 10:31:08 Nas3 kernel: </TASK>

Mar 30 10:31:08 Nas3 kernel: Mem-Info:

Mar 30 10:31:08 Nas3 kernel: active_anon:339895 inactive_anon:3160518 isolated_anon:0

Mar 30 10:31:08 Nas3 kernel: active_file:2622 inactive_file:15763 isolated_file:320

Mar 30 10:31:08 Nas3 kernel: unevictable:2640 dirty:88 writeback:0

Mar 30 10:31:08 Nas3 kernel: slab_reclaimable:152330 slab_unreclaimable:120562

Mar 30 10:31:08 Nas3 kernel: mapped:110892 shmem:476643 pagetables:29871

Mar 30 10:31:08 Nas3 kernel: sec_pagetables:523 bounce:0

Mar 30 10:31:08 Nas3 kernel: kernel_misc_reclaimable:0

Mar 30 10:31:08 Nas3 kernel: free:34899 free_pcp:791 free_cma:0

Mar 30 10:31:08 Nas3 kernel: Node 0 active_anon:1359580kB inactive_anon:12642072kB active_file:10488kB inactive_file:63052kB unevictable:10560kB isolated(anon):0kB isolated(file):1280kB mapped:443568kB dirty:352kB writeback:0kB shmem:1906572kB shmem_thp:8192kB shmem_pmdmapped:0kB anon_thp:1275904kB writeback_tmp:0kB kernel_stack:40640kB pagetables:119484kB sec_pagetables:2092kB all_unreclaimable? yes

Mar 30 10:31:08 Nas3 kernel: Node 0 DMA free:15360kB boost:0kB min:64kB low:80kB high:96kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB

Mar 30 10:31:08 Nas3 kernel: lowmem_reserve[]: 0 1668 15720 15720 15720

Mar 30 10:31:08 Nas3 kernel: Node 0 DMA32 free:62224kB boost:0kB min:7164kB low:8952kB high:10740kB reserved_highatomic:0KB active_anon:3236kB inactive_anon:1458696kB active_file:2112kB inactive_file:22896kB unevictable:0kB writepending:228kB present:1845684kB managed:1750412kB mlocked:0kB bounce:0kB free_pcp:2220kB local_pcp:612kB free_cma:0kB

Mar 30 10:31:08 Nas3 kernel: lowmem_reserve[]: 0 0 14052 14052 14052

Mar 30 10:31:08 Nas3 kernel: Node 0 Normal free:62012kB boost:2048kB min:62400kB low:77488kB high:92576kB reserved_highatomic:0KB active_anon:1356344kB inactive_anon:11183376kB active_file:8260kB inactive_file:40280kB unevictable:10560kB writepending:124kB present:14675968kB managed:14389496kB mlocked:32kB bounce:0kB free_pcp:944kB local_pcp:300kB free_cma:0kB

Mar 30 10:31:08 Nas3 kernel: lowmem_reserve[]: 0 0 0 0 0

Mar 30 10:31:08 Nas3 kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15360kB

Mar 30 10:31:08 Nas3 kernel: Node 0 DMA32: 392*4kB (UME) 300*8kB (UME) 225*16kB (UME) 78*32kB (UME) 155*64kB (UME) 72*128kB (UME) 57*256kB (UME) 32*512kB (UE) 2*1024kB (UE) 0*2048kB 0*4096kB = 62224kB

Mar 30 10:31:08 Nas3 kernel: Node 0 Normal: 1671*4kB (UM) 1136*8kB (UE) 2264*16kB (UE) 295*32kB (UE) 9*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 62012kB

Mar 30 10:31:08 Nas3 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB

Mar 30 10:31:08 Nas3 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB

Mar 30 10:31:08 Nas3 kernel: 495461 total pagecache pages

Mar 30 10:31:08 Nas3 kernel: 0 pages in swap cache

Mar 30 10:31:08 Nas3 kernel: Free swap = 0kB

Mar 30 10:31:08 Nas3 kernel: Total swap = 0kB

Mar 30 10:31:08 Nas3 kernel: 4134411 pages RAM

Mar 30 10:31:08 Nas3 kernel: 0 pages HighMem/MovableOnly

Mar 30 10:31:08 Nas3 kernel: 95594 pages reserved

Mar 30 10:31:08 Nas3 kernel: 0 pages cma reserved

Mar 30 10:31:08 Nas3 kernel: Tasks state (memory values in pages):

Mar 30 10:31:08 Nas3 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name

[...]

Mar 30 10:31:08 Nas3 kernel: [ 977805] 99 977805 1719297 1561138 1561118 20 0 12828672 0 0 python3

[...]

Mar 30 10:31:08 Nas3 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/b38595ff52534922b927890317bed1061f5e113561f8d1d382acbf0da8eda547,task=python3,pid=977805,uid=99

Mar 30 10:31:08 Nas3 kernel: Out of memory: Killed process 977805 (python3) total-vm:6877188kB, anon-rss:6244472kB, file-rss:80kB, shmem-rss:0kB, UID:99 pgtables:12528kB oom_score_adj:0

Mar 30 10:31:08 Nas3 kernel: oom_reaper: reaped process 977805 (python3), now anon-rss:0kB, file-rss:80kB, shmem-rss:0kB

Oops sorry what I was looking for was in fact right there under my nose: b38595ff52534922b927890317bed1061f5e113561f8d1d382acbf0da8eda547

That's my calibre web automated container, something went wrong with it.

Edited March 30Mar 30 by googleg