January 21, 20251 yr First time seeing this error. I would be very surprised if its actually an out of memory issue. Per the errors instructions, I am posting here to see if anyone can review my diagnostic logs, which are attached. Thanks! server-diagnostics-20250120-2051.zip
January 21, 20251 yr if hitting out of memory issues this is most likely because something ran over a memory limit or a service like a vm or docker was over provisioned.... per diag i'm not seeing issues.. total used free shared buff/cache available Mem: 124Gi 49Gi 6.2Gi 1.6Gi 72Gi 75Gi looks like you have 128 GB of ram... so this shouldn't be a problem... make sure your memory modules are setup and in the correct DIMMS for dual channel memeory... ATM this looks like you were runnign a backup taht hit memory limits? from syslog.... Quote Jan 18 16:32:27 Server kernel: free:259486 free_pcp:68 free_cma:0 Jan 18 16:32:27 Server kernel: Node 0 active_anon:10293536kB inactive_anon:110619744kB active_file:9916kB inactive_file:73324kB unevictable:240kB isolated(anon):0kB isolated(file):80kB mapped:816416kB dirty:104kB writeback:0kB shmem:1668556kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:24866816kB writeback_tmp:0kB kernel_stack:46272kB pagetables:294140kB sec_pagetables:352kB all_unreclaimable? no Jan 18 16:32:27 Server kernel: Node 0 DMA free:15368kB boost:0kB min:4kB low:16kB high:28kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15368kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Jan 18 16:32:27 Server kernel: lowmem_reserve[]: 0 2088 127911 127911 127911 Jan 18 16:32:27 Server kernel: Node 0 DMA32 free:504432kB boost:0kB min:1100kB low:3236kB high:5372kB reserved_highatomic:10240KB active_anon:4112kB inactive_anon:1625820kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:2268416kB managed:2174388kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Jan 18 16:32:27 Server kernel: lowmem_reserve[]: 0 0 125823 125823 125823 Jan 18 16:32:27 Server kernel: Node 0 Normal free:518144kB boost:486228kB min:552700kB low:681540kB high:810380kB reserved_highatomic:53248KB active_anon:10289424kB inactive_anon:108993924kB active_file:9916kB inactive_file:73324kB unevictable:240kB writepending:104kB present:131037696kB managed:128843184kB mlocked:240kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Jan 18 16:32:27 Server kernel: lowmem_reserve[]: 0 0 0 0 0 Jan 18 16:32:27 Server kernel: Node 0 DMA: 0*4kB 1*8kB (U) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15368kB Jan 18 16:32:27 Server kernel: Node 0 DMA32: 905*4kB (UM) 704*8kB (UM) 497*16kB (UM) 452*32kB (UM) 419*64kB (UM) 314*128kB (UM) 199*256kB (UM) 213*512kB (UM) 122*1024kB (UM) 37*2048kB (M) 11*4096kB (M) = 504436kB Jan 18 16:32:27 Server kernel: Node 0 Normal: 1904*4kB (UME) 6329*8kB (UME) 10920*16kB (UME) 8862*32kB (UME) 12*64kB (ME) 1*128kB (E) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 517448kB Jan 18 16:32:27 Server kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Jan 18 16:32:27 Server kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 18 16:32:27 Server kernel: 438748 total pagecache pages Jan 18 16:32:27 Server kernel: 0 pages in swap cache Jan 18 16:32:27 Server kernel: Free swap = 0kB Jan 18 16:32:27 Server kernel: Total swap = 0kB Jan 18 16:32:27 Server kernel: 33330527 pages RAM Jan 18 16:32:27 Server kernel: 0 pages HighMem/MovableOnly Jan 18 16:32:27 Server kernel: 572292 pages reserved Jan 18 16:32:27 Server kernel: 0 pages cma reserved Jan 18 16:32:27 Server kernel: Tasks state (memory values in pages): Jan 18 16:32:27 Server kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name Jan 18 16:32:27 Server kernel: [ 1821] 0 1821 5713 861 61440 0 -1000 udevd Jan 18 16:32:27 Server kernel: [ 2131] 0 2131 666 256 40960 0 0 cgroup2-unraid Jan 18 16:32:27 Server kernel: [ 2516] 81 2516 1613 514 45056 0 0 dbus-daemon Jan 18 16:32:27 Server kernel: [ 2538] 0 2538 1582 833 49152 0 0 elogind-daemon Jan 18 16:32:27 Server kernel: [ 2652] 44 2652 19468 755 65536 0 0 ntpd Jan 18 16:32:27 Server kernel: [ 2670] 0 2670 677 320 40960 0 0 acpid Jan 18 16:32:27 Server kernel: [ 2716] 0 2716 671 384 45056 0 0 crond Jan 18 16:32:27 Server kernel: [ 2729] 0 2729 686 384 40960 0 0 atd Jan 18 16:32:27 Server kernel: [ 2790] 61 2790 1323 448 45056 0 0 avahi-daemon Jan 18 16:32:27 Server kernel: [ 2791] 61 2791 1188 320 45056 0 0 avahi-daemon Jan 18 16:32:27 Server kernel: [ 2806] 0 2806 696 320 40960 0 0 avahi-dnsconfd Jan 18 16:32:27 Server kernel: [ 2837] 0 2837 21625 2493 147456 0 0 smbd Jan 18 16:32:27 Server kernel: [ 2843] 0 2843 21040 1407 131072 0 0 smbd-notifyd Jan 18 16:32:27 Server kernel: [ 2844] 0 2844 21042 1215 131072 0 0 smbd-cleanupd Jan 18 16:32:27 Server kernel: [ 2851] 0 2851 674 448 40960 0 0 wsdd2 Jan 18 16:32:27 Server kernel: [ 2855] 0 2855 20390 1391 135168 0 0 winbindd Jan 18 16:32:27 Server kernel: [ 2857] 0 2857 20381 1587 135168 0 0 wb[SERVER] Jan 18 16:32:27 Server kernel: [ 2938] 0 2938 20565 1843 135168 0 0 wb-idmap Jan 18 16:32:27 Server kernel: [ 2979] 0 2979 1040 576 40960 0 0 monitor_nchan Jan 18 16:32:27 Server kernel: [ 4009] 0 4009 1272 768 45056 0 0 atd Jan 18 16:32:27 Server kernel: [ 4013] 0 4013 1032 768 40960 0 0 sh Jan 18 16:32:27 Server kernel: [ 4014] 0 4014 1150 704 40960 0 0 sh Jan 18 16:32:27 Server kernel: [ 4015] 0 4015 1625 576 49152 0 0 inotifywait Jan 18 16:32:27 Server kernel: [ 6323] 0 6323 1590 448 40960 0 0 usbhid-ups Jan 18 16:32:27 Server kernel: [ 6327] 0 6327 980 384 45056 0 0 upsd Jan 18 16:32:27 Server kernel: [ 6330] 0 6330 695 448 40960 0 0 upsmon Jan 18 16:32:27 Server kernel: [ 10601] 0 10601 52874 832 65536 0 0 rsyslogd Jan 18 16:32:27 Server kernel: [ 10898] 0 10898 1101 320 45056 0 0 inetd Jan 18 16:32:27 Server kernel: [ 10903] 0 10903 70011 761 86016 0 0 emhttpd Jan 18 16:32:27 Server kernel: [ 10907] 0 10907 688 448 40960 0 0 agetty Jan 18 16:32:27 Server kernel: [ 10908] 0 10908 688 448 40960 0 0 agetty Jan 18 16:32:27 Server kernel: [ 10909] 0 10909 688 448 45056 0 0 agetty Jan 18 16:32:27 Server kernel: [ 10910] 0 10910 688 448 40960 0 0 agetty Jan 18 16:32:27 Server kernel: [ 10911] 0 10911 688 448 45056 0 0 agetty Jan 18 16:32:27 Server kernel: [ 10912] 0 10912 688 448 40960 0 0 agetty Jan 18 16:32:27 Server kernel: [ 11309] 0 11309 25135 2168 163840 0 0 php-fpm Jan 18 16:32:27 Server kernel: [ 11745] 0 11745 40236 1523 65536 0 0 nginx Jan 18 16:32:27 Server kernel: [ 11746] 0 11746 43633 4787 110592 0 0 nginx Jan 18 16:32:27 Server kernel: [ 12077] 0 12077 2861558 49251 4169728 0 0 unraid-api Jan 18 16:32:27 Server kernel: [ 12131] 0 12131 1338 896 45056 0 0 rc.flash_backup Jan 18 16:32:27 Server kernel: [ 16655] 0 16655 35098 385 57344 0 0 shfs Jan 18 16:32:27 Server kernel: [ 16667] 0 16667 172509 15757 299008 0 0 shfs Jan 18 16:32:27 Server kernel: [ 17812] 0 17812 1222228 51685 1900544 0 0 dockerd Jan 18 16:32:27 Server kernel: [ 17840] 0 17840 318885 6277 241664 0 0 containerd Jan 18 16:32:27 Server kernel: [ 19637] 0 19637 9307 1996 110592 0 0 virtlockd Jan 18 16:32:27 Server kernel: [ 19669] 0 19669 9341 2188 110592 0 0 virtlogd Jan 18 16:32:27 Server kernel: [ 19729] 0 19729 309824 1027 131072 0 1 containerd-shim Jan 18 16:32:27 Server kernel: [ 19807] 1000 19807 582 128 40960 0 0 dumb-init Jan 18 16:32:27 Server kernel: [ 19857] 0 19857 416067 3480 323584 0 0 libvirtd Jan 18 16:32:27 Server kernel: [ 19995] 1000 19995 81862 9717 184320 0 0 python Jan 18 16:32:27 Server kernel: [ 20263] 99 20263 1808 640 49152 0 0 dnsmasq Jan 18 16:32:27 Server kernel: [ 20264] 0 20264 1746 192 49152 0 0 dnsmasq Jan 18 16:32:27 Server kernel: [ 21463] 1000 21463 5693 1920 86016 0 0 Xvfb Jan 18 16:32:27 Server kernel: [ 21506] 0 21506 3105525 2147482 19062784 0 0 qemu-system-x86 Jan 18 16:32:27 Server kernel: [ 21695] 0 21695 2409793 968060 9330688 0 0 qemu-system-x86 Jan 18 16:32:27 Server kernel: [ 25956] 0 25956 309824 909 126976 0 1 containerd-shim Jan 18 16:32:27 Server kernel: [ 25977] 0 25977 9283 4352 110592 0 0 supervisord Jan 18 16:32:27 Server kernel: [ 29009] 0 29009 1835 384 49152 0 0 heartbeat.sh Jan 18 16:32:27 Server kernel: [ 29010] 0 29010 1835 512 53248 0 0 startnginx.sh Jan 18 16:32:27 Server kernel: [ 29013] 0 29013 140642 1746 106496 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29021] 33 29021 157888 14170 380928 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29022] 33 29022 157557 18765 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29023] 33 29023 157491 18957 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29024] 33 29024 157491 16717 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29032] 33 29032 157491 18844 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29050] 33 29050 157524 19405 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29078] 33 29078 157491 18253 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29115] 33 29115 157524 19405 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29156] 33 29156 157491 16461 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29194] 33 29194 157491 16269 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29228] 33 29228 157491 16717 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29273] 33 29273 157491 16525 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29303] 33 29303 157491 17211 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29338] 33 29338 157491 18894 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29365] 33 29365 157491 17933 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29401] 33 29401 157491 18893 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29431] 33 29431 157491 18765 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29458] 33 29458 157491 18957 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29483] 33 29483 157491 16846 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29522] 33 29522 157491 19213 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29577] 33 29577 157524 19237 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29620] 33 29620 157524 17702 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29712] 33 29712 157491 19352 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29740] 33 29740 157524 16333 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29756] 33 29756 157524 17549 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29779] 33 29779 157491 15414 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29809] 33 29809 157491 19413 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29824] 33 29824 157491 16012 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29850] 33 29850 157491 18845 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29889] 33 29889 157491 16013 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29917] 33 29917 157491 17869 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29942] 33 29942 157491 18701 376832 0 0 nginx Jan 18 16:32:27 Server kernel: [ 29992] 33 29992 140684 19357 237568 0 0 nginx Jan 18 16:32:27 Server kernel: [ 918732] 0 918732 309824 1831 126976 0 1 containerd-shim Jan 18 16:32:27 Server kernel: [ 918751] 100 918751 1902 448 53248 0 0 chronyd Jan 18 16:32:27 Server kernel: [ 938544] 0 938544 309760 971 126976 0 1 containerd-shim Jan 18 16:32:27 Server kernel: [ 938564] 0 938564 56 0 24576 0 0 dumb-init Jan 18 16:32:27 Server kernel: [ 938608] 0 938608 9495 4608 110592 0 0 supervisord Jan 18 16:32:27 Server kernel: [ 938746] 99 938746 56 0 24576 0 0 start.sh Jan 18 16:32:27 Server kernel: [ 938747] 99 938747 1947 576 53248 0 0 bash Jan 18 16:32:27 Server kernel: [ 940673] 99 940673 579255 32998 618496 0 0 python3 Jan 18 16:32:27 Server kernel: [ 944593] 0 944593 309760 1506 131072 0 1 containerd-shim Jan 18 16:32:27 Server kernel: [ 944614] 99 944614 417159 151063 1540096 0 0 python Jan 18 16:32:27 Server kernel: [ 944748] 0 944748 309824 1105 131072 0 1 containerd-shim Jan 18 16:32:27 Server kernel: [ 944767] 0 944767 110 0 28672 0 0 s6-svscan Jan 18 16:32:27 Server kernel: [ 944833] 0 944833 54 0 24576 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 944834] 0 944834 51 0 24576 0 0 s6-linux-init-s Jan 18 16:32:27 Server kernel: [ 944853] 0 944853 54 0 24576 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 944854] 0 944854 54 0 24576 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 944855] 0 944855 54 0 24576 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 944856] 0 944856 54 0 24576 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 944864] 0 944864 52 0 24576 0 0 s6-ipcserverd Jan 18 16:32:27 Server kernel: [ 944982] 0 944982 407 0 40960 0 0 busybox Jan 18 16:32:27 Server kernel: [ 945097] 0 945097 309760 1051 126976 0 1 containerd-shim Jan 18 16:32:27 Server kernel: [ 945116] 0 945116 109 0 24576 0 0 s6-svscan Jan 18 16:32:27 Server kernel: [ 945180] 0 945180 55 0 24576 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 945181] 0 945181 52 0 24576 0 0 s6-linux-init-s Jan 18 16:32:27 Server kernel: [ 945200] 0 945200 55 0 24576 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 945201] 0 945201 55 0 24576 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 945202] 0 945202 55 0 24576 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 945203] 0 945203 55 0 24576 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 945212] 0 945212 52 0 24576 0 0 s6-ipcserverd Jan 18 16:32:27 Server kernel: [ 945343] 99 945343 166733 8319 901120 0 0 node Jan 18 16:32:27 Server kernel: [ 945344] 0 945344 410 64 40960 0 0 busybox Jan 18 16:32:27 Server kernel: [ 945401] 99 945401 2831763 54901 6680576 0 0 node Jan 18 16:32:27 Server kernel: [ 946105] 0 946105 309760 1055 131072 0 1 containerd-shim Jan 18 16:32:27 Server kernel: [ 946126] 0 946126 109 0 24576 0 0 s6-svscan Jan 18 16:32:27 Server kernel: [ 946210] 0 946210 54 0 24576 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 946212] 0 946212 51 0 24576 0 0 s6-linux-init-s Jan 18 16:32:27 Server kernel: [ 946231] 0 946231 54 0 24576 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 946232] 0 946232 54 0 24576 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 946233] 0 946233 54 0 24576 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 946234] 0 946234 54 0 24576 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 946242] 0 946242 52 0 24576 0 0 s6-ipcserverd Jan 18 16:32:27 Server kernel: [ 946351] 99 946351 312599 2578 155648 0 0 syncthing Jan 18 16:32:27 Server kernel: [ 946352] 0 946352 407 128 40960 0 0 busybox Jan 18 16:32:27 Server kernel: [ 946380] 99 946380 346889 12837 413696 0 0 syncthing Jan 18 16:32:27 Server kernel: [ 946507] 0 946507 309760 1011 131072 0 1 containerd-shim Jan 18 16:32:27 Server kernel: [ 946531] 0 946531 310348 2445 135168 0 0 unpackerr Jan 18 16:32:27 Server kernel: [ 947159] 0 947159 309824 1309 126976 0 1 containerd-shim Jan 18 16:32:27 Server kernel: [ 947212] 0 947212 51 0 28672 0 0 s6-svscan Jan 18 16:32:27 Server kernel: [ 947428] 0 947428 51 0 28672 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 947671] 0 947671 51 0 28672 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 947674] 99 947674 1239335 31935 884736 0 0 Plex Media Serv Jan 18 16:32:27 Server kernel: [ 947933] 99 947933 16292 8205 143360 0 0 Plex Script Hos Jan 18 16:32:27 Server kernel: [ 948110] 99 948110 10723 320 90112 0 0 Plex Tuner Serv Jan 18 16:32:27 Server kernel: [ 948462] 0 948462 309696 1482 131072 0 1 containerd-shim Jan 18 16:32:27 Server kernel: [ 948489] 0 948489 110 0 28672 0 0 s6-svscan Jan 18 16:32:27 Server kernel: [ 948623] 0 948623 54 0 24576 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 948624] 0 948624 51 0 24576 0 0 s6-linux-init-s Jan 18 16:32:27 Server kernel: [ 948644] 0 948644 54 0 24576 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 948645] 0 948645 54 0 24576 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 948646] 0 948646 54 0 24576 0 0 s6-supervise Jan 18 16:32:27 Server kernel: [ 948654] 0 948654 52 0 24576 0 0 s6-ipcserverd Jan 18 16:32:27 Server kernel: [ 948801] 0 948801 407 0 36864 0 0 busybox Jan 18 16:32:27 Server kernel: [ 949222] 0 949222 309760 1575 126976 0 1 containerd-shim Jan 18 16:32:27 Server kernel: [ 949241] 0 949241 536 256 40960 0 0 dumb-init Jan 18 16:32:27 Server kernel: [ 949279] 0 949279 2863934 31176 4304896 0 0 node Jan 18 16:32:27 Server kernel: [ 949915] 0 949915 309824 1114 135168 0 1 containerd-shim Jan 18 16:32:27 Server kernel: [ 949968] 0 949968 69305697 60232 1425408 0 0 CodeProject.AI. Jan 18 16:32:27 Server kernel: [ 950394] 0 950394 3845683 963230 12333056 0 0 python3.8 Jan 18 16:32:27 Server kernel: [ 950514] 0 950514 3676564 970116 12132352 0 0 python3 Jan 18 16:32:27 Server kernel: [ 950591] 0 950591 69626539 254541 3088384 0 0 dotnet Jan 18 16:32:27 Server kernel: [ 950719] 0 950719 3505401 958273 12247040 0 0 python3.8 Jan 18 16:32:27 Server kernel: [3270663] 1001 3270663 25597 4524 184320 0 0 smbd[192.168.20 Jan 18 16:32:27 Server kernel: [2190598] 99 2190598 649 64 40960 0 0 EasyAudioEncode Jan 18 16:32:27 Server kernel: [3144777] 0 3144777 309824 956 131072 0 1 containerd-shim Jan 18 16:32:27 Server kernel: [3144799] 0 3144799 445 128 40960 0 0 sudo Jan 18 16:32:27 Server kernel: [3144885] 99 3144885 22443870 22175971 179695616 0 0 python Jan 18 16:32:27 Server kernel: [ 603697] 99 603697 407 192 40960 0 0 jackett_launche Jan 18 16:32:27 Server kernel: [ 603709] 99 603709 68280290 36327 888832 0 0 jackett Jan 18 16:32:27 Server kernel: [1803578] 99 1803578 4201586 780697 8048640 0 0 java Jan 18 16:32:27 Server kernel: [1827790] 99 1827790 1397737 95558 999424 0 0 Plex Transcoder Jan 18 16:32:27 Server kernel: [1916589] 1001 1916589 25539 4930 180224 0 0 smbd[192.168.0. Jan 18 16:32:27 Server kernel: [1923501] 0 1923501 25888 3648 180224 0 0 notify_poller Jan 18 16:32:27 Server kernel: [1923503] 0 1923503 25886 3392 176128 0 0 session_check Jan 18 16:32:27 Server kernel: [1923505] 0 1923505 25903 3392 176128 0 0 system_temp Jan 18 16:32:27 Server kernel: [1933492] 0 1933492 3730 1152 61440 0 0 ttyd Jan 18 16:32:27 Server kernel: [1963834] 0 1963834 25886 3328 176128 0 0 wg_poller Jan 18 16:32:27 Server kernel: [1963836] 0 1963836 25924 3520 180224 0 0 update_1 Jan 18 16:32:27 Server kernel: [1963838] 0 1963838 25931 3712 180224 0 0 update_2 Jan 18 16:32:27 Server kernel: [1963840] 0 1963840 25905 3200 176128 0 0 update_3 Jan 18 16:32:27 Server kernel: [1963844] 0 1963844 26415 3712 184320 0 0 vm_dashusage Jan 18 16:32:27 Server kernel: [1964475] 0 1964475 25902 3392 176128 0 0 zfs_master Jan 18 16:32:27 Server kernel: [1964477] 0 1964477 25932 4096 180224 0 0 device_list Jan 18 16:32:27 Server kernel: [1964479] 0 1964479 1042 832 40960 0 0 disk_load Jan 18 16:32:27 Server kernel: [1964481] 0 1964481 25903 3392 180224 0 0 parity_list Jan 18 16:32:27 Server kernel: [1965565] 0 1965565 671 320 45056 0 0 sleep Jan 18 16:32:27 Server kernel: [1967461] 0 1967461 25903 3392 180224 0 0 file_manager Jan 18 16:32:27 Server kernel: [1971082] 0 1971082 1421 320 49152 0 0 sleep Jan 18 16:32:27 Server kernel: [2099073] 0 2099073 25915 1799 167936 0 0 php-fpm Jan 18 16:32:27 Server kernel: [2099595] 0 2099595 25915 1913 167936 0 0 php-fpm Jan 18 16:32:27 Server kernel: [2099597] 0 2099597 26427 1730 172032 0 0 php-fpm Jan 18 16:32:27 Server kernel: [2125776] 0 2125776 19724 704 110592 0 0 notify Jan 18 16:32:27 Server kernel: [2125802] 0 2125802 1031 832 40960 0 0 sh Jan 18 16:32:27 Server kernel: [2125803] 0 2125803 312532 0 110592 0 0 docker Jan 18 16:32:27 Server kernel: [2125804] 0 2125804 880 384 45056 0 0 grep Jan 18 16:32:27 Server kernel: [2125866] 0 2125866 1031 768 40960 0 0 sh Jan 18 16:32:27 Server kernel: [2125867] 0 2125867 1781 128 49152 0 0 smartctl Jan 18 16:32:27 Server kernel: [2125868] 0 2125868 896 448 45056 0 0 grep Jan 18 16:32:27 Server kernel: [2125879] 0 2125879 1032 832 40960 0 0 sh Jan 18 16:32:27 Server kernel: [2125880] 0 2125880 1513 64 49152 0 0 awk Jan 18 16:32:27 Server kernel: [2125891] 0 2125891 1513 64 45056 0 0 awk Jan 18 16:32:27 Server kernel: [2125905] 0 2125905 692 384 45056 0 0 crond Jan 18 16:32:27 Server kernel: [2125906] 0 2125906 692 384 45056 0 0 crond Jan 18 16:32:27 Server kernel: [2125919] 0 2125919 3739 256 57344 0 0 zfs Jan 18 16:32:27 Server kernel: [2125945] 0 2125945 1091 192 40960 0 0 bash Jan 18 16:32:27 Server kernel: [2125946] 0 2125946 1040 386 40960 0 0 monitor_nchan Jan 18 16:32:27 Server kernel: [2125947] 0 2125947 405 0 40960 0 0 curl Jan 18 16:32:27 Server kernel: [2125948] 0 2125948 896 576 45056 0 0 grep Jan 18 16:32:27 Server kernel: [2125975] 0 2125975 1168 64 49152 0 0 nvidia-smi Jan 18 16:32:27 Server kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=ce7e0da9db06e6e57d77db7b6c96ef5d85e28591849e07737a32a0bf11be0928,mems_allowed=0,global_oom,task_memcg=/docker/ce7e0da9db06e6e57d77db7b6c96ef5d85e28591849e07737a32a0bf11be0928,task=python,pid=3144885,uid=99 Jan 18 16:32:27 Server kernel: Out of memory: Killed process 3144885 (python) total-vm:89775480kB, anon-rss:88702348kB, file-rss:1536kB, shmem-rss:0kB, UID:99 pgtables:175484kB oom_score_adj:0 Jan 18 16:32:34 Server kernel: oom_reaper: reaped process 3144885 (python), now anon-rss:0kB, file-rss:1020kB, shmem-rss:0kB Jan 18 16:32:34 Server kernel: veth618d9ce: renamed from eth0 Jan 18 16:36:48 Server flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update Jan 18 16:53:08 Server emhttpd: shcmd (195): /usr/local/sbin/mover start &> /dev/null & While I have seen this with mover and database dockers... This deont' seem to be the case here... To me this seems like a cron scheduling conflict of trying to run mutiple things at once.... as mover and falsh backup and some other heavy ram resource was loaded...
January 21, 20251 yr Author Certainly no overprovisioning. Its a fairly powerfull system with lots of ram as you mentioned. Looking at the time stamp... It was not a backup task. I was performing a data dump on the server. In the 200-300GB range. That mover event was manually triggered to get some files off my cache drive. Its possible the cache drive was still being written to during that time. Not sure why the flash backup was running then. Thats an odd time. In unraid connect there is a flash backup setting, with no date or time options. Maybe this triggered it? My back up runs every monday at 2 or 3 AM, so that was not happening then. Its 2 sets of 2x32gb sticks in the correct corresponding spots for dual channel. While I do not have any database specific Dockers. I am sure some of them run databases in their normal operation. But again, this is a fairly powerful system that normally idles around 5% cpu. If its running out of memory, is there a way to allocate more to the system from available/cache? Kinda a silly problem to have for someone who has as much ram as I do. I only have 2 vm's with 12GB allocated. And the dockers which I dont think there is any control over. Knowing that this system idles quite low, I still built it up to be powerfull enough with 10gb nics to take large data dumps at speed. I have never seen this issue before, and updated to 7.0 8 days ago. I am slightly skeptical of that, unless the reporting of the error is a new feature :s Thanks.
January 21, 20251 yr 7 hours ago, Cody Peters said: First time seeing this error. I would be very surprised if its actually an out of memory issue. Per the errors instructions, I am posting here to see if anyone can review my diagnostic logs, which are attached. Thanks! server-diagnostics-20250120-2051.zip 221.74 kB · 1 download Whatever docker container was running Python back on the 18th around 16:30 ran the server OOM - if you remember doing something at the time, then you can narrow it down to which one. Else, check the result of ps auxf or run ps auxf > ps.txt making sure to sanitize any user names and passwords it may contain, then you can check which container is running what. Since it only happened once, it shouldn't be too much to worry about - if it happens again you can limit the memory to the container. Remember to reboot the server to clear the log so FCP stops warning you
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.