October 9, 2025Oct 9 I'm sorry if this should be posted in else where as I am running 7.2. I had been running 7.2.0-beta.2 for about a month with no problems until about 5 days ago. I have tried upgrading to RC1 same issue. Memory usage seems to increase very quickly. I wasn't able to determine what was using the memory. I tried install the swap plugin but my cache drive is zfs not btrfs so maybe it wont work, it doesn't show any usage.Dell R720ProcessorIntel® Xeon® CPU E5-2690 v2 @ 3.00GHzMemory: 96 GiB DDR3 Multi-bit ECCIntel GPU Arc A380zfs storage, cache pool, no array unserverd-diagnostics-20251008-2137.zip
October 9, 2025Oct 9 Community Expert 5 hours ago, allanbennett said:I'm sorry if this should be posted in else where as I am running 7.2. I had been running 7.2.0-beta.2 for about a month with no problems until about 5 days ago. I have tried upgrading to RC1 same issue. Memory usage seems to increase very quickly. I wasn't able to determine what was using the memory. I tried install the swap plugin but my cache drive is zfs not btrfs so maybe it wont work, it doesn't show any usage.Dell R720ProcessorIntel® Xeon® CPU E5-2690 v2 @ 3.00GHzMemory: 96 GiB DDR3 Multi-bit ECCIntel GPU Arc A380zfs storage, cache pool, no arrayunserverd-diagnostics-20251008-2137.zipLooks like you have tdarr and plex transcodes going at the same time, if they're both set to transcode to RAM try running it without tdarr for a bit and see if it stops running oom
October 9, 2025Oct 9 Author 3 hours ago, Michael_P said:Looks like you have tdarr and plex transcodes going at the same time, if they're both set to transcode to RAM try running it without tdarr for a bit and see if it stops running oomI can certainly try this. I tdarr/ffmpeg had been complaining about out of device space and encodes would fail. So I had not been sending jobs to it for "about a week." I was only able to fix it by uninstalling tdarr and deleting appdata files. So tonight was first time in tdarr had been running transcodes when server oom'd. I also had this problem on another of my unraid servers very similar specs as this but I haven't had any oom on it. Plex and tdarr still running this morning, qbittorrent still non-functional after it was oom killed. Rebooting server now with tdarr removed. Let's see if it can go through this evening without oom'ing.I capture memory usage right after oom and then about 5 min later. I wasn't able to tell what was using the memory between the two,root@unServerD:~# free -h total used free shared buff/cache availableMem: 94Gi 91Gi 1.8Gi 787Mi 2.7Gi 2.7GiSwap: 0B 0B 0Broot@unServerD:~# htoproot@unServerD:~# free -h total used free shared buff/cache availableMem: 94Gi 16Gi 44Gi 782Mi 35Gi 78GiSwap: 0B 0B 0BDo i need to setup swap manually to get it to work?
October 9, 2025Oct 9 Community Expert 6 minutes ago, allanbennett said:Do i need to setup swap manually to get it to work?Seeing as how the kernel was reaping different processes, that suggests it wasn't something running away and instead was just the sever being over-utilized - instead of trying to swap it out, just configure either your tdarr or plex to transcode to disk since that's where it will end up anyway with the swap file.
October 9, 2025Oct 9 Author Just oom'd again about an hour after reboot. tdarr docker was removed before reboot. Plex might have had 1 or 2 people connected but probably not. I had htop running load went over 200 before it became unresponsive. tail on syslog shows me oom messages after waiting about 20 min. ssh sessions basically unresponsive for last 30 min. Remote console loging times out. Ill give the system a hour to see if I can get the latest diagnosis. well only took another 5min as I as writing this up.Only thing different with this server compared to other server is zfs pool. I wanted to use reflinks. is this a possible culprit? unserverd-diagnostics-20251009-0843.zip
October 9, 2025Oct 9 Community Expert 4 minutes ago, allanbennett said:Just oom'd again about an hour after reboot. tdarr docker was removed before reboot. Plex might have had 1 or 2 people connected but probably not. I had htop running load went over 200 before it became unresponsive. tail on syslog shows me oom messages after waiting about 20 min. ssh sessions basically unresponsive for last 30 min. Remote console loging times out. Ill give the system a hour to see if I can get the latest diagnosis. well only took another 5min as I as writing this up.Only thing different with this server compared to other server is zfs pool. I wanted to use reflinks. is this a possible culprit?unserverd-diagnostics-20251009-0843.zipJust for shits and giggles, put a memory limit on your nginx proxy manager and see if that gets killed before the whole host goes oom - start with 1GB Edited October 9, 2025Oct 9 by Michael_P
October 9, 2025Oct 9 Author 23 minutes ago, Michael_P said:Just for shits and giggles, put a memory limit on your nginx proxy manager and see if that gets killed before the whole host goes oom - start with 1GBFunny you suggest this. This docker has always had --memory=1G --no-healthcheck. I use this for cert renewal. This docker can be off for a month.
October 9, 2025Oct 9 Community Expert Try leaving it off for a bit and see if it stops going OOM, I've seen these processes in a couple OOM reports in the last week and both times it was nginx proxy manager (its node process in particular)Oct 9 08:07:39 unServerD kernel: [ 22401] 0 22401 320314 8283 7968 315 0 1306624 0 0 node Oct 9 08:07:39 unServerD kernel: [ 22447] 0 22447 13121 1127 961 166 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22448] 0 22448 13121 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22449] 0 22449 13188 1484 961 523 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22450] 0 22450 13153 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22451] 0 22451 13155 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22452] 0 22452 13112 1141 1041 100 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22453] 0 22453 13112 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22454] 0 22454 13078 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22455] 0 22455 13078 1015 961 54 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22456] 0 22456 13011 1463 961 502 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22457] 0 22457 12978 1321 961 360 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22458] 0 22458 13011 1123 961 162 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22459] 0 22459 13011 961 961 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22460] 0 22460 13011 965 961 4 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22461] 0 22461 13011 1067 961 106 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22462] 0 22462 13012 961 961 0 0 81920 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22463] 0 22463 13012 1126 961 165 0 81920 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22464] 0 22464 13011 961 961 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22465] 0 22465 13064 961 961 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22466] 0 22466 13054 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22467] 0 22467 13012 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22468] 0 22468 13011 1146 961 185 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22469] 0 22469 13011 961 961 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22470] 0 22470 13011 1314 961 353 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22471] 0 22471 13011 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22472] 0 22472 13011 961 961 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22473] 0 22473 13011 1234 1041 193 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22474] 0 22474 13088 1049 961 88 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22475] 0 22475 13088 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22476] 0 22476 13012 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22477] 0 22477 13046 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22478] 0 22478 13120 1582 961 621 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22479] 0 22479 13087 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22480] 0 22480 13046 961 961 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22481] 0 22481 13046 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22482] 0 22482 13087 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22483] 0 22483 13046 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22484] 0 22484 13089 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22485] 0 22485 13045 1237 1041 196 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22486] 0 22486 13121 961 961 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22487] 0 22487 12720 1555 835 720 0 81920 0 0 nginx Edited October 9, 2025Oct 9 by Michael_P
October 9, 2025Oct 9 Author Definitely worth a try. proxy-manager off. I am also removing unused dockers and seeing if i can get swap working. I shouldn't need it, but if the issue is zfs, I've read this can be a work-around.
October 9, 2025Oct 9 Community Expert 9 minutes ago, allanbennett said:seeing if i can get swap workingWith 96G of RAM you don't need swap, I run an absolute ton of services and a couple of chunky VMs on 32
October 10, 2025Oct 10 Author I think it just did it again but this time no oom but server locked up for a good 15 min. proxy-manager still off.root@unServerD:/boot/config# free -h total used free shared buff/cache availableMem: 94Gi 86Gi 2.2Gi 198Mi 6.7Gi 7.5GiSwap: 63Gi 9.2Gi 54GiI was able to get swap working.Can't use swapfile on zfs so the plugin swapfile plugin doesn't work.Here is what I did in case it helps others:https://askubuntu.com/questions/1468817/how-to-solve-error-on-swap-creation-skipping-it-appears-to-have-holes-ubutldr;/dev/zvol/cache/swapfile none swap discard 0 0root@unServerD:~# zfs listNAME USED AVAIL REFER MOUNTPOINTcache 446G 454G 378G /mnt/cachezfs create -V 64G -b $(getconf PAGESIZE) -o logbias=throughput -o sync=always -o primarycache=metadata -o com.sun:auto-snapshot=false cache/swapfilemkswap -f /dev/zvol/cache/swapfileadded this is go fileswapon /dev/zvol/cache/swapfile unserverd-diagnostics-20251009-2119.zip
October 10, 2025Oct 10 Community Expert 4 hours ago, allanbennett said:I think it just did it again but this time no oom but server locked up for a good 15 min. proxy-manager still off.root@unServerD:/boot/config# free -htotal used free shared buff/cache availableMem: 94Gi 86Gi 2.2Gi 198Mi 6.7Gi 7.5GiSwap: 63Gi 9.2Gi 54GiI was able to get swap working.Can't use swapfile on zfs so the plugin swapfile plugin doesn't work.Here is what I did in case it helps others:https://askubuntu.com/questions/1468817/how-to-solve-error-on-swap-creation-skipping-it-appears-to-have-holes-ubutldr;/dev/zvol/cache/swapfile none swap discard 0 0root@unServerD:~# zfs listNAME USED AVAIL REFER MOUNTPOINTcache 446G 454G 378G /mnt/cachezfs create -V 64G -b $(getconf PAGESIZE) -o logbias=throughput -o sync=always -o primarycache=metadata -o com.sun:auto-snapshot=false cache/swapfilemkswap -f /dev/zvol/cache/swapfileadded this is go fileswapon /dev/zvol/cache/swapfileunserverd-diagnostics-20251009-2119.zipThe slowdowns are likely because you're very close to running OOM, that's where these are coming from and will make the server unresponsive:Oct 9 20:46:14 unServerD php-fpm[12266]: [WARNING] [pool www] child 3078784 exited on signal 9 (SIGKILL) after 163.772122 seconds from start Oct 9 20:46:28 unServerD php-fpm[12266]: [WARNING] [pool www] child 3080607 exited on signal 9 (SIGKILL) after 76.074364 seconds from start Oct 9 20:46:38 unServerD php-fpm[12266]: [WARNING] [pool www] child 3081694 exited on signal 9 (SIGKILL) after 24.022172 seconds from start Oct 9 20:46:40 unServerD php-fpm[12266]: [WARNING] [pool www] child 3082525 exited on signal 9 (SIGKILL) after 12.009674 seconds from start Oct 9 20:46:50 unServerD php-fpm[12266]: [WARNING] [pool www] child 3082530 exited on signal 9 (SIGKILL) after 12.010067 seconds from start Oct 9 20:46:52 unServerD php-fpm[12266]: [WARNING] [pool www] child 3082531 exited on signal 9 (SIGKILL) after 12.011153 seconds from start Oct 9 20:47:16 unServerD php-fpm[12266]: [WARNING] [pool www] child 3082557 exited on signal 9 (SIGKILL) after 26.023875 seconds from start Oct 9 20:47:18 unServerD php-fpm[12266]: [WARNING] [pool www] child 3082633 exited on signal 9 (SIGKILL) after 26.023854 seconds from startIt also looks like plex is choking on one of your files, could be corrupt or just because of the OOM issuesOct 9 21:06:19 unServerD kernel: Plex Transcoder[3143683]: segfault at 18 ip 0000150df840ea23 sp 00007ffc332bb280 error 4 in libavcodec.so.60[4f1a23,150df80a0000+406000] likely on CPU 17 (core 11, socket 1)
October 10, 2025Oct 10 Author I agree that something starts using huge amounts of memory. I suspect its zfs as no process shows abnormal memory usage when this happens. The unraid dashboard page doesn't update when this happens. Is there a way to see/track zfs cache?I will look to see if I can set a zfs cache limit but i have read it can be ignored. When I created zfs pool I choose raid z1 and compression. Very little on the disks is compressible but i read it would still be a performance boost.Any zfs experts out there?My next step is to stop creating reflinks and possibly removing the ones I already have. Plex transcode segfaults while not common are not uncommon either. My servers see one or two a week. That segfault happened about 20 min after the server returned to normal. At the time of the lockup I had 12 plex streams playing.
October 10, 2025Oct 10 Community Expert 28 minutes ago, allanbennett said:At the time of the lockup I had 12 plex streams playing.If they were all transcoding to RAM, then that'd do it. You can try caching to disk instead for a while and/or limiting memory to the container. FWIW I have my plex container limited to 4G since if it finds a file it doesn't like during a scan it will spike to infinite RAM usage in a couple of seconds.
October 10, 2025Oct 10 Author 1 hour ago, Michael_P said:If they were all transcoding to RAM, then that'd do it. You can try caching to disk instead for a while and/or limiting memory to the container. FWIW I have my plex container limited to 4G since if it finds a file it doesn't like during a scan it will spike to infinite RAM usage in a couple of seconds.I do transcode to my cache pool and my previous oom was when there was nothing streaming on plex. But sure I can set limits on all my dockers. Did I mention I have another unraid server with similar dockers? It has regular array with xfs drives (rather than zfs pool) that has never had oom issues.
October 10, 2025Oct 10 Community Expert 44 minutes ago, allanbennett said:I do transcode to my cache pool and my previous oom was when there was nothing streaming on plex. But sure I can set limits on all my dockers.Did I mention I have another unraid server with similar dockers? It has regular array with xfs drives (rather than zfs pool) that has never had oom issues.And you haven't change the default zfs cache amount? (1/8th system ram)what does ps -auxf look like? Edited October 10, 2025Oct 10 by Michael_P
October 10, 2025Oct 10 Author This is taken now, I can supply again next time it happens.What are you looking for? ps_auxf.txt
October 10, 2025Oct 10 Community Expert I don't see anything crazy, qbittorrent is using a lot of CPU but not too out there, try to run it again next time it acts up
October 12, 2025Oct 12 Author Stopped making reflinks and haven't had an issue since. arcstat shows my max cache memory set to 12G. Not sure how unraid determines what this value should be but for 40TB this seems low. With arcstat and zpool iostat, I now have the tools to do further testing but it wont be this week. If I find any setting that allow me to use reflinks again, I'll update my post. Thank you Michael_P for your help!
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.