So this is wild...
I had been running this for awhile without issues and I've had a non docker instance running on a dedicated machine before this. Zoneminder was writing to cache but it looks like mover had issues moving the files from cache to array. I turned off ZM and allowed mover to move everything and then set use cache : no. Maybe this issue has been since then but not certain.
2x Xeon 2630L V3 (7C+14T pinned to ZM)
40GB DDR4
3x 4TB SAS drives + Parity
1TB NVME Cache
All cameras (7) are configured for record only and total 3.5MB/s.
After an hour or 2 I go from about 6GB baseline RAM usage to 100% and the system binds up so hard that I can't even access the server's BMC to kill it and have to smother via the power button. It doesn't seem to happen gradually though either. At startup it's fine but seems like I leave for a few minutes and it's locked up. ZM Logs don't seem very helpful either.
8/28/2021 12:13 zmdc 1000 WAR 'zmc -m 5' has not stopped at 21/08/28 12:13:57 after 10 seconds. Sending KILL to pid 1097 zmdc.pl '-
8/28/2021 12:13 zmdc 1000 INF 'zmc -m 5' sending stop to pid 1097 at 21/08/28 12:13:57 zmdc.pl '-
8/28/2021 12:13 zmwatch 1121 INF Restarting analysis daemon for 5 BackYardSouth zmwatch.pl '-
8/28/2021 12:13 zmwatch 1121 WAR Analysis daemon for 5 BackYardSouth needs restarting, time since last analysis 58 seconds (1630170837-1630170779) zmwatch.pl '-
8/28/2021 12:13 zmdc 1000 WAR 'zmc -m 4' has not stopped at 21/08/28 12:13:57 after 10 seconds. Sending KILL to pid 1092 zmdc.pl '-
8/28/2021 12:13 zmdc 1000 INF 'zmc -m 4' sending stop to pid 1092 at 21/08/28 12:13:57 zmdc.pl '-
8/28/2021 12:13 zmwatch 1121 INF Restarting analysis daemon for 4 BackYardNorth zmwatch.pl '-
8/28/2021 12:13 zmwatch 1121 WAR Analysis daemon for 4 BackYardNorth needs restarting, time since last analysis 59 seconds (1630170837-1630170778) zmwatch.pl '-
8/28/2021 12:13 zmdc 1000 WAR 'zmc -m 5' has not stopped at 21/08/28 12:13:57 after 10 seconds. Sending KILL to pid 1097 zmdc.pl '-
8/28/2021 12:13 zmdc 1000 WAR 'zmc -m 2' has not stopped at 21/08/28 12:13:57 after 10 seconds. Sending KILL to pid 1068 zmdc.pl '-
8/28/2021 12:13 zmdc 1000 WAR 'zmc -m 4' has not stopped at 21/08/28 12:13:57 after 10 seconds. Sending KILL to pid 1092 zmdc.pl '-
8/28/2021 12:13 zmdc 1000 INF 'zmc -m 2' sending stop to pid 1068 at 21/08/28 12:13:57 zmdc.pl '-
8/28/2021 12:13 zmwatch 1121 INF Restarting analysis daemon for 2 FrontNorth zmwatch.pl '-
8/28/2021 12:13 zmwatch 1121 WAR Analysis daemon for 2 FrontNorth needs restarting, time since last analysis 56 seconds (1630170836-1630170780) zmwatch.pl '-
8/28/2021 12:13 zmdc 1000 INF 'zmc -m 1' sending stop to pid 1064 at 21/08/28 12:13:56 zmdc.pl '-
8/28/2021 12:13 zmwatch 1121 INF Restarting analysis daemon for 1 FrontSouth zmwatch.pl '-
8/28/2021 12:13 zmwatch 1121 WAR Analysis daemon for 1 FrontSouth needs restarting, time since last analysis 51 seconds (1630170836-1630170785) zmwatch.pl '-
8/28/2021 12:13 zmc_m6 1102 INF BabyCam: 1800 - Capturing at 4.00 fps, capturing bandwidth 65690bytes/sec zm_monitor.cpp 1663
8/28/2021 12:13 zmdc 1000 INF 'zmc -m 5' sending stop to pid 1097 at 21/08/28 12:13:46 zmdc.pl '-
8/28/2021 12:13 zmc_m7 1107 INF Laview: 1800 - Capturing at 4.00 fps, capturing bandwidth 111802bytes/sec zm_monitor.cpp 1663
8/28/2021 12:13 zmwatch 1121 INF Restarting analysis daemon for 5 BackYardSouth zmwatch.pl '-
8/28/2021 12:13 zmwatch 1121 WAR Analysis daemon for 5 BackYardSouth needs restarting, time since last analysis 47 seconds (1630170826-1630170779) zmwatch.pl '-
8/28/2021 12:13 zmdc 1000 INF 'zmc -m 4' sending stop to pid 1092 at 21/08/28 12:13:46 zmdc.pl '-
8/28/2021 12:13 zmwatch 1121 INF Restarting analysis daemon for 4 BackYardNorth zmwatch.pl '-
8/28/2021 12:13 zmwatch 1121 WAR Analysis daemon for 4 BackYardNorth needs restarting, time since last analysis 48 seconds (1630170826-1630170778) zmwatch.pl '-
8/28/2021 12:13 zmdc 1000 INF 'zmc -m 2' sending stop to pid 1068 at 21/08/28 12:13:46 zmdc.pl '-
8/28/2021 12:13 zmwatch 1121 INF Restarting analysis daemon for 2 FrontNorth zmwatch.pl '-
8/28/2021 12:13 zmwatch 1121 WAR Analysis daemon for 2 FrontNorth needs restarting, time since last analysis 46 seconds (1630170826-1630170780) zmwatch.pl '-
8/28/2021 12:13 zmc_m2 1068 INF FrontNorth: 1800 - Capturing at 4.00 fps, capturing bandwidth 805629bytes/sec zm_monitor.cpp 1663
8/28/2021 12:13 zmc_m3 1072 INF SideYard: 1800 - Capturing at 4.00 fps, capturing bandwidth 512321bytes/sec zm_monitor.cpp 1663
8/28/2021 12:13 zmc_m4 1092 INF BackYardNorth: 1500 - Capturing at 3.39 fps, capturing bandwidth 765902bytes/sec zm_monitor.cpp 1663
8/28/2021 12:13 zmc_m1 1064 INF FrontSouth: 1500 - Capturing at 3.35 fps, capturing bandwidth 768390bytes/sec zm_monitor.cpp 1663
8/28/2021 12:13 zmc_m6 1102 INF BabyCam: 1700 - Capturing at 4.00 fps, capturing bandwidth 56713bytes/sec zm_monitor.cpp 1663
8/28/2021 12:13 zmc_m7 1107 INF Laview: 1700 - Capturing at 4.00 fps, capturing bandwidth 130066bytes/sec zm_monitor.cpp 1663
HTOP offers the biggest clue. The last time every core was 100% utilized.
At one point I was able to get into the web interface long enough to see the Unraid log. Looks like it just freaked out because there was no useable RAM.
Any Suggestions?