It would be possible write a script that checks memory usage using the linux 'free' command and set this to run regularly with the User Scripts plugin. The 'ps' command might also be relevant if you want to monitor particular processes. What you do when you detect a trigger level will be up to the script - although it could either write entries to the syslog, or alternatively raise a notification at the Unraid level.