March 12, 20224 yr A simple but powerful script which produces Unraid notifications if specific error words are found in the syslog. It can be executed for example hourly by the user scripts plugin. Donate? 🤗 #!/bin/bash # ##################################### # Name: Syslog notify v1.4 # Description: Creates notification if log contains errors or its size exceeds 90% of the available space # Author: Marc Gutt # ##################################### # ##################################### # Settings # ##################################### # get most recent syslog file syslog_file=$(ls -t /var/log/syslog{.[0-9],} 2>/dev/null | head -n 1) # words that should cause a notification words="corrupt|error|fail|tainted|mismatch|invalid" # store line number of last found error in this file line_number_file="/tmp/syslog-notify-last-error-line-number.log" # ignore these phrases (you can't use more than 4 wildcards per line!) ignore_lines=( 'kernel: CIFS: VFS: \\*\* error -9 on ioctl to get interface list' # unsolvable message from UD plugin 'sshd[*]: Read error from remote host * port *: Connection reset by peer' # interrupted ssh connection 'sshd[*]: Read error from remote host * port *: Connection timed out' # interrupted ssh connection 'lsof[*] general protection fault ip:*' # unknown error which can't be solved 'kernel: NFS: server * error: fileid changed' # if a file on an nfs mounted server is moved from one disk to another (not critical) ) # ##################################### # Script # ##################################### # make script race condition safe if [[ -d "/tmp/${0//\//_}" ]] || ! mkdir "/tmp/${0//\//_}"; then echo "Script is already running!" && exit 1; fi; trap 'rmdir "/tmp/${0//\//_}"' EXIT; # obtain line number of last check if [[ -f "$line_number_file" ]]; then line_number_start=$(cat "$line_number_file") # syslog has been truncated if [[ ! "$line_number_start" =~ ^[0-9]+$ ]] || [[ $line_number_start -gt $(grep -c ^ "$syslog_file") ]]; then line_number_start=0 fi # store last line number on first execution else line_number_start=$(grep -c ^ "$syslog_file") line_number_start=$((line_number_start-100)) echo "$line_number_start" >"$line_number_file" fi echo "check for errors in $syslog_file after line $line_number_start" # parse logs EOL=$'\n' errors="" while IFS= read -r line; do # ignore specific lines for ignore_line in "${ignore_lines[@]}"; do IFS=\* read -r one two three four <<< "$ignore_line" if [[ $line == *"$one"*"$two"*"$three"*"$four" ]]; then continue 2 fi done # remember last line last_line="$line" # combine multiple error messages errors="$errors$EOL$line" done < <(tail -n +"$((line_number_start+1))" "$syslog_file" | grep --text --ignore-case --extended-regexp "($words)") # create notification for new errors if [[ $errors ]]; then # remember line number of last error line_number_start=$(grep --text --fixed-strings --line-regexp --line-number "$last_line" "$syslog_file" | cut -f 1 -d ":") echo "$line_number_start" >"$line_number_file" # send notification /usr/local/emhttp/webGui/scripts/notify -i "alert" -s "syslog $(echo "$errors" | grep -ioP "($words)" | tr '[:upper:]' '[:lower:]' | sort -u | xargs)" -d "${errors:1}" exit # store last line number if no error has been found else line_number_start=$(grep -c ^ "$syslog_file") echo "$line_number_start" > "$line_number_file" fi # create notificaton if log exceeds usage of 90% log_size=$(df | grep -oP "[0-9]+(?=% /var/log)") if [[ ! -f /tmp/syslog-notify.size ]] && [[ $log_size -gt 90 ]]; then touch /tmp/syslog-notify.size /usr/local/emhttp/webGui/scripts/notify -i "alert" -s "log utilizes more than 90%!" -d "$(du -h /var/log/* | sort -h | tail)" elif [[ -f /tmp/syslog-notify.size ]]; then rm /tmp/syslog-notify.size fi E-mail notification example send by Unraid: You can test it by creating a custom error message in your syslog: logger Errortest
May 26, 20224 yr Great script! I'll be modifying it at the weekend to alert me to BTRFS scrub errors/corrupt files, and was exactly what I was looking for!
May 31, 20224 yr Hi, thanks for creating this it's really useful in combination with a discord webhook. Currently I am only getting notifications which say "Failed sending notification". The syslog reflects this through the following entry. Quote May 31 19:06:07 Tower emhttpd: cmd: /usr/local/emhttp/plugins/user.scripts/startScript.sh /tmp/user.scripts/tmpScripts/SyslogNotify/script May 31 19:06:12 Tower Discord.sh: Failed sending notification Any idea what would be causing the script to fail? If I run a test notification from the notification settings panel it works fine. Edited May 31, 20224 yr by wolfNZ
August 7, 20223 yr Author On 5/31/2022 at 9:15 AM, wolfNZ said: Currently I am only getting notifications which say "Failed sending notification". I don't know what you target is, but this script finds the word "fail" in the syslog line "May 31 19:06:12 Tower Discord.sh: Failed sending notification" and sends this line as a notification. So it works as expected.
August 7, 20223 yr Author Released v1.2 - now it is much faster as it does not parse the complete log file on every execution. Instead it uses the line number of the last found error as the starting point. - in addition it sends a notification if the syslog file exceeds 90% of the available space
November 19, 20223 yr Author At the moment I'm experimenting with log entries which occur extremely often and maybe should be added as a new feature to the above script: log_last=3 log_path="/var/log/syslog" while read -r count word; do echo -e "\nLast $log_last log entries which end to the word '$word' and appeared $count times:" grep "[ ']$word$" $(ls -tr "$log_path"*) | tail -n "$log_last" done < <(grep -hoP "[^ ']+$" "$log_path"* | sort | uniq -c | sort -nr | grep -P '^[ ]+[0-9]{4,}') For me it returns the following, which tells me that I should add further filtering regarding the time frame as the "worker process" do not happen anymore and maybe adding some whitelisting as "disabled state" (happens every night as I'm stopping containers to created backups) and "RAM-Disk synced" (custom entry I create on my own) log entries are expected: Last 3 log entries which end to the word '6' and appeared 3966 times: /var/log/syslog.1:Oct 30 22:36:31 thoth nginx: 2022/10/30 22:36:31 [alert] 11544#11544: worker process 19012 exited on signal 6 /var/log/syslog.1:Oct 30 22:36:33 thoth nginx: 2022/10/30 22:36:33 [alert] 11544#11544: worker process 19167 exited on signal 6 /var/log/syslog.1:Oct 30 22:36:35 thoth nginx: 2022/10/30 22:36:35 [alert] 11544#11544: worker process 19301 exited on signal 6 Last 3 log entries which end to the word 'state' and appeared 2538 times: /var/log/syslog:Nov 19 02:30:28 thoth kernel: docker0: port 5(veth9338c0f) entered disabled state /var/log/syslog:Nov 19 02:30:28 thoth kernel: docker0: port 5(veth9338c0f) entered blocking state /var/log/syslog:Nov 19 02:30:28 thoth kernel: docker0: port 5(veth9338c0f) entered forwarding state Last 3 log entries which end to the word 'synced' and appeared 2377 times: /var/log/syslog:Nov 19 11:30:01 thoth docker: RAM-Disk synced /var/log/syslog:Nov 19 12:00:01 thoth docker: RAM-Disk synced /var/log/syslog:Nov 19 12:30:01 thoth docker: RAM-Disk synced I executed the same code on a log of a broken server and it looks like this: Last 3 log entries which end to the word 'write-back' and appeared 8550 times: Nov 11 04:35:29 Tower kernel: x86/PAT: ipmiseld:4588 map pfn expected mapping type uncached-minus for [mem 0xbcdcb000-0xbcdcbfff], got write-back Nov 11 04:35:29 Tower kernel: x86/PAT: ipmiseld:4588 map pfn expected mapping type uncached-minus for [mem 0xbcdca000-0xbcdcafff], got write-back Nov 11 04:35:29 Tower kernel: x86/PAT: ipmiseld:4588 map pfn expected mapping type uncached-minus for [mem 0xbcdc9000-0xbcdc9fff], got write-back Last 3 log entries which end to the word 'failed' and appeared 4227 times: Nov 10 05:23:21 Tower nginx: 2022/11/10 05:23:21 [error] 6364#6364: shpool alloc failed Nov 10 05:23:21 Tower nginx: 2022/11/10 05:23:21 [error] 6364#6364: shpool alloc failed Nov 10 05:23:21 Tower nginx: 2022/11/10 05:23:21 [error] 6364#6364: shpool alloc failed Last 3 log entries which end to the word 'nchan_max_reserved_memory.' and appeared 4226 times: Nov 10 05:23:21 Tower nginx: 2022/11/10 05:23:21 [error] 6364#6364: nchan: Out of shared memory while allocating message of size 3623. Increase nchan_max_reserved_memory. Nov 10 05:23:21 Tower nginx: 2022/11/10 05:23:21 [error] 6364#6364: nchan: Out of shared memory while allocating message of size 4506. Increase nchan_max_reserved_memory. Nov 10 05:23:21 Tower nginx: 2022/11/10 05:23:21 [error] 6364#6364: nchan: Out of shared memory while allocating message of size 233. Increase nchan_max_reserved_memory. Last 3 log entries which end to the word 'memory' and appeared 4226 times: Nov 10 05:23:21 Tower nginx: 2022/11/10 05:23:21 [crit] 6364#6364: ngx_slab_alloc() failed: no memory Nov 10 05:23:21 Tower nginx: 2022/11/10 05:23:21 [crit] 6364#6364: ngx_slab_alloc() failed: no memory Nov 10 05:23:21 Tower nginx: 2022/11/10 05:23:21 [crit] 6364#6364: ngx_slab_alloc() failed: no memory Last 3 log entries which end to the word '"localhost"' and appeared 4226 times: Nov 10 05:23:21 Tower nginx: 2022/11/10 05:23:21 [error] 6364#6364: *195093 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/var?buffer_length=1 HTTP/1.1", host: "localhost" Nov 10 05:23:21 Tower nginx: 2022/11/10 05:23:21 [error] 6364#6364: *195094 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/disks?buffer_length=1 HTTP/1.1", host: "localhost" Nov 10 05:23:21 Tower nginx: 2022/11/10 05:23:21 [error] 6364#6364: *195095 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/wireguard?buffer_length=1 HTTP/1.1", host: "localhost" Last 3 log entries which end to the word '/devices' and appeared 2013 times: Nov 10 05:23:18 Tower nginx: 2022/11/10 05:23:18 [error] 6364#6364: MEMSTORE:00: can't create shared message for channel /devices Nov 10 05:23:19 Tower nginx: 2022/11/10 05:23:19 [error] 6364#6364: MEMSTORE:00: can't create shared message for channel /devices Nov 10 05:23:20 Tower nginx: 2022/11/10 05:23:20 [error] 6364#6364: MEMSTORE:00: can't create shared message for channel /devices Last 3 log entries which end to the word '/disks' and appeared 2005 times: Nov 10 05:23:19 Tower nginx: 2022/11/10 05:23:19 [error] 6364#6364: MEMSTORE:00: can't create shared message for channel /disks Nov 10 05:23:20 Tower nginx: 2022/11/10 05:23:20 [error] 6364#6364: MEMSTORE:00: can't create shared message for channel /disks Nov 10 05:23:21 Tower nginx: 2022/11/10 05:23:21 [error] 6364#6364: MEMSTORE:00: can't create shared message for channel /disks My dev server, which runs for 6 days now, returns nothing, which probably means its healthy, but I check only log entries, which appear more than 1000 times and maybe it should instead use the total size of the log file and use a variable value. Like if the log file has 1000 lines, it should return log entries which appear more than 100 times and if it has 5000 lines, than 500 times, and so on... Maybe some other users can test the code snippet and return some feedback? 😎
November 21, 20223 yr I tried to get your 3 entry part to work. Is there a specific placement for the code to work? I'm not seeing it in the output of the script nor in the log file. So I'm guessing its in the wrong place. I just placed it in the very bottom of the code. Lol
November 21, 20223 yr Author 2 hours ago, kizer said: Is there a specific placement for the code to work? Copy & Paste it into the Terminal. If you don't get any output, than your logs have a good condition. You could replace {4,} against {3,}. By that it finds all lines which appear only 100 times. But those should be normal.
February 7, 20233 yr On 5/27/2022 at 12:22 AM, boomam said: Great script! I'll be modifying it at the weekend to alert me to BTRFS scrub errors/corrupt files, and was exactly what I was looking for! Can you post your Modifications? @boomam Edited February 8, 20233 yr by Revan335
February 12, 20233 yr My Script with 70% Log and BTRFS include. I hope this working by BTRFS Notifications/Problems. #!/bin/bash # ##################################### # Name: Syslog notify v1.2 # Description: Creates notification if log contains errors or its size exceeds 70% of the available space # Author: Marc Gutt # ##################################### # ##################################### # Settings # ##################################### # get most recent syslog file syslog_file=$(ls -t /var/log/syslog{.[0-9],} 2>/dev/null | head -n 1) # words that should cause a notification words="corrupt|error|fail|tainted|BTRFS" # store line number of last found error in this file log_file="/tmp/syslog-notify-last-error-line-number.log" # log what is being done verbose=false # ##################################### # Script # ##################################### # make script race condition safe if [[ -d "/tmp/${0//\//_}" ]] || ! mkdir "/tmp/${0//\//_}"; then echo "Script is already running!" && exit 1; fi; trap 'rmdir "/tmp/${0//\//_}"' EXIT; # check user settings [[ $verbose == 0 ]] || [[ $verbose == false ]] && unset verbose # obtain line number of last check if [[ -f "$log_file" ]]; then line_number_start=$(cat "$log_file") # syslog has been truncated if [[ $line_number_start -gt $(grep -c ^ "$syslog_file") ]]; then [[ $verbose ]] && echo "Monitoring of a new syslog file begins" line_number_start=0 fi # store last line number on first execution else line_number_start=$(grep -c ^ "$syslog_file") line_number_start=$((line_number_start-100)) [[ $verbose ]] && echo "Monitoring syslog starts from line $line_number_start" echo "$line_number_start" > "$log_file" fi # parse logs EOL=$'\n' errors="" while read -r line; do # remember last line last_line="$line" # combine multiple error messages errors="$errors$EOL$line" done < <(tail -n +"$((line_number_start+1))" "$syslog_file" | grep -iP "($words)") # create notification for new errors if [[ $errors ]]; then # remember line number of last error line_number_start=$(grep -nFx "$last_line" "$syslog_file" | cut -f 1 -d ":") [[ $verbose ]] && echo "Monitoring syslog continues from line $line_number_start" echo "$line_number_start" > "$log_file" # send notification /usr/local/emhttp/webGui/scripts/notify -i "alert" -s "syslog $(echo "$errors" | grep -ioP "($words)" | tr '[:upper:]' '[:lower:]' | sort -u | xargs)" -d "${errors:1}" exit else # store last line number if no error has been found line_number_start=$(grep -c ^ "$syslog_file") [[ $verbose ]] && echo "Monitoring syslog continues from last line $line_number_start" echo "$line_number_start" > "$log_file" fi # create notificaton if log exceeds usage of 70% log_size=$(df | grep -oP "[0-9]+(?=% /var/log)") if [[ ! -f /tmp/syslog-notify.size ]] && [[ $log_size -gt 70 ]]; then touch /tmp/syslog-notify.size /usr/local/emhttp/webGui/scripts/notify -i "alert" -s "log utilizes more than 90%!" -d "$(du -h /var/log/* | sort -h | tail)" elif [[ -f /tmp/syslog-notify.size ]]; then rm /tmp/syslog-notify.size fi
March 11, 20233 yr Author 12 hours ago, Revan335 said: Have the Script a ignored Option for example this Entry's? You can now add ignore strings. I already added your case and a different one which annoyed me.
July 20, 20232 yr Is this script still working properly in Unraid 6.12? I've noticed that the functionality that stores previous errors isnt working properly anymore. It emails me every time its executed and finds old errors. I've also noticed that the /tmp/syslog-notify-last-error-line-number.log file is always blank. Thanks!
April 26, 20251 yr Author Released version 1.4- added the new errors words "mismatch" and "invalid"- added two new error lines, which are ignored by default (see comments in code)- solved a bug which caused not finding new errors because last_line="$line" did not contain trailing whitespaces (solved by adding "IFS=")- solved a possible bug by adding "--text" to both greps (if syslog lines contain binary data)- added a check to verify that "$line_number_file" contains a number and if not set "line_number_start=0" so it does not break the following code (instead it restarts searching from the first line, but I think this is acceptable). @jordanchinI think the new version fixes your problem. Sorry for the late bug fix, but I wasn't able to reproduce it until I faced it on my own 😅
June 23, 20251 yr Instead of scheduling the execution of this script, is it possible to execute it in the background using the User script plugin?
June 23, 20251 yr 1 hour ago, Vitek said:Instead of scheduling the execution of this script, is it possible to execute it in the background using the User script plugin?Yes! So I'm running this on my Systems. Edited June 23, 20251 yr by Revan335
June 23, 20251 yr 19 minutes ago, Revan335 said:Yes! So I'm running this on my Systems.How?I have tried to use User script plugin run in background option but the script runs only once and exit.
June 23, 20251 yr 31 minutes ago, Vitek said:How?I have tried to use User script plugin run in background option but the script runs only once and exit.Its running every hour.
June 23, 20251 yr 5 minutes ago, Revan335 said:Its running every hour.Seems like you don't know difference between running in background and schedules run...
June 23, 20251 yr 1 minute ago, Vitek said:Seems like you don't know difference between running in background and schedules run...Maybe!Running in the Background is for me a single manuel Run that's are not in the front when he click on running.What do you do to running this in the Background? The Running in the Background Button?For me its scheduled a background process that automatic startet to his time than he was only showed when I manuell click on running. Edited June 23, 20251 yr by Revan335
June 23, 20251 yr Author 3 hours ago, Vitek said:execute it in the background using the User script plugin?The user scripts plugin is not able to run scripts infinite in the background. Or to be more precise: The user scripts plugin starts scripts by PHP and PHP has a maximum execution time. So the script will still run in the background, but the user scripts plugin looses control of it.If you are fine with this, we could of course change the script, so it runs infinite in the background, but how do you control its run status? Let's say a bug or OS error kills the script. How do you monitor this scenario? Wouldn't it be better to schedule the script instead?What is your main target to have it run permanently? Live notifications on new log entries? So something in combination with inotifywait? I'm not a huge fan of having a permanent process monitoring a file for rare new log entries, but yes of course this would be possible to build.
June 23, 20251 yr Hey @mgutt thanks for replying. Yeah, I want to run this script "forever" in the background so I can get notification immediately when errors occur. But I see, I have to stay with scheduled runs. Btw. I have a script that runs the inotifywait command inside, and using the user scripts plugin, I am able to run it using "Run in background" to monitor folders.
September 10, 2025Sep 10 @mgutt , thanks for putting this together. I use discord for alerting and a lot of the times it would fail to send the notification to discord because it would parse the JSON incorrectly with this error {"message": "The request body contains invalid JSON.", "code": 50109}.I've patched this section of the script to ensure Newlines, quotes, and backslashes in syslog entries are escaped safely. I'm posting it here in case you want to add it into your script as well. # create notification for new errors if [[ $errors ]]; then # remember line number of last error line_number_start=$(grep --text --fixed-strings --line-regexp --line-number "$last_line" "$syslog_file" | cut -f 1 -d ":") echo "$line_number_start" >"$line_number_file" # escape subject keywords subject="syslog $(echo "$errors" | grep -ioP "($words)" \ | tr '[:upper:]' '[:lower:]' | sort -u | xargs)" # escape description safely desc=$(echo "${errors:1}" | jq -Rs . | sed 's/^"//;s/"$//') # send notification /usr/local/emhttp/webGui/scripts/notify \ -i "alert" \ -s "$subject" \ -d "$desc" exit else # store last line number if no error has been found line_number_start=$(grep -c ^ "$syslog_file") echo "$line_number_start" > "$line_number_file" fi
September 12, 2025Sep 12 Author On 9/10/2025 at 5:24 PM, bobokun said:it would fail to send the notification to discord because it would parse the JSON incorrectly with this error {"message": "The request body contains invalid JSON.", "code": 50109}.Thank you for your input. I don't think that my script should handle this. Instead I opened a pull request to let the Discord notification agent handle the escaping:https://github.com/unraid/webgui/pull/2364
September 21, 2025Sep 21 Author @bobokun I fixed a small bug and tested the new version of the Discord Agent. Feel free to test it by yourself:https://github.com/unraid/webgui/pull/2364#issuecomment-3300620990Install via Unraid Web UI:Go to Plugins → Install PluginCopy and paste this URL:https://preview.dl.unraid.net/pr-plugins/pr-2364/webgui-pr-2364.plgI think it will be part of the next unraid release.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.