August 17, 2025Aug 17 I thought I'd share a script I setup to do daily monitoring of my ZFS pools. I run it with the User Scripts plug-in on a Daily schedule.The script will examine the output of zpool status and generate a message to send as a notification to the web GUI. All the configuration is in a block at the head of the script.#!/bin/bash # ==================== PARAMETERS ==================== # Set AUTO_DISCOVER to "true" to automatically find all pools, # or "false" to use the manual list below AUTO_DISCOVER="true" # Manual pool list (only used if AUTO_DISCOVER="false") # Use space-separated quoted strings like: ("hdd_main" "nvme_cache") POOLS=("hdd_main" "nvme_cache") # Notification settings NOTIFY_SUCCESS="true" # Set to "false" to only notify on errors/warnings ERROR_THRESHOLD=0 # Notify if READ/WRITE/CHECKSUM errors > this value SLOW_IO_THRESHOLD=10 # Notify if slow I/O count > this value (if supported) # Debug mode - set to "true" to see detailed parsing output DEBUG="false" # ==================================================== # Function to get all vdev error counts recursively get_vdev_errors() { local json="$1" local pool="$2" local vdev_path="$3" local read_errors=$(echo "$json" | jq -r "$vdev_path.read_errors // \"0\"") local write_errors=$(echo "$json" | jq -r "$vdev_path.write_errors // \"0\"") local checksum_errors=$(echo "$json" | jq -r "$vdev_path.checksum_errors // \"0\"") # Sum up errors from child vdevs if they exist local child_vdevs=$(echo "$json" | jq -r "$vdev_path.vdevs // empty | keys[]?" 2>/dev/null) for child in $child_vdevs; do local child_path="$vdev_path.vdevs[\"$child\"]" local child_errors=$(get_vdev_errors "$json" "$pool" "$child_path") read_errors=$((read_errors + $(echo "$child_errors" | cut -d: -f1))) write_errors=$((write_errors + $(echo "$child_errors" | cut -d: -f2))) checksum_errors=$((checksum_errors + $(echo "$child_errors" | cut -d: -f3))) done echo "$read_errors:$write_errors:$checksum_errors" } # Build pool list if [[ "$AUTO_DISCOVER" == "true" ]]; then mapfile -t POOLS < <(zpool list -H -o name) echo "Auto-discovered pools: ${POOLS[*]}" else echo "Using manual pool list: ${POOLS[*]}" fi # Get JSON status for all pools # JSON=$(zpool status -j) # Worked fine on my PC... :D JSON=$(zpool status -j | sed 's/\\/\\\\/g') # Clean up control characters in zpool's erronous JSON output if [[ $DEBUG == "true" ]]; then echo "JSON Output:" echo "$JSON" | jq . fi # Check each pool for POOL in "${POOLS[@]}"; do echo "=========================================" echo "Checking pool: $POOL" # Parse JSON for pool information POOL_STATE=$(echo "$JSON" | jq -r --arg POOL "$POOL" '.pools[$POOL].state // "UNKNOWN"') if [[ "$POOL_STATE" == "null" || "$POOL_STATE" == "UNKNOWN" ]]; then echo "ERROR: Pool $POOL not found in status output" /usr/local/emhttp/webGui/scripts/notify -e "ZFS POOL STATUS" -s "Pool Not Found" -d "Pool $POOL not found - check pool name" -i alert continue fi # Get total error counts for the pool POOL_ERRORS=$(get_vdev_errors "$JSON" "$POOL" ".pools[\"$POOL\"].vdevs[\"$POOL\"]") TOTAL_READ_ERRORS=$(echo "$POOL_ERRORS" | cut -d: -f1) TOTAL_WRITE_ERRORS=$(echo "$POOL_ERRORS" | cut -d: -f2) TOTAL_CHECKSUM_ERRORS=$(echo "$POOL_ERRORS" | cut -d: -f3) # Check for scan information (scrub/resilver in progress) SCAN_STATE=$(echo "$JSON" | jq -r --arg POOL "$POOL" '.pools[$POOL].scan_stats.state // "none"') SCAN_FUNCTION=$(echo "$JSON" | jq -r --arg POOL "$POOL" '.pools[$POOL].scan_stats.function // "none"') # Determine notification level and message DESCRIPTION="" SUBJECT="" ICON="normal" if [[ $DEBUG == "true" ]]; then echo " State: $POOL_STATE" echo " Read Errors: $TOTAL_READ_ERRORS" echo " Write Errors: $TOTAL_WRITE_ERRORS" echo " Checksum Errors: $TOTAL_CHECKSUM_ERRORS" echo " Scan State: $SCAN_STATE" echo " Scan Function: $SCAN_FUNCTION" fi # Analyze pool health case "$POOL_STATE" in "ONLINE") if [[ $TOTAL_READ_ERRORS -gt $ERROR_THRESHOLD || $TOTAL_WRITE_ERRORS -gt $ERROR_THRESHOLD || $TOTAL_CHECKSUM_ERRORS -gt $ERROR_THRESHOLD ]]; then SUBJECT="$POOL Errors Detected" DESCRIPTION="Pool <b>$POOL</b> is ONLINE but has errors: READ=$TOTAL_READ_ERRORS, WRITE=$TOTAL_WRITE_ERRORS, CHECKSUM=$TOTAL_CHECKSUM_ERRORS" ICON="warning" else SUBJECT="$POOL Healthy" DESCRIPTION="Pool <b>$POOL</b> is ONLINE and healthy" ICON="normal" fi ;; "DEGRADED") SUBJECT="$POOL Degraded" DESCRIPTION="Pool <b>$POOL</b> is DEGRADED" if [[ $TOTAL_READ_ERRORS -gt 0 || $TOTAL_WRITE_ERRORS -gt 0 || $TOTAL_CHECKSUM_ERRORS -gt 0 ]]; then DESCRIPTION+=" with errors: READ=$TOTAL_READ_ERRORS, WRITE=$TOTAL_WRITE_ERRORS, CHECKSUM=$TOTAL_CHECKSUM_ERRORS" fi DESCRIPTION+=" - May be rebuilding or need attention" ICON="warning" ;; "FAULTED"|"UNAVAIL") SUBJECT="$POOL CRITICAL" DESCRIPTION="Pool <b>$POOL</b> is $POOL_STATE - IMMEDIATE ATTENTION REQUIRED" if [[ $TOTAL_READ_ERRORS -gt 0 || $TOTAL_WRITE_ERRORS -gt 0 || $TOTAL_CHECKSUM_ERRORS -gt 0 ]]; then DESCRIPTION+=" Errors: READ=$TOTAL_READ_ERRORS, WRITE=$TOTAL_WRITE_ERRORS, CHECKSUM=$TOTAL_CHECKSUM_ERRORS" fi ICON="alert" ;; "OFFLINE") SUBJECT="$POOL Offline" DESCRIPTION="Pool <b>$POOL</b> is OFFLINE" ICON="warning" ;; *) SUBJECT="$POOL Unknown State" DESCRIPTION="Pool <b>$POOL</b> has unknown state: $POOL_STATE" ICON="warning" ;; esac # Add scan information if relevant if [[ "$SCAN_STATE" == "SCANNING" ]]; then DESCRIPTION+=" ($SCAN_FUNCTION in progress)" elif [[ "$SCAN_STATE" == "FINISHED" && "$SCAN_FUNCTION" == "SCRUB" ]]; then SCAN_ERRORS=$(echo "$JSON" | jq -r --arg POOL "$POOL" '.pools[$POOL].scan_stats.errors // "0"') if [[ "$SCAN_ERRORS" != "0" ]]; then DESCRIPTION+=" (Last scrub found $SCAN_ERRORS errors)" if [[ "$ICON" == "normal" ]]; then ICON="warning" fi fi fi # Send notification based on settings if [[ "$ICON" != "normal" || "$NOTIFY_SUCCESS" == "true" ]]; then /usr/local/emhttp/webGui/scripts/notify -e "ZFS POOL STATUS" -s "$SUBJECT" -d "$DESCRIPTION" -i "$ICON" fi echo "$DESCRIPTION" # Display detailed vdev states if there are issues if [[ "$ICON" != "normal" ]]; then echo "Detailed vdev information:" zpool status "$POOL" fi echo "" done echo "=========================================" echo "Pool monitoring complete." Edited August 19, 2025Aug 19 by NIronwolf
August 18, 2025Aug 18 Thanks for sharing. One of my pools is currently resilvering a disk and it showed this:Is this expected? I was expecting to see "degraded".
August 19, 2025Aug 19 Author Interesting. If you run it in debug mode or do a zpool status -j does the node for pools[tank].state show as "DEGRADED"? I haven't had a degraded pool yet so it's not a case I've been able to test.
August 19, 2025Aug 19 3 hours ago, NIronwolf said:zpool status -j does the node for pools[tank].state show as "DEGRADED"?This?root@Test5:~# zpool status -j tank{"output_version":{"command":"zpool status","vers_major":0,"vers_minor":1},"pools":{"tank":{"name":"tank","state":"DEGRADED","pool_guid":"13192947393055723064","txg":"32","spa_version":"5000","zpl_version":"5","status":"One or more devices are faulted in response to persistent errors.\n\tSufficient replicas exist for the pool to continue functioning in a\n\tdegraded state.\n","action":"Replace the faulted device, or use 'zpool clear' to mark the device\n\trepaired.\n","vdevs":{"tank":{"name":"tank","vdev_type":"root","guid":"13192947393055723064","class":"normal","state":"DEGRADED","alloc_space":"984K","total_space":"444G","def_space":"323G","read_errors":"0","write_errors":"0","checksum_errors":"0","vdevs":{"raidz1-0":{"name":"raidz1-0","vdev_type":"raidz","guid":"7898802336716648820","class":"normal","state":"DEGRADED","alloc_space":"984K","total_space":"444G","def_space":"323G","rep_dev_size":"444G","read_errors":"0","write_errors":"0","checksum_errors":"0","vdevs":{"sdh1":{"name":"sdh1","vdev_type":"disk","guid":"3970299765321604050","path":"/dev/sdh1","class":"normal","state":"ONLINE","rep_dev_size":"111G","phys_space":"112G","read_errors":"0","write_errors":"0","checksum_errors":"0","slow_ios":"0"},"sdi1":{"name":"sdi1","vdev_type":"disk","guid":"7219673904986742476","path":"/dev/sdi1","class":"normal","state":"FAULTED","rep_dev_size":"111G","phys_space":"112G","read_errors":"0","write_errors":"0","checksum_errors":"0","slow_ios":"0","aux":"ERR_EXCEEDED"},"sdk1":{"name":"sdk1","vdev_type":"disk","guid":"11182709466181077249","path":"/dev/sdk1","class":"normal","state":"ONLINE","rep_dev_size":"111G","phys_space":"112G","read_errors":"0","write_errors":"0","checksum_errors":"0","slow_ios":"0"},"sdl1":{"name":"sdl1","vdev_type":"disk","guid":"5032241952026091966","path":"/dev/sdl1","class":"normal","state":"ONLINE","rep_dev_size":"111G","phys_space":"112G","read_errors":"0","write_errors":"0","checksum_errors":"0","slow_ios":"0"}}}}}},"error_count":"0"}}}Note that this is a different pool, as the other resilver already finished.
August 19, 2025Aug 19 Author Yes. This line should have made $POOL_STATUS set to "DEGRADED" and it should run the different case section.POOL_STATE=$(echo "$JSON" | jq -r --arg POOL "$POOL" '.pools[$POOL].state // "UNKNOWN"')running"DEGRADED") SUBJECT="$POOL Degraded" DESCRIPTION="Pool <b>$POOL</b> is DEGRADED" if [[ $TOTAL_READ_ERRORS -gt 0 || $TOTAL_WRITE_ERRORS -gt 0 || $TOTAL_CHECKSUM_ERRORS -gt 0 ]]; then DESCRIPTION+=" with errors: READ=$TOTAL_READ_ERRORS, WRITE=$TOTAL_WRITE_ERRORS, CHECKSUM=$TOTAL_CHECKSUM_ERRORS" fi DESCRIPTION+=" - May be rebuilding or need attention" ICON="warning" ;;If you run something like POOL_STATE=$(zpool status -j | jq -r --arg POOL "$POOL" '.pools[$POOL].state // "UNKNOWN"')echo $POOL_STATEIt should print DEGRADEDAnd then I asked AI about it.The issue is that your JSON contains unescaped control characters (specifically newlines \n and tabs \t in the strings).It suggested processing the zpool command output. Can you verify by changing the line that stores the JSON?-- JSON=$(zpool status -j)++ JSON=$(zpool status -j | sed 's/\\/\\\\/g')
August 19, 2025Aug 19 Well, this pool works correctly even without the change. Sorry, I didn't test before, I'll see if I can reproduce the issue again with a different pool and then retest with the changes.
August 20, 2025Aug 20 Author Thanks. The change should be fine either way. If anyone runs across issues I haven't been able to test, I'm happy to update.
August 23, 2025Aug 23 Had a chance to take another look at this; turns out, the script was not the problem.The pool I originally posted about was running on Unraid 7.0.1, and that OpenZFS doesn't yet support the zpool status -j option, and that was the reason for the unknown state; the script was fine.So as long as it's used with Unraid 7.1.0+, it should always work correctly.Posted a link to this script from the FAQ page, assume that's OK with you.
January 18Jan 18 This seems very nice. Is there a way to make Unraid notify via "agents" if it fails? Might need some official work to make third party notifications work?Edit: Nevermind, I see it actually works! 👏 Edited January 18Jan 18 by Ademar
January 19Jan 19 It should use whatever you have set to send the warning/alert notification in Unraid (Settings - Notifications)
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.