Skip to content
View in the app

A better way to browse. Learn more.

Unraid

A full-screen app on your home screen with push notifications, badges and more.

To install this app on iOS and iPadOS
  1. Tap the Share icon in Safari
  2. Scroll the menu and tap Add to Home Screen.
  3. Tap Add in the top-right corner.
To install this app on Android
  1. Tap the 3-dot menu (⋮) in the top-right corner of the browser.
  2. Tap Add to Home screen or Install app.
  3. Confirm by tapping Install.

Monitor ZFS Pools

Featured Replies

I thought I'd share a script I setup to do daily monitoring of my ZFS pools. I run it with the User Scripts plug-in on a Daily schedule.

The script will examine the output of zpool status and generate a message to send as a notification to the web GUI. All the configuration is in a block at the head of the script.

#!/bin/bash

# ==================== PARAMETERS ====================
# Set AUTO_DISCOVER to "true" to automatically find all pools, 
# or "false" to use the manual list below
AUTO_DISCOVER="true"

# Manual pool list (only used if AUTO_DISCOVER="false")
# Use space-separated quoted strings like: ("hdd_main" "nvme_cache")
POOLS=("hdd_main" "nvme_cache")

# Notification settings
NOTIFY_SUCCESS="true"     # Set to "false" to only notify on errors/warnings
ERROR_THRESHOLD=0         # Notify if READ/WRITE/CHECKSUM errors > this value
SLOW_IO_THRESHOLD=10      # Notify if slow I/O count > this value (if supported)

# Debug mode - set to "true" to see detailed parsing output
DEBUG="false"
# ====================================================

# Function to get all vdev error counts recursively
get_vdev_errors() {
    local json="$1"
    local pool="$2"
    local vdev_path="$3"
    
    local read_errors=$(echo "$json" | jq -r "$vdev_path.read_errors // \"0\"")
    local write_errors=$(echo "$json" | jq -r "$vdev_path.write_errors // \"0\"")
    local checksum_errors=$(echo "$json" | jq -r "$vdev_path.checksum_errors // \"0\"")
    
    # Sum up errors from child vdevs if they exist
    local child_vdevs=$(echo "$json" | jq -r "$vdev_path.vdevs // empty | keys[]?" 2>/dev/null)
    
    for child in $child_vdevs; do
        local child_path="$vdev_path.vdevs[\"$child\"]"
        local child_errors=$(get_vdev_errors "$json" "$pool" "$child_path")
        read_errors=$((read_errors + $(echo "$child_errors" | cut -d: -f1)))
        write_errors=$((write_errors + $(echo "$child_errors" | cut -d: -f2)))
        checksum_errors=$((checksum_errors + $(echo "$child_errors" | cut -d: -f3)))
    done
    
    echo "$read_errors:$write_errors:$checksum_errors"
}

# Build pool list
if [[ "$AUTO_DISCOVER" == "true" ]]; then
    mapfile -t POOLS < <(zpool list -H -o name)
    echo "Auto-discovered pools: ${POOLS[*]}"
else
    echo "Using manual pool list: ${POOLS[*]}"
fi

# Get JSON status for all pools
# JSON=$(zpool status -j)   # Worked fine on my PC...  :D
JSON=$(zpool status -j | sed 's/\\/\\\\/g') # Clean up control characters in zpool's erronous JSON output

if [[ $DEBUG == "true" ]]; then
    echo "JSON Output:"
    echo "$JSON" | jq .
fi

# Check each pool
for POOL in "${POOLS[@]}"; do
    echo "========================================="
    echo "Checking pool: $POOL"
    
    # Parse JSON for pool information
    POOL_STATE=$(echo "$JSON" | jq -r --arg POOL "$POOL" '.pools[$POOL].state // "UNKNOWN"')
    
    if [[ "$POOL_STATE" == "null" || "$POOL_STATE" == "UNKNOWN" ]]; then
        echo "ERROR: Pool $POOL not found in status output"
        /usr/local/emhttp/webGui/scripts/notify -e "ZFS POOL STATUS" -s "Pool Not Found" -d "Pool $POOL not found - check pool name" -i alert
        continue
    fi
    
    # Get total error counts for the pool
    POOL_ERRORS=$(get_vdev_errors "$JSON" "$POOL" ".pools[\"$POOL\"].vdevs[\"$POOL\"]")
    TOTAL_READ_ERRORS=$(echo "$POOL_ERRORS" | cut -d: -f1)
    TOTAL_WRITE_ERRORS=$(echo "$POOL_ERRORS" | cut -d: -f2)
    TOTAL_CHECKSUM_ERRORS=$(echo "$POOL_ERRORS" | cut -d: -f3)
    
    # Check for scan information (scrub/resilver in progress)
    SCAN_STATE=$(echo "$JSON" | jq -r --arg POOL "$POOL" '.pools[$POOL].scan_stats.state // "none"')
    SCAN_FUNCTION=$(echo "$JSON" | jq -r --arg POOL "$POOL" '.pools[$POOL].scan_stats.function // "none"')
    
    # Determine notification level and message
    DESCRIPTION=""
    SUBJECT=""
    ICON="normal"
    
    if [[ $DEBUG == "true" ]]; then
        echo "  State: $POOL_STATE"
        echo "  Read Errors: $TOTAL_READ_ERRORS"
        echo "  Write Errors: $TOTAL_WRITE_ERRORS"
        echo "  Checksum Errors: $TOTAL_CHECKSUM_ERRORS"
        echo "  Scan State: $SCAN_STATE"
        echo "  Scan Function: $SCAN_FUNCTION"
    fi
    
    # Analyze pool health
    case "$POOL_STATE" in
        "ONLINE")
            if [[ $TOTAL_READ_ERRORS -gt $ERROR_THRESHOLD || $TOTAL_WRITE_ERRORS -gt $ERROR_THRESHOLD || $TOTAL_CHECKSUM_ERRORS -gt $ERROR_THRESHOLD ]]; then
                SUBJECT="$POOL Errors Detected"
                DESCRIPTION="Pool <b>$POOL</b> is ONLINE but has errors: READ=$TOTAL_READ_ERRORS, WRITE=$TOTAL_WRITE_ERRORS, CHECKSUM=$TOTAL_CHECKSUM_ERRORS"
                ICON="warning"
            else
                SUBJECT="$POOL Healthy"
                DESCRIPTION="Pool <b>$POOL</b> is ONLINE and healthy"
                ICON="normal"
            fi
            ;;
        "DEGRADED")
            SUBJECT="$POOL Degraded"
            DESCRIPTION="Pool <b>$POOL</b> is DEGRADED"
            if [[ $TOTAL_READ_ERRORS -gt 0 || $TOTAL_WRITE_ERRORS -gt 0 || $TOTAL_CHECKSUM_ERRORS -gt 0 ]]; then
                DESCRIPTION+=" with errors: READ=$TOTAL_READ_ERRORS, WRITE=$TOTAL_WRITE_ERRORS, CHECKSUM=$TOTAL_CHECKSUM_ERRORS"
            fi
            DESCRIPTION+=" - May be rebuilding or need attention"
            ICON="warning"
            ;;
        "FAULTED"|"UNAVAIL")
            SUBJECT="$POOL CRITICAL"
            DESCRIPTION="Pool <b>$POOL</b> is $POOL_STATE - IMMEDIATE ATTENTION REQUIRED"
            if [[ $TOTAL_READ_ERRORS -gt 0 || $TOTAL_WRITE_ERRORS -gt 0 || $TOTAL_CHECKSUM_ERRORS -gt 0 ]]; then
                DESCRIPTION+=" Errors: READ=$TOTAL_READ_ERRORS, WRITE=$TOTAL_WRITE_ERRORS, CHECKSUM=$TOTAL_CHECKSUM_ERRORS"
            fi
            ICON="alert"
            ;;
        "OFFLINE")
            SUBJECT="$POOL Offline"
            DESCRIPTION="Pool <b>$POOL</b> is OFFLINE"
            ICON="warning"
            ;;
        *)
            SUBJECT="$POOL Unknown State"
            DESCRIPTION="Pool <b>$POOL</b> has unknown state: $POOL_STATE"
            ICON="warning"
            ;;
    esac
    
    # Add scan information if relevant
    if [[ "$SCAN_STATE" == "SCANNING" ]]; then
        DESCRIPTION+=" ($SCAN_FUNCTION in progress)"
    elif [[ "$SCAN_STATE" == "FINISHED" && "$SCAN_FUNCTION" == "SCRUB" ]]; then
        SCAN_ERRORS=$(echo "$JSON" | jq -r --arg POOL "$POOL" '.pools[$POOL].scan_stats.errors // "0"')
        if [[ "$SCAN_ERRORS" != "0" ]]; then
            DESCRIPTION+=" (Last scrub found $SCAN_ERRORS errors)"
            if [[ "$ICON" == "normal" ]]; then
                ICON="warning"
            fi
        fi
    fi
    
    # Send notification based on settings
    if [[ "$ICON" != "normal" || "$NOTIFY_SUCCESS" == "true" ]]; then
        /usr/local/emhttp/webGui/scripts/notify -e "ZFS POOL STATUS" -s "$SUBJECT" -d "$DESCRIPTION" -i "$ICON"
    fi
    
    echo "$DESCRIPTION"
    
    # Display detailed vdev states if there are issues
    if [[ "$ICON" != "normal" ]]; then
        echo "Detailed vdev information:"
        zpool status "$POOL"
    fi
    
    echo ""
done

echo "========================================="
echo "Pool monitoring complete."

Edited by NIronwolf

Thanks for sharing. One of my pools is currently resilvering a disk and it showed this:

image.png

Is this expected? I was expecting to see "degraded".

image.png

  • Author

Interesting. If you run it in debug mode or do a zpool status -j does the node for pools[tank].state show as "DEGRADED"? I haven't had a degraded pool yet so it's not a case I've been able to test.

3 hours ago, NIronwolf said:

zpool status -j does the node for pools[tank].state show as "DEGRADED"?

This?

root@Test5:~# zpool status -j tank

{"output_version":{"command":"zpool status","vers_major":0,"vers_minor":1},"pools":{"tank":{"name":"tank","state":"DEGRADED","pool_guid":"13192947393055723064","txg":"32","spa_version":"5000","zpl_version":"5","status":"One or more devices are faulted in response to persistent errors.\n\tSufficient replicas exist for the pool to continue functioning in a\n\tdegraded state.\n","action":"Replace the faulted device, or use 'zpool clear' to mark the device\n\trepaired.\n","vdevs":{"tank":{"name":"tank","vdev_type":"root","guid":"13192947393055723064","class":"normal","state":"DEGRADED","alloc_space":"984K","total_space":"444G","def_space":"323G","read_errors":"0","write_errors":"0","checksum_errors":"0","vdevs":{"raidz1-0":{"name":"raidz1-0","vdev_type":"raidz","guid":"7898802336716648820","class":"normal","state":"DEGRADED","alloc_space":"984K","total_space":"444G","def_space":"323G","rep_dev_size":"444G","read_errors":"0","write_errors":"0","checksum_errors":"0","vdevs":{"sdh1":{"name":"sdh1","vdev_type":"disk","guid":"3970299765321604050","path":"/dev/sdh1","class":"normal","state":"ONLINE","rep_dev_size":"111G","phys_space":"112G","read_errors":"0","write_errors":"0","checksum_errors":"0","slow_ios":"0"},"sdi1":{"name":"sdi1","vdev_type":"disk","guid":"7219673904986742476","path":"/dev/sdi1","class":"normal","state":"FAULTED","rep_dev_size":"111G","phys_space":"112G","read_errors":"0","write_errors":"0","checksum_errors":"0","slow_ios":"0","aux":"ERR_EXCEEDED"},"sdk1":{"name":"sdk1","vdev_type":"disk","guid":"11182709466181077249","path":"/dev/sdk1","class":"normal","state":"ONLINE","rep_dev_size":"111G","phys_space":"112G","read_errors":"0","write_errors":"0","checksum_errors":"0","slow_ios":"0"},"sdl1":{"name":"sdl1","vdev_type":"disk","guid":"5032241952026091966","path":"/dev/sdl1","class":"normal","state":"ONLINE","rep_dev_size":"111G","phys_space":"112G","read_errors":"0","write_errors":"0","checksum_errors":"0","slow_ios":"0"}}}}}},"error_count":"0"}}}

Note that this is a different pool, as the other resilver already finished.

  • Author

Yes. This line should have made $POOL_STATUS set to "DEGRADED" and it should run the different case section.

POOL_STATE=$(echo "$JSON" | jq -r --arg POOL "$POOL" '.pools[$POOL].state // "UNKNOWN"')

running


"DEGRADED") SUBJECT="$POOL Degraded" DESCRIPTION="Pool <b>$POOL</b> is DEGRADED" if [[ $TOTAL_READ_ERRORS -gt 0 || $TOTAL_WRITE_ERRORS -gt 0 || $TOTAL_CHECKSUM_ERRORS -gt 0 ]]; then DESCRIPTION+=" with errors: READ=$TOTAL_READ_ERRORS, WRITE=$TOTAL_WRITE_ERRORS, CHECKSUM=$TOTAL_CHECKSUM_ERRORS" fi DESCRIPTION+=" - May be rebuilding or need attention" ICON="warning" ;;


If you run something like

POOL_STATE=$(zpool status -j | jq -r --arg POOL "$POOL" '.pools[$POOL].state // "UNKNOWN"')
echo $POOL_STATE

It should print

DEGRADED


And then I asked AI about it.

The issue is that your JSON contains unescaped control characters (specifically newlines \n and tabs \t in the strings).

It suggested processing the zpool command output. Can you verify by changing the line that stores the JSON?

-- JSON=$(zpool status -j)
++ JSON=$(zpool status -j | sed 's/\\/\\\\/g')

Well, this pool works correctly even without the change. Sorry, I didn't test before, I'll see if I can reproduce the issue again with a different pool and then retest with the changes.

  • Author

Thanks. The change should be fine either way. If anyone runs across issues I haven't been able to test, I'm happy to update.

Had a chance to take another look at this; turns out, the script was not the problem.

The pool I originally posted about was running on Unraid 7.0.1, and that OpenZFS doesn't yet support the zpool status -j option, and that was the reason for the unknown state; the script was fine.

So as long as it's used with Unraid 7.1.0+, it should always work correctly.

Posted a link to this script from the FAQ page, assume that's OK with you.

  • Author

That's an honor. Thank you. I'm glad you found it useful!

  • 4 months later...

This seems very nice. Is there a way to make Unraid notify via "agents" if it fails? Might need some official work to make third party notifications work?

Edit: Nevermind, I see it actually works! 👏

Edited by Ademar

It should use whatever you have set to send the warning/alert notification in Unraid (Settings - Notifications)

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

Account

Navigation

Search

Search

Configure browser push notifications

Chrome (Android)
  1. Tap the lock icon next to the address bar.
  2. Tap Permissions → Notifications.
  3. Adjust your preference.
Chrome (Desktop)
  1. Click the padlock icon in the address bar.
  2. Select Site settings.
  3. Find Notifications and adjust your preference.