Hi,

This week I keep losing access to my WebUI after a fresh start. Everything else seems to work - SSH, containers, VMs etc I just can't access the WebUI. It's only been like this for the last 48 hours.

Diagnostics attached - can someone have a look please and see if they can spot the problem.

Thanks in advance!highlander-diagnostics-20260828-1224.zip

The diagnostics captured the failure in the previous syslog. nginx was still accepting WebGUI requests, but requests to PHP-FPM were timing out. This affected Docker, Main, login, and authentication requests. PHP-FPM was restarted at 02:43, but the same timeouts returned about 11 minutes later.

Many of the failed requests came from Tmux Terminal Manager polling, although the timeout occurs in the shared authentication request before the TTM endpoint runs, so this does not yet prove TTM is the cause.

As a test, please close all Tmux Terminal Manager browser tabs and temporarily disable that plugin after restoring access, then see whether the issue returns.

If it happens again, please do not restart PHP-FPM or reboot before capturing the following over SSH:

webui_state=/boot/logs/webui-stall-state.txt

{

date

ps -C php-fpm -o pid,ppid,stat,wchan:32,etime,cmd

ss -xapn | grep -E 'php-fpm|nginx'

for pid in $(pgrep -x php-fpm); do

echo "===== php-fpm PID $pid ====="

cat "/proc/$pid/stack"

done

curl --max-time 15 --unix-socket /var/run/nginx.socket -o /dev/null -sS -w 'HTTP %{http_code}, total %{time_total}s\n' http://localhost/Main

} >"$webui_state" 2>&1

diagnostics

Then please attach webui-stall-state.txt and the newly generated diagnostics.

Thanks for replying. I stopped using TTM as I saw that as well, but it's just happened again.

Here's the new diagnostics

The new file command hangs when I try to run it

highlander-diagnostics-20260828-1643.zip

highlander-diagnostics-20260829-0128.zip

It seems to happen when I'm running a script and Fix Common Problems is running at the same time????

The latest diagnostics confirm the same underlying condition: three pgrep processes are stuck in uninterruptible sleep, along with a Plex process. This can prevent WebGUI requests that inspect running processes from completing.

Fix Common Problems was running near the time of the capture, but the diagnostics do not show that it initiated the problem. Some of the blocked pgrep processes were already present before that scan, and on the previous boot another FCP scan occurred after the WebGUI had already begun timing out. Similarly, the User Scripts requests timed out during the shared authentication check, before the requested script handler could run.

When this happens again, please run the following from an existing SSH session before restarting anything:

echo w > /proc/sysrq-trigger

diagnostics

Also let us know the exact user script you were running, what it does, and whether it accesses the rclone mount or Plex/media paths. That will let us compare its activity with the blocked-task stacks.

10 minutes ago, JorgeB said:

When this happens again, please run the following from an existing SSH session before restarting anything:

echo w > /proc/sysrq-trigger

diagnostics

I just woke up to it being down again. I'm not sure if the command adds anything to the diagnostics as nothing was displayed in the terminal. It's driving me mad as all was good a few days ago!

root@Highlander:~# echo w > /proc/sysrq-trigger

root@Highlander:~# diagnostics

Starting diagnostics collection... done.

ZIP file '/boot/logs/highlander-diagnostics-20260829-0900.zip' created.

12 minutes ago, JorgeB said:

Also let us know the exact user script you were running, what it does, and whether it accesses the rclone mount or Plex/media paths. That will let us compare its activity with the blocked-task stacks.

I've been doing a lot of work to optimise my zfs snapshots, recordsizes, arr + plex + nzbdav/Decypharr stack over the last month as well as testing silo_server, so something in there is probably the issue, particularly zfs as I don't really understand it yet. At the time of the crash in that quote I think I was running this script I've been using on shares to find the best recordsizes - the silo directory is 200GB+ so that could be why, but I've run the script below in the past against large directories with no issues.

#!/bin/bash

# ====================================================================

# SCRIPT: ZFS performance-aligned File Size Distribution Analyzer

# VERSION: 1.0

# DESCRIPTION: Analyzes file size distributions in the current directory

# (or a specified target) to help align ZFS recordsizes

# and find block size mismatch bottlenecks.

# ====================================================================

TARGET_DIR="${1:-.}"

if [ ! -d "$TARGET_DIR" ]; then

echo "[-] ERROR: Directory '$TARGET_DIR' does not exist."

exit 1

fi

# Convert relative path to absolute for display

ABS_DIR=$(cd "$TARGET_DIR" && pwd)

echo "================================================================="

echo " ZFS BLOCK ALIGNMENT FILE SIZE DISTRIBUTION "

echo "================================================================="

echo "Target Directory: $ABS_DIR"

echo "Analyzing files... (This may take a moment for large directories)"

echo ""

# Scan target directory and pipe file counts/sizes into the awk bucket-analyzer

find "$TARGET_DIR" -type f -exec du -b {} + 2>/dev/null | awk '

BEGIN {

# ZFS Performance Buckets

b[1] = 4096; # 4KB

b[2] = 16384; # 16KB

b[3] = 65536; # 64KB

b[4] = 131072; # 128KB

b[5] = 1048576; # 1MB

lbl[1] = "< 4KB ";

lbl[2] = "4KB - 16KB ";

lbl[3] = "16KB - 64KB";

lbl[4] = "64KB-128KB ";

lbl[5] = "128KB - 1MB";

lbl[6] = "> 1MB ";

for (i=1; i<=6; i++) {

count[i] = 0;

sum[i] = 0;

}

}

{

size = $1;

found = 0;

for (i = 1; i <= 5; i++) {

if (size <= b[i]) {

count[i]++;

sum[i] += size;

found = 1;

break;

}

}

if (!found) {

count[6]++;

sum[6] += size;

}

}

END {

printf "%-15s | %10s | %15s | %10s\n", "SIZE BUCKET", "FILE COUNT", "TOTAL SIZE", "PERCENTAGE";

print "-----------------------------------------------------------------";

tot_files = 0; tot_bytes = 0;

for (i = 1; i <= 6; i++) {

tot_files += count[i];

tot_bytes += sum[i];

}

for (i = 1; i <= 6; i++) {

p = 0;

if (tot_bytes > 0) p = (sum[i] / tot_bytes) * 100;

h_bytes = sum[i];

unit = "B ";

if (h_bytes >= 1073741824) { h_bytes /= 1073741824; unit = "GB"; }

else if (h_bytes >= 1048576) { h_bytes /= 1048576; unit = "MB"; }

else if (h_bytes >= 1024) { h_bytes /= 1024; unit = "KB"; }

printf "%-15s | %10d | %11.2f %-2s | %9.1f%%\n", lbl[i], count[i], h_bytes, unit, p;

}

print "-----------------------------------------------------------------";

h_tot = tot_bytes; tot_unit = "B ";

if (h_tot >= 1073741824) { h_tot /= 1073741824; tot_unit = "GB"; }

else if (h_tot >= 1048576) { h_tot /= 1048576; tot_unit = "MB"; }

else if (h_tot >= 1024) { h_tot /= 1024; tot_unit = "KB"; }

printf "%-15s | %10d | %11.2f %-2s | %9.1f%%\n", "TOTALS", tot_files, h_tot, tot_unit, 100.0;

print "=================================================================";

# Output recommendations based on footprint (data density weight)

print "\n[ZFS RECORDSIZE ALIGNMENT RECOMMENDATIONS]:";

max_idx = 1;

max_val = sum[1];

for (i = 2; i <= 6; i++) {

if (sum[i] > max_val) {

max_val = sum[i];

max_idx = i;

}

}

if (tot_files == 0) {

print " -> No files detected in target directory. Recommended: Default 128K.";

} else if (max_idx 1 || max_idx 2) {

print " -> Majority data footprint: Small Files (< 16KB)\n RECOMMENDED RECORDSIZE: 16K (Perfect for SQLite/Postgres DB datasets!)";

} else if (max_idx == 3) {

print " -> Majority data footprint: Medium-Small Files (16KB - 64KB)\n RECOMMENDED RECORDSIZE: 64K";

} else if (max_idx == 4) {

print " -> Majority data footprint: Medium Files (64KB - 128KB)\n RECOMMENDED RECORDSIZE: 128K (Unraid default profile)";

} else {

print " -> Majority data footprint: Large Files (> 128KB)\n RECOMMENDED RECORDSIZE: 1M (Gold standard for media/large binary streaming datasets!)";

}

}'

Latest diags attached. For this crash I was just moving files from mnt/cache to /mnt/appdata as /mnt/cache was over 80% utliised and the AI said this could be causing problems for ZFS snapshots and behind my issues. Once I'd finished I went to bed but it looks like some weird stuff went off around 3:30am including my VM Buzz shutting down which wasn't initiated by me. Script wise, the only scripts running overnight were my hourly maintenance script below:

#!/bin/bash

### ====================================================================

### SCRIPT: Hourly Maintenance

### VERSION: 2.3 (Boot-Sync Gated & FUSE-Safe Bypass)

### DESCRIPTION: Handles hourly snapshots, database health monitoring,

### and intelligent FUSE cache safety audits for Highlander.

### Synchronized with the v8.0 Startup State Machine.

### ====================================================================

### --- 1. STARTUP SYNC GUARD ---

# Prevent Hourly Maintenance from triggering or throwing errors if the array is still booting.

# Loops and waits up to 5 minutes (300s) for the Master Startup flag to be created.

READY_FLAG="/var/run/highlander_ready"

MAX_BOOT_WAIT=300

BOOT_WAITED=0

while [ ! -f "$READY_FLAG" ]; do

if [ $BOOT_WAITED -ge $MAX_BOOT_WAIT ]; then

echo "[$(date '+%H:%M:%S')] ERROR: Master startup did not complete within ${MAX_BOOT_WAIT}s. Aborting Hourly Maintenance to prevent database/ZFS conflicts."

exit 1

fi

echo "[$(date '+%H:%M:%S')] WAIT: Master startup is still running. Waiting 15s before retrying..."

sleep 15

((BOOT_WAITED+=15))

done

### --- 2. USER CONFIGURATION ---

# FUSE-Bypass Logging: Writes directly to physical cache NVMe to prevent filesystem locks

LOG_FILE="/mnt/cache/appdata/other/scripts/hourly/hourly.log"

SANOID_CONF_DIR="/boot/config/plugins/sanoid"

VFS_CACHE_DIR="/mnt/samsung/streaming_cache/nzbdav"

CACHE_DRIVE_MAX_UTILISATION="70%"

# MULTI-DATABASE SENTINEL SETTINGS

PG_CONTAINERS=("postgresql16" "silo_postgres")

DB_USERS=("derycks" "silo")

DB_NAMES=("postgres" "silo")

MAX_CONNS=250

SLOW_QUERY_MAX_SEC=300

### --- 3. LOGGING & NOTIFICATIONS ---

mkdir -p "$(dirname "$LOG_FILE")"

log_event() {

local TYPE="$1"

local MSG="$2"

echo "[$(date '+%H:%M:%S')] $TYPE: $MSG" | tee -a "$LOG_FILE"

}

send_unraid_notify() {

local SUBJECT="$1"

local MESSAGE="$2"

local IMPORTANCE="$3" # normal, warning, alert

/usr/local/emhttp/webGui/scripts/notify -e "Highlander Sentinel" -s "$SUBJECT" -d "$MESSAGE" -i "$IMPORTANCE"

}

log_event "INFO" "--- STARTING HOURLY SYSTEM AUDIT (v2.3) ---"

### --------------------------------------------------------------------------------------

### 1. HOURLY SANOID SNAPSHOTS

### --------------------------------------------------------------------------------------

log_event "INFO" "Ensuring Sanoid configuration link..."

ln -sf "$SANOID_CONF_DIR/sanoid.conf" /etc/sanoid/sanoid.conf

log_event "INFO" "Starting Sanoid snapshot and prune process..."

/usr/local/sbin/sanoid --take-snapshots --prune

SANOID_EXIT=$?

if [ $SANOID_EXIT -ne 0 ]; then

log_event "ERROR" "Sanoid snapshot process FAILED (Exit Code: $SANOID_EXIT)."

send_unraid_notify "BACKUP FAILURE" "Sanoid failed to take snapshots. Check logs at $LOG_FILE" "alert"

else

log_event "INFO" "Sanoid snapshots completed successfully."

fi

### --------------------------------------------------------------------------------------

### 2. MULTI-POSTGRES SENTINEL (Auto-Correction & Maintenance)

### --------------------------------------------------------------------------------------

for i in "${!PG_CONTAINERS[@]}"; do

CONTAINER="${PG_CONTAINERS[$i]}"

DB_USER="${DB_USERS[$i]}"

DB_NAME="${DB_NAMES[$i]}"

log_event "INFO" "Running health audit for $CONTAINER..."

# Pre-flight check: Is the database container running?

if [ "$(docker inspect -f '{{.State.Running}}' "$CONTAINER" 2>/dev/null)" != "true" ]; then

log_event "ERROR" "Container $CONTAINER is DOWN! Attempting auto-restart..."

send_unraid_notify "CRITICAL" "Database $CONTAINER is DOWN. Auto-restart attempted." "alert"

docker start "$CONTAINER" > /dev/null 2>&1

continue

fi

# 1. Terminate Zombie/Stalled Queries

ZOMBIES=$(docker exec "$CONTAINER" psql -U "$DB_USER" -d "$DB_NAME" -t -c \

"SELECT count(*) FROM pg_stat_activity WHERE state != 'idle' AND (now() - query_start) > interval '$SLOW_QUERY_MAX_SEC seconds';" | xargs)

[[ $ZOMBIES =~ ^[0-9]+$ ]] || ZOMBIES=0

if [ "$ZOMBIES" -gt 0 ]; then

log_event "ACTION" "Detected $ZOMBIES hung queries in $CONTAINER. Terminating..."

docker exec "$CONTAINER" psql -U "$DB_USER" -d "$DB_NAME" -c \

"SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state != 'idle' AND (now() - query_start) > interval '$SLOW_QUERY_MAX_SEC seconds';" > /dev/null

send_unraid_notify "Maintenance" "Terminated $ZOMBIES zombie queries stalled over $SLOW_QUERY_MAX_SEC seconds in $CONTAINER." "warning"

fi

# 2. Connection Pool Monitoring

CONNS=$(docker exec "$CONTAINER" psql -U "$DB_USER" -d "$DB_NAME" -t -c "SELECT count(*) FROM pg_stat_activity;" | xargs)

[[ $CONNS =~ ^[0-9]+$ ]] || CONNS=0

if [ "$CONNS" -gt "$MAX_CONNS" ]; then

log_event "WARNING" "High connection count in $CONTAINER: $CONNS / $MAX_CONNS"

send_unraid_notify "High Load" "Database $CONTAINER connections ($CONNS) nearing capacity limits." "warning"

fi

# 3. Auto-Maintenance: Vacuum Analyze

log_event "INFO" "Running background maintenance (Analyze) on $CONTAINER..."

if ! docker exec "$CONTAINER" psql -U "$DB_USER" -d "$DB_NAME" -c "VACUUM ANALYZE;" > /dev/null 2>&1; then

log_event "ERROR" "Maintenance (VACUUM ANALYZE) failed on $CONTAINER."

send_unraid_notify "Maintenance Error" "Database $CONTAINER VACUUM ANALYZE failed." "warning"

else

log_event "INFO" "Database $CONTAINER vacuum maintenance completed successfully."

fi

done

### --------------------------------------------------------------------------------------

### 3. VFS-SMART CACHE PRUNING AUDIT

### --------------------------------------------------------------------------------------

log_event "INFO" "VFS CACHE: Analyzing active mount settings..."

# Detect the active VFS cache mode from the running rclone process

RCLONE_PID=$(pgrep -f "rclone mount" | head -n1)

if [ -n "$RCLONE_PID" ]; then

VFS_MODE_DETECTED=$(ps -o args= -p "$RCLONE_PID" | grep -oE -- '--vfs-cache-mode [a-z]+' | awk '{print $2}')

log_event "INFO" "VFS CACHE: Detected active VFS mode: '${VFS_MODE_DETECTED:-unknown}'"

else

VFS_MODE_DETECTED="unknown"

log_event "WARNING" "VFS CACHE: No active rclone mount process found."

fi

# Apply targeted pruning behavior based on VFS mode

if [[ "$VFS_MODE_DETECTED" "writes" ]] || [[ "$VFS_MODE_DETECTED" "minimal" ]]; then

log_event "INFO" "VFS CACHE: Handing cache management to rclone's internal GC. Manual pruning skipped to protect active uploads."

# Emergency warning check: If the SSD is critically full, alert the user instead of deleting files

CURRENT_UTIL=$(df -P "$VFS_CACHE_DIR" | awk 'NR==2 {print $5}' | sed 's/%//')

if [ "$CURRENT_UTIL" -gt 95 ]; then

log_event "CRITICAL" "Samsung SSD is over 95% full ($CURRENT_UTIL%) under 'writes' mode! Uploads may stall."

send_unraid_notify "CRITICAL: Storage Exhaustion" "VFS Cache Drive is critically full ($CURRENT_UTIL%). Manual pruning bypassed to protect uploads. Please inspect upload queues." "alert"

fi

elif [[ "$VFS_MODE_DETECTED" == "full" ]]; then

if [[ -d "$VFS_CACHE_DIR/vfs" ]]; then

log_event "INFO" "VFS CACHE: Initiating smart cache drive prune for 'full' read-cache..."

TARGET_UTIL_PCT="${CACHE_DRIVE_MAX_UTILISATION%%%}"

CURRENT_UTIL=$(df -P "$VFS_CACHE_DIR" | awk 'NR==2 {print $5}' | sed 's/%//')

if [ "$CURRENT_UTIL" -gt "$TARGET_UTIL_PCT" ]; then

log_event "INFO" "VFS CACHE: Utilization ($CURRENT_UTIL%) exceeds threshold ($TARGET_UTIL_PCT%). Purging oldest cache items..."

find "$VFS_CACHE_DIR/vfs" -type f -printf '%T@ %p\n' | sort -n | cut -d' ' -f2- | while read -r file; do

rm -f "$file"

CURRENT_UTIL=$(df -P "$VFS_CACHE_DIR" | awk 'NR==2 {print $5}' | sed 's/%//')

if [ "$CURRENT_UTIL" -le "$TARGET_UTIL_PCT" ]; then

log_event "INFO" "VFS CACHE: Target utilization achieved ($CURRENT_UTIL%)."

break

fi

done

else

log_event "INFO" "VFS CACHE: Utilization ($CURRENT_UTIL%) is safe under target ($TARGET_UTIL_PCT%)."

fi

fi

else

log_event "INFO" "VFS CACHE: Manual pruning skipped. Cache is either inactive or unmanaged."

fi

log_event "INFO" "--- HOURLY MAINTENANCE COMPLETE ---"

exit 0

highlander-diagnostics-20260829-0348.zip

I think that last diags didn't have what you need - just run again

highlander-diagnostics-20260829-0956.zip

I have just realised that my hourly script runs at 47 mins past the hour - I just assumed it ran on the hour. This I think coincides with my crashes. If use of pgrep is the issue then this line in my hourly is the culprit as it was added a few days ago:

RCLONE_PID=$(pgrep -f "rclone mount" | head -n1)

I've just removed that whole section as it's pretty pointless and will see if I get stability back

@JorgeB


I sadly still lost webui access again after deleting the line from my script 😒

highlander-diagnostics-20260829-2035.zip

I have added:

--contimeout 5

--timeout 10s

--low-level-retries 3

to my rclone mount to try and get it to timeout quicker

These diagnostics contain the SysRq output we needed. The command does not display anything in the terminal; it writes the blocked-task information to the syslog.

The results show that this is not primarily an nginx or PHP-FPM failure. Before the WebGUI begins timing out, multiple media-related processes are already blocked in filesystem reads. In particular, one Plex thread is waiting in:

request_wait_answer

__fuse_simple_request

fuse_flush

Other Plex reads are blocked in the FUSE read path, while the latest capture also contains a find process waiting during ZFS directory traversal.

Removing the pgrep section from the hourly script was a useful test. The issue still recurred and blocked pgrep processes remain, confirming that the script line was not the cause.

Those processes are getting stuck while inspecting another blocked process, which can then prevent WebGUI PHP requests that inspect the process table from completing. nginx consequently times out while waiting for PHP-FPM.

The rclone timeout changes are reasonable as an experiment, although they do not isolate the cause. --contimeout only affects connection establishment, while the other timeouts may not resolve a request already blocked between the local backend, rclone/FUSE, and the backing filesystem. A 10-second I/O timeout may also be fairly aggressive, so please treat it as a test rather than a confirmed fix.

The next useful test would be a clean reboot with the rclone mount, the nzbdav/Decypharr backend, and all consumers of that mount, including Plex and Silo, disabled from startup.

Observe it for at least the normal recurrence period. If the server remains stable, restore the backend, mount, and consumers one layer at a time.

At this point, the evidence points more toward the third-party media/FUSE stack or its interaction with the backing ZFS path. It does not look to me like TTM, FCP, the removed pgrep command, nginx, or PHP-FPM are causing the issues.

Thanks.

I made some other changes to reclone's VFS caching that seemed to stabilise things as I didn't have a crash for 4 hours+. But, then the docker autoupdate ran and tried to update some of my containers that I'd moved to Docker Compose. I think this triggered another crash and I'm just waiting for a task to complete before I reboot the server.

Will come back if I get another crash.

Should I remove all these lines?

--contimeout 5

--timeout 10s

--low-level-retries 3

Yes, for now I would remove all three lines and return to rclone’s defaults:

Since you also changed the VFS-cache configuration, returning these settings to their defaults will give us a cleaner test of whether the VFS changes themselves helped.

The Docker auto-update timing is another useful observation, but it does not yet prove that the updater caused the stall. For the next test, I would also temporarily disable automatic container updates so that only one variable is being tested.

If WebGUI access is lost again, please run:

echo w > /proc/sysrq-trigger

diagnostics

before restarting services or rebooting, and attach the new diagnostics.

@JorgeB I had a good 12+ hour run the longest in a while, but I've just lost access again. I thought I'd made it 🙁

Can you have another look please.

highlander-diagnostics-20260901-1058.zip

The diagnostics include the requested blocked-task capture and show the same underlying condition as before. A Plex scanner is again blocked waiting for a FUSE request, while Plex, Silo, and ffprobe have additional filesystem reads blocked. Several pgrep processes are also blocked, after which nginx times out waiting for PHP-FPM.

There was no Docker auto-update during this boot before the failure, so that does not appear to be a required trigger. The VFS changes may have delayed the recurrence, but they have not resolved the underlying FUSE/media-path stall.

The next useful test is still a clean boot with the rclone mount, its nzbdav/Decypharr backend, Plex, Silo, and any other consumers of that mount disabled. Since this occurrence took around 14 hours, recommend leaving it in that state for at least 24 hours. If it remains stable, restore the backend, mount, and consumers one layer at a time.