June 12, 20251 yr I noticed this being displayed on my terminal screen when I walked by it this morning. I always seem to hit some type of issue at 30 days of uptime - I'll go to log into the server and cant hit the IP address - the terminal screen is completely dark. Only way to bring back online is hard reboot and ive been meaning to schedule and automated restart. I am currently at 27 days, 7 hours of uptime now, so I'm expecting a repeat soon. Maybe I'm catching the issue this time? But also want to make sure it is not something malicious. Server config has been moved around through quite a bit of hardware over the last 5 years, so maybe it could be something with my config. Diagnostics attached. Thanks in advance for any info on thismclovinii-diagnostics-20250612-1456.zip
June 12, 20251 yr Community Expert 500 sessions have successfully logged in yes...This could be web uI login but this apears to be ssh session and ssh keys... Mostlikly a bot / hack...Delete your ssh keys from the USB and disable ssh on unraid.use the fail2ban plugin or install the sftp-fail2ban docker to caputre and block bad actors...in /boot/config/sshdelete the ssh folder and reboot.This will reset all ssh keys...then in the web UI go to setting > managment accessdisable turn off ssh
June 12, 20251 yr Author 3 minutes ago, bmartino1 said:500 sessions have successfully logged in yes...This could be web uI login but this apears to be ssh session and ssh keys... Mostlikly a bot / hack...Delete your ssh keys from the USB and disable ssh on unraid.use the fail2ban plugin or install the sftp-fail2ban docker to caputre and block bad actors...in /boot/config/sshdelete the ssh folder and reboot.This will reset all ssh keys...then in the web UI go to setting > managment accessdisable turn off sshThank you! I do share SSH keys between two Unraid servers to replicate ZFS snapshots bi-directionally. Is there a certain log i can check to ensure its not from that?Thanks again for the quick reply and steps to follow
June 14, 20251 yr Community Expert mby the zfs replication plugin... Theeis apears to be a script...Unless in the scripts that you may or may not be using to replicate zfs snapshots. You could add a log for that. Syslog / dmesg command is the reason why we see it in syslog...Yes the connection could be form you zfs snapshot depending on how your sending them and how your script is setup to send them.still 500 different connection is a bit much...General Overview:Your syslog confirms a huge number of open SSH sessions, which is highly unusual unless explicitly intended. Based on your description and the Unraid post:ZFS snapshot replication is likely being done over SSH using custom scripts.It's bi-directional, meaning both systems may be initiating and accepting connections.Sessions may not be closing cleanly or could be leaking, causing buildup over time (hence the 27-day uptime correlation).Here some things you can do to check...Confirm SSH SourceTo isolate if it's really your ZFS replication doing this, run:who | wc -llsof -i :22 | grep ESTABLISHED | wc -lAnd this, to view and analyze the actice connections:lsof -i :22 | grep ESTABLISHED*Look for many repeated IPs or usernames (likely your replication account).Check the replication methodIt’s highly likely a cron or user script is using ssh or zfs send/receive in a way that does not exit properly, especially if you’re using something like:in your script if its the zfs replicationssh user@remote zfs receive ...--Make sure all such processes use:ControlMaster=autoControlPersist=60 (for reuse, not excess)Proper trap or exit handling in scriptsExample user scirpt:ssh session cleanup dead/closed connections...#!/bin/bash# Kill stale SSH sessions and logLOG="/boot/logs/ssh_prune.log"DATE=$(date "+%Y-%m-%d %H:%M:%S")# Find and kill idle SSH sessions older than 1 hourecho "[$DATE] Checking SSH sessions..." >> "$LOG"COUNT=$(ps -ef | grep '[s]shd: ' | wc -l)echo "[$DATE] Active SSHD sessions: $COUNT" >> "$LOG"# Optional: Kill excess (over 100)if [ "$COUNT" -gt 100 ]; then echo "[$DATE] Too many sessions, pruning..." >> "$LOG" ps -ef | grep '[s]shd: ' | awk '{print $2}' | xargs -n1 kill -9 echo "[$DATE] Sessions killed." >> "$LOG"else echo "[$DATE] Count acceptable." >> "$LOG"fisome side script tips for ssh zfs replication:If you’re using zfs-auto-snapshot + zfs-auto-send, use mbuffer or ssh multiplexing to manage connections better.Add -o ConnectionTimeout=60 -o ServerAliveInterval=30 -o ServerAliveCountMax=2 to your SSH options.Track with something like:journalctl -u ssh | grep "Accepted"This does not appear malicious, but is almost certainly automation gone leaky. Fixing the SSH process lifecycle and possibly batching or limiting sessions will resolve this.Example zfs replication scirpt:Uses SSH with connection controlSupports multiple datasetsHandles bidirectional ZFS snapshot replicationCleans up old SSH connectionsIncludes easy configuration at the topzfs-replicator.sh#!/bin/bash# ZFS Replication & SSH Management Script for Unraid# Supports bidirectional snapshot replication between two Unraid systems########################### ==== CONFIGURATION ====########################### List of datasets to replicate FROM this machine to the remoteDATASETS_TO_SEND=("tank/media""tank/backups")# List of datasets to replicate FROM remote to this machineDATASETS_TO_RECEIVE=("tank/documents")# IP or hostname of the remote Unraid serverREMOTE_HOST="192.168.1.20"# User for SSH (must have key-based auth)SSH_USER="root"# SSH Port if not defaultSSH_PORT=22# Base dataset path for receiving on local (optional, leave blank to use same names)LOCAL_RECEIVE_BASE=""# Set to true to show commands as they runDEBUG=false# SSH Control path for persistent sessionCONTROL_SOCKET="/tmp/ssh-zfs-ctl.sock"########################### ==== SSH FUNCTIONS ====##########################start_ssh_connection() {if $DEBUG; then echo " Starting persistent SSH connection..."; fissh -M -S "$CONTROL_SOCKET" -fnNT -o ControlPersist=300 -p $SSH_PORT "$SSH_USER@$REMOTE_HOST"}close_ssh_connection() {if $DEBUG; then echo "🧹 Closing SSH connection..."; fissh -S "$CONTROL_SOCKET" -O exit "$SSH_USER@$REMOTE_HOST" 2>/dev/null}run_ssh_command() {ssh -S "$CONTROL_SOCKET" -p $SSH_PORT "$SSH_USER@$REMOTE_HOST" "$1"}############################### ==== ZFS REPLICATION ====##############################replicate_to_remote() {for DATASET in "${DATASETS_TO_SEND[@]}"; doSNAP_NAME="replication-$(date +%Y%m%d-%H%M%S)"echo " Sending $DATASET to $REMOTE_HOST as snapshot $SNAP_NAME"zfs snapshot "${DATASET}@${SNAP_NAME}"zfs send -c "${DATASET}@${SNAP_NAME}" | ssh -S "$CONTROL_SOCKET" -p $SSH_PORT "$SSH_USER@$REMOTE_HOST" "zfs receive -F $DATASET"done}replicate_from_remote() {for DATASET in "${DATASETS_TO_RECEIVE[@]}"; doSNAP_NAME="replication-$(date +%Y%m%d-%H%M%S)"echo " Receiving $DATASET from $REMOTE_HOST as snapshot $SNAP_NAME"run_ssh_command "zfs snapshot ${DATASET}@${SNAP_NAME}"run_ssh_command "zfs send -c ${DATASET}@${SNAP_NAME}" | zfs receive -F "${LOCAL_RECEIVE_BASE}${DATASET}"done}############################# ==== MAIN EXECUTION ====############################echo " Starting ZFS Replication Script"start_ssh_connectionreplicate_to_remotereplicate_from_remoteclose_ssh_connectionecho " Replication completed!"NotesSSH control socket allows multiple commands over one connection.ZFS snapshots are timestamped (replication-YYYYMMDD-HHMMSS).You must have SSH key-based login between systems.This script assumes both servers have identical dataset names unless LOCAL_RECEIVE_BASE is set.so now lets add loging to the example script above and maybe make use of mbuffer..-mbuffer is a deb only pacakge...As this is something I use in my ring network...unraid user scirpt plugin...zfs-bidirectional-replicator.sh#!/bin/bash# ZFS Snapshot + Bidirectional Replication Script with SSH Key Auth and Logging##################################### ==== USER CONFIGURATION =========##################################### Local datasets to replicate to remoteDATASETS_TO_SEND=( "tank/media" "tank/backups")# Remote datasets to replicate to localDATASETS_TO_RECEIVE=( "tank/documents")# Remote host configREMOTE_HOST="192.168.1.20"SSH_USER="root"SSH_PORT=22# Optional: receive into base prefix (e.g., "remote/"), or leave blankLOCAL_RECEIVE_BASE=""# Number of snapshots to keep per datasetSNAPSHOT_RETENTION=12# Enable debug outputDEBUG=true# Log fileLOGFILE="/var/log/zfs-replication-$(date +%F).log"# SSH control socket for connection reuseCONTROL_SOCKET="/tmp/ssh-zfs-ctl.sock"##################################### ==== LOGGING UTILITY ============####################################log() { echo "[$(date '+%F %T')] $*" | tee -a "$LOGFILE"}run_debug() { if $DEBUG; then log "$*"; fi eval "$@"}##################################### ==== SSH CONNECTION MGMT ========####################################test_ssh_connection() { log "Testing SSH key connection to $REMOTE_HOST..." ssh -o BatchMode=yes -o ConnectTimeout=5 "$SSH_USER@$REMOTE_HOST" "echo SSH connection OK" >/dev/null 2>&1 if [ $? -ne 0 ]; then log "ERROR: SSH key authentication failed. Make sure SSH keys are set up and authorized_keys is configured." exit 1 fi log "SSH connection test successful."}start_ssh_connection() { log "Starting SSH control connection..." ssh -M -S "$CONTROL_SOCKET" -fnNT -o ControlPersist=300 -p "$SSH_PORT" "$SSH_USER@$REMOTE_HOST"}close_ssh_connection() { log "Closing SSH connection..." ssh -S "$CONTROL_SOCKET" -O exit "$SSH_USER@$REMOTE_HOST" 2>/dev/null || true}run_ssh_command() { ssh -S "$CONTROL_SOCKET" -p "$SSH_PORT" "$SSH_USER@$REMOTE_HOST" "$1"}##################################### ==== SNAPSHOT HANDLING ==========####################################create_snapshot_if_changed() { local dataset="$1" local written written=$(zfs get -H -o value written "$dataset") if [[ "$written" != "0" ]]; then local snap_name="replication-$(date '+%Y%m%d-%H%M%S')" zfs snapshot "$dataset@$snap_name" log "Snapshot created: $dataset@$snap_name" echo "$snap_name" else log "No changes in $dataset, skipping snapshot." echo "" fi}prune_snapshots() { local dataset="$1" local snapshots=($(zfs list -t snapshot -o name -s creation -r "$dataset" | grep "^$dataset@")) local total=${#snapshots[@]} if (( total > SNAPSHOT_RETENTION )); then local to_delete=$(( total - SNAPSHOT_RETENTION )) for (( i=0; i<to_delete; i++ )); do log "Deleting old snapshot: ${snapshots[$i]}" zfs destroy "${snapshots[$i]}" done fi}##################################### ==== ZFS SEND TO REMOTE =========####################################replicate_to_remote() { for dataset in "${DATASETS_TO_SEND[@]}"; do log "Sending dataset: $dataset" local snap_name snap_name=$(create_snapshot_if_changed "$dataset") if [[ -n "$snap_name" ]]; then prune_snapshots "$dataset" log "Sending $dataset@$snap_name to $REMOTE_HOST" zfs send -c "$dataset@$snap_name" | ssh -S "$CONTROL_SOCKET" -p "$SSH_PORT" "$SSH_USER@$REMOTE_HOST" "zfs receive -F $dataset" fi done}##################################### ==== ZFS RECEIVE FROM REMOTE ====####################################replicate_from_remote() { for dataset in "${DATASETS_TO_RECEIVE[@]}"; do log "Receiving dataset: $dataset from $REMOTE_HOST" snap_name="replication-$(date '+%Y%m%d-%H%M%S')" run_ssh_command "zfs snapshot $dataset@$snap_name && zfs list -t snapshot $dataset@$snap_name" || { log "Failed to snapshot remote dataset: $dataset" continue } run_ssh_command "prune() { list=(\$(zfs list -t snapshot -o name -s creation -r $dataset | grep \"^$dataset@\")); \ total=\${#list[@]}; \ if (( total > $SNAPSHOT_RETENTION )); then \ del=\$((total - $SNAPSHOT_RETENTION)); \ for ((i=0; i<del; i++)); do zfs destroy \"\${list[\$i]}\"; done; \ fi }; prune" run_ssh_command "zfs send -c $dataset@$snap_name" | zfs receive -F "${LOCAL_RECEIVE_BASE}${dataset}" done}##################################### ==== MAIN EXECUTION =============####################################log "===== ZFS Bidirectional Replication Job ====="test_ssh_connectionstart_ssh_connectionreplicate_to_remotereplicate_from_remoteclose_ssh_connectionlog "Replication complete"This script enables bidirectional ZFS snapshot replication between two Unraid servers over SSH, using machine SSH keys (key-based auth only, no passwords).Creates a new snapshot only if there’s changed data.Prunes older snapshots, keeping only a set number.Sends and receives snapshots using zfs send / zfs receive.Uses SSH ControlMaster sockets to reuse a single SSH connection safely.Logs all actions to /var/log/zfs-replication-YYYY-MM-DD.log.
July 11, 2025Jul 11 Author On 6/14/2025 at 2:20 PM, bmartino1 said:mby the zfs replication plugin... Theeis apears to be a script...Unless in the scripts that you may or may not be using to replicate zfs snapshots. You could add a log for that. Syslog / dmesg command is the reason why we see it in syslog...Yes the connection could be form you zfs snapshot depending on how your sending them and how your script is setup to send them.still 500 different connection is a bit much...General Overview:Your syslog confirms a huge number of open SSH sessions, which is highly unusual unless explicitly intended. Based on your description and the Unraid post:ZFS snapshot replication is likely being done over SSH using custom scripts.It's bi-directional, meaning both systems may be initiating and accepting connections.Sessions may not be closing cleanly or could be leaking, causing buildup over time (hence the 27-day uptime correlation).Here some things you can do to check...Confirm SSH SourceTo isolate if it's really your ZFS replication doing this, run:who | wc -llsof -i :22 | grep ESTABLISHED | wc -lAnd this, to view and analyze the actice connections:lsof -i :22 | grep ESTABLISHED*Look for many repeated IPs or usernames (likely your replication account).Check the replication methodIt’s highly likely a cron or user script is using ssh or zfs send/receive in a way that does not exit properly, especially if you’re using something like:in your script if its the zfs replicationssh user@remote zfs receive ...--Make sure all such processes use:ControlMaster=autoControlPersist=60 (for reuse, not excess)Proper trap or exit handling in scriptsExample user scirpt:ssh session cleanup dead/closed connections...#!/bin/bash# Kill stale SSH sessions and logLOG="/boot/logs/ssh_prune.log"DATE=$(date "+%Y-%m-%d %H:%M:%S")# Find and kill idle SSH sessions older than 1 hourecho "[$DATE] Checking SSH sessions..." >> "$LOG"COUNT=$(ps -ef | grep '[s]shd: ' | wc -l)echo "[$DATE] Active SSHD sessions: $COUNT" >> "$LOG"# Optional: Kill excess (over 100)if [ "$COUNT" -gt 100 ]; thenecho "[$DATE] Too many sessions, pruning..." >> "$LOG"ps -ef | grep '[s]shd: ' | awk '{print $2}' | xargs -n1 kill -9echo "[$DATE] Sessions killed." >> "$LOG"elseecho "[$DATE] Count acceptable." >> "$LOG"fisome side script tips for ssh zfs replication:If you’re using zfs-auto-snapshot + zfs-auto-send, use mbuffer or ssh multiplexing to manage connections better.Add -o ConnectionTimeout=60 -o ServerAliveInterval=30 -o ServerAliveCountMax=2 to your SSH options.Track with something like:journalctl -u ssh | grep "Accepted"This does not appear malicious, but is almost certainly automation gone leaky. Fixing the SSH process lifecycle and possibly batching or limiting sessions will resolve this.Example zfs replication scirpt:Uses SSH with connection controlSupports multiple datasetsHandles bidirectional ZFS snapshot replicationCleans up old SSH connectionsIncludes easy configuration at the topzfs-replicator.sh#!/bin/bash# ZFS Replication & SSH Management Script for Unraid# Supports bidirectional snapshot replication between two Unraid systems########################### ==== CONFIGURATION ====########################### List of datasets to replicate FROM this machine to the remoteDATASETS_TO_SEND=("tank/media""tank/backups")# List of datasets to replicate FROM remote to this machineDATASETS_TO_RECEIVE=("tank/documents")# IP or hostname of the remote Unraid serverREMOTE_HOST="192.168.1.20"# User for SSH (must have key-based auth)SSH_USER="root"# SSH Port if not defaultSSH_PORT=22# Base dataset path for receiving on local (optional, leave blank to use same names)LOCAL_RECEIVE_BASE=""# Set to true to show commands as they runDEBUG=false# SSH Control path for persistent sessionCONTROL_SOCKET="/tmp/ssh-zfs-ctl.sock"########################### ==== SSH FUNCTIONS ====##########################start_ssh_connection() {if $DEBUG; then echo " Starting persistent SSH connection..."; fissh -M -S "$CONTROL_SOCKET" -fnNT -o ControlPersist=300 -p $SSH_PORT "$SSH_USER@$REMOTE_HOST"}close_ssh_connection() {if $DEBUG; then echo "🧹 Closing SSH connection..."; fissh -S "$CONTROL_SOCKET" -O exit "$SSH_USER@$REMOTE_HOST" 2>/dev/null}run_ssh_command() {ssh -S "$CONTROL_SOCKET" -p $SSH_PORT "$SSH_USER@$REMOTE_HOST" "$1"}############################### ==== ZFS REPLICATION ====##############################replicate_to_remote() {for DATASET in "${DATASETS_TO_SEND[@]}"; doSNAP_NAME="replication-$(date +%Y%m%d-%H%M%S)"echo " Sending $DATASET to $REMOTE_HOST as snapshot $SNAP_NAME"zfs snapshot "${DATASET}@${SNAP_NAME}"zfs send -c "${DATASET}@${SNAP_NAME}" | ssh -S "$CONTROL_SOCKET" -p $SSH_PORT "$SSH_USER@$REMOTE_HOST" "zfs receive -F $DATASET"done}replicate_from_remote() {for DATASET in "${DATASETS_TO_RECEIVE[@]}"; doSNAP_NAME="replication-$(date +%Y%m%d-%H%M%S)"echo " Receiving $DATASET from $REMOTE_HOST as snapshot $SNAP_NAME"run_ssh_command "zfs snapshot ${DATASET}@${SNAP_NAME}"run_ssh_command "zfs send -c ${DATASET}@${SNAP_NAME}" | zfs receive -F "${LOCAL_RECEIVE_BASE}${DATASET}"done}############################# ==== MAIN EXECUTION ====############################echo " Starting ZFS Replication Script"start_ssh_connectionreplicate_to_remotereplicate_from_remoteclose_ssh_connectionecho " Replication completed!"NotesSSH control socket allows multiple commands over one connection.ZFS snapshots are timestamped (replication-YYYYMMDD-HHMMSS).You must have SSH key-based login between systems.This script assumes both servers have identical dataset names unless LOCAL_RECEIVE_BASE is set.so now lets add loging to the example script above and maybe make use of mbuffer..-mbuffer is a deb only pacakge...As this is something I use in my ring network...unraid user scirpt plugin...zfs-bidirectional-replicator.sh#!/bin/bash# ZFS Snapshot + Bidirectional Replication Script with SSH Key Auth and Logging##################################### ==== USER CONFIGURATION =========##################################### Local datasets to replicate to remoteDATASETS_TO_SEND=("tank/media""tank/backups")# Remote datasets to replicate to localDATASETS_TO_RECEIVE=("tank/documents")# Remote host configREMOTE_HOST="192.168.1.20"SSH_USER="root"SSH_PORT=22# Optional: receive into base prefix (e.g., "remote/"), or leave blankLOCAL_RECEIVE_BASE=""# Number of snapshots to keep per datasetSNAPSHOT_RETENTION=12# Enable debug outputDEBUG=true# Log fileLOGFILE="/var/log/zfs-replication-$(date +%F).log"# SSH control socket for connection reuseCONTROL_SOCKET="/tmp/ssh-zfs-ctl.sock"##################################### ==== LOGGING UTILITY ============####################################log() {echo "[$(date '+%F %T')] $*" | tee -a "$LOGFILE"}run_debug() {if $DEBUG; then log "$*"; fieval "$@"}##################################### ==== SSH CONNECTION MGMT ========####################################test_ssh_connection() {log "Testing SSH key connection to $REMOTE_HOST..."ssh -o BatchMode=yes -o ConnectTimeout=5 "$SSH_USER@$REMOTE_HOST" "echo SSH connection OK" >/dev/null 2>&1if [ $? -ne 0 ]; thenlog "ERROR: SSH key authentication failed. Make sure SSH keys are set up and authorized_keys is configured."exit 1filog "SSH connection test successful."}start_ssh_connection() {log "Starting SSH control connection..."ssh -M -S "$CONTROL_SOCKET" -fnNT -o ControlPersist=300 -p "$SSH_PORT" "$SSH_USER@$REMOTE_HOST"}close_ssh_connection() {log "Closing SSH connection..."ssh -S "$CONTROL_SOCKET" -O exit "$SSH_USER@$REMOTE_HOST" 2>/dev/null || true}run_ssh_command() {ssh -S "$CONTROL_SOCKET" -p "$SSH_PORT" "$SSH_USER@$REMOTE_HOST" "$1"}##################################### ==== SNAPSHOT HANDLING ==========####################################create_snapshot_if_changed() {local dataset="$1"local writtenwritten=$(zfs get -H -o value written "$dataset")if [[ "$written" != "0" ]]; thenlocal snap_name="replication-$(date '+%Y%m%d-%H%M%S')"zfs snapshot "$dataset@$snap_name"log "Snapshot created: $dataset@$snap_name"echo "$snap_name"elselog "No changes in $dataset, skipping snapshot."echo ""fi}prune_snapshots() {local dataset="$1"local snapshots=($(zfs list -t snapshot -o name -s creation -r "$dataset" | grep "^$dataset@"))local total=${#snapshots[@]}if (( total > SNAPSHOT_RETENTION )); thenlocal to_delete=$(( total - SNAPSHOT_RETENTION ))for (( i=0; i<to_delete; i++ )); dolog "Deleting old snapshot: ${snapshots[$i]}"zfs destroy "${snapshots[$i]}"donefi}##################################### ==== ZFS SEND TO REMOTE =========####################################replicate_to_remote() {for dataset in "${DATASETS_TO_SEND[@]}"; dolog "Sending dataset: $dataset"local snap_namesnap_name=$(create_snapshot_if_changed "$dataset")if [[ -n "$snap_name" ]]; thenprune_snapshots "$dataset"log "Sending $dataset@$snap_name to $REMOTE_HOST"zfs send -c "$dataset@$snap_name" | ssh -S "$CONTROL_SOCKET" -p "$SSH_PORT" "$SSH_USER@$REMOTE_HOST" "zfs receive -F $dataset"fidone}##################################### ==== ZFS RECEIVE FROM REMOTE ====####################################replicate_from_remote() {for dataset in "${DATASETS_TO_RECEIVE[@]}"; dolog "Receiving dataset: $dataset from $REMOTE_HOST"snap_name="replication-$(date '+%Y%m%d-%H%M%S')"run_ssh_command "zfs snapshot $dataset@$snap_name && zfs list -t snapshot $dataset@$snap_name" || {log "Failed to snapshot remote dataset: $dataset"continue}run_ssh_command "prune() {list=(\$(zfs list -t snapshot -o name -s creation -r $dataset | grep \"^$dataset@\")); \total=\${#list[@]}; \if (( total > $SNAPSHOT_RETENTION )); then \del=\$((total - $SNAPSHOT_RETENTION)); \for ((i=0; i<del; i++)); do zfs destroy \"\${list[\$i]}\"; done; \fi}; prune"run_ssh_command "zfs send -c $dataset@$snap_name" | zfs receive -F "${LOCAL_RECEIVE_BASE}${dataset}"done}##################################### ==== MAIN EXECUTION =============####################################log "===== ZFS Bidirectional Replication Job ====="test_ssh_connectionstart_ssh_connectionreplicate_to_remotereplicate_from_remoteclose_ssh_connectionlog "Replication complete"This script enables bidirectional ZFS snapshot replication between two Unraid servers over SSH, using machine SSH keys (key-based auth only, no passwords).Creates a new snapshot only if there’s changed data.Prunes older snapshots, keeping only a set number.Sends and receives snapshots using zfs send / zfs receive.Uses SSH ControlMaster sockets to reuse a single SSH connection safely.Logs all actions to /var/log/zfs-replication-YYYY-MM-DD.log.Really appreciate you taking the time to look into this. I do have nightly zfs replication using Space Invader's script and Squid's user scripts - probably 4 or 5 zfs shares on one side & maybe 10 or so on the other. So the 500 over the course of the month might be accurate in this case. I think the output being seen on the terminal through me off, but I think there are some memory issues as well. Sure enough, the server crashed after i posted this a few weeks ago. I turned on syslog and was able see a memory issue regarding unreadable sectors. I found that my plex contaniers metadata disk (standalone NVMe disk that was using zfs) had errors. I wiped the nvme ssd, formatted with xfs, and let plex rebuild. 11 days and no more of those issues found. Today, i was adding so more libraries and getting things back to normal and i had another crash. I dug back into logs and found errors about access to memory being restricted. I have 64GB installed, and usual usage is around mid-40%. sometimes spikes to 60ish%Below was the first error:Jul 9 00:38:28 McLovinII nginx: 2025/07/09 00:38:28 [crit] 18022#18022: ngx_slab_alloc() failed: no memoryJul 9 00:38:28 McLovinII nginx: 2025/07/09 00:38:28 [error] 18022#18022: shpool alloc failedJul 9 00:38:28 McLovinII nginx: 2025/07/09 00:38:28 [error] 18022#18022: nchan: Out of shared memory while allocating message of size 32484. Increase nchan_max_reserved_memory.Jul 9 00:38:28 McLovinII nginx: 2025/07/09 00:38:28 [error] 18022#18022: *3059388 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"Jul 9 00:38:28 McLovinII nginx: 2025/07/09 00:38:28 [error] 18022#18022: MEMSTORE:00: can't create shared message for channel /devicesI have a HD homerun connected to plex, so that is transcoding to memory quite a bit. But Google AI makes this error sound like it is related to unraid UI running out of memory. Which took me down the rabbit hole of learning that leaving your browser open in the Unraid UI can cause it to crash. I noticed that i got these same errors for the last few days and i always leave the browser open. But today between 1300 and 1400 i seemed capture more memory errors but i am unsure if they are related. I attached the syslog of the last few dayssyslog.txt Edited July 11, 2025Jul 11 by mclovin spelling
July 12, 2025Jul 12 Community Expert this tells me that your sytem ran out of resources, unraid web ui atempted to do something while under heavy load and ran out of memory.nchan is a comon one I have found with leaving a unraid dashbaord open for too long or doing thing when the mermoy of the host is maxed.(do you have swap/swap plugin). swap should have helped prevent or be used when this happens to recover.?what verion of unrad are you on. Lattest stable 7.1.4 has patched and fixed nginx nchan memory errors...https://forums.unraid.net/topic/178541-server-unresponsive-nchan-error-message/I wrote a script and helpd someone having nchan errors in the past... Haveing a hard time finding that forum post and code to try.I recall some mods helping others with simalr issues with a unresponsive unriad host. soemthings happens while underload where there a memory leak that casues this...@Squid you wanted someone to test something if this came up again?If you guys see this again, can you post the output ofls -ail /usr/local/emhttp/plugins/dynamix/scriptsSo, My first recommendation is make sure you on the latest Stable unraid version.https://docs.unraid.net/unraid-os/download_list/the error more shows the exhibition of errors with unraid emhttp and https nginx web server...this may be casued by a underlaods sytem busy when trying to do somethign int he web ui and unraid becoems unresponve as its already doing a task...
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.