j5i7

Truth be told. I haven't done anything new to fix the problem, but its become rare enough that I haven't tried anything new (looks like the last time the ethernet went down was six months ago). Since I failed to previously, I've attached the code for my userscript to this post. I modified another script (author Marc Gutt, but I don't remember what the script was originally). The script checks every hour if the Ethernet is down, and if it is down, it restarts the server. It also saves the syslog anytime the ethernet goes down. #!/bin/bash # ##################################### # Name: NIC bandaid # Description: Script to restart the server if the ethernet adapter goes down. Also saves the syslog file for reference. # Author: *** - modified from script by Marc Gutt # ##################################### # ##################################### # Settings # ##################################### # get most recent syslog file syslog_file=$(ls -t /var/log/syslog{.[0-9],} 2>/dev/null | head -n 1) # The error message to look for in the syslog words="eth0: PCIe link lost, device now detached" # store line number of last found error in this file log_file="/tmp/syslog-notify-last-error-line-number.log" # store the time & date of a shutdown on a shared folder which can be viewed later for reference. log_file_shutdown="/mnt/user/appdata/_nic_bandaid/nic_restarts.log" log_file_syslog="/mnt/user/appdata/_nic_bandaid/syslog_" # log what is being done verbose=false # ##################################### # Script # ##################################### # make script race condition safe if [[ -d "/tmp/${0//\//_}" ]] || ! mkdir "/tmp/${0//\//_}"; then echo "Script is already running!" && exit 1; fi; trap 'rmdir "/tmp/${0//\//_}"' EXIT; # check user settings [[ $verbose == 0 ]] || [[ $verbose == false ]] && unset verbose # obtain line number of last check if [[ -f "$log_file" ]]; then line_number_start=$(cat "$log_file") # syslog has been truncated if [[ $line_number_start -gt $(grep -c ^ "$syslog_file") ]]; then [[ $verbose ]] && echo "Monitoring of a new syslog file begins" line_number_start=0 fi # store last line number on first execution else line_number_start=$(grep -c ^ "$syslog_file") line_number_start=$((line_number_start-100)) [[ $verbose ]] && echo "Monitoring syslog starts from line $line_number_start" echo "$line_number_start" > "$log_file" fi # parse logs EOL=$'\n' errors="" while read -r line; do # remember last line last_line="$line" # combine multiple error messages errors="$errors$EOL$line" done < <(tail -n +"$((line_number_start+1))" "$syslog_file" | grep -iP "($words)") # create notification for new errors if [[ $errors ]]; then # remember line number of last error line_number_start=$(grep -nFx "$last_line" "$syslog_file" | cut -f 1 -d ":") [[ $verbose ]] && echo "Monitoring syslog continues from line $line_number_start" echo "$line_number_start" > "$log_file" datetime=$(date) datetime_safe=$(date +%F_%H-%M-%S) # send notification /usr/local/emhttp/webGui/scripts/notify -i "alert" -s "NIC down" -d "eth0 appears lost as of $datetime! Prepping to restart the server!" # copy the syslog to the set destinateion & write to the log -fiel log_file_syslog_name="$log_file_syslog$datetime_safe" cp /var/log/syslog "$log_file_syslog_name" echo "System to be restarted as NIC was found down on: $datetime. See $log_file_syslog_name" echo "System to be restarted as NIC was found down on: $datetime. See $log_file_syslog_name" >> "$log_file_shutdown" powerdown -r exit else # store last line number if no error has been found line_number_start=$(grep -c ^ "$syslog_file") [[ $verbose ]] && echo "Monitoring syslog continues from last line $line_number_start" echo "$line_number_start" > "$log_file" datetime=$(date) echo "NIC operational on: $datetime." echo "NIC operational on: $datetime." >> "$log_file_shutdown" fi # create notificaton if log exceeds usage of 90% log_size=$(df | grep -oP "[0-9]+(?=% /var/log)") if [[ ! -f /tmp/syslog-notify.size ]] && [[ $log_size -gt 90 ]]; then touch /tmp/syslog-notify.size /usr/local/emhttp/webGui/scripts/notify -i "alert" -s "log utilizes more than 90%!" -d "$(du -h /var/log/* | sort -h | tail)" elif [[ -f /tmp/syslog-notify.size ]]; then rm /tmp/syslog-notify.size fi

I’ll share it when I can, unfortunately it’s an unlucky time as I’m in the middle of a move and my server is packed away and I don’t think I have a copy of the script elsewhere. However, in general I searched for an existing script that monitored the syslog, then modified it to look for the Ethernet disconnected message and when found, restart the server. Then I set the script to run every hour. I also added logging etc, I think it’s run about twice in the last six months or so. If I don’t post it in the upcoming week, remind me.

Not a big update but I went ~20 days without an issue, then had it happen again. The only correlation I can fathom is that I turned on download overnight and it happened that same night. However, I haven't had any issues when streaming countless hours of TV off my NAS. Anyway, rather than throwing alternative hardware at it for the moment, I just created a script that scans the syslog for the ethernet crash message and will reboot the server if detected. Hopefully some driver update down the line fixes my issue, but for my current use case, it's not the end of the world for the server to gracefully restart every few weeks.

It happened again right before I changed the ethernet cable. The log is a bit different and attached here again and partially quoted below. I updated my mobo to the latest BIOS revision which should be the only change between this and the prior one. Since this error message I've changed my ethernet cable. If I get another error, I'm tempted to turn off the ASPM in the bios and my next step (hoping that won't significantly affect power draw). syslog2

I do have the pcie slot on my motherboard free and could put a half-height NIC in if need be (or a USB to ethernet adapter, but no clue of the USB overhead here). I don't currently own one, but may get one if need be. Do you think the the log looks like this was an intel driver issue? Or is this a PCI-E link issue that then resulted in a driver throwing an error (or hard to say)? The only 2.5gb cable that is isolated to this connection is a single ~6ft cable from my 2.5gb switch to my unraid box. I'll try swapping it for another cable that's known to be at least cat 6 and see from there (the current cable came from my steam link I think).

Hello all, I've been really enjoying Unraid the last couple months I've had it, but I'm running into an annoying hiccup I was hoping some log / network wizard could help me with. Sporadically (every 1-2 weeks) my Unraid server will completely lose network connectivity. The server itself is still running just fine if I physically access it with a mouse and keyboard, but no network connectivity goes in or out (docker/ping/dashboard inaccessible form LAN or WAN and I can't ping out from the server to my router). If I reboot the server, everything works fine until the error occurs again. I can't pinpoint anything that makes it happen. I finally caught it again today and have attached my syslog from the error. I've also attached my diagnostics zip but this is from after rebooting. The syslog attached is prior to a reboot and I've put the critical excerpt at the bottom of this message. I'm not great at understanding this, but is this a intel driver error or a hardware error? Thanks for any and all guidance. Worst case... maybe the server could just gracefully reboot if this error is detected. syslog n1-diagnostics-20230116-1703.zip