February 17, 20242 yr Over the last month, I have had issues with Unraid crashing and failing to reboot. When the system crashes, I am able to ping it but cannot reach the GUI or complete a connection via SSH. I can also not connect to the system directly as no video outputs to a monitor, although that may be because the default VGA was the video card, I have since switched to onboard so I can determine if that is still a problem in the future. Additionally, I was having an issue with the BIOS not recognizing the USB when I would go to reboot, until it was moved to another port. I have since purchased another usb and exchanged the mother board, as there was some question as to whether the USB ports on the board were failing, though Im having the same/similar issues. This most recent time, I was unable to determine if the USB disappeared completely on reboot, as I did not go direct to bios, however it did switch to network boot instead of the usb. When I went into the BIOS, the USB was there, though it was at the bottom of the list. I swear I had disabled other boots prior leaving only the usb, leading me to believe the USB disappeared entirely again, though I cannot be sure this time. This issue has persisted across 2 USBs and 2 motherboards, I am kind of running out of solutions. Possibly related, I have been running into an issue where certain crashes and reboots cause all of my docker containers to disappear.
February 18, 20242 yr Community Expert 13 hours ago, jpfeifer14 said: I swear I had disabled other boots prior leaving only the usb, leading me to believe the USB disappeared entirely again, though I cannot be sure this time. This issue has persisted across 2 USBs and 2 motherboards, I am kind of running out of solutions. This suggests a board issue, also make sure the CMOS battery is good. For the crashing enable the syslog server and post that after a crash.
February 18, 20242 yr Author 7 hours ago, JorgeB said: This suggests a board issue, also make sure the CMOS battery is good. For the crashing enable the syslog server and post that after a crash. Thanks, Ive just started mirroring the logs to flash, Ill see if it happens again. Im going to be real PO'd if this new motherboard is bad as well
February 22, 20242 yr Author The system quit responding again, so I had a look at the logs dont see anything in the last 12 hours or so that looked out of the ordinary. Really not sure what to make of this. Feb 21 00:00:01 cloud usbipd: usbipd: info: connection from 192.168.0.11:40930 Feb 21 00:09:07 cloud emhttpd: read SMART /dev/sdi Feb 21 01:00:01 cloud usbipd: usbipd: info: connection from 192.168.0.11:59164 Feb 21 01:14:01 cloud emhttpd: read SMART /dev/sdd Feb 21 01:56:40 cloud emhttpd: spinning down /dev/sdi Feb 21 02:00:01 cloud usbipd: usbipd: info: connection from 192.168.0.11:38164 Feb 21 02:01:04 cloud emhttpd: read SMART /dev/sdb Feb 21 02:14:14 cloud emhttpd: spinning down /dev/sdd Feb 21 02:49:35 cloud emhttpd: read SMART /dev/sdi Feb 21 03:00:01 cloud usbipd: usbipd: info: connection from 192.168.0.11:40484 Feb 21 03:01:19 cloud emhttpd: spinning down /dev/sdb Feb 21 03:33:18 cloud emhttpd: read SMART /dev/sdb Feb 21 04:00:01 cloud usbipd: usbipd: info: connection from 192.168.0.11:41562 Feb 21 04:00:17 cloud emhttpd: read SMART /dev/sdg Feb 21 04:00:17 cloud emhttpd: read SMART /dev/sde Feb 21 04:08:41 cloud emhttpd: spinning down /dev/sdi Feb 21 05:00:01 cloud usbipd: usbipd: info: connection from 192.168.0.11:54870 Feb 21 05:01:50 cloud emhttpd: spinning down /dev/sdg Feb 21 05:01:50 cloud emhttpd: spinning down /dev/sde Feb 21 05:31:36 cloud emhttpd: spinning down /dev/sdb Feb 21 06:00:01 cloud usbipd: usbipd: info: connection from 192.168.0.11:38926 Feb 21 06:57:52 cloud emhttpd: read SMART /dev/sdi Feb 21 07:00:01 cloud usbipd: usbipd: info: connection from 192.168.0.11:58790 Feb 21 07:57:57 cloud emhttpd: spinning down /dev/sdi Feb 21 08:00:01 cloud usbipd: usbipd: info: connection from 192.168.0.11:40614 Feb 21 08:28:11 cloud emhttpd: read SMART /dev/sdd Feb 21 09:00:01 cloud usbipd: usbipd: info: connection from 192.168.0.11:46250 Feb 21 09:28:16 cloud emhttpd: spinning down /dev/sdd Feb 21 10:00:01 cloud usbipd: usbipd: info: connection from 192.168.0.11:60950 Feb 21 10:18:50 cloud emhttpd: read SMART /dev/sdi Feb 21 11:00:01 cloud usbipd: usbipd: info: connection from 192.168.0.11:43262 Feb 21 11:27:13 cloud emhttpd: spinning down /dev/sdi Feb 21 12:00:01 cloud usbipd: usbipd: info: connection from 192.168.0.11:58098 Feb 21 12:00:01 cloud Docker Auto Update: Community Applications Docker Autoupdate running Feb 21 12:00:01 cloud Docker Auto Update: Checking for available updates Feb 21 12:00:15 cloud emhttpd: read SMART /dev/sdi Feb 21 12:00:29 cloud Docker Auto Update: Found update for flaresolverr. Not set to autoupdate Feb 21 12:00:29 cloud Docker Auto Update: Found update for nextclouddb. Not set to autoupdate Feb 21 12:00:29 cloud Docker Auto Update: Stopping binhex-jackett Feb 21 12:00:33 cloud kernel: br-8133cdf1cf98: port 5(veth9cb7ff6) entered disabled state Feb 21 12:00:34 cloud kernel: veth3b3355e: renamed from eth0 Feb 21 12:00:34 cloud kernel: br-8133cdf1cf98: port 5(veth9cb7ff6) entered disabled state Feb 21 12:00:34 cloud kernel: device veth9cb7ff6 left promiscuous mode Feb 21 12:00:34 cloud kernel: br-8133cdf1cf98: port 5(veth9cb7ff6) entered disabled state Feb 21 12:00:34 cloud Docker Auto Update: Installing Updates for binhex-jackett Feb 21 12:00:47 cloud Docker Auto Update: Restarting binhex-jackett Feb 21 12:00:47 cloud kernel: br-8133cdf1cf98: port 5(veth4f5f4c8) entered blocking state Feb 21 12:00:47 cloud kernel: br-8133cdf1cf98: port 5(veth4f5f4c8) entered disabled state Feb 21 12:00:47 cloud kernel: device veth4f5f4c8 entered promiscuous mode Feb 21 12:00:49 cloud kernel: eth0: renamed from veth412ec51 Feb 21 12:00:49 cloud kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth4f5f4c8: link becomes ready Feb 21 12:00:49 cloud kernel: br-8133cdf1cf98: port 5(veth4f5f4c8) entered blocking state Feb 21 12:00:49 cloud kernel: br-8133cdf1cf98: port 5(veth4f5f4c8) entered forwarding state Feb 21 12:00:50 cloud Docker Auto Update: Community Applications Docker Autoupdate finished Feb 21 12:57:38 cloud flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update Feb 21 13:00:01 cloud usbipd: usbipd: info: connection from 192.168.0.11:47186 Feb 21 13:00:22 cloud emhttpd: spinning down /dev/sdi Feb 21 13:06:12 cloud emhttpd: read SMART /dev/sdi Feb 21 14:00:01 cloud usbipd: usbipd: info: connection from 192.168.0.11:43620 Feb 21 14:06:16 cloud emhttpd: spinning down /dev/sdi Feb 21 14:15:56 cloud nginx: 2024/02/21 14:15:56 [alert] 6805#6805: worker process 21615 exited on signal 6 Feb 21 15:00:01 cloud usbipd: usbipd: info: connection from 192.168.0.11:47458 Feb 21 15:13:40 cloud nginx: 2024/02/21 15:13:40 [alert] 6805#6805: worker process 6761 exited on signal 6 Feb 21 15:21:26 cloud emhttpd: read SMART /dev/sdg Feb 21 15:23:30 cloud emhttpd: read SMART /dev/sde Feb 21 16:00:01 cloud usbipd: usbipd: info: connection from 192.168.0.11:42686 Feb 21 16:11:14 cloud emhttpd: read SMART /dev/sdb Feb 21 16:25:03 cloud emhttpd: spinning down /dev/sdg Feb 21 16:25:03 cloud emhttpd: spinning down /dev/sde Feb 21 17:00:01 cloud usbipd: usbipd: info: connection from 192.168.0.11:38194 Feb 21 17:15:35 cloud emhttpd: spinning down /dev/sdb Feb 21 18:00:01 cloud usbipd: usbipd: info: connection from 192.168.0.11:59242
February 22, 20242 yr Community Expert Unfortunately there's nothing relevant logged, this usually points to a hardware issue, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.
February 23, 20242 yr Author Any ideas what hardware issues I might look at? Memtest passed, and while i guess you can never rule out getting a brand new faulty motherboard, I find it strange that this one is running into the same issue that the other one did, only that one took a year to have the issue. I have a SAS card running a few drives, though Id assume that if something happened with that, it wouldnt kill the OS. The server has been set up on a LAGG connection since day one with no problems. As I said previously, even when the GUI cannot be reached, the system can be pinged. Other than that, I cannot think of anything hardware wise that changed recently. A surveillance drive was added as cache and a google coral was plugged in november, though these issues didnt arise until maybe January. I kept receiving a notification about MACVLAN being an issue so I changed it to ipvlan, but that didnt really seem to resolve anything. I believe the boot issue I was having is a quirk of the motherboard, it seems like if it is not cleanly shut down, occasionally it does not recognize the USB next boot, but will if power cycled. Edited February 23, 20242 yr by jpfeifer14
February 23, 20242 yr Community Expert 5 hours ago, jpfeifer14 said: Any ideas what hardware issues I might look at? If it's hardware it can be basically anything, but first I would recommend On 2/22/2024 at 9:05 AM, JorgeB said: one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.
February 23, 20242 yr Author Ok, Im setting it up in safe mode right now. It crashed again a bit earlier, so I pulled the latest logs, though I dont think this reveals any new information Feb 23 02:01:26 cloud emhttpd: read SMART /dev/sdb Feb 23 03:08:43 cloud emhttpd: spinning down /dev/sdb Feb 23 03:17:42 cloud emhttpd: read SMART /dev/sdb Feb 23 04:00:01 cloud emhttpd: read SMART /dev/sdi Feb 23 04:00:06 cloud emhttpd: read SMART /dev/sdg Feb 23 04:00:10 cloud emhttpd: read SMART /dev/sde Feb 23 04:17:57 cloud emhttpd: spinning down /dev/sdb Feb 23 05:01:37 cloud emhttpd: spinning down /dev/sdi Feb 23 05:02:45 cloud emhttpd: spinning down /dev/sdg Feb 23 05:02:45 cloud emhttpd: spinning down /dev/sde Feb 23 05:03:08 cloud sshd[17272]: Read error from remote host 192.168.0.111 port 62980: Connection timed out Feb 23 05:03:08 cloud sshd[17272]: pam_unix(sshd:session): session closed for user root Feb 23 06:12:50 cloud sshd[22939]: Connection from 192.168.0.111 port 52053 on 192.168.0.11 port 22 rdomain "" Feb 23 06:12:50 cloud sshd[22939]: Postponed keyboard-interactive for root from 192.168.0.111 port 52053 ssh2 [preauth] Feb 23 06:12:50 cloud sshd[22939]: Postponed keyboard-interactive/pam for root from 192.168.0.111 port 52053 ssh2 [preauth] Feb 23 06:12:50 cloud sshd[22939]: Accepted keyboard-interactive/pam for root from 192.168.0.111 port 52053 ssh2 Feb 23 06:12:50 cloud sshd[22939]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0) Feb 23 06:12:50 cloud sshd[22939]: Starting session: command for root from 192.168.0.111 port 52053 id 1 Feb 23 06:12:50 cloud sshd[22939]: Starting session: shell on pts/0 for root from 192.168.0.111 port 52053 id 0 Feb 23 06:12:50 cloud sshd[22939]: Close session: user root from 192.168.0.111 port 52053 id 1 Feb 23 06:12:50 cloud sshd[22939]: Starting session: command for root from 192.168.0.111 port 52053 id 1 Feb 23 06:12:50 cloud sshd[22939]: Starting session: command for root from 192.168.0.111 port 52053 id 2 Feb 23 06:12:50 cloud sshd[22939]: Close session: user root from 192.168.0.111 port 52053 id 1 Feb 23 06:12:50 cloud sshd[22939]: Close session: user root from 192.168.0.111 port 52053 id 2 Feb 23 08:28:28 cloud sshd[22939]: Read error from remote host 192.168.0.111 port 52053: Connection timed out Feb 23 08:28:28 cloud sshd[22939]: pam_unix(sshd:session): session closed for user root Feb 23 09:09:45 cloud emhttpd: read SMART /dev/sdg Feb 23 10:04:32 cloud emhttpd: read SMART /dev/sdd Feb 23 10:12:01 cloud emhttpd: spinning down /dev/sdg Feb 23 11:04:37 cloud emhttpd: spinning down /dev/sdd Feb 23 11:22:13 cloud sshd[11531]: Connection from 192.168.0.111 port 61662 on 192.168.0.11 port 22 rdomain "" Feb 23 11:22:13 cloud sshd[11531]: Postponed keyboard-interactive for root from 192.168.0.111 port 61662 ssh2 [preauth] Feb 23 11:22:13 cloud sshd[11531]: Postponed keyboard-interactive/pam for root from 192.168.0.111 port 61662 ssh2 [preauth] Feb 23 11:22:13 cloud sshd[11531]: Accepted keyboard-interactive/pam for root from 192.168.0.111 port 61662 ssh2 Feb 23 11:22:13 cloud sshd[11531]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0) Feb 23 11:22:13 cloud sshd[11531]: Starting session: command for root from 192.168.0.111 port 61662 id 1 Feb 23 11:22:13 cloud sshd[11531]: Starting session: shell on pts/0 for root from 192.168.0.111 port 61662 id 0 Feb 23 11:22:13 cloud sshd[11531]: Close session: user root from 192.168.0.111 port 61662 id 1 Feb 23 11:22:13 cloud sshd[11531]: Starting session: command for root from 192.168.0.111 port 61662 id 1 Feb 23 11:22:14 cloud sshd[11531]: Starting session: command for root from 192.168.0.111 port 61662 id 2 Feb 23 11:22:14 cloud sshd[11531]: Close session: user root from 192.168.0.111 port 61662 id 1 Feb 23 11:22:14 cloud sshd[11531]: Close session: user root from 192.168.0.111 port 61662 id 2
February 23, 20242 yr Author Didnt think so. I have the server running in safe mode with only the array going, no VMs or docker. Issue takes about 24-48 hours to present so we'll see
February 24, 20242 yr Author Still running with no crashes in safe mode 24 hours later so Im keeping an eye on it. I did stress test the CPU at one point with no crash, so theres that. I noticed that in safe mode the log repeatedly writes the following. I assume this is due to plugins being turned off but I wanted to make sure it isnt a bigger issue. Feb 24 14:21:43 cloud nginx: 2024/02/24 14:21:43 [error] 3667#3667: *876992 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 192.168.0.111, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.0.11", referrer: "http://192.168.0.11/Dashboard" Edited February 24, 20242 yr by jpfeifer14
February 25, 20242 yr I had these kind of issues eversince upgrading to 12.x. I reverted back to the last 11 version and was stable again. Then upgraded again after reading reports that crash issues were resolved, but still have them. Now thinking of going back again as there really seem to be something off with 12.x.
February 25, 20242 yr Author So far, it has not crashed. At 48 hours Im going to start adding services back in, I have an idea that frigate might be causing it, although others have said that they actually get an "Out of memory" error when they have the issue Ive experienced, and I have not gotten that.
February 25, 20242 yr Author Well, started adding back in services after no crashes in 48 hours. Ran the whole thing full force with a stress test on the cpu, all drives spinning, every container running and still only pulled 276w on the 750w psu, so Im kinda at a loss hardware wise. I guess it's not too late to RMA every component and pray it was one of those pieces.
March 2, 20242 yr Author This appears to be an issue with the Frigate docker container so rapidly blowing through 128 GB of RAM that the system halts and never records an error, I have submitted a request on their github
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.