wes.crockett Posted September 27, 2023 Share Posted September 27, 2023 As the title says, on Saturday, the server started acting up. I could access the unraid UI but not really od anything. Rebooting resulted in nothing. I could access the terminal but I couldn't get the reboot command to work there either. CPU was pegged at 100%. I had to hard boot it. Parity check passed fine after brining the array online. Tonight, I came home after a few hours away and I couldn't even access the unraid UI through web browser and I couldn't ping the box (so obviously I couldn't ssh in either.) iDrac showed no issues of note (but I could reach the idrac on the box) and I couldn't wake up the box to get direct video out. Naturally, I'm leaning to a dreaded hardware issue. I've attached Unraid logs... Hopefully someone more well versed than I can see a glaring issue in here? Syslogs attached kuiper-diagnostics-20230926-2310.zip Quote Link to comment
JorgeB Posted September 27, 2023 Share Posted September 27, 2023 Enable the syslog server and post that after a crash. 1 Quote Link to comment
wes.crockett Posted September 27, 2023 Author Share Posted September 27, 2023 I turned it on last night. I THINK I have it set up correctly. Thank you. Quote Link to comment
wes.crockett Posted September 27, 2023 Author Share Posted September 27, 2023 7 hours ago, JorgeB said: Enable the syslog server and post that after a crash. Hey @JorgeB, I should be seeing logs to the directory I specified, correct? I'm not seeing any generated at this point even though logging is enabled. Quote Link to comment
JorgeB Posted September 27, 2023 Share Posted September 27, 2023 36 minutes ago, wes.crockett said: correct? Correct, make sure you set the Unraid server IP in the remote server filed, it's a common mistake. 1 Quote Link to comment
itimpi Posted September 27, 2023 Share Posted September 27, 2023 1 hour ago, wes.crockett said: Hey @JorgeB, I should be seeing logs to the directory I specified, correct? I'm not seeing any generated at this point even though logging is enabled. It is often easier to set the “mirror to flash” option to get the output written to the ‘logs’ folder on the flash drive. 1 Quote Link to comment
wes.crockett Posted September 27, 2023 Author Share Posted September 27, 2023 5 hours ago, JorgeB said: Correct, make sure you set the Unraid server IP in the remote server filed, it's a common mistake. I'm all about the common mistakes... like this one... that I made. Thanks. 4 hours ago, itimpi said: It is often easier to set the “mirror to flash” option to get the output written to the ‘logs’ folder on the flash drive. So this will write them to my boot drive too then (being the flash drive)? Sounds good. Quote Link to comment
wes.crockett Posted September 28, 2023 Author Share Posted September 28, 2023 The log from the boot drive and the log on my cache drive aren't perfectly aligned, so I uploaded both... either way: I came into my office this morning to find the server totally unresponsive again. Had to perform a hard reboot. Logs attached. syslogs.zip Quote Link to comment
JorgeB Posted September 28, 2023 Share Posted September 28, 2023 Both logs only cover a few minutes, at what time was the crash? 1 Quote Link to comment
wes.crockett Posted September 28, 2023 Author Share Posted September 28, 2023 (edited) 15 minutes ago, JorgeB said: Both logs only cover a few minutes, at what time was the crash? Unknown. Sometime between 11pm and 630ish. I came in this morning and it was unusable. It became unusable overnight. I'm seeing several very similar threads. Going through them, looks like pretty identical symptoms. Hoping that, between all the threads, some light can come as to what may be causing it. Edited September 28, 2023 by wes.crockett Quote Link to comment
JorgeB Posted September 28, 2023 Share Posted September 28, 2023 35 minutes ago, JorgeB said: Both logs only cover a few minutes Forget that, I missed the first couple of lines, since I was expecting more activity logged before the crash: Sep 27 15:12:16 Kuiper monitor: Stop running nchan processes Sep 27 21:21:14 Kuiper webGUI: Successful login user root from 192.168.1.162 Sep 28 06:40:04 Kuiper kernel: Linux version 6.1.49-Unraid (root@Develop-612) (gcc (GCC) 12.2.0, Unfortunately there's nothing relevant logged, this usually points to a hardware issue, one thing you can try is to boot the server in safe mode with all docker/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one. 1 Quote Link to comment
wes.crockett Posted September 28, 2023 Author Share Posted September 28, 2023 I'm seeing several people mention downgrading and the issue going away... would this be worth pursuing to ensure it isn't the newest release resulting in the issue? Quote Link to comment
JorgeB Posted September 28, 2023 Share Posted September 28, 2023 You can try but when there's nothing logged usually it's not release related, unless there's some compatibility issue with the kernel itself. 1 Quote Link to comment
wes.crockett Posted September 28, 2023 Author Share Posted September 28, 2023 (edited) I just ran /sbin/reboot as a user script. Server rebooted fine but array didn't come back online on its own. Is that normal behavior when running /sbin/reboot? My thinking is trying nightly reboots to see if the issue persists every now and then. EDIT: I'm dumb... that just flat out reboots Linux. Is there a prebuilt script for safely rebooting Unraid in the proper order? Edited September 28, 2023 by wes.crockett Quote Link to comment
JorgeB Posted September 28, 2023 Share Posted September 28, 2023 17 minutes ago, wes.crockett said: Is that normal behavior when running /sbin/reboot? No. 17 minutes ago, wes.crockett said: EDIT: I'm dumb... that just flat out reboots Linux. Is there a prebuilt script for safely rebooting Unraid in the proper order? Reboot works with Unraid, it will start a clean shutdown then reboot. 1 Quote Link to comment
wes.crockett Posted September 28, 2023 Author Share Posted September 28, 2023 It would help if I had auto-start enabled in disk settings. Quote Link to comment
wes.crockett Posted October 2, 2023 Author Share Posted October 2, 2023 Ever since downgrading to 6.12.3, I have had no issues with my system. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.