Cannot Connect to Server

[SyberSin]

Hello,

 

I'm having a serious issue with my unraid server. I would greatly appreciate if somebody could help me. Please read, as this is a unique issue that I have not seen anywhere else on the forums. I have searched and read almost every post on the forums and other places online with no solution.

 

My unraid server keeps disappearing from my network, which is happening every day now. I cannot connect to it via ssh, gui, local network, or by my direct 10gb connection. It's as if the eth ports are not working (although lights are blinking on them and my unifi dream machine.) 

The server shows as offline in my dream machine as well.

 

I also cannot boot into safe mode or into the gui mode with a monitor attached, the output hangs if I try and it never comes online, cannot even reach the console. I have reset the network, and the server with no luck. When this happens, I essentially loose the whole server. Now here is where it gets stranger. I posted about this issue once a while back with no responses:

 

 

 

The solution I found in that post still 'works'. If I completely unplug the server from power, plug it back in, it boots and shows on my network just fine. I don't even need to replace the network config files. Just unplug. But unlike the first time, after the server is running for about 24 hours, it will disappear from my network again, and all the symptoms above come back. I need to shut down the server with the power button and pull the plug again, reboot and Ill have access again and it shows backup on my network like nothing is wrong.

 

I first had that issue 7 months ago, and the server has been running constantly since then with no issues. All of a sudden, this problem is happening every morning now. So my solution is obviously not ideal and this is making my server unusable.

 

I do not think this is related but I did make a few changes before this started happening.

One of my pool drives died, so I replaced that drive and I upgraded my parity drives, I rebuilt the parity, checked the parity and everything was fine. 24 hours later this problem started happening again. Luckly, I tried my 'solution' and got back my server, so I immediately upgrade to 6.12.4 (from 6.11) hoping that would work. This morning the issue happened again.

 

In my other post I mentioned what I originally changed that appears to have started this whole issue. (Setting up a 10gb connection between my computer and server following space invaders guide.)

 

attached is the diagnostics.

 

server-diagnostics-20230927-1216.zip

Edited September 27, 2023 by EyeOfMaze

[itimpi]

You should set up the syslog server so that we can get a log that survives a reboot to see if we can spot something leading up to your syste’ problem.   The one in the diagnostics only shows what happened since you last rebooted.

[SyberSin]

16 minutes ago, itimpi said:

You should set up the syslog server so that we can get a log that survives a reboot to see if we can spot something leading up to your syste’ problem.   The one in the diagnostics only shows what happened since you last rebooted.

 

Thank you for your help, I just setup the syslog server as follows:

 

local syslog server

rotation: 100mb

number of files: 4

 

Should that be fine? and do I need to reboot unraid or keep it running?

[itimpi]

4 hours ago, EyeOfMaze said:

 

Thank you for your help, I just setup the syslog server as follows:

 

local syslog server

rotation: 100mb

number of files: 4

 

Should that be fine? and do I need to reboot unraid or keep it running?

As mentioned in the link, have you set one of the last two options in the settings?    If not then the server is just listening - not actually writing any messages.    It is normally easiest to use the Mirror to flash option.

[SyberSin]

8 minutes ago, itimpi said:

As mentioned in the link, have you set one of the last two options in the settings?    If not then the server is just listening - not actually writing any messages.    It is normally easiest to use the Mirror to flash option.

 

Okay, yeah I just set it to use the flash option instead. Do I need to reboot for it to take effect? as the docs mention it captures everything from the start of the boot process.

 

So when I need to shutdown the server again and pull the plug, do I extract the logs first before I plug the server back in? Or would you like me to plug the server back in, reboot then extract the logs?

Edited September 28, 2023 by EyeOfMaze

[itimpi]

5 minutes ago, EyeOfMaze said:

Okay, yeah I just set it to use the flash option instead. Do I need to reboot for it to take effect? as the docs mention it captures everything from the start of the boot process.

It normally takes effect from the point at which you set that option.

 

6 minutes ago, EyeOfMaze said:

So when I need to shutdown the server again and pull the plug, do I extract the logs first before I plug the server back in? Or would you like me to plug the server back in, reboot then extract the logs?

You can get the syslog off the flash after you have rebooted the server if that is more convenient.   Ideally take new diagnostics zip file at that point to post and also post the syslog file from the logs folder on the flash drive.

[SyberSin]

On 9/27/2023 at 8:27 PM, itimpi said:

It normally takes effect from the point at which you set that option.

 

You can get the syslog off the flash after you have rebooted the server if that is more convenient.   Ideally take new diagnostics zip file at that point to post and also post the syslog file from the logs folder on the flash drive.

 

Okay it finally did it again, here is the syslog and new diagnostics.

 

It was last seen on my network at 10/1/2023 8:10AM pacific standard.

server-diagnostics-20231001-1456.zip syslog

Edited October 1, 2023 by EyeOfMaze

[SyberSin]

It looks like it might be an issue with the network drivers? also my bios is slightly out of date but the latest version doesn't list any changes made that seem significant?

 

I'm on bios 1003:

https://www.asus.com/us/motherboards-components/motherboards/workstation/pro-ws-wrx80e-sage-se-wifi/helpdesk_bios/?model2Name=Pro-WS-WRX80E-SAGE-SE-WIFI

Should I upgrade to 1201?

 

I also found the intel drivers for what's on my board (Intel® X550-AT2 dual 10Gb Ethernet) for Linux here:

https://www.intel.com/content/www/us/en/products/sku/84329/intel-ethernet-controller-x550at2/downloads.html

 

but there is so many options I'm not sure which on is correct, or even if I need to / should?

 

The options for linux are:

1. Complete Driver Pack All OS 8/25/2023
2. Ethernet Connections Boot Util, Preboot Images, and EFI Drivers 8/7/2023
3. PCIe 10 Gigabit Connections under Linux 8/7/2023
4. Virtual Function Driver for 10 Gigabit Connections under Linux 8/7/2023
5. Non-Volatile Memory Update Utility 3/29/2022

 

Plus some of these have 10 different Linux options to download when I click on them.

 

If I do, which one and how would I go about installing on unraid?

Edited October 1, 2023 by EyeOfMaze

[JorgeB]

Several call traces logged, some look similar to the ones caused by this:

https://forums.unraid.net/bug-reports/stable-releases/crashes-since-updating-to-v611x-for-qbittorrent-and-deluge-users-r2153/

 

Check if it applies to you.

[SyberSin]

9 hours ago, JorgeB said:

Several call traces logged, some look similar to the ones caused by this:

https://forums.unraid.net/bug-reports/stable-releases/crashes-since-updating-to-v611x-for-qbittorrent-and-deluge-users-r2153/

 

Check if it applies to you.

 

That does seem very similar to what I'm experiencing. So I changed my binhex deluge to use libtorrentv1 and I already notice the Unraid dashboard feels more responsive. We will see if that was the issue or not. Thank you