Hi, within the last three days I had multiple freezes or the system becomes inaccessible to me.

The reason for the crashes are unclear at the moment and I don't have much to go on. In order to troubleshoot I need so some logs or errors, which I'm unable to produce at the moment.

 

Symtomps:

• Server IP is remains pingable.
  • connected LAN port is on a pcie dual nic card (intel) and not on the mainboard
• all other protocols seem to be down, not http to the webgui and no ssh either
• server is totally headless without gpu, so I don't have a vga output yet to see locally what is going. Trying to find a gpu to get some insight.
• one disk a reallocated sectors. Not sure if related to manual power offs or relevant for this issue

 

Actions:

• I have to power off locally and reboot manually in order to get system responsive again
• I can't pinpoint my finger down 100%, but I've had reboots that didn't help either. I tried rebooting without the unraid flash drive and that seemed to worked 100%, but without vga output I couldn't really tell what was going on
• system diagnostic logs seem be useless (nonetheless I attached them) after reboots. I've read (https://docs.unraid.net/unraid-os/manual/troubleshooting/#persistent-logs-syslog-server) to setup the syslog mirror.
  • need some guidance here: mirror to flash means using my unraid license flash drive, right? Which in turn could wear it down 
• I've updated to the latest .6 release since the last crash, but I don't expect it to help

 

Please help me get some direction on how to troubleshoot something like this. I have more than enough IT experience to read docs and logs, I'm just not proficient enough with unraid and linux troubleshooting if it the error is not in my face enough :)

unraid01-diagnostics-20240106-2201.zip

Enable the syslog server and post that after a crash.

 

On 1/7/2024 at 11:32 AM, JorgeB said:

Enable the syslog server and post that after a crash.

 

Hi, so I've configured Syslog Mirror to Flash on Jan. 8th.

 

The Diagnostics now include a 

syslog-previous.txt

But I don't see much new, but maybe I'm wrong. The issue seems to have happened between yesterday (Jan 9th) evening on today (Jan 10th).

 

The last line in the previous syslog is 

Jan  9 19:35:14 unraid01 monitor: Stop running nchan processes

I was pretty much logged in the webgui since Jan 8th and am very confident that I did NOT log-out at this time manually, as this is around time I was putting the kids down to bed.
Since my last comment:

• I've organized a GPU and ran MEMTEST86. After 11 passes with 0 errors (I's just 16GB RAM) I've stopped continuing. 
• I've replaced Disk5 because of reallocated sectors and have started a rebuild, which seems to have completed before the "crash"

unraid01-diagnostics-20240110-0708.zip

Unfortunately there's nothing relevant logged, this usually points to a hardware issue, one thing you can try is to boot the server in safe mode with all docker/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

9 minutes ago, JorgeB said:

Unfortunately there's nothing relevant logged, this usually points to a hardware issue, one thing you can try is to boot the server in safe mode with all docker/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

Thought so too :(. 

I'll give it a try.

 

What I forgot to add in my previous comment, since a GPU was connected this time around:
While the server is unreachable (except PING), the local console is still visible and somewhat accessible. In practical terms this means:

• the sign in prompt is active/visible and accepts a username.
• when I type in root and hit ENTER, the cursor jumps to the next line, but
• the console is not really prompting me for/accepting my password, since everything I type in is in plaintext visible on the console (I don't remember any linux command line login, where the password is visibile while typing in, and my la)
• curiuosly, the sign in is visible aborted after 60s 

not sure if this helps in any way.

16 minutes ago, 1l25kj1 said:

not sure if this helps in any way.

Possibly suggests that the issue can be more software related to me, I would still recommend trying what I posted above, if it's a specific app/container causing the issue you may find out which.

On 1/10/2024 at 12:04 PM, JorgeB said:

Possibly suggests that the issue can be more software related to me, I would still recommend trying what I posted above, if it's a specific app/container causing the issue you may find out which.

Hi, so I'm back with an update, but still nothing helpful in the syslog.

I've made to observations:

• the local console is still accessible. The behaviour is the following
  • login prompt is active - I enter "root" and hit ENTER
  • for a few seconds nothing happens
  • then the "password" prompt appears in the next line - but I wasn't able to enter my stupidly long password fast enough before the 60s timeout occurs.
    • I've hard rebooted the machine and since then shortened the password and will probably get another chance tomorrow, as the "crashes" have been daily.
  • Video: 
    
     
• Resetting/Rebooting the machine does not work as I do not reach the unraid bootloader
  • since I now have an active GPU and Display connected to it, I was able to "see" some additional display prompts
  • after the reboot, even multiple reboots, I'm unable to boot into unraid - basically no boot device is detected
  • I have to once unplug the unraid usb drive, plug it back in immediately and then reboot
  • after this, unraid properly boots and is again accessible via http and all other protocols
  •  

I'm currently uploading some screenrecordings to youtube and will post the links tomorrow after the upload and processing.

Really interested what you make out of this information.

Edited January 16, 20242 yr by 1l25kj1

10 hours ago, 1l25kj1 said:

Resetting/Rebooting the machine does not work as I do not reach the unraid bootloader

This is an unrelated issue, looks like the board is not finding a bootable device, you can try recreating the flash drive or using a different one.

13 hours ago, JorgeB said:

This is an unrelated issue, looks like the board is not finding a bootable device, you can try recreating the flash drive or using a different one.

I will have to give this a try to be honest, as I‘m not ruling out a relation yet. Daily occurance just happened again.

• local console login was not possible, after entering root as user name, since the password prompt never appeared. The next message was always the login timed out
• while still running, I unplugged the unraid usb drive and plugged right back in and voila suddenly (for the first time) local console login works like charm
• ssh and the protocols still do not work though.
• pinging my gateway or other hosts does not work, but not in timeout, but rather a error that the command can‘t be executed due to missing file or so (don‘t have a photo of it right)

I don‘t know what to commands to run locally. I looked up https://docs.unraid.net/legacy/FAQ/console/ but half of those don‘t work for me in this mode. Tried powerdown, nada, another error.
Is there anything I can try locally in this stage, before wiping/restoring the usb drive?

10 hours ago, 1l25kj1 said:

before wiping/restoring the usb drive?

I would try that, and if the same try a different flash drive.

On 1/18/2024 at 10:15 AM, JorgeB said:

I would try that, and if the same try a different flash drive.

Hi, wanted to give an update.

• I plugged the unraid usb drive into my Mac and ran a disk repair on it and put it back into the unraid server.
• I since have uptime of over 10d with a manual reboot shortly before that, so somehwhere between 10d-15ds of uptime. 

Overall I still can't really tell if this was a working solution or just coincidence. I haven't changed any other settings yet.

So I'll keep an eye on it, post an update if something new develops.