Unraid Server turn on after power outage but not loading OS

[Rommel]

Hi all, hope someone in here can guide me to fix this.

 

Since the last month I have a problem with my unraid server, I noticed that after a power outage it turn on by itself but does not load the OS until I press the restart button in the machine.

 

Before that it turn on by itself but there was no problem loading the OS; the problem started since a power outage 1 month ago, that power outage cause some problems in my docker image but I fix it by deleting it and recreating the containers, so, I know that this power outage screw up somethin in my machine.

 

Is there something I can check in my logs to see what the problem was? I don't think it is possible since when it happens I cannot communicate with the server until restarted but at that moment everything works as always.

 

Note: I know I need a UPS, will try to buy it as soon as I can.

tower-diagnostics-20240507-2009.zip

[JorgeB]

Post a photo of the monitor showing where it's stuck.

[Rommel]

7 hours ago, JorgeB said:

Post a photo of the monitor showing where it's stuck.

Hi JorgeB, I noticed that what I believe the error was wrong. The system load every time, but it looks like after a couple of minutes of activity it just freezes; everything stop working, from WebUI to the CLI (I tried this because I have a screen pluged to the unraid machine).

 

I turn on Syslog Server (a copy is attached) and the last line looks suspicious to em:

 

May  8 08:43:22 Tower kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=libvirtd,pid=21996,uid=0
May  8 08:43:22 Tower kernel: Out of memory: Killed process 21996 (libvirtd) total-vm:1389456kB, anon-rss:4040kB, file-rss:4kB, shmem-rss:3248kB, UID:0 pgtables:276kB oom_score_adj:0
May  8 08:43:22 Tower file.activity: File Activity inotify exiting
May  8 08:43:22 Tower sudo: pam_unix(sudo:session): session closed for user root

 

It says that system is out of memory and that something was "killed". Any ideas?

syslog

[JorgeB]

Possibly a memory leak with Connect, try uninstalling the plugin and re-test

[Rommel]

Hi again JorgeB, I uninstalled the plugin and server have been up for around 1 hour and no issues yet. Before it the server freezes after 15-20 minutes.

 

I'll report back tomorrow if no more freezes happens.

 

Btw, after I turned on the server I got an error from one of my drives, it was associated to this: UDMA CRC error count. I read it could be because of a faulty sata connection/port; don't know if it is related or not to the freezes but will report back if the count increases, it currently is at 27.

[JorgeB]

That's usually a SATA cable issue, replace the cable and see it stops increasing, it should not make the server crash.

[Rommel]

Hi, I adjust the sata cable and error stop.

 

Now, regarding the OS freezing I tried 3 different things, all unsuccessful:

1. Stop docker service from settings.
2. Stop docker + VM service from settings.
3. Stop the array

In all 3 cases the OS stop working and freezes and logs always show as the last entry "OOM Kill inotifywait".

 

Additional symptoms: As long as I have the WebUI open it does not crash, but if I close it, in less than 5 minutes is freezes and don't let me access the webUI and nothing works.

 

At this point my guess is that it could be an issue with the USB Stick or RAM (?). It is weird for it to continue happening even when the array is stopped.

 

I'll try a new ram kit today and see how it goes.

[JorgeB]

Post a new syslog after a crash.

[Rommel]

Hi, thanks for continue answer to this post.

 

I'm attaching 2 of the syslogs from yesterday; the first one is the syslog file of the system as it works normally and the second one after turning docker service off. The following is what I noticed:

 

1. 0 Syslog All On: From 20:28 to 20:44 the system was working fine, then, nothing was logged from that moment until 21:39, then at 21:45 the system trimmed docker and cache drives and finally at 21:50 it started messing with the memory.
2. 1 syslog Docker off: From 22:20 to 22:34 the system registered me stopping docker service, looks like some packages were updated and finally it registered me triggering mover and scrubbing the VM drive. Then at 22:41 it starting messing with the system memory.

Right now the system is on and the array is stopped, I'm just monitoring if it freezes at some point. Syslog server is on and saving the file to the USB drive.

 

This is frustrating

0 syslog All On 1 syslog Docker Off

[JorgeB]

Disable VM and docker services and reboot in safe mode, do you still see the OOM issues?

[Rommel]

Weird behavior, my server was off most of the day, from 6 am to 8 pm. When I got home I open it and replug all sata cables, clean it and turn it on. Since then I haven't had any freezes; so, it have been around 16 hrs and it is working just fine.

 

I'm still monitoring the syslog if something changes. I'll wait some more time.

[Rommel]

Well, unsure of what really happens but I was able no notice how the ram usage indicator started to fill until the point it breaks the system. What I did was to disable Plex docker DLNA option and haven't had additional problems. Will mark this thread as solved.