[SOLVED] Discovered my Unraid in a frozen state

[chris_netsmart]

would someone mind having a look at my Diagnostics log file and confirm, what I think it is , as this has happened to me to twice this month, and I think one of my components is no playing nice.

 

I discovered my server was still running, but with no activity network light flashing, so I have to do a warm reboot. and once I did this all the services and hard drivers came back on line.

 

many thanks

tower-diagnostics-20200613-0908.zip

[itimpi]

Unfortunately those diagnostics are of limited use as they start from the reboot and do not cover the problem period (by default logs are only stored in RAM).  You can follow the steps shown here to get logs that can survive a reboot.

 

[JorgeB]

Like itimpi mentioned syslog before rebooting might provide more clues, still there are a few visible issues:

 

Jun 13 09:06:45 Tower kernel: usb 1-9: reset high-speed USB device number 2 using xhci_hcd
Jun 13 09:06:45 Tower kernel: sd 0:0:0:0: [sda] 30253056 512-byte logical blocks: (15.5 GB/14.4 GiB)

 

Flash is dropping, use a USB 2.0 port instead (if available).

 

 

You are overclocking the RAM for the CPU used, it's a known issue with Ryzen see here:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=819173

 

[chris_netsmart]

thanks for the reply.  I will look into the syslog saving procedure.  

 

@tee-tee jorge no I am not overclocking anything,. and  I will have a look at the port which my UNRAID is plug and it looks like a USB 2. here is a image of the back of my motherboard and I have plug it into the top USB port

 

[JorgeB]

18 hours ago, chris_netsmart said:

no I am not overclocking anything,

RAM is @ 3200Mhz, max supported speed with your CPU is 2667/2933Mhz depending on the board.

[chris_netsmart]

1 hour ago, johnnie.black said:

RAM is @ 3200Mhz, max supported speed with your CPU is 2667/2933Mhz depending on the board.

I have a MSI B450 TOMAHAWK Motherboard

[JorgeB]

 

I have no idea if your board has 4 or 6 PCB layers, you can try to find out or just use 2667 for now, and if it's stable then you can try higher speeds.

[chris_netsmart]

thanks for your advice, at the moment my server has now moved on to power shutdown as and when it feels like it, so I have changed my P{persistent logs to Mirror to Flash and hopefully this will catch something.

 

Update:

the server has been running for about 10 mins and then it auto shutdown.  I then gone to the USB to see if I can copy the log files but all I can I see are old files.

 

@tee-tee jorge  I don'r understand why this has just started as it has been stable for about 6 months with no issues and, only the past few days has it started to play up

 

I will do as you advice and lower my RAM speed to see if this helps, and I will report back my findings

 

Update:

 

Last night after about 4 hours the server again shut down, then only thing that was running was my MontionEye Docker.

 

so this morning, I have created a new Unraid USB Pen Drive and booted off this. I have have connect my RAID to the new OS

 

I will see how long this last before it turns off.  if it does then it will indicated that it is a hardware issue and not OS

 

Edited June 16, 2020 by chris_netsmart
Update - RAM Speed and System Stopped

[JorgeB]

If it's shutting down by itself there's likely a hardware problem, like overheating or bad power supply/board.

[chris_netsmart]

7 hours ago, johnnie.black said:

If it's shutting down by itself there's likely a hardware problem, like overheating or bad power supply/board.

If this is true then woundn't this happened early and not 6 months down the line. And when i monitor my temp the cpu and motherboard are both between 45c and i have good air flow.

 

But i will look into this @johnnie.black

 

Update: just checked my motherboard and CPU and both are bouncing around 49c to 53c

 

 

Edited June 16, 2020 by chris_netsmart

[JorgeB]

Any hardware can go bad at any point in time. 

[chris_netsmart]

5 hours ago, johnnie.black said:

Any hardware can go bad at any point in time. 

please don't say that as it is going to be very hard to fine.

 

here is my syslog which I hope will help to troubleshoot the issue

syslog

Edited June 16, 2020 by chris_netsmart

[JorgeB]

If the syslog is after a crash/shutdown there's nothing there, which further suggests a hardware issue.

[chris_netsmart]

no the syslog was running and saving to the flash drive.  as it set it to Mirror to Flash

[JorgeB]

Only need to post it after the server crashes, not much point in looking at it before that.

[chris_netsmart]

@johnnie.black this is a pre crash syslog.

Edited June 16, 2020 by chris_netsmart

[chris_netsmart]

On 6/14/2020 at 10:49 AM, johnnie.black said:

 

I have no idea if your board has 4 or 6 PCB layers, you can try to find out or just use 2667 for now, and if it's stable then you can try higher speeds.

@johnnie.black I just changed my RAM to 2667 as advise.

[chris_netsmart]

here is a up today SysLog File,  the server was running for for a few hours and it just crashed.

syslog Crashed 16-06-2020

[JorgeB]

Unfortunately there are no errors logged before the crash, most likely a hardware issue.

[chris_netsmart]

Good News I have found the issue:  the issue was a power supply issue that for some reason it kept on tripping out,

this was discovered after going through all the hardware tests for Memory - CPU and Hard Discs

 

if anyone else if having issues here is a quick work through of my testing.

 

removed all external connections ' GPU, HD's Raid Controllers

get a ubuntu boot USB and boot into test mode.

 

from here I ran the Commands to test the HD's  smartcrt -l  and  and also I found a online website to test the CPU, GPU.

then I started to add things like the Raid Controller PCI board ,  at which point I start to have issues so I through RAID Control so I

replace this with a spare and still have issues.

 

I even updated my Bios to the latest version.

 

so this only leaves Motherboard and PSU,  so I loan a PSU from a friend and retested.  and so far it has been stable for 2 days.

as all my HDs are running. so I have go out and order a new PSU which I will install into my Unraid. and for fun I am also going to add a few more fans to help with the cooding as at the moment it is stilling around 40c

 

thanks to all who posted in here.

[JorgeB]

Thanks for reporting back, if you don't mind I'm going to tag this solved.

[chris_netsmart]

Please do