My Unraid server, seemingly randomly, will stop responding and hang, and then require a hard reboot to get up and running again. During this hang, there is no output from the local video adapter, the server network interface is not ping-able, and nothing its running responds.

I've tried troubleshooting by setting the output of syslog to the flash, but there is nothing in there of any concern, in fact it doesn't seem to log any warnings about the hang.

 

The latest example shows the last log entry on the 21st at just after 8pm, however it was definitely working up until midnight of that day. this morning I woke up and it was fully hung, requiring a hard reboot, and this is where the log picks up again. the last few log lines are probably me updating a few containers I have running on it.

 

Could I get some more pointers on how to troubleshoot this please? is there a log I am missing somewhere that might be of some use?

 

Thanks

syslog.txt

Your diagnostics might provide more information.

Hi,

 yes I've already looked at some of the logs this generates, however they all start after the reboot, and dont contain any logs from before, unless I'm missing something?

 

Thanks

 

EDIT: attached diag logs

svalbard-diagnostics-20210922-0915.zip

Edited September 22, 20214 yr by john014

Yes, you are right, to have logs that survive a crash/reboot, you have to set up a syslog server.

 

However, the diagnostics provide a lot of information about the system and our experienced users can dig into this and provide suggestions.

Sorry to bump, but has any super duper Unraid person managed to look at my diag logs to point me in the right direction yet?

 

Thanks

Did you have any syslogs from syslog server?

I mirrored the syslog to flash as I don't have anything else that I can leave on permanently, and it looks like the syslog stops just before the server crashes as there is nothing of interest in there, excerpt is in OP.

 

Try downgrading to 6.8.3, I had issues with 6.9.x on intel hardware, freezing every 1-2 days, now have 60 day uptime back on 6.8.3

Thanks for the suggestion, but my server started on 6.9.2, so I'd need pretty hard evidence that there is a bug in 6.9.2 to roll back that far. I also don't know what configs I would have to redo so sounds like a big job to me.

Config stays the same, you may have to redo your cache drives but that's it. I downgraded by replacing everything on the usb drive except for the config folder. (ran out of rollbacks trying multiple 6.9 versions. Its cheaper than buying replacement hardware if you don't find another fix. MAke sure you have the ram at the recommended speed for the cpu, disable c states, make sure you don't have anything funky with power supply power control etc.

I too have the same thing. Randomly just reboots, nothing in the logs that pertains to any issue. @john014, did you ever figure out what was causing this? The last log entry for me is:

Mar 4 03:49:46 UnRaid emhttpd: read SMART /dev/sdo

And the next is after I powered it back up, and the system was starting. It has done this for quite sometime, and it is super annoying. Is there any sort of additional debugging we could enable?

How long should it take to reboot? A similar thing happened to me -- I was logged out, cache drives wouldn't show up, in-progress file transfers aborted, and the login popup wouldn't come back up. So, I figured I might as well reboot (I'm just getting things setup and transferred over so the server isn't really in use yet), and now it's been sitting on the REBOOT screen for "1251" seconds (?). That seems excessive, no?

 

Edit: I just did the "dangerous" thing and hard powered it off. After rebooting, logs indicated a problem with the attached USB drive I was copying from and after disconnecting it, it booted fine.

 

This has been a bit of a finicky setup process, but hopefully things smooth out once it's all done.

Edited March 4, 20224 yr by no-thanks

Yeah, somethings holding it up.  Do you have an attached monitor / keyboard to the system?  Reboot should take around 5 minutes max depending upon what you've got installed

Thanks! I do have an attached monitor, but there was nothing on the screen when I switched inputs over. I didn't have my monitor on the Unraid box's output when I rebooted, so I didn't see what may have happened in the process. I haven't gotten the onboard GUI working properly yet.

 

It seems something happened to the external HDD during the copy process, because now Unraid and my Mac won't recognize/mount it. It's possible it died, it was pretty old. Nothing of import on it, fortunately, just some easily replaceable media files.

Edited March 4, 20224 yr by no-thanks
clarification

On 3/4/2022 at 3:11 PM, hamish_18 said:

I too have the same thing. Randomly just reboots, nothing in the logs that pertains to any issue. @john014, did you ever figure out what was causing this? The last log entry for me is:

Mar 4 03:49:46 UnRaid emhttpd: read SMART /dev/sdo

And the next is after I powered it back up, and the system was starting. It has done this for quite sometime, and it is super annoying. Is there any sort of additional debugging we could enable?

Hi, 

 

I think we had different issues - as mine was a lockup. It seemingly fixed itself with no changes made by me. The only thing that would have changed would have been the docker containers I have running - but if a container can bring a system to its knees then somethings wrong...

 

12 minutes ago, john014 said:

if a container can bring a system to its knees then somethings wrong

A misconfigured container can fill rootfs and cause unpredictable OS behavior since the OS lives in rootfs and needs space to work. Any volume mapping that isn't actual host storage is in rootfs.

I think I have the same issue.

My servers was still running with an screen attached and I got no output. the web interface of unraid wasnt reachable and every webpage I am hosting had the same issue.
I had this problem before and also wasnt able to find the solution to it.
(While typing My screen turned black and i wasnt able to get it back, solution was replugging the usb dongle and turning the keyboard (logitech k400+) off/on after this typing made the screen come back.   Mayby this was also the problem I had before will try this out next time it happens.)

I wil list my server specs mayby someone in the future will find this topic and something simulair:
CPU: AMD Ryzen 5 PRO 4650G
MOBO: ASRock B550M Pro4 Bios L1.42 (updating the bios after this post to latest non beta 3.40)
Memory: 48GB DDR4 2x CMK16GX4M2A2666C16 and 2x CMK32GX4M2B3000C15 running @ 3000 MT/s
USB: Corsair Flash Voyager Slider X1 USB 3.0 64GB
Drives (these are al over the place):
Parity: Western Digital WD80EDAZ (8TB)
Disk1: Toshiba DT01ACA300 (3TB)
Disk2: Western Digital WD30EZRX (3TB)
Disk3: Western Digital WD30EZRX (3TB)
Disk4: Western Digital WD60EMAZ (6TB)
Cache: WD Blue SN570 (500GB)
Cache2: WD_Blue_SN570 (500GB)
PCIE: 6x sata card (aliexpress TSR781, please dont hate me) (https://nl.aliexpress.com/item/1005006974070561.html?spm=a2g0o.order_list.order_list_main.25.6db379d2Krro5q&gatewayAdapt=glo2nld)
PCIE: extra m.2 slot (aliexpress, JEYI M.2 NVME SSD) (https://nl.aliexpress.com/item/1005005924920287.html?spm=a2g0o.order_list.order_list_main.30.6db379d2Krro5q&gatewayAdapt=glo2nld)


Something weird I had before this problem is: I was not able to open shares with \\192.168.0.5\share but i was able to load \\unraid.local\share, after one reboot of the unraid server i was able to open the share with both option again.

Hope someone finds this intressting

syslog-previous.txt

5 minutes ago, Dannydvd21 said:

Ryzen

 

I’m having the same issue but on an intel 11th gen platform. Any ideas ?

4 hours ago, BaconDroid said:

I’m having the same issue but on an intel 11th gen platform. Any ideas ?

It's always best to start a new thread, but since this one is no longer active, we can continue here.

Enable the syslog server and post that after a crash, together with the diagnostics after a reboot.