[6.12.2] Intermittent Server lockup or inaccessible

[DanielCoffey]

I have been having infrequent issues with my server apparently dropping off the network or being inaccessible from a browser and having to be shut down by its own power switch and wondered whether I am developing a hardware issue with it. I would appreciate some advice. Specs in my signature.

 

Because the server is infrequently used for Plex once every day or two it spends a lot of time asleep to save power. I send it a Magic Packet using a WakeOnLAN tool to wake it up, check the Plex docker is active (it sometimes isn't - I suspect Plex timeout issues) and use it to watch a film. This always works without issue. Some time after the film is finished I open a browser on my main desktop or tablet and request it Sleep again. Most of the time it works exactly as expected (c. 90%).

 

Occasionally the browser fails to present the server's GUI. The server is still visible on the Router list of devices connected but nothing will access the server. I have to go over to it and press the Power button to make the device perform a shut down.

 

On a reboot it works perfectly... till next time. Until today.

 

Today the quick press of the power button did nothing. No drive activity, nothing happening at all. I had to press and hold to force the PSU to shut down.

 

Does this sound like a hardware issue to you? The motherboard perhaps? I do have a monitor I can connect to it for diagnosis if needed. Might there be logs stored in the Flash drive about possible causes?

[JorgeB]

You can enable the syslog server and post that after a crash to see if there's something there, should post the complete diagnostics at that time.

[DanielCoffey]

Thank you, I will do that. It may take a while to reproduce but I will run the log server until then.

[DanielCoffey]

Actually when it happens again it is unlikely that will be able to run the diagnostic tool because I won't be able to access the GUI to trigger it.

[JorgeB]

Diags you can get after a reboot, it's mostly to see the hardware.

[DanielCoffey]

Good to know. I have set up a cache-only share and pointed the syslog server to that. The system always comes back up after a power on so we should be able to get it. I will post back next time it occurs.

[DanielCoffey]

Got it, sort of. This time it failed to Sleep and left messages in the syslog.

 

I tried to command it to go to Sleep at Jul 14 18:27:28 and also 18:29:41 and it did not do so. I was able to get back in to the GUI on the browser which was helpful.

nas-plex-diagnostics-20230714-1933.zip syslog-192.168.1.11.log

[JorgeB]

Hi can only see PM related errors, if you don't suspend the server does it still crash?

[DanielCoffey]

I will perform a clean reboot and test the Plex docker. It should only take a couple of days to test and find out. Thank you for the help.

[DanielCoffey]

Alright - it has done it again - the server is inaccessible via the GUI after a period of inactivity and has to be rebooted.

 

Changes I made since last time are...

 

1. Upgraded to 6.12.3 and Rebooted the server

2. Set HDD spin down time from Never to 30 mins (no idea why I had set it to Never in the past)

3. Instead of Sleeping the server yesterday evening I left it active overnight.

 

Interesting events during the overnight inactivity (no idea if they are significant)...

 

Jul 16 03:53:26 NAS-PLEX apcupsd[2501]: Power failure.
Jul 16 03:53:26 NAS-PLEX apcupsd[2501]: Power is back. UPS running on mains.

 

 

Also...

 

Jul 16 05:15:50 NAS-PLEX kernel: nginx[14966]: segfault at 218b8 ip 00000000004e7630 sp 00007ffdaebced40 error 4 in nginx[424000+110000] likely on CPU 0 (core 0, socket 0)
Jul 16 05:15:50 NAS-PLEX kernel: Code: 40 00 48 8b 47 08 ba 80 33 e1 01 48 85 c0 48 0f 44 c2 c3 0f 1f 80 00 00 00 00 48 83 7f 20 00 74 19 48 83 ec 08 e8 30 48 02 00 <48> 8b 40 08 48 83 c4 08 c3 0f 1f 80 00 00 00 00 48 8b 47 10 c3 66
Jul 16 05:15:50 NAS-PLEX nginx: 2023/07/16 05:15:50 [alert] 2830#2830: worker process 14966 exited on signal 11
 

 

Also an unusual event this morning that spammed a lot of log file. Time stamp starts here and continues...

 

Jul 16 06:21:42 NAS-PLEX nginx: 2023/07/16 06:21:42 [alert] 2830#2830: worker process 18163 exited on signal 6
Jul 16 06:21:42 NAS-PLEX nginx: 2023/07/16 06:21:42 [alert] 2830#2830: shared memory zone "memstore" was locked by 18163

 

I was unable to access the GUI at about 07:38:00 so short-pressed the Power Button to trigger a shutdown which worked as expected.

nas-plex-diagnostics-20230716-0842.zip syslog-192.168.1.11.log

[bonienl]

Can you test with "IPv4 only" setting on br0 and see if that makes a differrence?

Please provide diagnostics of the new test.

Thx

 

[DanielCoffey]

Thank you - I have stopped the array, set IPv4 Only and applied that change followed by a shutdown and reboot of the server. Everything came up cleanly and a peek at the syslog shows that there are no references to IPv6.

 

Here is a Diagnostic if it is needed of the server in a healthy state and I will monitor it over the next couple of days allowing the server to remain powered on.

nas-plex-diagnostics-20230716-1125.zip syslog-192.168.1.11.log

[bonienl]

Quote

Here is a Diagnostic if it is needed of the server in a healthy state and I will monitor it over the next couple of days allowing the server to remain powered on.

 

This looks a lot better, it is a bit of a puzzle why IPv6 gives so much grief on your system.

 

[DanielCoffey]

I have an ASUS RT-AX86U running Merlin's firmware on a Giganet FTTP connection and the IPv6 page shows it is set to Native (PPPoE), PPP which seems correct according to ASUS' wiki.

 

The motherboard is about 6 years old as you can see from my sig so it might be something to do with IPv6 support on that. I am not too bothered if the change back to IPv4 corrects the issue with the GUI being inaccessible.

 

I'll watch it for a couple of days then revert to my normal "Sleep after using the server for the day" routine.

 

I do have a secondary issue of Plex failing to find my media after about a day which requires a Docker restart but perhaps it is also connected to this issue. I will monitor Plex and raise a separate Issue if I still have problems with that Docker.

 

Thank you everyone for the assistance and hopefully I can flag the post as Solved in a couple of days.

[DanielCoffey]

Well it has failed to Sleep again even under IPv4.

I left the server running for three days and it was perfect, no isses, no errors in the log so after using Plex today I slept the server. It failed to Sleep and I didn't notice for a few hours. Once I spotted the server awake I slept it again and this time it worked.

The issue occurred in the syslog at Jul 19 18:58:24 NAS-PLEX s3_sleep: Enter sleep mode

 

I really would prefer to have the server Sleeping when not in use as according to the UPS it is using about 40W with drives spun down.

 

Any thoughts as to the crash on sleep?

nas-plex-diagnostics-20230719-2300.zip syslog-192.168.1.11.log

[DanielCoffey]

Thanks bonienl for the tip on using "IPv4 only" for my specific server and router combination. It has certainly stopped any network errors in the logs and fixed the inability to access the GUI after inactivity.

I am still plagued by the issue of Sleep not working reliably but this seems to be an ancient unraid issue dating back nearly ten years according to Google. I would prefer not to Shutdown the server daily but would also like to avoid the electricity cost of running the server 24/7 when it is only actually used for about 2 hours a day. Looks like I will have to Sleep then go and check if it actually has done it.

[itimpi]

7 minutes ago, DanielCoffey said:

I would prefer not to Shutdown the server daily

Is there a reason for this?    If not it would seem easier than having to do any manual checking of the server.

 

i personally DO shutdown my server overnight.   This was one of the things that incentivised me to add a ‘restart from position reached’ for array operations to the Parity Check Tuning plugin as these can easily take over a day to complete with large drives in the array.

[DanielCoffey]

If Sleep worked perfectly every time, it makes bringing the server up very convenient for me as it is tucked away in a spare room. From my desktop PC I can issue a WakeOnLAN command from a small app and 30 seconds later the server is available for us to use. Of course I still have to use the GUI to Sleep and I agree that is the same amount of effort as selecting Shut Down.

 

Bringing the server up after a Shut Down requires going up to the server and pressing the power button so I may have to move it to a more prominent position. It also takes a good few minutes before it is ready to use.

 

I know it is really just a small inconvenience but it is an irritation when there is a function (Sleep) that is supposed to make it easier but fails to work reliably and does not appear to have been thoroughly discussed for almost ten years.

[itimpi]

10 minutes ago, DanielCoffey said:

Bringing the server up after a Shut Down requires going up to the server and pressing the power button so I may have to move it to a more prominent position. It also takes a good few minutes before it is ready to use.

 

Have you checked that your system does not have an option in the BIOS to do a Wake on LAN even from a complete shutdown as many systems do.   I agree with the time delay though so I have my own systems set to power on at a given time.

[DanielCoffey]

I will check, thank you. It is an ASUS H370 ITX board on the 2811 bios. I would have to connect a monitor to see what bios options were available but it could be well worth a rummage in the Sleep settings.