Hello All - Looking for some help/direction, I'm kind of at a loss.  For approximately the last month, my Unraid server has been rebooting overnight, almost everyday.  I upgrade the P/S from a 650W to a known working/brand new 850W unit, but there was no change. 

 

There are no scheduled tasks running on a time schedule (I have a couple that run on server start or shutdown), and the times aren't perfectly consistent.  Here are the examples from the last few days:

3/15 - 3:46 AM

3/16 - no reboot

3/17 - 2:32 AM

3/18 - 3:14 AM

 

Through some of the forum posts, I was able to set up a syslog server to capture the logs outside of the RAM, but there are gaps from the last event captured in the syslog to the time the reboot occurred:

 

Mar 15 03:01:52 Tower kernel: [Hardware Error]: cache level: L1, tx: INSN, mem-tx: IRD

Mar 15 03:46 REBOOT [not in syslog, added to show sequence]
Mar 15 07:50:44 Tower root: Delaying execution of fix common problems scan for 10 minutes

 

Mar 17 02:03:04 Tower  crond[1241]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null

Mar 17 02:32 REBOOT [not in syslog, added to show sequence]
Mar 17 07:25:44 Tower root: Delaying execution of fix common problems scan for 10 minutes

 

Mar 18 03:00:09 Tower  crond[1240]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null

Mar 18 03:14 REBOOT [not in syslog, added to show sequence]
Mar 18 07:51:26 Tower root: Delaying execution of fix common problems scan for 10 minutes

 

Hardware:

AMD FX™-8300 Eight-Core @ 3300 MHz

16GB RAM

NVIDIA GeForce GTX 1060 6GB 

Please let me know if there's other information I can provide to help.  Thanks very much.

 

C.

syslog-192.168.0.203.zip

A reboot without anything logged usually points to a hardware problem, one thing you can try is to boot the server in safe mode with all docker/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

I had an UNRAID server at a clients that crashed EVERY night or ~16hrs. New RAM, replaced PS, updated the OS, disabled almost everything. 

It had one bad failing HDD, I kept putting off replacing that failing drive. I wanted to fix the constant crashing first.  I replaced the failing drive, and it has NEVER crashed sense. I honestly wouldn't have thought a bad drive would cause it. But it did.  

Not saying that is your issue, just saying to check the drives.

 

Here's the update today...I ran the server for most of the week in Safe Mode with all plugins and dockers disabled, and the server ran 5 days without restarting.  The first day I re-enabled Plex/Sonarr/Radarr/NZBGet and the server again rebooted overnight.  So tonight I will run only Plex + NetData to see if I have a reboot and if NetData can capture anything relevant.  After @Dking00000's comment, I looked at the drives and while everything looks good on the Main tab:

 

There is a smart error (thumbs down) for Disk 4 on the Dashboard:

 

I've attached the log in case anyone can help with this error (and see if it's somehow related to me rebooting issues).

 

 

 

tower-diagnostics-20230326-1325.zip

I am now having this same exact issue.  But I have boiled it down to Plex running overnight.

 

Was originally running linuxservers version, tried reinstalling it, still rebooted overnight.  Tried removing that version and installing the official version, still reboots.

 

It always happens around 2am.

 

All other dockers can run overnight with no issue.  But as soon as I let Plex run overnight I wake up to a fresh reboot and a parity check running.

 

I've checked syslog messages but they are not helpful. It appears to be a hardware issue (due to lack of information in syslog) but is also very much tied to the Plex docker.  If I do not run Plex then my server can run for weeks with no reboot.

 

AMD Ryzen 5 3400G

ASRock X570M Pro4

Edited April 10, 20233 yr by ugnaught

Just realized that Plex scheduled tasks run at 2am by default.

---------------------------------------------------------------------------------------------

 

 

@ugnaught: Please turn on "mirror syslog to flash", then (after the next reboot) provide diagnostics and the resulting "logs" folder from your flash drive so that someone here can help you.

@EDACerton Unfortunately I tried that already to look for a hint, but no dice.  Just begins showing bootup around 2:03

 

Apr 10 00:00:11 Unraid-Server root: /var/lib/docker: 25.3 GiB (27136815104 bytes) trimmed on /dev/loop2
Apr 10 00:00:11 Unraid-Server root: /mnt/cache: 338.9 GiB (363936219136 bytes) trimmed on /dev/nvme0n1p1
Apr 10 00:16:01 Unraid-Server  crond[1142]: failed parsing crontab for user root: Invalid frequency setting of  /usr/local/emhttp/plugins/ca.update.applications/scripts/updateApplications.php >/dev/null 2>&1
Apr 10 00:20:01 Unraid-Server  sSMTP[16636]: Creating SSL connection to host
Apr 10 00:20:01 Unraid-Server  sSMTP[16636]: SSL connection using TLS_AES_256_GCM_SHA384
Apr 10 00:20:01 Unraid-Server  sSMTP[16636]: Authorization failed (535 5.7.8  https://support.google.com/mail/?p=BadCredentials o9-20020a0568080f8900b00389533b1cc8sm3982888oiw.18 - gsmtp)
Apr 10 01:03:01 Unraid-Server  crond[1142]: failed parsing crontab for user root: Invalid frequency setting of  /usr/local/emhttp/plugins/ca.update.applications/scripts/updateApplications.php >/dev/null 2>&1
Apr 10 01:50:01 Unraid-Server  crond[1142]: failed parsing crontab for user root: Invalid frequency setting of  /usr/local/emhttp/plugins/ca.update.applications/scripts/updateApplications.php >/dev/null 2>&1
Apr 10 02:03:16 Unraid-Server kernel: Linux version 5.19.17-Unraid (root@Develop) (gcc (GCC) 12.2.0, GNU ld version 2.39-slack151) #2 SMP PREEMPT_DYNAMIC Wed Nov 2 11:54:15 PDT 2022
Apr 10 02:03:16 Unraid-Server kernel: Command line: BOOT_IMAGE=/bzimage initrd=/bzroot
Apr 10 02:03:16 Unraid-Server kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
Apr 10 02:03:16 Unraid-Server kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
Apr 10 02:03:16 Unraid-Server kernel: x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
Apr 10 02:03:16 Unraid-Server kernel: x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
Apr 10 02:03:16 Unraid-Server kernel: x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'compacted' format.
Apr 10 02:03:16 Unraid-Server kernel: signal: max sigframe size: 1776
Apr 10 02:03:16 Unraid-Server kernel: BIOS-provided physical RAM map:
Apr 10 02:03:16 Unraid-Server kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
Apr 10 02:03:16 Unraid-Server kernel: BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved

 

----------------------------------------------------

 

But one thing I did notice while poking around is that for some reason my Plex appdata is spanning across the array when I have that share set to 'cache prefer'.  So I changed it to 'cache only' and made a copy of my plex folder (plexmediaserver_old) and pointed the container at that copy which is sitting on only the cache now.

 

 

I wonder if something has gotten corrupted with that appdata folder and the mover can't get it off the array.  Then potentially when the Plex scheduled tasks run at 2am it hits some pocket of corrupted data.  Why on earth it is causing the server to reboot I have no idea though. Going to see if moving the data helps. 

 

I should know by tomorrow morning

@ugnaught This is very interesting.  Following my last post, I was able to isolate the same thing, it's also Plex causing my reboot issues, and my scheduled tasks are set at 2AM as well.  My Plex appdata is currently set to Prefer : Cache, but I also confirmed that all the data is on the cache drive.

@cmac1982  so bad news, I still had issues with Plex this last night and had a reboot this morning. 

 

Another thing I noticed is that my Plex container is set to with "Privileged = off".  Most of the reading I can find says to set it to "on".  So I just made that change too and will see if results are any different. 

 

Another thing I found (finding lots of stuff) is that upon the reboot this morning my server name changed from a unique name to the default "Tower" which could be an indication of a bad USB drive used for my boot device. I just ordered a replacement and it should be here in a couple of days and I will also test swapping that out. 

 

Additionally one thing to look at is the last time Plex was able to backup your database.  The default action (as can be seen in one of my screenshots in an earlier message above) is to do it every three days.  Then it appears to only save the 3 most recent. 

 

Looking at my backup folder it hasn't be able to successfully do that since February.  Which is around the time I began having random issues.  I haven't always had a use for Plex everyday so I sometimes went weeks without turning it on which led me to taking a while to find the issue.  

 

All in all........ this is really lining up that when Plex runs it's scheduled maintenance at 2am something is catching and causing a reboot.  This doesn't seem like a hardware issue at all. 

 

So I recently replaced my boot drive USB (somewhat recently....not in the same timeframe as I started experiencing these issues).  I have also changed the Priviledged setting to ON, and I checked the backups.  It doesn't look like there's been a successful backup here since November.  The reboot problem only started within the last 6-8 weeks, so while it may be related to that, it didn't go as far back.

 

 

Given the size of the DB that it's trying to back up on my 500 GB cache drive, I'm wondering if there could be a problem with the backup maxing out the available drive space?

I have plenty of space on my side.  So it likely isn't the cause of my issues. 

 

quick update, changing docker privileged to "on" did not affect anything. it still rebooted.

Last update (I hope). 

 

I tried rolling back the version of Plex, trying different dockers (binhex, official, etc) and even tried a fresh Plex install.  None of that worked.  Kept rebooting every night at 2am. 

 

Gave up and wiped my cache pool clean and started over.  I unmounted the cache drives, started up array with no cache, wiped cache drives, then added them back with the warning "all data will be erased if you add these drives". 

 

Only thing I copied over after this was my appdata backup for some other dockers (not Plex). 100% started over with a new Plex docker/server config. 

 

Has been three days with no reboots with Plex running 24 hours. 

 

Wish I could say exactly what was causing the issue but I grew tired of the reboots before I went scorched earth.  Something somewhere on my cache had to have been corrupted/bad permissions/etc.  Sometimes it is easier to start over than deal with the troubleshooting headache.