Basically title. I am on 6.12.10 (was 6.12.4). Tried updating to see if it would fix. Issue persists. Last memtest was fine and syslog shows nothing before crash (neither does "enhanced syslog"). Had decent uptime for a couple months and then almost daily crashing for the last month or so.

 

Previous instability was due to c-states on AMD, but I disabled those in bios and that lead to happy times for a while. I recently added a new drive to increase storage and had some issues getting it to show up through the LSI HBA I have so wondering if possibly that is going offline and causing an issue?

 

Attached current syslog file (I have a bigger one as well that goes back to August of last year).

 

Looking for any ideas on troubleshooting this. It has been...frustrating to say the least (I run pi-hole and other networking items on it so when it goes down, my world halts for 5-10 minutes while waiting for the system to come back up)

 

 

tower-syslog-20240504-1656.zip

16 hours ago, seecs2011 said:

I recently added a new drive to increase storage and had some issues getting it to show up through the LSI HBA I have so wondering if possibly that is going offline and causing an issue?

Seems unlikely to me, but without anything logged there's not much to go on, and looks more like a hardware issue, since memtest is only definitive if it finds errors, if you have multiple sticks try with just one, if the same try with a different one, that will basically rule out bad RAM.

Ah the mysterious "Hardware issue" I see talked about everywhere. I have new RAM arriving today to try (same amount but only two dimms instead of all 4 to make troubleshooting a bit easier).

 

Any other ideas of hardware tests to run?

Some MBs are picky about RAM.   Be sure to use Google to see if your MB is one of those.  If it is, be sure to check to see what you purchased is on the 'approved' list before you open the packaging. 

19 minutes ago, Frank1940 said:

Some MBs are picky about RAM.   Be sure to use Google to see if your MB is one of those.  If it is, be sure to check to see what you purchased is on the 'approved' list before you open the packaging. 

TIL - thought about possibly doing a BIOS update as well to see if there was some compatibility thing happening. Thanks for the suggestion though!

That awkward moment when you replace the RAM and the next system uptime lasts about an hour before crashing...

 

The reboots seem to consistently be 10 minutes after a message stating

May  7 17:45:06 TOWER emhttpd: spinning down /dev/sde
May  7 17:45:12 TOWER emhttpd: read SMART /dev/sde

 

Edited May 8, 20242 yr by seecs2011

Well, I made it a little over 1 day and 5 hours this time. Really tired of this system. I'm out of ideas.

Got a neat new error on this boot though

May  8 22:38:30 TOWER root: error: /webGui/include/DashboardApps.php: wrong csrf_token

 

Shows about 30 times in a row.

 

Same context around the crash/reboot again as well

May  8 19:12:01 TOWER sSMTP[29961]: Creating SSL connection to host
May  8 19:12:02 TOWER sSMTP[29961]: SSL connection using TLS_AES_256_GCM_SHA384
May  8 19:12:02 TOWER sSMTP[29961]: Authorization failed (535 5.7.8  https://support.google.com/mail/?p=BadCredentials 6a1803df08f44-6a15f17963csm1182386d6.5 - gsmtp)
May  8 19:27:44 TOWER emhttpd: spinning down /dev/sde
May  8 19:27:49 TOWER emhttpd: read SMART /dev/sde
May  8 19:43:50 TOWER emhttpd: spinning down /dev/sde
May  8 19:43:56 TOWER emhttpd: read SMART /dev/sde
May  8 19:59:57 TOWER emhttpd: spinning down /dev/sde
May  8 20:00:03 TOWER emhttpd: read SMART /dev/sde
May  8 20:16:04 TOWER emhttpd: spinning down /dev/sde
May  8 20:16:10 TOWER emhttpd: read SMART /dev/sde
May  8 20:32:11 TOWER emhttpd: spinning down /dev/sde
May  8 20:32:16 TOWER emhttpd: read SMART /dev/sde
May  8 20:48:18 TOWER emhttpd: spinning down /dev/sde
May  8 20:48:20 TOWER emhttpd: read SMART /dev/sde
May  8 21:04:24 TOWER emhttpd: spinning down /dev/sde
May  8 21:04:30 TOWER emhttpd: read SMART /dev/sde
May  8 21:20:31 TOWER emhttpd: spinning down /dev/sde
May  8 21:20:36 TOWER emhttpd: read SMART /dev/sde
May  8 21:36:37 TOWER emhttpd: spinning down /dev/sde
May  8 21:36:43 TOWER emhttpd: read SMART /dev/sde
May  8 21:52:44 TOWER emhttpd: spinning down /dev/sde
May  8 21:52:49 TOWER emhttpd: read SMART /dev/sde
May  8 22:08:51 TOWER emhttpd: spinning down /dev/sde
May  8 22:08:56 TOWER emhttpd: read SMART /dev/sde
May  8 22:24:57 TOWER emhttpd: spinning down /dev/sde
May  8 22:25:03 TOWER emhttpd: read SMART /dev/sde
May  8 22:31:14 TOWER root: Delaying execution of fix common problems scan for 10 minutes
May  8 22:31:14 TOWER unassigned.devices: Mounting 'Auto Mount' Devices...
May  8 22:31:14 TOWER emhttpd: Starting services...
May  8 22:31:14 TOWER emhttpd: shcmd (75): /etc/rc.d/rc.samba restart
May  8 22:31:14 TOWER wsdd2[9158]: 'Terminated' signal received.
May  8 22:31:14 TOWER nmbd[9134]: [2024/05/08 22:31:14.344902,  0] ../../source3/nmbd/nmbd.c:59(terminate)
May  8 22:31:14 TOWER nmbd[9134]:   Got SIGTERM: going down...
May  8 22:31:14 TOWER winbindd[9172]: [2024/05/08 22:31:14.344934,  0] ../../source3/winbindd/winbindd_dual.c:1950(winbindd_sig_term_handler)
May  8 22:31:14 TOWER winbindd[9170]: [2024/05/08 22:31:14.344938,  0] ../../source3/winbindd/winbindd_dual.c:1950(winbindd_sig_term_handler)
May  8 22:31:14 TOWER winbindd[9172]:   Got sig[15] terminate (is_parent=0)
May  8 22:31:14 TOWER winbindd[9170]:   Got sig[15] terminate (is_parent=1)
May  8 22:31:14 TOWER wsdd2[9158]: terminating.

 

On 5/8/2024 at 1:14 AM, seecs2011 said:

The reboots seem to consistently be 10 minutes after a message stating

Server rebooting on its own is almost always a hardware issue, but can be caused by different components, if you've rule out the RAM, next suspects after that would be PSU, board, CPU.

3 minutes ago, JorgeB said:

Server rebooting on its own is almost always a hardware issue, but can be caused by different components, if you've rule out the RAM, next suspects after that would be PSU, board, CPU.

One comparatively common cause you have left off the list is 'thermal' if cooling is not working properly.

 

7 hours ago, seecs2011 said:

Adding anonymized diagnostic data here for review, sent a ticket request directly to unraid

tower-diagnostics-20240508-2253.zip 169.42 kB · 1 download

 

Looks like your VPN details may be in plain text (I've obfuscated it below), wonder if @limetech is going to get that sorted in the next release. For now, you should remove it from your post

 

@JorgeB

 

 

root       566  0.0  0.0  23824 12724 ?        S    22:53   0:00          |   \_ /usr/bin/openvpn --reneg-sec 0 --mute-replay-warnings --auth-nocache --setenv VPN_PROV protonvpn --setenv VPN_CLIENT openvpn --setenv DEBUG false --setenv VPN_DEVICE_TYPE tun0 --setenv VPN_ENABLED yes --setenv VPN_REMOTE_SERVER us-chicago.privacy.network --setenv APPLICATION qbittorrent --script-security 2 --writepid /root/openvpn.pid --remap-usr1 SIGHUP --log-append /dev/stdout --pull-filter ignore up --pull-filter ignore down --pull-filter ignore route-ipv6 --pull-filter ignore ifconfig-ipv6 --pull-filter ignore tun-ipv6 --pull-filter ignore dhcp-option DNS6 --pull-filter ignore persist-tun --pull-filter ignore reneg-sec --up /root/openvpnup.sh --up-delay --up-restart --keepalive 10 60 --setenv STRICT_PORT_FORWARD yes --setenv VPN_USER YWQpRx******** --setenv VPN_PASS SBKmfs********** --down /root/openvpndown.sh --disable-occ --auth-user-pass credentials.conf --cd /config/openvpn --config /config/openvpn/us_chicago.ovpn --remote ******* 1198 udp --remote ******* 1198 udp --remote ****** 1198 udp --remote-random

 

2 hours ago, Michael_P said:

 

Looks like your VPN details may be in plain text (I've obfuscated it below)

 

Thanks for catching that, probably not a huge deal given its use-case, but I deleted that part of the post to be safe.

4 hours ago, itimpi said:

One comparatively common cause you have left off the list is 'thermal' if cooling is not working properly.

 

Since its an easy test, I may start with thermal re-application. I have a PSU tester and didn't see any red flags there.

 

Of note, I had the dashboard pulled up last time it crashed and was able to see that all the reported information there looked normal.

Interesting troubleshooting find - system seems to stay up as long as a parity check is happening. If I cancel the check, I get a crash withing roughly the next hour--2hours. If I leave it running, crashes occur in the same window after finishing (hence the daily-ish crashes - my parity check takes roughly 22-25 hours. Combined with the disk on sde being a common thread, I'm wondering if that disk is going bad contrary to smart data reports or if somehow idle states are creeping back in as an issue.

Try disabling C-States

6 hours ago, JorgeB said:

Try disabling C-States

I'm planning to check this weekend on that - I had disabled them before so not sure how they would have gotten re-enabled unless bios auto-updated (which I didn't realize could happen until my gaming pc did that a few months ago). But the behavior is certainly pointing that direction (again).