Unraid 6.x Random problems, ended up being old SSD's? Hope this helps someone else.

Mods, don't know if this belongs here since its not a current issue. I just wanted to give others examples of random problems and a possible solution. If it helps you, please go through my past diagnostics for my other support threads/tickets, as I believe the solution was found below, may help you diagnose future issues with other systems?
To be clear, Unraid has largely been perfect for the past 2 months solid now. I still have a weekly reboot setup, may try to get monthly uptime. This is fine for now. Its now always ready to go when I need to access server.

Its been a while since Ive been having constant Unraid problems. Ive opened a couple of threads, done a bunch of research and the problems kept happening. Terrible server uptime/reliability since Dec 2022 until May 2024.

I now believe that I had a couple older SSD's (OCZ Vertex 3 120gb is oldest, also had a Samsung 500gb, and a 64gb one from a steam deck) that may have just been throwing weird edge case errors that I was never able to track down.
I have spent thousands of dollars on replacing hardware (cpu, ram (3 sets), mobo, psu, drives, ect) before just removing a couple of older drives that had 90+% life remaining according to SMART data.

If this helps someone else, I am posting my error log that I started recording a year into the errors in hopes of finding a solution. If you get random errors that happen to match some of mine, take a look at any older hardware?

Below are my notes for personal use and have spelling errors or abbreviations. Figured I would share if it helps, sorry for not being 100% thorough.
 

Quote

crashing daily or every few days

unraid safe mode: left up for 4 days no issues

1/16 regular startup: disable docker:  died in less than 5 hours with only plugins running
hard reset at 10pm, doing parity check, will see how long it lasts, lasted less than 24 hours, at 10pm not responding, but able to use keyboard to reboot.
1/17 removed a bunch of plugins at 11pm . Left ones that were added after server had problems. Drivers/ect. Need to keep user scripts and zfs stuff.
enabled docker but nothing running: on 1/20 11am. No crashes since the above. Only running netdata and nextcloud
1/22 restarted according to user script. Starting pihole/unbound at 3am.
1/23 1:30pm started tdarr which caused crashes in past, drivers were never uninstalled, installed gpu stats plugin
1/24 tdarr all night, no problems. adding back in open files and NUT. 
restart for updates 1/26 11am, added dynamix schedules and system buttons and share floor, system info, file manager, cache
1/29 add a couple more (active streams and system statistics. File integrity doesnt work with non xfs formats, so wont use this going forward.
1/30 crashed after copying a bunch of files. Temps are 46 for ssd, 35-40 for harddrives.  Deleting system statistics plugin. 
2/1 crashed overnight. hard reset. Deleting active streams
2/1 shares dissappeared, i didnt check when I hard reset earlier today.
2/4 shares disappeared. Removing share floor, system info, file manager. Restarted server.
2/4 shares dissappeared again about 12 hrs later. removing cache plugin. Restarted server.
2/7 all dockers dissappeared, yet I can still access pihole. removing dynamix schedules, rebooting (cannot unmount dockers....)
2/8 restarted overnight due to normal scheduling, tried accessing around 3pm to find docker not working again... removing system buttons and restarting, cant unmount dockers, had to give reboot command a second time.
Error seen in log:    nginx: 2024/02/08 02:03:35 [error] 8741#8741: *159 "/usr/local/emhttp/api/index.html" is not found (2: No such file or directory)
Error led me to a post to change "Power Supply Idle Control" in BIOS and set it to "typical current idle"
2/15 complete crash overnight (probably during scheduled restart), had to pull power, video/keyboards not working.
2/16 couldnt even get through parity rebuild, UI crashed/stopped responding. Able to use keyboard/monitor to order a reboot. disabling pihole/unbound as my friend says that recently made his unraid fck up
2/17 Dockers all crashed, not working. Applying the new unraid update for docker security.. Opened log, bunch of segfault errors on different cores?  Tried restarting server, tried unmounting disk shares alot. Got certain docker busy messages, but dockers not working. Had to open terminal and tell it to reboot many times. Finally restarted, finished the update to OS 6.12.8. Restarted 2pm.
2/21 Router updated, probably had issues resolving saved IP's. Internet issues. Restarted unraid to fix and get dockers back up properly.
2/23 Ich777 gameserver docker restarting constantly due to ipv6 issue even though ip6 is disabled. enabled and still doing it. Had to restart unraid. Disabling and reenabling docker didnt work.
2/25 one of the shares not working, hdd spun up, share visible in settings. restarted server. Disabled ipv6 since still getting that error.
2/27 docker service completely dead. Turned off and on, still not working. Restarting server manually.
2/28 same thing, docker dead, had to restart unraid
3/2 1st of month parity check, only got to 78% on 2nd day when shares dissappeared. "make_connection_snum: canonicalize_connect_path failed for service zz, path /mnt/user/zz" Had to reboot server.
3/3 dockers are dead again. Segfault error in log. I know my CPU is fine... Removing pihole and unbound even though they havent been started and are off for months.
3/5 docker service died in middle of updating SWAG. Orphaned that and Maradb even though they appear to be running still.. rebooted unraid, terminal not fully working, had to use reboot command multiple times to no effect. Manually stopped array, now unraid completely dead. Waited 15 minutes, had to pull power plug.
3/9 server went down at 2am, i figured it was restarting, went to bed. Tried loading the webgui at noon, completely crashed, no video/keyboiard, pulled power hard restart.
3/11 server hard crash at 11am? had to pull power, hard reset.
3/12 nextcloud docker orphan'd itself, cannot add container without getting error. Disabled docker in settings and re-enabled.  ich777 game server doing the promiscous mode/disabled port thing again. Restarting server took fckn forever. Setting server to restart every day... This is fckn dumb.
3/14 nightly server restart failed, had to hard reboot server (pulled power)
3/15 nightly server restart failed, checked server 3pm, webui non responsive. able to use keyboard/monitor to log in and reboot.
3/19 dont know, working earlier, checked at 5pm and server dead/unresponsive. No webgui, no keyboard/monitor. Had to pull power, reboot.
3/22 left home for a day, come back to dead server. Keyboard/monitor not working, pull power....
3/23 restarted fine last night. Stopped working between 11am and 1pm... full dead, keyboard/monitor not working. Pulled power... Doing another parity check. Deleted Open Files plugin.
3/24 verified that everything working, still running parity check. Server died during file transfer to cache.... Able to use keyboard to restart. But it hung on reboot, even waiting an hour. Typing reboot again does nothing. Pulled power plug....
3/24 went about 7 hours after that last crash. Crashed again, this time couldnt use keyboard/monitor to attempt restart, pulled power... REMOVING NUT which I really dont want to do. I have small power blips every couple months. I want the UPS to tell unraid to shut down.
3/24 only lasted another 3 hours till full crash, no kb/monitor. Reinstalling NUT since thats not the issue. Still havent finished the gdamn transfer. Reinstalling open files plugin and system buttons. Removing GPU statistics.
3/26 managed a day without a crash!!!!. But full hard crashed today. Few hours after transferring files onto. Went over network good, maybe a mover issue? Monitor/KB didnt work, pulled power.
3/27 server died overnight, or was unable to auto restart.Monitor/KB didnt work, pulled power.
3/30 Server got weird last night (docker page unresponsive) but reboot was this morning so i let it go. This morning, reboot hung, all 5 user scripts are still running after 4-6 hours even though they take seconds to run normally. Instigated reboot command via terminal, not working. NGINX errors in log. "[error] 13320#13320: *1410895 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.5.74, server: , request: "POST /webGui/include/DashboardApps.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock"" Moving all dockers onto array then to another ssd to test if its the ssd.
4/1 Server webgui failed sometime this morning/last night. Dockers were disabled since using mover to get files off SSD. Mover failed to work properly, had to manually move the files. Wasnt able to stop array because mover never quit. Was able to use keyboard/monitor to reboot server.
4/4 Server crashed during auto restart last night, parity was going this morning. Cancelled parity, got the remainder of the dockers switched to the new SSD. Restoring from a backup the one that got screwed up. Rest are working.
4/6 server dead in morning. kb/monitor not work, pulled power
4/9 server handled auto reboot yesterday just fine. Ran Tdarr overnight, appears fine. Server stuck doing mover on a single file (that I can tell) around noon. I told server to reboot at 1:30pm since mover still hasnt finished moving that 300mb file.
4/10 Updated to Unraid 6.12.10, rebooted. Unclean shutdown, running parity check on restart. Cancelled.
4/14 Server worked for a few days. Crashed overnight. KB/monitor didnt work, pulled power
4/17 Server worked for a couple days. Crashed midway through today. accessed it a few hours ago. Rolling spam of segfaults on monitor, couldnt log in and restart. Pulled power. Came back up but couldnt connect via gui. Used keyboard to reboot.
4/19 server working until now. Dockers still work. 11 threads are pegged 100%, memory still at 39%. Pulled up log and found a "kswapd0 Tainted " error, then the server stopped responding. Cant reboot, keyboard not working. After about an hour was able to use KB to log in and reboot. Stuck rebooting, waited 30 minutes then pulled power.
4/21 been working fine past couple days. shutting down server to remove all older ssd's (and steamdeck one), putting sata 1tb back in to use for vm's as a replacement for the ones im taking out. Updating mobo bios to latest version (month old)
4/22 Looked over at spare monitor and looks like the server crashed and rebooted? Or was caught in mobo reboot loop for 12 hrs. Pulled power, reboot.
5/9 havent had any crashes that i know of for a while, reducing reboots to once a week from 3x. Next one sch for 7 days from now.
5/10 installed immich Stopped weekly reboots (forget when)
5/29 installed a bunch of plugins a week or 2 ago. Server WebUI stopped responding (probably a VPN problem). Able to log in with keyboard/monitor to reboot. Starting weekly reboots again.

Edited July 11, 20241 yr by RaidUnnewb
edited sharename out