When i initially setup the server i did about 14 passes of Memtest with no errors.
Board is a W680-ACE and supports ECC, memory was chosen based off of the QVL
Prior to the constant crashing server was running perfectly stable for 28days up time, no changes made besides periodic updates to docker containers, plugin, etc. I wasn't even monitoring the server anymore. Only reason i found out it crashed is when i a friend asked if Plex was down as i have/had three instances (docker, 2 VMs) of it running on Unraid.
Still going through the process of elimination of turning off VMs and Docker container that i'm suspecting of crashing the system. Still haven't nailed it down yet.
First i disabled SWAG (duckDNS) since i'm getting a lot of blocked external attacks on the WAN (no fix)
Second disabled Plex docker since one of the logs the last fault mentioned a PMS LIBUpdater error (no fix)
Third disabled all instances of Tautulli dockers since of the faults mentioned SYN Flooding on one of the ports (not exposed)
Fourth and current test. Disabled one of my VMs running Plex as the last fault mentioned something about a qemu error and crashed when i tried to play something
other things i've tried is setting all network interfaces back to MTU1500. Although network is configured properly for MTU9000 (no fix)
changed bios c-state all to disabled (no fix)
changed CPU setting in bios and disabled any performance features (no fix)
Ran SMART test on every drive, all passed with no error
Ran Scrub on all cache pools, no errors detected/repaired
Rebuilt docker image, (no fix)
Disabled user defined schedules (zfs snapshots for all datasets). (no fix)
there's probably more i've attempted but at this point it's all jammed up.
ran a memtest about two crashes ago but only a single pass with no error
At the moment i'm moving all my data to a spare DS1821+ as my next move will probably just be to nuke it all and start with a new config and reinstall/reconfigure everything (of course after doing a proper memtest.. or i'll try a proper memtest prior to nuking it.
will keep posting my progress up until i nuke the system if no solution if discovered before that
a little more system information (all was stable prior to troubleshooting)
Docker
SWAG - pointing to Overseer (shutdown for troubleshooting atm)
Overseer
duckDNS (shutdown for troubleshooting atm)
Tdarr (server only, nodes external on three different computers/VMs. transcoded over 80k files successfully)
Tautulli 3x instances (shutdown for troubleshooting atm)
Heimdall
Jellyfin (shutdown for troubleshooting atm)
Plex (shutdown for troubleshooting atm)
(Removed from docker) All *arr apps and miscellaneous stuff i tested out
VMs (no cpu cores/threads overlapping)
Windows 11 - Plex, tdarr node (shutdown for troubleshooting atm)
Windows 11 - Plex, tdarr node
Windows 11 - *arr apps only
pfsense - intel nics passthrough, network behind main network via blackhole
(shutdown VMs) - multiple linux distros for testing use
Network
Intel X540 dual 10gbe (bonded LACP) host, bridge, vlan tagged
Onboard dual 2.5gbe (passthrough to pfsense, VLANd)
Realtek dual 2.5gbe (port down not used, will pull out next time i open the case)
*side note, removed network cards/networks still show up in "fix common problems" due to having set MTU to 9000
Main System Components
Mainboard - Asus WS W680-ACE
CPU - Intel 14900K stock no overclock
Memory - 128GB ECC Kingston 4x32GB set to rated stock speed/voltage
GPU 1 - Intel ARC A40 Pro
GPU 2 - Intel ARC A380
HBA - LSI 9211-8i P20 IT Mode
Storage
Unraid array - 3x Seagate X22 Exos 20TB, 1x Toshiba x300 8TB, single parity
Appdata_cache - 2x Samsung 990 Pro 2TB in zfs mirror (all docker and 3 vm)
Domaim_cache - 1x Samsung 990 Pro 4TB in zfs (2 vm)
Download cache - 2x Intel 480GB DC SSD zfs raid0 (tdarr transcode directory)
Raid0_zfs - 2x Samsung 870 Evo 250GB zfs raid0 (used for data transfer external sources and unraid)
Transfer_cache - 5x Samsung 870 Evo zfs1 (used for all *arr app/sabnzb data transfers)