February 12, 20242 yr I'm not sure what's going on. Server was stable for about a month and now it crashes anywhere between immediate and a several hours. Is there anything obvious in the log files?log_170681484.log Edited February 12, 20242 yr by 808mp5 mispelled
February 12, 20242 yr Author 18 minutes ago, trurl said: Attach Diagnostics to your NEXT post in this thread. thank you for the reply/advice meshifyunraid-diagnostics-20240211-1937.zip
February 12, 20242 yr Author Lasted about 8hrs then crashedlog_203493125.log Darn it and the log is empty.. i'll try again Edited February 12, 20242 yr by 808mp5
February 12, 20242 yr Author ok this one has some stuff lasted about an hour before it crashed log_233322968.log log_236763968.log Edited February 12, 20242 yr by 808mp5
February 12, 20242 yr Community Expert <13>Feb 12 02:42:05 MeshifyUnraid root: Fix Common Problems: Warning: Complex bonding mode on eth0 ** Ignored <13>Feb 12 02:42:05 MeshifyUnraid root: Fix Common Problems: Warning: Jumbo Frames detected on eth0 ** Ignored <13>Feb 12 02:42:05 MeshifyUnraid root: Fix Common Problems: Warning: Jumbo Frames detected on eth2 ** Ignored <13>Feb 12 02:42:05 MeshifyUnraid root: Fix Common Problems: Warning: Jumbo Frames detected on eth3 ** Ignored <13>Feb 12 02:42:05 MeshifyUnraid root: Fix Common Problems: Warning: Jumbo Frames detected on eth5 ** Ignored <13>Feb 12 02:42:05 MeshifyUnraid root: Fix Common Problems: Warning: Jumbo Frames detected on eth6 ** Ignored <13>Feb 12 02:42:05 MeshifyUnraid root: Fix Common Problems: Warning: Jumbo Frames detected on eth7 ** Ignored <13>Feb 12 02:42:05 MeshifyUnraid root: Fix Common Problems: Warning: Jumbo Frames detected on eth8 ** Ignored Looks like you have several things mentioned in 6.12.6 Release Notes: https://docs.unraid.net/unraid-os/release-notes/6.12.6/
February 13, 20242 yr Author 12 hours ago, trurl said: <13>Feb 12 02:42:05 MeshifyUnraid root: Fix Common Problems: Warning: Complex bonding mode on eth0 ** Ignored <13>Feb 12 02:42:05 MeshifyUnraid root: Fix Common Problems: Warning: Jumbo Frames detected on eth0 ** Ignored <13>Feb 12 02:42:05 MeshifyUnraid root: Fix Common Problems: Warning: Jumbo Frames detected on eth2 ** Ignored <13>Feb 12 02:42:05 MeshifyUnraid root: Fix Common Problems: Warning: Jumbo Frames detected on eth3 ** Ignored <13>Feb 12 02:42:05 MeshifyUnraid root: Fix Common Problems: Warning: Jumbo Frames detected on eth5 ** Ignored <13>Feb 12 02:42:05 MeshifyUnraid root: Fix Common Problems: Warning: Jumbo Frames detected on eth6 ** Ignored <13>Feb 12 02:42:05 MeshifyUnraid root: Fix Common Problems: Warning: Jumbo Frames detected on eth7 ** Ignored <13>Feb 12 02:42:05 MeshifyUnraid root: Fix Common Problems: Warning: Jumbo Frames detected on eth8 ** Ignored Looks like you have several things mentioned in 6.12.6 Release Notes: https://docs.unraid.net/unraid-os/release-notes/6.12.6/ thank you for the reference... I haven't had any network issues utilizing Jumbo Frames though. And for the complex bonding my switch is setup properly for LACP. I think i may have found the reason for the crashing maybe... I'm leaning towards not having the dGPU(s) not binded via IMMOU. I have binded them and so far this is the longest the server has been up (since the crashing started, I must have changed something at some point). I also didn't start PMS (not sure if one of the crashes was because of Plex docker but noticed in the logs PMS LIBupdater was there. Gonna see how long it lasts.. if the server doesn't crash then i'll try turning PLEX back on
February 13, 20242 yr Author just had another crash.. I'll try disabling Jumbo Frames... but still optimistic that is the issue log_249118015.log
February 15, 20242 yr Author set MTU to 1500 and problem still persists... i'm about ready to abandon ship i am at a loss
February 15, 20242 yr Community Expert <1>Feb 12 21:49:33 MeshifyUnraid kernel: BUG: unable to handle page fault for address: ffff8881b5a489c0 <1>Feb 12 21:49:33 MeshifyUnraid kernel: #PF: supervisor instruction fetch in kernel mode <1>Feb 12 21:49:33 MeshifyUnraid kernel: #PF: error_code(0x0011) - permissions violation <6>Feb 12 21:49:33 MeshifyUnraid kernel: PGD 4c01067 P4D 4c01067 PUD 1cbb41063 PMD 80000001b5a001e3 <4>Feb 12 21:49:33 MeshifyUnraid kernel: Oops: 0011 [#1] PREEMPT SMP NOPTI <2>Feb 12 21:49:33 MeshifyUnraid kernel: kernel tried to execute NX-protected page - exploit attempt? (uid: 0) <1>Feb 12 21:49:33 MeshifyUnraid kernel: BUG: unable to handle page fault for address: ffff8881b5a489c0 <1>Feb 12 21:49:33 MeshifyUnraid kernel: #PF: supervisor instruction fetch in kernel mode <1>Feb 12 21:49:33 MeshifyUnraid kernel: #PF: error_code(0x0011) - permissions violation Have you done memtest recently? The reason I haven't asked this before is because it looked like your RAM was ECC. Is your RAM ECC, and does the rest of the system actually support ECC?
February 15, 20242 yr Author 4 hours ago, trurl said: <1>Feb 12 21:49:33 MeshifyUnraid kernel: BUG: unable to handle page fault for address: ffff8881b5a489c0 <1>Feb 12 21:49:33 MeshifyUnraid kernel: #PF: supervisor instruction fetch in kernel mode <1>Feb 12 21:49:33 MeshifyUnraid kernel: #PF: error_code(0x0011) - permissions violation <6>Feb 12 21:49:33 MeshifyUnraid kernel: PGD 4c01067 P4D 4c01067 PUD 1cbb41063 PMD 80000001b5a001e3 <4>Feb 12 21:49:33 MeshifyUnraid kernel: Oops: 0011 [#1] PREEMPT SMP NOPTI <2>Feb 12 21:49:33 MeshifyUnraid kernel: kernel tried to execute NX-protected page - exploit attempt? (uid: 0) <1>Feb 12 21:49:33 MeshifyUnraid kernel: BUG: unable to handle page fault for address: ffff8881b5a489c0 <1>Feb 12 21:49:33 MeshifyUnraid kernel: #PF: supervisor instruction fetch in kernel mode <1>Feb 12 21:49:33 MeshifyUnraid kernel: #PF: error_code(0x0011) - permissions violation Have you done memtest recently? The reason I haven't asked this before is because it looked like your RAM was ECC. Is your RAM ECC, and does the rest of the system actually support ECC? When i initially setup the server i did about 14 passes of Memtest with no errors. Board is a W680-ACE and supports ECC, memory was chosen based off of the QVL Prior to the constant crashing server was running perfectly stable for 28days up time, no changes made besides periodic updates to docker containers, plugin, etc. I wasn't even monitoring the server anymore. Only reason i found out it crashed is when i a friend asked if Plex was down as i have/had three instances (docker, 2 VMs) of it running on Unraid. Still going through the process of elimination of turning off VMs and Docker container that i'm suspecting of crashing the system. Still haven't nailed it down yet. First i disabled SWAG (duckDNS) since i'm getting a lot of blocked external attacks on the WAN (no fix) Second disabled Plex docker since one of the logs the last fault mentioned a PMS LIBUpdater error (no fix) Third disabled all instances of Tautulli dockers since of the faults mentioned SYN Flooding on one of the ports (not exposed) Fourth and current test. Disabled one of my VMs running Plex as the last fault mentioned something about a qemu error and crashed when i tried to play something other things i've tried is setting all network interfaces back to MTU1500. Although network is configured properly for MTU9000 (no fix) changed bios c-state all to disabled (no fix) changed CPU setting in bios and disabled any performance features (no fix) Ran SMART test on every drive, all passed with no error Ran Scrub on all cache pools, no errors detected/repaired Rebuilt docker image, (no fix) Disabled user defined schedules (zfs snapshots for all datasets). (no fix) there's probably more i've attempted but at this point it's all jammed up. ran a memtest about two crashes ago but only a single pass with no error At the moment i'm moving all my data to a spare DS1821+ as my next move will probably just be to nuke it all and start with a new config and reinstall/reconfigure everything (of course after doing a proper memtest.. or i'll try a proper memtest prior to nuking it. will keep posting my progress up until i nuke the system if no solution if discovered before that a little more system information (all was stable prior to troubleshooting) Docker SWAG - pointing to Overseer (shutdown for troubleshooting atm) Overseer duckDNS (shutdown for troubleshooting atm) Tdarr (server only, nodes external on three different computers/VMs. transcoded over 80k files successfully) Tautulli 3x instances (shutdown for troubleshooting atm) Heimdall Jellyfin (shutdown for troubleshooting atm) Plex (shutdown for troubleshooting atm) (Removed from docker) All *arr apps and miscellaneous stuff i tested out VMs (no cpu cores/threads overlapping) Windows 11 - Plex, tdarr node (shutdown for troubleshooting atm) Windows 11 - Plex, tdarr node Windows 11 - *arr apps only pfsense - intel nics passthrough, network behind main network via blackhole (shutdown VMs) - multiple linux distros for testing use Network Intel X540 dual 10gbe (bonded LACP) host, bridge, vlan tagged Onboard dual 2.5gbe (passthrough to pfsense, VLANd) Realtek dual 2.5gbe (port down not used, will pull out next time i open the case) *side note, removed network cards/networks still show up in "fix common problems" due to having set MTU to 9000 Main System Components Mainboard - Asus WS W680-ACE CPU - Intel 14900K stock no overclock Memory - 128GB ECC Kingston 4x32GB set to rated stock speed/voltage GPU 1 - Intel ARC A40 Pro GPU 2 - Intel ARC A380 HBA - LSI 9211-8i P20 IT Mode Storage Unraid array - 3x Seagate X22 Exos 20TB, 1x Toshiba x300 8TB, single parity Appdata_cache - 2x Samsung 990 Pro 2TB in zfs mirror (all docker and 3 vm) Domaim_cache - 1x Samsung 990 Pro 4TB in zfs (2 vm) Download cache - 2x Intel 480GB DC SSD zfs raid0 (tdarr transcode directory) Raid0_zfs - 2x Samsung 870 Evo 250GB zfs raid0 (used for data transfer external sources and unraid) Transfer_cache - 5x Samsung 870 Evo zfs1 (used for all *arr app/sabnzb data transfers) Edited February 15, 20242 yr by 808mp5
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.