investing-tote3683

Ok guys! Server has been running strong for almost 2 days with no issues after I replaced the 14700k. The old 14700k i replaced had about 11 months of use on it, and used primarily to transcode via the iGPU on my plex server. It did amazing, up until it started crashing my server this past month. But it's kind of alarming it already failed, as utilization has been steady around 20-40% with temps around 40-50 degrees Celsius. So, the 14700k failure is to blame. Submitting an RMA request to Intel. Thanks JorgeB for suggesting a hardware failure, you were right! Thanks everyone.

Ok, It crashed again. Same symptoms as before. I finally decided to replace the CPU. I just did it after the last crash earlier today and so far so good. The server is up and running again. Hopefully this fixes the issue. I will report back if the crashes happen again! Still waiting on the new LSI 9305-24i, and will replace that when that comes in. But for now, at least I can see if it will crash again after replacing the CPU. Will report back soon!

Ok, Well, I'm not sure exactly what's going on. I didn't replace anything hardware related. I was going to replace the LSI 9207 and intel raid expander, and maybe swap the cpu, but it's running fine. After the parity sync was allowed to complete with Docker disabled, I re-enabled docker and turned on all my containers, and it has been running solid since. No crashes as of yet. Before it would crash every 4-6 hours, but has been running fine since the parity sync completed 24 hours ago. I will continue to check it over the weekend, and if everything is fine, chalk it up to an issue with the parity check/sync that wouldn't complete cause the containers were putting too much use on the array, and Mover not being able to move files from the cache drives to the array during the parity sync operation. I'm beginning to think that maybe there was an issue after I had an unclean shutdown, and the parity sync wasn't completing due to Mover not being allowed to move files over to the array for the last month or so, with sonarr and radarr still actively grabbing things through SabNZBD. I'll respond back after the weekend and hopefully give my last update on this issue. Thanks for following along!

Parity sync is complete. Turning on Docker and going to run Plex and all the other services, and see if I can get it to crash again, now that it's not stuck in some Parity sync/check loop while the containers are going. Will report back if/when it crashes.

Update! Parity sync is still going, no issues there. Parity check will be complete by later tomorrow night. I'm beginning to think that the 14700k is at fault. I'm going to replace the LSI controller, and the Intel raid expander, but I have a feeling this is more of a CPU problem, as I've read other threads that have pointed to similar issues with the 13th and 14th gen CPUs. I've also memtested the memory, and it passed. I know that doesn't necessarily rule out the ram, so I ran the system with 2 sticks instead of 4 a few days ago, and it still crashed, and then i tried the other 2 sticks of ram, and it crashed still. So I think it's more than likely a CPU issue after reading similar threads. And since I use the CPU as the transcoder and the crashes happen when Plex is running, I'm thinking it's more of a 14700k failure. No big deal, as I can RMA this one, and I have another LGA 1700 cpu I can throw in and test. But it could be a number of things. I have a spare mobo too, so I'm going to try numerous things, and get to the bottom of why this is crashing, but try a different CPU before I go down this rabbit hole. I will do this after the parity sync is complete, and test different hardware until I figure out what is going on. Thanks for following along!

Thanks JorgeB! I figured it would be ok, I just wanted to make sure those were the correct cables for the LSI 9305.

So far so good guys and gals. Parity sync is chugging along with Docker disabled. No crashes yet.

Hey guys and gals! I'm looking to update my system, and was wanting to limit the amount of Raid cards/Expanders I have in my system to limit possible hardware failures. My 24 hard drives vary from 16tb to 22 tb WD Red Pros. I'm using a RROYJ 24 Bay rackmount chassis, and it has a total of 6 backplanes, with 6 total SAS SFF-8087 ports. Currently I have a LSI 9207 8i, and an Intel Raid Expander res2sv240. So right now, I have 2 SFF-8087 cables going from the LSI 9207 (one going to one of the SAS backplanes, and another cable going to the Intel Raid Expander card). Then I have 5 cables going from the Intel Raid Expander, to the other SAS backplanes. I want to know if there is a single card I can use that will work right out of the box, that has 6 ports to connect to all 6 backplanes of the HD bays without needing to use an Intel Raid Expander card. I was looking at possibly getting this 9305-24i 12gbps IT mode card. This uses the SFF 8643 cable I think? I did see they made an SFF 8643 to SFF 8087 cable. Will this card and type of cord work alright? If so, I can just get 6 of these below cables, plug everything in and be good to go correct? I appreciate all the help!

With Docker disabled, it has been crash free so far *knock on wood*. The parity sync is running. 22TB parity sync, at 100mb/s is going to take 2-3 days to complete if the speeds stay the same. Uptime is almost at 8 hours, and it hasn't crashed. I still got quite some time before the Parity sync is complete. I made the mistake of removing parity drives and switching them around, reassigning them, so I have to do this parity sync before I proceed further so the array can be protected.

Thanks for that! I'm going to try swapping out the LSI 9207 and the Intel RAID Storage Expander RES2SV240 and see if that remedies the issue. The thing that makes me believe that it could be one of these cards, is that WHEN the server was running without the crashes last month, I still wasn't able to complete a parity check after an unclean shutdown i had, as it always slowed down to Kbs/sec durity the parity check, and would never actually finish (with 100's of days left to completion due to the painstakingly slow speeds of the parity check). And during this latest parity sync, I've noticed sporadic slowdowns while building parity (even with docker disabled) to the point where it goes from 100mbs/sec to 3-4mbs/sec where it will stay for awhile. It has done that just now, but it has climbed back to 100mbs/sec. I guess it could also be the backplate SAS adapters on my RROYJJ 4U Rackmount 24 Bay Server Case as well, but I don't know how to get replacement parts for that to swap out, without having to buy a whole new case for $500. Thanks for the replies! I'll let you know what I find out!

Still getting crashes. I can't get through the Parity sync before the server crashes again. This leads me to believe that I have a hardware problem. I have attached diagnostics. I did see these errors in the syslog I attached above. Oct 14 00:57:18 Tower kernel: md: recovery thread: multiple disk errors, sector=547446160 Oct 14 00:57:18 Tower kernel: ------------[ cut here ]------------ Oct 14 00:57:18 Tower kernel: kernel BUG at drivers/md/unraid.c:1617! Another thing is, I was trying to get through the Parity sync with Docker enabled, with all my containers running. I'm have since disabled docker, to see if I can even get through the parity sync. I also read somewhere about changing maclan to IPvlan, but I'm going to see if I can even get through the parity check with docker disabled, before I proceed further. If the system crashes with Docker disabled, I don't think the issue is with maclan/IPvlan according to some other threads I have read on here. I have an LSI 9207 card, and an Intel RAID Storage Expander RES2SV240. I'm going to order 1 more of each, to try and swap out, and also to have as a backup. Will report back findings. Thanks for following along! tower-diagnostics-20241015-0753.zip

I exported the log share and made it so I could view the syslog on my windows pc, and I was able to grab the syslog diagnostics on the fly as I had the log opened last time it crashed. Not sure if this is any help, but it's attached below. Thanks! syslog-192.168.86.27.log

Ok, it only lasted an hour, and it crashed again, same issue and same symptoms as all the other crashes. So I guess it wasn't the flash drive, but I'm glad I replaced it! I did have the WebGui open when it crashed, and I did scroll over to the dashboard, and saw that the cpu was pegged at 100% on many of the cores. I also removed 2 of the 4 32gb ram sticks, maybe to see if that would have any effect. Restarted and am trying again. Will report back.

I transferred everything over to the new flash drive. I also was able to update unraid now to the latest version! I'm pretty sure the problem has been fixed, but I will report back tomorrow and let you all know if I have any recurring issues. Thanks guys and gals.

I think i may be onto something. I saw i was running an older version of unraid (Unraid ver. 6.12.8), and I tried to update to the latest version. Attached is the error I got when I tried to update to the latest 6.12.13 It appears my usb flash drive is failing. Luckily I was able to backup the flash drive, so i will be moving over to a new one. Will post an update after I do this! Thanks everyone!

Hey guys! I'm trying to figure out what's wrong with my server. I use this server primarily as a Plex server. When it crashes, I can access the WebGui, but it hangs when I try to pull the logs from the Diagnostics tab, and I can't pull em before I have to manually power off the server and power it back on. Then I set up syslog on the problem server, but the problem I've had with that is the server rewrites over the logs I saved on the logs share I made, when I power the server back on. I've had a few unclean shutdowns, and I've tried to rebuild parity, but it crashes before I can rebuild the parity. I've tried to get logs, but they get written back over on my logs share I set up on the problem server (syslog). So I don't have logs to post. I did take a picture of the panic message on the screen before I powered it down via an unclean shutdown, and am attaching that as a starting point. I am running 4 x 32gb kingston sticks of ddr5 ECC ram that i've ran memtest on, and it's passed with zero errors. System specs Unraid ver. 6.12.8 Pro WS W680-ACE Intel W680 LGA 1700 ATX motherboard Intel® Core™ i7 processor 14700K Kingston Server Premier 4 x 32GB 4800MT/s DDR5 ECC CL40 DIMM 2Rx8 Hynix M Server Memory - KSM48E40BD8KM-32HM (128GB total) 24 Various hard drives 1600w T2 EVGA PSU 2 2tb SN700 Red NVME mirrored cache drives 10GB intel NIC LSI 9207 Intel Raid Expander res2sv240 RROYJJ 4U Rackmount Server Case Chassis with 24 HD bays Is there a way to write the logs via syslog so they don't get overwritten on the server when it crashes and I reboot it? How would I set the logs to write to a remote windows computer so I can look at them there, instead of having them be overwritten upon reboot of the problem server? But then again, I'm not exactly sure what some of the log lingo means, so if any of you guys and gals know more, I'd appreciate the help! Attached is the memtest diagnostics, and the panic message I got off the screen before I powered it off by holding the button down, and turning it back on. Any and all help is appreciated! Thanks everyone.

Ok, well it was less than a couple weeks. It happened faster this time around. Here are the logs I pulled after it happened, and before I powered down the server. Same thing happened as before. Cache Drive disappeared again, and I had to power off the server completely to get it to show back up again. Thanks for all your help. syslog.txt

Ok, thank you. I will remember to post the diagnostics when this happens again! See ya in a couple weeks lol.

Hey guys and gals, Just wondering what would be the first thing to look at. I've been dealing with this issue on Unraid for a few months now. About every 2-3 weeks, Plex and all my other containers will stop working, and in my docker tab, everything is stopped. When I try to manually Restart the containers, I'm met with a 403 error. I stop the array, and then when my disks refresh and populate, I notice that my 2TB Samsung 980 PRO Cache drive is now missing (why the 403 error is happening no doubt). I do a server reboot, and the cache drive is still missing. The only thing that gets it to show back up is a complete Power Cycle of the server. I turn it off, and turn it back on, and the Cache Drive is recognized again, and I can start the array and all my Containers work as they should. I would really like to get to the bottom of why the server is doing this, so any help would be appreciated. I'm just not sure what to try first. I didn't think to grab the diagnostics after this happened, but I did grab the diagnostics after the first reboot, when I noticed the cache drive was still missing, and the current diagnostics on startup after I completely Power Cycled the server. File system on all my Hard drives are xfs, my cache drive is setup as btrfs. Specs: Asus Pro WS W680-ACE Motherboard Intel 14700k 128GB DDR5 Kingston ECC Memory 1600 Watt EVGA Titanium PSU 2TB Samsung 980 Pro Cache Drive 24 WD Red Hard drives of various sizes Containers: Plex (linuxserver) Radarr Sonarr Overseerr Binhex-Sabnzbd Binhex-Qbittorrentvpn Tautulli I did increase my docker img size to 50gb, to see if that would solve the issue, but that didn't seem to work. That's all that I have tried so far. Any help is appreciated. Just trying to figure out where to look first. tower-diagnostics-20240225-1659 (2).zip

That was it! Thank you so much. I had old parameters set to use Intel's iGPU in the advanced view. I forgot about that. Thanks again. I deleted the parameters and plex started no problem.

Hey guys and gals. My server went down last night. Something happened with the Asus WS W680 Pro and 13900k. It was off when I went to check on it. I turned it back on, and it just shut off and on 3 times. Figured it was a power supply issue, so I replaced that and that didn't fix the issue. So i figured I would order another Asus mobo, and a new 14700k cpu to try out, while I diagnose this hardware a bit more and possibly RMA it. I'm thinking the motherboard may have went out. To get the server back up in the mean time, I installed the old hardware back in it (an AMD 5950x build). The server fired right up and all the drives showed up, and I'm going through a parity rebuild due to the unclean shutdown. I tried to start the plex container now, and I get a Server Execution error, Server error message. All my other dockers work just fine, it's just the plex one. I know the AMD 5950x doesn't have an iGPU and isn't setup for quicksync, but that's what I used before i made the switch to intel, and that's the replacement parts I had available to get the server back up and running in the meantime. I'm guessing the plex container is still thinking it want's to use the iGPU for quicksync, but it's no longer there? I uninstalled the Intel Top plugin, and I thought that would be enough to have plex start up, but that doesn't seem to work either. I'm using linuxservers version of Plex media server. Any ideas? Thanks!