Hey guys!

 

I'm trying to figure out what's wrong with my server. I use this server primarily as a Plex server.  When it crashes, I can access the WebGui, but it hangs when I try to pull the logs from the Diagnostics tab, and I can't pull em before I have to manually power off the server and power it back on.

 

Then I set up syslog on the problem server, but the problem I've had with that is the server rewrites over the logs I saved on the logs share I made, when I power the server back on.

 

I've had a few unclean shutdowns, and I've tried to rebuild parity, but it crashes before I can rebuild the parity. I've tried to get logs, but they get written back over on my logs share I set up on the problem server (syslog).  So I don't have logs to post.  

 

I did take a picture of the panic message on the screen before I powered it down via an unclean shutdown, and am attaching that as a starting point.

 

I am running 4 x 32gb kingston sticks of ddr5 ECC ram that i've ran memtest on, and it's passed with zero errors.

 

System specs

 

Unraid ver. 6.12.8

Pro WS W680-ACE Intel W680 LGA 1700 ATX motherboard

Intel® Core™ i7 processor 14700K

Kingston Server Premier 4 x 32GB 4800MT/s DDR5 ECC CL40 DIMM 2Rx8 Hynix M Server Memory - KSM48E40BD8KM-32HM (128GB total)

24 Various hard drives

1600w T2 EVGA PSU

2 2tb SN700 Red NVME mirrored cache drives

10GB intel NIC

LSI 9207

Intel Raid Expander res2sv240

RROYJJ 4U Rackmount Server Case Chassis with 24 HD bays

 

Is there a way to write the logs via syslog so they don't get overwritten on the server when it crashes and I reboot it?

 

How would I set the logs to write to a remote windows computer so I can look at them there, instead of having them be overwritten upon reboot of the problem server?  But then again, I'm not exactly sure what some of the log lingo means, so if any of you guys and gals know more, I'd appreciate the help!

 

Attached is the memtest diagnostics, and the panic message I got off the screen before I powered it off by holding the button down, and turning it back on. 

 

Any and all help is appreciated!  Thanks everyone.

Edited October 17, 20241 yr by investing-tote3683
Editing title for now that more testing was done.

I think i may be onto something.

 

I saw i was running an older version of unraid (Unraid ver. 6.12.8), and I tried to update to the latest version.

 

Attached is the error I got when I tried to update to the latest 6.12.13

 

It appears my usb flash drive is failing.

 

Luckily I was able to backup the flash drive, so i will be moving over to a new one.  Will post an update after I do this!  Thanks everyone!

 

 

 

 

Edited October 14, 20241 yr by investing-tote3683

A failing flash drive can be the source of the craziest symptoms.  Probably lucky you are finding it this quickly.

I transferred everything over to the new flash drive.

 

I also was able to update unraid now to the latest version!  I'm pretty sure the problem has been fixed, but I will report back tomorrow and let you all know if I have any recurring issues.  

 

Thanks guys and gals.

Ok, it only lasted an hour, and it crashed again, same issue and same symptoms as all the other crashes.

 

So I guess it wasn't the flash drive, but I'm glad I replaced it!  I did have the WebGui open when it crashed, and I did scroll over to the dashboard, and saw that the cpu was pegged at 100% on many of the cores.

 

I also removed 2 of the 4 32gb ram sticks, maybe to see if that would have any effect.

 

Restarted and am trying again.  Will report back.

 

 

I exported the log share and made it so I could view the syslog on my windows pc, and I was able to grab the syslog diagnostics on the fly as I had the log opened last time it crashed.

 

Not sure if this is any help, but it's attached below.  Thanks!

syslog-192.168.86.27.log

Still getting crashes.

 

I can't get through the Parity sync before the server crashes again.  This leads me to believe that I have a hardware problem.  I have attached diagnostics.

 

I did see these errors in the syslog I attached above.

 

Oct 14 00:57:18 Tower kernel: md: recovery thread: multiple disk errors, sector=547446160
Oct 14 00:57:18 Tower kernel: ------------[ cut here ]------------
Oct 14 00:57:18 Tower kernel: kernel BUG at drivers/md/unraid.c:1617!

 

Another thing is, I was trying to get through the Parity sync with Docker enabled, with all my containers running.

 

I'm have since disabled docker, to see if I can even get through the parity sync.  I also read somewhere about changing maclan to IPvlan, but I'm going to see if I can even get through the parity check with docker disabled, before I proceed further.

 

If the system crashes with Docker disabled, I don't think the issue is with maclan/IPvlan according to some other threads I have read on here.

 

I have an LSI 9207 card, and an Intel RAID Storage Expander RES2SV240.  I'm going to order 1 more of each, to try and swap out, and also to have as a backup.  Will report back findings.

 

Thanks for following along!

tower-diagnostics-20241015-0753.zip

25 minutes ago, investing-tote3683 said:

Oct 14 00:57:18 Tower kernel: kernel BUG at drivers/md/unraid.c:1617!

This means the Unraid driver is crashing, and that's almost always a hardware issue.

Thanks for that!

 

I'm going to try swapping out the LSI 9207 and the Intel RAID Storage Expander RES2SV240 and see if that remedies the issue.

 

The thing that makes me believe that it could be one of these cards, is that WHEN the server was running without the crashes last month, I still wasn't able to complete a parity check after an unclean shutdown i had, as it always slowed down to Kbs/sec durity the parity check, and would never actually finish (with 100's of days left to completion due to the painstakingly slow speeds of the parity check).

 

And during this latest parity sync, I've noticed sporadic slowdowns while building parity (even with docker disabled) to the point where it goes from 100mbs/sec to 3-4mbs/sec where it will stay for awhile.  It has done that just now, but it has climbed back to 100mbs/sec.

 

I guess it could also be the backplate SAS adapters on my RROYJJ 4U Rackmount 24 Bay Server Case as well, but I don't know how to get replacement parts for that to swap out, without having to buy a whole new case for $500.

 

Thanks for the replies!  I'll let you know what I find out!

With Docker disabled, it has been crash free so far *knock on wood*.

 

The parity sync is running.  22TB parity sync, at 100mb/s is going to take 2-3 days to complete if the speeds stay the same.

 

Uptime is almost at 8 hours, and it hasn't crashed.  I still got quite some time before the Parity sync is complete.

 

I made the mistake of removing parity drives and switching them around, reassigning them, so I have to do this parity sync before I proceed further so the array can be protected.

So far so good guys and gals.

 

Parity sync is chugging along with Docker disabled.  No crashes yet.

 

 

Update! Parity sync is still going, no issues there.

 

Parity check will be complete by later tomorrow night.  I'm beginning to think that the 14700k is at fault.  I'm going to replace the LSI controller, and the Intel raid expander, but I have a feeling this is more of a CPU problem, as I've read other threads that have pointed to similar issues with the 13th and 14th gen CPUs.

 

I've also memtested the memory, and it passed.  I know that doesn't necessarily rule out the ram, so I ran the system with 2 sticks instead of 4 a few days ago, and it still crashed, and then i tried the other 2 sticks of ram, and it crashed still.  So I think it's more than likely a CPU issue after reading similar threads.

 

And since I use the CPU as the transcoder and the crashes happen when Plex is running, I'm thinking it's more of a 14700k failure.  No big deal, as I can RMA this one, and I have another LGA 1700 cpu I can throw in and test.

 

But it could be a number of things.  I have a spare mobo too, so I'm going to try numerous things, and get to the bottom of why this is crashing, but try a different CPU before I go down this rabbit hole.

 

I will do this after the parity sync is complete, and test different hardware until I figure out what is going on.  

 

Thanks for following along!

 

 

Edited October 17, 20241 yr by investing-tote3683

Parity sync is complete.  

 

Turning on Docker and going to run Plex and all the other services, and see if I can get it to crash again, now that it's not stuck in some Parity sync/check loop while the containers are going.

 

Will report back if/when it crashes.

Ok,

 

Well, 

I'm not sure exactly what's going on.

 

I didn't replace anything hardware related.  I was going to replace the LSI 9207 and intel raid expander, and maybe swap the cpu, but it's running fine.

 

After the parity sync was allowed to complete with Docker disabled, I re-enabled docker and turned on all my containers, and it has been running solid since.  No crashes as of yet.  Before it would crash every 4-6 hours, but has been running fine since the parity sync completed 24 hours ago.  I will continue to check it over the weekend, and if everything is fine, chalk it up to an issue with the parity check/sync that wouldn't complete cause the containers were putting too much use on the array, and Mover not being able to move files from the cache drives to the array during the parity sync operation.

 

I'm beginning to think that maybe there was an issue after I had an unclean shutdown, and the parity sync wasn't completing due to Mover not being allowed to move files over to the array for the last month or so, with sonarr and radarr still actively grabbing things through SabNZBD.  

 

I'll respond back after the weekend and hopefully give my last update on this issue.

 

Thanks for following along!

Ok, 

 

It crashed again.  Same symptoms as before.

 

I finally decided to replace the CPU.  I just did it after the last crash earlier today and so far so good. The server is up and running again.  Hopefully this fixes the issue.

 

I will report back if the crashes happen again!  Still waiting on the new LSI 9305-24i, and will replace that when that comes in.  But for now, at least I can see if it will crash again after replacing the CPU.  Will report back soon!

Ok guys!  Server has been running strong for almost 2 days with no issues after I replaced the 14700k. 

 

The old 14700k i replaced had about 11 months of use on it, and used primarily to transcode via the iGPU on my plex server. 

 

It did amazing, up until it started crashing my server this past month.

 

But it's kind of alarming it already failed, as utilization has been steady around 20-40% with temps around 40-50 degrees Celsius. 

 

So, the 14700k failure is to blame.  Submitting an RMA request to Intel.  Thanks JorgeB for suggesting a hardware failure, you were right!  Thanks everyone.

Edited October 24, 20241 yr by investing-tote3683