Hey everyone - I've been having daily crashing for a little over a week straight now that I can't pin down. The system is generally stable with almost no hardware changes preceding the issues. I recently changed out a few drives, but there were no errors and subsequent parity rebuilds went fine. The logs have been almost empty as well, which makes me wonder if there is a hardware issue more than anything else. One other change that I made was to switch to ipvlan in docker which made no difference on the crashing (fwiw I've never had an issue on macvlan so I never changed). I've run memtest and got through 3 passes with no errors.  Despite that I've been pulling ram sticks between crashes to see if I just missed something. The only other hardware I could trade out at this point is the HBA which I've been meaning to do, but again, it's been good for a couple years now so I didn't want to introduce that as a potential.

 

I've attached the most recent logs and the diagnostics, but I'm not seeing much. In the logs you'll see that there's nothing right before it crashed at around 8:40AM. One last thing that I'm wondering about as I type this, I've recent put in two shucked drives for the first time - one of them became a parity drive. I put the first one in and the system picked it up right away - no need for the 3.3 volt tape thing. However, the second drive (same model same purchase time etc) did need the adjustment to be recognized by the system. Could that new parity drive actually need the power adjustment and when parity writes happen the whole system locks up? At this point I'll just make the change the next time it crashes, but I'm at a loss.

 

Thanks for any help you all can give!

manky-dreadful-diagnostics-20240429-0852.zip syslog-172.16.120.5.log

1 hour ago, noja said:

Could that new parity drive actually need the power adjustment and when parity writes happen the whole system locks up?

I don't think that would be a problem, does the server crash just running as a basic NAS or do you have VMs/dockers running? If the latter you can try stopping everything for a day or two, if it still crashes it's most likely hardware.

4 minutes ago, JorgeB said:

stopping everything for a day or two

I already pared it down to just a few services that haven't had any new updates and turned off the one vm. I haven't run it as a basic nas quite yet.  I'm 99% sure its hardware, I'm just a bit clueless how to track down what it is when there isn't anything in the logs.