So I’ve had a couple of instances where my server would completely lock up. It would drop off the network completely and the only way to bring it back up was a hard reboot. 
 

If someone is willing to dig through my diagnostics to help me figure this one out it would be extremely appreciated. I feel like I’m taking random shots in the dark and getting no closer to solving this. 
 

I had this happen a couple of times in the past, and have chalked it up to some dockers using all the ram and locking the system up. I’ve got no concrete proof of that, and have since built a new machine with more memory. It didn’t seem consistent with how often it locks up so I never know if I’m going to come home to a server or not. Sometimes it’s up for a week, and sometimes it locks up two days in a row.

All seemed well for the first couple of weeks when running in the new machine, but the issue has struck again a couple of times.

 

I started limiting some dockers that I suspected might be using too much memory, and have limited them to a couple of cores and a fraction of the systems memory. this hasn’t proved useful as I’ve just had two lockups this past week, so now I’ve limited the rest of the dockers, so I’m hoping that might make a difference, but I wanted to post here in case someone else spots something wrong right away.

 

I had added Netdata to see if maybe it would capture a cpu / ram spike from when things lock up, but nothing seems out of the ordinary there, at least in what gets written to its database before a lock up.

 

Something that may be of note, is I may have had a lockup yesterday that recovered. I’m not sure if it’s the same issue as all of the other lockups since it did seemingly recover, but it brought all of my dockers down, and I could not restart them until stoping and restarting the array.

 

the main thing I noticed in the logs from before I restarted the array was this:

 

Jan 24 16:44:55 Tower kernel: verify_parent_transid: 52 callbacks suppressed

Jan 24 16:44:55 Tower kernel: BTRFS error (device loop2: state EAL): parent transid verify failed on 7135576064 wanted 5280418 found 5280417

 

I’ve attached diagnostics from yesterday (taken a bit after restarting the array).

 

I had another lockup just a few minutes ago, which has prompted me to write this post, as I’ve started going a bit crazy because of this, and I’ve attached those diagnostics as well from right after booting up again. I’m guessing it won’t show anything but I figured it can’t hurt to add.

 

I also just enabled mirroring the syslog to my flash drive, so if this locks up again maybe that might have show something.

 

If anyone has any advice, I’m all ears! Thank you!

 

 

 

tower-diagnostics-20230125-2348.zip tower-diagnostics-20230126-2225.zip

first thing to rule out would be a basic hardware error. try running a long memtest session to rule out bad memory

if that turns out fine, try sending logs to a remote syslog server or mirror to flash so logs can survive a reboot after crash

Sounds good, I was hoping since It seems like this issue was present on the old machine it might rule that out, but worth a shot.