Kernel Panics - no logs or clues

[weirdcrap]

So as a follow up to my post here: 

around the same time I started having those parity errors the server started having kernel panics that would lock up the server, producing no logs.

 

I initially wrote this off as being related to the RAM issue, but its been two weeks of normal functionality since replacing the RAM and now the server again just dropped offline. I'm not seeing parity mismatches like before so I don't think this is the same issue. The monitor wasn't on when it went down so I have no clue what happened.

 

I've had the external rsyslog server option turned on in UnRAID for months but every single time the server hard locks, it locks up so thoroughly nothing useful ever gets logged. It's just whatever the last normal log entry was then the server startup again. Is there any other way to better capture what's happening? I'd like to avoid mirroring logs to the flash drive and given how I never get anything to my log server on the LAN I doubt mirroring locally would be any more productive...

 

2021-10-10T13:27:50-05:00 Node sshd[28719]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
2021-10-10T13:27:50-05:00 Node sshd[28719]: Starting session: command for root from xx.xx.xx.xx port 22016 id 0
2021-10-10T13:34:30-05:00 Node sshd[28719]: Close session: user root from xx.xx.xx.xx port 22016 id 0
2021-10-10T13:34:30-05:00 Node sshd[28719]: Received disconnect from xx.xx.xx.xx port 22016:11: disconnected by user
2021-10-10T13:34:30-05:00 Node sshd[28719]: Disconnected from user root xx.xx.xx.xx port 22016
2021-10-10T13:34:30-05:00 Node sshd[28719]: pam_unix(sshd:session): session closed for user root
2021-10-10T13:34:30-05:00 Node sshd[3724]: Connection from xx.xx.xx.xx port 9719 on xx.xx.xx.xx port 22 rdomain ""
2021-10-10T13:34:31-05:00 Node sshd[3724]: Accepted key RSA  found at /etc/ssh/root.pubkeys:1
2021-10-10T13:34:31-05:00 Node sshd[3724]: Accepted publickey for root from xx.xx.xx.xx port 9719 ssh2: RSA SHA256
2021-10-10T13:34:31-05:00 Node sshd[3724]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
2021-10-10T13:34:31-05:00 Node sshd[3724]: Starting session: command for root from xx.xx.xx.xx port 9719 id 0
2021-10-10T20:05:10-05:00 Node rsyslogd: [origin software="rsyslogd" swVersion="8.2002.0" x-pid="8183" x-info="https://www.rsyslog.com"] start
2021-10-10T20:05:10-05:00 Node root: plugin: skipping: /boot/config/plugins/enhanced.log/enhanced.log.cfg already exists
2021-10-10T20:05:10-05:00 Node root: plugin: skipping: /boot/config/plugins/enhanced.log/custom_syslog.conf already exists
2021-10-10T20:05:10-05:00 Node root: plugin: skipping: /boot/config/plugins/enhanced.log/syslog_filter.conf already exists
2021-10-10T20:05:10-05:00 Node root: plugin: running: anonymous
2021-10-10T20:05:10-05:00 Node root: 
2021-10-10T20:05:10-05:00 Node root: -----------------------------------------------------------
2021-10-10T20:05:10-05:00 Node root:  Plugin enhanced.log is installed.
2021-10-10T20:05:10-05:00 Node root:  Copyright 2015-2021, Dan Landon
2021-10-10T20:05:10-05:00 Node root:  Version: 2021.08.21
2021-10-10T20:05:10-05:00 Node root: -----------------------------------------------------------
2021-10-10T20:05:10-05:00 Node root: 
2021-10-10T20:05:10-05:00 Node root: plugin: enhanced.log.plg installed
2021-10-10T20:05:10-05:00 Node root: plugin: installing: /boot/config/plugins/fix.common.problems.plg
2021-10-10T20:05:10-05:00 Node root: plugin: skipping: /boot/config/plugins/fix.common.problems/fix.common.problems-2021.08.05-x86_64-1.txz already exists
2021-10-10T20:05:10-05:00 Node root: plugin: running: /boot/config/plugins/fix.common.problems/fix.common.problems-2021.08.05-x86_64-1.txz
2021-10-10T20:05:10-05:00 Node root: 
2021-10-10T20:05:10-05:00 Node root: +==============================================================================
2021-10-10T20:05:10-05:00 Node root: | Installing new package /boot/config/plugins/fix.common.problems/fix.common.problems-2021.08.05-x86_64-1.txz
2021-10-10T20:05:10-05:00 Node root: +==============================================================================
2021-10-10T20:05:10-05:00 Node root: 

 

I've asked that the monitor be left on so I can hopefully get some sort of a clue as to wtf is going on as I don't know what else to do besides just start replacing hardware. I guess I could upgrade to 6.10-RC1 but it was stable on 6.9.2 for months before this so I don't think it has anything to do with the OS version.

node-diagnostics-20211010-2007.zip

 

EDIT: I have this photo of a kernel panic from when I was still having the RAM issue, but its the only clue I've got right now.

Edited October 11, 2021 by weirdcrap