Upgraded from 6.11.5 to 6.12.0 and then had to revert back to 6.11.5 - Server crashes every 1-8 hours

[JorgeB]

8 hours ago, bullmoose20 said:

- stay in safe mode with plugins not running and turn on docker service which will then start up all my containers?

I would do this, since you'll still be running in safe mode and help confirm if a plugin is really the problem. 

[bullmoose20]

crashed again.

syslog (4)nzwhs01-diagnostics-20230624-0950.zip

[Craig Dennis]

18 minutes ago, bullmoose20 said:

crashed again.

syslog (4) 14.3 MB · 0 downloads nzwhs01-diagnostics-20230624-0950.zip 182.42 kB · 0 downloads

 

Are you concluding from this crash that the issue is with either the Docker service itself or a container you're running?
Would disabling 'auto start' for all containers and rebooting determine if it's a container or the service itself (if it stays up for 4+ hours)?
I'm by no means an expert but am experiencing similar issues with 6.12.

[bullmoose20]

i cannot conclude much at this point. Someone is telling me that I am likely experiencing kernel panics and that I should be doing a memtest for 24 hours to see. I am not at that point yet because with 168GB of RAM, that memtest is pretty painful

[bullmoose20]

 

 

Well... I thought that with my ilo4 board and console connected, I might get more details from the "screen" that might not be captured in the syslog locally on the usb drive or on the remote syslog server, however this reset just happens... literally nothing on the screen... as you can see. I am starting a memtest for the next 24 hours or so I guess? 

If anyone has an idea or sees a clue, let me know.

[JorgeB]

If downgrading back to v6.11.5 and leaving everything disabled doesn't help it suggests to me some hardware issue, and the upgrade was just a red haring.

[bullmoose20]

6 hours ago, JorgeB said:

If downgrading back to v6.11.5 and leaving everything disabled doesn't help it suggests to me some hardware issue, and the upgrade was just a red haring.

So… I took the opportunity to shutdown the server and move it to a new location. In doing so I took the time to get rid of dust that had accumulated everywhere and reseated all the ram. Since then… all looks good. 20 hours and counting of uptime. 
 

I agree… it was a red herring and just darn luck that the machine had been running so long. Talk about Murphy’a Law!