I can't seem to win lately. Fixed an issue with a docker hanging the system and now it appears it won't run for more than a few hours without crashing. I did just install more ram last night while working on the docker issue, but I am not sure if that is causing all the problems or not.

alexandria-diagnostics-20240622-1603.zip

Enable the syslog server and post that after a crash.

14 hours ago, JorgeB said:

Enable the syslog server and post that after a crash.

I have attached a syslog server capture that I setup on another machine that captured before crash and immediately after the crash.

 

Also attached the latest diagnostics from right after a boot

alexandria-diagnostics-20240623-0907.zip syslog.txt

The syslog before the crash contains entries like:

<4>Jun 22 23:22:04 Alexandria kernel: sd 10:0:2:0: Power-on or device reset occurred
<4>Jun 22 23:22:04 Alexandria kernel: sd 10:0:4:0: Power-on or device reset occurred
<4>Jun 22 23:22:04 Alexandria kernel: sd 10:0:5:0: Power-on or device reset occurred
<4>Jun 22 23:22:04 Alexandria kernel: sd 10:0:6:0: Power-on or device reset occurred
<4>Jun 22 23:22:04 Alexandria kernel: sd 10:0:7:0: Power-on or device reset occurred
<4>Jun 22 23:22:04 Alexandria kernel: sd 10:0:8:0: Power-on or device reset occurred
<4>Jun 22 23:22:04 Alexandria kernel: sd 10:0:13:0: Power-on or device reset occurred
<4>Jun 22 23:22:04 Alexandria kernel: sd 10:0:14:0: Power-on or device reset occurred
<4>Jun 22 23:22:04 Alexandria kernel: sd 10:0:15:0: Power-on or device reset occurred
<4>Jun 22 23:22:04 Alexandria kernel: sd 10:0:17:0: Power-on or device reset occurred
<4>Jun 22 23:22:04 Alexandria kernel: sd 10:0:18:0: Power-on or device reset occurred
<4>Jun 22 23:22:04 Alexandria kernel: sd 10:0:19:0: Power-on or device reset occurred
<4>Jun 22 23:22:04 Alexandria kernel: sd 10:0:20:0: Power-on or device reset occurred
<4>Jun 22 23:22:04 Alexandria kernel: sd 10:0:21:0: Power-on or device reset occurred
<4>Jun 22 23:22:04 Alexandria kernel: sd 10:0:22:0: Power-on or device reset occurred

which shows multiple drives simultaneously having issues so the problem is probably with something they all share.   I would either suspect power or the enclosure holding the drives although it could be the HBA I guess.

 

There are also call traces related to the HBA:

 

<4>Jun 22 23:21:04 Alexandria kernel: _base_build_sg_scmd_ieee+0x1d5/0x36d [mpt3sas]
<4>Jun 22 23:21:04 Alexandria kernel: ? _base_set_and_get_msix_index+0x3c/0x54 [mpt3sas]

 

this is quite uncommon, not sure if it would indicate a HBA problem or other hardware issue, always good to run memtest if you didn't yet.

8 hours ago, JorgeB said:

There are also call traces related to the HBA:

 

<4>Jun 22 23:21:04 Alexandria kernel: _base_build_sg_scmd_ieee+0x1d5/0x36d [mpt3sas]
<4>Jun 22 23:21:04 Alexandria kernel: ? _base_set_and_get_msix_index+0x3c/0x54 [mpt3sas]

 

this is quite uncommon, not sure if it would indicate a HBA problem or other hardware issue, always good to run memtest if you didn't yet.

So I actually spent a couple hours running memtest yesterday in various configurations. Initial test gave me a lot of errors, reran it with 1 stick, then 2 etc up to all 4 sticks and it passed. It’s been stable now for about 16 hrs or so. 
 

I think I might look at swapping out the HBA for further testing. Only down side is knowing which one might the culprit. I have 2 has, an internal and external one. Most likely start with the internal since it has more drives attached.