Last week I had one of my disks go down with an error during the mover. I woke up to the parity disk reporting thousand of errors per second (in the Main tab of the UI) and Disk 5 being emulated. The mover appeared to still be running but stuck, so I figured it was the source of the errors. I tried repeatedly to initiate a clean shutdown, including trying to manually kill the mover, but it would not respond to anything I did so I eventually pulled the plug.

 

After I powered back on I ran extended SMART tests on both the parity drive and Disk 5, which reported 0 errors after the test completed 4-5 days later.

 

I'm now trying to rebuild the contents of Disk 5. On the first 2 attempts, the ~30 hour operation was scheduled to complete overnight, and I woke up to the server having powered itself off. Upon power on it gives a notification that the data rebuild was successful, but then Disk 5 is still disabled/emulated.

 

I've attached diagnostics & a syslog that spans the crash. The most recent crash occurred sometime between 2024-12-08 22:00 and 2024-12-09 08:00, during which time nothing interesting appears in the syslog, so I am at a bit of a loss for what to try next.

 

Edited January 15, 20251 yr by icntoaun

Server powering down by itself, without a showdown event logged, basically can only be a hardware issue.

Is there any way to pinpoint which hardware component could be the problem? I'd blindly guess PSU but is there somewhere I can check for panic logs from other components or something?

 

One thing I'm going to try is setting up the syslog server on a separate machine - currently I use the "remote syslog server is just a share" strategy. I'm wondering if there are logs that I am missing based on the order of things that go down.

PSU or overheating CPU would be the main suspects.

Welp, I got the syslog server up on a raspberry pi just in time to see:

 

2024-12-09T12:02:00-07:00 khione kernel: mpt2sas_cm0: SAS host is non-operational !!!!
2024-12-09T12:02:02-07:00 khione kernel: mpt2sas_cm0: SAS host is non-operational !!!!
2024-12-09T12:02:03-07:00 khione kernel: mpt2sas_cm0: SAS host is non-operational !!!!
2024-12-09T12:02:04-07:00 khione kernel: mpt2sas_cm0: SAS host is non-operational !!!!
2024-12-09T12:02:05-07:00 khione kernel: mpt2sas_cm0: SAS host is non-operational !!!!
2024-12-09T12:02:06-07:00 khione kernel: mpt2sas_cm0: SAS host is non-operational !!!!
2024-12-09T12:02:06-07:00 khione kernel: mpt2sas_cm0: _base_fault_reset_work: Running mpt3sas_dead_ioc thread success !!!!
2024-12-09T12:02:06-07:00 khione kernel: md: disk5 write error, sector=1779862432
2024-12-09T12:02:06-07:00 khione kernel: md: disk5 write error, sector=1779861408
2024-12-09T12:02:06-07:00 khione kernel: md: disk5 write error, sector=1779861416
2024-12-09T12:02:06-07:00 khione kernel: md: disk5 write error, sector=1779861424
2024-12-09T12:02:06-07:00 khione kernel: md: disk5 write error, sector=1779861432
2024-12-09T12:02:06-07:00 khione kernel: md: disk5 write error, sector=1779861440

 

etc.

 

For anyone reading this in the future, my SAS controller is "LSI LOGIC SAS 9207-8i Storage Controller LSI00301" and it has caused so, so, so many headaches over the few years I've used it. It'll seem to work fine for months at a time then will randomly cause errors like this. Replacing it ASAP, and would definitely not recommend it.

Make sure the HBA is well seated and sufficiently cooled, you can also try using it in a different slot, but that issue should not cause the server to shutdown.