Looking for help diagnosing the problem(s) and figuring out next steps to fix them and get back to a stable system.

The problems started a few weeks ago with what seemed like random crashes every few days. These were hard crashes that left the system completely inaccessible and required a power cycle to get back running. After a few crashes I thought I had isolated the problem to media transcoding. It seemed that anytime the CPU/GPU usage would be high for an extended period of time the system would crash. I tried using Tdarr and Unmanic and transcoding with both my Quadro GPU and the iGPU of my Intel CPU, but would get the same crashes. For a while, if I disabled those programs the system would be stable for days, even a week. I also noticed that when I would restart the system after crashes I would have SMART errors in the array. They would always be on Disk 4 and sometimes Disk 2. I would acknowledge them to clear them out and I don't recall any additional ones popping up while the system was running. I just assumed these were caused by the crash while something was being read on the drives during the transcoding.

I thought I had isolated the problem to the Tdarr/Unmanic transcoding and just left those stopped and continued to use my server like normal. That was until a few days ago when I started getting crashes whenever Plex was causing heavy CPU usage during into/credit detection, audio analysis, chapter images, etc. tasks. I went into Plex and changed all of those tasks to Never in the hope that this would stabilize the system while I had to travel for a couple of days this week. Everything seemed to be working fine and I was able to access Plex while traveling. Now I get home late last night and the system is still running, but I log into the GUI this morning and see that Disk 2 has entered an error state and is disabled.

I am coming here for help in figuring out what the actual issue(s) is/are and what I need to do to fix them.

I did run the quick SMART tests on the 2 drives that were giving errors after crashes and didn't find anything. I have also booted into memtest and ran several cycles with no errors. I have attached the diagnostics ZIP file from this morning. The system status has not changed since I ran it, nor have i made any changes since then.

Any assistance is greatly appreciated. Please let me know if there is any other information needed to properly diagnose the issues. Thank you.

cerealkiller-diagnostics-20250911-0911.zip

Disk2 dropped offline, check/replace its cables and post new diags after array start.

I stopped the array, removed Disk2 from array, restarted array in maintenance mode, stopped array, put HDD back in at Disk 2 and restarted array. Now it is currently rebuilding Disk2. I have attached new diags.

cerealkiller-diagnostics-20250913-2311.zip

Looks good so far, I assumed nothing was done to the cables? Post new diags if there are any errors.

Sorry, I forgot to mention that I did replace the SATA cable on Disk2. There is still around 3 hours left on the rebuild, but no errors so far.

The rebuild finished yesterday with no issues and the system has been running normally since with no errors. To hopefully eliminate as many potential issues as possible, I replaced my PCI SATA card and all SATA cables with the below items:

SATA Card

Cables

Now all 12 drive bays (9 currently in use) are wired through this card, unlike before when 6 were wired through the MB. After the hardware swap the system started up normally, all disks were properly recognized and the array was started successfully.

My plan now is to do a parity check to get some action on the HDDs. Hopefully that will finish without any errors. If so, I will try to run through a library scan in Tdarr without any errors or crashes. Around 95-98% of my library is already converted to 265, so most of the work will just be health scans, but it will be a good test to determine if my issues were as simple as SATA ports/card and/or cables. I will update the thread as I progress.

I just ran new diags and attached them just in case.

Thank you again for your help.

cerealkiller-diagnostics-20250915-2110.zip

So I finished the parity check with no issues or warnings. I then attempted to run the library scan in Tdarr which is what I believe started my issues. After about 30-45 minutes the system crashed as it has in the past. This was a hard crash that left the system completely unresponsive and required a power cycle to get going again. I have attached new diags from after the system restart and the syslogs. The crash happened around 9/18/25 15:00.

cerealkiller-diagnostics-20250918-1559.zip syslog-192.168.50.50.log

There are some HBA issues logged before the crash:

Sep 18 14:31:55 CerealKiller kernel: mpt3sas_cm0 fault info from func: mpt3sas_base_make_ioc_ready

Sep 18 14:31:55 CerealKiller kernel: mpt3sas_cm0: fault_state(0x2810)!

Sep 18 14:31:55 CerealKiller kernel: mpt3sas_cm0: sending diag reset !!

Not sure this would make the server crash, but it's still a problem, make sure the HBA is well seated and sufficiently cooled.