My CPU is pegged at 100%, and I don't know what is causing it.  Here is the diagnostics report.  Any ideas?

server-diagnostics-20220520-1124.zip

Any ideas? It seems my syslog is getting filled up causing ram to filled up causing 100% cpu.  I'm attaching another diagnostics upon waking up this morning.

server-diagnostics-20220521-1045.zip

According to your diagnostics your syslog is being spammed with messages of the form:

May 21 10:45:25 server kernel: sd 7:0:18:0: [sdf] tag#110 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s
May 21 10:45:25 server kernel: sd 7:0:18:0: [sdf] tag#110 Sense Key : 0x5 [current] 
May 21 10:45:25 server kernel: sd 7:0:18:0: [sdf] tag#110 ASC=0x26 ASCQ=0x0 
May 21 10:45:25 server kernel: sd 7:0:18:0: [sdf] tag#110 CDB: opcode=0x42 42 00 00 00 00 00 00 00 18 00
May 21 10:45:25 server kernel: blk_update_request: critical target error, dev sdf, sector 941623360 op 0x3:(DISCARD) flags 0x0 phys_seg 1 prio class 0

with the sector number changing.   The sdf device appears to be your cache2 device.    This makes it sound as if it could be failing  

Wouldn't a disk error show up in a smart report?

7 minutes ago, Mlatx said:

Wouldn't a disk error show up in a smart report?

Not always. Run an extended SMART test on the disk

I'll try that.  I had ntop running.  I took some pics and noticed this: unraid /var/run/dockerid.pid --log-level=fatal --storagedriver=btrfs.  Does that mean something docker related could be jamming the log?

Extended SMART test showed no errors. Could it be the SATA port? I'm looking at a live log and still getting the blk_update_request: critical target error, dev sdf. 

Post new diagnostics

I'm using a SAS controller for my disks. I had a bunch of errors on my ryzen board with the internal controller, and going SAS eliminated them.  I wanted to test the SAS cables.  Switching them did nothing.  I put the cache drives back on the internal motherboard controller, and I no longer have the error in my log. 

 

Recently, disk 2 was giving me errors, and an extended SMART test showed no errors.  This happens every 3 months.  Restarting fixes the issue.

 

It looks like I'm going to have to by a new SAS controller.  First, I'll wait and see if the disk 2 error shows up again.  Maybe having the cache drives and share drives on the same controller causes some kind of issue. 

You're using an LSI RAID controller, get and LSI HBA instead, some recommended models here.

Thanks @JorgeB

I've ordered a 9207-8i.  I'm back to having the cpu pegged to 100 again.  The log is not full this time.  The memory is.  I'm attaching the latest log file.

server-diagnostics-20220522-1504.zip

Any VMs?

Yes but not running at the time. I'm thinking it could be the zoneminder docker.  htop with 100% cpu showed zmc process consuming 66% of the ram.  It seems like a docker issue. The other potential docker is crash plan pro.  I have it set to limit memory usage.  I'm going to try and monitor the ram and kill off a docker to see how much gets freed.

It looks like it was crashplan.  I've deleted and re-installed it.  The quick fill up seems to have went away.