CPU maxed and all disks being read

[ivanavich]

Hi all thanks in advance for the assistance.

 

 

Dashboard is showing 100% CPU usage for the past couple of hours and all drives are being accessed.

 

Dockers all quiet showing no more that 1% CPU usage.

 

NetData is showing constant 560MiB/sec disk reads. Applications chart shows ZFS is consuming the majority of that. All array and pool drives are formatted ZFS + LUKS.

 

htop unremarkable, top showing ksoftirqd/8 at 35.0 %CPU with 27 instances of z_rd_int_0 and z_rd_int_1.

Can anyone shed some light on what ZFS is doing? Diagnostics attached.

odin-diagnostics-20240605-1034.zip

[ivanavich]

After rebooting:

critical medium error, dev sda, sector 1453600

critical medium error, dev sda, sector 1453688

critical medium error, dev sda, sector 1453864

critical medium error, dev sda, sector 1453880

bzfirmware checksum error - press ENTER key to reboot ...

 

I did a Get-FileHash on the boot USB drive bzfirmware and CRC error unreadable. So copied over fresh bz files and the server booted.

 

Parity check kicked off automatically and is running at 1.7 MB/sec. ZFS still smashing CPU and disk access.

 

odin-diagnostics-20240605-1749.zip

[ivanavich]

Update:

 

ZFS is scrubbing all the drives as per cron entries in /config/plugins/dynamix that match the current timing. It appears the corrupted bzfirmware is inconsequential.

Anyone know where/how those cron schedules were created? I can't find any further information in my digging.

[JorgeB]

1 hour ago, ivanavich said:

critical medium error, dev sda, sector 1453600

critical medium error, dev sda, sector 1453688

critical medium error, dev sda, sector 1453864

critical medium error, dev sda, sector 1453880

These suggest a flash drive issue, if it happens again recommend replacing it.

 

38 minutes ago, ivanavich said:

Anyone know where/how those cron schedules were created?

They can be set by clicking on each zfs disk/poll, and by default scheduled scrubs are disabled.

[ivanavich]

3 hours ago, JorgeB said:

and by default scheduled scrubs are disabled.

Is that because a regular parity check should find any bit-errors?

[JorgeB]

In part, since it basically accomplishes a similar result, and the check also does a full disk test, unlike a scrub, but if you have any pools you should schedule a scrub.