6.12.6 smartctl (and a few other things) causing 100% CPU and unresponsivness

Closed

I have a media center NAS. It works flawlessly all day long for plex, transfers, general utilities etc. Even under quite heavy load (4-5 streams) it chugs along.

But it "crashes" every night. I've done extensive diagnosis and experimentation, and eliminated almost everything, including rebuilding the config from scratch multiple times.

I just installed newrelic for additional monitoring, and I believe it finally found the culprit. (Unfortunately netdata does not do a good job of tracking arbitrary processes that aren't in its list, otherwise this could have been a much quicker journey)

The crashes generally occur between 11pm my time and 3am my time, though there have been a few that have happened as late as 5-6am. Notably this is when the server is most idle, so may have something to do with either sleep kicking in, or some sort of scheduled task that is waiting until the system is unused.

They are not actually "crashes" always. For the example from last night, the server became unresponsive at 11:52pm (per a user script that pings status cake 1x/min) But as you can see in the netdata and newrelic graphs, both dockers continued to collect metrics until I rebooted the server in the morning. This does not line up with mover nor plex scheduled task windows (1 am and 2 am respectively)

startBackground, monitor, smartctl_type, php, startSchedule.p and samba-dcerpcd all kick in at roughly the same time and eat all the CPU

This morning, I ran smart self tests on all the drives in the system (array and cache) and all completed without error. And as I said, even under heavy disk usage everything works fine.

I have run a memtest for 3 hours, which was clean. I've also tried running the server with only the original 4gb that the machine came with.