High shfs IO/CPU utilization, diagnostics attached

[Ramshackleton]

Unraid v 6.6.3

 

I've had some issues with transfer speeds or even directory listings going slowly for the past couple months, never sure exactly what was causing it but noticed that after a fresh restart of my dockers the problem tended to go away at least for a little while.  But recently the problem has gotten worse, in that in trying to watch a movie (via Plex, over my own gigabit LAN), the movie would pause every 10-20 seconds.  Unfortunately I didn't take my diagnostics at that moment, but I will say that a server reboot at that point did not fix the problem.

 

I did see another thread suggesting that I move my docker containers' config directories directly to /mnt/cache/appdata, which I did do this morning.  But it didn't seem to address the issue, as shfs still reported reading around 1-2MB/sec on average.  Now I don't know if that's enough to cause problems, and the CPU utilization for the process was usually in the 10% range, but imagine that it was probably worse last night when my movie was stuttering.  

 

Bu today's troubleshooting activities have left me scratching my head: I first shut down my Linux VM (which usually has somewhat high CPU utilization but doesn't seem to have much effect on disk IO), then started killing docker containers one by one, keeping an eye on iotop to see if/when the IO would go down.  None of them touched it.  Then I killed the docker service.  Still no change.  Then I restarted the docker service which in turned started all the containers back up, and at some time in the next 10-15 minutes, disk utilization dropped back down to essentially zero.  Perhaps this was a delayed reaction, but I can't understand why a reboot last night wouldn't have fixed the problem.  

 

In any case I know this is fairly rambling but I'm trying to make sure I mention anything potentially relevant.  Many thanks in advance for ideas on what to try next.

tower-diagnostics-20190701-1512.zip