100% CPU in Dashboard and unresponsive, but top shows otherwise


boyd91

Recommended Posts

Sometimes (all) my services running in Docker containers are totally unresponsive and giving timeouts on every http request. Only the unraid GUI still works as fast as always, except for the Docker page, that one takes very long to load. But it will load eventually and there's no single container that's using much CPU or mem. I've experienced this 4-6 times already and only a reboot would fix it.

 

When this happens I can however log in to the unraid web GUI without any trouble. The dashboard bars show 100% CPU and memory usage on all my cores. But when I run `top` in the terminal, I see very different (and normal) CPU and mem usage.

 

I was hoping to be able to provide some diagnostics but it happens only every few months or so (will do it the next time it happens). I did look at the syslog last time and there was nothing unusual to see, only the timeouts that are caused by the unresponsiveness of the containers.

 

Does anyone recognize these symptoms? Or can anyone give me advice on how to pinpoint the issue, because this is seriously hurting the wife acceptance factor, considering Home Assistant is running on it.

  • Thanks 1
Link to comment

Im seeing similar symptoms, extremely high CPU usage in the UNRAID dashboard, by very normal cpu usage when i run 'top' or 'htop' in terminal.

 

Restarts didnt fix it for me, and upgrading the latest unraid did not fix it either. ( I went from 6.6.3 to 6.6.6).

 

I am running standard stuff, plex, sonarr, couch potato, sabnzbd, letsencrypt (with some wordpress sites), mysql, mariadb, nextcloud, rutorrent, unifi, muximux.

 

I have found the culprit in my case to be couchpotato,  it is running the mover on my completed torrents folder on repeat erroneously.. as in copy completed torrent xyz to the library location leaving a copy in the completed folder (as it should for seeding), however then not remembering that its processed that file already, and just doing it again and again, cycling through all the files in the folder....

 

I havent figured out what is wrong with couchpotato or how to fix it yet, but at least i know what the issue is and disabling couchpotato docker for now has stopped the super high CPU usage.

 

 

 

Link to comment
  • 2 weeks later...

Have you found out anything else about this?

I have the EXAKT same problem as you describe. Sometimes it however resolves itself.

I dont need to reboot, but I can actullay enter setting and disable dockers. This forces everything to stop and then I can restart docker agan and it works normally.

 

This is really annoying and I am not sure how to check what is causing this. 

I am not running Couchpotato but Radarr (and Sonarr). I have not made this connection. I have however sometimes files moving around which could seem to be forever but they dont use all this CPU when that has been occuring.

 

Most annoying things is that Plex is non responsive so my movies for the kids does not work (if I am not home to "fix" it).

 

-------------------------

Ok. Thanks for the tip. I logged into console. Killed off Sonarr docker with docker command. 

CPU dropped down to normal use in a few minutes (guess when all processes belonging to Sonarr where all killed).

 

I will try to disable Sonarr for the moment.

Edited by Johan76
New test - killing off containers
Link to comment
  • 4 months later...
  • 5 months later...
3 minutes ago, Timbiotic said:

is the wait the "wa" sorry dont use linux that often

yes

Cpu usage is always split in user/system/wait/idle . Together its ~100% (ignoring the other smaller indicators)

Meaning its x% of the time either serving the user, bussy with internal system activities, waiting for something , or idle doing nothing.

Edited by glennv
Link to comment

did you try killing them like normal linux processes with kill pid or kill -9 pid. ?

If they wont die with this , its difficult as likely completely hanging.

You can see this effect also when a process is hanging on an hard nfs mount that is not there anymore. Typicaly only a reboot can kill these sessions. You dont happen to have any mounted external shares that they can be hanging on ?

Link to comment
1 hour ago, Timbiotic said:

how can i see what specific is it waiting on

 

have you tried: ps l  (try ps -x instead this gives a better list with the current state of the process and the pid)

anything waiting should be in the (uninterruptible) D state I think... all of my processes are in the (interruptible) S sleep state.

Edited by Dissones4U
corrected
Link to comment

if killing the specific docker still did not clear the iowait situation then its not as simple to find the culprit.

check this for some ideas how to approach it https://bencane.com/2012/08/06/troubleshooting-high-io-wait-in-linux/

The required commands/tools for deeper troubleshooting you can install using the nerdtools plugin. Like iotop for example, which is in there and which may be usefull.

Its not a simple problem that can easily be identified remote. 

 

edit: iostat is part of the sysstat package of nerdtools

Edited by glennv
Link to comment
  • 1 year later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.