Dockers and or server becoming unresponsive for a short time.

[Michael Ganrer]

Recently I have migrated my unraid server from a desktop intel 10700 to an HP DL360 g10 server. Every thing seems to be working but I am getting a hiccup every couple minutes where all my dockers and basically entire server becomes unresponsive. If I have the dashboard open already the CPU usage will still update but I couldn't switch do the docker tab if I wanted to. It is a 48 core 96 thread system and I don't have any CPU pinning except for the VM that is limited to 8 cores 16 threads. When it is unresponsive, anywhere from 8-11 of the cpu threads are pegged at 100%. They drop back down to normal after about 30 seconds and the system returns to normal. Any ideas on what would cause this?

jinks-diagnostics-20240807-0712.zip

[Michael Ganrer]

Update..... I have tried to figure out what is using the cores but HTOP freezes as well.

Edited August 8 by Michael Ganrer

[JorgeB]

Does it still happen is you disable the docker service?

[Michael Ganrer]

Sorry for the late response. I have been doing a bit more diag on this. It seems that it is happening whenever I write to ether cache drive. The more docker containers I stop the less frequent it happens. Same thing with the VM that I am running. The VM is windows 11 running Blueiris camera software that dumps to the cache drive once an hour or so to be written to the array overnight. The cache drives were originally install in the front of the 1u server while the HDD are installed in 2 SC200 SAS expansion racks. I moved the cache drives to the SAS bays because I don't have any problem writing to the array. The Cache drives are still doing the same thing there. Both cache drives are new with no SMART errors. 1 cache drive is btrfs while the other is xfs. Nothing is set to write between either cache drive(anymore). Please let me know if anymore info is needed are if I can try something else.

 

Thanks everyone!

[JorgeB]

Try using a disk path/share every time that is a possibility, or use exclusive shares, sometimes bypassing FUSE (/mnt/user) helps a lot with that type of symptom.

[Michael Ganrer]

That is a change I made about a week ago. No change. I will attach a current diagnostic.

jinks-diagnostics-20240823-1050.zip

[Michael Ganrer]

Also, here is a screenshot of netdata.

[JorgeB]

Unfortunately there's nothing relevant logged that I can see.

[Michael Ganrer]

The netdata screenshot was to illustrate that when iowait is high is when everything becomes unresponsive. Hence the gap of unrecorded data. I was actively copying files(deluge was anyways) when this was taken.

[JorgeB]

Yes, but there's nothing logged in the syslog to explain that, if you are already using disk/exclusive shares, I don't see another reason for that, could be just slow devices, that won't leave anything logged.