CPU usage spikes when downloading at high speeds

[MammothJerk]

Unraid: 6.12.4

X10SL7-F - Dual Gigabit Ethernet LAN ports via Intel® i210AT

Intel® Xeon® CPU E3-1231 v3 @ 3.40GHz

32gb ddr3 ecc memory

 

My usual CPU usage is around 30-40% with 50 containers running and i've noticed that the cpu usage on all cores hits 100% whenever i'm downloading anything at high speeds (20MB/s+).

 

This is so bad that almost everything else on the server stops working, if i'm downloading and trying to update my docker containers the updates per container can take up to 10 minutes each.

It usually brings the rest of the containers to a crawl as well, i cannot open plex or jellyfin, i cannot open radarr/sonarr either, they will just open the webui "loading" pages where they say something like "loading battlestation" and this similarly will just keep loading upwards to 10 minutes. and forget about doing any operations like imports and renames.

 

The downloads are writing to the cache so the parity calculations shouldn't be a factor, even looking at the parity when the cpu is at 100% shows no writes being made.

 

I tried stopping all containers except for resilio sync and still all cores are pinned at 100% when downloading at 40-50MB/s to the cache pool.

pausing the downloads immediately brings down the cpu usage to about 10%.

 

I've tried looking at the CPU usage via top but it never shows more than 20% being used by resilio sync even though the unraid webui shows 95-100% on all cores

 

is this expected behavior? does this CPU always hit 100% usage when downloading at around 50MB/s in other systems? does it have something to do with the NIC? is it unraid? is it the filesystem?

 

Thanks.

Edited October 29, 2023 by MammothJerk

[MAM59]

you may have been fooled by a known UNRAID problem: the CPU Usage also counts in blocked IO.

If a disk cannot write fast enough to empty the buffers, the process is halted and cpu usage shows 100%.

Of course, all other processes have to wait too, so it looks like the machine is halted. But it will recover soon again.

(of course until the copy still exits, this will happen over and over again)

You can tell if your 100% are real 100% or come from the IO halt, if you look at the CPU temperature. If machine is stalled but temp is low to normal, you are facing the blocked IO problem. If CPU is almost glowing, you really see 100%.

 

If you have the IO problem, there are some ways to make it smoother. But for getting totally rid of it you usually need faster disks/controllers/cache drives.

 

[MammothJerk]

20 hours ago, MAM59 said:

you may have been fooled by a known UNRAID problem: the CPU Usage also counts in blocked IO.

If a disk cannot write fast enough to empty the buffers, the process is halted and cpu usage shows 100%.

Of course, all other processes have to wait too, so it looks like the machine is halted. But it will recover soon again.

(of course until the copy still exits, this will happen over and over again)

You can tell if your 100% are real 100% or come from the IO halt, if you look at the CPU temperature. If machine is stalled but temp is low to normal, you are facing the blocked IO problem. If CPU is almost glowing, you really see 100%.

 

If you have the IO problem, there are some ways to make it smoother. But for getting totally rid of it you usually need faster disks/controllers/cache drives.

 

The CPU temp is pretty much stable at around 50C, does this look like the IO problem then?

 

I have a 9201-16i coming in soon for expanding my setup as opposed to the onboard sata ports so maybe that will help?

 

my cache drive is a kingston a400 960g, maybe adding another one in to the pool would help with the speed?

 

my normal array drives are a bunch of wd red 20TB's but as the issue is only with the cache drives im not sure they are relevant.

 

Any ideas of tests that i could do to troubleshoot this better?