Containers limited to 1 thread if using cpu-setcpus on isolcpus threads

[-Daedalus]

So this is a weird one.

 

If you have your syslinux configured like mine, and have isolated some cores from unRAID, like so:

label unRAID OS
  menu default
  kernel /bzimage
  append isolcpus=8-15,24-31 initrd=/bzroot

Any time I pin any of these threads to a Docker container using:

--cpu-setcpus=14-15,30-31

That container is limited to only using the first thread.

 

VMs don't seem to be affected by this, only Docker stuff.

Running 6.5.2 at the moment, but will update to 6.5.3 when I can.

 

Can anyone else reproduce this?

Edited June 16, 2018 by -Daedalus

[-Daedalus]

*bump*

 

@limetech (or anyone else). Would someone care to see if they can reproduce this? If it's just me that's fine, but if not it would be nice if I could be investigated a bit.

[tjb_altf4]

Looking at the docker documentation, I think you need double quotes around the number set i.e.
 

--cpu-setcpus="14-15,30-31"

 

[-Daedalus]

Thanks for the input. 

 

Unfortunately, no difference.

 

Using quotation marks or not, one thread is used on isolated CPUs, all threads are used in non-isolated CPUs.

 

 

[Squid]

On 6/16/2018 at 1:40 PM, -Daedalus said:

--cpu-setcpus=14-15,30-31

should be

--cpuset-cpus=...

[-Daedalus]

That is what I have. I manually typed that bit from memory, apparently got it arseways. But this is what I have now, copy/pasted from container settings just now:

--cpuset-cpus="3-4,19-20"

 

[-Daedalus]

Has anyone else tried to reproduce this? I'm very curious if this is just me or not.

[Alphahelix]

On 6/22/2018 at 2:02 AM, -Daedalus said:

Has anyone else tried to reproduce this? I'm very curious if this is just me or not.

Yes! I was trying to isolate 2 cores to Unraid, 4 other cores to dockers, and the rest to VM.

But it looks like the cores that is isolated isolcpus=### really isolate cores for Unraid AND docker. My conclusions is based on tingering with "isolcpus=" and the fact that the assigned VM cores are outside the scope of the "isolcpus=".

If possible it would be great (at least for my setup) to be able to assign dedicated cores to: unraid, docker, VMs and to "<emulatorpin cpuset='##'/>"

[-Daedalus]

Thank you! Someone else tested this!

 

I think it makes sense that isolcpus are isolated from Docker as well; this runs within unRAID.

 

But, you should be able to manually assign them via pinning, and it does work; a container will run on CPUs that unRAID itself can't touch, it just only runs on the first one for some reason. Oh yeah, and this is still present in 6.5.3 as well.

[pwm]

Is it really good to use both isolcpus and cpuset for same cores?

 

If the core is isolcpu:ed then the scheduler will not touch it - but who will then schedule between multiple threads if there are multiple threads in the cgroup that needs to share the cores that have been cpuset?

[-Daedalus]

Good point. Honestly, I'm not sure on this one. It's more nuts-and-bolts info than I have, that's why I was hoping someone more knowledgeable could chime in.

 

If this is expected behaviour then that's fine, it just doesn't seem like it should be.

[Alphahelix]

If you Isolcpu'ed core 0-3 leaving 4-7 unused, you can then dedicate them to VM's (works in my setup), but you cannot dedicate them to docker containers.

[SSD]

I have no CPU limits on unRAID in the flash drive syslinux configuration.

 

I do allocate 11 of my 12 cores to my VM (leaving core0 for unRAID's use).

 

For Dockers, all my lighweight dockers (everything but Plex) are restricted to Core0 (shared with unRAID).

 

For Plex, i have the same 11 of 12 cores as the VM.

 

So my VM and Plex each have access to 11/12ths of my CPUs horsepower. They are seldom in heavy use at the same time, and I think the horsepower is such that it would be very rare to either of them would impact the other in any meaningful way, even with Plex transcoding (in software) a 4K x265 stream to 1080p (taking 30%-50% of the CPU).

 

On my prior server, with 4 cores, I tried allowing VM and Plex to share 3 cores. But I found Plex, when syncing to a tablet, could suck all the horsepower and literally leave the Windows unresponsive to mouse clicks for 10s of seconds. It was unusable. So I changed it so that VM and Plex each had access to two cores (one dedicated and the other one shared). So instead of having access to a max of 3 shared cores, they had access to a max of 2 cores, but always had one core dedicated. This prevented the issue I was having. BTW, syncing with my tablet allows Plex to transcode as fast as it can. With real-time watching, the transcodes are paced to support the playback. So the syncing is transcoding at a much faster rate, and therefore uses a lot more CPU, than real-time transcoding.

 

If I ever had a similar issue occur in my current setup (Windows or Plex lagging due to the other being used heavily), I think I'd do something similar - dedicating 1 core to the VM, 1 core to Plex, and allow them to share the other 9. A 4.5GHz core is plenty to allow Windows to be responsive to mouse clicks and even significant computing, even if all the other cores were tied up with a heavy transcode requirement on Plex. If I found I was needing a lot of horsepower from Windows and Plex at the same time (which almost never happens now), I may look at dedicating more cores until the issue is managable. I recommend this type of approach (allow max sharing and tweaked based on actual observed issues).  Rather than giving each the VM and Plex half of the cores. This might guarantee you won' t hit the starving situation, but also reduces the value of sharing your server asset across multiple VMs and Dockers.

 

Technically I think that core unRAID would have access to all the cores, but from observed behavior they tend to stick to core0. So I have not tried to mess with limiting unRAID to core0.  

 

One thing I did do, in the VM config, I have changed the mapped the VM's virtual threads to the actual threads. So what the VM sees as core0, is actually core11:

 

Original:

    <vcpupin vcpu='0' cpuset='1'/>

   ...

Revised:

    <vcpupin vcpu='0' cpuset='11'/>

   ...

 

My theory is that the VM would then tend to prefer core11 and use them in descending order, while Plex would tend to prefer core1 (core0 is not available to Plex), and use them ascending. And that this would keep them from interfering with each other. But I do notice core utilization bounces around a lot, so I'm not sure the rearranging is really having an effect if the OS is using something other that order to assign the cores, but couldn't hurt. If anyone else wants to do this, note that the only way to tweak these assignments is by editing the XML, and if you ever edit any VM settings through the GUI, it is going to reset the mappings.

To solve this and other risks of the VM config getting messed up, I actually have a backup configuration for this same VM, so if I ever forget (or need to use the GUI because I am doing something significant), I have the CPU mappings in the backup config, which I can copy and paste into the real config's XML. It's a good idea to have a duplicate configuration, so if ever your primary config stops working (e.g., you made some changes and now the VM isn't booting), you have your backup as a functional configuration to use until you can get the primary working again. (Or just copy and paste the entire backup configuration to the primary, and you are back to a known working state and can re-do whatever you were trying to do.) Copying the config to a .txt file is also a reasonable option. You just have to remember to back it up after any change. With the backup config staring you in the face, it is much more likely you'll remember to update it.

Cheers!

[-Daedalus]

Hi SSD!

 

Cheers for the reply!

 

What you're doing is what I'd like to move to: Leave all the lightweight containers on a couple of cores, along with unRAID, then manually assign all the other cores to containers and VMs as necessary.

 

The issue is, whenever I try to assign cores to a container that unRAID's not using (with isolcpus), the container only uses the first core. I've tried this with Plex and MineOS and the behaviour is the same.

 

Your VMs and containers are definitely using resources as you'd expect, based on allocation?

Do you have your CPUs isolated in the same way (--cpuset-cpus=xx)?

Are you running the same container (linuxserver) for Plex?

Are you on different hardware to me? (1950X, X399 Taichi)

Edited July 1, 2018 by -Daedalus
fixed typos

[Squid]

1 hour ago, -Daedalus said:

The issue is, whenever I try to assign cores to a container that unRAID's not using (with isolcpus), the container only uses the first core. I've tried this with Plex and MineOS and the behaviour is the same.

But, this all makes sense when you consider @pwm's comments about what isolcpus does is stop the task scheduler from using those cores, so it would only wind up using the first core since scheduler is forbidden from touching any of the other cores.  You should however be able to assign an individual docker container to which core you want (and other containers to other cores), but there's probably zero way to get the system to work the way you want as its by design on the Kernel

[-Daedalus]

Understood. pwm's comments made sense to me, that's why I was asking for clarification from @SSD, because it sounds like he has things running the way I'm hoping, but it would seem to fly in the face of what pwm suggested.

Edited July 2, 2018 by -Daedalus

[SSD]

On 7/1/2018 at 1:27 PM, Squid said:

But, this all makes sense when you consider @pwm's comments about what isolcpus does is stop the task scheduler from using those cores, so it would only wind up using the first core since scheduler is forbidden from touching any of the other cores.  You should however be able to assign an individual docker container to which core you want (and other containers to other cores), but there's probably zero way to get the system to work the way you want as its by design on the Kernel

 

 

All I can really explain is what I did and tried. If you have any questions about my approach, please ask. You are free to experiment to find the best solution for your particular situation. 

 

Again, my goal it's to share my CPU asset to greatest degree, tweaking the approach of I experience performance issues in my real world use of the server. Rather than try to isolate each core to a single use with little or no s sharing, which avoids contention, but might underutilizes the CPU.

[-Daedalus]

SSD, can you confirm that the Plex container is in fact using those two cores? Are those two cores also isolated from unRAID with isolcpus?

[SSD]

52 minutes ago, -Daedalus said:

SSD, can you confirm that the Plex container is in fact using those two cores? Are those two cores also isolated from unRAID with isolcpus?

 

Please go back and re-read my post. Plex is not confined to 2 cores. And I am not isolating cores from unRAID.

[-Daedalus]

Yeah, probably shouldn't have read that so quick...

 

Missed "no" on the first line, and confused a couple of things otherwise.

Alright. I guess I have to manually assign cores for all my containers then to get them off those cores. 

 

Thanks for all the help guys.

[Warrentheo]

So just to confirm, docker runs on top of the Unraid kernel, and so the only way to fix this is to turn off CPU isolation?

With Unraid-6.6.3, CPU pinning is much easier than it used to be so this is less of an issue, but with no isolation enabled does this cause minor performance issues or stuttering on the VM's when the Unraid kernel tries to process something on a pinned CPU?

For instance I currently have Unraid isolated down to just core 0 and its hyperthread mirror, and then use emulatorpin to pin the VM's to emulate on that core and pin the remaining cores as needed to VM's/Docker's...  But now that I am getting more into Docker, I am running into this issue because of it...

 

Am I just stuck with turning off CPU isolation and just dealing with random performance bumps in VM's and Docker's?

Edited November 1, 2018 by Warrentheo