I've got a 64 thread Epyc 7551p in my unraid server, with threads 0-7 & 32-39 isolated for use in a vm. This should constitute the first numa node, and the first 8 cores/16 threads.

 

It appears that something is always being scheduled for use on the first 2-3 threads, and it appears to be some process(es?) related to docker, or one of my docker containers.

 

I have tried moving the isolated threads to another numa node but the first 2-3 threads always seem to be in use still.

I have also tried pinning docker containers, but that didnt have any affect either.

 

Ive attached a diagnostic.

 

Any help would be appreciated!

 

wiggum-diagnostics-20240227-1038.zip

Good practice is to never pin/isolate Core 0 for any VM/Docker, as this will usually be used by the bare metal OS.  This applies to most every OS, including Unraid.  With the number of cores you have, there should be plenty of room for everyone to play nice and have the space they need.

35 minutes ago, ConnerVT said:

Good practice is to never pin/isolate Core 0 for any VM/Docker, as this will usually be used by the bare metal OS.  This applies to most every OS, including Unraid.  With the number of cores you have, there should be plenty of room for everyone to play nice and have the space they need.

 

Fair, that makes sense. I'll shift it off the first node to stick with best practice.

When I have moved the isolated threads for troubleshooting though, the processes seem to move with it. The first 2-3 isolated threads seem to be used, regardless of which ones I've got isolated.

 

For example, having just moved the isolation to the back 16 threads:

Edited February 27, 20242 yr by Flon22

Apolgies for the bump but I really am stuck with this if anyone has any ideas.

For a bit more information, here is what it looks like when I'm using the VM:

 

CPU 24 -> 31 and HT 56 -> 63 are isolated.

 

CPU 28 -> 31 and HT 58 -> 63 are pinned to the VM, the other 4 threads have nothing pinned.

 

No idea what is using CPU 24, 25 and HT 56.

 

This is what an average HTOP looks like with the VM running. Note whatever was pegging cpu 24 in the above screenshot had stopped at this point, but the unpinned, but isolated threads were still being used.:

 

Edited March 9, 20242 yr by Flon22

On 3/9/2024 at 6:18 PM, Flon22 said:

Apolgies for the bump but I really am stuck with this if anyone has any ideas.

For a bit more information, here is what it looks like when I'm using the VM:

 

CPU 24 -> 31 and HT 56 -> 63 are isolated.

 

CPU 28 -> 31 and HT 58 -> 63 are pinned to the VM, the other 4 threads have nothing pinned.

 

No idea what is using CPU 24, 25 and HT 56.

 

This is what an average HTOP looks like with the VM running. Note whatever was pegging cpu 24 in the above screenshot had stopped at this point, but the unpinned, but isolated threads were still being used.:

 

You may need to use Numa 1 for the VM as that is where most of your PCI passthru devices are.

 

I have exerimented on my system but only have 2 nodes and 64G per node.

 

I have added this to my xml and it does remain persistent even if you update on the gui.

  <numatune>
    <memory mode='strict' nodeset='1'/>
  </numatune>

 

This results in the following PID 18396 is the vm for the XML. not sure on impact if you set with 16Gb for the VM and only 16Gb is available on that node. Maybe try with 12Gb first, You can specify more that one node so you could do nodeset="1-2" or ="1,2".

 

Per-node process memory usage (in MBs)
PID              Node 0 Node 1 Total
---------------  ------ ------ -----
18396 (qemu-syst      2  32478 32480
31132 (qemu-syst   1183  15529 16712
---------------  ------ ------ -----
Total              1185  48007 49192
root@unraid:~# 

 

As you have somes PCI devices on different nodes you will see IO on some of the other cores not allocated to the VM.