Please Add NUMAD


Recommended Posts

  • 2 weeks later...
My understanding of it was that it was supposed to optimally allocate ram based on the assigned cores which for me it doesn't do. Say I only assigned cores from node02 it would put all my ram for the vm on node0. I would also assume if I had cores from node0 and node02 it would properly allocate ram based on that but it didn't sadly. And for some reason unraid or qemu doesn't listen either even though I define all that in the topology. Even without numad it doesn't allocate the ram based on the topoloty assigned so I have to make fake machines that suck up the right amount of ram to balance it myself.
 
Mine rarely split BTW without me pushing it to. It usually will just grab everything from node0 even when told to get it from node02. Or as much free ram as node0 has..
Same here. Somehow setting it to "preferred" to node 1 in xml will make it to take 10~20mb from node 0 and the rest in node 1. Sometimes it just split it randomly but i don't know what causes it (probably dockers or just plex... or something that is allocating ram in node 1 even though all dockers are pinned to the cpu0 cores).

If set to "strict" it will take random allocations that can be all to the node0 or split like 75/25 between the 2 CPUs. When this happens gaming vm will suffer performance hit with some stuttering and fps loss and sometimes audio latencies with usb headsets.

I hope someday we can get this fixed or at least we can enforce it properly to avoid the performance and latency issues.

In my case this is the last step to finally reach near full baremetal performance in vm's.
The times when ram allocation is almost 100% of the proper cpu node i am able to reach it. But, since this is sometimes just random, I have to stop/start array a couple of times to get it.

Enviado desde mi Mi MIX 2 mediante Tapatalk

Link to comment
  • 3 months later...
  • 2 weeks later...

For the people experiencing random numa allocation this is what i did to workaround it.

In the XML template had to add two things:

 

Inside the cpu TAG:

Quote

  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='7' threads='2'/>
    <numa>
      <cell id='0' cpus='0-13' memory='16777216' unit='KiB'/>
    </numa>

  </cpu>

And up in the numatune tag:

Quote

  <numatune>
    <memory mode='strict' nodeset='1'/>
    <memnode cellid='0' mode='strict' nodeset='1'/>

  </numatune>

With this i avoided the random numa node allocations between nodes making it forced to node 1.

Quote

~# numastat qemu

Per-node process memory usage (in MBs) for PID 11494 (qemu-system-x86)
                           Node 0          Node 1           Total
                  --------------- --------------- ---------------
Huge                         0.00            0.00            0.00
Heap                         0.00            0.05            0.05
Stack                        0.00            0.03            0.03
Private                      2.64        16471.49        16474.12
----------------  --------------- --------------- ---------------
Total                        2.64        16471.57        16474.21
 

Pastebin example here.

 

Only thing i noticed is that now my VM take 10~15 more seconds to start booting.  Gaming performance looks good.

Edited by Nephilgrim
typo
Link to comment
On 8/30/2019 at 3:29 PM, Nephilgrim said:

For the people experiencing random numa allocation this is what i did to workaround it.

In the XML template had to add two things:

 

Inside the cpu TAG:

And up in the numatune tag:

With this i avoided the random numa node allocations between nodes making it forced to node 1.

Pastebin example here.

 

Only thing i noticed is that now my VM take 10~15 more seconds to start booting.  Gaming performance looks good.

I was able to achieve what you are getting with only numatune set to strict e.g.

  <numatune>
    <memory mode='strict' nodeset='2'/>
  </numatune>

 

1 hour ago, m0ngr31 said:

You were able to do this without numad?

I was able to do THIS every boot without numad 😅. My VM core assignment spreads across 2 nodes so consistent 50-50 split is the most ideal.

Per-node process memory usage (in MBs) for PID 31136 (qemu-system-x86)
                           Node 0          Node 2           Total
                  --------------- --------------- ---------------
Heap                         0.02            0.04            0.05
Stack                        0.02            0.02            0.05
Private                  24412.36        24810.51        49222.87
----------------  --------------- --------------- ---------------
Total                    24412.40        24810.57        49222.96

 

 

 

 

Link to comment
4 hours ago, m0ngr31 said:

You were able to do this without numad?

 

Yes. This is what numad does but it does it automatically. I forced it manually creating the node allocations.

 

3 hours ago, testdasi said:

was able to achieve what you are getting with only numatune set to strict e.g

Strict only works for me with node 0. Node 1 uses full node 0 too.

Only work sometimes on first vm but it's inconsistent. 

Edited by Nephilgrim
Link to comment
  • 4 weeks later...

Is this still recommended in 2019?  I know newer bioses came out with more automatic management across the nodes.  I've just left mine automatic, but highly suspect and slightly confused about what to do.  I have one or two apps I'd like to assign across both nodes, but my assumption is it will be slower than running on a single node....

Link to comment
On 10/1/2019 at 4:48 PM, Marshalleq said:

Is this still recommended in 2019?  I know newer bioses came out with more automatic management across the nodes.  I've just left mine automatic, but highly suspect and slightly confused about what to do.  I have one or two apps I'd like to assign across both nodes, but my assumption is it will be slower than running on a single node....

NUMA is different.. This is more for threadripper systems and systems will multiple sockets. If your running say a Ryzen 3k series you don't have to worry about any of this.

Link to comment
  • 2 months later...

So in 6.8 it looks like NUMAD is now part of the base OS, putting in the;

  <numatune>
    <memory mode='preferred' nodeset='0'/>
  </numatune>

In to a VM's XML file looks to be working fine.

image.png.86d078a933953af24e212384a69b0fc0.png

 

@limetech can you confirm that is the case? I get a permissions error saying permission is denied to "/boot/numad" and if it's not needed anymore I would like to remove it from the "go" file and the flashdrive.

Link to comment
1 hour ago, AnnabellaRenee87 said:

I get a permissions error saying permission is denied to "/boot/numad"

Can't comment on whether it's needed or not, but the reason you're getting permission denied is that due to security, no file stored on the flashdrive is directly executable anymore.  You have to copy the file somewhere else (say /tmp) and then execute it from there.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.