So, after getting into RC2, I was trying to optimize my pinning using the new interface. I looked up my NUMA boundaries in the process:
numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23 node 0 size: 32040 MB node 0 free: 256 MB # <<< Make note of this value node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31 node 1 size: 32243 MB node 1 free: 19974 MB node distances: node 0 1 0: 10 16 1: 16 10
I'm currently running two VMs as of this command.
VM 1: 16gb RAM, CPUs 4-7, 20-23 (So, numa node 0, in CPU pairs)
VM 2: 16gb RAM, CPUs 8-11, 24-27 (Numa node 1)
But as you'll note, all the RAM is being allocated to node 0. Uh oh. Let's check:
numastat qemu
Per-node process memory usage (in MBs)
PID Node 0 Node 1 Total
----------------------- --------------- --------------- ---------------
13479 (qemu-system-x86) 16473.43 0.25 16473.68
27148 (qemu-system-x86) 13259.18 3204.48 16463.66
----------------------- --------------- --------------- ---------------
Total 29732.60 3204.74 32937.34
Well, crap. That's no good. I then tried to force it using the <numatune> tags. This works fine for VM 1, which is completely in it's own node, but for VM 2, this makes it take forever to start up, because it tries to force the second qemu instance onto node 1 (where it should be) and you get a bunch of numa misses when the memory is allocated to node 0 anyway. This can also cause some NVRAM corruption in combination with other numa-optimizations and xml configuration settings, though I'm not able to remember exactly which one borked up the VM so bad I had to restore the .img file, nvram and xml to get nvidia drivers working again.
I imagine this will be extra important for 2990 users as two of the cores have significantly better memory access than the others and you'd want to keep VMs nicely in line with these boundaries for optimum performance.
Obviously we don't want this boundary crossing to happen with other processors (like my 1950) for performance reasons as well.
Bonus bug: WebTerminal is really slow this release once you have some text in the window compared to last release.
Bonus question: Any chance of getting 'numad' baked in so we can use "auto" in numatune?
- 1
- 1
Recommended Comments
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.