July 14, 20205 yr I've recently picked up a Dual Xeon e5-2690 v3 server, and I appear to be having fairly poor NUMA memory allocation. These CPUs are a little funny, as they utilize a topology called "Cluster-on-Die". Each CPU has 12 cores, but those cores are divided into two distinct NUMA Nodes, each with access to their own memory controller. There was a whitepaper from NASA that covered this topology in great detail [here]. Super helpful read if you would like to know more about these processors! Here's a diagram from that article which shows the exact topology of these CPU dies. Each bi-directional "ring" is it's own NUMA Node with 6 cores associated: The output from numactl --hardware: # numactl --hardware available: 4 nodes (0-3) node 0 cpus: 0 1 2 3 4 5 24 25 26 27 28 29 node 0 size: 32124 MB node 0 free: 197 MB node 1 cpus: 6 7 8 9 10 11 30 31 32 33 34 35 node 1 size: 32253 MB node 1 free: 65 MB node 2 cpus: 12 13 14 15 16 17 36 37 38 39 40 41 node 2 size: 32253 MB node 2 free: 165 MB node 3 cpus: 18 19 20 21 22 23 42 43 44 45 46 47 node 3 size: 32253 MB node 3 free: 130 MB node distances: node 0 1 2 3 0: 10 11 21 21 1: 11 10 21 21 2: 21 21 10 11 3: 21 21 11 10 This is what I get from lstopo: And this is what I'm getting from numastat after 22 hours of uptime: # numastat -n Per-node numastat info (in MBs): Node 0 Node 1 Node 2 Node 3 --------------- --------------- --------------- --------------- Numa_Hit 2288524.80 1739432.91 1395431.20 815033.61 Numa_Miss 662912.27 1059782.66 983562.46 1219333.84 Numa_Foreign 2014864.54 942640.61 651756.69 316329.40 Interleave_Hit 196.36 196.60 196.48 196.50 Local_Node 2288502.47 1739217.54 1395220.26 814821.59 Other_Node 662934.60 1059998.02 983773.40 1219545.87 Total --------------- Numa_Hit 6238422.52 Numa_Miss 3925591.23 Numa_Foreign 3925591.23 Interleave_Hit 785.94 Local_Node 6237761.86 Other_Node 3926251.89 I have also used the Intel Memory Latency Checker to show a better understanding of how these NUMA Nodes interact with each other: # ./mlc Intel(R) Memory Latency Checker - v3.8 Measuring idle latencies (in ns)... Numa node Numa node 0 1 2 3 0 76.7 159.8 198.1 208.2 1 150.2 79.6 194.1 203.6 2 195.7 205.3 77.6 161.9 3 192.5 203.5 149.3 79.3 Measuring Peak Injection Memory Bandwidths for the system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using traffic with the following read-write ratios ALL Reads : 111205.5 3:1 Reads-Writes : 107675.1 2:1 Reads-Writes : 108332.7 1:1 Reads-Writes : 91567.6 Stream-triad like: 104022.0 Measuring Memory Bandwidths between nodes within system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using Read-only traffic type Numa node Numa node 0 1 2 3 0 31591.2 18212.4 14887.9 14160.6 1 18748.3 31443.5 14796.8 14171.0 2 14965.8 14210.9 30433.0 18081.5 3 14961.6 14270.1 18111.1 31309.2 Measuring Loaded Latencies for the system Using all the threads from each core if Hyper-threading is enabled Using Read-only traffic type Inject Latency Bandwidth Delay (ns) MB/sec ========================== 00000 310.44 101637.3 00002 301.04 101744.6 00008 322.89 102328.3 00015 333.16 103578.6 00050 292.31 105430.2 00100 252.14 102857.1 00200 190.95 79047.3 00300 139.05 57093.7 00400 149.08 43520.2 00500 128.57 35536.9 00700 139.25 25450.3 01000 126.08 18124.3 01300 124.63 13950.7 01700 126.16 10748.6 02500 121.35 7475.3 03500 123.91 5374.5 05000 126.66 3895.5 09000 117.32 2418.6 20000 128.93 1341.4 Measuring cache-to-cache transfer latency (in ns)... Local Socket L2->L2 HIT latency 28.5 Local Socket L2->L2 HITM latency 31.8 Remote Socket L2->L2 HITM latency (data address homed in writer socket) Reader Numa Node Writer Numa Node 0 1 2 3 0 - 56.6 93.1 95.2 1 107.1 - 124.2 109.7 2 153.7 141.3 - 85.4 3 130.8 113.1 63.0 - Remote Socket L2->L2 HITM latency (data address homed in reader socket) Reader Numa Node Writer Numa Node 0 1 2 3 0 - 95.6 104.3 108.9 1 101.6 - 149.2 160.7 2 123.4 133.5 - 78.3 3 150.4 134.5 88.5 - This is my first system that actually implements NUMA Nodes, and I have done a lot of research into what NUMA Nodes are, and how they function, but I am at a loss why I am getting so many numa_miss and numa_foreign results. At this point I have many containers running, but no VM's active at this time. I am hoping to improve the NUMA performance within unRAID itself before I start trying to tackle VM NUMA configuration. Any advice would be greatly appreciated.
Archived
This topic is now archived and is closed to further replies.