November 27, 201411 yr I have 4 Xen VMs running and one of them is a Windows 7 Media Center with a Ceton ETH6 TV tuner used to serve three Media Center Extenders. I have been having performance issues when I run multiple VMs. The video and audio on the MCE would pause and stutter. If I stopped all but the media center VM, there would be no stutter. I have found through some research and experimentation that I can fix this with some CPU pinning and managing of the CPUs. I have a Xeon 4 core processor that supports Hyper threading, so I have a total of 8 CPUs. If I don't manage the CPUs, all 8 will go to Dom0 and Dom0 will manage the CPUs for the client VMs. The management of the CPUs comes with overhead that appears to have an effect on the performance of the VMs. Most of the time this is not noticeable in a VM, but is causing a problem with WMC when streaming live TV. I ended up pinning one CPU to Dom0 and then managing the rest of the CPUs for each client. The following is what I put in my syslinux file to pin one CPU to Dom0: append / dom0_max_vcpus=1 dom0_vcpus_pin --- /bzimage --- /bzroot When unRAID is booted, CPU 0 is pinned to Dom0. I then added the following to my Windows 7 Media Center VM: vcpus = '2' cpus='1-2' This will pin CPUs 1 and 2 to the Windows 7 VM. I added the following to the rest of the VMs; vcpus = '2' cpus='all,^0-2' This assigns 2 VCPUs and they can be any of the remaining CPUs, but not 0, 1, or 2. Dom0 will not assign CPUs 0, 1, or 2 to any clients. In a telnet session you can see the assignment of your CPUs by the command 'xl vcpu-list' resulting in the following output: Name ID VCPU CPU State Time(s) CPU Affinity Domain-0 0 0 0 r-- 39214.4 0 DebianVM 18 0 3 -b- 382.3 3-7 MediaCenterVM 19 0 2 -b- 1694.6 1-2 MediaCenterVM 19 1 1 -b- 1620.0 1-2 DesktopVM 20 0 7 -b- 410.2 3-7 DesktopVM 20 1 5 -b- 272.3 3-7 BasementVM 21 0 6 -b- 59.4 3-7 BasementVM 21 1 7 -b- 59.5 3-7 Windows10VM 22 0 4 -b- 484.6 3-7 Windows10VM 22 1 3 -b- 385.3 3-7 You can of course assign more or less VCPUS to the VM as you feel necessary. I would suggest that you should not over assign VCPUS. Assigning too many will result in a lot of CPU switching by Xen and will result in a performance hit. I can run a parity check at full speed and there are no issues with the Windows 7 VM or any other VMs with the parity check running. EDIT: You can also adjust the priority of Dom0 using: /usr/sbin/xl sched-credit -d 0 -w 512 This will increase the priority of Dom0. EDIT: It's been pointed out that there may be a performance issue when splitting cores. When Hyper Threading is enabled, one core has two threads sharing cache resources. Each thread is a CPU to Dom0. If only one CPU of the core is pinned, the other is managed by Dom0 and is included in the pool of CPUs available for other VMs. It might be best to be sure both CPUs of one core are pinned. In my case I only needed to pin three CPUs so CPU 3 was in the VM CPU pool. This would cause context switching on core 1 CPU 3 causing a performance hit on core 1 CPU 2. I've decided to pin two CPUs to Dom0 (0-1), two to the Windows VM (2-3) and the rest in the CPU pool to all the other VMs. The CPU assignment will then be: Name ID VCPU CPU State Time(s) CPU Affinity Domain-0 0 0 0 -b- 173.7 0 Domain-0 0 1 1 r-- 147.7 1 DebianVM 1 0 5 -b- 9.5 4-7 MediaCenterVM 2 0 2 r-- 109.1 2-3 MediaCenterVM 2 1 3 -b- 89.9 2-3 DesktopVM 3 0 6 r-- 48.7 4-7 DesktopVM 3 1 4 --- 31.3 4-7 BasementVM 4 0 7 -b- 35.4 4-7 BasementVM 4 1 5 -b- 37.7 4-7
November 28, 201411 yr Author Over assigning CPUs to any VM is not a good idea. There is a cost for each CPU context switch. Not only does the CPU state have to be saved and restored, but the cache(s) also need to be saved and restored. The caches in todays processors can be quite large. There is a cost in time to do that. Minimizing these context switches and even eliminating them by pinning CPUs can really improve performance. Dom0 does not really need any more that one CPU. Some might think this is not enough, but it is sufficient. I have a few Dockers running at the same time as the 4 VMs without any performance issues. The better approach is to pin one CPU to Dom0 and then one or two VCPUs to each VM. A VM really does not need any more than that. Pinning CPUs to a time sensitive VM like I have done here with my Windows 7 WMC can improve the performance for that VM with no detrimental affect on other VMs.
November 28, 201411 yr How do the virtual CPUs relate to the actual cores? Are vCpu 0 and 1 related to the first actual physical Core? Is it every 2 virtual Cpu to one physical Cpu core? Or does it map vCpu 0 and vCpu 4 to the same physical core? Right now its highly likely that you are not getting the best performance that you can. You are forcing some physical cache core thrashing when you have one physical core split between 2 tasks (Dom 0 and WinVM). If so then you would want to keep the physical cores grouped together since they share physical cpu cache lines. Given that, you should see even better performance and less cache thrashing to pin vCpus 2 and 3 to your Windows VM. Or which ever vCpu mapping priced the same full physical core.
November 28, 201411 yr Author How do the virtual CPUs relate to the actual cores? Are vCpu 0 and 1 related to the first actual physical Core? Is it every 2 virtual Cpu to one physical Cpu core? Or does it map vCpu 0 and vCpu 4 to the same physical core? Right now its highly likely that you are not getting the best performance that you can. You are forcing some physical cache core thrashing when you have one physical core split between 2 tasks (Dom 0 and WinVM). If so then you would want to keep the physical cores grouped together since they share physical cpu cache lines. Given that, you should see even better performance and less cache thrashing to pin vCpus 2 and 3 to your Windows VM. Or which ever vCpu mapping priced the same full physical core. Yea. I'm still pondering this. Actually because CPU 0, 1, and 2 are pinned (core 0 and half of core 1), I think the issue is with CPU 3 because it is not pinned and is split between the Windows VM and other VMs. I'm going to experiment more with this.
November 28, 201411 yr hmmm odd. I decided to pin nothing. Dom-0 and Dom-1 (Arch w/ Plex and transmission) work perfectly and I haven't noticed any performance problems. I ran a "stress test" of three full rate BD rip transcodes + parity sync without a stumble. I know that doesn't mean it works for every other use case, just saying it can work and Xen does a decent enough job self managing left to its own devices.
November 28, 201411 yr Author Your generalization may not be valid. It depends on how many VCPUs you have compared to CPUs in the system, how they are assigned when Dom0 starts, which ones are shared, and if shared, which VMs are sharing the CPUs. If you don't have more VCPUs assigned to VMs as the number of CPUs you have in your system, there will not be any CPU context switching. i.e. if you have 8 CPUs in your system and less than or equal to 8 VCPUs assigned, there won't be any performance issues because Dom0 doesn't have to share any CPUs. You also might not have any CPUs being shared with any VMs that make any difference. It also depends on how CPUs are assigned as the VMs are started. At a telnet session use the command 'xl vcpu-list' and look at the output. You can see which VMs are using which CPUs and if there are any CPUs being shared and which VMs are sharing the CPUs. The strategy proposed here is not necessary for Xen to perform properly, it just controls the assignment of CPUs in a controlled fashion so the results are predictable.
November 28, 201411 yr root@Tower:~# xl vcpu-list Name ID VCPU CPU State Time(s) CPU Affinity Domain-0 0 0 5 -b- 79255.4 all Domain-0 0 1 4 -b- 23310.8 all Domain-0 0 2 1 -b- 20357.6 all Domain-0 0 3 0 -b- 19034.6 all Domain-0 0 4 7 r-- 16085.0 all Domain-0 0 5 0 -b- 17803.9 all Domain-0 0 6 2 -b- 18040.7 all Domain-0 0 7 3 -b- 18301.8 all arch 27 0 3 -b- 52.2 all arch 27 1 5 -b- 52.2 all arch 27 2 1 -b- 45.7 all arch 27 3 6 -b- 79.8 all arch 27 4 4 -b- 100.9 all arch 27 5 7 -b- 59.0 all arch 27 6 4 -b- 57.4 all arch 27 7 6 -b- 51.9 all
November 28, 201411 yr Author root@Tower:~# xl vcpu-list Name ID VCPU CPU State Time(s) CPU Affinity Domain-0 0 0 5 -b- 79255.4 all Domain-0 0 1 4 -b- 23310.8 all Domain-0 0 2 1 -b- 20357.6 all Domain-0 0 3 0 -b- 19034.6 all Domain-0 0 4 7 r-- 16085.0 all Domain-0 0 5 0 -b- 17803.9 all Domain-0 0 6 2 -b- 18040.7 all Domain-0 0 7 3 -b- 18301.8 all arch 27 0 3 -b- 52.2 all arch 27 1 5 -b- 52.2 all arch 27 2 1 -b- 45.7 all arch 27 3 6 -b- 79.8 all arch 27 4 4 -b- 100.9 all arch 27 5 7 -b- 59.0 all arch 27 6 4 -b- 57.4 all arch 27 7 6 -b- 51.9 all You only have 1 VM - I have four. That's not an equal comparison. You have good performance because of the number of processors total being shared between VMs. You would be better off pinning 2 CPUs to Dom0 and let Arch have the rest. You have Dom0 doing a lot of unnecessary context switching. Albeit your performance is good, but it's serious overkill.
November 29, 201411 yr fair enough on comparing you and me. But I disagree I would be better pinning 2 to dom-0. Not least because I tried it and it got me nothing extra except for depriving Arch (with plex) of 2 CPUs which limits transcoding speeds. In the meantime unraid gained nothing except being limited to only 2 (not that it needed more). No extra stability, not faster parity ... There is only excessive context switching if Xen isn't doing its job well enough. At this point, after discussions on the Xen mailing list I'm unlikely to trust myself more than Xen. The one situation where Xen suggests pinning for dom-0, heavy IO, my email to them on the topic generated a "to-do" item to reconsider that wording because they felt it was too strong. http://lists.xen.org/archives/html/xen-users/2014-05/msg00181.html Obviously it is all situation and voodoo dependent. I figure I'll let Xen do it's thing until I have a reason to consider otherwise.
November 29, 201411 yr It should also be pointed out that AMD cpus do not have SMT / Hyperthreading, they have CMT so the benefits of pinning vCPUs and cache thrashing on AMD CPUs is different.
Archived
This topic is now archived and is closed to further replies.