Why are my OSX VMs so sluggish (Threadripper)

[meep]

I'm at a bit of a loss here and hoping for some pointers. Rather than immediately post config files etc., I thought I'd outline my predicament and can respond with specific info if people take the time to make suggestions......

 

I've been successfully running Windows and OSX VMs with satisfactory performance on an AMD FX based system for several years. (8x cores, 32GB RAM). I recently upgraded to a Threadripper 2950X based system with 64Gb to allow me run more simultaneous VMs and/or dedicate more resources to discrete VMs and hopefully benefit from a speed bump.

 

In use, however, it's become clear that my VMs have reduced, (or perceived reduced)  performance on the new system. I'm really working with OSX only at the moment but here's what I observe;
 

• General sluggishness in the OS.
• Longer than expected delay when clicking on menus.
• Longer than expected application load times.
• Slower than expected network performance
• All assigned CPUs pinned at 100% for several hours after boot (they calm down after that until next reboot)

 

While just about usable, It's generally an underwhelming experience, and a lot less than I would have expected.  I migrated a couple of Mojave VMs from my old system, assigned more CPU cores and memory but overall, the in-OS experience feels generally slower and less satisfactory. (booting time etc. is acceptable, if not stellar).

 

My VMs run from either a dedicated SSD, or an image file on my cache SSD. (nothing on the array). I have 24GB memory assigned and 8x CPUs (from the 32 available on the 2950X). The CPUs are isolated, and pinned to the VM only. I have GPU and dedicated USB cards passed through. One VM runs on a GTX 770, the other on an AMD 570, though which GPU I use appear to have negligible impact on performance - both show Metal support.  Regardless of whether I run a single VM or 2x simultaneously, the same issues apply.

 

Here's a few things I've tried;

• Installed a completely new Catalina VM using MacInABox. No real difference.
• Updated from 6.7.x to 6.8.rc5 - this fixed an issue I had with VM use seeming to cause my system to crash / shut down. I thought I saw a marked improvement in my fresh Catalina VM after this update, but unfortunately, I'd made my disk image too small to transfer my Mojave user account & files. After re-createding the VM with a bigger disk image, it was back to uninspiring performance. 
• I've tried to get my head around Numas and tweaked assigned CPUs a bit, with no real affect.
• I've read all the threads around improving VM performance etc. but don't see anything obvious.
• I've switched the network type in my VM XML, but no joy

 

So, I'm at a bit of a loss. One thing I'm unsure of is whether of not my CPU is actually throttling up. It's a 3.5Ghz rated speed and I believe I've got my bios and unRaid set up to allow it scale up, but I'm not sure where I can see if that's working in UnRaid.

 

I've been using unRaid and VMs for several years and like to think I'm reasonably knowledgable. However, I'm just not sure where to focus my efforts 0- MB Bios?, CPU config?, unRaid boot config?, VM config? somewhere else? Part of my problem, I think,  is a degree on unfamiliarity with the TR platform and how I might optimise it

 

I hope the above adequately explains my observations. I've refrained from posting a bunch of XML and logs etc. as I'm just not sure which ones to focus on, but I can and will if some kind soul has any thoughts on a direction of attack.

 

Here's some system highlights:

CPU: Threadripper 2850X (16 cores, 32 threads)

MB: Taicahi X399

RAM: 64GB

OS: Unraid 6.8rc5

 

Many thanks for reading!

 

 

 

 

 

[meep]

So here's the ugly truth;

 

 

That's the pitiful Geekbench score I obtained having spent the weekend reading all the threads on OSX VM Optimisation, CPU pinning, Performance Enhancements etc.  This VM will often cause UnRaid to report the CPUs at full throttle, yet inside the VM, not a whole lot is happening;

 

 

 

Here's my setup, maybe someone can spot something.....

 

CPU: Threadripper 2850X (16 cores, 32 threads)

MB: Taicahi X399

RAM: 64GB

OS: Unraid 6.8rc6

VM: OSX Catalina 10.15.1

 

My Bios is updated to the latest (3.80) and all the bios VM optimised settings are turned on, as far as I can see. Memory is set to channel and here's my LSTOPO

 

 

 

Unraid is configured to boot as follows;

 

default menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 30
label Unraid OS
  menu default
  kernel /bzimage
  append isolcpus=4-7,12-15,20-23,28-31 vfio-pci.ids=1b73:1100,dd01:0003,1412:1712 pcie_aspm=off kvm_amd npt=1 nested=1 avic=1 initrd=/bzroot
label Unraid OS GUI Mode

 

The VM Configuration is;

 

Which looks like this in XML;

 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='18' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>MacinaboxCatalina</name>
  <uuid>faaaf6f5-246f-4a88-bf26-111de687651f</uuid>
  <description>MacOS Catalina</description>
  <metadata>
    <vmtemplate xmlns="unraid" name="MacOS" icon="/mnt/user/domains/MacinaboxCatalina/icon/catalina.png" os="Catalina"/>
  </metadata>
  <memory unit='KiB'>25165824</memory>
  <currentMemory unit='KiB'>25165824</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <iothreads>1</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='20'/>
    <vcpupin vcpu='2' cpuset='5'/>
    <vcpupin vcpu='3' cpuset='21'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='22'/>
    <vcpupin vcpu='6' cpuset='7'/>
    <vcpupin vcpu='7' cpuset='23'/>
    <emulatorpin cpuset='0,16'/>
    <iothreadpin iothread='1' cpuset='0,16'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-4.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/mnt/user/domains/MacinaboxCatalina/ovmf/OVMF_CODE.fd</loader>
    <nvram>/mnt/user/domains/MacinaboxCatalina/ovmf/OVMF_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'/>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writeback'/>
      <source file='/mnt/user/domains/MacinaboxCatalina/Clover.qcow2'/>
      <backingStore/>
      <target dev='hdc' bus='sata'/>
      <boot order='1'/>
      <alias name='sata0-0-2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source dev='/dev/disk/by-id/ata-KINGSTON_SV300S37A120G_50026B775B0777DE'/>
      <backingStore/>
      <target dev='hdd' bus='sata'/>
      <alias name='sata0-0-5'/>
      <address type='drive' controller='0' bus='0' target='0' unit='5'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source dev='/dev/disk/by-id/ata-TS64GSSD340_20140731B61710144112'/>
      <backingStore/>
      <target dev='hdf' bus='sata'/>
      <alias name='sata0-0-3'/>
      <address type='drive' controller='0' bus='0' target='0' unit='3'/>
    </disk>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x8'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:ae:04:3c'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='e1000-82545em'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-18-MacinaboxCatalina/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
  <qemu:commandline>
    <qemu:arg value='-usb'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='usb-kbd,bus=usb-bus.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='isa-applesmc,osk='/>
    <qemu:arg value='-smbios'/>
    <qemu:arg value='type=2'/>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='IvyBridge,vendor=GenuineIntel,+hypervisor,-erms,+invtsc,kvm=on,+topoext,+svm,+invtsc,+fma,+mmxext,+avx,+avx2,+aes,+xsave,+xsaveopt,+ssse3,+sse4_2,+popcnt,+sse4a,+bmi1,+bmi2,+arat,+abm,+3dnowprefetch,+adx,+clflushopt,+cr8legacy,+fsgsbase,+fxsr_opt,+misalignsse,+movbe,+osvw,+pclmuldq,+pdpe1gb,+rdrand,+rdseed,+rdtscp,+sha-ni,+smap,+smep,+svm,+vme,+xgetbv1,+xsave,+xsavec,+clwb,+umip,+topoext,+perfctr-core,+amd-ssbd,+wbnoinvd'/>
    <qemu:arg value='-overcommit'/>
    <qemu:arg value='cpu-pm=on'/>
  </qemu:commandline>
</domain>

 

Setup in Clover as iMacPro1,1, with the following kext versions;

 

 

 

 

Here's some info from the running system;
 

 

 

 

Very frustrating to have such a dog slow experience. All help & insights appreciated.

 

 

 

 

 

 

 

 

 

Edited November 17, 2019 by meep
additional info

[meep]

Hmm, so I ran this in unRaid command line;

 

watch -n 1 grep MHz /proc/cpuinfo

 

This doesn't look right for a 3.5GHz, does it? CPU under load (VM running benchmakrs on 8x coes);

 

 

Guess I need to track this issue down.

[testdasi]

Try installing the Tips and Tweaks plugin, and use it to tweak Unraid to turn on boost + change CPU profile to Performance.

 

Also 500 MHz seems exceptionally low (my TR4 idles in the GHz) so perhaps you have some power saving thingie in the BIOS turned on.

[meep]

33 minutes ago, testdasi said:

Try installing the Tips and Tweaks plugin, and use it to tweak Unraid to turn on boost + change CPU profile to Performance.

 

Also 500 MHz seems exceptionally low (my TR4 idles in the GHz) so perhaps you have some power saving thingie in the BIOS turned on.

Thanks for the input. I already had T&T installed and have tried all the profiles already. I've also been through the BIOS several times and everything is pretty much on auto.

 

This is starting to look like a Thermal Throttling issue. With unRaid idling (no load), the CPU cores hover around 3GHz, but as soon as some load is applied (booting/running my VM, for example), all the cores immediately drop to ~500Hz and stay there until activity ceases.

 

My GoogleFu finds similar reports for TRs that are either under-powered or exceeding temps. unRaid reports CPU temps between 55-60 degrees Celsius, so not too excessive. Though I do note I've only got the 8pin MB power connected on my TaiChi X399 - there's another 4 pin connector idle. I jury-rigged a second PSU into that last night and with MB getting full power, the TR temps shot right up. I'm guessing the CPU got an extra shot of power and started operating properly but this exacerbated whatever thermal problems it has.

 

I currently have an AIO water cooling solution on the CPU but I really don't trust it. I have a hefty air cooler ready to install. I've ordered a beefier PSU and will swap both items in when that arrives. Hopefully this will solve any underpower and overheating issues.

 

 The silver lining is that I've become very familiar with OSX VM XML configuration! 😵

 

 

[testdasi]

14 minutes ago, meep said:

unRaid reports CPU temps between 55-60 degrees Celsius, so not too excessive... the TR temps shot right up...

 

Could very well be thermal throttling.

• 55-60 is excessive. My rig, even at full load + Precision Boost On is in the low 50s.
• My rig temp change under load is also gradual, or at least certainly not qualify for "shot right up".

 

 

[meep]

Good news, (for me).

 

I replaced my AIO cooler with an air cooler and unraid now reports idle temps at a positively icy 17 degrees. Booting my VM sees this rise to 18 degrees and cores are no longer throttling. 

 

Geekbench results inside the vm have increased fourfold. 

 

Either my AIO was faulty, or it had been incorrectly fitted. In any case, I now have my expected system perfromance. 

 

Yay! 

[testdasi]

14 minutes ago, meep said:

Good news, (for me).

 

I replaced my AIO cooler with an air cooler and unraid now reports idle temps at a positively icy 17 degrees. Booting my VM sees this rise to 18 degrees and cores are no longer throttling. 

 

Geekbench results inside the vm have increased fourfold. 

 

Either my AIO was faulty, or it had been incorrectly fitted. In any case, I now have my expected system perfromance. 

 

Yay! 

Good to hear.

 

Just out of curiosity, your Mac VM, was it just standard MacinaBox + pass through GPU?

[bastl]

19 hours ago, meep said:

I currently have an AIO water cooling solution on the CPU but I really don't trust it.

Please tell me it was an Enermax Liqtech 😂

 

I had nearly the same issues a year ago after using the 360 from Enermax for 2 months. Remember the first gen gunked up after a couple months? Basically there wasn't any waterflow anymore. One corner of the rad became extremly hot and the CPU under even extrem low loads throttled down to 500-600MHz all the time.

[meep]

3 hours ago, testdasi said:

Good to hear.

 

Just out of curiosity, your Mac VM, was it just standard MacinaBox + pass through GPU?

Yes, though I did a lot of manual reconfiguration of the XML as I went along  as my first point of attack was the VM config.

 

I have other Mojave and Windows VMs, but I decided to focus attention on this one as it's the one the family use and there I was under most pressure to 'fix'

 

[meep]

3 hours ago, bastl said:

Please tell me it was an Enermax Liqtech 😂

 

I had nearly the same issues a year ago after using the 360 from Enermax for 2 months. Remember the first gen gunked up after a couple months? Basically there wasn't any waterflow anymore. One corner of the rad became extremly hot and the CPU under even extrem low loads throttled down to 500-600MHz all the time.

Yup.

 Though I believe it's the VII.

 

I purchased the CPU, Cooler and Motherboard 'slightly used' and all 3 were connected when i dropped them into my case. I should have checked, of course, but I saw them running before I bought and there were no obvious problems. The AIO might still be good, I just don't want to be pfaffing around with it, especially given the prevalence that the Gen1 had for failing.

 

There's a BeQuiet Dark rock Pro TR4 in there now and so far, so good. Hopefully the CPU itself did not get too stressed with all of this.

 

[bastl]

@meep there shouldn't be an issue with your CPU if the auto downclock safety mechanics did their job. For me I only noticed it as I tried to play a game after 2 months without stressing the CPU really. Only did some small office workloads during that time. Can't really tell for how long I had a temperature issue, but since than 18 months later with a Noctua air cooler the system is still rock stable overclocked to 4GHz 😉

[DayspringGaming]

I'm sitting at 29C at idle on a 1920x stock with a Dark Rock Pro TR4. 

Wonder what my OC temps would be - I've been hesitant to overclock it for right now.

[bastl]

@DayspringGaming At idle with dockers and 2 VMs up running it usually sitting in the low 30's for me. Using my main VM for browsing the web, media onsumption etc. the temps go up to 40-45°C and if I play games the temps go up to 50+ depending on the game scretching the 60°C. On stress tests under full load what I never see on a daily basis it goes into the high 60

[testdasi]

3 hours ago, DayspringGaming said:

I'm sitting at 29C at idle on a 1920x stock with a Dark Rock Pro TR4. 

Wonder what my OC temps would be - I've been hesitant to overclock it for right now.

Don't overclock TR. Just use Precision Boost. Because it raises clock (and voltage) on-demand, it will run cooler and consume less energy. I have found it to be superior than any effort to manually overclock.

 

47 minutes ago, bastl said:

@DayspringGaming At idle with dockers and 2 VMs up running it usually sitting in the low 30's for me. Using my main VM for browsing the web, media onsumption etc. the temps go up to 40-45°C and if I play games the temps go up to 50+ depending on the game scretching the 60°C. On stress tests under full load what I never see on a daily basis it goes into the high 60

That's still a bit hot I think. I'm writing this post while transcoding on half of the cores in the background and I'm at 38 (with whisper-level fan noise). You might want to change the fan curve in your BIOS a little bit to get a bit more juice out of Precision Boost. (or maybe your temp was already with Precision Boost tuned up, which is then normal  ).

[bastl]

@testdasi Not sure how different the "precision boost" works on your 2nd gen, but on first gen on stock the CPU runs at 3.4GHz with boost up to 4GHz depending on the temperature. On a bare metal system I rarely saw that 4GHz. Btw only 1 core boosts up to that speed. Most of the cores sitting around 3.7GHz and 2-4 up to 3.9GHz for a short period under load. The vcore jumpes around like crazy during all the boosting up to 1.6V on stock. The temperatures under full load stress tests on stock are higher and reaches the max what ends up in throttling the CPU. 68°C is the point where it starts throttling. On my all core 4GHz 1.275V OC I never get to that temperature, where as on stock it will happen. Keep in mind on stock only 1 core boosts up to 4GHz. XFR on first gen was a mess and I think nothing really change since than with BIOS updates for that chip. The used vcores on stock are way to high  at least on my system. I wrote about it a couple times that maybe I was lucky with my chip running stable 4GHz at 1.275V. Most reviews from back than used 1.45 and somtetimes 1.5V to get it stable at 4GHz. And also to be mentioned is on stock Unraid only boosts up to 3.7GHz on my chip and starts to throttle down because of the high temps it reaches. Fan curves are tuned so I can't really hear the fan on daily use. I've set it up to kick in more aggressively at around 48°C. Thats the area where in my test for more heavy loads like gaming I need more cooling. Temperatures in the 40's doesn't hurt any CPU even 50 isn't really high.

 

46 minutes ago, testdasi said:

or maybe your temp was already with Precision Boost tuned up, which is then normal

46 minutes ago, testdasi said:

Don't overclock TR. Just use Precision Boost. Because it raises clock (and voltage) on-demand, it will run cooler and consume less energy. I have found it to be superior than any effort to manually overclock.

As soon as you dial in some manual OC settings, precision boost will be dissabled on 1st gen. Keep in mind XFR was extremly improved on seceond gen chips and makes OC obsolete. On first gen it had a lot of issues like explained the extreme aggressive vcore curves.

Edited November 23, 2019 by bastl

[testdasi]

On 11/23/2019 at 12:30 PM, bastl said:

As soon as you dial in some manual OC settings, precision boost will be dissabled on 1st gen. Keep in mind XFR was extremly improved on seceond gen chips and makes OC obsolete. On first gen it had a lot of issues like explained the extreme aggressive vcore curves.

Ahh, sorry I completely forgot about 1st gen vs 2nd gen. My 2990WX used to swing quite hot with early BIOS too. Ironically, AMD recent BIOS went a bit too far the other way so there isn't enough voltage leading to instability. I had to tune it up a bit LOL.