The Black Bear - Threadripper 2990WX build


Recommended Posts

7 hours ago, skois said:

Mine goes to idle on logout fine, i have a ups connected and i can see the power and also on webui unraid shows 1-2% usage on pinned cpus.
i was more intested in the hyperthreadding part! Would this give more performance? 
after setting cores 6 threads 2 on xml. CPU-Z  reports 6 cores 12 threads. so i guess hyper T is on!

I believe it primarily helps with stuttering. Basically what is very important is for you to tell the VM as much as you can about the host topology. By doing that you optimize numerous parts of the system. When it "assumes" things I've found it generally assumes wrong sadly. It is why if you look at my config I have all the NUMA stuff there. So the VM knows where the boundaries are and for what cores.

Link to comment
On 1/7/2020 at 7:38 AM, Jerky_san said:

@testdasi@jbartlett

Good news! I recently discovered that we no longer need the EPYC fix =0~~. Cache all comes through right and identifies as threadripper.. This should also translate for Ryzen as well.

All you need to do is change it to something like below(of course adjusting cores). All cache shows up properly, CPUZ launches and everything, things seem slightly more responsive and latency tests show same as baremetal. Happy days!

 


  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='8' threads='2'/>
    <cache mode='passthrough'/>

VM

image.png.4242de805a864fd07dac72bf72188523.png

Can you install Microsoft's CoreInfo and see if it detects a hyperthreaded setup?

 

I use this Batch file for a quick report since it's a command line utility and the screen will close after running

Coreinfo.bat

@echo off
coreinfo.exe -ncs
pause

 

My Results with the EPYC hyperthreading workaround (weird numa mapping due to cross-numa node testing)

Coreinfo v3.31 - Dump information on system CPU and memory topology
Copyright (C) 2008-2014 Mark Russinovich
Sysinternals - www.sysinternals.com

Logical to Physical Processor Map:
**----------  Physical Processor 0 (Hyperthreaded)
--**--------  Physical Processor 1 (Hyperthreaded)
----**------  Physical Processor 2 (Hyperthreaded)
------**----  Physical Processor 3 (Hyperthreaded)
--------**--  Physical Processor 4 (Hyperthreaded)
----------**  Physical Processor 5 (Hyperthreaded)

Logical Processor to Socket Map:
************  Socket 0

Logical Processor to NUMA Node Map:
**----------  NUMA Node 0
--**********  NUMA Node 1
Press any key to continue . . .

 

Link to comment
1 hour ago, jbartlett said:

Can you install Microsoft's CoreInfo and see if it detects a hyperthreaded setup?

 

I use this Batch file for a quick report since it's a command line utility and the screen will close after running

Coreinfo.bat


@echo off
coreinfo.exe -ncs
pause

 

My Results with the EPYC hyperthreading workaround (weird numa mapping due to cross-numa node testing)


Coreinfo v3.31 - Dump information on system CPU and memory topology
Copyright (C) 2008-2014 Mark Russinovich
Sysinternals - www.sysinternals.com

Logical to Physical Processor Map:
**----------  Physical Processor 0 (Hyperthreaded)
--**--------  Physical Processor 1 (Hyperthreaded)
----**------  Physical Processor 2 (Hyperthreaded)
------**----  Physical Processor 3 (Hyperthreaded)
--------**--  Physical Processor 4 (Hyperthreaded)
----------**  Physical Processor 5 (Hyperthreaded)

Logical Processor to Socket Map:
************  Socket 0

Logical Processor to NUMA Node Map:
**----------  NUMA Node 0
--**********  NUMA Node 1
Press any key to continue . . .

 

Coreinfo v3.31 - Dump information on system CPU and memory topology
Copyright (C) 2008-2014 Mark Russinovich
Sysinternals - www.sysinternals.com

Logical to Physical Processor Map:
**--------------  Physical Processor 0 (Hyperthreaded)
--**------------  Physical Processor 1 (Hyperthreaded)
----**----------  Physical Processor 2 (Hyperthreaded)
------**--------  Physical Processor 3 (Hyperthreaded)
--------**------  Physical Processor 4 (Hyperthreaded)
----------**----  Physical Processor 5 (Hyperthreaded)
------------**--  Physical Processor 6 (Hyperthreaded)
--------------**  Physical Processor 7 (Hyperthreaded)

Logical Processor to Socket Map:
****************  Socket 0

Logical Processor to NUMA Node Map:
********--------  NUMA Node 0
--------********  NUMA Node 1

 

Edited by Jerky_san
Link to comment
1 minute ago, jbartlett said:

Ya know, you just invalidated 4 benchmark runs with 25 passes each run. At least you caught be before the next two runs! 😁

 

Might as well upgrade to 6.8.1 RC1 too. Though this would also mean a lot less edits in the XML every time I needed to use the GUI editor.

lol sorry 😃 I've been researching all the changes that QEMU and virtio have been doing.

Link to comment

CONFIRMED: On 6.8.1 RC1, I don't need the EPYC workaround to get hyperthreadding enabled but the TR does need the following CPU feature:

<feature policy='require' name='topoext'/>

 

I was also able to get CPU-Z to run with CPU passthrough and hyperthreading detected though it takes a bit at the 10%/Processors stage where in the past it apparently hung.

 

I'm going to look to see if HT support has already been recommended for Threadripper CPUs and request it if not.

Link to comment

Thanks everyone for the help! I Tweaked my VM XML and i got near bare metal single core cpu and reasonable multicore (since i only use 6 cores i cant really compare it with another tr2920x) score is a lil bit more than 50% of a real 2920x.
Also my latency of memory is at ~80ns (geekbench 4) 
I enabled channel mode memory interleaveing on bios, so i could also put the numa nodes tweaks.
 

Should i upgrade to 6.8.1 rc1 (kernel 4.19)? Or Threadrippers need the kernel 5.3?

Im on 6.8.0 rc7 right now!
i

Here is my XML for anyone might want it
 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='6'>
  <name>Windows 10</name>
  <uuid>1998758b-9ce5-e718-d030-bc8fc18b314a</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows_teamred.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>18874368</memory>
  <currentMemory unit='KiB'>18874368</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>12</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='3'/>
    <vcpupin vcpu='1' cpuset='15'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <vcpupin vcpu='3' cpuset='16'/>
    <vcpupin vcpu='4' cpuset='5'/>
    <vcpupin vcpu='5' cpuset='17'/>
    <vcpupin vcpu='6' cpuset='6'/>
    <vcpupin vcpu='7' cpuset='18'/>
    <vcpupin vcpu='8' cpuset='7'/>
    <vcpupin vcpu='9' cpuset='19'/>
    <vcpupin vcpu='10' cpuset='8'/>
    <vcpupin vcpu='11' cpuset='20'/>
    <emulatorpin cpuset='0,12'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0-1'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
    <memnode cellid='1' mode='strict' nodeset='1'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-4.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/1998758b-9ce5-e718-d030-bc8fc18b314a_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vpindex state='on'/>
      <synic state='on'/>
      <stimer state='on'/>
      <reset state='on'/>
      <vendor_id state='on' value='none'/>
      <frequencies state='on'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='6' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='disable' name='svm'/>
    <feature policy='disable' name='x2apic'/>
    <numa>
      <cell id='0' cpus='0-5' memory='9437184' unit='KiB'/>
      <cell id='1' cpus='6-11' memory='9437184' unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/disks/Samsung_SSD_970_EVO_Plus_500GB_S4EVNF0M862976E/Windows 10/vdisk1.img'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/disks/Samsung_SSD_970_EVO_Plus_500GB_S4EVNF0M862976E/Windows 10/vdisk2.img'/>
      <backingStore/>
      <target dev='hdd' bus='virtio'/>
      <alias name='virtio-disk3'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.160-1.iso'/>
      <backingStore/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <alias name='sata0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0xe'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0xf'/>
      <alias name='pci.8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.9'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='10' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='10' port='0x10'/>
      <alias name='pci.10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='11' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='11' port='0x11'/>
      <alias name='pci.11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='12' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='12' port='0x12'/>
      <alias name='pci.12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='13' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='13' port='0x13'/>
      <alias name='pci.13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='usb' index='0' model='qemu-xhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:5f:a7:7d'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-6-Windows 10/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0f' slot='0x00' function='0x3'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x43' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x43' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>

 

 

Link to comment
49 minutes ago, skois said:

Should i upgrade to 6.8.1 rc1 (kernel 4.19)? Or Threadrippers need the kernel 5.3?

I'm passing through my TR on 6.8.1 RC1 with no hacks or workarounds (but does need the topoext cpu flag) and getting hyperthreadding. I recommend upgrading. I haven't finished my benchmarking yet but am seeing some small improvements.

Link to comment
17 minutes ago, jbartlett said:

I'm passing through my TR on 6.8.1 RC1 with no hacks or workarounds (but does need the topoext cpu flag) and getting hyperthreadding. I recommend upgrading. I haven't finished my benchmarking yet but am seeing some small improvements.

Thanks ill consider upgrading in the next days and give it some tests.
Now i must learn how to build a kernel so i can include navi reset patch v2.
This  bug  annoys me a lot!

Link to comment
On 1/9/2020 at 11:12 PM, jbartlett said:

I'm passing through my TR on 6.8.1 RC1 with no hacks or workarounds (but does need the topoext cpu flag) and getting hyperthreadding. I recommend upgrading. I haven't finished my benchmarking yet but am seeing some small improvements.

Tested 6.8.1 stable all was working fine, didnt noticed anything weird. But when i launched to play a game. Oh boy.. From ~100fps i had on 6.8.0 rc7, on 6.8.1 i get 50fps while moving only forward. If i just look around just standing i go as alow as 10 fps. 
Reverted back to 6.8.0 rc7 and back on ~100 fps. 

I guess it might be the kernel. 6.8.1 has 4.19.x not 5.x.x

All this is on a Win 10 VM // 1909 update // 20.1.2 adrenalin driver.

 

Other than that is super stable

Link to comment
9 hours ago, skois said:

Tested 6.8.1 stable all was working fine, didnt noticed anything weird. But when i launched to play a game. Oh boy.. From ~100fps i had on 6.8.0 rc7, on 6.8.1 i get 50fps while moving only forward. If i just look around just standing i go as alow as 10 fps. 
Reverted back to 6.8.0 rc7 and back on ~100 fps. 

I guess it might be the kernel. 6.8.1 has 4.19.x not 5.x.x

All this is on a Win 10 VM // 1909 update // 20.1.2 adrenalin driver.

 

Other than that is super stable

It's the same kernel. 6.8.1 RC mainly only has fixes for the hypervisor. The 5.x kernel is being introduced in 6.9

Link to comment

Updates:

  • Bought another 3.84TB Samsung PM983 NVMe so all of my VM data is now PCIe-passed-through (e.g. no need to save data on UD share of SATA SSD).
  • I have been backing up my data offline to a 2.5" USB external (holding the Seagate BarraCuda 5TB i.e. SMR). It is filled to 95%, making it very slow (even slower than it has always been!) and creating the need to split my backup among multiple external drives. So I thought since I need to split my offline backup, I may as well solve the speed issue as well.
    • Now all my large capacity SATA SSD's (Samsung 860 Evo 4TB, Samsung 850 Evo 2TB and Crucial MX300 2TB) are in external enclosures serving as offline backup.
    • One of the i750 is in cache, the other one mounted as UD. I use both for write-heavy activities. 1 is at 94% reserve and 1 at 98% so still have some way to go.
    • I mounted the Seagate BarraCuda 5TB + 2 of my old Toshiba laptop HDD's (320GB and 80GB - yep they are still alive and kicking) as UD. The 2 laptop HDDs will be used as online backup for appdata, flash, vdisk etc. The 5TB will be used for infrequently-accessed static data (to keep them off my write-heavy SSD's).
  • I'm trying to resist the temptation to build another server to house my SATA SSD's as offline backup LOL.
  • Updated Unraid to 6.8.2. No problem to report.
Link to comment
6 hours ago, testdasi said:

Updates:

  • Bought another 3.84TB Samsung PM983 NVMe so all of my VM data is now PCIe-passed-through (e.g. no need to save data on UD share of SATA SSD).
  • I have been backing up my data offline to a 2.5" USB external (holding the Seagate BarraCuda 5TB i.e. SMR). It is filled to 95%, making it very slow (even slower than it has always been!) and creating the need to split my backup among multiple external drives. So I thought since I need to split my offline backup, I may as well solve the speed issue as well.
    • Now all my large capacity SATA SSD's (Samsung 860 Evo 4TB, Samsung 850 Evo 2TB and Crucial MX300 2TB) are in external enclosures serving as offline backup.
    • One of the i750 is in cache, the other one mounted as UD. I use both for write-heavy activities. 1 is at 94% reserve and 1 at 98% so still have some way to go.
    • I mounted the Seagate BarraCuda 5TB + 2 of my old Toshiba laptop HDD's (320GB and 80GB - yep they are still alive and kicking) as UD. The 2 laptop HDDs will be used as online backup for appdata, flash, vdisk etc. The 5TB will be used for infrequently-accessed static data (to keep them off my write-heavy SSD's).
  • I'm trying to resist the temptation to build another server to house my SATA SSD's as offline backup LOL.
  • Updated Unraid to 6.8.2. No problem to report.

Why not backup to gdrive?

Link to comment

Updates:

  • Found a great deal on a brand new Intel Optane 905p 960GB 2.5" U.2 NVMe SSD and made the jump. It's a great drive for boot + app + scratch disk due to extremely low latency (and therefore fast random IO). Sequential is actually slower than all my Samsung NVMe but that's not a big concern for its intended use.
    • Stubbed and passed through the 905p to the workstation VM and decided to reinstall Windows to re-optimise app + boot (boot drive used to the be the SM951 AHCI M.2).
    • Move all the "scratch" stuff off the 970 Evo to the Optane. Ideally I would want to have a separate Optane for this purpose but life is not perfect, I guess.
      • The Optane being boot + app + scratch is still faster than the 970 Evo being scratch-dedicated.
    • The SM951 is mounted UD to take some write-heavy duty off cache.
      • This old little M.2 has been rambo-ing through all my abuse over the years and spit it back to my face so we'll see how much Sylvester Stallone it still has in its tank.
    • Removed the Toshiba 80GB (again) to make room for the 905p.
  • My workstation workflow is kinda optimised as:
    • Data ingest onto 2x Samsung PM983 NVMe
    • Actively-being-worked-on is on the 970 Evo (for best speed as the PM983 is relatively slower with random IO).
      • By having scratch stuff on the Optane 905p, it actually makes the 970 Evo faster as mixed read-write is reduced.
    • Finished content go back to the 2x Samsung PM983 NVMe for storage
  • I completely forgot that the Gigabyte X399 Designare comes with a M.2 -> U.2 adapter and almost pulled the trigger on a new one. Fortunately, got my sense back in time and checked the box.
    • The U.2 connector blocks the Zotac 1070 Mini GPU heatpipe so it can't be used on the 1st M.2 slot.
    • It works perfectly on the 2nd M.2 slot. My Asus PCIe bifurcation card heat sink just touches the U.2 connector.
  • I'm now keeping an eye out for another good deal on another Optane.
    • The view is to replace the 970 Evo with an Optane, move the 970 Evo to cache and decommission either the SM951 or one of the two i750.
Edited by testdasi
Link to comment

Important updates re NVMe storage devices in a Threadripper Windows VM:

  • So I ran CDM 7.0.0g benchmarks for all my VM NVMe. I found the 905p latency for 4k Q1T1 to be about 60us (micro-seconds).
    • The number is more or less the same for both read, write and mixed (70-30) load. That is great because NAND SSD performance tends to dip quite a bit with mixed load, which tends to happen more frequently in real life workload.
    • This about half that of my best performance NVMe (the 970 Evo) i.e. twice as fast.
  • While doing this test, I also discovered how the 970 Evo kinda cheats the benchmarks. Write caching is allowed and turned on!
    • With write-caching disabled, write latency went from about 60us to around a whopping 2000us (2ms)!
    • With write-caching enabled, write latency is about 60us while read at about 120us and mixed at about 115us.
    • The 905p and PM983 do not allow enabling write-caching at all.
    • Of course I'll still enable write-caching for the 970 Evo as cheating or not, it does positively impact performance.
  • Those who read the 905p spec sheet will immediately notice 60us is more than 5x the headline latency of 11us.
    • I certainly don't expect headline performance in real life but my estimate says something closer to 16-17ms is reasonable.
      • That means my test is still 3.5x or so higher than the expected latency.
    • This is bugging me quite a bit. My goofu gave some clues:
      • Just Threadripper being Threadripper, there's a latency floor of about 40us - 60us due to vfio latency (source 1)
      • Disabling vapic may help but potentially need kernel 4.7 to support AMD AVIC (source 2)
        • I can test removing vapic hyperv tag but kinda am reluctant to downgrade to 6.8.3-rc7 for 5.x kernel. Need the Soon™ 6.9.0-rc1 😅
      • Removing Hyper-V from Windows may help with latency (source 3)
        • This, however, runs counter to my previous tests showing that Hyper-V helps with storage performance
        • Or perhaps, it's about uninstalling Hyper-V feature from Windows VM (and not a host xml parameter).
      • Installing Hyper-V role (feature) on Windows (Server) 2016 increases latency (lower IOPS) (source 4)
        • This is the only source that relates directly to the 905p + Threadripper (albeit 1950X)
        • The change in performance surprisingly (or not) matches my case.
      • I'm still using EPYC emulation so wondering if perhaps that plays a role.
  • TO DO LIST
    • Retest with Hyper-V feature uninstalled in Windows guest
    • Retest hyper-v on vs off in xml
    • Retest with vapic off
    • Retest without EPYC emulation
    • Just be happy that a bottlenecked 905p is still 2x faster than my fastest NAND NVMe.

 

Source 1: https://www.reddit.com/r/VFIO/comments/b5qyza/any_tips_on_reducing_vfiopci_latency/

Source 2: https://www.redhat.com/archives/vfio-users/2017-March/msg00005.html

Source 3: https://superuser.com/questions/1368076/severe-input-latency-lag-on-threadripper-2990wx

Source 4: https://social.technet.microsoft.com/Forums/en-US/13ff0892-38e2-4505-b923-c89caeafaaf8/hyperv-performance-hit-on-intel-optane?forum=winserverhyperv

Useful tool: convert IOPS to MB/s: https://wintelguy.com/iops-mbs-gbday-calc.pl

(note: IOPS is just 1/latency in seconds)

 

Edited by testdasi
Link to comment

More goofu over lunch about 905p in a VM

  • marktech.us reported something similar about 1.5 years ago. - Source 1
    • Performance in a VM (albeit Hyper-V and not KVM/qemu) drops by about 3.5x
    • 80% bare metal can be achieved with Q32T1.
    • This points towards 905p just having higher latency in a VM by default e.g. vfio latency again?
  • Reddit suggests to run qemu with fifo scheduler to reduce latency (15-50us albeit with Intel). - Source 2
  • That reminded of a recent topic on here which user uses vcpusched in the xml to reduce latency to 20us to 200us (again albeit with Intel) - Source 3
    • The parameters for vcpusched can be found on the official libvirt wiki - Source 4
    • The user reported rr to seems to work better than fifo
    • Probably will avoid the 2 commands in the post if possible. Particularly, kernel.sched_rt_runtime_us=-1 carries the risk of system lock up - Source 8
  • That brought me to another topic on here which also mentioned fifo optimisation, specifically for Threadripper - Source 5
    • This method changes the real time scheduler of the qemu process using chrt command
    • Details on chrt parameters - Source 6
    • Which priority to select (hint 99 is highest) - Source 7
    • The user, however, reported that this fix is not effective in reducing latency, unlike the vcpusched topic above.
  • UPDATED TO DO LIST
    • Retest with Hyper-V feature uninstalled in Windows guest
      • Not holding out high hope for this but it's the easiest quickest one to retest
    • Redo and retest xml using vcpusched, iothreadpin, and remove EPYC emulation
    • Retest with vapic off with hyper-v on
    • Retest hyper-v on vs off in xml
    • Just be happy that a bottlenecked 905p is still 2x faster than my fastest NAND NVMe.

 

 

Source 1: https://marktech.us/tag/optane/

Source 2:  https://www.reddit.com/r/VFIO/comments/6ze435/high_dpc_latency_in_guest/

Source 3: https://forums.unraid.net/topic/87948-windows-10-kvm-high-dpc-latency/?tab=comments#comment-817403

Source 4: https://libvirt.org/formatdomain.html#elementsCPUTuning

Source 5: https://forums.unraid.net/topic/84095-terrible-perfomance-on-threadripper/?tab=comments#comment-778928

Source 6: https://linux-tips.com/t/how-to-use-chrt-command/268

Source 7: https://stackoverflow.com/questions/8887531/which-real-time-priority-is-the-highest-priority-in-linux

Source 8: https://access.redhat.com/solutions/1604133

 

Edited by testdasi
Link to comment

More updates:

 

4k random Q1T1 average latency in micro seconds (read - write - mixed)

  • 905p
    • Bare metal: 26.54 - 31.03 - 27.20
    • VM: 59.99 - 64.73 - 63.13
    • Numa-aligned: 57.92 - 62.35 - 60.40
  • 970 Evo write cache enabled
    • Bare metal: 80.19 - 29.23 - 74.88
    • VM: 118.66 - 61.50 - 109.67
  • PM983
    • Bare metal: 111.90 - 36.39 - 91.22
    • VM: 133.52 - 60.47 - 114.69
  • 970 Evo write cache disabled
    • Bare metal: 80.63 - 1986.41 - 716.47
    • VM: 117.98 - 1940.51 - 685.75

Comments

  • We all know latency isn't a strength of the Threadripper platform. Now we can have an idea of the extent.
    • The PCIe access latency floor for Threadripper is about 30us BM and 60us in VM.
    • This can be seen in the write figures. Write that hits low-latency medium (e.g. Optane or DRAM write cache) will be bottlenecked by this latency floor.
  • None of the tuning considered in the previous posts made any difference at all. In fact, turning on FIFO even slows things down!
  • The only tweak that makes a difference to the 905p performance is to align the cores CDM uses to the same numa node that the 905p is connected to. I did this by using Process Lasso. What isn't shown on the data is the consistency of the numa-aligned results.
    • I ran it multiple times and the results are within 0.03us of each other.
    • Non numa-aligned results can vary within about 3us!
  • I can't make up my mind how I should view the BM vs VM diff for the 905p. The overhead is "only" 30us but that is still a 2x performance ratio.
    • At higher queue depths (starting from Q4T1), the BM:VM performance ratio converges to about 1.5x (or 1.3x for numa-aligned VM).

 

Doing all these benchmarks sort of opened a can of worms.

  • I'm now considering running Windows 10 bare metal and Unraid as a VM under a type 2 hypervisor (e.g. VMWare Workstation).
  • I should then be able to use Process Lasso to artificially "pin" the no-RAM numa cores to the Unraid VM, which I probably use mainly for slow storage and dockers.
  • Then perhaps I'll eventually move stuff from Unraid dockers to Windows dockers, leaving Unraid purely as a NAS solution.

A rather large can of worms indeed!

 

 

 

 

Edited by testdasi
Link to comment

So the worms are crawling out of the can.

I'm writing this post with Unraid running in VMWare workstation in the background serving Plex from the cloud.

Things are still in the process of getting sorted but what I've found so far:

  • I wasted half a day trying and tweaking VirtualBox first but gave up.
    • Storage performance was terrible. 50MB/s dd from within the server.
    • Network performance was terrible. 10-20MB/s!
    • Cannot minimise the VM window to system tray. This feature was requested like a decade ago and still yet not implemented.
  • So next comes VMWare Workstation.
    • I qemu-img convert the cache drive to a vmdk image..
      • Dockers are all running fine.
      • Storage performance is great (can reach GB/s NVMe).
    • I turned on nested virtualisation and "HVM" is now enabled so theoretically I can run a nested VM but find no need to.
      • My main Linux "Macbuntu" VM is also running on VMWare.
    • Network performance is peculiar (write about 80MB/s so giabit-ish but read is like 500MB/s).
    • Annoyance: Unraid partition alignment sort of messes up Windows ability to make drives offline so I end up with a catch 22.
      • A drive formatted with Unraid can't be attached to the VM (because it can't be made offline properly, to be exact, the partition still mounts)
      • A drive that is formatted with Windows can be attached to the VM but doesn't work with Unraid (once reformatted).
      • Probably will have to do vmdk for my slow storage but we'll see how it goes.
  • My 905p somehow decides to drop latency a further 5ms down to 25ms.

 

 

 

 

 

Link to comment

So after running it for a while, below are the Pros and Cons (with workaround suggestions) of running Unraid in VMWare Workstation.

I am now debating moving back to the way it was LOL.

 

Premise: No n-gamer-1-PC scenario   -   Unraid wins, end of in this scenario.

 

Pros:

  • No need to mess about with PCIe pass-through.
    • Have a single GPU and have trouble passing it through to the VM?
    • Have a Navi GPU that just refuses to work in a VM?
    • Have strange audio issues that can't seem to be fixed?
  • No VM limitation for the machine that matters
    • USB device disconnecting?
    • VM overhead affecting performance?
    • Disappointed with your VM fps variance / stuttering?

 

Cons:

  • No physical drive pass through   -   have to use vmdk vdisk
    • That means no protection by isolation e.g. a Windows cryptovirus can encrypt the entire vmdk
  • Drive size limited to 8TB   -   can break large drives into multiple smaller vdisks but parity sync / check will be terrible.
  • Network access is limited to gigabit (i.e. cap at 125MB/s) regardless of interface
    • This is a VMWare limitation so no workaround
  • No SMART monitoring   -   need to use 3rd-party software e.g. HDSentinel
  • Cannot run Hyper-V based software at the same time
    • Docker for windows is also affected as it requires Hyper-V
    • Another VMWare limitation so no workaround. In fact, it requires this command in PowerShell to work at all.
      bcdedit /set hypervisorlaunchtype off
  • No CPU core pinning on host   -   can use Process Lasso to set core affinity for vmware-vmx.exe process

  • Additional cost of VMWare Workstation.
    • I think it would work on VMWare Player but can't confirm as VMWare does not allow running both softwares in the same machine for me to test.
  • Need plob / rEFInd iso files to boot Unraid from USB stick.
    • VMWare does not support booting from USB device (but can connect USB device to VM). Note: I can't set it to connect automatically so have to manually connect at boot.
    • While it's possible to boot Hyper-V VM from a USB device, this is done via the disk controller i.e. the GUID isn't sent to Unraid.
  • Restart (e.g. due to Windows update) will require rebooting the server
    • Need to run task scheduler as system user (e.g. using PSTools) to disable the 2 reboot tasks

 

 

 

Edited by testdasi
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.