jbartlett

Here's the controller link report for the 0000:08:00.0 device 08:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05) Capabilities: [68] Express (v2) Endpoint, MSI 00 LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- I have a lookup table that for PCI3 with a transfer rate of 8 and a width of 8 returns "7.88 GB/s". It looks like this reference table is in error. Based on information from https://paolozaino.wordpress.com/2013/05/21/converting-gts-to-gbps/ I should be able to compute it. The transfer rate of 8 GT/s identifies it as a PCI 3 controller. 8 GT/s multiplied by 8 lanes equals 64 GT/s, multiplied by the overhead (LineCodeL/LineCodeH) of 128b/130b equals 63.02 GB/s.

@icemansid - Please upload/email a debug file from the DiskSpeed app (link at the bottom of the page). Use the left button to create a regular/smaller report. The link speed uses the results from a lspci -vmm command and the debug file will have what it returned.

Changed Status to Retest

The DiskSpeed app simply reports what the dd command is outputting. If you're having issues reading all the drives at once, try removing half of the drives and then run a controller benchmark to see if you get the same results or better. If the same, remove the existing and hook the others back up. If better, then add a couple drives and do another benchmark. Rinse & repeat. See if there's a magic number or a given drive that's causing the issue.

@bonienl The VM GUI editor is hard coded to set the thread count to 1 if it detects an AMD processor in libvirt.php // detect if the processor is AMD, and if so, force single threaded $strCPUInfo = file_get_contents('/proc/cpuinfo'); if (strpos($strCPUInfo, 'AuthenticAMD') !== false) { $intCPUThreadsPerCore = 1; } This was due to AMD reporting no support for hyperthreadding in a VM. With UNRAID 6.8.1 RC1, hyperthreadding is supported with CPU passthrough as is (and CPU cache) if the CPU feature topoext is enabled. Previously, the CPU had to be forced to report as an EPYC to get it to support hyperthreadding. <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='6' threads='2'/> <cache mode='passthrough'/> <feature policy='require' name='topoext'/> </cpu> Microsoft's CoreInfo returns Coreinfo v3.31 - Dump information on system CPU and memory topology Copyright (C) 2008-2014 Mark Russinovich Sysinternals - www.sysinternals.com Logical to Physical Processor Map: **---------- Physical Processor 0 (Hyperthreaded) --**-------- Physical Processor 1 (Hyperthreaded) ----**------ Physical Processor 2 (Hyperthreaded) ------**---- Physical Processor 3 (Hyperthreaded) --------**-- Physical Processor 4 (Hyperthreaded) ----------** Physical Processor 5 (Hyperthreaded) Note that changes to the CPU layout may not be detected in the VM until the VM is rebooted from inside the VM itself (for example: Start > Power > Restart) Prior to 6.8.1 RC1, I could not get CPU-Z to run, it would always hang at the 10%/Processors on load. It still takes a bit but does return now.

CONFIRMED: On 6.8.1 RC1, I don't need the EPYC workaround to get hyperthreadding enabled but the TR does need the following CPU feature: <feature policy='require' name='topoext'/> I was also able to get CPU-Z to run with CPU passthrough and hyperthreading detected though it takes a bit at the 10%/Processors stage where in the past it apparently hung. I'm going to look to see if HT support has already been recommended for Threadripper CPUs and request it if not.

Thanks for mentioning "virtio", that reminded me to check for a new virtio-win ISO file. New one out last October, version 0.1.173.2

Ya know, you just invalidated 4 benchmark runs with 25 passes each run. At least you caught be before the next two runs! 😁 Might as well upgrade to 6.8.1 RC1 too. Though this would also mean a lot less edits in the XML every time I needed to use the GUI editor.

Can you install Microsoft's CoreInfo and see if it detects a hyperthreaded setup? I use this Batch file for a quick report since it's a command line utility and the screen will close after running Coreinfo.bat @echo off coreinfo.exe -ncs pause My Results with the EPYC hyperthreading workaround (weird numa mapping due to cross-numa node testing) Coreinfo v3.31 - Dump information on system CPU and memory topology Copyright (C) 2008-2014 Mark Russinovich Sysinternals - www.sysinternals.com Logical to Physical Processor Map: **---------- Physical Processor 0 (Hyperthreaded) --**-------- Physical Processor 1 (Hyperthreaded) ----**------ Physical Processor 2 (Hyperthreaded) ------**---- Physical Processor 3 (Hyperthreaded) --------**-- Physical Processor 4 (Hyperthreaded) ----------** Physical Processor 5 (Hyperthreaded) Logical Processor to Socket Map: ************ Socket 0 Logical Processor to NUMA Node Map: **---------- NUMA Node 0 --********** NUMA Node 1 Press any key to continue . . .

AMD Threadripper doesn't support hyperthreadding out of the box to VM's but it can be tricked into working with the EPYC workaround. As such, the code behind the VM Editor is hard-coded to set the CPU threads to 1.

I have not but AMD agrees with you on the latency issues having to utilize the Infinityfabric. They're not creating multiple numa nodes for TR3.

(Ref "on node": Numa 0 & 2 with direct access to PCI & RAM. "off node": Numa 1 or 3) Some interesting findings I had last night. I had three VM's running, Cam1 (hogging numa 0 & 2) had four Brios connected, each set to output the cam over NDI. Cam2 running OBS taking in a NDI feed from Cam1. CPU % steady. When I started OBS on Cam3 taking in a NDI feed from Cam1, Cam2's CPU utilization jumped. When I closed OBS on Cam3, the CPU % on Cam2 returned back. The effect could also be seen in reverse. Cam2 & Cam3 were running "off node" with accessing the memory. I just ran a benchmark and showed that memory latencies as well as read/write/copy times were negatively affected by around 50% if the memory access had to utilize the Infinityfabric of the TR2.

2990WX memory speeds are seriously impacted if the VM has to utilize the Infinityfabric. This is likely true for all TR2 models. Memory latency is 58.8% slower and the read/write/copy scores jump by 47% / 48.9% / 54% respectively.

1. I got slightly better CPU scores under i440fx SeaBIOS than Q35 OVMF. 2. Still fine tuning CPU placement. One live test I ran last night showed CPU utilization steady on Cam2 taking in a NDI video feed jumped when Cam3 also started taking in a NDI video feed. Cam1 is hogging the CPUs with direct access to the memory and I may spread it out so it takes a numa node on memory and one off so I can move Cam2 & Cam3 on the numa nodes with memory. Ran a test just now and memory latency is 58.8% slower when it has to cross the Infinityfabric and the read/write/copy scores jump by 47% / 48.9% / 54% respectively.

Cam1VM.xml

I saw a 0.73% drop with the additional hyper-v settings than what the GUI put in.

Test it without passing through any other devices. I have a MB that one of the LAN ports may not always survive a VM reboot if I was also passing through a graphics card - stopping passing through the card and it survived over & over. Just something to troubleshoot.

My guess is that the read speed of the faster drives were capped by the OS in order to maximize the output of the other slower drives. Gives better overall performance. You're only utilizing 2GB of the stated 7.8GB bandwidth so you may have a bottleneck elsewhere in the PCI chain.

Ah, I missed that it was under Process Lasso. My mind clicked in on the unraid VM editor.

You're the second person I've seen mention a NUMA box. I've never seen one, can you send a screen shot?

Hmm what work around? I pass it through as an EPYC so it sees the L3 cache right and it does threading correctly that way as well. Also posted video of with and without but feel its not as defined. Both times the machine was just idle/logged in I have this block in my XML <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>EPYC</model> <topology sockets='1' cores='4' threads='2'/> <cache level='3' mode='emulate'/> <feature policy='require' name='topoext'/> <feature policy='disable' name='monitor'/> <feature policy='require' name='hypervisor'/> <feature policy='disable' name='svm'/> <feature policy='disable' name='x2apic'/> </cpu> With this setup, likely driven mostly by the cpu mode/matach and model EPYC, CoreInfo shows the TR as hyperthreaded. Logical to Physical Processor Map: **------ Physical Processor 0 (Hyperthreaded) --**---- Physical Processor 1 (Hyperthreaded) ----**-- Physical Processor 2 (Hyperthreaded) ------** Physical Processor 3 (Hyperthreaded)

I use the Hyperthreading workaround in my VM's, CPUZ hangs when it scans the CPUs.

You're definitely going to have to enable the PCIe override to even have a prayer. I didn't do any benchmarks with it on & off because I was able to utilize my use case with it off. Your performance may be less with the override on. Please refer to the first post on this thread for PCie slot/memory to NUMA slot assignments as well as what controllers can be passed through to a VM and if it needs an override or not along with quirks I've found to date. This is true. Note that I haven't been able to find which USB controller will let the onboard USB2 ports pass through. Other notes: The Infinityfabric for internode communication is fast enough on TR2 that having off-node graphics card will cause a slight CPU usage bump but no meaningful difference in GPU scores. I haven't tested off-node memory speeds but I have two VM's where one is on node and one off so I can do that. I have a GeForce GT 1030 (single slot) for unraid to grab in slot #1 since the MB demands a GPU. I have a Quadro P2000 in slot #2 for my Main VM. I was able to pass through a USB3 PCIe card to a VM in slot #3 but none of the other slots worked with override=off. I was able to connect several Brio's and pass them through but I had the occasional video glitch with a 1080p@60 video feed which hinted that the data stream wasn't operating at a constant speed. This almost ruined my planned VM buildout until I discovered that I had no issues with passing through the onboard USB3 controllers that don't require an override. Not ideal because those USB3 ports only have one controller but it is what it is. I have cores CPU's 1-59 isolated and while I do see CPU 0 bump that doesn't seem to mach the VM's task manager, I haven't noticed any impact yet. I still need to try out @Jerky_san clock/hyperv tweaks. Here's my pinning layout. Cam 1 is running Livestream Studio and it exports multiple Brio's via NDI to the other VM's to upload to YouTube using OBS.

Adding for posterity that RC8 also updated the Aquantia 10G drivers (which is a good thing). Probably from the downgrade of the kernel. I don't have the network dropping out issues any longer. RC7 root@VM1:~# ethtool -i eth0 driver: atlantic version: 5.3.8-Unraid-kern firmware-version: 1.5.44 RC8 root@VM1:~# ethtool -i eth0 driver: atlantic version: 2.0.3.0-kern firmware-version: 3.1.44