bastl

@Nooke I have 4x16GB. 2x16 each die and if i have nothing run except of 1 VM with 16GB please explain me why I Only reason could be Unraid claiming more than 16GB from node1. My GPU and the NVME controller i pass through to the VM are connected to node1 which cores and RAM I assigned to the the VM and still it uses RAM from node0 ^^

The 16GB VM i tried to use RAM and cores from node0 only and it shows the same. It always uses a couple megs from the other node. Don't know why. Something is off. 64GB in total and not even running any Docker or other stuff it happens. Can you check one of your VMs with "preferred" instead of "interleaved" how it behaves? @DZMM Also i don't know why you using "interleaved" anyways. This forces Qemu to spread the RAM across all nodes. The point for pinning down the RAM to a specific node/die is to reduce latency and improve the performance of a VM.

numastat qemu Shows current running VMs and their use of RAM. As i said earlier, you can tell the VM with the numatune option which node to use the memory from but it looks like it doesn't. There are always a couble megs used from the other node. First VM is set to use 8GB and 4 cores from node0, second VM is set to use 16GB and 14 cores from node1. <memory mode='strict' nodeset='1'/> Strict should limit the VM only to a specific node. But it doesn't. "prefered" or "interleaved" doesn't change anything for me.

With that tweaks the VM shows the correct L1-L3 caches of the CPU in CPUz. It feels a bit smoother, might be to the lower latency i see in AIDA same as @Jerky_san reported earlier.

@Jerky_san Whatever i set the topology to, 8cores 1 thread or 4 cores 2 threads benchmarks showing the same performance. Even windows shows 8 virtual CPUs no matter what i set. There is no difference with these setting.

The core topology doesn't matter.

It doesn't matter performance wise if set it to cores='4' threads='1' or cores='2' threads='2'. For me in my tests it always shows the same performance. I did a couple tests on the current 6.6.5 with different benchmarks (Cinebench, Aida, CPUz) and games (GTA, BF1, Rust) and all scores are nearly the same. The issue with the L1, L2 and L3 cache thats reported wrong to the VM, I don't know if this a Unraid specific thing that @limetech can fix or has to be implemented in the Linux kernel, Libvirt or Qemu.

@Chamzamzoo Every first gen Threadripper has 2 dies. Max cores per die are 8. The smallest TR4 (1900x) has only 4 cores per die enabled. 0-3 + HT are on one die and 4-7 + HT on the second die. The increase of memory bandwith you see in AIDA is to the fact that you're using both dies each with it's own memory controller with 2 channels each. So with 2 Ryzen dies you get quad channel. There is no actual "best setting" all depends on your needs. If you need the memory bandwith for your applications, use both dies. If you need the lower latency use only cores from one die and set you memory in your BIOS to NUMA or channel in most cases. There are still some quirks with KVM and the memory setting. You can "strictly" set your VM to use only memory from a specific node but a couple people reported that a bit of memory is still used from the other die which inreases the latency again. Also tweaking your xml to present the CPU as an Epyc to the VM can improve the performance a bit. In this case the actual CPU cache is presented to the VM in the correct way. Unraid itself for some reason changes the L1 L2 and L3 cache that the VM sees with it's standard settings for CPU model passthrough. Also noted, you only passing through the HT cores to the VM. The usual way is to passthrough the main core + it's hyperthread. I can't really tell if this makes any big differences in performance. I never tested it like you have set it up.

I bet you have this setting to. All the x399 BIOSes I saw so far have tons of settings more or less good structures. If you search long enough you might find it 😁 Should be pretty similar. Both have 2 dies each with 8 cores.

If you're running in NUMA mode you gain improvement in memory bandwith cause you're accessing both memory controllers at the same time, 1 each die. But you will have slightly higher latency in memory access. All i tested for me and normal use of a VM (browsing, office stuff, gaming) i could't see any big differences. Switching the GPU in another slot or using a NVME in another one so the device isn't connected to the die i passthrough directly for me shows the same. No big noticable differences. I don't really care about +-5fps in games as long as everything runs smooth and it does. Read and write speeds of the NVME also kinda the same. You might see some hickups if the device is connected to the other die which is under heavy load. For me it never happens that my dockers or the Unraid itself gets to the point to use all the CPU ressources. Might be different with a plex container or some other transcoding dockers pushing the CPU to the limit. It all depends on how you're using your rig. I don't really know where you can find the BIOS setting on a MSI board. On AsRock you can find it under CBS / DF common options. Default for memory interleaving setting for me is Auto. Channel switches it to Numa. Besides from setting my XMP profile for the RAM i only enabled the hardware virtualization support (SVM Mode and SR-IOU) and enabled IOMMU. Not really sure how the last one exactly is called. Something with IOMMU. No matter how you set your memory you should always use cores/threads from the same die for a single VM and don't mix up the cores to reduce the latency. On my 1950x it looks like this: The die1 is only for my main VM with GPU and NVME passthrough. Cores 8 and 24 are the emulatorpins and the rest is isolated and only used by a Win10 VM. On die0 I have all my dockers running and a couple VMs which i use from time to time. I am playing around currently with the emulatorpin setting, but i can't see any difference as without it. Isolating the emu pins, not isolating, no pinning at all for me i can't really see a difference. Using the emulatorpin on the other die i didn't checked yet. Worth a try.

@naik You only can run 1 VM at the time using the GPU. You can configure multiple VMs using the GPU no problem, put you can't run them at the same time.

Isolating cores is to prevent unraid to use them for backround tasks or for docker. So for best performance in a VM you have to isolate the cores you wanna pin to your VM. Keep in mind core 0 is always used by unraid and you can't isolate this one. Also keep in mind always isolate the core + its HT. In my case i have a 1950x. 2 dies each with 8cores. I isolated all 8 cores from the second die which I use for my main Win10 VM. The rest of the cores are used by Unraid, Docker and some VMs i use from time to time.

@Jerky_san I wouldn't trust CPUz in the first place inside a VM as well as other software. Cinebench never reads the correct core clock nore does any tool show me the right Vcore used. CPUz also shows me the core clock always on its max speed where in the backround on unraid you can see it's running on the idle speeds as it should be. I guess thats the way Qemu/KVM showing/emulating the CPU to the guest OS.

The averrage read speed for that disk is 119MB/s and write is 111MB/s and the mixed IO speed is around 70MB/s. If you only transferring big files i guess you will see higher speeds and yours are basically the speeds for the mixed IO. So it looks normal. http://hdd.userbenchmark.com/WD-Red-2TB-2012/Rating/1789

I did a couple tests. I have a EVGA 1050ti in my first PCIE slot and a 1080ti in my third slot. The 1080ti is mainly used for a gaming VM and the 1050ti for Linux VMs i use. For the Linux VMs i don't have to use a extra VBIOS. Only choosing Q35 and it works out of the box. The 1080ti i can passthrough to a Win10 VM without any modifications. Create the VM (i440fx; OVMF) with VNC, install the OS, add the GPU+Audio later, install the driver, remove VNC done. For the card in the first slot if i want that to passthrough to a Win10 VM i have to do the same as before excepts that i have to passthrough the VBIOS of the card to get it working without the ERROR 43. I got my VBIOS from TechpowerUP for the EVGA 1050ti and modified it like SpaceInvader described it in his video. Passthrough the modified VBIOS remove VNC, done and it works. But if VNC is enabled i'm still getting the Error. So make sure after installinng your OS and enabling RDP or installing Teamviewer to remotly accessing your VM, that you remove VNC. Another thing to mention it makes no difference for me turning the Hyper-V option on or off or manual edit the xml, it works.

@Squid only a typo? or is there another UnRaid update around the corner?

@Jerky_san what board are you using?

@testdasi Yesterday I reduced my Gaming VM to 6 cores + 6 threads on node 1 with all cores isolated and did a couple of benchmarks without running anything else on that die. Than i switched all my dockers and other VMs from node 0 to node 1, isolated the last 6 out of 8 cores and their threads on node 0 from unraid and switched over the gaming VM to node 0 where still my 1080ti should be attached to (if lstopo is correct). I don't flipped the cards around yet, because for now i don't need any vbios to pass through. The performance is basically the same except from small stutters/hickups and sound bugs every 30-40 seconds. Every game (BF5, Farcry5, DayZ, Superposition + Heaven benchmark) i tested gave me nearly the same performance as on node 1 + that weird stuttering. Don't exactly know why. I never had that issue if i isolate the second die and use these cores only. This gets me back again to my initiall idea, that maybe the BIOS is reporting the core pairings wrong to the OS. Why should i get stutters if the GPU is connected to the cores which are used directly and no stutters across the infitity fabric. Weird! I didn't retested in NUMA mode. I did that before and as long as i don't mix up the dies for one VM it makes no difference in gaming performance. Using the UMA mode showed me in my tests that i get a higher memory bandwith with no real performance losts.

@stormense you might missed something in the video.

Thanks @SpaceInvaderOne the symlink fixed it for me Why the hell is the first PCIE slot connected to the second die and the third slot to the first die? In the first slot i have a 1050ti which is used by a Linux VM which uses some cores from the first die. The 1080ti on the 3rd slot is mainly used for a gaming VM and using all cores (8-15;24-31 isolated) on the second die. I wish i could flip a switch in the BIOS to reverse that. I guess there is no chance for such an option, right?

Same error on RC4

Ok, now it gets interesting. I already watched almost all videos from Wendell, but thanks for mentioning it here for people stumbling across this thread. @tjb_altf4 I might overlooked something by doin all my tests and the presented core pairings are alright. I assumed that the better memory performance depends on the cores and from which die they are. By switching between the options auto, die, channel, socket and none in the BIOS under AMD CBS settings, I should have already noticed that as soon as I limit a VM to only 1 die I get the memory bandwith from this specific memory controller. I basically cut the bandwith in half from quad channel (both dies) to dual channel. Makes perfectly sense. How could i miss that? If you need the memory bandwith for your applications, the UMA mode is the way to go. For me i have to set it to Auto, Socket or Die for the memory to get interleaved over all 4 channels and the CPU gets reported as only 1 node. By choosing the option Channel (Numa mode) I basically limit the memory access to the 2 channels from the specific die. The latency in this case should be reduced because you removed the hop to the other die. Option None will limit it to single channel memory and cuts the bandwith even further as shown in the pictures above. I'am actually not sure whats the difference between Auto, Die and Socket are. They all show similar results in the tests. And it should be also mentioned that it looks like Cinebench is more memory bandwith related as most people are reporting. Wendell mentioned in that video by using the lstopo to check which PCIE slots are directly connected to which die. Is there a way to check this without lstopo, which isn't available on Unraid? Right now my 1080ti is placed on the third PCIE slot x16 (1st slot 1050ti x16, second slot empty x8) and I'am not sure if it's directly attached to the correct die in my gaming VM. Maybe there is something already implemented in Unraid for listing the topology in a way lstopo did. Any ideas? Edit: Another thing i should have checked earlier are the behaviour of the clock speeds. Damn i feel so stupid right now. watch grep \"cpu MHz\" /proc/cpuinfo Checking this command during the tests would have shown that as soon as i choose cores from both dies for a VM the clocks on all cores ramp up. If i assign the core paires Unraid gives me, only one die ramps up to full speed and the other stays on idle clocks. 🙄

As reported earlier for the 1950x on a ASRock Fatal1ty x399 Gaming Pro something is reported differently. Looks like the same happened for Jcloud on his Asus Board. Currently I'am on the 6.6 RC2. I couldn't realy find a BIOS setting to change the behaviour how the dies are reported to the OS. It always been reported as 1 node. Edit: @testdasi It looks like your RAM usage for your VMs isn't optimized either. If I understand the shown scheme right, for example your VM with PID 33117 uses half the RAM from 2 different nodes which have a memory controller build in. In case u have more than 1 die assigned to the VM thats ok, but if you use lets say 4 cores from 1 die, it should use the 4GB RAM from the same node and not from another node.

Force shutdown is exactly what he is complaining about. It will be forced after some time if the clean shutdown isn't working. Except from a sheduled shutdown inside the VM i have no idea. I quickly tested it and it doesn't work for me either. Sorry