testdasi

Some more updates: 1. Updated to F12e BIOS and found out it changes the CPU core numbering!!!. Previously it was 0 + 1 numbering scheme. F12e changed it to 0 + 32 numbering scheme - if I can trust numactl. numactl --hardware available: 4 nodes (0-3) node 0 cpus: 0 1 2 3 4 5 6 7 32 33 34 35 36 37 38 39 node 0 size: 48268 MB node 0 free: 570 MB node 1 cpus: 16 17 18 19 20 21 22 23 48 49 50 51 52 53 54 55 node 1 size: 0 MB node 1 free: 0 MB node 2 cpus: 8 9 10 11 12 13 14 15 40 41 42 43 44 45 46 47 node 2 size: 48355 MB node 2 free: 352 MB node 3 cpus: 24 25 26 27 28 29 30 31 56 57 58 59 60 61 62 63 node 3 size: 0 MB node 3 free: 0 MB node distances: node 0 1 2 3 0: 10 16 16 16 1: 16 10 16 16 2: 16 16 10 16 3: 16 16 16 10 2. On the bright side, it gave me the motivation to reoptimise my core assignment and VM/docker usage. Shamelessly forked binhex's privoxy VPN and consol's VNC Web Browser to make some changes to make those dockers work better for my very niche purposes. That helps kill off 3 VMs. 3. Bought a M.2 -> PCIe adapter to see if I can move the PM983 to the vacant bottom PCIe slot (so I can add more M.2 drives) and then found out that slot is in the same group as the middle PCIe slot and the bottom right M.2 slot and the LAN ports. #facepalm. So have to revert and wasted 2 hours of my life.

2 thumbs up from me Maester Squid. Just installed and will reboot as soon as this render is done.

Updated from... wait for it... 6.5.3 with no issue! Glad to see the GUI whiners are still whining while I resolved my previously inexplicable lag problem with 6.6.0+ on my own and now get to quietly enjoy the new GUI. #smugface

No magic mate. Just a lot of trial and error. And the zen acceptance that some stuff is impossible to do (e.g. passing through the middle PCIe 2.0 slot to a VM due to IOMMU group needing ACS multifunction override which lags on unRAID 6.6.0+).

This PM983 just runs hot. I tested both upper (under the GPU) and lower slots and the lower one is actually hotter by 3-4 degrees. My hypothesis is that the GPU fan creates air flow in just the right direction which helps the SSD dissipate more heat than it receives from the GPU. That is a cool idea! Will see if I can get something similar on this side of the pond.

Some long overdue updates: As nobody has noticed, I have been whining a lot about being unable to update to unRAID above 6.5.3 due to terrible inexplicable lags to my main workstation VM which renders it (no pun intended) useless. How terrible? Think running Windows 10 on a Pentium MMX, if that's even possible. So I spent a lot of time on and off to scientifically eliminate all possible causes and eventually arrived at the only possibility, which is ACS multifunction override. So another month spent on rejigging stuff to eliminate the need for it and voila, I just managed to update to 6.7.0 any my workstation VM is now lag free. Along the way: Discovered that the bottom right M.2 slot (the 2280 size), for whatever the F reason, is in the same IOMMU group as (a) both wired LAN ports, (b) wireless LAN, (c) SATA controller and (d) the middle PCIe slot. Hence, it practically cannot be passed through via the PCIe method (need ACS multifunction override which lags - see above). My Toshiba 3TB HDD used for temp space died, likely due to being overworked. Writing TBs weekly will kill a HDD just as much as a SSD (probably even more so considering the 4K-ness of it). Made (a series of) mistakes rejigging stuff and long story short, accidentally ran a pre-clear on my SM951 to about 10%. Promptly have reallocated sectors and unrecoverable errors. So I now decided to run the SM951 to the ground (i.e. use it for temp space). It's an AHCI PCIe M.2 (i.e. basically a glorified SATA controller before NVMe was a thing - I was an early adopter of the M.2 PCIe SSD you see) so it's not too bad. Added a Samsung PM983 3.84TB NVMe SSD. This is a 22110 form factor and it runs hot - on average 10 degrees hotter than the 970 EVO. Both shows up with identical ID, probably because they use the same controller, so I reckon the extra heat is due to the extra capacity. My workstation VM now only runs on PCIe-passed-through NVMe (well except the boot vdisk). The Crucial MX300 2TB is now cache (it just had its first reallocated sector a few weeks ago). The Samsung 850 EVO 2TB is used as a NAS disk (basically an overflow for the workstation VM).

Yes, running MacOS VM and remote into it using NoMachine. Performance-wise, it's ok for general uses (I only assign 4 cores + GT 710), with a bit of lag when watching Youtube. It was more a fun experience than actual need for me.

Interesting. Was the xml in the post part of your unraid VM xml? I may try it when I have some time. Wonder if I can actually have the best of both worlds by using Process Lasso to restrict games to the same node while having my VM cross NUMA nodes for things that need more cores than low latency.

Sorry for late response. Have been in over my head with work. I have Plex and didn't have any of your problem. What unraid version do u use? Did you pin the VM to the node directly connected to the GPU? I have voice distortion on my HDMI audio output + general lag and stuttering if the node directly connected to the GPU is under super heavy load, which Plex certainly will do, even when the VM cores are not shared. That's why I always leave at least 1 core free, which seems to have solved the issue.

Below are my core assignments for the main workstation VM. I tested a lot of configs and this is the best one. The idea is to divide the CPU into 4 NUMA nodes, each node has 2 CCX, each CCX has 4 pairs of HT logical cores. So you can see for each CCX, I assign 3 non-linked logical cores to my workstation VM. Based on my testing, you want to spread the core assignments across as few nodes as possible but as evenly across as many CCX as possible and always leave core 0 not used by anything. So for instance, if you assign 6 cores to a VM, best config is to assign 3 to each CCX with both CCX on the same numa node. The side effect of that is an odd number of cores on a node, at least based on my workload, performs worse than having 1 fewer core assigned but spread evenly. You also might want to watch SpaceInvaderOne video on how to identify which PCIe slot is assigned to which numa node + how your cores are displayed. Mine shows 0 + 1 as a HT pair but I remember someone on the forum reported Asus mobo reports 0 + 32 as a pair. For gaming VM, you want to assign only the cores linked to your PCIe slot which has the GPU. That will minimise latency when playing (although I don't see any diff b/w gaming VM and my workstation VM, but then I use Process Lasso (to pin cores WITHIN a VM for a certain process) so that may be why). And everything turbo to 3.8GHz automatically (all cores) and drop to about 1.7 GHz when idle. <cputune> <vcpupin vcpu='0' cpuset='3'/> <vcpupin vcpu='1' cpuset='5'/> <vcpupin vcpu='2' cpuset='7'/> <vcpupin vcpu='3' cpuset='11'/> <vcpupin vcpu='4' cpuset='13'/> <vcpupin vcpu='5' cpuset='15'/> <vcpupin vcpu='6' cpuset='19'/> <vcpupin vcpu='7' cpuset='21'/> <vcpupin vcpu='8' cpuset='23'/> <vcpupin vcpu='9' cpuset='27'/> <vcpupin vcpu='10' cpuset='29'/> <vcpupin vcpu='11' cpuset='31'/> <vcpupin vcpu='12' cpuset='35'/> <vcpupin vcpu='13' cpuset='37'/> <vcpupin vcpu='14' cpuset='39'/> <vcpupin vcpu='15' cpuset='43'/> <vcpupin vcpu='16' cpuset='45'/> <vcpupin vcpu='17' cpuset='47'/> <vcpupin vcpu='18' cpuset='51'/> <vcpupin vcpu='19' cpuset='53'/> <vcpupin vcpu='20' cpuset='55'/> <vcpupin vcpu='21' cpuset='59'/> <vcpupin vcpu='22' cpuset='61'/> <vcpupin vcpu='23' cpuset='63'/> <emulatorpin cpuset='9,25'/> </cputune> No problem at all with GPU pass through. It depends on motherboard and GPU combi. My combi works well enough, except for the fact that Gigabyte mobo doesn't like unRAID v6.6+. My VMs take up to 10 minutes to boot and once booted, it runs ridiculously slow. So I downgraded to 5.5.3 and expect to stay there for years to come, or at least until my current 5-year upgrade cycle ends. Nobody seems to know why. Apparently other users reported no problem (e.g. with ASRock X399 Taichi) so I do recommend not to buy Gigabyte mobo. The 2950X is basically 2 Ryzen CPU glued together so you need to make sure to passthrough the node connected directly to your GPU PCIe slot for best performance. Alternatively use Process Lasso to limit the cores used by your games (or Adobe Lightroom <-- does not like more than 6 cores) within the VM can use similar effect.

Your description of what you did is pretty confusing. Can you described, step-by-step what you did.

That means upgrading to 6.6.0 was a red herring. There is a deeper issue with your server. As I originally asked and @bonienl reiterated, please boot in safe mode (non GUI) to rule out any plugin issue.

That's a good first step. Perhaps the acpid spam was due to the GUI having problems. Maybe try reboot in non-GUI to see if the hang goes away.

@Can0nfan: Perhaps try booting in safe mode and see if you still have the same issue? It cut out the possibility of misconfigured plugins. Your log is (still) flooded with acpid events every 2 seconds but no longer because of csrf_token.

According to ASRock spec, the bluetooth module is part of the Intel Wifi card so you will need to pass through the Wifi Card (it's a PCIe device) and see how it goes from there. It's probably in the same IOMMU group as the Intel Wired LAN so I'm pretty sure ACS Overide is needed.

After additional tuning (and BIOS updates) I have managed to make my 24-core VM (all odd numbers except 1, 9, 17, 25, 33, 41, 49, 57) to come pretty dang close to barebone 32-core non-SMT (F10 BIOS). As previous benchmarks, % is above barebone 32-core non-SMT so lower is better. Emulator core pinned on 9, 25, 41, 57, NUMA optimisation: 20% Emulator core pinned on 9, 25, with NUMA optimisation: 8% Emulator core pinned on 9, 25, with NUMA optimisation, F11e BIOS: 2% 😁 unRAID doesn't come with numatune and numad to optimise RAM for NUMA design but it's possible to work around. I created an MSDOS "blocker" VM with core pinned on NUMA node 0 and just enough allocated RAM to reduce my node 0 free RAM down to roughly half of what my actual VM needs. Then I start the main VM, resulting in an almost perfect 50-50 split between node 0 and node 2.

@Can0nfan: you will want to raise a separate bug report with details and especially need to attach diagnostics. I noticed you are posting your issues under various unrelated bugs (no, docker GUI shows more than 100% has nothing to do with VM cant mount volumes or pinning cores forcing a hard lock up). Doing that pretty much guarantee Limetech will not fix it because there isn't much details to work on.

Love F11e BIOS (AGESA 1.1.0.1a). All-core Turbo is now 3.8GHz! ~# cat /proc/cpuinfo | grep "MHz" cpu MHz : 3838.883 cpu MHz : 3842.082 cpu MHz : 3839.437 cpu MHz : 3838.451 cpu MHz : 3841.782 cpu MHz : 3842.523 cpu MHz : 3842.181 cpu MHz : 3835.253 cpu MHz : 3843.495 cpu MHz : 3834.948 cpu MHz : 3840.813 cpu MHz : 3841.923 cpu MHz : 3841.034 cpu MHz : 3841.805 cpu MHz : 3835.424 cpu MHz : 3841.688 cpu MHz : 3833.602 cpu MHz : 3842.614 cpu MHz : 3842.236 cpu MHz : 3842.521 cpu MHz : 3843.175 cpu MHz : 3842.522 cpu MHz : 3841.415 cpu MHz : 3842.517 cpu MHz : 3831.458 cpu MHz : 3842.367 cpu MHz : 3842.650 cpu MHz : 3842.180 cpu MHz : 3842.218 cpu MHz : 3841.945 cpu MHz : 3842.148 cpu MHz : 3840.602 cpu MHz : 3832.557 cpu MHz : 3842.336 cpu MHz : 3842.060 cpu MHz : 3840.882 cpu MHz : 3841.774 cpu MHz : 3840.777 cpu MHz : 3842.270 cpu MHz : 3842.064 cpu MHz : 3842.156 cpu MHz : 3835.171 cpu MHz : 3841.963 cpu MHz : 3840.519 cpu MHz : 3839.358 cpu MHz : 3833.257 cpu MHz : 3830.856 cpu MHz : 3840.741 cpu MHz : 3834.879 cpu MHz : 3842.435 cpu MHz : 3841.519 cpu MHz : 3840.938 cpu MHz : 3842.043 cpu MHz : 3840.830 cpu MHz : 3841.720 cpu MHz : 3837.862 cpu MHz : 3841.364 cpu MHz : 3840.644 cpu MHz : 3824.251 cpu MHz : 3840.582 cpu MHz : 3842.038 cpu MHz : 3840.441 cpu MHz : 3841.091 cpu MHz : 3840.340

I can vouch for the Noctua. It performs about the same level as a 360 rad all-in-one cooler. One small thing to note is that you may have to raise the fan to clear the RAM (especially the slots nearest to the socket), which makes the height a bit taller than spec. Will be curious what your benchmark figures is. I'm already seeing higher all-core turbo. ~# cat /proc/cpuinfo | grep "MHz" cpu MHz : 3838.883 cpu MHz : 3842.082 cpu MHz : 3839.437 cpu MHz : 3838.451 cpu MHz : 3841.782 cpu MHz : 3842.523 cpu MHz : 3842.181 cpu MHz : 3835.253 cpu MHz : 3843.495 cpu MHz : 3834.948 cpu MHz : 3840.813 cpu MHz : 3841.923 cpu MHz : 3841.034 cpu MHz : 3841.805 cpu MHz : 3835.424 cpu MHz : 3841.688 cpu MHz : 3833.602 cpu MHz : 3842.614 cpu MHz : 3842.236 cpu MHz : 3842.521 cpu MHz : 3843.175 cpu MHz : 3842.522 cpu MHz : 3841.415 cpu MHz : 3842.517 cpu MHz : 3831.458 cpu MHz : 3842.367 cpu MHz : 3842.650 cpu MHz : 3842.180 cpu MHz : 3842.218 cpu MHz : 3841.945 cpu MHz : 3842.148 cpu MHz : 3840.602 cpu MHz : 3832.557 cpu MHz : 3842.336 cpu MHz : 3842.060 cpu MHz : 3840.882 cpu MHz : 3841.774 cpu MHz : 3840.777 cpu MHz : 3842.270 cpu MHz : 3842.064 cpu MHz : 3842.156 cpu MHz : 3835.171 cpu MHz : 3841.963 cpu MHz : 3840.519 cpu MHz : 3839.358 cpu MHz : 3833.257 cpu MHz : 3830.856 cpu MHz : 3840.741 cpu MHz : 3834.879 cpu MHz : 3842.435 cpu MHz : 3841.519 cpu MHz : 3840.938 cpu MHz : 3842.043 cpu MHz : 3840.830 cpu MHz : 3841.720 cpu MHz : 3837.862 cpu MHz : 3841.364 cpu MHz : 3840.644 cpu MHz : 3824.251 cpu MHz : 3840.582 cpu MHz : 3842.038 cpu MHz : 3840.441 cpu MHz : 3841.091 cpu MHz : 3840.340

I would hypothesize that your motherboard reports the core incorrectly as a "display problem". i.e. underlying it's still 0 + 1 = 1 pair. Hence when you assign all the odd cores, you effectively running your VM in non-SMT mode (e.g. 16 non-SMT = 16 physical cores > 16 SMT = 8 physical cores). I made the above educated guess based on the fact that I see the exact same result i.e. assigning odd cores to VM gives me better performance. I think we'll need to have the right expectation. This is not a magical cure. Assigning all cores in the same die would only improve latency (assuming memory allocation is also to the same numa node). The processing power of a core (+ its SMT sister) doesn't change. If your workload scales well with multiple cores (e.g. transcoding), having more cores spread across multiple dies will almost always help more than having fewer (physical) cores trying to ring-fence the process within the same node. If your workload doesn't scale well and/or is bottle-necked by a pass-through device (e.g. gaming) then you will benefit from better consistency of performance (e.g. less stuttering i.e. more consistency fps) by ring-fencing the node with the connected device. What we lacked previously is the ability to make educated tuning for the 2nd use case.

Is there any harm to use these high values for relatively smaller arrays (e.g. let's say 5-disk 30TB)?

Great vid! I was super doped realising that my first slot is connected to node 2. That means I still can leave unRAID to do unRAID things in node 0 like it prefers and have my workstation VM isolated on node 2. Now if only I can force it to only use the RAM connected to node 2. You can always flip the cards and pass them through via vbios method. For gaming VM, it's better to lock everything to the same node (although you are running in UMA not NUMA so not sure how much benefit it's gonna get). How do you that please? It's always a "hope and pray" situation with me to get RAM assigned correctly. Now that I isolated the entire node 2 for my VM, it would allocate 99% of my RAM to node 2, which is good (enough) - but ONLY after a clean reboot. The longer I wait after the reboot, the more it allocates to node 0 (sometimes it would go nut and allocate 100% to node 0, even though there's nothing else running). Don't know how to do it without rebooting.

Please can you upload screenshot of your organizr docker template (in Advanced mode, remember to click "Show more settings" to show where your appdata is) screenshot of your docker appdata folder.

Then you didn't do it right. Did you delete the appdata folder AND the container? Check your docker set up to see if you have changed the config to some other paths (or perhaps deleted the wrong folder). Deleting appdata and container is the IT equivalent of burning one's letters. There's no way there's anything left, unless it hasn't been burnt.

@rinseaid reported same problem with 2950X so it probably is not due to core count. More likely AGESA since those with problems use Gigabyte mobo which is still on 1.1.0.0 (instead of the newer 1.1.0.1A). @gridrunner given problem is also reported on 2950X, perhaps it's better to change bug title to 2nd gen Threadripper, isn't it?