Do you some custom things in the go file on your flash drive in the config folder?

1 minute ago, bastl said:

Do you some custom things in the go file on your flash drive in the config folder?

disabled c6 with zenstates as mentioned by spaceinvader by one in his videos

 

root@Friday:~# cat /boot/config/go
#!/bin/bash
# Start the Management Utility
/usr/local/sbin/zenstates --c6-disable
/usr/local/sbin/emhttp &
 

 

I still don't get these custom edits. There are a couple TR users here in the forum and i never saw these used by someone. 🙄

 

Another thing you might can test. In my case passing through a Samsung 960 NVME i first had a couple issues with it where it showed not the performance it might have shown. I didn't tested any games back than or did any other benchmarks than read and write tests. The default driver loaded was the windows one. I had to install Samsungs Magician Tool and the driver from Samsung to fix that. The read and write speeds randomly dropped with the windows driver.

2 minutes ago, bastl said:

I still don't get these custom edits. There are a couple TR users here in the forum and i never saw these used by someone. 🙄

 

Another thing you might can test. In my case passing through a Samsung 960 NVME i first had a couple issues with it where it showed not the performance it might have shown. I didn't tested any games back than or did any other benchmarks than read and write tests. The default driver loaded was the windows one. I had to install Samsungs Magician Tool and the driver from Samsung to fix that. The read and write speeds randomly dropped with the windows driver.

 

I have the Samsung driver installed

 

There are alot of edits if you check SpaceinvaderOne's youtube or posts here.

I think i watched all of his Ryzen related videos. The specific EPYC part you are using and the feature policy settings are kinda new to me. Maybe @eschultz has any idea what can cause your problems. If i remeber it right he also has a TR system playing around with. 

Maybe it is worth it If I make some screenshots from BIOS settings?

 

Dunno if anyone else in here has that particular Board

The more information the better. You may also post your diagnostics for more help. 

alright will do some screenshots later on.

 

diagnostics attached 

friday-diagnostics-20181028-2133.zip

Edited October 28, 20187 yr by Nooke

22 hours ago, bastl said:

@Symon What cores do you have assigned to the VM and what motherboard are you using?

The whole die 2 (8-15/24-31). Although my graphic card is connected to die 1, I seem to get less stutters with using die 2. (I guess it has something to do with the ram)

I have a Asus Rog Zenith Extreme.

11 hours ago, Nooke said:

disabled c6 with zenstates as mentioned by spaceinvader by one in his videos

 

root@Friday:~# cat /boot/config/go
#!/bin/bash
# Start the Management Utility
/usr/local/sbin/zenstates --c6-disable
/usr/local/sbin/emhttp &
 

 

Is this also needed with Threadripper (thought its a Ryzen thing)? I never had any problems without..

11 hours ago, bastl said:

I still don't get these custom edits. There are a couple TR users here in the forum and i never saw these used by someone. 🙄

 

Another thing you might can test. In my case passing through a Samsung 960 NVME i first had a couple issues with it where it showed not the performance it might have shown. I didn't tested any games back than or did any other benchmarks than read and write tests. The default driver loaded was the windows one. I had to install Samsungs Magician Tool and the driver from Samsung to fix that. The read and write speeds randomly dropped with the windows driver.

I pass through an SSD for my games partition (Samsung 850 Pro). I just checked the Samsung magician tool and it doesn't recognize the drive. Any Idea how to fix that? The Bus of the disk is Sata.

1 hour ago, Symon said:

Is this also needed with Threadripper (thought its a Ryzen thing)? I never had any problems without..

TR had the same issues as the normal Ryzen chips. I never checked if the newer BIOS in combination with the newer Kernel maybe fixed that issue. Might have to have a look into that. 

 

56 minutes ago, Symon said:

I pass through an SSD for my games partition (Samsung 850 Pro). I just checked the Samsung magician tool and it doesn't recognize the drive. Any Idea how to fix that? The Bus of the disk is Sata.

My issue was NVME related. A passed through SSD works fine but isn't recognized by Samsung Magician either. It's attached as block device and uses the Virtio SCSI driver from RedHat.

 

1 hour ago, Symon said:

Although my graphic card is connected to die 1, I seem to get less stutters with using die 2. (I guess it has something to do with the ram)

 

I have tested a lot of combinations GPU in slot 1 Die0 or Die1 attached, GPU in slot 3 and Die0 or Die1 attached to a VM. I didn't noticed any big differences +-5fps no matter which memory configuration i tested with. UMA or NUMA didn't made any big differences for me. With "numactl -H" you can check which config you currently using. If it shows only 1 node you're in the default UMA mode. For memory bandwith intensive tasks it showed in some benchmarks thats the better mode. Better memory latency you gain in NUMA mode. 

 

In NUMA mode you can force unraid to only use the memory attached to a specific die. But for some reason it doesn't work for me. It's also reported by other users in the forum. My GPU in the first PCI slot is attached to the second die where also my NVME is attached to and the cores i use for a VM but as i said, 'strict' should only use memory from the node you set. It doesn't do it for me. I tested all the memory settings in the BIOS, nothing worked. Only if i set it to 'preferred' it uses memory of the node 1 + a couple MB from node 0 even if enough memory is available on node 1. I haven't found a fix for it yet. It looks like a couple people reporting that something might not work as expected. 

  <numatune>
    <memory mode='strict' nodeset='1'/>
  </numatune>

 

22 hours ago, bastl said:

I have tested a lot of combinations GPU in slot 1 Die0 or Die1 attached, GPU in slot 3 and Die0 or Die1 attached to a VM. I didn't noticed any big differences +-5fps no matter which memory configuration i tested with. UMA or NUMA didn't made any big differences for me. With "numactl -H" you can check which config you currently using. If it shows only 1 node you're in the default UMA mode. For memory bandwith intensive tasks it showed in some benchmarks thats the better mode. Better memory latency you gain in NUMA mode. 

 

In NUMA mode you can force unraid to only use the memory attached to a specific die. But for some reason it doesn't work for me. It's also reported by other users in the forum. My GPU in the first PCI slot is attached to the second die where also my NVME is attached to and the cores i use for a VM but as i said, 'strict' should only use memory from the node you set. It doesn't do it for me. I tested all the memory settings in the BIOS, nothing worked. Only if i set it to 'preferred' it uses memory of the node 1 + a couple MB from node 0 even if enough memory is available on node 1. I haven't found a fix for it yet. It looks like a couple people reporting that something might not work as expected. 

  <numatune>
    <memory mode='strict' nodeset='1'/>
  </numatune>

 

The whole testing just shows me how much I'm not undestanding yet ....

But thats also why I built this system to learn something new 😁

I still feel like there was a performance drop when upgrading from 6.5.X to 6.6.2 but I did not do enough performance test before such as I could verfiy that. It just feels like it from the feeling in games.

I'm using the default UMA mode.

How do you check what memory the VM is currently using?

3 minutes ago, Symon said:

How do you check what memory the VM is currently using?

numastat qemu

 

Some new information from my side.

 

I've found an issue but can't seem to fix it.

 

I checked the PCIe Speed of my GPU and my NMVe Drive.

Both running in PCIe Gen 2.

 

GPU

root@Friday:~# lspci -s 42:00.0 -vv | grep Lnk
                LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-

NVMe

root@Friday:~# lspci -s 41:00.0 -vv | grep Lnk
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-

 

 

I already tried fiddling around with pcie_aspm=off and using pcie_aspm.policy=performance instead.

Changing PCH State in Bios to force gen3. Seems nothing is working.

 

 

The GPU is plugged in the first PCIe Slot.

The NMVe drive is installed right below this PCIe Slot.

 

Is it possible that the Link is not working as PCIe Gen 3 cause both share the same PCIe Lanes?

 

Couldn't find any documentation regarding the M.2 port and PCIe port

 

 

 

UPDATE:

seems the BIOS on the MSI x399 SLI Plus is bugged.

As soon as I choose a specific CPU multiplier the PCIe Gen switches from 3.0 to 2.0

Even changing back to Auto not restoring normal behaviour.

I tested all kind of settings and could point out it that it has to be CPU multiplier and bkcl frequency.

 

CMOS clear is necessary to get PCIe 3.0 functionality back. Restoring in BIOS to default settings didn't work for me.

 

 

 

 

cheers
 

Edited November 3, 20187 yr by Nooke

I'll be honest with you guys. I got a 2990wx and I get this strange lag every once in a while. Sometimes its "fast" other times its dog slow. Its almost like other processes are using my pinned CPUs even though they aren't supposed to.

55 minutes ago, Jerky_san said:

Sometimes its "fast" other times its dog slow.

What is fast or slow? The server itself? A VM only? The server and the VM? Some Dockers?

Edited November 4, 20187 yr by bastl

12 minutes ago, bastl said:

What is fast or slow? The server itself? A VM only? The server and the VM? Some Dockers?

The gaming vm starts to get a bunch of hiccups and slow downs.

Which cores did you selected for the VM? Did you checked with lstopo which dies are directly connected to the RAM? Did you tweaked your gaming VM? What are you passing through to the VM? I can ask you tons of questions. It might be useful you provide a couple more informations if you need help. Btw there are a couple 2990 specific threads you should check first. Search the forum first and you will find a couple threads where people reported kinda similar behaviour "slow downs" or "slow startups" what it looks like are specific for the 16+ core Threadripper parts.

1 minute ago, bastl said:

Which cores did you selected for the VM? Did you checked with lstopo which dies are directly connected to the RAM? Did you tweaked your gaming VM? What are you passing through to the VM? I can ask you tons of questions. It might be useful you provide a couple more informations if you need help. Btw there are a couple 2990 specific threads you should check first. Search the forum first and you will find a couple threads where people reported kinda similar behaviour "slow downs" or "slow startups" what it looks like are specific for the 16+ core Threadripper parts.

I've been through every which way.. settled on the same cores as another user on here that went through all the numbers. It runs fast most of the time but sudden slow downs seem to happen.

1 hour ago, Jerky_san said:

Its almost like other processes are using my pinned CPUs

Btw pinning a core to the VM doesn't prevent unraid to use the core for other processes or VMs. Try to isolate the cores you wanna passthrough and restart the server. 

@bastl

I've seen that you have an ASRock x399 Board instead of MSI.

 

I guess you have a fixed CPU multiplier right?

Can you check via GPU-Z if your GPU is running at PCIe 3.0 ?

 

MSI provided a new BIOS update to my support request but the issue persists.

As soon as I set the CPU multiplier or BLCK from "auto" to somethign else the GPU and my NVMe SSD going back to PCIe 2.0

Even after changing back to "auto" or loading defaults via BIOS menu doesn't fix that.

I have to clear CMOS to get PCIe 3.0 back.

 

Just want to figure out if that is a regular x399 issue or only on my specific Motherboard.

 

 

cheers

Nooke

@Nooke I'am currently running on a fixed multi 4GHz with the 1080ti in the first slot using cores from the second die, where the first slot is connected to. As soon as i have load on the GPU, GPUZ shows PCIe x16 3.0 @x16 3.0. Without load PCIe x16 3.0 @x16 1.1. I tested slot 3 couple weeks ago which should also provide x16. With BIOS set to auto for PCIe lanes settings or however its called the card only runs at x4 on slot 3. NVME i didn't noticed any speed impacts. But one of my 3 NVME slots is wired to the chipset itself where also are 2 SATA ports are connected to. Chipset is connect with 4 lanes to the CPU. You might be using a slot which isn't directly connected to the CPU and it auto slows down.

 

Thanks for sharing the infos

 

I'm using PCIe and M.2 slots directly connected to the Die inside the VM. Anyways I tried with native Windows 10 and got the same issue.

 

So I suspect this is a Hardware or Software failure with the MSI X399 SLI Plus.

Hopefully the MSI support can help in this case - as of right now I've just got a new BIOS Version (BETA I guess) from them which didn't help.

 

 

cheers

Which manufacturer is your NVME? I use a 960 pro for passthrough and it had a firmware on it when shipped which was bugged and couldn't reach expected speeds. I had a couple hickups in Windows from time to time till i got a new FW for it.