Summary of current Threadripper (2950x) issues with VMs please

[AntaresUK]

Hi guys

 

I am considering upgrading from a xeon to a 2950x. Im sure a lot of other members are considering the same. Could someone be so kind as to catalogue any outstanding issues with threadripper that have not got a solution. I know there is a @SpaceInvaderOne video on Ryzen, but I think it would aid a lot of people with this kind of decision if the current state of threadripper and VM is documented.

 

Thanks for any assistance on this matter

[bastl]

With an up-to-date motherboard BIOS and newer kernel, kvm and qemu versions you can basically say, everything is working now. The early day issues with RAM compatibility, passthrough reset, IOMMU groupings and power stage issues are mostly fixed. I'am running an 1950x on an ASRock x399 Fatal1ty Gaming board, 64GB TrydentZ 3200MHz and a 1050ti and a 1080ti, both passed through. One Windows VM uses an dedicated M.2 and an extra SSD passed through as well as the onboard audio chip and the 1080ti. For me everything works fine.

 

There are a couple things you can tweak to improve the performance (UMA-NUMA memory settings, CPU model set to EPYC, emulator-thread IO-thread pinning), but on default everything should work. Something to note, there will be always some issues with passthrough, depending on the components you use, doesn't matter if Intel or AMD. Some people have issues with some specific M.2 NVMEs and others struggle to install newest drivers. AMD GPUs from the 5xx series lately have some issues with the up-to-date drivers, same as NVidia on some VM configurations giving you an error code 43. Vega cards are another thing to mention. Passthrough works, but if you restart the VM the card won't reset and you can't startup the VM again or the server becomes unstable and has to be restart to fix it.

 

The more complex a platform gets, the more features it provides the more you have to configure and tweak to get the best out of it. There is no "stick it all together" and everything is working at max performance and stability, neither on Intel nor on AMD side. End of 2017 I jumped onto the TR4 hype, knowing that most of the time it isn't a good idea to be an early adopter. Who knows what happens with the platform in the future? Will it pay off? Will there be support? I did my research first, ordered a RAM kit from the QVL, not the newest NVME on the market and put it all together. Et voila, 4x16GB 3200MHz quad channel working straight out of the box. Never had any issues with instability, couple people reported the early days on Rayzen/TR4. Early days I had to use the ACS override patch to get my GPUs separated in it's own groups and the onboard audio to be able to pass it through a VM. One or two kernel updates later and an BIOS update and this fix wasn't necessary anymore. Out of the box passthrough are working. 

 

Linux VM with GPU for usual web based stuff, media consumption etc and a Windows 10 VM for gaming. Dockers for my own Nextcloud, Unifi Controller, Duplicati and some scripts to backup regularly to an Synology NAS. Everything is working for me. I run this machine for almost 1 year now overclocked at 4GHz stable without one single issue in stability. I had a couple people testing the gaming VM on my machine and none of them noticed anything that they are playing in an virtual environment. 

 

I'am talking about one of the first gen boards and the first gen TR4 I'am using. I can guarantee you, there will be issues if you buy the newest from the newest. The 2950x shouldn't be an issue. It's basically the same as my chip with slightly higher clocks. You should be fine with that. But if you choose one of the newer boards you might see the same issues I had in the early days. 

 

 

[AntaresUK]

Brilliant, thats the kind of response I was after thanks @bastl

I have been looking at the specs 1950x vs 2950x and there doesnt seem to be much between them. The 2950x is higher clocked and has improved infinity fabric I believe but probably not worth the extra £300. Might see if I can find a 1950x on ebay

 

[bastl]

It all depends on your needs and on the money you wanna spent, right. The IPC from the 2950x is higher in single core and in multicore benchmarks. Memory latency is reduced to improved infinity fabric. Power consumption is optimized. All the usual platform improvements you'll always see. The question is, do you wanna pay more for this improvements? It's the same question as, should I wait for the next generation to upgrade? Upgrade if you need to  

[Jerky_san]

On my 2990wx I still get stuttering here and there sadly. Seems more noticeable when plex streams are going on in the non memory controller dies. Sound will get out of sync if you use onboard sound card while playing videos when running other VMs. I Have a an ASUS Zenith Board that doesn't like my ram much honestly. I had a Taichi x399 but that board wouldn't boot my LSI 9201 card into an OS. I am hoping the 3k series threadrippers with chiplet designs solve a lot of issues around memory access and such.

[jordanmw]

I've got the TaiChi x399 and a 1920x currently setup as a 4 player gaming computer- you could save a little money and go with that CPU.  If it is plenty for my setup, yours seems like an even lighter use case.  I just passed through disks directly for each vm for a data location, then put the main image on cache.  Then, just backup the image for the OS and all the data will stay safe.  I do that process nightly, and a restore of all 4 vms takes about 20 minutes.  I have 4 kiddos and each of them have their own workstation.  

[jordanmw]

On 3/18/2019 at 6:05 AM, AntaresUK said:

Brilliant, thats the kind of response I was after thanks @bastl

I have been looking at the specs 1950x vs 2950x and there doesnt seem to be much between them. The 2950x is higher clocked and has improved infinity fabric I believe but probably not worth the extra £300. Might see if I can find a 1950x on ebay

 

I just pulled the trigger on a 2950x upgrade from my 1920x for the extra cores and speed- might be looking to get rid of my 1920x this weekend if you are interested.

[AntaresUK]

2 hours ago, jordanmw said:

I just pulled the trigger on a 2950x upgrade from my 1920x for the extra cores and speed- might be looking to get rid of my 1920x this weekend if you are interested.

Saving up for a 2950x now...just pulled the plug on some g.skill trident z 64gb cl14 4 x 16gb. Once you have it installed I'd be interested in any tips you have for getting the most out of it @jordanmw Go AMD!

[testdasi]

On 3/20/2019 at 1:45 AM, Jerky_san said:

On my 2990wx I still get stuttering here and there sadly. Seems more noticeable when plex streams are going on in the non memory controller dies. Sound will get out of sync if you use onboard sound card while playing videos when running other VMs. I Have a an ASUS Zenith Board that doesn't like my ram much honestly. I had a Taichi x399 but that board wouldn't boot my LSI 9201 card into an OS. I am hoping the 3k series threadrippers with chiplet designs solve a lot of issues around memory access and such.

Interesting. How did you pass through the onboard soundcard to the VM? My IOMMU group has it together with a SATA controller. A quick google says it's the M.2 SATA controller and lstopo says my SATA drives are all NOT connected to it (naturally since I only use NVMe M.2). However, it still seems rather risky to vfio it to use it for pass through.

How did you do it?

 

With regards to Plex, I explicitly pin it to the dies without memory controller and have not had any stuttering. It only stutters when I run 4 simultaneous transcodes on top of Plex (so 1x4k HDR stream, 4x1080p streams) on the same set of cores - that is to be expected I guess. The other time it stutters was because the transcoding temp was on a failing HDD - switch it to SSD and it's gone.

Edited June 5, 2019 by testdasi

[bastl]

18 minutes ago, testdasi said:

How did you pass through the onboard soundcard to the VM?

I guess it depends on the motherboard manufacturer how they configure the IOMMU groups in the BIOS. On earlier BIOS revisions on my ASRock I had the same issue like you. Onboard sound was grouped with other devices (Sata, Network, USB). ACS Override on the first BIOS doesn't helped, on the second version I could split it into it's own group and with later BIOS revisions the ACS Override Patch wasn't necesary any more. Maybe check if Gigabyte released a newer version for your board.

[testdasi]

6 minutes ago, bastl said:

I guess it depends on the motherboard manufacturer how they configure the IOMMU groups in the BIOS. On earlier BIOS revisions on my ASRock I had the same issue like you. Onboard sound was grouped with other devices (Sata, Network, USB). ACS Override on the first BIOS doesn't helped, on the second version I could split it into it's own group and with later BIOS revisions the ACS Override Patch wasn't necesary any more. Maybe check if Gigabyte released a newer version for your board.

Already on latest BIOS. Same problem with previous BIOS.

[bastl]

@testdasi I guess you also tried the ACS multi-function option? Maybe?

[jbartlett]

On 3/19/2019 at 6:45 PM, Jerky_san said:

On my 2990wx I still get stuttering here and there sadly. Seems more noticeable when plex streams are going on in the non memory controller dies. Sound will get out of sync if you use onboard sound card while playing videos when running other VMs. I Have a an ASUS Zenith Board that doesn't like my ram much honestly. I had a Taichi x399 but that board wouldn't boot my LSI 9201 card into an OS. I am hoping the 3k series threadrippers with chiplet designs solve a lot of issues around memory access and such.

I fixed the stuttering sound issue by passing through a USB sound card though yesterday I saw a Spaceinvader One video on YouTube titled "The best way to install and setup a windows 10 vm Part 2 Hardware Passthrough" that shows how to correct what he calls "demon sounds". It's on my To Do list to try.

[jordanmw]

Just wanted to chime in- just changed from a 1920x to a 2950x and still having the same success I was before.  4 gaming machines with CPU/sound passed through that are functioning well with no stutters or jitters.  I have only one issue with my rig currently that wasn't there before I upgraded 2 of my GPUs.  The issue is explained here: 

 

[MrGreen1717]

On 6/5/2019 at 9:43 AM, jordanmw said:

I just pulled the trigger on a 2950x upgrade from my 1920x for the extra cores and speed- might be looking to get rid of my 1920x this weekend if you are interested.

you still selling that 1920x?