Ryzen - Build & Progress with GPU & NVMe pass through


Recommended Posts

Ok so new to unRaid but really wanted to give it a go for an all in one media solution... I read of all the Ryzen issues but still couldn't stop myself from investing in Ryzen even though i knew it was going to be a headache! but so far made some good progress using a lot of guides and forums in this community and others and thought I would give back a little as information and benchmarks on this stuff is still pretty sparse (unless I just don't know where to look :) )

 

Also if any of you more experienced guys and gals have any recommendations on changes\ what else I should be doing please let me know!

 

Ok so current build is

 

Case: NZXT H440 
CPU: AMD Ryzen 5 1600 @ 3.6 
Motherboard: Gigabyte Gaming 5 x370

BIOS Version F7
RAM: Ripjaws IV 16GB 3000 (4x8GB) DDR4 @2933 XMP profile

GPU: Radeon HD 6450 

GPU 2: Gigabyte Windforce GTX 970 / Sapphire VaporX 280X
HDD's 3xHGST 8TB, 4xWD Red 3tb, 1x 120GB SSD cache
SSD: Samsung 960 EVO NVMe M.2 250GB
PSU: Corsair RM850x

 

OS: Unraid 6.4.0-rc7a

VM OS: Windows 10 home x64

 

current progress is actually pretty positive here are the benchmarks I have achieved on with the 970 GTX passed through, NTP off, Emulated CPU, NVMe Passthrough. I have yet to complete the Bare metal benchmarks but looking at other 970 benches out there these don't seem far out.

 

Cinebench 15

 

cinebench-npt-off-nvidia-allcores-3.6.thumb.PNG.ef3bf53d908c05dcd5b776f4bd2e0078.PNG

 

Firestrike (normal)

 

firestrike-npt-off-nvidia-allcores-3.6.thumb.PNG.cf682dd88f0cf1ded61ac820368488fb.PNG

 

Timespy

 

timespy-npt-off-nvidia-allcores-3.6.thumb.PNG.9aede21ca26597b56b575b9afaa0b2af.PNG

 

Shadow of Mordor (1920x1080 Ultra settings)

 

shadowofmordor-npt-off-nvidia-allcore-3.6.thumb.PNG.8b5888e4dd9fe4264343290b27c8717e.PNG

 

Crystal Disk Mark

 

crystaldiskmark-npt-off-nvidia-allcores-3.6.PNG.e1a0cd76f45f506e549c9a0be165463e.PNG

 

 

Edited by Tyranian
Link to comment

With NPT on with the 970 GTX CPU performance went through the floor (around 2000) but GPU performance was good (still above 12000) in Firestrike however, the cinebench CPU results are still over 900.

 

With NPT off the majority of benchmarks work well however, Kodi stutters occasionally on 4k video playback and the system seems a little unresponsive at times in general use, with NPT on this is not an issue and playback was flawless and general usage feels snappy but the benchmarks tank...

 

Another thing of note is SMT doesn't not seem to be supported in the VM. I have found a bug has already been logged here that may be related  https://bugs.launchpad.net/qemu/+bug/1703506

 

My experience with the AMD Radeon 280x was a little different  . 

 

(will update more with more time)

Edited by Tyranian
Link to comment
  • 2 weeks later...

Ok so progress so far is positive but its still not perfect,

 

I now have fast responsive machine most of the time with the above benchmarks but it still suffers occasionally from micro stutter I am hoping to try and resolve\reduce this with CPU isolation and dedicating the cpu's to the VM but currently have an issue with my bootloader causing the VM not to load when cpu's are isolated.

 

My settings are as follows

 

Bios

CPU is clocked to a fixed 3.6 ghz as dynamic boost doesn't seem to work currently 

 

Boot 

 

append kvm-amd.npt=0 kvm-amd.nested=0 kvm.ignore_msrs=1 initrd=/bzroot 

 

VM 

 

NVME & 970 GTX passed through - all cores passed through for testing (will reduce to 6-8) 8Gb memory

 

<cpu mode='host-passthrough' check='none'>
    .....
    <feature policy='disable' name='smep'/>
 </cpu>

 

I can provide the VM XML if that helps anyone.

 

 

 

 

Link to comment

Heres my current XML as requested i have lowered the core count recently is the only change from the benchmark results above

 

<domain type='kvm'>
  <name>WindowsMedia</name>
  <uuid>452eed65-1dfc-9329-7212-62bde7d85385</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='5'/>
    <vcpupin vcpu='2' cpuset='6'/>
    <vcpupin vcpu='3' cpuset='7'/>
    <vcpupin vcpu='4' cpuset='8'/>
    <vcpupin vcpu='5' cpuset='9'/>
    <vcpupin vcpu='6' cpuset='10'/>
    <vcpupin vcpu='7' cpuset='11'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.9'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/452eed65-1dfc-9329-7212-62bde7d85385_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='8' threads='1'/>
    <feature policy='disable' name='smep'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/isos/spaces_win_clover.img'/>
      <target dev='hdc' bus='sata'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <controller type='usb' index='0' model='nec-xhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:a4:6d:f2'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc52b'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc531'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x0b05'/>
        <product id='0x17cb'/>
      </source>
      <address type='usb' bus='0' port='3'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </memballoon>
  </devices>
</domain>
 

Link to comment
  • 2 weeks later...
On 9/5/2017 at 4:15 PM, Tyranian said:

Ok so progress so far is positive but its still not perfect,

 

I now have fast responsive machine most of the time with the above benchmarks but it still suffers occasionally from micro stutter I am hoping to try and resolve\reduce this with CPU isolation and dedicating the cpu's to the VM but currently have an issue with my bootloader causing the VM not to load when cpu's are isolated.

 

My settings are as follows

 

Bios

CPU is clocked to a fixed 3.6 ghz as dynamic boost doesn't seem to work currently 

 

Boot 

 

append kvm-amd.npt=0 kvm-amd.nested=0 kvm.ignore_msrs=1 initrd=/bzroot 

 

VM 

 

NVME & 970 GTX passed through - all cores passed through for testing (will reduce to 6-8) 8Gb memory

 

<cpu mode='host-passthrough' check='none'>
    .....
    <feature policy='disable' name='smep'/>
 </cpu>

 

I can provide the VM XML if that helps anyone.

 

 

 

 

 

Did you have any further luck with the micro-stuttering?  I'm on a Threadripper machine, so the progress in this department definitely interests me.T

Link to comment

This page has some tips:

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

 

"Using huge pages for guest and bigger huge page size(e.g. 1 GB) could reduce periodical micro-freezes of whole VM introduced by disabled NPT.

If periodical stuttering still occurs try removing smep feature from vCPU..."

 

Nonetheless, I still haven't gotten a GPU to pass through yet. :(  Thanks for the advice above.  Will try ASAP. 

Link to comment
  • 4 weeks later...

Hi all, I'm wondering if dimes007 or Tyranian have any new updates? I'm really interested to see how your testing/tweaking has gone with your Ryzen builds? I recently just swapped out my Xeon components for new a new Ryzen 7 1700 build, but I'm running into severe micro stutter issues with video playback. Benchmarks and gaming seem fine though.


I've read about the NPT bug associated with AMD CPUs and KVM... however it seems like you guys have come up with some tweaks to mitigate that issue?


Thanks!

  • Like 1
Link to comment

Hi,

 

I have not been online for a while so apologies for not responding sooner. I actually just use NPT now that removes micro stuttering and kodi works perfectly, I won't get the same gaming performance, but I don't actually need too in this build it as it's for media pc / dockers / storage.

 

I have no stutters at all on video playback what are you wanting to use your VM for Swiguy?

 

Also think there is some good news on the whole subject here...

 

https://www.phoronix.com/scan.php?page=news_item&px=AMD-NPT-GPU-Pass-Through

 

 

this is what the suggested fix is I believe

 

By just changing the line in `init_vmcb` that reads:
>>>
>>>    save->g_pat = svm->vcpu.arch.pat;
>>>
>>> To:
>>>
>>>    save->g_pat = 0x0606060606060606;


 

Edited by Tyranian
Link to comment
10 minutes ago, Tyranian said:

just tested and works so much better now! thanks for applying the patch so quickly. quick tests show very close performance to the NPT off results in my first post

 

I have this same board with a 1600x. Any issues passing thru a GPU or sound? Also why use a emulated cpu?

Edited by david279
Link to comment

no issues passing through GPU or sound use both from the graphics card, the only thing I do have is on TV I have to remove HDMI cable and reinsert it when I restart the VM dont have this issue with my monitor though....

 

I was messing around with both emulated and host passthrough CPU and can't remember why I left on emulated I'm guessing because it either gave better performance or I tested it last that way... I will try again with host passthrough when I get a chance.

Link to comment
On 10/28/2017 at 1:16 PM, Tyranian said:

no issues passing through GPU or sound use both from the graphics card, the only thing I do have is on TV I have to remove HDMI cable and reinsert it when I restart the VM dont have this issue with my monitor though....

 

I was messing around with both emulated and host passthrough CPU and can't remember why I left on emulated I'm guessing because it either gave better performance or I tested it last that way... I will try again with host passthrough when I get a chance.

 

 

Curious,  do you have CEC-ON on the TV?  Does turning it off fix this?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.