Jump to content
jbartlett

ASUS ROG Zenith Extreme Alpha X399

65 posts in this topic Last Reply

Recommended Posts

Hey all. I have an issue. However, I have just read the first post. I will get back to you once I have made the changes to the BIOS. 


 

Execution error

internal error: qemu unexpectedly closed the monitor: 2019-06-14T16:42:54.367135Z qemu-system-x86_64: -device vfio-pci,host=0b:00.0,id=hostdev0,bus=pci.0,addr=0x5: vfio 0000:0b:00.0: failed to setup container for group 20: failed to set iommu for container: Operation not permitted

 

This was 1 VM I setup with 2 video card, GTX 1080 and a GTX 650TI.

 

I will keep you updated. 

 

Thank you

Share this post


Link to post
4 hours ago, authorleon said:

Hey all. I have an issue. However, I have just read the first post. I will get back to you once I have made the changes to the BIOS. 


 


Execution error

internal error: qemu unexpectedly closed the monitor: 2019-06-14T16:42:54.367135Z qemu-system-x86_64: -device vfio-pci,host=0b:00.0,id=hostdev0,bus=pci.0,addr=0x5: vfio 0000:0b:00.0: failed to setup container for group 20: failed to set iommu for container: Operation not permitted

 

This was 1 VM I setup with 2 video card, GTX 1080 and a GTX 650TI.

 

I will keep you updated. 

 

Thank you

Hi All. Okay, I did what John said on the first post. Thank you John. 

 

So, I have disabled ACS. 

 

The VM boots but I get not output from the GTX 650TI. 

 

Am I missing something ?

 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='1'>
  <name>Windows 10 - TEST2</name>
  <uuid>f9c5482f-3fc5-e50d-d81d-87ad181944af</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='32'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-3.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/f9c5482f-3fc5-e50d-d81d-87ad181944af_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='2' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10 - TEST2/vdisk1.img'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/W10X64.MULTi6.OCT2018.iso'/>
      <backingStore/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:0c:e3:8e'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-Windows 10 - TEST2/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x413c'/>
        <product id='0x2003'/>
        <address bus='9' device='2'/>
      </source>
      <alias name='hostdev2'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>

 

Thank You 

Share this post


Link to post
Posted (edited)

Did some numa node & infinity fabric benchmarking CPU's and a graphics card (GTX 980ti Classified). I ran 3D Mark Timespy with the CPU's on the same numa node as the card and off and what I found was interesting. The GPU scores were in the same range but the CPU scores dropped off node . 2.2% slower on Timespy Extreme 4K and 9.5% slower on the regular Timespy.

 

A Passmark CPU benchmark with no graphical aspects other than Windows itself scored 0.6% faster on node.

 

Each benchmark was repeated 5 times and the results averaged.

Edited by jbartlett
  • Like 1
  • Upvote 1

Share this post


Link to post
17 hours ago, authorleon said:

Hi All. Okay, I did what John said on the first post. Thank you John. 

 

So, I have disabled ACS. 

 

The VM boots but I get not output from the GTX 650TI. 

 

Am I missing something ?

 


<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='1'>
  <name>Windows 10 - TEST2</name>
  <uuid>f9c5482f-3fc5-e50d-d81d-87ad181944af</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='32'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-3.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/f9c5482f-3fc5-e50d-d81d-87ad181944af_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='2' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10 - TEST2/vdisk1.img'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/W10X64.MULTi6.OCT2018.iso'/>
      <backingStore/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:0c:e3:8e'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-Windows 10 - TEST2/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x413c'/>
        <product id='0x2003'/>
        <address bus='9' device='2'/>
      </source>
      <alias name='hostdev2'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>

 

Thank You 

Can anyone help me wit this please or point me in the right direction. 

 

Thank you

Share this post


Link to post

Sorry, I haven't done anything with passing multiple GPU's to a VM before.

Share this post


Link to post
1 hour ago, jbartlett said:

Sorry, I haven't done anything with passing multiple GPU's to a VM before.

 Hello John, I'm not passing multiple GPUs.

 

PCI1 - 1080 GTX is for UNraid GUI at present

 

PCI4 - GTX 650 TI Boost

 

1VM using the GTX650 Ti Boost.  Your box standard set up .  When turning on the VM. Nothing is outputted from the GTX 650

Share this post


Link to post
6 hours ago, jbartlett said:

Sorry, I haven't done anything with passing multiple GPU's to a VM before.

I switched the cards and now it is working. It does not like the 650 TI boost...

 

I will keep you all updated. 

 

Thank you

Share this post


Link to post

Not all video cards can be passed through. I have a GTX 1050 that doesn't like it.

Share this post


Link to post

I benchmarked a Win10 VM with PassMark's PerformanceTest assigned to the different Numa nodes on the 2990WX, separately and different combinations. The last column was truncated, it's all four numa nodes except CPU 15/47.

 

Each test was repeated five times and the results averaged.

 

641152592_2990WXNumaNodePassmarkCPUBenchmark.png.2a7384b28b91039b6c9d2ca970e5be6b.png

 

1169678436_2990WXNumaNodePassmarkMemoryBenchmark.png.349e454562f104248ac1915645fc690c.png

 

495364568_2990WXNumaNodePassmarkMemoryBenchmarkJitter.png.8a831da21f9d4a9f4712cfb43ca1d6e9.png

Share this post


Link to post
3 hours ago, jbartlett said:

I benchmarked a Win10 VM with PassMark's PerformanceTest assigned to the different Numa nodes on the 2990WX, separately and different combinations. The last column was truncated, it's all four numa nodes except CPU 15/47.

 

Each test was repeated five times and the results averaged.

When running the benchmark, did you also set numatune to force memory load to the right NUMA node too?

 

Your Node 0 vs Node 1-3 looks suspiciously like your VM is fully allocated to node 0 only.

 

For VM that spread across multiple nodes, you might even want to force it so that memory is loaded evenly across different nodes. In my case, I have had to go as far as running 2 dummy VMs to load up 2 nodes to the right amount so that my main VM splits almost 50-50 across 2 nodes (and then shutdown the dummy VMs - thank goodness for CA User Scripts).

Share this post


Link to post

I didn't do any memory placement testing, just loaded up one VM with 8 GB of RAM assigned to it. This strikes me as how the vast majority of people will launch their VM's letting KVM do the memory assignments.

Share this post


Link to post
53 minutes ago, jbartlett said:

I didn't do any memory placement testing, just loaded up one VM with 8 GB of RAM assigned to it. This strikes me as how the vast majority of people will launch their VM's letting KVM do the memory assignments.

Indeed your tests make it clear that these WX Threadripper need some intervention for best performance. Unraid KVM is not smart enough to deal with it.

Share this post


Link to post
1 hour ago, testdasi said:

Unraid KVM is not smart enough

Even Windows isn't smart enough to handle the multi-Numa-node design. I remember watching Wendell in one of his videos showing that W10 in a VM on TR4 with extra Numa configuration runs way faster in multiple software and close to bare metal linux performance, than bare metal windows on the same system. I hope for the Ryzen 3000 series Microsoft is doin their homework.

Share this post


Link to post

Hi guys,
First of all, very good documentation work you have done here. Very appreciated!


I am also planning to build an unraid system, where as one build option is the Threadripper 2920 or 50 with the discussed mainboard. This option should serve as an Unraid server but also contain a VM used as my normal PC. This VM should not only be capable for gaming but also not be too high maintenance due to whatever reason (e.g. Windows updates). The alternative to this option would be two separate builds which would be more expensive.

 

Now I have some questions relating to the Threadripper build if you would be so kind to answer:

1.      From your experience, would I, as a newbie to Unraid and advanced bios settings, be able to set up such a machine and keep it working? (as mentioned e.g. due to software updates or pass through stability issues)

2.      Threadripper has its handicaps with gaming in comparison to other modern CPU‘s. Do you think, adding another difficulty level (running games on the Windows VM instead on the base platform) will result in material performance losses?

3.      As far as I understood you were not able to pass through a GPU in PCIe 1 to a VM. Thus PCIe 1 is just usable for an Unraid GPU or another PCIe card other than a passed through GPU. Is this still correct or would it change with an AMD GPU?

4.      In connection to question 3. Is it correct that it is not possible to put the GPU dedicated to Unraid somewhere else than in the very valuable PCIe 1 slot (x16 + space)? Ideal would be the PCIe 4 (x4) but that does not work I guess.

 

Thank you very much in advance!

Share this post


Link to post

@Georg

1. OS updates are outside the scope of unraid as you would still face them if you built a bare metal system. In the case of Windows 10, you can "defer" updates for up to three months with Windows Pro.

2. Intel has handicaps too. You can expect to lose a few FPS running a gaming rig as a VM but not anywhere near you'd be able to notice assuming your video card is sufficiently powerful. I have not noticed any video performance drops if the video card & pinned CPU's aren't on the same NUMA node but I did see a CPU hit.

3. I haven't check for new BIOS updates in the past month or so but there was no option to select which PCIe slot the BIOS used for it's primary video card. I would believe that if you could change it, you would still experience the same thing but with whatever slot is given. Note: This is a known issue with nVidia cards, I have not tried any AMD video cards. The problem is that the nVidia driver can detect that the card was in an active state when it took over and then is coded to not load (Error 43)

4. You still need a video card in PCIe #1 for the BIOS to bind to. Omitting a video card in that slot would just cause the BIOS to bind to the next available slot with a video card.

 

 

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.