trx40 - 3960x setup questions

luca2

By luca2
in Hardware

Recommended Posts

luca2 Enthusiast

luca2

Posted
Posted

Hi,

I would appreciate if you owners of a similar setup can tell me you experiences with unraid so far.

I am interested in the following:

1) best iommu grouping out of the box for pass through...pls recommend which motherboard would you buy now

2) any problems still with linux kernel lower than 5, I am thinking of still sticking to latest unraid stable version

3) motherboard that accepts pciexpress bifurcation (not only for raid but also for running 2xgpus@x8_4.0), so that I can put 2 cards in 1 slot

4) any other issues I must be aware of (cpu-core tuning, ....)

I considering making a move to invest in such a platform to run 3 vm's with fully dedicated gpu's + 1/2 more to run via VNC. Also some dockers. And also running some nvme raid to be used as disk pool for the vm's.

X570 offers only limited pciexpress lanes I cannot use it for this setup. I doubt x670 will soon come out and even then will add more lines.

Since it is a big investment for me I would like to make sure in a certain degree that it will work before I pull the trigger.

Hope to hear your thoughts.

Rgds

 

 

Sent from my NX569J using Tapatalk

 

 

Link to comment
  • 9 months later...
  • 7 months later...
ddk-input Noob

ddk-input

Posted
Posted (edited)

I have AMD Ryzen 3960x , the settings is a bit tricky,

3960x To topology all cores 24x2=48 threads:

 

# Unraid 6.10.0-rc2

# Windows 11 3960x topology :  

<cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='24' cores='1' threads='2'/>  # threads = dies X 2 = 48 
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>

 

# macOS 3960x topology: 

<cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='24' dies='1' cores='1' threads='2'/>  # threads = sockets X 2 = 48 
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>

 

# Machine type tricky, last thing I expected to make pain boot failed: 

<type arch='x86_64' machine='pc-q35-4.2'>hvm</type> macOS tested ok

<type arch='x86_64' machine='pc-q35-6.1'>hvm</type> macOS tested failed, Windows 11 ok

 

# GPU RX 6900 XT:

# Windows ok HDMI audio ok, macOS ok HDMI audio failed, 'bus' can't be changed.

# All GPU instences must be passthough and must be added to xml otherwise = failed 

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x4f' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x4f' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x4f' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x4f' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>

 

..................................

 

# macOS args: 

<qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='isa-applesmc,osk=YOUR OSK HERE'/>
    <qemu:arg value='-smbios'/>
    <qemu:arg value='type=2'/>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check'/>
  </qemu:commandline>
</domain>

 

Edited by ddk-input
Link to comment
ghost82 Proficient

ghost82

Posted
Posted
8 hours ago, ddk-input said: 
<type arch='x86_64' machine='pc-q35-6.1'>hvm</type> macOS tested failed

This can be solved by adding this to custom qemu args: 

    <qemu:commandline>
      <qemu:arg value='-global'/>
      <qemu:arg value='ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'/>
    </qemu:commandline>

Mac os fails without this on machine type >=6.0 because in these versions acpi tables change and mac os doesn't like them.

 

8 hours ago, ddk-input said:

# GPU RX 6900 XT:

# macOS ok HDMI audio failed

I have a 6900 xt too (sapphire nitro+ se), hdmi audio works, but not dp audio:

https://github.com/acidanthera/bugtracker/issues/1924

 

Quote

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x4f' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x4f' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x4f' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x4f' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>

 

Multifunction is wrong, change to this: 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x4f' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x4f' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x4f' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x4f' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x3'/>
    </hostdev>

 

and I think your hdmi audio will start to work if you have lilu/weg/applealc

  • Like 1
Link to comment
ddk-input Noob

ddk-input

Posted
Posted
9 hours ago, ghost82 said:

This can be solved by adding this to custom qemu args: 

    <qemu:commandline>
      <qemu:arg value='-global'/>
      <qemu:arg value='ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'/>
    </qemu:commandline>

Mac os fails without this on machine type >=6.0 because in these versions acpi tables change and mac os doesn't like them.

 

I have a 6900 xt too (sapphire nitro+ se), hdmi audio works, but not dp audio:

https://github.com/acidanthera/bugtracker/issues/1924

 

 

Multifunction is wrong, change to this: 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x4f' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x4f' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x4f' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x4f' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x3'/>
    </hostdev>

 

and I think your hdmi audio will start to work if you have lilu/weg/applealc

Yes it works like charm 

pc-q35-6.1  ok

Multifunction ok

DisplayPort Audio works even on Monterey .

Thanks very much

 

 

Screen Shot 2022-02-27 at 7.10.11 PM.png

Link to comment
ghost82 Proficient

ghost82

Posted
Posted
9 minutes ago, ddk-input said:

DisplayPort Audio works even on Monterey

Still not sure why dp audio doesn't work for the sapphire nitro+ se version, even forcing the reference card vbios doesn't work (not a big issue since I'm using onboard hdef audio).

This is not a problem for reference cards, that you seems to have.

Link to comment
ddk-input Noob

ddk-input

Posted
Posted (edited)
37 minutes ago, ghost82 said:

Still not sure why dp audio doesn't work for the sapphire nitro+ se version, even forcing the reference card vbios doesn't work (not a big issue since I'm using onboard hdef audio).

This is not a problem for reference cards, that you seems to have.

I think it should work also on your sapphire nitro+

Try use USB3 (qemu XHCI) instead of USB2

and try my EFI attached it not the same as Macinabox

I don't use vbios at all.

EFI.zip

Edited by ddk-input
  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing