IOMMU and PCIe lanes correlation (?)


5 posts in this topic Last Reply

Recommended Posts

Hi,
I'll try to explain as simple as i can.

My server runs on a Gigabyte B450 Aorus M board with a Ryzen 2400G CPU (12 pcie lanes).
I disabled in bios all the things i don't need (serial, the embedded GPU and so on), ACS enabled (in bios).

PCIexpress slots are populated by 1 Gfx card and a SATA3 card (4 sata ports), so i still have one PCIe slot free.

Problem is, whatever i insert in the free pcie slot (i tried some 'spare' cards i have, like a Firewire one and another sata3 controller) it always ends up in one of the 'common' IOMMU groups (likely, the one handled by the chipset), so i can't passthrough what i plug there to any VM (actually, the only thing fully 'isolated' i can safely passthroug is the GPU).

Total number or IOMMU groups is 10 (from 0 to 9).

The question is: if i upgrade the CPU with one with more PCIe lanes (eg: the 3700x has 24 PCIe lanes) is there any chance i'll get more IOMMU groups (and maybe get the card i plug in in an isolated one)?

More in dept:
I'm running a VM with Windows 10 and an audio plugins server (AudioGridder), wich basically streams back to the client (my main computer) the audio streams generated by the hosted plugins. It works, but packet losses are too frequent to be accettable/usable. Isolating the cores used by the VM leads to no differences, as stopping everything else (dockers, vms).
o i guess the problem relies to some overhead introduced in the  Virtio (or Virtio-net) bridging (i tried both).
Even tried to update the virtio drivers to no avail.
If i boot the sever directly into windows (the installation is on a dedicated SSD), bypassing Unraid, it works perfectly.
So, my idea was to buy a PCIe gigabit card and fit it in the free PCIe slot to pass it throug the VM, bypassing the (supposed) virtio overhead.
But, being the situation what it is, i can't.

For reference, below is my System Devices situation (with the 'target' pci-e slot not populated; if i put anything there it ends in IOMMU group 8.

Thanks to anyone who can give me an idea.
image.thumb.png.0fe20e59809ae5a428db54bdcc7144ad.png


 

Link to post

You can also try one of the ACS override options to try and break apart the IOMMU groups

 

As an aside, you *may* have problems with using the Marvel controller on your drives.

Link to post
9 hours ago, Squid said:

You can also try one of the ACS override options to try and break apart the IOMMU groups

As an aside, you *may* have problems with using the Marvel controller on your drives.

 

Thanks for the suggestion but ACS is already active (via bios) - the effect is the very same as activating it in the VM options in Unraid.

C.

Link to post
14 hours ago, dhstsw said:

The question is: if i upgrade the CPU with one with more PCIe lanes (eg: the 3700x has 24 PCIe lanes) is there any chance i'll get more IOMMU groups (and maybe get the card i plug in in an isolated one)?

Nope.

 

In AMD platform, it relate PCIe come from CPU or Chipset, if come from CPU then it should natively in standalone group. So you need those motherboard with divide x16 to two x8 feature.

As @squid said, you need try software ACS setting even BIOS have/haven't enable ACS.

Link to post
Posted (edited)
45 minutes ago, Vr2Io said:

Nope.

 

In AMD platform, it relate PCIe come from CPU or Chipset, if come from CPU then it should natively in standalone group. So you need those motherboard with divide x16 to two x8 feature.

As @squid said, you need try software ACS setting even BIOS have/haven't enable ACS.


Thanks for the answer, but now i'm more confused than ever.
I found this reddit post with a guy with the same motherboard i have (and a CPU without embedded GPU, so i guess it's one of the 24 pcie lanes ones) wich shows
20 IOMMU groups:


Anyway, i'll try ACS in unraid a report back.

Thanks.

EDIT: even with ACS enabled on Unraid, no joy.

Edited by dhstsw
Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.