same PCI ID in multiple IOMMU groups [Solved]

[Marshalleq]

Yes, I have this exact issue.  All sorts of network weirdness actually.  But trying to stub all or one of my interfaces in a dual port card.  No dice.

 

Edit:  Actually I have found previously, I can't stub any device.  In cast that's helpful.  The NVIDIA card is available to be passed through, and works successfully, but that is automatic.  I CAN use the ata-device line to pass through my SATA card, but that's not a stub as far as I know.

 

Must be something to do with the motherboard BIOS - Asus X399-A / Threadripper 1950X

Edited May 7, 2019 by Marshalleq

[scubieman]

Did you disable the second port in flash/startup config?

[Marshalleq]

Hi thanks for the response - I simply added as below

 

kernel /bzimage
append isolcpus=12-15,28-31,xen-pciback.hide=(0b:00.0) initrd=/bzroot

 

So only one port - thought I only had to do one - will try two!

[Marshalleq]

Sadly, that makes no difference, both still show up in Unraid.

Edited May 7, 2019 by Marshalleq

[scubieman]

2 minutes ago, Marshalleq said:

Sadly, that makes no difference, both still show up in Unraid.

If you go to Network settings Does it show 2 or 1 port?

[scubieman]

I appended two.

 

Also I had to turn on "VFIO allow unsafe interrupts:"

 

Quote

default menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label Unraid OS
  menu default
  kernel /bzimage
  append xen-pciback.hide=(02:00.0)(02:00.1) vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot
label Unraid OS GUI Mode
  kernel /bzimage
  append xen-pciback.hide=(02:00.0)(02:00.1) vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot,/bzroot-gui
label Unraid OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label Unraid OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest

 

Edited May 7, 2019 by scubieman

[Marshalleq]

It shows three, each under their own interface heading.

[Marshalleq]

Will try the unsafe interupts, have heard about that, never seen anything suggesting that I should use it, until now!   Thanks for the example config, will try it.

[scubieman]

Explain NIC setup. You have 2 NIC cards with 2 ports each? 1 4-port card? 

[Marshalleq]

I have an onboard NIC and a PCI card with two ports in it.  I've been trying to track down what I suspected were network errors.  Installing net data dashboard seems to confirm this as there is quite a decent amount of packet loss.  I'd still like to passthrough a card though - to rule out Unraid drivers as it seems it's assigning the wrong driver to the intel board and I'm not sure I trust the onboard.

 

However disabling the onboard did not remove the card from Unraid either, (go figure).  Further Unraid insisted on sending traffic out it's gateway and kept reassigning the gateway to that card until I pushed a foreign subnet into it.  I've since removed all config altogether and am at where I'm at today.

 

The Ethernet interfaces are listed below:

 

[scubieman]

The Network cards should be in different IOMMU groups so you can isolate entire group...

 

Or you can try breaking the IOMMU groups down and isolate by that.

 

Not sure if this makes difference but I have integrated 4 port NIC. I isolated 2 of them. Unraid doesnt see them at all.

 

Then I configure VM to pick them up.

 

Edited May 7, 2019 by scubieman

[Marshalleq]

One would think that disabling it in the BIOS would be sufficient.  LOL.  I'm just ignoring it for now and trying to pass through the other cards.

 

Just tried the allowing unsafe interrupts - didn't seem to change anything - all three interfaces still show up in Unraid.

 

kernel /bzimage
append isolcpus=12-15,28-31,xen-pciback.hide=(0b:00.0)(0b:00.1),vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot

 

I think I did this right.  You didn't have the comma before vfio, but I'm pretty sure that doesn't matter, in fact I'd read it should be there, could try it I suppose.

Edited May 7, 2019 by Marshalleq

[scubieman]

4 minutes ago, Marshalleq said:

One would think that disabling it in the BIOS would be sufficient.  LOL.  I'm just ignoring it for now and trying to pass through the other cards.

 

Just tried the allowing unsafe interrupts - didn't seem to change anything - all three interfaces still show up in Unraid.

 

kernel /bzimage
append isolcpus=12-15,28-31,xen-pciback.hide=(0b:00.0)(0b:00.1),vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot

 

I think I did this right.

where did you get this from?

 

(0b:00.0)(0b:00.1)

 

Should be these.

 

Edited May 7, 2019 by scubieman

[scubieman]

Should be these.

 

[Marshalleq]

If you scroll up to one of my previous pictures, you can see them by IOMMU Group 18 and 19.  It is my understand that those are the correct ones - they looked similar.  An you can see it is the second field in that picture, just as you've highlighted.

[scubieman]

1 minute ago, Marshalleq said:

If you scroll up to one of my previous pictures, you can see them by IOMMU Group 18 and 19.  It is my understand that those are the correct ones - they looked similar.  An you can see it is the second field in that picture, just as you've highlighted.

For giggles maybe try 05:00.0 See which it disabled. Just hope it doesnt disable your main connection.

[Marshalleq]

At this point I may as well attach logs in case someone sees something - and have a look at them myself in the process.

obi-wan-diagnostics-20190507-0155.zip

[Marshalleq]

1 minute ago, scubieman said:

For giggles maybe try 05:00.0 See which it disabled. Just hope it doesnt disable your main connection.

I have got a screen plugged in so not a biggie.  Will try.  It helps having someone else to throw ideas at me!

[scubieman]

Not sure if you watched this yet, but it may help out in figuring out what one you need to disable. I'd like to assume you may be trying to disable wrong IOMMU group.

 

 

[Marshalleq]

I believe I've seen it, but it was early days.  This stuff worked quite well on my Ryzen 1800x.  It's when the threadripper came that I had issues with it.  I'll have another look.

 

Also, disabling that 05:00.0 did nothing that I can see.  Still have three NICS showing up in UNRAID.  Just going to have a poke around the logs now in case there's something syntax related.  Might try poking around the BIOS too, I seem to remember there was something in there.

[Marshalleq]

This seemed interesting in the logs - though removing it didn't help.  Nevertheless incorrect CPU range is an odd error to have and googling turns up nothing.

 

May  7 13:43:44 Obi-Wan kernel: Kernel command line: BOOT_IMAGE=/bzimage isolcpus=12-15,28-31,xen-pciback.hide=(0b:00.0)(0b:00.1),vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot
May  7 13:43:44 Obi-Wan kernel: Housekeeping: nohz_full= or isolcpus= incorrect CPU range

[Marshalleq]

Will start a new thread for this.

[scubieman]

Just now, Marshalleq said:

Will start a new thread for this.

How many cores you have?

[Marshalleq]

16 with 32 threads.

[scubieman]

2 minutes ago, Marshalleq said:

16 with 32 threads.

Could try removing the isolated CPU cores.

 

But it should still work