only pcie devices in CPU slot showing up, not on the chipset lanes (B550) [SOLVED]

[Milos]

Hi there, first post here so if I did some format wrong please say!

 

I am new to unraid and am using a 3900X on a gigabyte B550 mobo, I have succesfully passed through my 3070 to a linux fedora vm (haven't had much luck with windows yet but that's another problem)

 

I also have a 1030 I want to pass through, but any pcie device plugged in any other slot then the main pcie 4x16 slot from the CPU (all others are chipset) don't show up, I have already tried all three different pcie cards (3070, 1030 and some random network card)

 

I am slowly suspecting this might be a motherboard BIOS thing, otherwise I am in the dark

 

[SOLVED]

so I had multiple problems, I will list them here with the solutions:

 

PCIE devices not showing up in any IOMMU group in system devices

-cheaper/consumer motherboards/CPU's have pcie slots on a bus, but only one of the devices on that bus can be used. in my case my second m.2 slot was on a bus with the 4x pcie slot, after removing the m.2 drive it showed up in an IOMMU group

 

PCIE devices in one IOMMU group

-use ACS override under settings>vm manager>advanced view>ACS override

 

3070 still not giving display output in windows (but in linux it did)

-in my case I had a 1030 (slated for linux) that I temporarely used for windows to install nvidia 3070 drivers, after that I could just switch back to the 3070 in unraid and it worked

 

--please post it if this thread helped you out, it would make my day!--

Edited May 22, 2021 by Milos

[SimonF]

1 hour ago, Milos said:

Hi there, first post here so if I did some format wrong please say!

 

I am new to unraid and am using a 3900X on a gigabyte B550 mobo, I have succesfully passed through my 3070 to a linux fedora vm (haven't had much luck with windows yet but that's another problem)

 

I also have a 1030 I want to pass through, but any pcie device plugged in any other slot then the main pcie 4x16 slot from the CPU (all others are chipset) don't show up, I have already tried all three different pcie cards (3070, 1030 and some random network card)

 

I am slowly suspecting this might be a motherboard BIOS thing, otherwise I am in the dark

Which exact model do you have any other devices suuch as nvme

[Milos]

9 minutes ago, SimonF said:

Which exact model do you have any other devices suuch as nvme

mobo: gigabyte b550 pro

nvme storage: 970 evo 500g and another samsung nvme 128g drive

ram: 32gb 3600mhz corsair LPX ram

CPU: AMD 3900x

[SimonF]

13 minutes ago, Milos said:

mobo: gigabyte b550 pro

nvme storage: 970 evo 500g and another samsung nvme 128g drive

ram: 32gb 3600mhz corsair LPX ram

CPU: AMD 3900x

 

The slots share bandwidth, I suspect that is why is doesn't work. 

 

1 x PCI Express x16 slot (PCIEX16), integrated in the CPU:
3rd Generation AMD Ryzen™ processors support PCIe 4.0 x16 mode
New Generation AMD Ryzen™ with Radeon™ Graphics processors support PCIe 3.0 x16 mode
* For optimum performance, if only one PCI Express graphics card is to be installed, be sure to install it in the PCIEX16 slot.
1 x PCI Express x16 slot (PCIEX4), integrated in the Chipset:
Supporting PCIe 3.0 x4 mode
* The M2B_SB connector shares bandwidth with the PCIEX4 slot. The PCIEX4 slot will become unavailable when an SSD is installed in the M2B_SB connectors.
1 x PCI Express x16 slot (PCIEX2), integrated in the Chipset:
Supporting PCIe 3.0 x2 mode
* The PCIEX2 slot shares bandwidth with the SATA3 4, 5 connectors. The PCIEX2 slot will become unavailable when a device is installed in the SATA3 4 or SATA3 5 connector.
2 x PCI Express x1 slots

 

[Milos]

does this mean I have to buy a different motherboard (x570) or is there any way to use this slot:

* For optimum performance, if only one PCI Express graphics card is to be installed, be sure to install it in the PCIEX16 slot.

1 x PCI Express x16 slot (PCIEX4), integrated in the Chipset:

[Vr2Io]

Key point is

 

slot will become unavailable when an SSD is installed

[SimonF]

If you have a device plugged into M2B_SB PCIEX4 Is disabled.

 

So you would need to move stuff around or look for a different mb.

[Milos]

thanks everyone, I got the 1030 to show up, only it now shows up in IOMMU group 0, together with all other chipset devices and when trying to launch a VM with it I get this error:

 

internal error: qemu unexpectedly closed the monitor: 2021-05-22T12:17:02.533336Z qemu-system-x86_64: -device pcie-pci-bridge,id=pci.7,bus=pci.1,addr=0x0: Bus 'pci.1' not found

[SimonF]

You may need to try diffierent acs override options to see if you can get into its iommu group.

[Milos]

9 minutes ago, SimonF said:

You may need to try diffierent acs override options to see if you can get into its iommu group.

how do I do that, I've heard of the term but do I do that in bios or unraid?

[SimonF]

Goto settings VM its one of the options.

 

There are 3 options

 

PCIe ACS override allows various hardware components to expose themselves as isolated devices. Typically it is sufficient to isolate Downstream ports. A hardware component may need the setting Multi-function or Both to further isolate

[Milos]

thanks! I can now boot linux fedora from the 1030 or 3070 (awesome!) but I am still having problems with windows, I can boot windows from the 1030 but not from the 3070, should I try a different vBIOS maybe? I am now just using space invaders script

[SimonF]

14 minutes ago, Milos said:

thanks! I can now boot linux fedora from the 1030 or 3070 (awesome!) but I am still having problems with windows, I can boot windows from the 1030 but not from the 3070, should I try a different vBIOS maybe? I am now just using space invaders script

Which are you using for Machine Type and BIOS.

 

Q35 OVMF,

Q35 Seabios

I440fx OVMF

i400fx SeaBios.

 

Do you have the latest Nvidia drivers install in the WIN10 machine?

Edited May 22, 2021 by SimonF

[Milos]

i440fx-5.1, just tried Q35-5.1 (both also with 5.0) OVMF and SeaBios and both don't show an image. I will install the drivers through vnc now to check if that works

 

edit

when trying to switch back from Q35 or i440fx to the opposite, I get this error. forcing me to go back to the option first selected:

The PCI controller with index='0' must be model='pcie-root' for this machine type, but model='pci-root' was found instead

Edited May 22, 2021 by Milos

[ghost82]

52 minutes ago, Milos said:

The PCI controller with index='0' must be model='pcie-root' for this machine type, but model='pci-root' was found instead

That is because i440fx and Q35 have different xml templates.

Fastest option is to create from scratch the vm and point to the same (v)disk.

But take into account that ovmf boots uefi, so if you installed windows without the efi partition (seabios) it will probably not boot, you need to reinstall (?)

I would bet for Q35+ovmf+drivers

Edited May 22, 2021 by ghost82